Dynamically presenting augmented reality content generator

Abstract

The technique requests a group identifier (ID) from an extended application programming interface (API) based on the item identification indicator. The technique requests a set of augmented reality (AR) content generator IDs from a shopping AR content generator service based on an item identification indicator using an extended API. The technique receives a set of AR content generator IDs based on a mapping of item identification indicators to a set of AR content generator IDs. The technique requests first metadata associated with a set of AR content generators and an item identification indicator from a shopping AR content generator service using an extended API. The technique requests a set of products based on the first metadata using the extended API. The technique generates a group ID based on a group of products using an extended API. The technique receives a group ID.

Description

[0001] Priority declaration

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 295,450, filed on December 30, 2021, the entire contents of which are incorporated herein by reference for all purposes. Background Art

[0003] The present application generally relates to utilizing electronic devices to provide an augmented reality experience of physical products. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] To easily identify discussion of any particular element or act, the most significant digit or digits in a reference number refer to the figure number in which the element is first introduced.

[0005] Figure 1 is a pictorial representation of a networking environment in which the present disclosure may be deployed, according to some example implementations.

[0006] Figure 2 is a graphical representation of a messaging client application according to some example embodiments.

[0007] Figure 3 is a graphical representation of a data structure as maintained in a database according to some example implementations.

[0008] Figure 4 is a graphical representation of a message according to some example embodiments.

[0009] Figure 5 is a flow diagram of an access restriction process according to some example implementations.

[0010] Figure 6 is a block diagram illustrating an example network environment for accessing a product catalog service for products (eg, physical items) according to some example implementations, the example network environment including the product catalog service described above in Figure 1 The portion of the messaging system discussed in .

[0011] Figure 7 is a block diagram illustrating various modules of an annotation system according to certain example implementations.

[0012] Figure 8 is a diagram showing an example of including additional information corresponding to product metadata of a physical item (eg, a product) according to some embodiments. Figure 4 Schematic diagram of the structure of the message annotation described in .

[0013] Fig. 9 is a diagram showing storage in accordance with some embodiments Figure 3Schematic diagram of the structure of product metadata corresponding to information of a physical item (eg, product) in the product table described in .

[0014] Fig.10 Example interfaces are shown in accordance with various implementations.

[0015] Fig.11 Example interfaces are shown in accordance with various implementations.

[0016] Fig.12 Example interfaces are shown in accordance with various implementations.

[0017] Fig.13 An example interface 1300 of an application for developing an augmented reality content generator (eg, for providing an AR experience for a given product) based on a user profile is shown.

[0018] Fig.14 An example interface 1400 of an application for developing an augmented reality content generator (eg, for providing an AR experience for a given product) based on a user profile is shown.

[0019] Fig.15 Example interfaces are shown in accordance with various implementations.

[0020] Fig.16 Conceptually illustrates example components and data flow diagrams for applying an augmented reality (AR) content generator and metadata delivery for the AR content generator in accordance with some implementations.

[0021] Fig.17 Conceptually illustrates example components and data flow diagrams for resolving a group identifier for a group of AR content generators in accordance with some implementations.

[0022] Fig.18 Conceptually illustrated are example components and data flow diagrams for delivering product metadata in accordance with some implementations.

[0023] Fig.19 Conceptually illustrates example components and data flow diagrams for applying an augmented reality content generator in accordance with some implementations.

[0024] Fig. 20 Conceptually illustrates example components and data flow diagrams for applying an augmented reality content generator in accordance with some implementations.

[0025] Fig.21 is a flow chart illustrating a method according to certain example embodiments.

[0026] Fig. 22 is a flow chart illustrating a method according to certain example embodiments.

[0027] Fig.23 is a flow chart illustrating a method according to certain example embodiments.

[0028] Fig.24 is a flow chart illustrating a method according to certain example embodiments.

[0029] Fig.25 is a block diagram illustrating a software architecture in which the present disclosure may be implemented according to some example implementations.

[0030] Fig.26 is a diagrammatic representation of a machine in the form of a computer system according to some examples within which a set of instructions may be executed, for causing the machine to perform any one or more of the methodologies discussed. DETAILED DESCRIPTION

[0031] With the increase in the use of digital images, the affordability of portable computing devices, the availability of increased capacity of digital storage media, and the increased bandwidth and accessibility of network connections, digital images have become part of the daily lives of more and more people. Users with a wide range of interests from all over the world can capture digital images of various objects and make the captured images available to others via a network (e.g., the Internet). In order to use augmented reality to enhance the user's experience of digital images and content, and enable computing devices to recognize various objects or features captured under various changing conditions (e.g., changes in image scale, noise, lighting, motion, or geometric distortion) may be challenging and computationally intensive. On the one hand, the present technology provides an improved system for providing an augmented reality experience of a product (which can be purchased), which is then applied to such an object (e.g., a user's face) as further described herein.

[0032] In various settings, users of mobile computing devices often use and increasingly utilize messaging systems to provide different types of functionality in a convenient manner. As described herein, the messaging system provides a practical application that enables the provision of an augmented reality experience of a product. The messaging system performs processing to access additional information related to a given product based on product metadata while utilizing the capabilities of the messaging system. The messaging system can present an augmented reality (AR) experience related to a product in a scene with an object (e.g., a user's face) along with providing product information or other information. In an example, the AR experience provides AR content applied to an object (e.g., a representation of a user's face or other parts of the user's body (e.g., arms, legs, etc.) for display on a given client device. Such AR content can be related to the product, such as presenting how a beauty product will appear on a representation of the user's face.

[0033] Also as described herein, the present messaging system provides a practical application that enables identification of physical items or products based on image data captured by a given client device (e.g., a mobile computing device) and provided to the present messaging system for analysis. In particular, a given user of the present messaging system can capture an image or collection of images of a physical item that includes a physical identification indicator (e.g., a barcode) that can identify the physical item. While utilizing the capabilities of the present messaging system, the present messaging system extracts product metadata from the physical identification indicator and performs processing to determine additional product metadata. Therefore, the present technology is understood to advantageously provide technical improvements in presenting AR content items to facilitate natural and intuitive presentation of AR content (e.g., without lags and dropped frames due to graphics bottlenecks and inefficient processing of AR content), and thereby increase the sense of presence in an environment that includes a mix of (real) physical items and AR content items.

[0034] In an example, the technical improvement also involves utilizing a server or remote computing resource to process and present AR content, while in another example, AR content can be presented by a given computing device and then transmitted to other computing devices for presentation. Through the combination and utilization of the various techniques described herein, the waiting time for displaying an AR content item (e.g., the duration for generating AR content and then presenting the AR content on a corresponding display of a given device) can be reduced compared to other existing implementations to provide a more immersive and convincing user experience.

[0035] As referred to herein, the phrases "augmented reality experience," "augmented reality content item," and "augmented reality content generator" include or refer to various image processing operations corresponding to image modifications, filtering, LENSES, media overlays, transformations, etc. as further described herein.

[0036] Figure 1 1 is a block diagram illustrating an example of a messaging system 100 for exchanging data (e.g., messages and associated content) over a network. The messaging system 100 includes multiple instances of client devices 102, each hosting multiple applications including a messaging client application 104. Each messaging client application 104 is communicatively coupled to other instances of the messaging client application 104 and a messaging server system 108 via a network 106 (e.g., the Internet).

[0037] The messaging client application 104 is able to communicate and exchange data with another messaging client application 104 and a messaging server system 108 via a network 106. The data exchanged between messaging client applications 104 and between messaging client applications 104 and messaging server system 108 includes functions (e.g., commands to call functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0038] The messaging server system 108 provides server-side functionality to specific messaging client applications 104 via the network 106. Although certain functions of the messaging system 100 are described herein as being performed by the messaging client application 104 or by the messaging server system 108, the location of certain functions within the messaging client application 104 or the messaging server system 108 is a design choice. For example, it may be technically preferable to initially deploy certain technologies and functions within the messaging server system 108, but later migrate the technologies and functions to the messaging client application 104 where the client device 102 has sufficient processing power.

[0039] The messaging server system 108 supports various services and operations provided to the messaging client applications 104. Such operations include sending data to the messaging client applications 104, receiving data from the messaging client applications 104, and processing data generated by the messaging client applications 104. As examples, the data may include message content, client device information, geographic location information, media annotations and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the messaging system 100 is invoked and controlled by functions available via the user interface (UI) of the messaging client applications 104.

[0040] Turning now specifically to the messaging server system 108, an application program interface (API) server 110 is coupled to and provides a programming interface to an application server 112. The application server 112 is communicatively coupled to a database server 118, which facilitates access to a database 120 in which data associated with messages processed by the application server 112 is stored.

[0041] The application program interface (API) server 110 receives and sends message data (e.g., commands and message payloads) between the client device 102 and the application server 112. Specifically, the application program interface (API) server 110 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the messaging client application 104 in order to invoke the functions of the application server 112. The application program interface (API) server 110 exposes various functions supported by the application server 112, including account registration, login functionality, sending messages from a particular messaging client application 104 to another messaging client application 104 via the application server 112, sending media files (e.g., images or videos) from the messaging client application 104 to the messaging server application 114 and possible access by another messaging client application 104, setting of collections of media data (e.g., stories), retrieval of a friend list of a user of the client device 102, retrieval of such collections, retrieval of messages and content, adding and removing friends from a social graph, location of friends within a social graph, and opening application events (e.g., involving the messaging client application 104).

[0042] The application server 112 hosts multiple applications and subsystems, including a messaging server application 114, an image processing system 116, and a social networking system 122, a product catalog system 124, and an augmented reality system 126. The messaging server application 114 implements many message processing techniques and functions, particularly relating to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the messaging client application 104. As will be described in more detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or libraries). The messaging server application 114 then makes these collections available to the messaging client application 104. In view of the hardware requirements for such processing, the messaging server application 114 can also perform other processor- and memory-intensive data processing on the server side.

[0043] The application server 112 also includes an image processing system 116 dedicated to performing various image processing operations, typically on images or videos received within the payload of messages at the messaging server application 114 .

[0044] The social networking system 122 supports various social networking functional services and makes these functional services available to the messaging server application 114. To this end, the social networking system 122 maintains and accesses the entity graph 304 (e.g., Figure 3). Examples of functionality and services supported by the social networking system 122 include identification of other users of the messaging system 100 with whom a particular user has relationships or “follows” and identification of other entities and particular users of interest.

[0045] The messaging client application 104 includes a set of functions that allow the client device 102 to access the augmented reality system 126. The augmented reality system 126 generates and maintains a list of augmented reality content generators. A given augmented reality content generator may correspond to an augmented reality content item (or some other image modification, image processing, transformation, modification, etc.) for supplementing captured image data or video data with augmented reality content to provide an augmented reality experience.

[0046] In one or more embodiments, the augmented reality system 126 identifies an object depicted in one or more images captured by the messaging client application 104 and determines one or more attributes of the object. The augmented reality system 126 searches for one or more augmented reality content items (e.g., virtual objects) associated with the one or more attributes of the object and ranks the virtual objects (e.g., based on the association and the weight assigned to each attribute). The augmented reality system 126 causes one or more virtual objects or graphical elements of the highest ranked augmented reality content items to be presented on top of the captured image.

[0047] The application server 112 is communicatively coupled to a database server 118 , which facilitates access to a database 120 in which data associated with messages processed by the messaging server application 114 is stored.

[0048] The product catalog system 124 performs operations for verifying products based at least on image data of such products provided by the messaging client application 104. In embodiments, the product catalog system 124 may store information (e.g., metadata) related to the manufacturer of such products and information for verifying the authenticity of the products in the database 120. Figure 6 The product catalog system 124 is discussed in more detail.

[0049] Figure 2 1 is a block diagram showing additional details about the messaging system 100 according to an example implementation. Specifically, the messaging system 100 is shown to include a messaging client application 104 and an application server 112, which in turn contains a number of subsystems, namely, a transient timer system 202, a collection management system 204, and an annotation system 206.

[0050] The transient timer system 202 is responsible for implementing transient access to content permitted by the messaging client application 104 and the messaging server application 114. To this end, the transient timer system 202 incorporates a plurality of timers that selectively display messages and associated content and enable access to the messages and associated content via the messaging client application 104 based on durations and display parameters associated with a message or a collection of messages (e.g., a story). Additional details regarding the operation of the transient timer system 202 are provided below.

[0051] The collection management system 204 is responsible for managing collections of media (e.g., collections of text, image video, and audio data). In some examples, collections of content (e.g., messages, including images, videos, text, and audio) may be organized into "event libraries" or "event stories." Such collections may be made available for a specified period of time (e.g., the duration of the event to which the content relates). For example, content related to a concert may be made available as a "story" for the duration of the concert. The collection management system 204 may also be responsible for publishing icons that provide notification of the existence of a particular collection to the user interface of the messaging client application 104.

[0052] The collection management system 204 also includes a curation interface 208 that allows a collection manager to manage and curate a particular collection of content. For example, the curation interface 208 enables an event organizer to curate a collection of content related to a particular event (e.g., to remove inappropriate content or redundant messages). In addition, the collection management system 204 employs machine vision (or image recognition technology) and content rules to automatically curate content collections. In some embodiments, compensation may be paid to users to include user-generated content in a collection. In such a case, the curation interface 208 operates to automatically pay such users for use of their content.

[0053] The annotation system 206 provides various functions that enable users to annotate or otherwise modify or edit media content associated with a message. For example, the annotation system 206 provides functions related to the media overlay of messages generated and published for processing by the messaging system 100. The annotation system 206 can operably supply media overlay or supplement (e.g., image filtering) to the messaging client application 104 based on the geographic location of the client device 102. In another example, the annotation system 206 can operably supply media overlay to the messaging client application 104 based on other information (e.g., social network information of the user of the client device 102). Media overlay can include audio and visual content and visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlay. Audio and visual content or visual effects can be applied to media content items (e.g., photos) at the client device 102. For example, media overlay can include text that can be superimposed on photos taken by the client device 102. In another example, the media overlay includes a location identification (e.g., Venice Beach) overlay, the name of a live event, or a business name (e.g., Beach Cafe) overlay. In another example, the annotation system 206 uses the geographic location of the client device 102 to identify a media overlay that includes the name of the business at the geographic location of the client device 102. The media overlay may include other tags associated with the business. The media overlay may be stored in the database 120 and accessed through the database server 118.

[0054] In one example implementation, the annotation system 206 provides a user-based publishing platform that enables a user to select a geographic location on a map and upload content associated with the selected geographic location. The user can also specify the context in which a particular media overlay should be provided to other users. The annotation system 206 generates a media overlay including the uploaded content and associates the uploaded content with the selected geographic location.

[0055] In another example implementation, the annotation system 206 provides a merchant-based publishing platform that enables merchants to select specific media overlays associated with geographic locations via a bidding process. For example, the annotation system 206 associates the media overlays of the highest bidding merchant with the corresponding geographic location within a predefined amount of time.

[0056] Figure 3 is a schematic diagram illustrating a data structure 300 that may be stored in a database 120 of a messaging server system 108 according to certain example implementations. Although the contents of the database 120 are illustrated as including a plurality of tables, it should be understood that data may be stored in other types of data structures (e.g., as an object-oriented database).

[0057] The database 120 includes message data stored in a message table 314. The entity table 302 stores entity data, including an entity graph 304. The entities whose records are maintained in the entity table 302 may include individuals, corporate entities, organizations, objects, places, events, etc. Regardless of the type, any entity that stores data with respect to the messaging server system 108 may be an identified entity. Each entity is provided with a unique identifier as well as an entity type identifier (not shown).

[0058] The entity graph 304 also stores information about the relationships and associations between entities. As examples only, such relationships may be social relationships, professional relationships (e.g., working in a common company or organization) based on interests or activities.

[0059] The database 120 also stores annotation data in the form of an example of a filter in the annotation table 312. The filter whose data is stored in the annotation table 312 is associated with a video (whose data is stored in the video table 310) or an image (whose data is stored in the image table 308) and is applied to the video (whose data is stored in the video table 310) or the image (whose data is stored in the image table 308). In one example, the filter is displayed as an overlay superimposed on the image or video during presentation to the recipient user. The filter can be of various types, including a user-selected filter from a library of filters presented to the sending user by the messaging client application 104 when the sending user is composing a message. Other types of filters include geographic location filters (also referred to as geographic filters), which can be presented to the sending user based on geographic location. For example, based on geographic location information determined by the GPS unit of the client device 102, the messaging client application 104 can present a neighborhood or specific location-specific geographic location filter within the user interface. Another type of filter is a data filter, which can be selectively presented to the sending user by the messaging client application 104 based on other input or information collected during the message creation process by the client device 102. Examples of data filters include the current temperature at a particular location, the current speed at which the sending user is traveling, the battery life of the client device 102, or the current time.

[0060] Other annotation data that can be stored in the image table 308 is augmented reality content generators (eg, corresponding to application LENSES, augmented reality experiences, or augmented reality content items). Augmented reality content generators can be real-time special effects and sounds that can be added to images or videos.

[0061] As described above, augmented reality content generators, augmented reality content items, overlays, image transformations, AR images, and similar terms refer to modifications that can be made to a video or image. This includes real-time modifications that modify the image when it is captured using a device sensor and then display the modified image on the device's screen. This also includes modifications to stored content, such as video clips in a library that can be modified. For example, in a device that accesses multiple augmented reality content generators, a user can use a single video clip with multiple augmented reality content generators to see how different augmented reality content generators will modify the stored clips. For example, by selecting different augmented reality content generators for content, multiple augmented reality content generators that apply different pseudo-random motion models can be applied to the same content. Similarly, real-time video capture can be used with the modifications shown to show how the video image currently captured by the device's sensor will modify the captured data. Such data can simply be displayed on the screen and not stored in memory, or the content captured by the device sensor can be recorded and stored in memory with or without modifications (or both). In some systems, a preview feature can show how different augmented reality content generators look in different windows in the display at the same time. This may, for example, enable viewing of multiple windows with different pseudo-random animations on the display simultaneously.

[0062] Thus, data and various systems that use an augmented reality content generator or other such transformation systems to modify content using the data may involve: detection of objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.); tracking such objects as they leave, enter, and move around a field of view in a video frame; and modifying or transforming such objects while tracking them. In various embodiments, different methods for implementing such transformations may be used. For example, some embodiments may involve generating a three-dimensional mesh model of one or more objects and using transformations and animated textures of the models within a video to implement the transformations. In other embodiments, tracking of points on an object may be used to place an image or texture (which may be two-dimensional or three-dimensional) at the tracked location. In yet another embodiment, a neural network analysis of a video frame may be used to place an image, model, or texture in content (e.g., an image or frame of a video). Thus, an augmented reality content generator involves both images, models, and textures used to create transformations in content, and additional modeling and analysis information required to implement such transformations using object detection, tracking, and placement.

[0063] Real-time video processing can be performed using any type of video data (e.g., video streams, video files, etc.) stored in the memory of any type of computerized system. For example, a user can load video files and store them in the memory of the device, or a sensor of the device can be used to generate a video stream. In addition, computer animation models can be used to process any object, such as human faces and parts of the human body, animals, or inanimate objects (e.g., chairs, cars, or other objects).

[0064] In some embodiments, when a specific modification is selected together with the content to be transformed, the computing device identifies the elements to be transformed, and then detects and tracks them if the elements are present in the video frame. The elements of the object are modified according to the request for modification, thereby transforming the frame of the video stream. The transformation of the frame of the video stream can be performed by different methods for different types of transformations. For example, for the transformation of the frame mainly involving the form of the elements of the object, the feature points of each element of the object are calculated (for example, using the active shape model (ASM) or other known methods). Then, a grid based on the feature points is generated for each element of at least one element of the object. The grid is used for the subsequent stage of tracking the elements of the object in the video stream. During the tracking process, the grid mentioned for each element is aligned with the position of each element. Then, additional points are generated on the grid. A first set of first points is generated for each element based on the request for modification, and a set of second points is generated for each element based on the set of first points and the request for modification. Then, the frame of the video stream can be transformed by modifying the elements of the object based on the set of first points and the second point and the grid. In such a method, the background of the modified object can also be changed or distorted by tracking and modifying the background.

[0065] In one or more embodiments, a transformation that changes some areas of an object using elements of an object can be performed by calculating feature points for each element of the object and generating a grid based on the calculated feature points. Points are generated on the grid, and then various areas based on these points are generated. The elements of the object are then tracked by aligning the area for each element with the position for each element of at least one element, and the characteristics of the area can be modified based on a request for modification, thereby transforming the frame of the video stream. Depending on the specific request for modification, the characteristics of the mentioned area can be transformed in different ways. Such modifications can involve changing the color of the area; removing at least some parts of the area from the frame of the video stream; including one or more new objects into the area based on the request for modification; and modifying or distorting the elements of the area or object. In various embodiments, any combination of such modifications or other similar modifications can be used. For certain models to be animated, some feature points can be selected as control points for the entire state space to be used to determine options for model animation.

[0066] In some embodiments of computer animation models that use face detection to transform image data, faces are detected on images using a specific face detection algorithm (e.g., Viola-Jones). Then, an active shape model (ASM) algorithm is applied to the face area of ​​the image to detect facial feature reference points.

[0067] In other embodiments, other methods and algorithms suitable for face detection may be used. For example, in some embodiments, landmarks representing distinguishable points present in most of the images considered are used to locate features. For example, for facial landmarks, the location of the left pupil may be used. In the event that the initial landmarks are not identifiable (e.g., if the person has an eye mask), secondary landmarks may be used. Such a landmark identification process may be used for any such object. In some embodiments, a collection of landmarks forms a shape. The shape may be represented as a vector using the coordinates of the points in the shape. One shape is aligned with another shape using a similarity transformation (allowing translation, scaling, and rotation) that minimizes the average Euclidean distance between the shape points. The average shape is the average of the aligned training shapes.

[0068] In some embodiments, a landmark search begins with an average shape aligned with the position and size of the face determined by the global face detector. Such a search then repeats the steps of suggesting tentative shapes by adjusting the positions of shape points by template matching the image texture around each point, and then conforming the tentative shapes to the global shape model until convergence occurs. In some systems, template matching alone is unreliable, and the shape model aggregates the results of weak template matchers to form a stronger overall classifier. The entire search is repeated at each level in the image pyramid from coarse to fine resolution.

[0069] Embodiments of the transformation system can capture an image or video stream on a client device (e.g., client device 102) and perform complex image manipulations locally on the client device 102 while maintaining an appropriate user experience, computational time, and power consumption. Complex image manipulations can include size and shape changes, emotional transformations (e.g., changing a face from a frown to a smile), state transformations (e.g., aging a subject, reducing apparent age, changing gender), style transformations, application of graphical elements, and any other suitable image or video manipulations that have been configured to be efficiently executed on the client device 102 by a convolutional neural network.

[0070] In some example embodiments, a computer animation model for transforming image data may be used by a system in which a user may capture an image or video stream of the user (e.g., a selfie) using a client device 102 having a neural network operating as part of a messaging client application 104 operating on the client device 102. A transformation system operating within the messaging client application 104 determines the presence of a face within the image or video stream and provides a modification icon associated with the computer animation model for transforming the image data, or the computer animation model may exist in association with an interface described herein. The modification icon includes changes that may be the basis for modifying the user's face within the image or video stream as part of the modification operation. Once the modification icon is selected, the transformation system initiates a process of transforming the user's image to reflect the selected modification icon (e.g., generating a smiling face about the user). In some embodiments, once the image or video stream is captured and the specified modification is selected, the modified image or video stream may be presented in a graphical user interface displayed on the mobile client device. The transformation system may implement a complex convolutional neural network on a portion of the image or video stream to generate and apply the selected modification. That is, a user can capture an image or video stream, and once a modification icon is selected, the modified result can be presented in real time or near real time. Furthermore, the modification can be persistent while the video stream is captured and the selected modification icon remains toggled. Machine-taught neural networks can be used to implement such modifications.

[0071] In some embodiments, a graphical user interface presenting the modifications performed by the transformation system may provide additional interactive options to the user. Such options may be based on an interface for initiating selection of a particular computer animation model and content capture (e.g., initiated from a content creator user interface). In various embodiments, the modifications may be persistent after the initial selection of the modification icon. The user may toggle the modification on or off by tapping or otherwise selecting the face being modified by the transformation system, and store it for later viewing or browsing other areas of the imaging application. In the case where the transformation system modifies multiple faces, the user may globally toggle the modification on or off by tapping or selecting a single face modified and displayed within the graphical user interface. In some embodiments, each face in a group of multiple faces may be modified separately, or such modifications may be toggled separately by tapping or selecting each face or a series of each face displayed within the graphical user interface.

[0072] In some example embodiments, a graphics processing pipeline architecture is provided that enables different augmented reality experiences (e.g., AR content generators) to be applied in corresponding different layers. Such a graphics processing pipeline provides a scalable rendering engine for providing multiple augmented reality experiences included in composite media (e.g., images or videos) for presentation by the messaging client application 104 (or messaging system 100).

[0073] As mentioned above, the video table 310 stores video data, which in one embodiment is associated with messages whose records are maintained in the message table 314. Similarly, the image table 308 stores image data associated with messages whose message data is stored in the entity table 302. The entity table 302 may associate various annotations from the annotation table 312 with the various images and videos stored in the image table 308 and the video table 310.

[0074] The story table 306 stores data about a collection of messages and associated images, videos, or audio data that are compiled into a collection (e.g., a story or library). The creation of a particular collection can be initiated by a particular user (e.g., each user whose record is maintained in the entity table 302). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcasted by the user. To this end, the user interface of the messaging client application 104 may include an icon that the user can select to enable the sending user to add specific content to his or her personal story.

[0075] A collection may also constitute a "Live Story," which is a collection of content from multiple users that is created manually, automatically, or using a combination of manual and automatic techniques. For example, a "Live Story" may constitute a curated stream of content submitted by users from different locations and events. Users whose client devices have location services enabled and who are at a common location event at a particular time may be presented with an option, such as via a user interface of the messaging client application 104, to contribute content to a particular Live Story. Live Stories may be identified to users by the messaging client application 104 based on their location. The end result is a "Live Story" told from a community perspective.

[0076] Another type of content collection is called a "location story," which enables users whose client devices 102 are located in a particular geographic location (e.g., on a college or university campus) to contribute to a particular collection. In some implementations, contributions to location stories may require a second degree of authentication to verify that the end user belongs to a particular organization or other entity (e.g., is a student on a university campus).

[0077] The database 120 also stores data for products in a product table 316, which enables the product catalog system 124 to perform operations related to providing an augmented reality experience for a product (e.g., a given physical item available for purchase or sale). In an example, the product table 316 includes a catalog (e.g., a list) of products and their associated product identifiers, which can be compared to product metadata provided by the product catalog system 124. As further described herein, the product catalog system 124 can determine product metadata associated with a particular product. As further described below, such product metadata includes a product identifier that can be used as part of a process for providing an augmented reality experience related to the product.

[0078] Figure 4 is a schematic diagram illustrating the structure of a message 400 generated by a messaging client application 104 for transmission to another messaging client application 104 or a messaging server application 114 according to some embodiments. The content of a particular message 400 is used to populate a message table 314 stored in a database 120 accessible to a messaging server application 114. Similarly, the content of the message 400 is stored in memory as "in-flight" or "in-transit" data of a client device 102 or application server 112. The message 400 is shown to include the following elements:

[0079] ●Message identifier 402: A unique identifier that identifies the message 400.

[0080] • Message text payload 404 : Text to be generated by a user via the user interface of the client device 102 and included in the message 400 .

[0081] • Message Image Payload 406 : Image data captured by a camera component of the client device 102 or retrieved from a memory component of the client device 102 and included in the message 400 .

[0082] • Message video payload 408 : Video data captured by the camera component or retrieved from the memory component of the client device 102 and included in the message 400 .

[0083] • Message audio payload 410 : audio data captured by a microphone or retrieved from a memory component of the client device 102 and included in the message 400 .

[0084] • Message annotation 412: Annotation data representing an annotation (eg, a filter, sticker, or other enhancement) to be applied to the message image payload 406, the message video payload 408, or the message audio payload 410 of the message 400.

[0085] ●Message duration parameter 414: A parameter value indicating the amount of time in seconds that the content of the message (e.g., message image payload 406, message video payload 408, message audio payload 410) will be presented or made accessible to the user via the messaging client application 104.

[0086] Message geolocation parameters 416: Geolocation data (e.g., latitude and longitude coordinates) associated with the content payload of the message. A plurality of message geolocation parameter values ​​416 may be included in the payload, each of which is associated with information about a content item included in the content (e.g., a specific image within a message image payload 406 or a specific video in a message video payload 408).

[0087] Message story identifier 418: An identifier value that identifies one or more content collections (e.g., "stories") associated with a particular content item in message image payload 406 of message 400. For example, multiple images within message image payload 406 may each be associated with multiple content collections using an identifier value.

[0088] Message tags 420: Each message 400 may be tagged with a plurality of tags, each of which indicates the subject of the content included in the message payload. For example, where a particular image included in the message image payload 406 depicts an animal (e.g., a lion), a tag value may be included within the message tags 420 indicating the relevant animal. The tag values ​​may be manually generated based on user input or may be automatically generated using, for example, image recognition.

[0089] • Message sender identifier 422: An identifier (eg, a messaging system identifier, an email address, or a device identifier) ​​that indicates the user of the client device 102 on which the message 400 was generated and from which the message 400 was sent.

[0090] • Message recipient identifier 424: An identifier (eg, a messaging system identifier, an email address, or a device identifier) ​​that indicates the user of the client device 102 to which the message 400 is addressed.

[0091] The content (e.g., value) of various elements of message 400 may be pointers to locations in a table in which content data values ​​are stored. For example, an image value in message image payload 406 may be a pointer to a location (or address) within image table 308. Similarly, a value within message video payload 408 may point to data stored within video table 310, a value stored within message annotation 412 may point to data stored in annotation table 312, a value stored within message story identifier 418 may point to data stored in story table 306, and values ​​stored within message sender identifier 422 and message recipient identifier 424 may point to user records stored within entity table 302.

[0092] Figure 5 is a schematic diagram illustrating an access restriction process 500 according to which access to content (e.g., a multimedia payload of an ephemeral message 502 and associated data) or a collection of content (e.g., an ephemeral message group 504) can be time-restricted (e.g., ephemeral).

[0093] The ephemeral message 502 is shown to be associated with a message duration parameter 506, the value of which determines the amount of time that the messaging client application 104 will display the ephemeral message 502 to the receiving user of the ephemeral message 502. In one embodiment, the receiving user may view the ephemeral message 502 for up to 10 seconds, depending on the amount of time specified by the sending user using the message duration parameter 506.

[0094] The message duration parameter 506 and the message recipient identifier 424 are shown as inputs to a message timer 512, which is responsible for determining the amount of time that the transient message 502 is shown to a particular receiving user identified by the message recipient identifier 424. In particular, the transient message 502 is shown to the associated receiving user only for a period of time determined by the value of the message duration parameter 506. The message timer 512 is shown as providing an output to a more generalized implementation of the transient timer system 202, which is responsible for the overall timing of displaying content (e.g., the transient message 502) to the receiving user.

[0095] Figure 5 502 is included in a transient message group 504 (e.g., a collection of messages in a personal story or an event story). The transient message group 504 has an associated group duration parameter 508, the value of which determines the duration for which the transient message group 504 is presented and accessible to users of the messaging system 100. For example, the group duration parameter 508 may be the duration of a concert, wherein the transient message group 504 is a collection of content belonging to the concert. Alternatively, a user (an owning user or a curator user) may specify the value of the group duration parameter 508 when performing setup and creation of the transient message group 504.

[0096] In addition, each ephemeral message 502 within the ephemeral message group 504 has an associated group participation parameter 510, the value of which determines the duration for which the ephemeral message 502 is accessible within the context of the ephemeral message group 504. Thus, a particular ephemeral message group 504 may "expire" and become inaccessible within the context of the ephemeral message group 504 before the ephemeral message group 504 itself expires according to the group duration parameter 508. The group duration parameter 508, the group participation parameter 510, and the message recipient identifier 424 each provide inputs to a group timer 514, which is operable to first determine whether a particular ephemeral message 502 of the ephemeral message group 504 is to be displayed to a particular receiving user, and if so, how long to display it. Note that the identity of the particular receiving user is also known to the ephemeral message group 504 due to the message recipient identifier 424.

[0097] Thus, the group timer 514 is operable to control the total lifespan of the associated ephemeral message group 504 and the individual ephemeral messages 502 included in the ephemeral message group 504. In one embodiment, each ephemeral message 502 within the ephemeral message group 504 remains viewable and accessible for a period of time specified by the group duration parameter 508. In another embodiment, within the context of the ephemeral message group 504, a particular ephemeral message 502 may expire based on the group participation parameter 510. Note that even within the context of the ephemeral message group 504, the message duration parameter 506 may still determine the duration for which a particular ephemeral message 502 is displayed to a receiving user. Thus, the message duration parameter 506 determines the duration for which a particular ephemeral message 502 is displayed to a receiving user, regardless of whether the receiving user is viewing the ephemeral message 502 within or outside the context of the ephemeral message group 504.

[0098] The transient timer system 202 may also be operable to remove a particular transient message 502 from the transient message group 504 based on a determination that an associated group participation parameter 510 has been exceeded. For example, when a sending user has established a group participation parameter 510 of 24 hours from posting, the transient timer system 202 will remove the associated transient message 502 from the transient message group 504 after the specified 24 hours. The transient timer system 202 is also operable to remove a transient message group 504 when the group participation parameter 510 for each transient message 502 within the transient message group 504 has expired or when the transient message group 504 itself has expired according to the group duration parameter 508.

[0099] In some use cases, the creator of a particular ephemeral message group 504 may specify an indefinite group duration parameter 508. In this case, the expiration of the group participation parameter 510 for the last remaining ephemeral message 502 within the ephemeral message group 504 will determine when the ephemeral message group 504 itself expires. In this case, new ephemeral messages 502 with new group participation parameters 510 added to the ephemeral message group 504 effectively extend the life of the ephemeral message group 504 to a value equal to the group participation parameter 510.

[0100] In response to the transient timer system 202 determining that the transient message group 504 has expired (e.g., is no longer accessible), the transient timer system 202 communicates with the messaging system 100 (and, for example, in particular, the messaging client application 104) to cause the indicia (e.g., icon) associated with the relevant transient message group 504 to no longer be displayed within the user interface of the messaging client application 104. Similarly, when the transient timer system 202 determines that the message duration parameter 506 for a particular transient message 502 has expired, the transient timer system 202 causes the messaging client application 104 to no longer display the indicia (e.g., icon or text identification) associated with the transient message 502.

[0101] Figure 6 is a block diagram illustrating an example network environment 600 for accessing information from a product catalog service for products (e.g., physical items for sale) according to some example implementations, the example network environment 600 including the above-described Figure 1 100 discussed in the foregoing. However, not all of the components depicted may be used in all implementations, and one or more implementations may include additional or different components than those shown in the figures. Changes may be made to the arrangement and type of components without departing from the spirit or scope of the claims set forth herein. Additional components, different components, or fewer components may be provided.

[0102] For illustrative purposes, the network environment 600 is Figure 6 6. The network environment 600 is shown as including the client device 102, the messaging server system 108, and the product catalog service system 602; however, in some implementations, the network environment 600 may include any number of electronic devices and any number of servers or a data center including multiple servers.

[0103] In one or more implementations, the network 106 may be an interconnected network that may include the Internet or electronic devices that may be communicatively coupled to the Internet. The network 106 may communicatively couple, for example, the client device 102 with the messaging server system 108 or the product catalog service system 602 (directly or indirectly).

[0104] The messaging server system 108 or product catalog service system 602 may be part of a computer network or server group such as in a cloud computing or data center implementation. As further described below, the messaging server system 108 or product catalog service system 602 stores or generates information requested or received by the client device 102.

[0105] As shown, a client device 102 hosting a plurality of applications, including those previously described in the prior art, is communicatively coupled to a messaging server system 108 and other instances of a messaging client application 104 (e.g., on another client device 102) via a network 106. Figure 1 The messaging client application 104 described in . The messaging server system 108 includes the messaging client application 104 described in . Figure 1 The product catalog system 124 described in .

[0106] In an example implementation, the client device 102 may select or access an augmented reality (AR) experience for a given product, such as an augmented reality (AR) content generator, etc., by using the product catalog system 124. The product catalog system 124 may access a product catalog service system 602 that provides product information / metadata about the corresponding product in one or more databases 620.

[0107] The product catalog system 124 may receive at least an indicator of the selected AR experience (e.g., a product AR experience ID as further described herein) from the client device 102 and perform a lookup, search, or selection operation on the product table 316 to retrieve product metadata from the database 120. In an example implementation, such product metadata includes a product identifier (ID). The product catalog system 124 provides the aforementioned product metadata to the client device 102, which enables the client device 102 to access or launch the selected AR experience for display on the client device 102.

[0108] In an example implementation, client device 102 may provide (e.g., captured using a camera provided by client device 102) image data including a representation of a physical item including an identification indicator of the physical item (e.g., a physical barcode, etc.), and analyze the image data to extract information from the identification indicator (e.g., extract verification metadata from the barcode). The extracted information may include a manufacturer identifier (ID), a product ID, or an item unique ID (e.g., a unique serial number, etc.).

[0109] The product catalog system 124 may receive the previously mentioned extracted information from the client device 102 and determine additional product information (e.g., product metadata). The additional product metadata for a given physical item may be determined based at least in part on a set of signals (e.g., provided by the client device 102 or the product catalog system 124) that include information about the respective product that the respective manufacturer has maintained, including, but not limited to, a barcode, a universal product code (UPC) code, a QR code, a Snap code, and a Snap ID. TM, unique individual serial numbers, inventory unit numbers, vehicle identification numbers, European article numbers (EANs), international standard book numbers (ISBNs), manufacturer part numbers (MPNs), global trade item numbers (GTINs), Japanese article numbers (JANs), watermarks, etc. Other signals that may be used may include location information (e.g., GPS coordinates for determining a specific distributor or retail, or a geographic region corresponding to a physical item), network information (e.g., a Wi-Fi network), etc. In addition, it should be understood that a physical item may include a corresponding physical identification indicator based on one or more of the previously mentioned standards or formats for identifying individual products.

[0110] The product catalog system 124 may receive at least the product ID from the client device 102 (or the product catalog system 124 as described below) and perform a lookup, search, or select operation on the product table 316 to retrieve the product metadata from the database 120. The product catalog system 124 then provides the client device 102 or the product catalog system 124 with the previously mentioned product metadata.

[0111] For the purpose of populating the product table 316, the product catalog system 124 can communicate with the product catalog service system 602. In an example, the product catalog system 124 can send a request message to a corresponding server for obtaining metadata related to a given physical item. The request message can include, for example, a product ID. In an embodiment, as further shown in the network environment 600, such a server is the product catalog service system 602. In an embodiment, the product catalog service system 602 is implemented as an electronic-based service (e.g., a network service) that can be accessed by a given client electronic device (e.g., client device 102) based on a request message including a product ID in a specific message protocol. In response to the request message, the product catalog service system 602 can perform a search or database query for information or metadata related to the physical item based on the product ID included in the request message.

[0112] In response to the request message, the product catalog service system 602 may provide a response message to the product catalog system 124, which, in an example embodiment, includes information from the database 620 in response to the request. The product catalog system 124 may store the received information in the database 120. Based at least on the product-related information retrieved from the database 120, the client device 102 may provide (e.g., present on the UI of the messaging client application 104) an AR experience of the product for display.

[0113] In an embodiment, as described above in Figure 6The client device 102 described in the can be, for example, a portable computing device, such as a laptop computer, a smart phone, a peripheral device (e.g., a digital camera), a tablet device, a wearable device (e.g., a watch, a bracelet, a headset, etc.), or any other suitable device. Each of the aforementioned devices can also support touch screen gestures (e.g., by using a touch screen) or non-touch gestures (e.g., by using a gyroscope or motion sensor). In one or more implementations, the client device 102 can support touch screen-like gestures, for example, in a virtual reality or augmented reality environment. In Figure 6 In the embodiment of the present invention, the client device 102 is depicted as a mobile electronic device with a touch screen as an example. In one or more implementations, the client device 102 may be a machine as further described below, or may include all or a portion of the machine. The client device 102 may include components that perform operations for providing an AR experience of a physical item (e.g., a product) as described in further detail below.

[0114] In an embodiment, the client device 102 is a head-mounted portable system wearable by a user that includes a system capable of capturing images, audio, video, multimedia content, etc. based on the user's surrounding physical environment.

[0115] As further described herein, the client device 102 can perform image collection, text, or object recognition processing to extract product metadata from captured image data, where the image data can include a representation of a physical item corresponding to the product and a physical identification indicator (e.g., a barcode) corresponding to the physical item.

[0116] As previously discussed, client device 102 may provide (e.g., captured using a camera provided by client device 102) image data including a representation of a physical item including an identification indicator of the physical item (e.g., a physical barcode, etc.), and analyze the image data to extract information from the identification indicator (e.g., extract product metadata from the barcode). The extracted information may include, among other things, a manufacturer identifier (ID), a product ID, or an item unique ID (e.g., a unique serial number, etc.).

[0117] Figure 7700 is a block diagram illustrating various modules of the annotation system 206 according to certain example implementations. The annotation system 206 is shown to include an image data receiving module 702, a product identification module 704, an augmented reality content generator module 706, and a product recommendation module 708. The various modules of the annotation system 206 are configured to communicate with each other (e.g., via a bus, a shared memory, or a switch). Any one or more of these modules can be implemented using one or more computer processors 710 (e.g., by configuring such one or more computer processors to perform the functions described for the module), and thus can include one or more of the computer processors 710 (e.g., a collection of processors provided by the client device 102).

[0118] Any one or more of the described modules may be implemented using hardware alone (e.g., one or more of the computer processors 710 of a machine (e.g., machine 2600)) or a combination of hardware and software. For example, any described module of the annotation system 206 may physically include an arrangement of one or more computer processors 710 (e.g., a subset of one or more computer processors of a machine (e.g., machine 2600) or one or more computer processors 710 therein) configured to perform the operations described herein for that module. As another example, any module of the annotation system 206 may include software, hardware, or both software and hardware that configures an arrangement of one or more computer processors 710 (e.g., among one or more computer processors of a machine (e.g., machine 2600)) to perform the operations described herein for that module. Thus, different modules of the annotation system 206 may include and configure different arrangements of such computer processors 710 or a single arrangement of such computer processors 710 at different points in time. Furthermore, any two or more modules of the annotation system 206 may be combined into a single module, and the functionality described herein for a single module may be subdivided between multiple modules. Furthermore, according to various example embodiments, modules described herein as being implemented in a single machine, database, or device may be distributed across multiple machines, databases, or devices.

[0119] The image data receiving module 702 receives an image captured by the client device 102. For example, the image is a photograph captured by an optical sensor (e.g., a camera) of the client device 102. The image includes one or more real-world features, such as a user's face or a physical object detected in the image. In some embodiments, the image includes metadata describing the image.

[0120] The product identification module 704 uses different object detection processes to detect objects in the image, such as a physical item corresponding to the product from which the user wants to extract product metadata or a physical indicator (e.g., a barcode) of an identification corresponding to the physical item. In an example, imaging processing algorithms and recognition techniques can be used to detect objects of the image. For example, optical character recognition (OCR) can be used as the main image analysis technique or to enhance other processes. Features of the image (e.g., shape, size, color, and text) can be extracted. In some embodiments, the image processing process can include sub-processes such as thresholding (converting a grayscale image to black and white, or using separation based on grayscale values), segmentation, blob extraction, pattern recognition, barcode and data matrix code reading, measurement (measuring object size), positioning, edge detection, color analysis, filtering (e.g., morphological filtering), and template matching (finding, matching, or counting specific patterns).

[0121] In addition, in an example embodiment, the product identification module 704 utilizes a set of classifiers that classify images received from a camera device of a mobile computing device into one or more categories. In an embodiment, an example set of image classifiers determines whether an image includes a physical identification indicator including text, a barcode pattern, or a QR code pattern, etc. In an embodiment, another example set of image classifiers performs object classification based on a library of 3D models or images (e.g., stored in the database 120). Various techniques (e.g., OCR and other geometric recognition processes) may also be used as image analysis techniques to identify one or more potential objects of an image (e.g., a physical item corresponding to a product and a physical identification indicator of a physical item). In addition, it will be appreciated that the product identification module 704 may utilize a machine learning model (e.g., a predictive model that classifies image data into a specific object classification or type) to perform object recognition.

[0122] The product identification module 704 can perform operations (e.g., processing) for extracting product metadata from the identified object corresponding to the physical identification indicator of the physical item in the image and determining additional product metadata for the physical item. In addition, the product identification module 704 can receive additional product metadata from the product catalog system 124 described herein.

[0123] In an embodiment, the augmented reality content generator module 706 enables the display of optional graphical items to be presented in a carousel arrangement (e.g., a carousel interface). As an example, a user can utilize various inputs to rotate the optional graphical items onto and off the display screen in a manner corresponding to a carousel providing a circular view of the graphical items. The carousel arrangement allows multiple graphical items to occupy a specific graphical area on the display screen. In an example, corresponding AR experiences corresponding to different AR content generators can be organized into corresponding groups to be included in the carousel arrangement, so that the AR content generators can be rotated in groups. Although a carousel interface is provided as an example, it is understood that other graphical interfaces can also be used. For example, a collection of augmented reality content generators can include a graphic list, a scrolling list, a scrolling graphic, or another graphical interface that enables navigation through various graphical items for selection, etc. As used herein, a carousel interface refers to displaying graphical items in an arrangement similar to a circular list, so that navigation through a circular list can be performed based on user input (e.g., touch or gesture) to select or scroll graphical items. In an example, a collection of graphic items can be presented on a horizontal (or vertical) line or axis, wherein each graphic item is represented as a specific thumbnail image (or icon, avatar, etc.). At any one moment, some graphic items in the carousel interface can be hidden. In an example, if a user wants to view hidden graphic items, the user can provide user input (e.g., touch, gesture, etc.) to scroll the graphic items in a specific direction (e.g., left, right, up or down, etc.). Afterwards, a subsequent view of the carousel interface is displayed, wherein an animation is provided or presented to present one or more additional graphic items to be included in the interface, and wherein some of the previously presented graphic items can be hidden in the subsequent view. In an embodiment, in this way, the user can navigate back and forth the collection of graphic items in a circular manner. Therefore, it can be understood that the carousel interface can optimize screen space by displaying only a subset of images from a collection of graphic items in a circular view.

[0124] The augmented reality content generator module 706 utilizes different object or face detection processes to detect objects or faces in the image. In an example, imaging processing algorithms and recognition techniques can be used to detect the user's face in the image. Based on the selected AR content generator, the augmented reality content generator module 706 can generate and present an AR experience based on the AR content generator selected from the carousel interface for display on a given client device (e.g., client device 102). In an embodiment, information from the product identification module 704 (e.g., product ID, etc.) is used by the augmented reality content generator module 706 to present such AR content.

[0125] The augmented reality content generator module 706 performs presentation of content based on data provided by at least one of the previously mentioned modules for display by the messaging client application 104. For example, the augmented reality content generator module 706 performs various operations based on algorithms or techniques corresponding to animation or providing visual or auditory effects to received image data according to product information, which will be further described herein.

[0126] The augmented reality content generator module 706 performs rendering of the message for display by the messaging client application 104 based on data provided by at least one of the previously mentioned modules. In an example, the augmented reality content generator module 706 utilizes a graphics processing pipeline to perform graphics operations to render the message for display. In an example, the augmented reality content generator module 706 implements an extensible rendering engine that supports multiple image processing operations corresponding to corresponding media overlays.

[0127] In some implementations, the augmented reality content generator module 706 provides a graphics system that renders two-dimensional (2D) objects or objects from a three-dimensional (3D) world (real or imaginary) onto a 2D display screen. In some implementations, such a graphics system (e.g., a graphics system included on the client device 102) includes a graphics processing unit (GPU) for performing image processing operations and rendering graphical elements for display.

[0128] In an implementation, the GPU includes a logical graphics processing pipeline that can receive a representation of a 2D or 3D scene and provide an output of a bitmap representing a 2D image for display. Existing application programming interfaces (APIs) have implemented a graphics pipeline model. Examples of such APIs include the Open Graphics Library (OPENGL) API and the METAL API. The graphics processing pipeline includes multiple stages that convert a set of vertices, textures, buffers, and state information into image frames on the screen. In an implementation, one of the stages of the graphics processing pipeline is a shader, which can be used as part of a specific augmented reality content generator applied to an input frame (e.g., an image or video). The shader can be implemented as code running on a dedicated processing unit (also referred to as a shader unit or shader processor) that typically executes several computing threads, and the code is programmed to generate appropriate levels of color and / or specific effects for the fragment being presented. For example, a vertex shader processes the attributes of a vertex (position, texture coordinates, color, etc.), and a pixel shader processes the attributes of a pixel (texture value, color, z depth, and alpha value). In some instances, a pixel shader is referred to as a fragment shader.

[0129] It should be understood that other types of shader processing may be provided. In an example, a particular sampling rate is utilized within a graphics processing pipeline to render an entire frame, or pixel shading is performed at a particular per-pixel rate. In this manner, a given client device (e.g., client device 102) operates the graphics processing pipeline to convert information corresponding to an object into a bitmap that can be displayed by an electronic device.

[0130] Figure 8 is a diagram showing an example of an augmented reality experience including additional information corresponding to a product according to some embodiments. Figure 4 Schematic diagram of the structure of message annotation 412 described in . In particular, the additional information corresponds to the information in the message used to present the AR experience.

[0131] In an embodiment, comprising Figure 8 The content of the particular message 400 of the additional data shown is used to populate the message table 314 for the given message stored in the database 120, which can then be accessed by the product catalog system 124 or the messaging client application 104. As shown in the example embodiment, the message annotation 412 includes the following elements corresponding to the data of the message:

[0132] Product AR experience identifier 852: Identifier of the AR experience utilized in the message (e.g., AR content generator, animation, or effect, including effects, lenses, filters, image modifiers, etc.)

[0133] Message identifier 854: Identifier of the message

[0134] Asset identifiers 856: A collection of identifiers of assets in the message. For example, corresponding asset identifiers may be included for multiple assets determined by a particular AR content generator. In an embodiment, such assets are created by an AR content generator on the sender side, uploaded to the messaging server application 114, and used by an AR content generator on the receiver side to recreate the message.

[0135] Metadata 862 corresponding to the additional metadata

[0136] Fig. 9 is a diagram showing storage in accordance with some embodiments Figure 3 Schematic diagram of the structure of product metadata 900 corresponding to information of a physical item (eg, product) in the product table 316 described in FIG.

[0137] In an embodiment, Fig. 9The data shown is used to populate the product table 316 stored in the database 120, which can then be accessed by the product catalog system 124 or the messaging client application 104. As shown in the example embodiment, the product metadata 900 includes the following elements:

[0138] Product Identifier 902: An identifier of the product (e.g., model number, part number, unique serial number, etc.) extracted from the physical identification identifier from the image data.

[0139] Product AR experience identifier 904: An identifier of the AR experience utilized in the message (e.g., AR content generator, animation, or effect, including effects, lenses, filters, image modifiers, etc.), which can be used to communicate with Figure 8 The products discussed in the Augmented Reality Experience Identifier 852 are compared

[0140] Product metadata 906: information related to the product corresponding to product identifier 902

[0141] Embodiments of the present technology provide a consistent user interface and user experience framework for an AR content generator for a brand (e.g., corresponding to a corresponding company or entity that manufactures or provides a physical product), wherein the AR content generator is built at least in part based on product information from a product catalog (e.g., provided by the product catalog system 124 or the product catalog service system 602). More specifically, a product card can be generated, which refers to a combination of AR content and product information, and the product information is provided for display to the user in an appropriate interface (e.g., a graphical interface or as part of an AR experience when using a mobile device or an eyewear device such as an AR head-mounted device). As previously described in Figure 8 and Fig. 9 As discussed in , product information can be associated with a specific AR experience (e.g., an AR content generator, etc.).

[0142] Fig.10 Example interfaces are shown in accordance with various implementations. Fig.10 The example interface of can be provided for display on a client device (e.g., client device), for example, through one or more interfaces of the messaging client application 104.

[0143] The client device 102 may have at least one camera. Each camera may be, for example, an active pixel sensor in a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or an N-type metal oxide semiconductor (NMOS), an infrared or ultrasonic image sensor, or an image sensor utilizing other types of image capture technology. In an example, a user may position the client device 102 so that one or more physical objects including a physical item (e.g., a beauty product box) are within the field of view of at least one camera of the client device 102. The at least one camera may capture an image such that a representation of the physical item with a representation of a physical identification indicator is displayed on a display screen of the client device 102. In some embodiments, at least one camera captures video, thereby providing a "live" view of the captured video information. Frames of an image or video may be analyzed, for example, by analyzing on the client device 102 or sending to a server or service (e.g., a messaging server system 108 or an image processing system 116) for analyzing image data over a network (e.g., network 106).

[0144] Using the product identification module 704, the client device 102 can extract product metadata, such as a product ID, based on the representation of the physical identification indicator 1002. The client device 102 can then send the extracted product metadata to the product catalog service system 124 to receive additional product metadata. Based at least in part on the received manufacturer verification metadata, the client device 102 can send a request to the product catalog service system 602 to obtain the additional product metadata and other types of information that can be included as discussed herein.

[0145] In embodiments, the product recommendation module 708 may select a set of AR experiences (e.g., AR content generators, etc.) related to an item or product based on user profile information, user activity, ranking signals, and other signals for display by the client device 102. As also discussed further herein, an AR experience may be selected based on a domain or other parameters associated with a given AR content generator.

[0146] exist Fig.10 In the example of , a product corresponding to a particular brand may have been scanned, which causes the interface discussed below to be provided. In another example, when accessing a social network or some other server, the interface described below is provided, where the user can navigate and explore various products or content, including AR content.

[0147] exist Fig.10 In the example shown, an interface 1000 showing information based at least in part on the product metadata described above is provided and may be received by the augmented reality content generator module 706 and displayed on the client device 102 .

[0148] As shown, profile information 1010 corresponding to a given brand (e.g., a cosmetics manufacturer) is shown for display in interface 1000. Similar to viewing a profile in a social network, interface 1000 can provide content related to the brand, and in this example, provide AR experiences of different products associated with the brand. In this example, a set of product cards including product card 1020, product card 1022, and product card 1024 are included in interface 1000 below profile information 1010. In an embodiment, when included in interface 1000, each product card can have a size up to a predetermined size (e.g., a fixed maximum width or a fixed maximum height, etc.). In addition, each product card provides an AR experience (e.g., an AR content generator) for a specific product (e.g., corresponding to a product SKU or other identifier). In an embodiment, each product card in interface 1000 is randomly selected to populate interface 1000. In an embodiment, each product card in interface 1000 is selected based on a ranking of each product card in the product card (e.g., a score assigned to each product card).

[0149] Additional product cards are shown partially below product card 1020, product card 1022, and product card 1024, and it should be understood that any number of product cards may be provided in response to appropriate input such as a scroll gesture (e.g., swiping down or swiping up) or similar interactions with interface 1000. As also shown, each product card may include a corresponding label or text information describing the product(s) associated with the product card.

[0150] In embodiments, a given product card may include various functions that enable the subject system to provide:

[0151] SKU selector for products

[0152] Call to action buttons that can link to different product URLs

[0153] Display of relevant purchase information (e.g., price, SKU name, merchant, etc.)

[0154] A way to favorite, save, or save individual products displayed in an AR content generator or experience

[0155] Fig.11 Example interfaces are shown in accordance with various implementations. Fig.11 The example interface of can be provided for display on a client device (e.g., client device), for example, through one or more interfaces of the messaging client application 104.

[0156] In this example, the Fig.101022). As shown, product information 1160 (e.g., product description, price, manufacturer, supplier, selection graphic items for viewing additional information, etc.) is included. As further shown, additional information 1150 related to the product (e.g., description or product name) can be included in interface 1100. To return to interface 1000, input can be received anywhere other than additional information 1150 or product information 1160.

[0157] A set of selectable graphical items corresponding to respective product cards associated with different beauty products including selectable graphical item 1170 is displayed. In an embodiment, the set of selectable graphical items is selected and presented in a carousel interface in interface 1100 based on information from a product catalog (e.g., associated with a particular brand mentioned above). In this example, selectable graphical item 1170 corresponding to a particular product card (e.g., product card 1022) is shown as being selected by a user. A different product card may be selected from a set of selectable graphical items such as selectable graphical item 1176 or selectable graphical item 1178 from a carousel interface using appropriate input (e.g., touch input, tap, etc.). When a different product card is selected, the corresponding selectable graphical item may be positioned in the center of the carousel interface, which causes the other selectable graphical items to shift in the left or right direction accordingly.

[0158] Fig.12 Example interfaces are shown in accordance with various implementations. Fig.12 The example interface of can be provided for display on a client device (e.g., client device), for example, through one or more interfaces of the messaging client application 104.

[0159] In this example, in response to selecting Fig.11 1200 ). In an embodiment, the set of product cards is scrollable, which causes additional product cards to be included in interface 1200 (also causing product card 1250, product card 1255, and product card 1260 to disappear from interface 1200). Each product card in interface 1200 is represented by a selectable graphical item, and each of the selectable graphical items can be selected (e.g., touch input, or hovering over with a cursor or finger, etc.) to cause additional information to be displayed for the selected product card. As also shown, Fig.12 Each product card in includes a selectable graphical item (e.g., a “Try Now” button) that, when selected, causes the corresponding AR content generator of the product card to be displayed on the display of the client device (e.g., similar to Fig.11). The AR content generator can then render based on the corresponding product of the product card (e.g., a beauty product such as lipstick, makeup, etc.) to display the AR content on the user's facial features.

[0160] In embodiments, the present technology may provide an API (application programming interface) that will allow a user to view an augmented reality experience (e.g., an AR content generator, etc.) related to a particular product while the user is browsing a product detail page (PDP). As further discussed herein, the present API may enable the creation of dynamic AR experiences (e.g., dynamic AR content generators) that provide several advantages. In embodiments, this requires efficiently finding (e.g., searching a database) augmented reality experiences (e.g., an AR content generator, etc.) that reference completed items / products at scale (e.g., high network traffic, number of requests, number of products, etc.).

[0161] As referred to herein, an "item" is stored by a product catalog service (e.g., see the description of the product catalog system). Figure 6 ). In embodiments, a product catalog contains items and is displayed in an online advertisement. Items may have specific attributes. For example, a hotel has a neighborhood, a product has a size, and a car has a vehicle identification number. As referred to herein, an item set may refer to a collection of products that match a filter provided by an advertiser, and in some embodiments, an online advertisement may refer to a specific item set.

[0162] In some embodiments, static augmented reality experiences (e.g., AR content generators, etc.), for example, 3D assets are "baked in" (e.g., stored therein) during creation of the augmented reality content generator, and the items that the augmented reality content generator may include may also be pre-defined during creation of the augmented reality content generator, which 1) is subject to predetermined size constraints, limiting the number of items / assets that the augmented reality content generator can have, and 2) thereby increasing the difficulty of mass-creating augmented reality content generators for each "set" of items, and a user may be required to create a new augmented reality content generator.

[0163] Fig.13 An example interface 1300 of an application for developing an augmented reality content generator (eg, for providing an AR experience for a given product) based on a user profile is shown.

[0164] In an embodiment, the application may be a program that enables editing of a given augmented reality content generated (including modifying various properties and interactive characteristics of a given augmented reality content generator, including AR content). In addition, the application may enable editing of product metadata (e.g., product ID, etc.) of a product associated with the augmented reality content generator.

[0165] As shown in interface 1300, various graphical elements are included to modify and configure various aspects of the AR content generator. Information related to the AR content generator is shown in interface 1300, including the name of the AR content generator and its identifier. As further shown in the figure, information related to the user profile is shown. In addition, statistical information related to the user profile is provided in interface 1300 (e.g., play, share, view, favorite, swipe up, scan subscribed users, etc.). As also shown in the figure, product-related information (e.g., domain, location, product name) is provided in interface 1300.

[0166] As shown in interface 1300, a graphical element is included to provide an option for indicating a user profile corresponding to an augmented reality content generator (e.g., which provides an AR experience) for a product (e.g., a "shopping" AR experience). In an example, interface 1300 is displayed when managing user profiles in an application or tool for developing an augmented reality experience.

[0167] In an embodiment, for a dynamic shopping augmented reality experience (e.g., an AR content generator, etc.), 3D assets are stored in a corresponding product catalog (e.g., see Figure 6 ). The AR content generator is used as a "container" where brands can define the AR product type (e.g., "shoes") and custom fields (e.g., "oversized boots") of the AR content generator as filters to determine the product types that are eligible to be displayed in the product catalog in the AR content generator (e.g., the set of these eligible products is called the "AR content generator item set"). At runtime, these assets will be dynamically loaded into the client (e.g., client device 102) and can undergo sorting or (random) selection (e.g., by a product recommendation service such as product recommendation module 708) to determine the product types to be displayed in the interface.

[0168] Based on how items are specified in the dynamic AR content generator, shopping-related AR content generators can have two subcategories:

[0169] Dynamic fixed: Fixed membership is controlled by a limited list of item identifiers allowed in the item set. Only these items are allowed in the item set. In the product catalog system, the item set membership used by the dynamic fixed shopping AR content generator is called fixed membership.

[0170] Dynamic and Flexible: Search filters are provided by the creator of the item set. If an item matches the filter, it will be placed in the item set. The supported filter properties depend on the catalog type associated with the item set. In a product catalog system, the item set membership used by the Dynamic and Flexible Shopping AR Content Generator is called dynamic membership.

[0171] Fig.14 An example interface 1400 of an application for developing an augmented reality content generator (eg, for providing an AR experience for a given product) based on a user profile is shown.

[0172] In some examples, domains define how products are grouped for selection and presented to the client. From a UI perspective, domains can be understood as multi-select elements. An AR content generator can have one or more domains, for example, if the AR content generator wants to show products in 2 areas (lips, eyeshadow), these areas will occupy 2 domains.

[0173] As shown in interface 1400, a set of graphical elements are provided to modify and edit products selected for domain(s) associated with an AR content generator. In this example, an option is provided for specifying a sales rank that determines the selection criteria for the product. Other graphical items provide the ability to add custom fields to the AR content generator and to add additional domains.

[0174] Fig.15 Example interfaces are shown in accordance with various implementations. Fig.15 The example interface of can be provided for display on a client device (e.g., client device), for example, through one or more interfaces of the messaging client application 104.

[0175] In this example, product information 1560 is included (e.g., product description, price, manufacturer, supplier, selection graphic item for viewing additional information, etc.) As further shown, additional information 1550 related to the product (e.g., description or product name) can be included in interface 1500.

[0176] A set of selectable graphic items including selectable graphic item 1570 is displayed, the set of selectable graphic items corresponding to respective product cards associated with different beauty products. In this example, the product cards are associated with the product catalog (e.g., selected by the product card) that has been selected based on the domain or other selection parameters as previously discussed. Figure 6 In an embodiment, the group of selectable graphic items is selected in this manner and presented in a carousel interface in interface 1500. In this example, selectable graphic item 1570 corresponding to a particular product card and an AR content generator is shown as being selected by the user. Different product cards can be selected from a group of selectable graphic items such as selectable graphic item 1576 or selectable graphic item 1578 from a carousel interface using appropriate input (e.g., touch input, tap, etc.). When a different product card is selected, a corresponding AR content generator can be provided for displaying its own associated information, AR content, AR effects, etc.

[0177] In embodiments, the present technology provides an API and SDK (Software Development Kit) to enable applications (e.g., e-commerce applications involving product catalogs and associated items) to provide an AR shopping experience.

[0178] More specifically, the present technology provides an extension to an existing camera API in the form of an API (e.g., a "shopping camera extension API" or "camera extension API" or "extension API"), which can be understood as enabling (external or third-party) applications to utilize and provide AR experiences, including basic camera functionality, as well as providing management and analytics related to such AR experiences. In an implementation, the functionality provided by the extension API does not overlap with the functionality provided by the existing camera API as further discussed herein, and the extension API works in conjunction with the camera API to facilitate (external) applications to provide shopping-related AR content generators and AR experiences in a manner that makes it easier for such AR content to be provided to users who are, for example, interested in products of a given brand (e.g., items offered in a product catalog via a product catalog service).

[0179] In an embodiment, the extension API processes product information (e.g., SKU) and starts the camera and provides a graphical interface that applies a given AR content generator in an automated manner without extensive development or configuration by the application to facilitate displaying products or items (e.g., from a product catalog) as AR content for viewing within the application. In an embodiment, this can be provided as a selectable graphical item in the graphical interface (e.g., a "Try it with AR" button or a "Try it now" button, etc.).

[0180] Fig.16 Conceptually illustrates example components and data flow diagrams for applying an augmented reality (AR) content generator and metadata delivery for the AR content generator in accordance with some implementations.

[0181] In data flow 1600, in order to apply an AR content generator using the camera API, an object referred to as an AR content generator (or "AR object" discussed in connection with the following figures) is used. In embodiments, the AR content generator object may include a simple descriptor containing all the information necessary for the camera API (e.g., at least partially using the camera API) to perform a set of operations (e.g., extracting content, assets, processing metadata, etc.) that ultimately results in AR content being rendered for display on a given client device.

[0182] In data flow 1650, the camera API repository is responsible for metadata delivery. The camera API repository provides the ability to retrieve AR objects by their identifiers and group identifiers. In an embodiment, the camera API repository may be implemented as a dedicated service for camera API partners (e.g., applications) that delivers metadata for AR content generators. In an embodiment, AR objects may be stored in a database (e.g., database 120).

[0183] Fig.17 Conceptually illustrates example components and data flow diagrams for resolving a group identifier for a group of AR content generators in accordance with some implementations.

[0184] In data flow 1700, the extension API also utilizes functionality provided by the camera API repository. In an implementation, the camera API repository accepts (e.g., receives input or as a parameter) a group identifier, which is a unique identifier representing a group of AR content generators. Such a group identifier can be parsed to determine the shopping AR content generator, which is then provided to the user (e.g., displayed in an interface). As shown, a product identifier such as a SKU is provided to the application, which proceeds to use the camera API repository to determine the group ID corresponding to the SKU. In an implementation, the camera API repository stores information associating SKUs with group IDs (e.g., by utilizing a lookup table (or database table or registry) that includes information associating a group of SKUs with their corresponding group IDs).

[0185] In an embodiment, a repository for the extended API is based on a service that provides AR content generator metadata for a given SKU, and such a service can inject such metadata to provide to the camera API, which then utilizes an auxiliary API ("private API") to store the group ID to the shopping AR content generator service (e.g., associated with one or more AR objects). In subsequent requests, the application then receives a special identifier associated with the SKU, which determines the group ID corresponding to the SKU by querying the extended API's repository. In turn, the camera API provides the group ID to the private API, which communicates with the shopping AR content generator service, which retrieves the AR content generator based on the group ID and enables the AR content generator to be accessed in the camera API repository. As shown, the AR object is provided from the private API to the camera API, which then sends the AR object to the application for display. In an implementation, the auxiliary API is a private API (as shown) that is only accessible by the camera API or the extended API (e.g., not publicly accessible). Therefore, in order to access Fig.17, the camera API sends (multiple) requests to the private API (e.g., with the group ID), which in turn communicates with the shopping AR content generator service and returns metadata corresponding to the group ID (e.g., the (multiple) AR content generators associated with the group ID). Therefore, the private API can provide an additional (abstract) layer to facilitate greater interoperability between the camera API and the shopping AR content generator service, while also improving security, because invalid requests are not handled by the private API, or vice versa (e.g., the private API does not accept invalid requests from the shopping AR content generator service).

[0186] Fig.18 Conceptually illustrated are example components and data flow diagrams for delivering product metadata in accordance with some implementations.

[0187] In data flow 1800, the extension API may query the shopping AR content generator service to determine the availability of a product, which results in providing relevant metadata (e.g., SKU) that can be used to query a group ID corresponding to a group of AR content generators (e.g., from at least Fig.17 In an implementation, a shopping camera configuration reference artifact (e.g., a type of data object) is provided that aggregates camera settings, shopping UI implementation, authorization, carousel UI, and basic business logic for AR shopping experience. In addition, the shopping camera configuration reference artifact will present a virtual try-on button to open the camera with the shopping AR content generator and related products.

[0188] Fig.19 Conceptually illustrates example components and data flow diagrams for applying an augmented reality content generator in accordance with some implementations.

[0189] In data flow 1900, the technology delivers product metadata from the backend to the extension API in order to draw (e.g., render to display) a set of product cards on the partner's application and provide product options to the URI handler (discussed further herein). In embodiments, this is accomplished by providing a product repository. In embodiments, such a product repository may be implemented in a manner similar to the product catalog system 124 (or product catalog service system 602) described above, and may provide one or more databases or tables (e.g., data structure 300 stored in database 120 of message server system 108).

[0190] Fig. 20 Conceptually illustrates example components and data flow diagrams for applying an augmented reality content generator in accordance with some implementations.

[0191] In data flow 2000, a shopping AR content generator component is provided to "glue" anything needed to handle the AR content generator. The URI handler corresponds to a mechanism for communication between a script for a given AR content generator (e.g., a component that enables a given AR content generator to respond to touch input, play animations and audio, modify scene objects, etc.) and a client application in an HTTP-like manner. In an implementation, the URI handler can be used by the shopping AR content generator to provide product metadata that can be included in (one or more) product cards. The processor corresponds to a mechanism required to listen to the AR content generator lifecycle through the shopping AR content generator service to attach / detach internal components and notify the consumer application, such as: prepare URI handlers, prepare card UI, etc. The shopping session is a session from the camera device API and reflects the ongoing session with the shopping AR content generator and is the holder of key API components.

[0192] Fig.21 21 is a flowchart illustrating a method according to certain example embodiments. The method 2100 may be implemented in computer-readable instructions for execution by one or more computer processors, such that the operations of the method 2100 may be performed by the messaging client application 104 (particularly with respect to the above-described Figure 7 ) or an application executed on a given client device (e.g., client device 102) in cooperative communication with the messaging server system 108 and its components; therefore, the method 2100 is described below by way of example with reference thereto. However, it should be understood that at least some of the operations of the method 2100 can be deployed on various other hardware configurations, and the method 2100 is not intended to be limited to the messaging client application 104 or any of the components or systems mentioned above.

[0193] At operation 2102 , the client device 102 generates an initial session to a camera using a camera API.

[0194] At operation 2104 , the client device 102 initiates frame processing using the camera API.

[0195] At operation 2106 , the client device 102 subscribes to events for frame processing using the camera API.

[0196] At operation 2108, the client device 102 generates a second session for shopping using the extension API.

[0197] At operation 2110 , the client device 102 configures a second session based on the first session.

[0198] Fig. 22 2200 according to some example embodiments. The method 2200 may be implemented in computer-readable instructions for execution by one or more computer processors, such that the operations of the method 2200 may be performed by the messaging client application 104 (particularly with respect to the above-described method 2200). Figure 7 The method 2200 is performed in part or in whole by an application executing on a given client device (e.g., client device 102) in cooperative communication with the messaging server system 108 and its components; therefore, the method 2200 is described below by way of example with reference thereto. However, it should be understood that at least some of the operations of the method 2200 can be deployed on various other hardware configurations, and the method 2200 is not intended to be limited to the messaging client application 104 or any of the components or systems mentioned above.

[0199] At operation 2202 , the client device 102 requests a group identifier (ID) from an extension application programming interface (API) based on the item identification indicator.

[0200] At operation 2204 , the client device 102 uses the extension API to request a set of augmented reality (AR) content generator IDs from the shopping AR content generator service based on the item identification indicator.

[0201] At operation 2206 , the client device 102 receives a set of AR content generator IDs based on the mapping of item identification indicators to the set of AR content generator IDs.

[0202] At operation 2208 , the client device 102 requests first metadata associated with a set of AR content generators and item identification indicators from the shopping AR content generator service using the extension API.

[0203] At operation 2210 , the client device 102 uses the extension API to request a set of products based on the first metadata.

[0204] At operation 2212, the client device 102 generates a group ID based on a set of products using the extension API.

[0205] At operation 2214, the client device 102 receives the group ID.

[0206] In an embodiment, the item identification indicator includes a stock keeping unit (SKU) of a product that is associated with a specific domain of products, the specific domain including beauty products.

[0207] In an embodiment, performing additional operations includes: mapping the item identification indicator to a set of AR content generator IDs through a shopping AR content generator service; and sending first metadata including the mapping of the item identification indicator to the set of AR content generator IDs through the shopping AR content generator service.

[0208] In an embodiment, performing the additional operations includes receiving the first metadata through an extension API.

[0209] In an embodiment, performing additional operations includes: determining, by the shopping AR content generator service, a set of products based on a set of AR content generator IDs; and mapping, by the shopping AR content generator service, the set of AR content generator IDs to second metadata including information related to the set of products.

[0210] In an embodiment, performing the additional operations includes sending specific information including the mapping to the extension API through the shopping AR content generator service.

[0211] In an embodiment, performing additional operations includes: requesting first metadata associated with a set of AR content generators and an item identification indicator from a shopping AR content generator service using an extension API; retrieving the first metadata based on the first metadata and the item identification indicator through the shopping AR content generator service; sending the first metadata to the extension API through the shopping AR content generator service, the first metadata including information about a second set of AR content generators associated with a camera device API; requesting registration of the second set of AR content generators to the camera device API using the extension API; and receiving confirmation of the registration of the second set of AR content generators from the camera device API through the extension API.

[0212] In an embodiment, the group ID is received by a third-party application provided by a third-party vendor that is independent of the extension API and the shopping AR content generator service.

[0213] In an embodiment, performing additional operations includes: requesting a second group of AR content generators from the camera device API by a third-party application based on the group ID; receiving the second group of AR content generators from the camera device API by the third-party application; and rendering a group of selectable graphic items in a carousel interface by the third-party application for display on the client device, the group of selectable graphic items corresponding to the second group of AR content generators.

[0214] In an embodiment, the second set of AR content generators is associated with a domain, the domain including beauty products.

[0215] Fig.232300 according to some example embodiments. The method 2300 may be implemented in computer-readable instructions for execution by one or more computer processors, such that the operations of the method 2300 may be performed by the messaging client application 104 (particularly with respect to the above-described method 2300). Figure 7 ) or an application executed on a given client device (e.g., client device 102) in cooperative communication with the messaging server system 108 and its components; therefore, the method 2300 is described below by way of example with reference thereto. However, it should be understood that at least some of the operations of the method 2300 can be deployed on various other hardware configurations, and the method 2300 is not intended to be limited to the messaging client application 104 or any of the components or systems mentioned above.

[0216] At operation 2302 , the client device 102 uses a camera API to request a group of augmented reality (AR) content generators based on a group ID generated by an extended application programming interface (API).

[0217] At operation 2304 , the client device 102 receives a set of AR content generators.

[0218] At operation 2306 , the client device 102 provides representations of a set of AR content generators for display in the interface.

[0219] At operation 2308 , the client device 102 receives a selection of a first AR content generator from a group of AR content generators.

[0220] At operation 2310 , the client device 102 renders a first AR content generator for display using the camera API.

[0221] At operation 2312 , the client device 102 uses the extension API to request metadata for a set of products based on the selected first AR content generator.

[0222] At operation 2314, the client device 102 receives metadata from the extension API.

[0223] At operation 2316 , the client device 102 provides for display a representation of a set of products based on the received metadata.

[0224] In an embodiment, the extension API is an API different from the camera API, and the extension API is used by a third party to provide an AR content generator related to shopping.

[0225] In an embodiment, the first AR content generator includes a descriptor including information for a camera API to perform a set of operations to render AR content for display on a client device.

[0226] In an embodiment, performing the additional operations includes: querying, by the extension API, a product repository to determine product availability; and receiving, by the extension API, metadata for a set of products.

[0227] In an embodiment, metadata for a set of products includes one or more of product information, product description, product price, or product image.

[0228] In an embodiment, performing additional operations includes: sending, by the product repository, product metadata to an extension API; sending, by the extension API, product status to a uniform resource identifier (URI) handler; and sending, by the URI handler, the product status to an AR content generator service.

[0229] In an embodiment, performing the additional operations includes: receiving, by the URI handler from the AR content generator service, second product metadata, the second product metadata being different from the product metadata; and sending, by the URI handler, the second product metadata to the extension API.

[0230] In an embodiment, performing the additional operations includes: receiving second metadata from the extension API; and providing for display a representation of the set of products based on the received second metadata.

[0231] In an embodiment, the second product metadata includes a set of additional AR content in response to interactions received from the user with representations of the set of products.

[0232] In an embodiment, the group ID is derived from a product identifier, which comprises a stock keeping unit (SKU).

[0233] Fig.24 2400 according to some example embodiments. The method 2400 may be implemented in computer-readable instructions for execution by one or more computer processors, such that the operations of the method 2400 may be performed by the messaging client application 104 (particularly with respect to the above-described method 2400). Figure 7206) or an application executed on a given client device (e.g., client device 102) in cooperative communication with the messaging server system 108 and its components; therefore, the method 2400 is described below by way of example with reference thereto. However, it should be understood that at least some of the operations of the method 2400 can be deployed on various other hardware configurations, and the method 2400 is not intended to be limited to the messaging client application 104 or any of the components or systems mentioned above.

[0234] At operation 2402 , the client device 102 receives a selection of a first representation of a product from a set of representations of products.

[0235] At operation 2404 , the client device 102 uses an extended application programming interface (API) to render based on the metadata to display a first representation of the product, the rendering including the following.

[0236] At operation 2406 , the client device 102 determines remotely stored augmented reality (AR) content using the extension API.

[0237] At operation 2408 , the client device 102 sends a first request for remote AR content to the camera API using the extension API.

[0238] At operation 2410 , the client device 102 sends a second request for retrieving remote AR content using the camera API.

[0239] At operation 2412 , the client device 102 renders the remote AR content using the camera API.

[0240] At operation 2414 , the client device 102 receives notification from the extension API that the remote AR content is rendered for display.

[0241] In an embodiment, the extension API is an API different from the camera API, and the extension API is used by a third party to provide an AR content generator related to shopping.

[0242] In an embodiment, performing the further operations includes requesting the remote AR content based on the request by the camera API.

[0243] In an embodiment, the remote AR content is provided by a third-party supplier associated with a product in the representation from a set of products.

[0244] In an embodiment, the remote AR content includes AR objects associated with products provided by third-party vendors.

[0245] In an embodiment, the remote AR content includes dynamic content that is updated regularly.

[0246] In an embodiment, performing additional operations includes: rendering representations of a group of products based on first metadata associated with the representations of the group of products before receiving a selection of the first representation, and wherein rendering to display the first representation also includes: sending a specific request to an extension API to apply remote AR content based on second metadata associated with the first representation of the product; in response to the specific request, mapping the second metadata to the remote AR content via the extension API, the remote AR content being stored at a separate storage device via a network hosted by a third-party provider; generating third metadata including a mapping of the second metadata to the remote AR content; and sending the third metadata by the extension API to the camera device API, the third metadata being included in the first request.

[0247] In an embodiment, rendering the first representation for display also includes: receiving, by the camera device API, third metadata included in the first request; retrieving the remote AR content based on the third metadata in response to the first request; sending, by the camera device API, a specific notification to the extension API confirming that the remote AR content is rendered; and sending, by the extension API, a notification that the remote AR content is rendered for display in response to the specific notification.

[0248] In an embodiment, the selection of the first representation is received by a third party application, the third party application being provided by a third party vendor, the third party application being independent of the camera API and the extension API.

[0249] In an embodiment, rendering the first representation of the product for display is initiated by a third party application provided by a third party vendor, the third party application being independent of the camera API and the extension API.

[0250] Fig.25 is a block diagram illustrating an example software architecture 2506 that may be used in conjunction with the various hardware architectures described herein. Fig.25 is a non-limiting example of a software architecture, and it will be appreciated that many other architectures may be implemented to facilitate the functionality described herein. Software architecture 2506 may be implemented in a variety of environments such as Fig.26 2600, which includes a processor 2604, a memory 2614, and an (input / output) I / O component 2618, etc. A representative hardware layer 2552 is shown and may represent, for example Fig.26The representative hardware layer 2552 includes a processing unit 2554 with associated executable instructions 2504. The executable instructions 2504 represent executable instructions of the software architecture 2506, including implementations of the methods, components, etc. described herein. The hardware layer 2552 also includes a memory or storage module memory / storage device 2556 also having executable instructions 2504. The hardware layer 2552 may also include other hardware 2558.

[0251] exist Fig.25 In the example architecture of , software architecture 2506 can be conceptualized as a stack of layers, in which each layer provides specific functionality. For example, software architecture 2506 may include layers such as operating system 2502, library 2520, framework / middleware 2518, application 2516, and presentation layer 2514. In operation, application 2516 or other components within these layers may call API call 2508 through the software stack and receive a response to API call 2508 as in message 2512. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile operating systems or dedicated operating systems may not provide framework / middleware 2518, while other operating systems may provide such a layer. Other software architectures may include additional or different layers.

[0252] The operating system 2502 can manage hardware resources and provide public services. The operating system 2502 may include, for example, a kernel 2522, a service 2524, and a driver 2526. The kernel 2522 may be used as an abstraction layer between the hardware and other software layers. For example, the kernel 2522 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. The service 2524 may provide other public services to other software layers. The driver 2526 is responsible for controlling or interfacing with the underlying hardware. For example, depending on the hardware configuration, the driver 2526 may include a display driver, a camera driver, drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0253] The library 2520 provides a common infrastructure used by the application 2516 or other components or layers. The library 2520 provides the following functions, which allow other software components to perform tasks in an easier way than directly interfacing with the functions of the underlying operating system 2502 (e.g., the kernel 2522, the service 2524, or the driver 2526). The library 2520 may include a system library 2544 (e.g., a C standard library), which may provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 2520 may include an API library 2546, such as a media library (e.g., a library that supports the presentation and manipulation of various media formats such as MPREG4, H.264, MP3, AAC, AMR, JPG, PNG), a graphics library (e.g., an OpenGL framework that can be used to present 2D and 3D graphics content on a display), a database library (e.g., SQLite that can provide various relational database functions), a web library (e.g., WebKit that can provide web browsing functions), etc. The library 2520 may also include various other libraries 2548 to provide numerous other APIs to the application 2516 and other software components / modules.

[0254] Framework / middleware 2518 (also sometimes referred to as middleware) provides a higher level common infrastructure that can be used by applications 2516 and / or other software components / modules. For example, framework / middleware 2518 can provide various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. Framework / middleware 2518 can provide a wide range of other APIs that can be used by applications 2516 or other software components / modules, some of which can be specific to a particular operating system 2502 or platform.

[0255] Applications 2516 include built-in applications 2538 or third-party applications 2540. Examples of representative built-in applications 2538 may include, but are not limited to, contact applications, browser applications, book reader applications, location applications, media applications, messaging applications, and / or game applications. Third-party applications 2540 may include applications that are developed by entities other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) for developing applications and can be used on platforms such as IOS TM ANDROID TM , Mobile software running on the mobile operating system of the phone or other mobile operating systems. The third-party application 2540 can call API calls 2508 provided by the mobile operating system (e.g., operating system 2502) to facilitate the functions described herein.

[0256] Applications 2516 may use built-in operating system functionality (e.g., kernel 2522, services 2524, or drivers 2526), ​​libraries 2520, and framework / middleware 2518 to create user interfaces to interact with users of the system. Alternatively or additionally, in some systems, interaction with the user may be done through a presentation layer such as presentation layer 2514. In these systems, the application / component "logic" may be separated from aspects of the application / component that interact with the user.

[0257] Fig.26 2600 is a block diagram illustrating components of a machine 2600 capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any one or more of the methodologies discussed herein, according to some example embodiments. Specifically, Fig.26 A diagrammatic representation of a machine 2600 is shown in the form of an example of a computer system in which instructions 2610 (e.g., software, programs, applications, applet, app, or other executable code) for causing the machine 2600 to perform any one or more of the methods discussed herein may be executed. Thus, the instructions 2610 may be used to implement the modules or components described herein. The instructions 2610 convert a general unprogrammed machine 2600 into a specific machine 2600 that is programmed to perform the functions described and shown in the described manner. In alternative embodiments, the machine 2600 operates as a standalone device or may be coupled (e.g., networked) to other machines. In a networked deployment, the machine 2600 may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 2600 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 2610 specifying actions to be taken by the machine 2600. In addition, while only a single machine 2600 is shown, the term "machine" should also be taken to include a collection of machines that individually or jointly execute instructions 2610 to perform any one or more of the methodologies discussed herein.

[0258] The machine 2600 may include a processor 2604 (including processors 2608 to 2612), a memory / storage device 2606, and an I / O component 2618, which may be configured to communicate with each other, for example, via a bus 2602. The memory / storage device 2606 may include a memory 2614, such as a main memory or other storage device, and a storage unit 2616, both of which the processor 2604 can access, for example, via the bus 2602. The storage unit 2616 and the memory 2614 store instructions 2610 that implement any one or more of the methods or functions described herein. The instructions 2610 may also reside completely or partially in the memory 2614, in the storage unit 2616, in at least one of the processors 2604 (e.g., in a cache memory of a processor), or in any suitable combination thereof during execution thereof by the machine 2600. Therefore, the memory 2614, the storage unit 2616, and the memory of the processor 2604 are examples of machine-readable media.

[0259] I / O components 2618 may include a variety of components that receive input, provide output, generate output, transmit information, exchange information, capture measurements, etc. The specific I / O components 2618 included in a particular machine 2600 will depend on the type of machine. For example, a portable machine such as a mobile phone will likely include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It will be appreciated that I / O components 2618 may include Fig.26 26. Many other components not shown in the drawings. The I / O components 2618 are grouped according to function only for the purpose of simplifying the following discussion and the grouping is by no means limiting. In various example embodiments, the I / O components 2618 may include output components 2626 and input components 2628. The output components 2626 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The input components 2628 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens that provide positioning and / or force of touch or touch gestures, or other tactile input components), audio input components (e.g., microphones), etc.

[0260] In other example embodiments, the I / O component 2618 may include a biometric component 2630, a motion component 2634, an environmental component 2636, or a positioning component 2638, as well as various other components. For example, the biometric component 2630 may include a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 2634 may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. Environmental component 2636 may include, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects concentrations of hazardous gases for safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. Positioning component 2638 may include a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure that can obtain altitude), an orientation sensor component (e.g., a magnetometer), etc.

[0261] Various technologies may be used to achieve communication. I / O components 2618 may include communication components 2640 that are operable to couple machine 2600 to network 2632 or device 2620 via coupling 2624 and coupling 2622, respectively. For example, communication components 2640 may include a network interface component or other suitable device to interface with network 2632. In other examples, communication components 2640 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Components (e.g., low power ), Device 2620 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).

[0262] In addition, the communication component 2640 can detect an identifier or include a component operable to detect an identifier. For example, the communication component 2640 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting the following items: a one-dimensional barcode such as a universal product code (UPC) barcode; a multi-dimensional barcode such as a Quick Response (QR) code, an Aztec code, a data matrix, a data glyph, a MaxiCode, a PDF417, a supercode, a UCC RSS-2D barcode, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 2640, such as a location via an Internet Protocol (IP) geolocation, ... Signal triangulation to obtain location, location obtained via detection of NFC beacon signals that can indicate a specific location, etc.

[0263] The following discussion refers to various terms or phrases that are referred to throughout the subject disclosure.

[0264] "Signal medium" refers to any intangible medium that is capable of storing, encoding or carrying instructions executed by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" should be deemed to include any form of modulated data signals, carrier waves, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode the information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure.

[0265] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, The coupling may be a network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other types of cellular or wireless couplings. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0266] A "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and produces corresponding output signals that are applied to operate a machine. For example, a processor may be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0267] "Machine storage medium" refers to a single or multiple storage devices or media (e.g., a centralized or distributed database, or associated cache memory and servers) that store executable instructions, routines, or data. Therefore, the above terms should be considered to include, but are not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, or device storage media include: non-volatile memory, including, for example, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium", "device storage medium", and "computer storage medium" mean the same thing and are used interchangeably in this disclosure. The terms "machine storage medium", "computer storage medium", and "device storage medium" explicitly exclude carrier waves, modulated data signals, and other such media, at least some of which are covered by the term "signal media".

[0268] "Component" refers to a device, physical entity, or logic with the following boundaries, which are defined by function or subroutine calls, branch points, APIs, or other technologies provided for partitioning or modularizing specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. Components can be packaged functional hardware units designed for use with other components and can be part of a program that generally performs specific functions in related functions. Components can constitute software components (e.g., codes embodied on machine-readable media) or hardware components. "Hardware components" are tangible units that can perform certain operations and can be configured or arranged in a certain physical manner. In various example embodiments, one or more computer systems (e.g., independent computer systems, client computer systems, or server computer systems) or one or more hardware components of a computer system (e.g., processors or processor groups) can be configured by software (e.g., applications or application parts) to operate as hardware components that perform certain operations as described herein. Hardware components can also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component may include a dedicated circuit system or logic that is permanently configured to perform certain operations. The hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component may also include a programmable logic or circuit system that is temporarily configured to perform certain operations by software. For example, the hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is uniquely customized to perform the configured function, and is no longer a general-purpose processor. It will be understood that the decision to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a circuit system that is temporarily configured (e.g., configured by software) can be promoted for cost and time considerations. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in some way or perform certain operations described herein. Considering an embodiment in which the hardware component is temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each hardware component in the hardware component at any time. For example, where a hardware component includes a general purpose processor that is configured as a special purpose processor by software, the general purpose processor may be configured as different special purpose processors (e.g., including different hardware components) at different times. The software configures a specific processor or multiple specific processors accordingly to, for example, constitute a specific hardware component at one time and constitute different hardware components at different times. A hardware component may provide information to other hardware components and receive information from other hardware components.Accordingly, the hardware components described can be considered to be coupled in communication. In the case of multiple hardware components simultaneously, communication can be achieved by signal transmission (e.g., by appropriate circuits and buses) between or among two or more hardware components. In an embodiment in which multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessible to multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device coupled in communication. Then, other hardware components can access the memory device at a subsequent time to retrieve and process the stored output. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (e.g., the collection of information). The various operations of the example methods described herein can be performed at least in part by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform related operations. Whether it is a temporary configuration or a permanent configuration, such a processor can constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described herein can be implemented at least in part by a processor, wherein one or more specific processors are examples of hardware. For example, at least some operations of the method can be performed by one or more processors or components implemented by the processor. In addition, one or more processors can also operate to support the execution of related operations in a "cloud computing" environment or operate as "software as a service" (SaaS). For example, at least some operations in the operation can be performed by a group of computers (as an example of a machine including a processor), wherein these operations can be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., API). The execution of certain operations can be distributed between processors, not only residing in a single machine, but also deployed across multiple machines. In some example embodiments, a processor or a component implemented by a processor can be located in a single geographical location (e.g., in a home environment, an office environment, or a server cluster). In other example embodiments, a processor or a component implemented by a processor can be distributed across multiple geographical locations.

[0269] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine, and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be sent or received over a network using a transmission medium via a network interface device.

[0270] "Computer-readable media" refers to both machine storage media and transmission media. Therefore, these terms include both storage devices / media and carrier waves / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure.

[0271] "Client device" refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. The client device can be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, a laptop computer, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communication device that a user can use to access the network. In the present disclosure, the client device is also referred to as an "electronic device". "Short-lived message" refers to a message that is accessible within a limited duration of time. A short-lived message can be a text, an image, a video, etc. The access time for a short-lived message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technology, the message is temporary.

Claims

1. A method comprising: requesting a group identifier (ID) from an extended application programming interface (API) based on the item identification indicator; Using the extension API, requesting a set of augmented reality (AR) content generator IDs from a shopping AR content generator service based on the item identification indicator; receiving the set of AR content generator IDs based on the mapping of the item identification indicators to the set of AR content generator IDs; requesting, using the extension API, from the shopping AR content generator service first metadata associated with the set of AR content generators and the item identification indicator; requesting a set of products based on the first metadata using the extended API; generating a group ID based on the set of products using the extension API; as well as The group ID is received.

2. The method according to claim 1, wherein: The item identification indicator includes a stock keeping unit (SKU) of a product associated with a specific domain of products, the specific domain including beauty products.

3. The method according to claim 1, further comprising: Mapping, by the shopping AR content generator service, the item identification indicator to the set of AR content generator IDs; as well as The first metadata including a mapping of the item identification indicators to the set of AR content generator IDs is sent by the shopping AR content generator service.

4. The method according to claim 3, further comprising: The first metadata is received by the extension API.

5. The method according to claim 1, further comprising: determining, by the shopping AR content generator service, the set of products based on the set of AR content generator IDs; as well as The set of AR content generator IDs are mapped, by the shopping AR content generator service, to second metadata including information related to the set of products.

6. The method according to claim 5, further comprising: The shopping AR content generator service sends specific information including the mapping to the extension API.

7. The method according to claim 1, further comprising: requesting, using the extension API, from the shopping AR content generator service first metadata associated with the set of AR content generators and the item identification indicator; retrieving, by the shopping AR content generator service, the first metadata based on the first metadata and the item identification indicator; The shopping AR content generator service sends the first metadata to the extension API, wherein the first metadata includes information related to a second group of AR content generators associated with the camera API; Registering the second set of AR content generators to the camera device API using the extended API request; as well as A confirmation of the registration of the second set of AR content generators is received by the extension API from the camera API.

8. The method according to claim 1, wherein: The group ID is received by a third-party application, which is provided by a third-party vendor, and the third-party application is independent of the extension API and the shopping AR content generator service.

9. The method according to claim 8, further comprising: The third-party application requests a second group of AR content generators from the camera device API based on the group ID; Receiving, by the third-party application, the second set of AR content generators from the camera device API; as well as The third-party application renders a set of selectable graphic items in a carousel interface for display on the client device, the set of selectable graphic items corresponding to the second set of AR content generators.

10. The method according to claim 9, wherein: The second group of AR content generators is associated with a domain, and the domain includes beauty products.

11. A system comprising: processor; as well as a memory including instructions that, when executed by the processor, cause the processor to perform operations including: requesting a group identifier (ID) from an extended application programming interface (API) based on the item identification indicator; Using the extension API, requesting a set of augmented reality (AR) content generator IDs from a shopping AR content generator service based on the item identification indicator; receiving the set of AR content generator IDs based on the mapping of the item identification indicators to the set of AR content generator IDs; requesting, using the extension API, from the shopping AR content generator service first metadata associated with the set of AR content generators and the item identification indicator; requesting a set of products based on the first metadata using the extended API; generating a group ID based on the set of products using the extension API; and The group ID is received.

12. The system according to claim 11, wherein: The item identification indicator includes a stock keeping unit (SKU) of a product associated with a specific domain of products, the specific domain including beauty products.

13. The system according to claim 11, wherein: The operations also include: mapping, by the shopping AR content creator service, the item identification indicator to the set of AR content creator IDs; and The first metadata including a mapping of the item identification indicators to the set of AR content generator IDs is sent by the shopping AR content generator service.

14. The system according to claim 13, wherein: The operations also include: The first metadata is received by the extension API.

15. The system according to claim 11, wherein: The operations also include: determining, by the shopping AR content generator service, the set of products based on the set of AR content generator IDs; and The set of AR content generator IDs are mapped, by the shopping AR content generator service, to second metadata including information related to the set of products.

16. The system of claim 15, wherein: The operations also include: The shopping AR content generator service sends specific information including the mapping to the extension API.

17. The system of claim 11, wherein: The operations also include: requesting, using the extension API, from the shopping AR content generator service first metadata associated with the set of AR content generators and the item identification indicator; retrieving, by the shopping AR content generator service, the first metadata based on the first metadata and the item identification indicator; The shopping AR content generator service sends the first metadata to the extension API, wherein the first metadata includes information related to a second group of AR content generators associated with the camera API; registering the second set of AR content generators to the camera API using the extended API request; and A confirmation of the registration of the second set of AR content generators is received by the extension API from the camera API.

18. The system of claim 11, wherein: The group ID is received by a third-party application, which is provided by a third-party vendor, and the third-party application is independent of the extension API and the shopping AR content generator service.

19. The system of claim 18, wherein: The operations also include: The third-party application requests a second group of AR content generators from the camera device API based on the group ID; receiving, by the third-party application, the second set of AR content generators from the camera API; and The third-party application renders a set of selectable graphic items in a carousel interface for display on the client device, the set of selectable graphic items corresponding to the second set of AR content generators.

20. A non-transitory computer-readable medium comprising instructions that, when executed by a computing device, cause the computing device to perform operations comprising: requesting a group identifier (ID) from an extended application programming interface (API) based on the item identification indicator; Using the extension API, requesting a set of augmented reality (AR) content generator IDs from a shopping AR content generator service based on the item identification indicator; receiving the set of AR content generator IDs based on the mapping of the item identification indicators to the set of AR content generator IDs; requesting, using the extension API, from the shopping AR content generator service first metadata associated with the set of AR content generators and the item identification indicator; requesting a set of products based on the first metadata using the extended API; generating a group ID based on the set of products using the extension API; as well as The group ID is received.

Images