Intake pipeline for augmented reality content generator

Abstract

The subject technology receives information about a product. The subject technology generates a 3D model file of the product in a first format. The subject technology converts the 3D model file into a 3D object file in a second format. The subject technology associates the 3D object file with the product in a product catalog service. The subject technology publishes an augmented reality (AR) content generator corresponding to the product.

Description

[0001] Priority requirements

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 085593, filed September 30, 2020, the entire contents of which are incorporated herein by reference for all purposes. Background Technology

[0003] With the increasing use of digital images, the affordability of portable computing devices, the availability of increased capacity digital storage media, and the increased bandwidth and accessibility of network connections, digital images have become an integral part of the daily lives of more and more people.

[0004] Brief description of several views in the attached figure

[0005] To facilitate identification of any discussion of a particular element or action, one or more of the highest-order digits in the figure references indicate the figure number in which the element was first introduced.

[0006] Figure 1 It is a graphical representation of a networked environment in which the present disclosure can be deployed, based on some example implementations.

[0007] Figure 2 It is a graphical representation of a message sending and receiving client application based on some example implementations.

[0008] Figure 3 It is a graphical representation of a data structure maintained in a database, based on some example implementations.

[0009] Figure 4 It is a graphical representation of a message based on some example implementations.

[0010] Figure 5 This is a flowchart of an access restriction process based on some example implementations.

[0011] Figure 6 This illustrates, according to some embodiments, additional information corresponding to a given message, such as... Figure 4 A schematic diagram of the structure of the message annotation described in the document.

[0012] Figure 7 This is a block diagram illustrating various modules of an annotation system according to certain example implementations.

[0013] Figure 8 An example interface for an application for developing an augmented reality content generator (e.g., for providing AR experiences) based on some implementations is shown.

[0014] Figure 9Examples of procedural techniques for generating assets, according to some implementations, are shown.

[0015] Figure 10 An example of a mask generated by a program according to some implementations is shown.

[0016] Figure 11 An example of a facial appearance generated programmatically according to some implementations is shown.

[0017] Figure 12 Examples of shape primitives (“primitives”) for a signed distance field (SDF) according to some implementations are shown.

[0018] Figure 13 An example of plotting using the signed distance function (SDF) according to some implementations is shown.

[0019] Figure 14 Other examples of plotting using the symbolic distance function (SDF) according to some implementations are shown.

[0020] Figure 15 An example of a mirrored RXGY mapping according to some implementations is shown.

[0021] Figure 16 and Figure 17 Examples of mapping types according to some implementation methods are shown.

[0022] Figure 18 This is a flowchart illustrating a method according to some example implementations.

[0023] Figure 19 This is a flowchart illustrating a method according to some example implementations.

[0024] Figure 20 This is a block diagram illustrating a software architecture in which the present disclosure can be implemented according to some example embodiments.

[0025] Figure 21 It is a graphical representation of a machine in the form of a computer system according to some example embodiments, in which a set of instructions can be executed to cause the machine to perform any or more of the methods discussed herein. Detailed Implementation

[0026] Users with broad interests from various locations can capture digital images of a wide range of objects and make the captured images available to others via a network (e.g., the Internet). Enhancing the user experience with digital images and providing various features so that computing devices can perform image processing operations on the various objects and / or features captured under varying conditions (e.g., changes in image scale, noise, lighting, motion, or geometric distortion) can be challenging and computationally intensive.

[0027] As described in the embodiments herein, augmented reality (AR) experiences can be provided in messaging client applications (or messaging systems). As further discussed herein, ingestion pipelines are provided for assets used by a specific group of AR content generators (e.g., facial makeup). AR content generators that can be categorized into this specific group are those AR content generators that can be applied to faces to convey a particular look or style, typically for beautifying purposes, and are products or sets of products.

[0028] In the example, developing an AR content generator for facial makeup products might require collecting a large number of discrete pieces of information, as well as collecting and / or generating a large number of creative assets to meet the needs of such an AR content generator. Therefore, a standardized data format for facial makeup products related to AR content generators can reduce discrepancies in the onboarding process and development process.

[0029] As further described herein, the subject matter advantageously provides a format known as the internal facial makeup format (IFM format), which enables the definition and construction of AR content items for facial makeup appearances using individual primitive shapes, which can be combined using techniques further described below to create a specific appearance.

[0030] As mentioned in this article, the phrases “augmented reality experience,” “augmented reality content item,” and “augmented reality content generator” include or refer to various image processing operations corresponding to image modification, filtering, LENSES, media overlay, transformation, etc., as further described in this article.

[0031] Figure 1 This is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages and associated content) over a network. The messaging system 100 includes multiple instances of client devices 102, each instance hosting multiple applications including a messaging client application 104. Each messaging client application 104 is communicatively coupled to other instances of the messaging client application 104 and a messaging server system 108 via a network 106 (e.g., the Internet).

[0032] The messaging client application 104 can communicate and exchange data with another messaging client application 104 and the messaging server system 108 via network 106. The data exchanged between messaging client applications 104 and between messaging client application 104 and messaging server system 108 includes functions (e.g., commands that call functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0033] Message transceiver server system 108 provides server-side functionality to a specific message transceiver client application 104 via network 106. While some functions of message transceiver system 100 are described herein as being performed by message transceiver client application 104 or by message transceiver server system 108, the location of certain functions within message transceiver client application 104 or message transceiver server system 108 is a design choice. For example, it may technically be preferred to initially deploy certain technologies and functions within message transceiver server system 108, but subsequently migrate those technologies and functions to message transceiver client application 104, in which client device 102 has sufficient processing capabilities.

[0034] The messaging server system 108 supports various services and operations provided to the messaging client application 104. Such operations include sending data to and receiving data from the messaging client application 104, and processing data generated by the messaging client application 104. As an example, this data may include message content, client device information, geolocation information, media annotations and overlays, message content persistence conditions, social network information, and live event information. Data exchange within the messaging system 100 is invoked and controlled via functions available through the user interface (UI) of the messaging client application 104.

[0035] Now, specifically to message transceiver server system 108, application programming interface (API) server 110 is coupled to application server 112 and provides a programming interface to application server 112. Application server 112 is communicatively coupled to database server 118, which facilitates access to database 120, which stores data associated with messages processed by application server 112.

[0036] Application Programming Interface (API) server 110 receives and sends message data (e.g., commands and message payloads) between client device 102 and application server 112. Specifically, API server 110 provides a set of interfaces (e.g., routines and protocols) that can be invoked or queried by messaging client application 104 to invoke functions of application server 112. Application Programming Interface (API) server 110 exposes various functions supported by application server 112, including account registration, login functionality, sending messages from one messaging client application 104 to another messaging client application 104 via application server 112, sending media files (e.g., images or videos) from messaging client application 104 to messaging server application 114 and possible access for another messaging client application 104, setting up collections of media data (e.g., stories), retrieving the friend list of the user of client device 102, retrieving such collections, retrieving messages and content, adding and deleting friends to the social graph, the location of friends within the social graph, and opening application events (e.g., involving messaging client application 104).

[0037] Application server 112 hosts multiple applications and subsystems, including message transceiver server application 114, image processing system 116, and social networking system 122. Message transceiver server application 114 implements several message processing techniques and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of message transceiver client application 104. As will be described in further detail, text and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or libraries). Message transceiver server application 114 then makes these collections available to message transceiver client application 104. Considering the hardware requirements for additional processor- and memory-intensive processing of data, such processing can also be performed on the server side by message transceiver server application 114.

[0038] Application server 112 also includes an image processing system 116, which is dedicated to performing various image processing operations typically on images or videos received within the payload of messages at message transceiver server application 114.

[0039] Social networking system 122 supports various social networking functions and services, and makes these functions and services available to message sending and receiving server application 114. To this end, social networking system 122 maintains and accesses entity graph 304 (such as...) within database 120. Figure 3(As shown). Examples of the functions and services supported by the social networking system 122 include identifying other users of the messaging system 100 with whom a particular user has a relationship or who a particular user “follows”, as well as identifying the interests of a particular user and other entities.

[0040] Application server 112 is communicatively coupled to database server 118, which facilitates access to database 120, in which data associated with messages processed by message sending and receiving server application 114 is stored.

[0041] Figure 2 This is a block diagram illustrating further details of a message sending and receiving system 100 according to an example implementation. Specifically, the message sending and receiving system 100 is shown as including a message sending and receiving client application 104 and an application server 112, which in turn include several subsystems, namely a short-timer system 202, a collection management system 204, and an annotation system 206.

[0042] The short-lived timer system 202 is responsible for implementing temporary access to content permitted by the message sending client application 104 and the message sending server application 114. To this end, the short-lived timer system 202 combines multiple timers that selectively display messages and associated content based on durations and display parameters associated with messages or sets of messages (e.g., stories), enabling access to messages and associated content via the message sending client application 104. Further details regarding the operation of the short-lived timer system 202 are provided below.

[0043] The collection management system 204 is responsible for managing collections of media (e.g., collections of text, image, video, and audio data). In some examples, collections of content (e.g., messages including images, videos, text, and audio) can be organized into “event galleries” or “event stories.” Such collections can be made available for a specified time period, such as the duration of the event to which the content relates. For example, content related to a concert can be made available as a “story” for the duration of a concert. The collection management system 204 can also be responsible for publishing icons that notify the user interface of the messaging client application 104 of the existence of a specific collection.

[0044] Furthermore, the collection management system 204 includes a curation interface 208, which enables the collection manager to manage and curate specific content collections. For example, the curation interface 208 allows an event organizer to curate content collections related to a specific event (e.g., removing inappropriate content or redundant messages). Additionally, the collection management system 204 employs machine vision (or image recognition technology) and content rules to automatically curate content collections. In some implementations, users may be paid compensation to include user-generated content in the collection. In such cases, the curation interface 208 operates to automatically pay such users for using their content.

[0045] Annotation system 206 provides various functions that enable users to annotate or otherwise modify or edit media content associated with messages. For example, annotation system 206 provides functions related to generating and publishing media overlays for messages processed by messaging system 100. Annotation system 206 can operablely supply media overlays or supplements (e.g., image filtering) to messaging client application 104 based on the geographic location of client device 102. In another example, annotation system 206 can operablely supply media overlays to messaging client application 104 based on other information (e.g., social network information of the user of client device 102). Media overlays can include audio and visual content as well as visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects can be applied to media content items (e.g., photos) at client device 102. For example, media overlays can include text that can be overlaid on a photograph taken by client device 102. In another example, media overlays include location identifiers (e.g., Venice Beach), names of live events, or business names (e.g., Beach Cafe). In yet another example, annotation system 206 uses the geographic location of client device 102 to identify media overlays that include the business name at the geographic location of client device 102. Media overlays may include other identifiers associated with the business. Media overlays may be stored in database 120 and accessed via database server 118.

[0046] In one example implementation, annotation system 206 provides a user-based publishing platform that allows users to select geographic locations on a map and upload content associated with those locations. Users can also specify the environment in which a particular media overlay should be provided to other users. Annotation system 206 generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geographic location.

[0047] In another example implementation, annotation system 206 provides a merchant-based publishing platform that enables merchants to select specific media overlays associated with geographic locations during the bidding process. For example, annotation system 206 associates the media overlay of the highest bidder with the corresponding geographic location within a predefined time period.

[0048] Figure 3 This is a schematic diagram illustrating a data structure 300 that can be stored in a database 120 of a message transceiver server system 108 according to some example embodiments. Although the contents of the database 120 are shown as including multiple tables, it should be understood that the data can be stored in other types of data structures (e.g., as an object-oriented database).

[0049] Database 120 includes message data stored in message table 314. Entity table 302 stores entity data, including entity diagram 304. Entities maintaining records in entity table 302 can include individuals, company entities, organizations, objects, locations, events, etc. Regardless of type, any entity storing data in message transceiver server system 108 can be an identifiable entity. Each entity is assigned a unique identifier and an entity type identifier (not shown).

[0050] Entity Graph 304 also stores information about the relationships and associations between entities. As an example only, such relationships could be social or professional relationships based on interests or activities (e.g., working in a common company or organization).

[0051] Database 120 also stores annotation data in annotation table 312 in the form of filters. Filters storing data in annotation table 312 are associated with and applied to videos (whose data is stored in video table 310) and / or images (whose data is stored in image table 308). In one example, a filter is an overlay displayed as an image or video during presentation to the receiving user. Filters can be of various types, including user-selected filters from a library of filters presented to the sending user by message transceiver client application 104 when the sending user is composing a message. Other types of filters include geolocation filters (also known as geographic filters), which can be presented to the sending user based on geographic location. For example, based on geographic location information determined by the GPS unit of client device 102, message transceiver client application 104 can present neighborhood-specific or location-specific geolocation filters within the user interface. Another type of filter is a data filter, which can be selectively presented to the sending user by message transceiver client application 104 based on other input or information collected by client device 102 during the message creation process. Examples of data filters include the current temperature at a specific location, the current speed of the user's movement, the battery life of the client device 102, or the current time.

[0052] Other annotation data that can be stored in image table 308 is augmented reality content generators (e.g., corresponding to application LENSES, augmented reality experiences, or augmented reality content items). Augmented reality content generators can be real-time special effects and sounds that can be added to images or videos.

[0053] As described above, augmented reality content generators, augmented reality content items, overlays, image transformations, AR images, and similar terms refer to modifications that can be made to a video or image. This includes real-time modifications, which modify an image as it is captured using the device's sensors and then display the modified image on the device's screen. It also includes modifications to stored content, such as video clips in a library that can be modified. For example, in a device that accesses multiple augmented reality content generators, a user can use a single video clip with multiple generators to see how different generators will modify the stored clip. For instance, by selecting different generators for the content, multiple generators applying different pseudo-random motion models can be applied to the same content. Similarly, real-time video capture can be used with the shown modifications to show how the video image currently captured by the device's sensors will modify the captured data. Such data can simply be displayed on the screen without being stored in memory, or the content captured by the device's sensors can be recorded and stored in memory with or without modification (or both). In some systems, preview features can show how different augmented reality content generators look simultaneously in different windows on the display. For example, this can make it possible to view multiple windows with different pseudo-random animations on the monitor at the same time.

[0054] Therefore, data, and various systems that use augmented reality content generators or other such transformation systems to modify content using that data, can involve: the detection of objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.); tracking such objects as they leave or enter the field of view in a video frame and move around the field of view; and modifying or transforming such objects while they are being tracked. In various implementations, different methods can be used to implement such transformations. For example, some implementations may involve generating a three-dimensional mesh model of one or more objects and using transformations of the models in the video and animated textures to implement the transformation. In other implementations, tracking points on objects can be used to place images or textures (which can be two-dimensional or three-dimensional) at the tracked locations. In yet another implementation, neural network analysis of video frames can be used to place images, models, or textures into content (e.g., images or frames of a video). Thus, augmented reality content generators involve both images, models, and textures for creating transformations in content and additional modeling and analysis information required to implement such transformations using object detection, tracking, and placement.

[0055] Real-time video processing can be performed using any type of video data (e.g., video streams, video files, etc.) stored in the memory of any type of computerized system. For example, a user can load a video file and store it in the device's memory, or the device's sensors can be used to generate a video stream. Additionally, computer-animated models can be used to process any object, such as a human face and body parts, animals, or inanimate objects (e.g., chairs, cars, or other objects).

[0056] In some implementations, when a specific modification is selected along with the content to be transformed, a computing device identifies the element to be transformed and then detects and tracks the element to be transformed if it exists in a frame of the video. The elements of the object are modified according to the modification request, thus transforming the frames of the video stream. The transformation of the video stream frames can be performed using different methods for different types of transformations. For example, for a transformation of frames that primarily involves changing the form of object elements, feature points are calculated for each element of the object (e.g., using an Active Shape Model (ASM) or other known methods). Then, a feature point-based mesh is generated for each of at least one element of the object. This mesh is used for subsequent stages of tracking object elements in the video stream. During tracking, the mesh for each mentioned element is aligned with the location of each element. Then, additional points are generated on the mesh. A first set of first points is generated for each element based on the modification request, and a second set of points is generated for each element based on the first set of points and the modification request. The frames of the video stream can then be transformed by modifying the elements of the object based on the first set of points, the second set of points, and the mesh. In such a method, the background of the modified object can also be changed or deformed by tracking and modifying the background.

[0057] In one or more embodiments, transformations of some regions of an object can be performed by calculating feature points for each element of the object and generating a mesh based on the calculated feature points. Points are generated on the mesh, and then various regions based on these points are generated. The elements of the object are then tracked by aligning the regions of each element with the positioning of at least one of the elements, and the properties of the regions can be modified based on modification requests, thereby transforming frames of the video stream. Depending on the specific modification request, the properties of the mentioned regions can be transformed in different ways. Such modifications may involve changing the color of the region; removing at least some portions of the region from the frames of the video stream; including one or more new objects into the region based on the modification request; and modifying or deforming the elements of the region or object. In various embodiments, any combination of such modifications or other similar modifications may be used. For certain models to be animated, some feature points can be selected as control points to determine the entire state space for options used in model animation.

[0058] In some implementations of computer animation models that use face detection to transform image data, a specific face detection algorithm (e.g., Viola-Jones) is used to detect faces in the image. Then, an Active Shape Modeling (ASM) algorithm is applied to the facial regions of the image to detect facial feature reference points.

[0059] In other implementations, other methods and algorithms suitable for face detection can be used. For example, in some implementations, landmarks are used to locate features that represent distinguishable points present in most of the images considered. For example, for facial landmarks, the location of the left pupil can be used. Secondary landmarks can be used if the initial landmark is not recognizable (e.g., if the person is wearing an eye patch). Such a landmark recognition process can be used for any such object. In some implementations, the set of landmarks forms a shape. The shape can be represented as a vector using the coordinates of points in the shape. One shape is aligned with another shape using a similarity transformation (allowing translation, scaling, and rotation) that minimizes the average Euclidean distance between the points of the shapes. The average shape is the mean of the aligned training shapes.

[0060] In some implementations, a search for landmarks begins from an average shape aligned with the position and size of the face determined by a global face detector. This search then repeats the steps of adjusting the localization of shape points by template matching of the image texture around each point to suggest a provisional shape, and then conforming the provisional shape to a global shape model until convergence occurs. In some systems, individual template matching is unreliable, and the shape model aggregates the results of weak template matchers to form a stronger overall classifier. The entire search is repeated at each level of the image pyramid, from coarse to fine resolution.

[0061] The transformation system can be implemented to capture image or video streams on a client device (e.g., client device 102) and perform complex image manipulations locally on client device 102 while maintaining an appropriate user experience, computation time, and power consumption. Complex image manipulations can include size and shape changes, emotion shifts (e.g., changing a face from frowning to smiling), state shifts (e.g., aging a subject, reducing apparent age, changing gender), style shifts, application of graphic elements, and any other suitable image or video manipulations implemented by a convolutional neural network that has been configured to execute efficiently on client device 102.

[0062] In some example implementations, a computer animation model for transforming image data may be used by a system in which a user can use a client device 102 with a neural network to capture an image or video stream of the user (e.g., a selfie), the neural network operating as part of a messaging client application 104 operating on the client device 102. A transformation system operating within the messaging client application 104 determines the presence of faces within the image or video stream and provides a modification icon associated with the computer animation model to transform the image data, or the computer animation model may exist in association with the interface described herein. The modification icon includes changes that can serve as the basis for modifying the user's face in the image or video stream as part of a modification operation. Once a modification icon is selected, the transformation system initiates processing to transform the user's image to reflect the selected modification icon (e.g., generating a smile for the user). In some implementations, once the image or video stream has been captured and the specified modification has been selected, the modified image or video stream can be presented in a graphical user interface displayed on a mobile client device. The transformation system may implement a sophisticated convolutional neural network on a portion of the image or video stream to generate and apply the selected modification. In other words, once the edit icon is selected, the user can capture an image or video stream and the results of the edits can be presented in real-time or near real-time. Furthermore, the edits can be continuous while the video stream is captured and the selected edit icon remains active. Machine-trained neural networks can be used to achieve such edits.

[0063] In some implementations, the graphical user interface (GUI) presenting the modifications performed by the transformation system can provide the user with additional interactive options. Such options may be based on an interface used to initiate content capture and selection for a specific computer animation model (e.g., initiated from a content creator user interface). In various implementations, modifications can persist after the initial selection of the modification icon. The user can turn the modification on or off by tapping or otherwise selecting the face modified by the transformation system and save it for later viewing or browsing to other areas of the imaging application. In the case of multiple faces being modified by the transformation system, the user can globally turn the modification on or off by tapping or selecting a single face modified and displayed within the GUI. In some implementations, individual faces within a set of multiple faces can be modified individually, or such modifications can be turned on individually, by tapping or selecting a single face or a series of individual faces displayed within the GUI.

[0064] In some example implementations, a graphics processing pipeline architecture is provided that enables the application of different augmented reality experiences (e.g., AR content generators) at corresponding different layers. Such a graphics processing pipeline provides a scalable rendering engine for providing multiple augmented reality experiences included in composite media (e.g., images or videos) for rendering by messaging client application 104 (or messaging system 100).

[0065] As mentioned above, video table 310 stores video data, which in one embodiment is associated with messages whose records are maintained in message table 314. Similarly, image table 308 stores image data associated with messages related to the message data stored in entity table 302. Entity table 302 can associate various annotations from annotation table 312 with various images and videos stored in image table 308 and video table 310.

[0066] Story table 306 stores data about messages and collections of associated image, video, or audio data, compiled into collections (e.g., stories or libraries). The creation of a specific collection can be initiated by a specific user (e.g., each user maintaining a record in entity table 302). Users can create "personal stories" in the form of collections of content that have already been created and sent / broadcast by that user. For this purpose, the user interface of messaging client application 104 may include user-selectable icons that allow the sending user to add specific content to his or her personal story.

[0067] Collections can also constitute "live stories," which are collections of content from multiple users created manually, automatically, or using a combination of manual and automatic technologies. For example, a "live story" can constitute a curated stream of user-submitted content from different locations and events. Users whose client devices have location services enabled and are at a common location event at a specific time can be presented with options to contribute content to a specific live story, for example, via the user interface of messaging client application 104. The messaging client application 104 can identify live stories to users based on their location. The end result is a "live story" told from a community perspective.

[0068] Another type of content collection is called a "location story," which allows users whose client devices 102 are located in a specific geographic location (e.g., at a university or on a university campus) to contribute to a specific collection. In some implementations, contributing to a location story may require a second level of authentication to verify that the end user belongs to a specific organization or other entity (e.g., is a student on a university campus).

[0069] Figure 4 This is a schematic diagram illustrating the structure of a message 400 according to some embodiments, which is generated by a message transceiver client application 104 for transmission to another message transceiver client application 104 or a message transceiver server application 114. The content of a particular message 400 is used to populate a message table 314 stored in a database 120, which is accessible by the message transceiver server application 114. Similarly, the content of the message 400 is stored in memory as “in transit” or “in flight” data of the client device 102 or application server 112. The message 400 is shown as including the following components:

[0070] Message Identifier 402: A unique identifier that identifies message 400.

[0071] Message text payload 404: The text to be generated by the user via the user interface of the client device 102 and included in message 400.

[0072] Message image payload 406: Image data captured by the camera component of the client device 102 or retrieved from the memory component of the client device 102 and included in message 400.

[0073] Message video payload 408: Video data captured by the camera device component or retrieved from the memory component of the client device 102 and included in message 400.

[0074] Message audio payload 410: Audio data captured by the microphone or retrieved from the memory component of the client device 102 and included in message 400.

[0075] Message annotation 412: Annotation data (e.g., filters, stickers, or other enhancements) representing annotations to be applied to message image payload 406, message video payload 408, or message audio payload 410 of message 400.

[0076] Message duration parameter 414: A parameter value, in seconds, indicating the amount of time that the content of the message (e.g., message image payload 406, message video payload 408, message audio payload 410) will be presented to the user or made accessible to the user via the message sending and receiving client application 104.

[0077] Message geolocation parameter 416: Geographic location data (e.g., latitude and longitude coordinates) associated with the message's content payload. Multiple message geolocation parameter values ​​416 may be included in the payload, each of which is associated with a content item included in the content (e.g., a specific image within the message image payload 406 or a specific video within the message video payload 408).

[0078] Message Story Identifier 418: An identifier value that identifies one or more sets of content (e.g., "story") associated with a specific content item in the message image payload 406 of message 400. For example, multiple images within the message image payload 406 may each be associated with multiple sets of content using the identifier value.

[0079] Message Tag 420: Each message 400 can be labeled with multiple tags, each tag indicating the subject of the content included in the message payload. For example, in the case where a specific image depicts an animal (e.g., a lion) is included in the message image payload 406, the tag value can be included within the message tag 420 indicating the relevant animal. The tag value can be manually generated based on user input, or it can be automatically generated using, for example, image recognition.

[0080] Message sender identifier 422: An identifier (e.g., message sending system identifier, email address, or device identifier) ​​indicating the user of the client device 102 on which message 400 is generated and from which message 400 is sent.

[0081] Message recipient identifier 424: An identifier (e.g., a messaging system identifier, email address, or device identifier) ​​indicating the user of the client device 102 to which message 400 is addressed.

[0082] The content (e.g., values) of each component of message 400 can be pointers to locations in tables where content data values ​​are stored. For example, image values ​​in message image payload 406 can be pointers to locations (or addresses) within image table 308. Similarly, values ​​in message video payload 408 can point to data stored in video table 310, values ​​in message annotation 412 can point to data stored in annotation table 312, values ​​in message story identifier 418 can point to data stored in story table 306, and values ​​in message sender identifier 422 and message receiver identifier 424 can point to user records stored in entity table 302.

[0083] Figure 5 This is a schematic diagram illustrating an access restriction process 500, according to which access to content (e.g., a short message 502 and a multimedia payload of associated data) or a collection of content (e.g., a short message group 504) can be time-restricted (e.g., short-lived).

[0084] A brief message 502 is shown to be associated with a message duration parameter 506, the value of which determines the amount of time the message sending and receiving client application 104 will display the brief message 502 to the receiving user. In one implementation, depending on the amount of time specified by the sending user using the message duration parameter 506, the receiving user may view the brief message 502 for up to 10 seconds.

[0085] Message duration parameter 506 and message receiver identifier 424 are shown as inputs to message timer 512, which is responsible for determining the amount of time for which brief message 502 is shown to a specific receiving user identified by message receiver identifier 424. Specifically, brief message 502 is shown to the relevant receiving user only within the time period determined by the value of message duration parameter 506. Message timer 512 is shown as providing output to a more generalized brief timer system 202, which is responsible for the overall timing of displaying content (e.g., brief message 502) to the receiving user.

[0086] Brief message 502 Figure 5The message group 504 is shown as a collection of messages included within a short message group 504 (e.g., a collection of messages in a personal story or event story). The short message group 504 has an associated group duration parameter 508, the value of which determines the duration for which the short message group 504 is presented and accessible to a user of the messaging system 100. For example, the group duration parameter 508 could be the duration of a concert, where the short message group 504 is a collection of content belonging to that concert. Alternatively, when setting up and creating the short message group 504, the user (owner user or curator user) can specify the value of the group duration parameter 508.

[0087] Additionally, each short message 502 within a short message group 504 has an associated group participation parameter 510, the value of which determines the duration for which the short message 502 is accessible within the context of the short message group 504. Therefore, a particular short message group 504 can "expire" and become inaccessible within its context before the short message group 504 itself expires according to the group duration parameter 508. The group duration parameter 508, the group participation parameter 510, and the message receiver identifier 424 each provide input to a group timer 514, which is operable to first determine whether a particular short message 502 of the short message group 504 will be displayed to a specific receiving user, and if so, determine for how long. Note that, as a result of the message receiver identifier 424, the short message group 504 also knows the identity of the specific receiving user.

[0088] Therefore, the group timer 514 operatively controls the total usage period of the associated ephemeral message group 504 and the individual ephemeral messages 502 included in the ephemeral message group 504. In one embodiment, each ephemeral message 502 within the ephemeral message group 504 remains viewable and accessible for a period of time specified by the group duration parameter 508. In another embodiment, a ephemeral message 502 may expire within the context of the ephemeral message group 504 based on the group participation parameter 510. Note that even within the context of the ephemeral message group 504, the message duration parameter 506 can still determine the duration for which a particular ephemeral message 502 is displayed to the receiving user. Therefore, the message duration parameter 506 determines the duration for which a particular ephemeral message 502 is displayed to the receiving user, regardless of whether the receiving user views the ephemeral message 502 within or outside the context of the ephemeral message group 504.

[0089] The short-lived timer system 202 can also operatively remove a specific short-lived message 502 from the short-lived message group 504 based on determining that the associated group participation parameter 510 has expired. For example, when the sending user has established a group participation parameter 510 for 24 hours from the date of publication, the short-lived timer system 202 will remove the relevant short-lived message 502 from the short-lived message group 504 after the specified 24 hours. The short-lived timer system 202 also operates to remove the short-lived message group 504 when the group participation parameter 510 for each short-lived message 502 within the short-lived message group 504 has expired, or when the short-lived message group 504 itself has expired according to the group duration parameter 508.

[0090] In certain use cases, the creator of a specific ephemeral message group 504 can specify an indefinite group duration parameter 508. In this case, the expiration of the group participation parameter 510 for the last remaining ephemeral message 502 within the ephemeral message group 504 will determine when the ephemeral message group 504 itself expires. In this case, adding a new ephemeral message 502 with a new group participation parameter 510 to the ephemeral message group 504 effectively extends the lifetime of the ephemeral message group 504 to a value equal to the group participation parameter 510.

[0091] In response to the short timer system 202 determining that a short message group 504 has expired (e.g., is no longer accessible), the short timer system 202 communicates with the messaging system 100 (and, in particular, the messaging client application 104) to cause the indicium (e.g., icon) associated with the relevant short message group 504 to no longer be displayed in the user interface of the messaging client application 104. Similarly, when the short timer system 202 determines that the message duration parameter 506 for a particular short message 502 has expired, the short timer system 202 causes the messaging client application 104 to no longer display the indicium (e.g., icon or text identifier) ​​associated with the short message 502.

[0092] As described above, media overlays such as LENSES, overlay, image transformation, AR image, and similar terms refer to modifications that can be made to a video or image. This includes real-time modifications, which modify an image as it is captured using the device's sensors and then display the modified image on the device's screen. It also includes modifications to stored content, such as video clips in a library that can be modified. For example, in a device that accesses multiple media overlays (e.g., LENSES), a user can use a single video clip with multiple LENSES to see how different LENSES will modify the stored clip. For example, by selecting different LENSES for the same content, multiple LENSES applying different pseudo-random motion models can be applied to the same content. Similarly, real-time video capture can be used in conjunction with illustrated modifications to show how the video image currently captured by the device's sensors will modify the captured data. Such data can be simply displayed on the screen without being stored in memory, or the content captured by the device's sensors can be recorded and stored in memory with or without modification (or both). In some systems, preview functionality can simultaneously show how different LENSES look in different windows of the display. For example, this can make it possible to view multiple windows with different pseudo-random animations on the monitor at the same time.

[0093] The data and various systems using LENSES or other such transformation systems to modify content can therefore involve: detection of objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.); tracking of such objects as they leave, enter, and move within the field of view of a video frame; and modification or transformation of such objects while they are being tracked. In various implementations, different methods can be used to implement such transformations. For example, some implementations may involve generating a 3D mesh model of one or more objects and using transformations of the models and animated textures in the video to implement the transformation. In other implementations, tracking points on objects can be used to place images or textures (which can be two-dimensional or three-dimensional) at the tracked locations. In further implementations, neural network analysis of video frames can be used to place images, models, or textures within content (e.g., images or video frames). Therefore, lens data involves both images, models, and textures used to create content transformations and additional modeling and analysis information required to implement such transformations through object detection, tracking, and placement.

[0094] Real-time video processing can be performed using any type of video data (e.g., video streams, video files, etc.) stored in the memory of any type of computerized system. For example, a user can load a video file and store it in the device's memory, or the device's sensors can be used to generate a video stream. Furthermore, computer-animated models can be used to process any object, such as a human face and parts of the human body, animals, or inanimate objects such as chairs, cars, or other objects.

[0095] In some implementations, when a specific modification is selected along with the content to be transformed, a computing device identifies the element to be transformed and then detects and tracks the element to be transformed if it exists in a frame of the video. The elements of the object are modified according to the modification request, thus transforming the frames of the video stream. The transformation of the video stream frames can be performed using different methods for different types of transformations. For example, for frame transformations primarily involving changing the form of object elements, feature points of each of the object's elements are calculated (e.g., using an Active Shape Model (ASM) or other known methods). A feature-point-based mesh is then generated for each of at least one element of the object. This mesh is used for subsequent stages of tracking the object elements in the video stream. During tracking, the mesh for each mentioned element is aligned with the location of each element. Additional points are then generated on the mesh. A first set of first points is generated for each element based on the modification request, and a second set of points is generated for each element based on the first set of points and the modification request. The frames of the video stream can then be transformed by modifying the elements of the object based on the first and second sets of points and the mesh. In such methods, the background of the modified object can also be changed or deformed by tracking and modifying the background.

[0096] In one or more embodiments, transformations of some regions of an object can be performed by calculating feature points for each element of the object and generating a mesh based on the calculated feature points. Points are generated on the mesh, and then various regions based on these points are generated. The elements of the object are then tracked by aligning the regions of each element with the positioning of at least one of the elements, and the properties of the regions can be modified based on modification requests, thereby transforming frames of the video stream. Depending on the specific modification request, the properties of the mentioned regions can be transformed in different ways. Such modifications may involve: changing the color of the region; removing at least some portions of the region from the frames of the video stream; including one or more new objects in the region based on the modification request; and modifying or deforming the elements of the region or object. In various embodiments, any combination of such modifications or other similar modifications can be used. For certain models to be animated, some feature points can be selected as control points to determine the entire state space for options used in model animation.

[0097] In some implementations of computer animation models that use face detection to transform image data, a specific face detection algorithm (e.g., Viola-Jones) is used to detect faces in the image. Then, an Active Shape Modeling (ASM) algorithm is applied to the facial regions of the image to detect facial feature reference points.

[0098] In other implementations, other methods and algorithms suitable for face detection can be used. For example, in some implementations, landmarks are used to locate features that represent distinguishable points present in most of the images considered. For example, for a face landmark, the location of the left pupil could be used. Secondary landmarks can be used if the initial landmark is not recognizable (e.g., if the person is wearing an eye patch). Such a landmark recognition process can be used for any such object. In some implementations, the set of landmarks forms a shape. The shape can be represented as a vector using the coordinates of points in the shape. One shape is aligned with another shape using a similarity transformation (allowing translation, scaling, and rotation) that minimizes the average Euclidean distance between the points of the shapes. The average shape is the mean of the aligned training shapes.

[0099] In some implementations, a search for landmarks begins from an average shape aligned with the position and size of the face determined by a global face detector. This search then repeats the steps of adjusting the localization of shape points by template matching of the image texture around each point to suggest a provisional shape, and then conforming the provisional shape to a global shape model until convergence occurs. In some systems, individual template matching is unreliable, and the shape model aggregates the results of weak template matchers to form a stronger overall classifier. The entire search is repeated at each level of the image pyramid, from coarse to fine resolution.

[0100] The transformation system can be implemented by capturing image or video streams on a client device and performing complex image manipulations locally on the client device, such as client device 102, while maintaining an appropriate user experience, computation time, and power consumption. Complex image manipulations can include size and shape changes, emotion transfers (e.g., changing a face from frowning to smiling), state transfers (e.g., aging a subject, reducing apparent age, changing gender), style transfers, application of graphic elements, and any other suitable image or video manipulations implemented by a convolutional neural network that has been configured to execute efficiently on the client device.

[0101] In some example implementations, a computer animation model for transforming image data may be used by a system in which a user can use a client device 102 with a neural network to capture an image or video stream of the user (e.g., a selfie), the neural network operating as part of a messaging client application 104 operating on the client device 102. A transformation system operating within the messaging client application 104 determines the presence of faces within the image or video stream and provides a modification icon associated with the computer animation model to transform the image data, or the computer animation model may exist in association with the interface described herein. The modification icon includes changes that can serve as the basis for modifying the user's face in the image or video stream as part of a modification operation. Once a modification icon is selected, the transformation system initiates processing to transform the user's image to reflect the selected modification icon (e.g., generating a smile for the user). In some implementations, once the image or video stream has been captured and the specified modification has been selected, the modified image or video stream can be presented in a graphical user interface displayed on a mobile client device. The transformation system may implement a sophisticated convolutional neural network on a portion of the image or video stream to generate and apply the selected modification. In other words, once the edit icon is selected, the user can capture an image or video stream and the results of the edits can be presented in real-time or near real-time. Furthermore, the edits can be continuous while the video stream is captured and the selected edit icon remains active. Machine-trained neural networks can be used to achieve such edits.

[0102] In some implementations, the graphical user interface (GUI) presenting the modifications performed by the transformation system can provide the user with additional interactive options. Such options may be based on an interface used to initiate content capture and selection for a specific computer animation model (e.g., an initiation from a content creator user interface). In various implementations, modifications can persist after the initial selection of the modification icon. The user can turn modifications on or off by tapping or otherwise selecting a face modified by the transformation system and save it for later viewing or browsing to other areas of the imaging application. In the case of multiple faces being modified by the transformation system, the user can globally turn modifications on or off by tapping or selecting a single face modified and displayed within the GUI. In some implementations, individual faces within a group of multiple faces can be modified individually, or such modifications can be turned on individually, by tapping or selecting a single face or a series of individual faces displayed within the GUI.

[0103] In some example implementations, a graphics processing pipeline architecture is provided that enables the application of different media overlays in corresponding different layers. Such a graphics processing pipeline provides a scalable rendering engine for providing multiple augmented reality content generators included in composite media (e.g., images or videos) for rendering by messaging client application 104 (or messaging system 100).

[0104] As discussed herein, the infrastructure of this topic supports the creation and sharing of interactive messages with interactive effects across various components of messaging system 100. In the example, to provide such interactive effects, a given interactive message may include image data as well as 2D or 3D data. The infrastructure described herein enables the provision of other forms of 3D and interactive media (e.g., 2D media content) across the topic system, allowing such interactive media to be shared across messaging system 100 and along with photo and video messages. In the example implementation described herein, messages may enter the system from a real-time camera device or via a storage device (e.g., messages with 2D or 3D content or augmented reality (AR) effects (e.g., 3D effects or other interactive effects) are stored in memory or a database). In the example of interactive messages with 3D data, the topic system supports motion sensor input and manages the transmission and storage of 3D data, as well as the loading of external effects and asset data.

[0105] As described above, the interactive message includes an image combining 2D or 3D effects and depth data. In the example implementation, in addition to conventional image textures, this subject system is used to render the message to visualize the spatial details / geometry seen by the camera device. When a viewer interacts with the message by moving a client device, the movement triggers a corresponding change in the viewer's perspective of the rendered image and geometry.

[0106] In implementation, the theme system provides the following AR effects (which may include 3D effects using 3D data or interactive 2D effects without using 3D data): it works with other components of the system to provide 3D geometry, particles, shaders, and 2D assets that can occupy different 3D planes in the message. In the example, the AR effects described herein are rendered to the user in real time.

[0107] As mentioned in this article, gyroscope-based interaction refers to a type of interaction in which the rotation of a given client device is used as input to change aspects of the effect (e.g., rotating a phone along the x-axis to change the color of light in a scene).

[0108] As mentioned in this article, augmented reality content generators refer to real-time special effects and / or sounds that can be added to messages and that utilize AR effects and / or other 3D content (such as 3D animated graphic elements), 3D objects (such as non-animated objects), etc., to modify images and / or 3D data.

[0109] The following discussion involves example data of such message storage according to some implementation methods.

[0110] Figure 6 This illustrates, according to some embodiments, the additional information generated by the message sending and receiving client application 104, including information corresponding to a given message, as shown above. Figure 4 A schematic diagram of the structure of message annotation 412 as described in the document.

[0111] In the implementation method, such as Figure 3 The following are included Figure 6 The content of the specific message 400, as shown in the additional data, is used to populate the message table 314 for the given message, which is stored in database 120 and can then be accessed by the message sending and receiving client application 104. For example... Figure 6 As shown, message annotation 412 includes the following components corresponding to various types of data:

[0112] Augmented Reality (AR) Content Identifier 652: Identifier of the AR content generator used in the message.

[0113] o Message identifier 654: The identifier of the message

[0114] o Asset Identifier 656: A set of identifiers for assets in the message. For example, corresponding asset identifiers may be included for assets determined by a specific AR content generator. In an implementation, such assets are created by the AR content generator on the sending client device, uploaded to the message transceiver server application 114, and used on the receiving client device to recreate the message. Examples of typical assets include:

[0115] ■ Raw still RGB images captured by a camera device

[0116] ■ Post-processed images with AR content generator effects applied to the original images

[0117] ○ Augmented Reality (AR) Content Metadata 658: Additional metadata associated with the AR content generator corresponding to AR identifier 652, such as:

[0118] ○AR Content Generator Category: Corresponds to the type or classification of a specific AR content generator.

[0119] ○AR Content Generator Conveyor Index

[0120] ○ Conveyor Group: When a qualified captured AR content generator is inserted into the conveyor interface, a conveyor group can be entered and used. In the implementation, a new value "AR_DEFAULT_GROUP" (for example, the default group assigned to a specific AR content generator) can be added to the list of valid group names, and other selected AR content generators can be included in the group.

[0121] ○ This corresponds to the captured metadata 660, such as:

[0122] ○Image metadata of camera device

[0123] ■ Internal data of the camera device

[0124] ●focal length

[0125] ●Main Point

[0126] ■ Other camera device information (e.g., camera device location)

[0127] ○ Sensor Information

[0128] ■Gyroscope sensor data

[0129] ■ Positioning sensor data

[0130] ■Accelerometer sensor data

[0131] ■Other sensor data

[0132] ■ Position sensor data

[0133] Figure 7 This is a block diagram 700 illustrating various modules of an annotation system 206 according to some example embodiments. Annotation system 206 is shown as including an image data receiving module 702, a sensor data receiving module 704, an image data processing module 706, an augmented reality (AR) effects module 708, a rendering module 710, and a sharing module 712. The various modules of annotation system 206 are configured to communicate with each other (e.g., via a bus, shared memory, or switch). Any one or more of these modules can be implemented using one or more computer processors 720 (e.g., by configuring one or more such computer processors to perform the functions described for that module), and therefore may include one or more computer processors 720 (e.g., a collection of processors provided by client device 102).

[0134] Any one or more modules described may be implemented individually using hardware (e.g., one or more of the computer processors 720 of a machine (e.g., machine 2100)) or a combination of hardware and software. For example, any described module of annotation system 206 may physically include an arrangement of one or more computer processors 720 (e.g., a subset of one or more computer processors of a machine (e.g., machine 2100) or one or more computer processors thereof) configured to perform the operations described herein for that module. As another example, any module of annotation system 206 may include software, hardware, or both software and hardware that configures an arrangement of one or more computer processors 720 (e.g., in one or more computer processors of a machine (e.g., machine 2100) to perform the operations described herein for that module. Thus, different modules of annotation system 206 may include and configure different arrangements of such computer processors 720 or a single arrangement of such computer processors 720 at different points in time. Furthermore, any two or more modules of annotation system 206 may be combined into a single module, and the functionality described herein for a single module may be subdivided among multiple modules. Furthermore, based on various example implementations, modules described herein as being implemented in a single machine, database, or device can be distributed across multiple machines, databases, or devices.

[0135] Image data receiving module 702 receives image data and depth data captured by client device 102. For example, the image is a photograph captured by an optical sensor (e.g., a camera) of client device 102. The image includes one or more real-world features, such as a user's face or a real-world object detected in the image. In some implementations, the image includes metadata describing the image.

[0136] Sensor data receiving module 704 receives sensor data from client device 102. Sensor data is any type of data captured by sensors of client device 102. In this example, sensor data may include motion of client device 102 collected by a gyroscope, touch input or gesture input from a touch sensor (e.g., a touchscreen), GPS, or another sensor of client device 102 describing its current geographic location and / or motion. As another example, sensor data may include temperature data indicating the current temperature, such as that detected by sensors of client device 102. As yet another example, sensor data may include light sensor data indicating whether client device 102 is in a dark or bright environment.

[0137] The image data processing module 706 performs operations on the received image data. For example, the image data processing module 706 performs various image processing operations, which will be discussed further herein.

[0138] The AR effects module 708 performs various operations based on algorithms or techniques corresponding to animation and / or providing visual and / or auditory effects to received image data, which will be described further herein. In one implementation, a given augmented reality content generator may utilize the AR effects module 708 to perform operations for rendering AR effects (e.g., including 2D or 3D effects).

[0139] Rendering module 710 performs message rendering for display by message sending and receiving client application 104 based on data provided by at least one of the previously mentioned modules. In the example, rendering module 710 utilizes a graphics processing pipeline to perform graphics operations to render the message for display. In the example, rendering module 710 implements a scalable rendering engine that supports multiple image processing operations corresponding to various augmented reality content generators.

[0140] In some implementations, the rendering module 710 provides a graphics system for rendering two-dimensional (2D) objects or objects from a three-dimensional (3D) world (real or fictional) onto a 2D display screen. In some implementations, such a graphics system (e.g., a graphics system included on the client device 102) includes a graphics processing unit (GPU) for performing image processing operations and rendering graphical elements for display.

[0141] In implementations, the GPU includes a logical graphics processing pipeline that can receive a representation of a 2D or 3D scene and provide a bitmap output representing a 2D image for display. Existing application programming interfaces (APIs) have implemented graphics pipeline models. Examples of such APIs include the Open Graphics Library (OPENGL) API and the Metal API. The graphics processing pipeline comprises several stages that transform a set of vertices, textures, buffers, and state information into image frames on the screen. In implementations, one stage of the graphics processing pipeline is the shader, which can be used as part of a specific augmented reality content generator applied to the input frame (e.g., an image or video). Shaders can be implemented as code running on a dedicated processing unit (also called a shader unit or shader processor) that typically executes several computation threads, programmed to generate appropriate levels of color and / or special effects for the fragment being rendered. For example, a vertex shader processes the properties of vertices (position, texture coordinates, color, etc.), and a pixel shader processes the properties of pixels (texture values, color, z-depth, and alpha (α) values). In some instances, the pixel shader is called a fragment shader.

[0142] It should be understood that other types of shader processing can be provided. In the example, the entire frame is rendered using a specific sampling rate within the graphics processing pipeline, and / or pixel shading is performed at a specific per-pixel rate. In this way, a given electronic device (e.g., client device 102) operates the graphics processing pipeline to transform information corresponding to an object into a bitmap that can be displayed by the electronic device.

[0143] The sharing module 712 generates messages for storage and / or sending to the message transceiver server system 108. The sharing module 712 enables the sharing of messages with other users and / or client devices of the message transceiver server system 108.

[0144] In one implementation, the augmented reality content generator module 714 displays optional graphic items in a conveyor belt arrangement. As an example, a user can use various inputs to rotate optional graphic items onto and off the display screen in a manner corresponding to a conveyor belt providing a cyclical view of the graphic items. The conveyor belt arrangement allows multiple graphic items to occupy specific graphic areas on the display screen. In this example, the augmented reality content generator can be organized into corresponding groups to be included on the conveyor belt arrangement, thereby enabling rotation of the augmented reality content generator by group.

[0145] In the implementation described herein, by using depth data and image data, 3D face and scene reconstruction can be performed by adding a Z-axis dimension (e.g., depth dimension) to a regular 2D photograph (e.g., X-axis and Y-axis dimensions). This format allows viewers to interact with the message, change the angle / viewpoint of the message rendered by this subject system, and affect the particles and shaders used in rendering the message.

[0146] In the example, viewer interaction input comes from motion while viewing the message (e.g., from the motion sensors of the device displaying the message to the viewer), which is then translated into changes in perspective regarding how content, particles, and shaders are rendered. Interaction can also come from touch gestures on the screen and other device motions.

[0147] In such a user interface implementation, optional graphic items can be presented in a conveyor belt arrangement, in which a portion or subset of the optional graphic items is visible on the display screen of a given computing device (e.g., client device 102). As an example, a user can utilize various inputs to rotate optional graphic items onto and off the display screen in a manner corresponding to a conveyor belt providing a cyclical view of the graphic items. Therefore, the conveyor belt arrangement provided in the user interface allows multiple graphic items to occupy specific graphic areas on the display screen.

[0148] In the example, corresponding AR experiences for different AR content generators can be organized into groups to be included on a conveyor belt arrangement, allowing rotation through the media overlay by group. While a conveyor belt interface is provided as an example, it is understood that other graphical interfaces can also be used. For example, a group of augmented reality content generators could include a list of graphics, a scrolling list, scrolling graphics, or another graphical interface that allows navigation through various graphic items for selection. As used herein, a conveyor belt interface refers to displaying graphic items in an arrangement similar to a circular list, allowing navigation through the circular list based on user input (e.g., touch or gesture) to select or scroll through graphic items. In the example, a set of graphic items can be presented on a horizontal (or vertical) line or axis, where each graphic item is represented as a specific thumbnail image (or icon, avatar, etc.). At any given time, some graphic items in the conveyor belt interface can be hidden. If a user wants to view the hidden graphic items, in the example, the user can provide user input (e.g., touch, gesture, etc.) to scroll through the graphic items in a specific direction (e.g., left, right, up, or down). Subsequently, a follow-up view of the conveyor belt interface is displayed, wherein animations are provided or rendered to present one or more additional graphic items to be included on the interface, and wherein some of the previously presented graphic items may be hidden in this follow-up view. In this implementation, the user can navigate back and forth through the set of graphic items in a circular manner. Therefore, it can be understood that the conveyor belt interface can optimize screen space by displaying only a subset of images from the set of graphic items in the cyclical view.

[0149] In one implementation, the augmented reality content generator is included on a conveyor belt arrangement (or another interface as discussed above), allowing rotation through the generator. Furthermore, the augmented reality content generator can be selected for inclusion based on various signals, including, for example, time, date, geographic location, and metadata associated with the media content.

[0150] Figure 8 An example interface of an application 800 (e.g., development tool) for developing augmented reality content generators (e.g., for providing AR experiences) based on some implementations is shown.

[0151] As shown in the interface of application 800, various graphical elements are included for modifying and configuring various aspects of the AR content generator. As further shown, a preview of the AR content generator is displayed in the set of graphical elements 810, which, when provided for display on a given client device (e.g., client device 102), can advantageously provide a simulated representation of the AR content generator. In an implementation, the application can be a program that enables editing of the generated given augmented reality content, including modifying various attributes and interactive features of the given augmented reality content generator, including the AR content. Furthermore, the application can enable editing of product metadata (e.g., product ID, etc.) associated with the augmented reality content generator.

[0152] As described below, the subject technique provides an implementation of a procedural graph that can be used by application 800 to programmatically (e.g., procedurally) describe masks and textures applied to a given facial mesh (or a general mesh). For example, this approach advantageously enables programmatic updates, allowing for faster adoption of new appearances while reducing the space occupied by images serving as maps of specific regions and avoiding the manual creation of the graph.

[0153] Figure 9 Examples of procedural techniques for generating assets, according to some implementation methods, are shown. Figure 9 Examples of this can be performed by the application 800 discussed above. However, it should be understood that in some implementations, such examples can be performed by the message sending and receiving client application 104 or by components of the message sending and receiving system 100.

[0154] exist Figure 9 In the example, the process flow is executed by application 800, which is configured to perform procedural methods using a shader mirrored position map across the face mesh. This position map covers all areas of the face and scales appropriately as the face mesh moves closer to or further away from the camera device of client device 102 in the scene. This method abandons the operation of individually associating the point positions of the face mesh UV map with different areas of the face and avoids the (manual) setting of mapped face regions.

[0155] At 900, the target face is received by application 800. At 920, a mesh is applied by application 800. At 940, a color map is used to map the mirrored region to the mesh. In this implementation, a first region is mapped to an X-axis value and associated with a first color (e.g., red), and a second region is mapped to a Y-axis and associated with a second color (e.g., green). At 960, application 800 generates JSON and provides this JSON to the shader pre-mapped to the face mesh. As described herein, JSON is JavaScript Object Syntax, a lightweight data-interchange format. In this example, JSON is a text-based format used to represent structured data using JavaScript object syntax and can be used to transfer data in web applications (e.g., sending data from a server to a client for display on a webpage and vice versa).

[0156] In existing implementations, the set of (manual) operations that occur after performing the above steps involve (manually or temporarily) creating an image using mapped facial regions. Specifically, this may require an image authoring application or image editor to generate the image. Furthermore, the image is exported at a specific resolution and then imported into the application. The image is then associated with a specific facial mesh using a material with UV-mapped points.

[0157] To avoid performing the above operations, generating specific masks directly through shaders provides an alternative solution. In this implementation, mirror mapping using the red-X-green-Y method allows for simultaneous shader-based procedural rendering on both sides of the face to create a symmetrical appearance. As described herein, an "appearance" or "multiple appearances" refers to (e.g., using the techniques described herein) a collection of augmented reality content items (e.g., graphic items, graphic effects including 2D or 3D effects, etc.) generated and rendered onto facial features or facial regions. As further mentioned herein, an "appearance" or "multiple appearances" may also be referred to as an "AR facial pattern" or "AR facial makeup pattern."

[0158] In the example, the creation process for generating augmented reality content items occurs within a given augmented reality content generator and is scaled to the mesh to which the augmented reality content generator is being applied (e.g., a face mesh). Therefore, the mask is rendered at the optimal resolution, and specific rendering of the mask outside the augmented reality content generator is avoided.

[0159] Figure 10 An example of a mask generated by a program according to some implementations is shown. Figure 10 Examples of this can be performed by the application 800 discussed above. However, it should be understood that in some implementations, such examples can be performed by the message sending and receiving client application 104 or by components of the message sending and receiving system 100.

[0160] exist Figure 10 In the example, the process flow is executed by application 800 to provide a procedurally generated mask using a symbolic distance field (SDF) as described below. As described herein, an SDF is a function that takes a location as input and outputs the distance from that location to the nearest part of the shape. To create reusable pre-structured masks, techniques are employed as described in the following discussion.

[0161] like Figure 10 As shown, at 1000, a triangle SDF using customizable points is illustrated. At 1020, the triangle is mapped in RX-GY space and blurred. At 1040, RX-GY space is mapped to the face mesh. At 1060, a smoothed triangle appears on the representation of the face (e.g., rendered).

[0162] In this implementation, a triangular SDF is used to create a variety of shapes beyond the general triangle. SDF, as the name suggests, is a distance field, and therefore distances outside the triangle can also be used to create fainter edges, as is used in many appearance applications. This technique is referred to herein as smoothing and provides unique use cases for leveraging SDF.

[0163] Figure 11 An example of a facial appearance generated programmatically according to some implementations is shown. Figure 11 Examples of this can be performed by the application 800 discussed above. However, it should be understood that in some implementations, such examples can be performed by the message sending and receiving client application 104 or by components of the message sending and receiving system 100.

[0164] As shown, the eyeshadow appearance 1100 was reconstructed using only two SDF triangles mapped using the techniques described above. Some appearances can use more smoothing than others. The curvature appearing on some of these appearances can be achieved by relying on the natural curvature of the mesh and a combination of superimposing two triangles in the appropriate positions. In this example, the cutout of the eye is due to the absence of triangles in that particular area of ​​the mesh.

[0165] As further shown, appearance 1120 includes AR content items (e.g., eyeshadow, blush, and lipstick) and is procedurally generated using multiple (e.g., 5) mirrored smooth triangles.

[0166] Figure 12 Examples of shape primitives (“primitives”) for the symbolic distance function (SDF) according to some implementations are shown.

[0167] As shown, the SDF primitives include a triangle primitive 1200, a circle primitive 1220, a circular intersection primitive 1240 (e.g., an ellipse with pointed ends), and an irregular sac primitive 1260. In implementations, to create different complex shapes, the application 800 can manipulate and combine different primitives to achieve the desired effect.

[0168] While the triangle primitive 1200 offers sufficient flexibility in practical implementations to serve as the default starting point for masks that approximate the appearance of facial makeup, alternative SDF patterns also have advantages. Some of the aforementioned SDF primitives can better approximate a given appearance in terms of shape with fewer adjustments to placement and smoothing, while others may require less computational overhead and thus enable more complex masks with a wider range of shapes.

[0169] As described above, an ingestion pipeline is provided for assets used by a specific group of AR content generators (e.g., facial makeup). AR content generators that can be categorized into this specific group are those AR content generators that can be applied to faces to convey a particular look or style, typically for the purpose of beautification, and are products or product sets.

[0170] Furthermore, as mentioned earlier, the subject matter technology advantageously provides a format known as the Internal Facial Makeup Format (IFM format), which enables the definition and construction of AR content items for facial makeup appearances using individual primitive shapes, which can be combined using techniques further described below to create a specific look.

[0171] As described in this article, the IFM format is a shader-based method for programmatically generating facial beauty patterns. In its implementation, IFM uses formatted JSON to parameterize fragment elements that can be combined in layers to create patterns on the face similar to those in a makeup application.

[0172] An appearance can be defined as the result of applying one or more IFM-format masks to a facial mesh that has its own IFM-format properties. Applying appearances to facial meshes in augmented reality provides consumers with the ability to virtually try on facial cosmetics. The individual masks described here are created separately but will be combined later to create the appearance. Each of these individual masks has the ability to accept certain available properties. Examples of available properties include the following:

[0173]

[0174] In the example, the IFM format mask is a carefully crafted shape using the values ​​mentioned above. IFM format mask elements can be targeted to specific fields, and not all masks are able to accept all available IFM format attributes. Examples of mask fields include the following:

[0175]

[0176]

[0177] Using these masks as ratio maps applied to IFM format attributes allows these attributes (e.g., color) to be applied to corresponding areas in the desired mapping pattern. In this way, scalable solutions for creating masks allow for the automated creation of AR content generators.

[0178] In the example, the IFM format is a shader-based method for programmatically generating facial beauty patterns. In its implementation, the IFM format uses formatted JSON to parameterize fragment elements that can be combined in layers to create patterns on the face resembling makeup applications.

[0179] Figure 13 An example using the symbolic distance function (SDF) is shown according to some implementations. Figure 13 Examples of this can be performed by the application 800 discussed above. However, it should be understood that in some implementations, such examples can be performed by the message sending and receiving client application 104 or by components of the message sending and receiving system 100.

[0180] The UV mapping quadrilateral—which is colored to represent linearly interpolated UV values ​​as colors—can use red values ​​as the X-axis and green values ​​as the Y-axis, keeping Z constant at 0. Now, this is a pixel map showing all locations on this quadrilateral. In this case, the bottom left corner represents the RGB value (0,0,0), ignoring the blue value; red and green values ​​can be used as X and Y, and in this case, (0,0). Similarly, the top right corner converts rgb(1,1,0) to xy(1,1). This mapping format is referred to as RXGY in this paper.

[0181] In the example, the implicit shape can be drawn internally within the shader based on the RXGY graph above using the signed distance function (SDF)—sometimes also called the signed distance field. This is discussed below. Figure 13 The example illustrates a series of mathematical steps performed to draw a smooth circle inside the shader based on the circle SDF, which can be described as "GLSL" below:

[0182] / / GLSL Signed Distance Function for a circle

[0183] float SDF_circle(vec2 point,float radius){

[0184] return length(point)-radius;

[0185] }

[0186] exist Figure 13 In the example, the process flow is executed by application 800 to provide what is discussed below.

[0187] At 1300, the RXGY graph is generated by application 800. At 1320, the offset to the center is determined by application 800, for example, by subtracting (0.5, 0.5) from each point. At 1340, the length of each vector represented by the graph is determined by application 800.

[0188] Figure 14 Other examples of plotting using the symbolic distance function (SDF) according to some implementations are shown. Figure 14 Examples of this can be performed by the application 800 discussed above. However, it should be understood that in some implementations, such examples can be performed by the message sending and receiving client application 104 or by components of the message sending and receiving system 100. Figure 14 Including those discussed above Figure 13 The example continues the discussion.

[0189] exist Figure 14 In the example, the process flow is executed by application 800 to provide what is discussed below.

[0190] At 1400, application 800 moves the edge of the circle by subtracting the zero value to the radius. At 1420, application 800 uses a smoothing step function to limit the mask to values ​​below zero. At 1440, application 800 uses the adjusted value as the alpha channel to mask the color onto the mask shape when the blending mode is set to normal (e.g., default or default setting).

[0191] Figure 15 An example of a mirrored RXGY mapping according to some implementations is shown. Figure 15 Examples of this can be performed by the application 800 discussed above. However, it should be understood that in some implementations, such examples can be performed by the message sending and receiving client application 104 or by components of the message sending and receiving system 100.

[0192] In the example, facial mesh symmetry can be used for augmented reality. If the face is UV-mapped to the center of the image that will be mapped along with the face, this method can be used to color it symmetrically. This means that the graph discussed above is also transformed to be symmetrical to take into account the symmetry of the facial mesh.

[0193] exist Figure 15In the example, the process flow is executed by application 800 to provide a mirrored RXGY map as described below. At 1500, application 800 performs an operation to mirror the RXGY graphic into a square based on mathematical operations within the shader, which in turn generates a symmetrical graphic. At 1520, application 800 generates a face mesh shaded using a physically based rendering (PBR) shader, where the underlying color mapping is unaffected.

[0194] Figure 15 The example in the text shows an image that approximates a symmetrical shading map. In the example, such a mapping can be made if the planar face mesh represented by the UV map is centered relative to the boundary of UV space. If this is done properly (e.g., the planar face mesh is centered relative to the boundary of UV space), then the inverted symmetrical mapping will thus be properly mapped, as... Figure 15 As shown.

[0195] Figure 16 and Figure 17 Examples of mapping types according to some implementation methods are shown. Figure 16 and Figure 17 Examples of this can be performed by the application 800 discussed above. However, it should be understood that in some implementations, such examples can be performed by the message sending and receiving client application 104 or by components of the message sending and receiving system 100.

[0196] In the example, how the graph is used with the facial mesh changes how drawing occurs on the face. Figure 16 and Figure 17 The example shown is a graph format available in IFM format. When the graph covers the vertical length of the face, the range of mapped values ​​can be from (0,0) to (1,2). When the graph represents half of the vertical face, the range of values ​​can be from (0,0) to (1,1). Figure 16 The left and right referenced in the image name refer to the left and right sides of the screen.

[0197] In implementation, the formatted JSON used in IFM can determine the location of color mapping to the face using mapping type and other parameters. The following is an example... Figure 16 and Figure 17 The following is a list of all available parameters in IFM: Fully symmetrical 1600, Upper half symmetrical 1620, Lower half symmetrical 1640, Upper left symmetrical 1700, Upper right symmetrical 1720, Left asymmetrical 1740, and Right asymmetrical 1760.

[0198] Examples of top-level element parameters include the following:

[0199]

[0200]

[0201] Examples of primitive formatting parameters include the following:

[0202]

[0203]

[0204] Examples of primitive fill parameters include the following:

[0205]

[0206] Examples of filling formatting parameters include the following:

[0207]

[0208]

[0209] Examples of gradient scale parameters include the following:

[0210]

[0211] Figure 18 This is a flowchart illustrating method 1800 according to some example implementations. Method 1800 may be contained in computer-readable instructions for execution by one or more computer processors, such that operations of method 1800 may be performed partially or entirely by application 800; therefore, method 1800 is described below by way of example. However, it should be understood that at least some operations of method 1800 may be deployed on various other hardware configurations, and method 1800 is not intended to be limited to application 800, but may be performed by message transceiver client application 104 or components of message transceiver system 100.

[0212] In implementation, creating an AR content generator for generating facial makeup effects and AR content items involves describing the appearance as a combination of shape, color, and smoothness to be applied to a mask area. As previously mentioned, such a combination is described as JSON—which can be referenced by a product catalog service (PCS)—and ingested using an 800 application to create a given AR content generator. In some examples, the PCS is a standalone server providing a database with product information (e.g., product descriptions, product information, various assets, metadata, etc.). In some cases, such a PCS may be provided by a third party or by a theme system (e.g., as a service or component of the system). The following discussion describes an example process for ingesting such an appearance based on the IFM format.

[0213] At operation 1802, application 800 identifies a set of graphic elements in an augmented reality (AR) facial pattern.

[0214] The first step in creating an appearance is to identify the graphic elements within it and break them down into individual parts. These individual parts should be broken down into basic fragments (e.g., primitives) whenever possible, because the IFM format relies on the fidelity of these approximations to properly convey a given appearance (e.g., an AR face pattern). If a pattern for a given appearance is already available in an appearance library and has already been created, it is beneficial to start with an appearance that approximates the desired appearance and update it appropriately.

[0215] At operation 1804, application 800 determines at least one primitive shape based on the set of elements.

[0216] At operation 1806, application 800 uses at least one primitive shape to generate a JavaScript object simplification (JSON) file.

[0217] After breaking down the AR facial pattern into its components, the appearance can be easily reconstructed using IFM primitives. A JSON format describing each element that makes up the appearance will then be able to convey the appearance through these fragment elements via an IFM-based template.

[0218] At operation 1808, application 800 uses a JSON file to generate internal facial makeup format (IFM) data.

[0219] After breaking down the appearance into its components, it can be easily reconstructed using IFM primitives. A JSON format describing each element that makes up the appearance will be able to convey the appearance through these fragment elements via an IFM-based template. Below is an example of a suggested JSON format. In this example, only two primitives are used. These two example elements use different lighting models and different SDF primitives. This format of JSON will be considered an IFM format configuration. The deeper technical specifications of the IFM format configuration specification will be documented separately.

[0220] After converting the appearance to IFM format using JSON, the JSON can be viewed as an appearance configuration. This appearance configuration can be saved as a JSON file and hosted in a basic bucket that contains other such appearance configurations. In the example, the Product Catalog Service (PCS) can add specific fields to products that support appearance representation. This field can represent a URL pointing to the aforementioned IFM format configuration file.

[0221] At operation 1810, application 800 publishes IFM data to the product catalog service.

[0222] Advanced mapping techniques may require less internal knowledge about the IFM format and how the mapping technique procedurally generates the diagrams. This technique enables the subject system to map the appearance of profiles that third-party vendors may have pre-defined. In this example, the necessary step for the third-party vendor is to appropriately map its appearance to an approximate pre-defined one in the subject system, so that the diagrams provided by the vendor are automatically converted into IFM format profiles that will be associated with the products in the PCS.

[0223] A more robust, high-fidelity option is for a third-party vendor to directly output IFM format configurations. In the example, the third-party vendor has the resources to learn how to use IFM formats and templates internally on its end. As the library of such third-party vendors grows, they will be able to continuously output the necessary JSON to be interpreted by IFM-based templates.

[0224] Below is an example of a suggested JSON format. This example uses only two primitives.

[0225]

[0226]

[0227]

[0228] AR shopping brings the benefits of augmented reality to the shopping experience. A key component of the AR shopping experience is the ability to overlay graphic elements onto real-world space to enhance the perceived reality. Graphic elements come in various formats, including images, 3D models, and more. The techniques used to ingest 3D model assets are discussed below.

[0229] In the example, the 3D AR experience is packaged into relatively small, standalone files called AR content generators (e.g., Lenses), which are delivered to a given client device. The process of building the AR content generator has been formalized through Application 800, as previously described. This particular process of creating the AR content generator sets some implicit boundaries. Because the AR content generator is created using Application 800, there is no need to dynamically load the model into Application 800.

[0230] In the use case of AR shopping, comparing the AR content generator use case here to a typical e-commerce website yields a very similar experience in terms of the end result; users are able to make choices, learn about products, and then make a purchase decision. In e-commerce websites, the most common form of loading products onto a given page is by dynamically loading that data when the user requests it. This means that any webpage itself is essentially an empty shell that requests data when the page is ready to display an item.

[0231] Figure 19 This is a flowchart illustrating method 1900 according to some example implementations. Method 1900 may be contained in computer-readable instructions for execution by one or more computer processors, such that operations of method 1900 may be performed partially or entirely by application 800; therefore, method 1900 is described below by way of example. However, it should be understood that at least some operations of method 1900 may be deployed on various other hardware configurations, and method 1900 is not intended to be limited to application 800, but may be performed by message transceiver client application 104 or components of message transceiver system 100.

[0232] The following process for acquiring 3D assets involves several operations. The product is converted into a 3D model file, which can be in a binary format, such as a binary file representation of a 3D model saved in GL (Graphics Language) Transfer Format (glTF). The next step is to set up this model file in Application 800 so that it can be exported as a second model file, i.e., a 3D object, in a second format (e.g., FBX (Filmbox) export format or OBJ file format (e.g., representing 3D geometry)). Once the model file is created, it must be hosted in a specific model “bucket” storage device and tagged with the finest-grained attributes of the product, which in most cases is the stock unit (SKU) associated with the product.

[0233] This is then added to the product data associated with the specific product in the Product Catalog Service (PCS). In the PCS, 3D objects are represented by URLs that link to the storage location where the 3D object files are currently located. In the example, this is then used by application 800 to associate that specific product with a Product Type Template (PTT) that has been customized by the brand into a Storefront Lens template (SLT). Once the publisher or publishing service references the SLT and 3D object files, along with any other configuration files, an AR content generator is created and then published.

[0234] Once an AR content generator is published, a reporting and analysis mechanism for that specific AR content generator is available by creating a template for that AR content generator. The model file is displayed (e.g., rendered for display) in the appropriate location within the AR content generator, given appropriate selections, and all placement of the model is handled by the AR content generator. This means that any animations or display configurations requiring positional adjustments are handled internally within the Lens template and do not depend on other configurations.

[0235] At operation 1902, application 800 receives information about the product.

[0236] At operation 1904, application 800 generates a 3D model file of the product in the first format.

[0237] At operation 1906, application 800 converts the 3D model file into a 3D object file in the second format.

[0238] At operation 1908, application 800 associates the 3D object file with a product in the product catalog service.

[0239] At operation 1910, application 800 publishes an augmented reality (AR) content generator corresponding to the product.

[0240] Figure 20 This is a block diagram illustrating an example software architecture 2006, which is designed for use in conjunction with various hardware architectures described herein. Figure 20 This is a non-limiting example of software architecture, and it should be understood that many other architectures can be implemented to facilitate the functionality described herein. Software Architecture 2006 can be applied to, for example... Figure 21 The execution occurs on the hardware of machine 2100, which includes processor 2104, memory 2114, and (input / output) (I / O) components 2118, etc. A representative hardware layer 2052 is shown, and this representative hardware layer can represent, for example... Figure 21 The machine 2100. A representative hardware layer 2052 includes a processing unit 2054 having associated executable instructions 2004. The executable instructions 2004 represent executable instructions of the software architecture 2006, including implementations of the methods, components, etc., described herein. Hardware layer 2052 also includes a memory and / or storage module memory / storage device 2056 that also has executable instructions 2004. Hardware layer 2052 may also include other hardware 2058.

[0241] exist Figure 20 In the example architecture, software architecture 2006 can be conceptualized as a stack of layers, where each layer provides specific functionality. For example, software architecture 2016 may include layers such as operating system 2002, library 2020, framework / middleware 2018, application 2016, and presentation layer 2014. Operationally, application 2016 and / or other components within a layer can call API call 2008 via the software stack and receive responses to API call 2008, such as message 2012. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile operating systems or dedicated operating systems may not provide framework / middleware 2018, while other operating systems may provide such a layer. Other software architectures may include additional or different layers.

[0242] Operating System 2002 can manage hardware resources and provide public services. Operating System 2002 may include, for example, a kernel 2022, services 2024, and drivers 2026. Kernel 2022 can serve as an abstraction layer between hardware and other software layers. For example, kernel 2022 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. Services 2024 can provide other public services to other software layers. Drivers 2026 are responsible for controlling or interfacing with the underlying hardware. For example, depending on the hardware configuration, drivers 2026 may include display drivers, camera drivers, etc. Drivers, flash memory drivers, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0243] Library 2020 provides common infrastructure used by Application 2016 and / or other components and / or layers. Library 2020 provides functionality that allows other software components to perform tasks more easily than by directly interfaceing with the functions of the underlying operating system 2002 (e.g., kernel 2022, service 2024, and / or driver 2026). Library 2020 may include system libraries 2044 (e.g., the C standard library), which provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. Additionally, Library 2020 may include API libraries 2046, such as media libraries (e.g., libraries supporting the rendering and manipulation of various media formats such as MPREG4, H.264, MP3, AAC, AMR, JPG, and PNG), graphics libraries (e.g., OpenGL frameworks that can be used to render 2D and 3D graphical content on a display), database libraries (e.g., SQLite that provides various relational database functionalities), web libraries (e.g., WebKit that provides web browsing functionality), etc. Library 2020 may also include various other libraries 2048 to provide many other APIs to Application 2016 and other software parts / modules.

[0244] Framework / Middleware 2018 (sometimes also called middleware) provides a higher level of common infrastructure that can be used by Application 2016 and / or other software components / modules. For example, Framework / Middleware 2018 can provide various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. Framework / Middleware 2018 can provide a wide range of other APIs that can be used by Application 2016 and / or other software components / modules, some of which may be specific to a particular operating system 2002 or platform.

[0245] Application 2016 includes built-in applications 2038 and / or third-party applications 2040. Examples of representative built-in applications 2038 may include, but are not limited to, contact applications, browser applications, book reader applications, location applications, media applications, messaging applications, and / or game applications. Third-party applications 2040 may include those used by entities other than platform-specific vendors using Android. TM or iOS TM Applications developed using a Software Development Kit (SDK) can be used on platforms such as iOS. TM ANDROID TM , Mobile software running on the phone's mobile operating system or other mobile operating systems. Third-party applications 2040 can call API calls 2008 provided by the mobile operating system (e.g., OS 2002) to facilitate the functions described herein.

[0246] Application 2016 can use built-in operating system features (e.g., kernel 2022, service 2024, and / or driver 2026), libraries 2020, and frameworks / middleware 2018 to create user interfaces for interaction with the system's users. Alternatively or additionally, in some systems, interaction with the user can occur through a presentation layer, such as presentation layer 2014. In these systems, the application / component "logic" can be separated from the application / component's user-interacting aspects.

[0247] Figure 21 This is a block diagram illustrating components of a machine 2100 according to some example embodiments, which is capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and executing any or more of the methods discussed herein. Specifically, Figure 21A schematic representation of a machine 2100 in the form of an example computer system is shown, in which instructions 2110 (e.g., software, programs, applications, applets, or other executable code) can be executed to cause the machine 2100 to perform any or more of the methods discussed herein. Similarly, instructions 2110 can be used to implement the modules or components described herein. Instructions 2110 transform a general, unprogrammed machine 2100 into a specific machine 2100 programmed to perform the described and illustrated functions in the described manner. In alternative embodiments, machine 2100 operates as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, machine 2100 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 2100 may include, but is not limited to: server computers, client computers, personal computers (PCs), tablet computers, laptop computers, netbooks, set-top boxes (STBs), personal digital assistants (PDAs), entertainment media systems, cellular phones, smartphones, mobile devices, wearable devices (e.g., smartwatches), smart home devices (e.g., smart appliances), other smart devices, web home appliances, network routers, network switches, network bridges, or any machine capable of sequentially or otherwise executing instructions 2110 specifying actions to be taken by machine 2100. Furthermore, although only a single machine 2100 is shown, the term "machine" should also be considered as a collection of machines that individually or jointly execute instructions 2110 to perform any one or more of the methods discussed herein.

[0248] Machine 2100 may include processor 2104—which includes processors 2108 to 2112, memory / storage device 2106, and I / O components 2118, which may be configured to communicate with each other, for example, via bus 2102. Memory / storage device 2106 may include memory 2114, such as main memory or other storage device, and storage cells 2116, which processor 2104 can access, for example, via bus 2102. Storage cells 2116 and memory 2114 store instructions 2110 embodying any one or more of the methods or functions described herein. Instructions 2110 may also reside wholly or partially within memory 2114, storage cells 2116, at least one of processors 2104 (e.g., the processor's cache memory), or any suitable combination thereof during execution by machine 2100. Thus, memory 2114, storage cells 2116, and the memory of processor 2104 are examples of machine-readable media.

[0249] I / O component 2118 may include a wide variety of components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, etc. The specific I / O component 2118 included in a particular machine 2100 will depend on the type of machine. For example, a portable machine such as a mobile phone will likely include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It will be understood that I / O component 2118 may include... Figure 21 Many other components are not shown. The I / O components 2118 are grouped according to function only for the sake of simplifying the discussion below, and the grouping is by no means limiting. In various example embodiments, the I / O components 2118 may include output components 2126 and input components 2128. Output components 2126 may include visual components (e.g., displays such as plasma display panels (PDPs), light-emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tube (CRT) displays), auditory components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. Input components 2128 may include alphanumeric input components (e.g., keyboards, touchscreens configured to receive alphanumeric input; photo-optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touchpads, trackballs, joysticks, motion sensors, or other pointing instruments), haptic input components (e.g., physical buttons, touchscreens that provide position and / or force for touch or touch gestures, or other haptic input components), audio input components (e.g., microphones), etc.

[0250] In another example implementation, I / O component 2118 may include various other components such as biometric component 2130, motion component 2134, environmental component 2136, or positioning component 2138. For example, biometric component 2130 may include components for detecting expressions (e.g., hand gestures, facial expressions, vocal expressions, body posture, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), and identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition). Motion component 2134 may include accelerometer components (e.g., accelerometers), gravity sensor components, rotation sensor components (e.g., gyroscopes), etc. Environmental component 2136 may include, for example, a lighting sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an hearing sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects the concentration of hazardous gases for safety purposes or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. Positioning component 2138 may include a position sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer from which altitude can be obtained), an orientation sensor component (e.g., a magnetometer), etc.

[0251] A wide variety of technologies can be used to implement communication. I / O component 2118 may include communication component 2140, which is operable to couple machine 2100 to network 2132 or device 2120 via coupling 2124 and coupling 2122, respectively. For example, communication component 2140 may include a network interface component or other suitable device to interface with network 2132. In other examples, communication component 2140 may include wired communication component, wireless communication component, cellular communication component, near field communication (NFC) component, etc. Components (e.g.) ), Components and other communication components that provide communication via other modes. Device 2120 can be another machine or any of various peripheral devices (e.g., a peripheral device coupled via USB).

[0252] Furthermore, the communication component 2140 may detect identifiers or include components operable to detect identifiers. For example, the communication component 2140 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes; multi-dimensional barcodes such as Quick Response (QR) codes, Aztec codes, data matrices, dataglyphs, MaxiCodes, PDF417, hypercodes, UCC RSS-2D barcodes, and other optical codes), or an auditory detection component (e.g., a microphone for identifying audio signals from the tag). Additionally, various information can be obtained via the communication component 2140, such as location via Internet Protocol (IP) geolocation, etc. Location can be obtained through signal triangulation or by detecting NFC beacon signals that indicate a specific location.

[0253] The following discussion involves various terms or phrases mentioned throughout the publicly available content of this topic.

[0254] "Signal medium" means any intangible medium capable of storing, encoding, or carrying instructions executable by a machine, and includes digital or analog communication signals or other intangible media that facilitate the communication of software or data. The term "signal medium" should be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal whose characteristics are set or altered in a manner that encodes information in the signal. The terms "transmission medium" and "signal medium" refer to the same thing and may be used interchangeably in this disclosure.

[0255] "Communication network" refers to one or more parts of a network, which can be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a POTS (Plain Old-Style Telephone Service) network, a cellular telephone network, a wireless network, etc. A network, other types of networks, or a combination of two or more such networks. For example, a network or part of a network may include a wireless network or a cellular network, and the coupling may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other types of cellular or wireless coupling. In this example, the coupling can implement any data transmission technology of various types, such as Single Carrier Radio Transmission (1xRTT), Evolved Data Optimization (EVDO), General Packet Radio Service (GPRS), Enhanced Data Rate Evolution (EDGE) technology of GSM, the 3rd Generation Partnership Project (3GPP) including 3G, fourth-generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Global Microwave Access Interoperability (WiMAX), Long Term Evolution (LTE) standards, other data transmission technologies defined by various standards setting organizations, other long-distance protocols, or other data transmission technologies.

[0256] A "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and generates corresponding output signals applied to operate a machine. For example, a processor can be a Central Processing Unit (CPU), a Reduced Instruction Set Computing (RISC) processor, a Complex Instruction Set Computing (CISC) processor, a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Radio Frequency Integrated Circuit (RFIC), or any combination thereof. A processor can also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously.

[0257] "Machine storage medium" refers to one or more storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. Therefore, this term should be considered to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium," "device storage medium," and "computer storage medium" refer to the same thing and are used interchangeably in this disclosure. The terms "machine storage medium," "computer storage medium," and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are covered by the term "signal medium."

[0258] A “component” refers to a device, physical entity, or logic having boundaries defined by functional or subroutine calls, branch points, APIs, or other technologies that provide partitioning or modularity for specific processing or control functions. Components can interface with other components via their interfaces to perform machine processing. A component can be part of a program that is an encapsulated functional hardware unit designed for use with other components and typically performs a specific function. Components can constitute software components (e.g., code implemented on a machine-readable medium) or hardware components. A “hardware component” is a tangible unit capable of performing certain operations and can be configured or arranged in some physical manner. In various example implementations, one or more computer systems (e.g., standalone computer systems, client computer systems, or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., an application or application portion) to perform certain operations as described herein. Hardware components can also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component can include a dedicated circuit system or logic permanently configured to perform certain operations. Hardware components can be dedicated processors, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). Hardware components can also include programmable logic or circuitry temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific part of a machine) uniquely tailored to perform the configured function and is no longer a general-purpose processor. It will be understood that decisions to implement hardware components mechanically in dedicated and permanently configured circuitry or in temporarily configured (e.g., software-configured) circuitry may be driven by cost and time considerations. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to encompass tangible entities, i.e., entities physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain way or perform certain operations described herein. Consider implementations where hardware components are temporarily configured (e.g., programmed), eliminating the need to configure or instantiate each of the hardware components at any given time. For example, in cases where the hardware components include a general-purpose processor configured as a dedicated processor via software, this general-purpose processor can be configured as different dedicated processors (e.g., including different hardware components) at different times. The software accordingly configures one or more specific processors to constitute a specific hardware component, for example, at one moment and as different hardware components at different times. Hardware components can provide information to and receive information from other hardware components. Therefore, the described hardware components can be considered communicatively coupled.In the presence of multiple hardware components, communication can be achieved through signal transmission between or among two or more hardware components (e.g., via appropriate circuitry and buses). In embodiments where multiple hardware components are configured or instantiated at different times, such communication between hardware components can be achieved, for example, by storing information in a memory structure accessed by the multiple hardware components and retrieving information from that memory structure. For example, a hardware component can perform an operation and store the output of that operation in a memory device communicatively coupled to it. Other hardware components can then access the memory device at a subsequent time to retrieve and process the stored output. Hardware components can also initiate communication with input or output devices and can operate on resources (e.g., information collection). The various operations of the example methods described herein can be performed at least in part by one or more processors that are temporarily or permanently configured (e.g., via software) to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be implemented at least in part by processors, with one or more specific processors being examples of hardware. For example, at least some operations of the methods can be performed by one or more processors or processor-implemented components. Furthermore, the one or more processors can also be configured to support the execution of related operations in a “cloud computing” environment or to operate as “Software as a Service” (SaaS). For example, at least some operations can be performed by a group of computers (as an example of machines including processors), wherein these operations are accessible via a network (e.g., the Internet) and via one or more suitable interfaces (e.g., APIs). The execution of some operations can be distributed among processors, not residing within a single machine, but deployed across multiple machines. In some example implementations, the processor or processor-implemented component may reside in a single geographic location (e.g., in a home environment, office environment, or server cluster). In other example implementations, the processor or processor-implemented component may be distributed across several geographic locations.

[0259] "Carrier signal" refers to any intangible medium capable of storing, encoding, or carrying instructions to be executed by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of such instructions. Instructions can be sent or received over a network using a transmission medium via a network interface device.

[0260] "Computer-readable medium" refers to both machine storage media and transmission media. Therefore, these terms encompass both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" refer to the same thing and may be used interchangeably in this disclosure.

[0261] "Client device" means any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. Client devices can be, but are not limited to, mobile phones, desktop computers, laptop computers, portable digital assistants (PDAs), smartphones, tablet computers, ultrabooks, netbooks, laptop computers, multiprocessor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access the network. In this disclosure, client devices are also referred to as "electronic devices."

[0262] A "brief message" is a message that is accessible for a limited period of time. Brief messages can be text, images, videos, etc. The access time for a brief message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting method, the message is temporary.

[0263] "Signal medium" means any intangible medium capable of storing, encoding, or carrying instructions executable by a machine, and includes digital or analog communication signals or other intangible media that facilitate the communication of software or data. The term "signal medium" should be considered to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal whose characteristics are set or altered in a manner that encodes information in the signal. The terms "transmission medium" and "signal medium" refer to the same thing and may be used interchangeably in this disclosure.

[0264] "Communication network" refers to one or more parts of a network, which can be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a part of the Internet, a part of the Public Switched Telephone Network (PSTN), a POTS (Plain Old-Style Telephone Service) network, a cellular telephone network, a wireless network, etc. A network, other types of networks, or a combination of two or more such networks. For example, a network or part of a network may include a wireless network or a cellular network, and the coupling may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other types of cellular or wireless coupling. In this example, the coupling can implement any data transmission technology of various types, such as Single Carrier Radio Transmission (1xRTT), Evolved Data Optimization (EVDO), General Packet Radio Service (GPRS), Enhanced Data Rate Evolution (EDGE) technology of GSM, the 3rd Generation Partnership Project (3GPP) including 3G, fourth-generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Global Microwave Access Interoperability (WiMAX), Long Term Evolution (LTE) standards, other data transmission technologies defined by various standards setting organizations, other long-distance protocols, or other data transmission technologies.

[0265] A "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and generates corresponding output signals applied to operate a machine. For example, a processor can be a Central Processing Unit (CPU), a Reduced Instruction Set Computing (RISC) processor, a Complex Instruction Set Computing (CISC) processor, a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Radio Frequency Integrated Circuit (RFIC), or any combination thereof. A processor can also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") capable of executing instructions simultaneously.

[0266] "Machine storage medium" refers to one or more storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. Therefore, this term should be considered to include, but is not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium," "device storage medium," and "computer storage medium" refer to the same thing and are used interchangeably in this disclosure. The terms "machine storage medium," "computer storage medium," and "device storage medium" expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are covered by the term "signal medium."

[0267] A “component” refers to a device, physical entity, or logic having boundaries defined by functional or subroutine calls, branch points, APIs, or other technologies that provide partitioning or modularity for specific processing or control functions. Components can be combined with other components via their interfaces to perform machine processing. A component can be part of a program that is an encapsulated functional hardware unit designed for use with other components and typically performs a specific function. Components can constitute software components (e.g., code implemented on a machine-readable medium) or hardware components. A “hardware component” is a tangible unit capable of performing certain operations and can be configured or arranged in some physical manner. In various example implementations, one or more computer systems (e.g., standalone computer systems, client computer systems, or server computer systems) or one or more hardware components (e.g., processors or processor groups) of a computer system can be configured by software (e.g., an application or application portion) to perform certain operations as described herein. Hardware components can also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component can include a dedicated circuit system or logic permanently configured to perform certain operations. Hardware components can be dedicated processors, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). Hardware components can also include programmable logic or circuitry temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes uniquely tailored to perform the configured function and is no longer a machine-specific component (or a machine-specific part) of a general-purpose processor. It will be understood that decisions to mechanically implement hardware components in dedicated and permanently configured circuitry or in temporarily configured (e.g., software-configured) circuitry may be driven by cost and time considerations. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to encompass tangible entities, i.e., entities physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain way or perform certain operations described herein. Consider implementations where hardware components are temporarily configured (e.g., programmed), eliminating the need to configure or instantiate each of the hardware components at any given time. For example, in cases where the hardware components include a general-purpose processor configured as a dedicated processor via software, this general-purpose processor can be configured as different dedicated processors (e.g., including different hardware components) at different times. The software accordingly configures one or more specific processors to constitute a specific hardware component, for example, at one moment and as different hardware components at different times. Hardware components can provide information to and receive information from other hardware components. Therefore, the described hardware components can be considered communicatively coupled.In the presence of multiple hardware components, communication can be achieved through signal transmission between or among two or more hardware components (e.g., via appropriate circuitry and buses). In embodiments where multiple hardware components are configured or instantiated at different times, such communication between hardware components can be achieved, for example, by storing information in a memory structure accessed by the multiple hardware components and retrieving information from that memory structure. For example, a hardware component can perform an operation and store the output of that operation in a memory device communicatively coupled to it. Other hardware components can then access the memory device at a subsequent time to retrieve and process the stored output. Hardware components can also initiate communication with input or output devices and can operate on resources (e.g., information collection). The various operations of the example methods described herein can be performed at least in part by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, "processor-implemented component" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be implemented at least in part by processors, with one or more specific processors being examples of hardware. For example, at least some operations of the methods can be performed by one or more processors or processor-implemented components. Furthermore, the one or more processors can also be configured to support the execution of related operations in a “cloud computing” environment or to operate as “Software as a Service” (SaaS). For example, at least some operations can be performed by a group of computers (as an example of machines including processors), wherein these operations are accessible via a network (e.g., the Internet) and via one or more suitable interfaces (e.g., APIs). The execution of some operations can be distributed among processors, not residing within a single machine, but deployed across multiple machines. In some example implementations, the processor or processor-implemented component may reside in a single geographic location (e.g., in a home environment, office environment, or server cluster). In other example implementations, the processor or processor-implemented component may be distributed across several geographic locations.

[0268] "Carrier signal" refers to any intangible medium capable of storing, encoding, or carrying instructions to be executed by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of such instructions. Instructions can be sent or received over a network using a transmission medium via a network interface device.

[0269] "Computer-readable medium" refers to both machine storage media and transmission media. Therefore, these terms encompass both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" refer to the same thing and may be used interchangeably in this disclosure.

[0270] "Client device" refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. Client devices can be, but are not limited to, mobile phones, desktop computers, laptop computers, portable digital assistants (PDAs), smartphones, tablet computers, ultrabooks, netbooks, laptop computers, multiprocessor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access the network.

[0271] A "brief message" is a message that is accessible for a limited period of time. Brief messages can be text, images, videos, etc. The access time for a brief message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting method, the message is temporary.

Claims

1. A method for an augmented reality (AR) content generator, comprising: Receive information about the product; Generate a 3D model file of the product in the first format; The 3D model file is converted into a 3D object file in a second format; The 3D object file is associated with the product in the product catalog service, wherein the product catalog service is a standalone server that provides a database of product information, and a specific field on the product represents a Uniform Resource Locator URL pointing to an internal facial makeup format IFM configuration file; as well as Release the augmented reality (AR) content generator corresponding to the product.

2. The method according to claim 1, wherein, The product catalog service associates the IFM profile with the product, wherein the IFM profile uses at least one primitive shape to store the appearance data of the product.

3. The method according to claim 1, wherein, In the product catalog service, the product is represented by the URL, and the Uniform Resource Locator links to the storage location where the 3D object file is stored.

4. The method according to claim 1, wherein, Associating the 3D object file with the product in the product catalog service includes: Associate the 3D object file with a product type template modified to the Storefront Lens template file format.

5. The method according to claim 1, wherein, Generating the 3D model file of the product in the first format includes: The 3D model file is converted into a binary format, which includes a binary file format representation of the 3D model in GL transfer format.

6. The method according to claim 1, wherein converting the 3D model file into the 3D object file in the second format comprises: Convert the 3D model file into Filmbox file format.

7. The method according to claim 1, wherein converting the 3D model file into the 3D object file in the second format comprises: Convert the 3D model file into OBJ file format.

8. The method according to claim 1, wherein, Associating the 3D object file with the product includes: tagging the 3D object file to an inventory unit (SKU) associated with the product.

9. The method according to claim 1, further comprising: Receive a selection of the AR content generator corresponding to the product.

10. The method of claim 9, further comprising: Provide a 3D object file corresponding to the 3D model file of the product.

11. The method according to claim 1, wherein, Information about the product is received from a product catalog service, which includes a server provided by a third party.

12. A system for an augmented reality (AR) content generator, comprising: processor; as well as The memory includes instructions that, when executed by the processor, cause the processor to perform operations, the operations including: Receive information about the product; Generate a 3D model file of the product in the first format; The 3D model file is converted into a 3D object file in a second format; The 3D object file is associated with the product in a product catalog service, wherein the product catalog service is a standalone server providing a database of product information, and a specific field on the product represents a Uniform Resource Locator (URL) pointing to an internal facial makeup format (IFM) configuration file; and Release the augmented reality (AR) content generator corresponding to the product.

13. The system according to claim 12, wherein, In the product catalog service, the product is represented by the URL, and the Uniform Resource Locator links to the storage location where the 3D object file is stored.

14. The system according to claim 12, wherein, Associating the 3D object file with the product in the product catalog service includes: Associate the 3D object file with a product type template modified to the Storefront Lens template file format.

15. The system according to claim 12, wherein converting the 3D model file into the 3D object file in the second format comprises: Convert the 3D model file into Filmbox file format.

16. The system according to claim 12, wherein converting the 3D model file into the 3D object file in the second format comprises: Convert the 3D model file into OBJ file format.

17. The system according to claim 12, wherein, Associating the 3D object file with the product includes: tagging the 3D object file to an inventory unit (SKU) associated with the product.

18. The system of claim 12, further comprising: Receive a selection of the AR content generator corresponding to the product.

19. The system of claim 18, further comprising: Provide a 3D object file corresponding to the 3D model file of the product.

20. A non-transitory computer-readable medium comprising instructions that, when executed by a computing device, cause the computing device to perform an operation, the operation comprising: Receive information about the product; Generate a 3D model file of the product in the first format; The 3D model file is converted into a 3D object file in a second format; The 3D object file is associated with the product in the product catalog service, wherein the product catalog service is a standalone server that provides a database of product information, and a specific field on the product represents a Uniform Resource Locator URL pointing to an internal facial makeup format IFM configuration file; as well as Release an augmented reality (AR) content generator corresponding to the product.

Images