Post-capture processing in the messaging system

Abstract

The subject technology retrieves, by a client device, first image data captured by a client device at a previous time from a storage device. The subject technology receives first metadata corresponding to a selected image processing operation. The subject technology generates second image data based on the first metadata and the image processing operation performed on the first image data. The subject technology generates second metadata including information related to the image processing operation, the second metadata including a first identifier associated with the first image data and a second identifier associated with the second image data. The subject technology generates a message including the second metadata, the second image data, and the first image data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application serial number 16 / 732,025 filed on December 31, 2019, U.S. patent application serial number 16 / 732,040 filed on December 31, 2019, U.S. patent application serial number 16 / 732,051 filed on December 31, 2019, and U.S. patent application serial number 16 / 732,001 filed on December 31, 2019, the contents of each of the above U.S. patent applications are incorporated herein by reference in their entirety for all purposes. Background Art

[0003] With the increase in the use of digital images, the affordability of portable computing devices, the availability of increased capacity of digital storage media, and the increased bandwidth and accessibility of network connections, digital images have become part of the daily lives of more and more people. Interested users from a variety of locations can capture digital images of various objects and make the captured images available to others via a network (e.g., the Internet). To enhance the user's experience of digital images and provide various features, it can be challenging and computationally intensive to enable computing devices to perform image processing operations on various objects and / or features captured under various varying conditions (e.g., variations in image scale, noise, lighting, motion, or geometric distortion). BRIEF DESCRIPTION OF THE DRAWINGS

[0004] To easily identify the discussion of any particular element or act, the highest digit or digits in a reference number refer to the figure number in which the element is first introduced.

[0005] Figure 1 is a pictorial representation of a networked environment in which the present disclosure may be deployed, according to some example implementations.

[0006] Figure 2 is a graphical representation of a messaging client application according to some example embodiments.

[0007] Figure 3 is a graphical representation of a data structure as maintained in a database according to some example implementations.

[0008] Figure 4 is a graphical representation of a message according to some example embodiments.

[0009] Figure 5 is a flow diagram of an access restriction process according to some example implementations.

[0010] Figure 6is a block diagram illustrating various modules of an annotation system implementing a post-processing graphics pipeline, according to certain example implementations.

[0011] Figure 7 is a diagram showing how, according to some embodiments, Figure 4 A schematic diagram of the structure of a message annotation described that includes additional information corresponding to metadata for post-processing an image (eg, an input frame).

[0012] Fig. 8A and Figure 8B An example of a user capturing an image for post-processing in a messaging client application (or messaging system) is shown in accordance with some embodiments.

[0013] Fig.9A and Fig. 9B An example of post-processing in a messaging client application (or messaging system) according to some embodiments is shown.

[0014] Fig. 10A and Fig. 10B Other examples of post-processing in a messaging client application (or messaging system) according to some embodiments are shown.

[0015] Fig.11 An example of a user interface for selecting a media overlay to apply post-processing to content (eg, an image or video) in a messaging client application (or messaging system) is shown in accordance with some implementations.

[0016] Fig.12 An example of a user interface for selecting a media overlay for applying post-processing to content (e.g., an image or video) in a messaging client application (or messaging system) according to some embodiments is shown, following Fig.11 The examples described in .

[0017] Fig.13 is a schematic diagram of an example of a graphics processing pipeline, ie, a post-processing pipeline implemented for a component (eg, a GPU) of a client device, according to some example embodiments.

[0018] Fig.14 is a schematic diagram of an example of a graphics processing pipeline, ie, a post-processing pipeline implemented for a component (eg, a GPU) of a client device, according to some example embodiments.

[0019] Fig.15 According to some example embodiments, the graphics processing pipeline (i.e., Fig.13 and / or Fig.14 Schematic diagram of an example of a stack of media overlays implemented using the post-processing pipeline described in .

[0020] Fig.16 An example of a user interface (eg, a carousel) for selecting and stacking multiple media overlays to apply post-processing to media content (eg, images or videos) in a messaging client application (or messaging system) is shown in accordance with some embodiments.

[0021] Fig.17 Additional examples of user interfaces (eg, carousels) for selecting and stacking multiple media overlays to apply post-processing to media content (eg, images or videos) in a messaging client application (or messaging system) are shown in accordance with some embodiments.

[0022] Fig.18 Examples of media overlays that may be selected to be applied to media content during post-processing are shown.

[0023] Fig.19 Examples of media overlays that may be selected to be applied to media content during post-processing are shown.

[0024] Fig. 20 Examples of media overlays that may be selected to be applied to media content during post-processing are shown.

[0025] Fig.21 Examples of media overlays that may be selected to be applied to media content during post-processing are shown.

[0026] Fig. 22 Examples of media overlays that may be selected to be applied to media content during post-processing are shown.

[0027] Fig.23 Examples of media overlays that may be selected to be applied to media content during post-processing are shown.

[0028] Fig.24 Examples of media overlays that may be selected to be applied to media content during post-processing are shown.

[0029] Fig.25 Examples of media overlays that may be selected to be applied to media content during post-processing are shown.

[0030] Fig.26 is a flow chart illustrating a method of generating a message based on at least one post-processing operation on image data, according to certain example embodiments.

[0031] Fig. 27 is a flow chart illustrating a method of generating messages based on a layering of post-processing operations on image data, according to certain example embodiments.

[0032] Fig.28is a flow chart illustrating a method of providing a carousel interface for post-processing operations according to certain example embodiments.

[0033] Fig.29 is a flow chart illustrating a method for determining a set of media overlays to populate a set of media overlays that may be used by a client electronic device during a post-processing stage to modify media content, according to certain example embodiments.

[0034] Fig.30 is a block diagram illustrating a software architecture in which the present disclosure may be implemented according to some example embodiments.

[0035] Fig.31 is a diagrammatic representation of a machine in the form of a computer system within which a set of instructions may be executed, for causing the machine to perform any one or more of the discussed methodologies, according to some example embodiments. DETAILED DESCRIPTION

[0036] As mentioned above, with the increase in the use of digital images, the affordability of portable computing devices, the availability of the increased capacity of digital storage media, and the increased bandwidth and accessibility of network connections, digital images have become part of the daily lives of more and more people. Users with a wide range of interests from various locations can capture digital images of various objects, and make the captured images available to others via a network (e.g., the Internet). In order to enhance the user's experience with digital images and provide various features, it may be challenging and computationally intensive to enable computing devices to perform image processing operations on various objects and / or features captured under various changing conditions (e.g., changes in image scale, noise, lighting, motion, or geometric distortion). The embodiments described herein provide an improved system for image processing in the post-capture stage of image data or media content.

[0037] As mentioned herein, the post-capture phase refers to the time period in which the media content has been captured by a given device (e.g., using a capture sensor such as a camera) and has been stored in a given storage device and / or a persistent storage device. In addition, such media content may have undergone transcoding and / or compression to convert the media content into an appropriate format for storage. As further described herein, a user of a messaging client application can retrieve the stored media content at a later time period to apply a set of post-processing image operations further described herein, which advantageously utilizes the new graphics processing pipeline, system, method, technique, and user interface as described in more detail herein.

[0038] In various settings, users of mobile computing devices often use and increasingly utilize message systems to provide different types of functions in a convenient manner. As described herein, the message system of this theme includes a practical application that provides an improvement in presenting media overlays on media content (e.g., images, videos, etc.) by performing at least post-processing operations on the media content. Such post-processing operations are implemented by at least an extensible graphics rendering pipeline software architecture (as further described herein), which supports the layering of post-processing operations to provide synthetic media content. Via such an extensible architecture, it is possible to achieve processing and calculation improvements to existing graphics rendering pipelines. Compared with image processing operations performed in conjunction with image and / or video capture operations (e.g., via the camera device of a given electronic computing device), the extensible architecture can further reduce the time delay of presenting post-processing operations.

[0039] As referred to herein, the phrase "media overlay" or "media overlays" includes various image processing operations corresponding to image modifications, filtering, lenses, etc. as further described herein.

[0040] Figure 1 1 is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages and associated content) over a network. The messaging system 100 includes multiple instances of client devices 102, each hosting multiple applications including a messaging client application 104. Each messaging client application 104 is communicatively coupled to other instances of the messaging client application 104 and a messaging server system 108 via a network 106 (e.g., the Internet).

[0041] The message client application 104 is capable of communicating and exchanging data with another message client application 104 and a message server system 108 via a network 106. The data exchanged between the message client applications 104 and between the message client applications 104 and the message server system 108 includes functions (e.g., commands to call functions) and payload data (e.g., text, audio, video, or other multimedia data).

[0042] The message server system 108 provides server-side functionality to a particular message client application 104 via the network 106. Although certain functions of the message system 100 are described herein as being performed by the message client application 104 or by the message server system 108, the location of certain functions within the message client application 104 or the message server system 108 is a design choice. For example, it may be technically preferable to initially deploy certain technologies and functions within the message server system 108, but later migrate the technologies and functions to the message client application 104 where the client device 102 has sufficient processing power.

[0043] The message server system 108 supports various services and operations provided to the message client applications 104. Such operations include sending data to the message client applications 104, receiving data from the message client applications 104, and processing data generated by the message client applications 104. As examples, the data may include message content, client device information, geographic location information, media annotations and overlays, message content persistence conditions, social network information, and live event information. The data exchange within the message system 100 is invoked and controlled by functions available through the user interface (UI) of the message client application 104.

[0044] Turning now specifically to the message server system 108, an application program interface (API) server 110 is coupled to and provides a programming interface to an application server 112. The application server 112 is communicatively coupled to a database server 118, which facilitates access to a database 120 in which data associated with messages processed by the application server 112 is stored.

[0045] The application program interface (API) server 110 receives and sends message data (e.g., commands and message payloads) between the client device 102 and the application server 112. Specifically, the application program interface (API) server 110 provides a set of interfaces (e.g., routines and protocols) that can be called or queried by the message client application 104 to invoke functions of the application server 112. The application program interface (API) server 110 exposes various functions supported by the application server 112, including account registration, login functionality, sending messages from a particular message client application 104 to another message client application 104 via the application server 112, sending media files (e.g., images or videos) from a message client application 104 to a message server application 114 and possible access for another message client application 104, setting a collection of media data (e.g., a story), retrieving a friend list of a user of the client device 102, retrieving such a collection, retrieving messages and content, adding and removing friends from a social graph, locating a friend's position within a social graph, and opening application events (e.g., involving the message client application 104).

[0046] The application server 112 hosts multiple applications and subsystems, including a message server application 114, an image processing system 116, and a social networking system 122. The message server application 114 implements a number of message processing techniques and functions, particularly those related to the aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of the message client application 104. As will be described in further detail, text and media content from multiple sources can be aggregated into collections of content (e.g., called stories or libraries). The message server application 114 then makes these collections available to the message client application 104. Given the hardware requirements for other processor and memory intensive processing of data, such processing can also be performed on the server side by the message server application 114.

[0047] The application server 112 also includes an image processing system 116 that is dedicated to performing various image processing operations, typically with respect to images or videos received within the payload of messages at the message server application 114 .

[0048] The social networking system 122 supports various social networking functional services and makes these functional services available to the message server application 114. To this end, the social networking system 122 maintains and accesses the entity graph 304 (e.g., Figure 3 ). Examples of functionality and services supported by the social networking system 122 include identifying other users of the messaging system 100 with whom a particular user has relationships or that a particular user "follows," and identifying interests and other entities of a particular user.

[0049] The application server 112 is communicatively coupled to a database server 118 , which facilitates access to a database 120 in which data associated with messages processed by the message server application 114 is stored.

[0050] Figure 2 1 is a block diagram showing additional details about the messaging system 100 according to an example implementation. Specifically, the messaging system 100 is shown to include a messaging client application 104 and an application server 112, which in turn contain a number of certain subsystems, namely, a transient timer system 202, a collection management system 204, and an annotation system 206.

[0051] The transient timer system 202 is responsible for implementing temporary access to content permitted by the message client application 104 and the message server application 114. To this end, the transient timer system 202 incorporates multiple timers that selectively display messages and associated content and enable access to the messages and associated content via the message client application 104 based on durations and display parameters associated with the message or collection of messages (e.g., a story). Additional details regarding the operation of the transient timer system 202 are provided below.

[0052] The collection management system 204 is responsible for managing collections of media (e.g., collections of text, image video, and audio data). In some examples, collections of content (e.g., messages including images, videos, text, and audio) may be organized into "event libraries" or "event stories." Such collections may be made available for a specified period of time (e.g., the duration of an event related to the content). For example, content related to a concert may be made available as a "story" for the duration of the concert. The collection management system 204 may also be responsible for publishing an icon to the user interface of the messaging client application 104 that provides notification that a particular collection exists.

[0053] In addition, the collection management system 204 also includes a curation interface 208, which enables collection managers to manage and curate specific content collections. For example, the curation interface 208 enables event organizers to curate content collections related to specific events (e.g., delete inappropriate content or redundant messages). In addition, the collection management system 204 uses machine vision (or image recognition technology) and content rules to automatically curate content collections. In some embodiments, compensation can be paid to users to include user-generated content in the collection. In such a case, the curation interface 208 operates to automatically pay such users for using their content.

[0054] The annotation system 206 provides various functions that enable users to annotate or otherwise modify or edit media content associated with a message. For example, the annotation system 206 provides functions related to generating and publishing media overlays for messages processed by the messaging system 100. The annotation system 206 is operable to supply media overlays or supplements (e.g., image filtering) to the messaging client application 104 based on the geographic location of the client device 102. In another example, the annotation system 206 is operable to supply media overlays to the messaging client application 104 based on other information (e.g., social network information of the user of the client device 102). The media overlay can include audio and visual content and visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects can be applied to media content items (e.g., photos) at the client device 102. For example, the media overlay can include text that can be superimposed on a photo taken by the client device 102. In another example, the media overlay includes a location identifier (e.g., Venice Beach) overlay, a name of a live event, or a business name (e.g., Beach Cafe) overlay. In another example, the annotation system 206 uses the geographic location of the client device 102 to identify a media overlay that includes the name of a business at the geographic location of the client device 102. The media overlay may include other tags associated with the business. The media overlay may be stored in the database 120 and accessed through the database server 118.

[0055] In one example implementation, the annotation system 206 provides a user-based publishing platform that enables a user to select a geographic location on a map and upload content associated with the selected geographic location. The user can also specify the circumstances under which a particular media overlay should be provided to other users. The annotation system 206 generates a media overlay including the uploaded content and associates the uploaded content with the selected geographic location.

[0056] In another example implementation, the annotation system 206 provides a merchant-based publishing platform that enables merchants to select specific media overlays associated with geographic locations through a bidding process. For example, the annotation system 206 associates the media overlays of the highest bidding merchant with the corresponding geographic location within a predefined amount of time.

[0057] Figure 3 is a schematic diagram illustrating a data structure 300 that may be stored in a database 120 of a message server system 108 according to certain example implementations. Although the contents of the database 120 are illustrated as including several tables, it should be understood that the data may be stored in other types of data structures (e.g., as an object-oriented database).

[0058] The database 120 includes message data stored in a message table 314. The entity table 302 stores entity data, including an entity graph 304. The entities whose records are maintained in the entity table 302 may include individuals, corporate entities, organizations, objects, places, events, etc. Regardless of the type, any entity about which the message server system 108 stores data may be an identified entity. Each entity is provided with a unique identifier, as well as an entity type identifier (not shown).

[0059] The entity graph 304 also stores information about relationships and associations between entities. As examples only, such relationships may be social relationships, professional relationships (e.g., working in a common company or organization) based on interests or based on activities.

[0060] The database 120 also stores annotation data in the form of examples of filters in the annotation table 312. The filters whose data are stored in the annotation table 312 are associated with videos (whose data are stored in the video table 310) and / or images (whose data are stored in the image table 308) and are applied to the videos and / or images. In one example, the filter is displayed as an overlay on an image or video during presentation to the recipient user. The filter can be of various types, including a user-selected filter from a library of filters presented to the sender user by the message client application 104 when the sender user is composing a message. Other types of filters include geographic location filters (also referred to as geographic filters), which can be presented to the sender user based on geographic location. For example, based on geographic location information determined by the GPS unit of the client device 102, the message client application 104 can present a neighborhood or specific location-specific geographic location filter in the user interface. Another type of filter is a data filter, which can be selectively presented to the sender user by the message client application 104 based on other inputs or information collected by the client device 102 during the message creation process. Examples of data filters include the current temperature at a particular location, the current speed at which the sending user is traveling, the battery life of the client device 102, or the current time.

[0061] Other annotation data that can be stored in the image table 308 is data corresponding to a filter (e.g., filter). As mentioned herein, a filter is a type of media overlay. An example of a filter is real-time special effects and sounds that can be added to an image or video.

[0062] As described above, the video table 310 stores video data, which in one embodiment is associated with messages whose records are maintained in the message table 314. Similarly, the image table 308 stores image data, which is associated with messages whose message data is stored in the entity table 302. The entity table 302 may associate various annotations from the annotation table 312 with the various images and videos stored in the image table 308 and the video table 310.

[0063] The story table 306 stores data about a collection of messages and associated image, video, or audio data that are compiled into a collection (e.g., a story or library). The creation of a particular collection can be initiated by a particular user (e.g., each user whose record is maintained in the entity table 302). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcasted by the user. To this end, the user interface of the messaging client application 104 may include an icon that the user can select to enable the sending user to add specific content to his or her personal story.

[0064] A collection may also constitute a "live story" that is a collection of content from multiple users, which is created manually, automatically, or using a combination of manual and automatic techniques. For example, a "live story" may constitute a curated stream of user-submitted content from different locations and events. Users whose client devices have enabled location services and are at a common location event at a particular time may be presented with an option to contribute content to a particular live story, for example, via the user interface of the messaging client application 104. Live stories may be identified to the user by the messaging client application 104 based on his or her location. The end result is a "live story" told from a community perspective.

[0065] Another type of content collection is called a "location story," which enables users whose client devices 102 are located within a particular geographic location (e.g., on a college or university campus) to contribute to a particular collection. In some implementations, contributions to location stories may require secondary authentication to verify that the end user belongs to a particular organization or other entity (e.g., is a student on a university campus).

[0066] Figure 4 is a schematic diagram illustrating the structure of a message 400 generated by a message client application 104 for transmission to another message client application 104 or a message server application 114 according to some embodiments. The content of a particular message 400 is used to populate a message table 314 stored in a database 120, which is accessible by a message server application 114. Similarly, the content of the message 400 is stored in memory as "in transit" or "in flight" data of a client device 102 or application server 112. The message 400 is shown as including the following components:

[0067] Message identifier 402 : a unique identifier that identifies the message 400 .

[0068] Message text payload 404 : Text to be generated by a user via the user interface of the client device 102 and included in the message 400 .

[0069] Message image payload 406 : Image data captured by a camera component of the client device 102 or retrieved from a memory component of the client device 102 and included in the message 400 .

[0070] Message video payload 408 : Video data captured by the camera component or retrieved from the memory component of the client device 102 and included in the message 400 .

[0071] Message audio payload 410 : audio data captured by a microphone or retrieved from a memory component of the client device 102 and included in the message 400 .

[0072] Message annotation 412: Annotation data representing an annotation (eg, a filter, sticker, or other enhancement) to be applied to the message image payload 406, the message video payload 408, or the message audio payload 410 of the message 400.

[0073] Message duration parameter 414: A parameter value indicating the amount of time in seconds that the content of the message (eg, message image payload 406, message video payload 408, message audio payload 410) will be presented to the user or made accessible to the user via the messaging client application 104.

[0074] Message geolocation parameters 416: Geolocation data (e.g., latitude and longitude coordinates) associated with the content payload of the message. A plurality of message geolocation parameter values ​​416 may be included in the payload, each of which is associated with a content item included in the content (e.g., a specific image within message image payload 406 or a specific video in message video payload 408).

[0075] Message story identifier 418: An identifier value that identifies one or more content collections (e.g., "story") associated with a particular content item in message image payload 406 of message 400. For example, multiple images within message image payload 406 may each be associated with multiple content collections using an identifier value.

[0076] Message tags 420: Each message 400 may be tagged with a plurality of tags, each of which indicates the subject of the content included in the message payload. For example, where a particular image included in the message image payload 406 depicts an animal (e.g., a lion), a tag value indicating the relevant animal may be included in the message tags 420. The tag values ​​may be manually generated based on user input, or may be automatically generated using, for example, image recognition.

[0077] • Message sender identifier 422: An identifier (eg, a messaging system identifier, an email address, or a device identifier) ​​that indicates the user of the client device 102 on which the message 400 was generated and from which the message 400 was sent.

[0078] • Message recipient identifier 424: Indicates an identifier (eg, a messaging system identifier, an email address, or a device identifier) ​​of the user of the client device 102 to which the message 400 is addressed.

[0079] The content (e.g., value) of the various components of message 400 may be pointers to locations in a table in which the content data values ​​are stored. For example, the image value in message image payload 406 may be a pointer to a location (or address) in image table 308. Similarly, the value in message video payload 408 may point to data stored in video table 310, the value stored in message annotation 412 may point to data stored in annotation table 312, the value stored in message story identifier 418 may point to data stored in story table 306, and the values ​​stored in message sender identifier 422 and message recipient identifier 424 may point to user records stored in entity table 302.

[0080] Figure 5 is a schematic diagram illustrating an access restriction process 500 according to which access to content (e.g., a multimedia payload of an ephemeral message 502 and associated data) or a collection of content (e.g., an ephemeral message group 504) can be time-restricted (e.g., ephemeral).

[0081] The ephemeral message 502 is shown associated with a message duration parameter 506, the value of which determines the amount of time that the message client application 104 will display the ephemeral message 502 to the recipient user of the ephemeral message 502. In one embodiment, the recipient user may view the ephemeral message 502 for up to a maximum of 10 seconds, depending on the amount of time specified by the sending user using the message duration parameter 506.

[0082] The message duration parameter 506 and the message recipient identifier 424 are shown as inputs to a message timer 512, which is responsible for determining the amount of time that the transient message 502 is shown to a particular recipient user identified by the message recipient identifier 424. In particular, the transient message 502 is shown to the associated recipient user only for a period of time determined by the value of the message duration parameter 506. The message timer 512 is shown as providing an output to a more generalized implementation of the transient timer system 202, which is responsible for the overall timing of displaying content (e.g., the transient message 502) to the recipient user.

[0083] Figure 5 502 is included in an ephemeral message group 504 (e.g., a collection of messages in a personal story or event story). The ephemeral message group 504 has an associated group duration parameter 508, the value of which determines the duration for which the ephemeral message group 504 is presented and accessible to users of the messaging system 100. For example, the group duration parameter 508 may be the duration of a concert, wherein the ephemeral message group 504 is a collection of content belonging to the concert. Alternatively, a user (an owning user or a curator user) may specify a value for the group duration parameter 508 when performing setup and creation of the ephemeral message group 504.

[0084] In addition, each ephemeral message 502 within the ephemeral message group 504 has an associated group participation parameter 510, the value of which determines the duration for which the ephemeral message 502 is accessible within the context of the ephemeral message group 504. Thus, a particular ephemeral message group 504 may "expire" and become inaccessible within the context of the ephemeral message group 504 before the ephemeral message group 504 itself expires according to the group duration parameter 508. The group duration parameter 508, the group participation parameter 510, and the message recipient identifier 424 each provide inputs to a group timer 514, which is operable to first determine whether a particular ephemeral message 502 of the ephemeral message group 504 is to be displayed to a particular recipient user, and if so, how long to display it. Note that the identity of the particular recipient user is also known to the ephemeral message group 504 due to the message recipient identifier 424.

[0085] Thus, the group timer 514 is operable to control the total lifespan of the associated ephemeral message group 504 and the individual ephemeral messages 502 included in the ephemeral message group 504. In one embodiment, each ephemeral message 502 within the ephemeral message group 504 remains viewable and accessible for a period of time specified by the group duration parameter 508. In another embodiment, within the context of the ephemeral message group 504, a particular ephemeral message 502 may expire based on the group participation parameter 510. Note that even within the context of the ephemeral message group 504, the message duration parameter 506 may still determine the duration for which a particular ephemeral message 502 is displayed to a recipient user. Thus, the message duration parameter 506 determines the duration for which a particular ephemeral message 502 is displayed to a recipient user, regardless of whether the recipient user is viewing the ephemeral message 502 within or outside the context of the ephemeral message group 504.

[0086] The transient timer system 202 may also be operable to remove a particular transient message 502 from a transient message group 504 based on a determination that an associated group participation parameter 510 has been exceeded. For example, when a sender user has established a group participation parameter 510 of 24 hours from posting, the transient timer system 202 will remove the associated transient message 502 from the transient message group 504 after the specified 24 hours. The transient timer system 202 is also operable to remove a transient message group 504 when the group participation parameter 510 for each transient message 502 within the transient message group 504 has expired, or when the transient message group 504 itself has expired according to the group duration parameter 508.

[0087] In some use cases, the creator of a particular ephemeral message group 504 may specify an indefinite group duration parameter 508. In this case, the expiration of the group participation parameter 510 for the last remaining ephemeral message 502 within the ephemeral message group 504 will determine when the ephemeral message group 504 itself expires. In this case, a new ephemeral message 502 with a new group participation parameter 510 added to the ephemeral message group 504 effectively extends the life of the ephemeral message group 504 to a value equal to the group participation parameter 510.

[0088] In response to the transient timer system 202 determining that the transient message group 504 has expired (e.g., is no longer accessible), the transient timer system 202 communicates with the messaging system 100 (and, for example, in particular, the messaging client application 104) so ​​that the indicia (e.g., icon) associated with the relevant transient message group 504 is no longer displayed within the user interface of the messaging client application 104. Similarly, when the transient timer system 202 determines that the message duration parameter 506 for a particular transient message 502 has expired, the transient timer system 202 causes the messaging client application 104 to no longer display the indicia (e.g., icon or text identification) associated with the transient message 502.

[0089] Figure 6 6 is a block diagram 600 illustrating various modules of the annotation system 206 implementing a post-processing graphics pipeline according to certain example embodiments. The annotation system 206 is shown as including a tracking engine 610, a post-processing engine 620, a rendering engine 630, and a sharing engine 640. The various modules of the annotation system 206 are configured to communicate with each other (e.g., via a bus, a shared memory, or a switch). Any one or more of these modules can be implemented using one or more computer processors 605 (e.g., by configuring such one or more computer processors to perform the functions described for the module), and thus can include one or more of the computer processors 605 (e.g., a set of processors provided by the client device 102). In another embodiment, the computer processor 605 refers to a set of processors provided by a server or server system (e.g., the message server system 108).

[0090] Any one or more of the described modules may be implemented using hardware alone (e.g., one or more of the computer processors 605 of a machine (e.g., machine 3100)) or a combination of hardware and software. For example, any described module of the annotation system 206 may physically include an arrangement of one or more computer processors 605 (e.g., a subset of one or more computer processors of a machine (e.g., machine 3100) or one or more of the computer processors therein) configured to perform the operations described herein for that module. As another example, any module of the annotation system 206 may include software, hardware, or both software and hardware that configures an arrangement of one or more computer processors 605 (e.g., in one or more computer processors of a machine (e.g., machine 3100)) to perform the operations described herein for that module. Thus, different modules of the annotation system 206 may include and configure different arrangements of such computer processors 605 or a single arrangement of such computer processors 605 at different points in time. Furthermore, any two or more modules of the annotation system 206 may be combined into a single module, and the functionality described herein for a single module may be subdivided between multiple modules. Furthermore, according to various example embodiments, modules described herein as being implemented within a single machine, database, or device may be distributed across multiple machines, databases, or devices.

[0091] For the purpose of clearly explaining the following technical concepts, the following discussion relates to a single input frame. However, it should be understood that the following discussion will also be applicable to media content including multiple frames (eg, video).

[0092] The post-processing engine 620 receives an input frame 602 captured by the client device 102 (e.g., included in a captured image or video). For example, the input frame 602 can be an image captured by an optical sensor (e.g., a camera) of the client device 102 during the capture of the image or video. In an example, the image includes one or more real-world features, such as a physical object detected in the image. In some implementations, the input frame 602 includes metadata describing the image and / or other information as further discussed herein.

[0093] As further shown, the tracking engine 610 receives the input frame 602 (or alternatively, a copy of the input frame 602 in embodiments). The tracking engine 610 may include various tracking functions based on the type of object to be tracked. In an example, the tracking engine 610 includes tracking capabilities for surface tracking, face tracking, object tracking, etc. In an implementation, the tracking engine 610 may perform only one of each of the multiple tracking processes at a time to facilitate management of computing resources at the client device 102. In addition, the tracking engine 610 may perform one or more object recognition operations on the input frame 602.

[0094] As mentioned herein, tracking refers to an operation for determining the spatial properties (e.g., position and / or orientation) of a given object (or part thereof) during a post-processing stage. In an implementation, during tracking, the position and orientation of an object are measured in a continuous manner. Different objects, such as a user's head, eyes or limbs, surfaces, or other objects can be tracked. Tracking involves dynamic sensing and measurement to enable virtual objects and / or effects to be presented relative to physical objects in a three-dimensional space corresponding to a scene (e.g., input frame 602). Therefore, the tracking engine 610 determines metrics corresponding to at least the relative position and orientation of one or more physical objects in the input frame 602, and includes these metrics in the tracking data 606 provided to the post-processing engine 620. In an example, the tracking engine 610 updates (e.g., tracks over time) such metrics from frame to subsequent frame.

[0095] In implementations, the annotation system 206 may utilize techniques that combine information from a device's motion sensors (e.g., accelerometers and gyroscope sensors, etc.) with analysis of the scene provided in the input frame 602. For example, the annotation system 206 detects features in the input frame 602 and, accordingly, tracks differences in corresponding positions of such features across several input frames using information derived at least in part based on data from the device's motion sensors.

[0096] As described herein, surface tracking refers to operations for tracking one or more representations of a surface corresponding to a plane in an input frame (e.g., a given horizontal plane, a floor, a table). In an example, surface tracking is accomplished using hit testing and / or ray casting techniques. In an example, a hit test determines whether a selected point (e.g., a pixel or a collection of pixels) in an input frame intersects a surface or plane of a representation of a physical object in the input frame. In an example, ray casting utilizes a Cartesian-based coordinate system (e.g., x and y coordinates) and projects rays (e.g., vectors) into a camera's view of the world as captured in the input frame to detect planes that the rays intersect.

[0097] As referred to herein, face tracking refers to operations for tracking representations of facial features (e.g., portions of a user's face) in input frames. In some implementations, the tracking engine 610 includes face tracking logic to identify all or a portion of a face within one or more images and track landmarks of the face across the set of images of a video stream. As referred to herein, object tracking refers to tracking representations of physical objects in input frames.

[0098] In an implementation, the tracking engine 610 provides tracking data 606 corresponding to the aforementioned metrics (e.g., position and orientation) as output. In some cases, the tracking engine 610 includes logic for shape recognition, edge detection, or any other suitable object detection mechanism. The tracking engine 610 may also determine that the object of interest is an example of a predetermined object type, match a shape, edge, or marker within a range to an object type in a set of predetermined object types.

[0099] The post-processing engine 620 modifies the virtual content (e.g., changes its size, scale, direction / orientation, color, shape), such as performing operations that affect the visual appearance and / or position of a virtual object, which may be anchored to a representation of a physical object in the scene of the input frame 602.

[0100] Post-processing engine 620 receives media overlay metadata 604 and input frame 602. In an example, media overlay metadata 604 includes information about a set of media overlays applied as effects to input frame 602 by post-processing engine 620. In an example, media overlay metadata 604 includes the following in combination: Figure 7 Further describing the information, this information may be included as part of the input frame 602 (or stored separately).

[0101] In an embodiment, the post-processing engine 620 utilizes machine learning techniques to apply image processing to facial features in the input frame 602 using the media overlay metadata 604. An example of a machine learning model is a machine learning model (e.g., one or more neural networks) that has been trained to generate an "average face" by marking the boundaries of facial features based on training data (e.g., thousands of images). In this example, the machine learning model generates an "average face" that can be used to align with a face (e.g., an object of interest) detected in the input frame 602 for performing image processing to modify the input frame to generate an output frame 608. In addition, the post-processing engine 620 can perform one or more modifications to the portion of the object of interest described above to generate a modified input frame (e.g., output frame 608).

[0102] In one embodiment, the post-processing engine 620 cooperates with the sharing engine 640 to generate a message. The message includes the output frame 608, the media overlay metadata 609 and / or other metadata (e.g., as described herein), and in some cases, the original input frame 602. The sharing engine 640 sends the generated message from the client device 102 to one or more other client devices, the message server system 108, or a storage device of the client device 102. In addition, it should be understood that by including the original input frame 602, example embodiments enable non-destructive editing of the message to be provided, whereby different image processing operations can be applied to the original input frame 602, which replaces the previous image processing operations performed on the output frame 608.

[0103] Additionally, rendering engine 630 uses media overlay metadata 609 to perform rendering of the content for display by messaging client application 104. This rendering may be performed in conjunction with a graphics processing pipeline and / or media overlay stacking (e.g., applying multiple media overlays to media content) as further described herein.

[0104] Although the above Figure 6 The discussion of relates to input frames, but it should be understood that the aforementioned components of the annotation system can perform similar operations on a set of images (e.g., a video) corresponding to individual frames of such video content.

[0105] Figure 7 702 according to some embodiments includes additional information corresponding to metadata for post-processing an image (eg, input frame 702). Figure 4 Schematic diagram of the structure of message annotation 412) described in.

[0106] In an embodiment, comprising Figure 7 The additional data shown (such as Figure 3 The content of a particular message 400 (shown) is used to populate a message table 314 for a given message stored in the database 120, which can then be accessed by the message client application 104. Figure 7 As shown, the message annotation 412 includes the following components corresponding to data related to the post-processing operation:

[0107] o Media Overlay Identifier 752: Identifier of the media overlay used in messages with post-processing

[0108] oMessage identifier 754: Identifier of the message with post-processing

[0109] o Asset identifiers 756: A set of identifiers for assets in messages with post-processing. For example, corresponding asset identifiers may be included for assets determined by a particular media overlay. In an embodiment, such assets are created by a media overlay on a sender-side client device, uploaded to the message server application 114, and used on a recipient-side client device to recreate the message. Examples of typical assets include:

[0110] ■Raw static RGB images captured by the camera

[0111] ■Post-processed image with media overlay effects applied to the original image

[0112] o Media overlay metadata 758: Additional metadata associated with the media overlay corresponding to media overlay identifier 752, such as:

[0113] oMedia Overlay Category: corresponds to the type or classification of a specific media overlay

[0114] oMedia Overlay Carousel Index

[0115] o Carousel Group: A carousel group may be populated and used when a post-capture media overlay that meets the criteria is inserted into the carousel interface. In an implementation, a new value "POST_CAPTURE_LENS_DEFAULT_GROUP" (e.g., the default group assigned to post-capture media overlays) may be added to the list of valid group names, and post-capture media overlays may be included in this group unless another group has been explicitly set to correspond to the metadata of a geolocation entity in the data store.

[0116] o Metadata 760 corresponding to additional metadata, for example:

[0117] oCamera Image Metadata

[0118] ■ Camera device internal data

[0119] Focal length

[0120] Main point

[0121] ■Other camera information (e.g. camera location)

[0122] oSensor information

[0123] ■Gyro sensor data

[0124] ■Position sensor data

[0125] ■Accelerometer sensor data

[0126] ■Other sensor data

[0127] Positioning sensor data

[0128] Fig. 8A and Figure 8B An example of a user capturing an image for post-processing in the messaging client application 104 (or messaging system 100) is shown in accordance with some embodiments.

[0129] Fig. 8A An example situation is shown in which a user 802 is capturing an image of a physical item according to various embodiments. Fig. 8A Only a portable client device (e.g., client device 102) is shown, but it should be understood that various types of electronic or computing devices can capture, receive and / or process images according to various embodiments discussed herein. These client devices may include, for example, a desktop PC, a laptop computer, a tablet computer, a personal digital assistant (PDA), a smart phone, a portable media file player, an e-book reader, a portable computer, a head-mounted display, an interactive kiosk (kiosk), a mobile phone, a netbook, a single board computer (SBC), an embedded computer system, a wearable computer (e.g., a watch or glasses), a game console, a home theater PC (HTPC), a TV, a DVD player, a digital cable box, a digital video recorder (DVR), a computer system capable of running a web browser, or a combination of any two or more of these, etc.

[0130] The client device 102 may have at least one camera 806. Each camera may be, for example, an active pixel sensor in a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or an N-type metal oxide semiconductor (NMOS), an infrared or ultrasonic image sensor, or an image sensor utilizing other types of image capture technology.

[0131] like Fig. 8A As shown in the example of , the user 802 can position the client device 102 so that the user's face, including various facial features, is within the field of view 808 of at least one camera 806 of the client device 102. The at least one camera 806 can capture a still image, which can be stored to a local and / or remote storage device for post-processing at a later time. In some embodiments, the at least one camera 806 captures video, thereby providing a "live" view of the captured video information, which can be stored to a local and / or remote storage device for post-processing at a later time.

[0132] In the post-capture phase, previously captured images are accessed from storage for further processing (eg, post-processing). Figure 8BAs shown in the example of , an image 812 including a facial representation of a user 802 is displayed on a display screen 852 of a client device 102. The user can then select various media overlays to apply effects and modifications to the displayed image.

[0133] Fig.9A and Fig. 9B Examples of post-processing in the messaging client application 104 (or messaging system 100) according to some embodiments are shown. As shown in these examples, example embodiments enable various media overlays to be applied to previously captured images and / or videos in different layers, such that corresponding graphic elements and / or effects are superimposed on each other.

[0134] like Fig.9A and Fig. 9B As shown, the client device 102 may provide a display of a media overlay, which may include visual and audio content corresponding to animations, media content, text content, etc. It should be understood that the media overlay displayed by the client device 102 may include the above-mentioned media overlays. Figure 1 and Figure 2 The audio and visual content and visual effects described.

[0135] like Fig.9A As shown, image 812 including a facial representation of user 802 is shown in display screen 852 of client device 102. First media overlay 910 is applied to image 812. Implementations of the subject technology enable multiple media overlays to be applied to a given image, such as image 812.

[0136] like Fig. 9B As shown, a second media overlay 912 is applied to an image including a first media overlay. The second media overlay 912 is displayed in display screen 852 along with the first media overlay 910. In this manner, multiple media overlays may be applied to an image, thereby generating effects and graphic elements that are displayed with the image to provide a composite image having several applied media overlays.

[0137] Fig. 10A and Fig. 10B Other examples of post-processing in the messaging client application 104 (or messaging system 100) according to some embodiments are shown. As shown in these additional examples, example embodiments enable various media overlays to be applied to previously captured images and / or videos in different layers such that corresponding graphic elements and / or effects are superimposed on each other. More specifically, Fig. 10A and Fig. 10B The example shown in includes images of multiple users on which respective media overlays may be applied to varying degrees as post-processing operations to provide visual and / or audio effects that may be provided for display in conjunction with one another.

[0138] like Fig. 10A and Fig. 10B As shown, the client device 102 provides a display of a media overlay, which may include visual and audio content corresponding to animations, media content, text content, etc. As previously described, it should be understood that the media overlay displayed by the client device 102 may include the above-mentioned Figure 1 and Figure 2 The audio and visual content and visual effects described.

[0139] like Fig. 10A As shown, image 1010 includes a representation of a scene including several users (e.g., different people) shown in display screen 852 of client device 102. A first media overlay 1012 is applied to a first user at the center of the scene, and as shown, the first media overlay provides a graphical representation of virtual glasses (e.g., virtual glasses or mirrors).

[0140] like Fig. 10B As shown, a second media overlay 1014 and a third media overlay 1016 are applied to the image 1010 with the first media overlay. The second media overlay 1014 and the third media overlay 1016 are displayed with the first media overlay 1012. In this example, the second media overlay 1014 includes an animation of a graphical element (e.g., a virtual representation of leaves), and the third media overlay 1016 includes a graphical element that obscures the user's corresponding face. Thus, multiple media overlays can be applied to an image with multiple people, resulting in effects and graphical elements displayed with the image to provide a composite image with several applied media overlays.

[0141] In order to enable the selection of media overlays for applications as one or more post-processing operations, example embodiments provide various user interface implementations to facilitate an interactive user experience responsive to various inputs provided by a given user. In some embodiments, the messaging client application 104 may provide the following information about Fig.11 The sample user interface discussed.

[0142] In the discussion below, example embodiments enable delivery and organized presentation of media overlays for post-processed media content. In some examples, media overlays can be displayed during the post-processing phase with previously captured media content (e.g., images and / or videos). As further described herein, when a user performs a specific touch input gesture (e.g., a slide or drag gesture) within a user interface, a specific media overlay is recognized and presented to the user.

[0143] Fig.11An example of a user interface for selecting a media overlay to apply post-processing to content (eg, an image or video) in a messaging client application 104 (or messaging system 100) is shown in accordance with some implementations.

[0144] like Fig.11 As shown, the client device 102 provides a display of a media overlay, which may include visual and audio content corresponding to animations, media content, text content, etc. As previously mentioned, it should be understood that the media overlay displayed by the client device 102 may include the above-mentioned Figure 1 and Figure 2 The audio and visual content and visual effects described.

[0145] As shown in user interface 1100, an image 1105 corresponding to a scene representation with a portrait of a person is displayed in display screen 852 of client device 102, the image 1105 including a first media overlay 1110 (e.g., graphical text indicating the day of the week). In the example, the first media overlay 1110 is selected based on a gesture input (e.g., a swipe, a touch input, etc.), which provides for presentation of the selected first media overlay 1110 to display a preview of the media overlay on the image. Fig.11 In the example of , gesture input can be received by the messaging client application 10 (or messaging system 100), the gesture input triggering selection of a corresponding media overlay, which is then provided for display, and the gesture input can also cause the current corresponding media overlay to be removed from the presentation of a composite image that includes one or more media overlays.

[0146] As further shown, in the user interface 1100, a second media overlay 1120 is provided for display in the display screen 852 of the client device 102. In the example, the second media overlay 1120 is displayed in response to a gesture input (e.g., a swipe), and then displayed in a progressive manner as the gesture input moves across the display screen 852 of the client device 102 (e.g., as indicated by a left arrow). The second media overlay 1120 also includes a text and icon overlay 1130 (e.g., "Bearded Guy on Bicycle"), which indicates a description of the selected media overlay for application during post-processing. In this example, the second media overlay 1120 is used as a transition effect for the transition between the first media overlay 1110 and the newly selected media overlay. As further shown, the first media overlay 1110 is shown as moving outward as the gesture input progresses across the display screen 852 and eventually disappearing from the display screen 852.

[0147] Fig.12An example of a user interface for selecting a media overlay to apply post-processing to content (e.g., an image or video) in a messaging client application 104 (or messaging system 100) is shown in accordance with some embodiments, following Fig.11 The examples described in .

[0148] like Fig.12 As shown, the client device 102 can provide a display of a media overlay, which can include visual and audio content corresponding to animations, media content, text content, etc. As previously mentioned, it should be understood that the media overlay displayed by the client device 102 can include the above-mentioned Figure 1 and Figure 2 The audio and visual content and visual effects described.

[0149] As shown in user interface 1100, a representation of a scene 1205 with a portrait of a person (e.g., similar to the above) is displayed on display screen 852 of client device 102. Fig.11 The image includes a second media overlay 1120 (eg, a transition effect) and a text and icon overlay 1130. In the example, in the above Fig.11 After the gesture input (e.g., swipe, touch input, etc.) in the display is completed, the second media overlay 1120 is displayed in the entire display screen 852. As further shown, a graphical indication 1210 is displayed to indicate that a post-processing operation related to the selected media overlay is currently being performed (e.g., as a background process) to prepare for presentation of the selected media overlay.

[0150] As further shown in user interface 1100, the selected media overlay is now displayed in display screen 852 of client device 102 as third media overlay 1230 in representation of scene 1205 along with the portrait of the person. In this manner, different selected media overlays may be provided for display using one or more gesture inputs. In another example, a subsequent gesture input may be received that repeats the gesture input. Fig.11 and Fig.12 Similar operations as described in where another media overlay is applied to a representation of a scene.

[0151] Although in Fig.11 and Fig.12 Gesture input is discussed in the examples of , but it should be understood that other inputs may be received, including non-touch-based input and / or movement-based input (e.g., from a head-mounted display such as a virtual reality (VR), mixed reality (MR), or augmented reality (AR) device).

[0152] As described above, media overlays, overlays, image transformations, AR images, and similar terms such as filters refer to modifications that can be made to a video or image. This includes real-time modifications that modify the image when it is captured using a device sensor, and then display the modified image on the device's screen. This also includes modifications to stored content, such as modifications to video clips in a library that can be modified. For example, in a device that accesses multiple media overlays (e.g., filters), a user can use a single video clip with multiple filters to see how different filters will modify the stored clips. For example, multiple filters that apply different pseudo-random movement models can be applied to the same content by selecting different filters for the content. Similarly, real-time video capture can be used with the modifications shown to show how the video image currently captured by the device's sensor will modify the captured data. Such data can be simply displayed on the screen without being stored in memory, or the content captured by the device sensor can be recorded and stored in memory with or without modification (or both). In some systems, a preview function can show how different filters look in different windows of the display at the same time. For example, this can enable multiple windows with different pseudo-random animations to be viewed on the display at the same time.

[0153] Thus, data and various systems for utilizing the data to modify content using filters or other such transformation systems may involve detecting objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.), tracking such objects as they leave, enter, and move around a field of view in a video frame, and modifying or transforming such objects as they are tracked. In various embodiments, different methods for implementing such transformations may be used. For example, some embodiments may involve generating a three-dimensional mesh model of one or more objects and using transformations and animated textures of the models in a video to implement the transformations. In other embodiments, tracking of points on an object may be used to place an image or texture (which may be two-dimensional or three-dimensional) at the tracked location. In yet another embodiment, a neural network analysis of a video frame may be used to place an image, model, or texture in content (e.g., an image or video frame). Thus, filter data refers to both images, models, and textures used to create content transformations, as well as additional modeling and analysis information required to implement such transformations through object detection, tracking, and placement.

[0154] Real-time video processing can be performed using any type of video data (e.g., video streams, video files, etc.) stored in the memory of any type of computerized system. For example, a user can load a video file and save it in the memory of the device, or a video stream can be generated using the device's sensors. In addition, computer animation models can be used to process any object, such as a human face and parts of a human body, an animal, or a non-living thing, such as a chair, a car, or other objects.

[0155] In some embodiments, when a specific modification is selected together with the content to be converted, the elements to be converted are identified by the computing device, and then the elements to be converted are detected and tracked when the elements to be converted are present in the frame of the video. The elements of the object are modified according to the modification request, so the frame of the video stream is converted. The conversion of the frame of the video stream can be performed by different methods for different kinds of conversions. For example, for frame conversions that mainly involve changing the form of object elements, the feature points of each of the elements of the object are calculated (for example, using active shape models (ASM) or other known methods). Then, a grid based on the feature points is generated for each of at least one element of the object. The grid is used for the subsequent stage of tracking the object elements in the video stream. During the tracking process, the grid of each element mentioned is aligned with the position of each element. Then, additional points are generated on the grid. A first set of first points is generated for each element based on the modification request, and a set of second points is generated for each element based on the set of first points and the request for modification. Then, the frame of the video stream can be converted by modifying the elements of the object based on the set of first points and the set of second points and the grid. In such a method, by tracking and modifying the background, the background of the modified object can also be changed or deformed.

[0156] In one or more embodiments, the transformation of changing some areas of the object using the elements of the object can be performed by calculating the feature points of each element of the object and generating a grid based on the calculated feature points. Points are generated on the grid, and then various areas based on the points are generated. Then, the elements of the object are tracked by aligning the area of ​​each element with the position of each of the at least one element, and the properties of the area can be modified based on the request for modification, thereby transforming the frame of the video stream. Depending on the specific modification request, the properties of the mentioned area can be transformed in different ways. Such modifications may involve: changing the color of the area; removing at least some parts of the area from the frame of the video stream; including one or more new objects in the area based on the modification request; and modifying or deforming the elements of the area or object. In various embodiments, any combination of such modifications or other similar modifications can be used. For certain models to be animated, some feature points can be selected as control points to determine the entire state space of options for model animation.

[0157] In certain embodiments of a computer animation model for converting image data using face detection, faces are detected on an image using a specific face detection algorithm (e.g., Viola-Jones). An active shape model (ASM) algorithm is then applied to the face region of the image to detect facial feature reference points.

[0158] In other embodiments, other methods and algorithms suitable for face detection may be used. For example, in some embodiments, features are located using markers that represent distinguishable points that are present in most of the images considered. For example, for facial markers, the location of the left eye pupil can be used. When the initial markers are not recognizable (for example, if the person is wearing an eye patch), secondary markers can be used. Such a marker recognition process can be used for any such object. In some embodiments, a collection of markers forms a shape. The shape can be represented as a vector using the coordinates of the points in the shape. One shape is aligned with another shape using a similarity transformation (allowing translation, scaling, and rotation) that minimizes the average Euclidean distance between the shape points. The average shape is the mean of the aligned training shapes.

[0159] In some embodiments, a search for landmark points begins from an average shape that aligns with the position and size of a face determined by a global face detector. Such a search then repeats the steps of suggesting tentative shapes by adjusting the positioning of shape points through template matching of the image texture around each point, and then conforming the tentative shapes to a global shape model until convergence occurs. In some systems, individual template matches are unreliable, and the shape model brings together the results of weak template matchers to form a stronger overall classifier. The entire search is repeated at each level of the image pyramid from coarse resolution to fine resolution.

[0160] Embodiments of the conversion system may capture an image or video stream on a client device and perform complex image manipulations locally on a client device, such as client device 102, while maintaining a suitable user experience, computational time, and power consumption. Complex image manipulations may include size and shape changes, emotion transfer (e.g., changing a face from a frown to a smile), state transfer (e.g., aging a subject, reducing apparent age, changing gender), style transfer, application of graphical elements, and any other suitable image or video manipulation enabled by a convolutional neural network that has been configured to execute efficiently on a client device.

[0161] In some example embodiments, a computer animation model for converting image data may be used by a system in which a user may capture an image or video stream of the user (e.g., a selfie) using a client device 102 having a neural network running as part of a messaging application 104 running on the client device 102. A conversion system running within the messaging application 104 determines the presence of a face in the image or video stream and provides a modification icon associated with the computer animation model to convert the image data or the computer animation model may be presented in association with an interface described herein. The modification icon includes a change that may be a basis for modifying the user's face in the image or video stream as part of a modification operation. Once the modification icon is selected, the conversion system initiates a process for transforming the user's image to reflect the selected modification icon (e.g., generating a smiley face for the user). In some embodiments, once the image or video stream is captured and the specified modification is selected, the modified image or video stream may be presented in a graphical user interface displayed on a mobile client device. The conversion system may implement a complex convolutional neural network on a portion of the image or video stream to generate and apply the selected modification. That is, once the modification icon is selected, the user may capture the image or video stream and be presented with the result of the modification in real time or near real time. Furthermore, the modification may be persistent while the video stream is captured and the selected modification icon remains open. A machine taught neural network may be used to implement such modification.

[0162] In some embodiments, a graphical user interface presenting the modifications performed by the conversion system may supply the user with additional interactive options. Such options may be based on an interface for initiating content capture and selection of a particular computer animation model (e.g., initiated from a content creator user interface). In various embodiments, the modification may persist after the initial selection of the modification icon. The user may turn the modification on or off by tapping or otherwise selecting a face modified by the conversion system. And store it for later viewing or browsing other areas of the imaging application. In the case of multiple faces modified by the conversion system, the user may globally turn the modification on or off by tapping or selecting a single face modified and displayed within the graphical user interface. In some embodiments, each face in a group of multiple faces may be modified individually by tapping or selecting an individual face or a series of individual faces displayed within the graphical user interface, or such modifications may be performed individually.

[0163] In some example embodiments, a graphics processing pipeline architecture is provided that enables different media overlays to be applied in corresponding different layers. Such a graphics processing pipeline provides an extensible rendering engine for providing multiple media overlays included in composite media (e.g., images or videos) for presentation by the message client application 104 (or the message system 100).

[0164] In implementations described herein, computer graphics systems that can render two-dimensional (2D) objects or objects from a three-dimensional (3D) world (real or imaginary) onto a 2D display screen are currently used in a wide variety of applications. In some implementations, such a graphics system (e.g., a graphics system included on a client device 102) includes a graphics processing unit (GPU) for performing image processing operations and rendering graphical elements for display. Fig.13 and Fig.14 The following discussion of describes an example graphics processing pipeline. In some embodiments, in conjunction with the above description of at least Fig. 8A , Figure 8B , Fig.9A , Fig. 9B , Fig. 10A , Fig. 10B , Fig.11 and Fig.12 Operations are described to utilize an example graphics processing pipeline to provide layering of various media overlays for post-processing of media content.

[0165] In an implementation, the GPU includes a logical graphics processing pipeline that can receive a representation of a 2D or 3D scene and provide an output of a bitmap representing a 2D image for display. Existing application programming interfaces (APIs) have implemented a graphics pipeline model. Examples of such APIs include the Open Graphics Library (OPENGL) API and the METAL API. The graphics processing pipeline includes several stages that convert a set of vertices, textures, buffers, and state information into image frames on a screen. In an implementation, one of the stages of the graphics processing pipeline is a shader, which can be used as a part of a specific media overlay applied to an input frame (e.g., an image or video). The shader can be implemented as code running on a dedicated processing unit (also referred to as a shader unit or shader processor) that typically executes several computing threads, and the code is programmed to generate appropriate levels of color and / or specific effects for the fragment being presented. For example, a vertex shader processes the attributes of a vertex (position, texture coordinates, color, etc.), and a pixel shader processes the attributes of a pixel (texture value, color, z depth, and alpha value). In some instances, a pixel shader is called a fragment shader.

[0166] It should be understood that other types of shader processing may be provided. In an example, an entire frame is rendered using a particular sampling rate within a graphics processing pipeline, and / or pixel shading is performed at a particular per-pixel rate. In this manner, a given electronic device (e.g., client device 102) operates a graphics processing pipeline to convert information corresponding to an object into a bitmap that can be displayed by the electronic device. In some implementations, example embodiments provide aspects of a graphics processing pipeline as described above to provide post-processing (as further described below) for layering (e.g., stacking) individual media overlays onto media content to render composite media content.

[0167] Fig.13 1 is a schematic diagram of an example of a graphics processing pipeline, i.e., a post-processing pipeline 1300 implemented for a component (e.g., a GPU) of a client device 102, according to some example implementations. More specifically, Fig.13 An example of a graphics processing pipeline is shown in which graphics operations performed on a given input frame are accumulated such that the input frame is processed according to a selected first media overlay, and the output of that processing is then provided as input for processing according to a selected second media overlay, and so on throughout the remaining stages of the graphics processing pipeline.

[0168] In an embodiment, Fig.13 (and the following discussion Fig.14) implements an extensible rendering engine that supports multiple image processing operations corresponding to each media overlay. Such an extensible rendering engine enables a reduction in memory resource consumption of a given electronic device (e.g., client device 102) because a single rendering engine can be loaded into memory for execution, rather than having multiple different rendering engines in memory that each occupy different memory spaces as in some existing implementations.

[0169] In one example, the client device 102 is configured to implement one or more of the stages of the post-processing pipeline 1300, which are illustrated as various media overlays 1310-1330, each corresponding to a respective media overlay. In an example, each media overlay 1310-1330 may be configured, for example, to perform one or more shader and / or image processing operations corresponding to a particular media overlay.

[0170] As shown in media overlay 1310 , the post-processing pipeline 1300 receives an original frame 1312 and performs post-processing operations 1314 (e.g., transformations, color effects, shader effects (e.g., distortion), face effects, etc.) based on the selected first media overlay, and outputs a processed frame 1316 .

[0171] As shown in media overlay 1320, post-processing pipeline 1300 receives processed frame 1316 and performs a second post-processing operation 1322 based on the selected second media overlay. Second processed frame 1324 is provided as output to a subsequent media overlay corresponding to media overlay 1330.

[0172] In addition, as shown in the media overlay 1330, the post-processing pipeline 1300 receives the second processed frame 1324 and performs a third post-processing operation 1332 based on the selected third media overlay. In another example, the media overlay 1330 is instead an output layer that generates pipeline output data (e.g., the second processed frame 1324) for presentation, whereby the third post-processing operation 1332 corresponds to an operation on the pipeline output data.

[0173] Fig.14 1 is a schematic diagram of an example of a graphics processing pipeline, i.e., a post-processing pipeline 1400 implemented for a component (e.g., a GPU) of a client device 102, according to some example implementations. More specifically, Fig.14The example of shows a graphics processing pipeline in which graphics operations are performed separately on a given input frame, so that the input frame is processed according to a selected first media overlay, and the same (original) input frame is provided as input for processing according to a selected second media overlay, and the respective processed input frames are then composited into a single composite image for presentation. Therefore, it should be understood that Fig.14 and Fig.13 The difference is Fig.14 Each media overlay in the post-processing pipeline is applied to the original input frame.

[0174] As shown, the client device 102 is configured to implement one or more of the stages of the post-processing pipeline 1400, one or more of which are shown as various media overlays 1410-1430 corresponding to respective media overlays. In an example, each media overlay 1410-1430 may be configured to, for example, perform one or more shader and / or image processing operations corresponding to a particular media overlay.

[0175] As shown in media overlay 1410 , the post-processing pipeline 1400 receives a raw frame 1412 and performs a first post-processing operation 1414 (e.g., conversion, color effects, shader effects (e.g., distortion), face effects, etc.), and outputs a processed frame 1416 .

[0176] As shown in media overlay 1420, post-processing pipeline 1400 receives original frame 1412 and performs a second post-processing operation 1422 based on the selected second media overlay, and outputs processed frame 1424. Before continuing to media overlay 1430, post-processing pipeline 1400 generates composite frame 1426, which blends, merges or otherwise combines processed frame 1416 and processed frame 1424.

[0177] In addition, the post-processing pipeline 1400 receives the synthesized frame 1426 and performs a third post-processing operation 1432 based on the selected third media overlay, as shown in the media overlay 1430. In another example, the media overlay 1430 is instead an output layer that generates pipeline output data (e.g., the synthesized frame 1426) for presentation, whereby the third post-processing operation 1432 corresponds to an operation on the pipeline output data.

[0178] Fig.15 According to some example embodiments, the graphics processing pipeline (i.e., Fig.13 and Fig.14 Schematic diagram of an example of a stack of media overlays implemented using the post-processing pipeline described in .

[0179] like Fig.15As shown in the example of , the stacking of media overlays can be implemented and executed on a computer processor 1510 (for example, when provided by a given electronic device (for example, client device 102)). The computer processor 1510 applies a first media overlay corresponding to the first media overlay 1501 to the media content. The computer processor 1510 applies a second media overlay corresponding to the second media overlay 1502 to the media content. The computer processor 1510 applies a third media overlay corresponding to the third media overlay 1503 to the media content. In addition, the computer processor 1510 applies a fourth media overlay corresponding to the fourth media overlay 1504 to the media content. In this way, various media overlays can be stacked and applied to media content to be included in a message.

[0180] Although in Fig.15 Four different layers are described as an example in FIG. 1 , but it should be understood that fewer or more layers may be provided without departing from the scope of the subject technology.

[0181] As previously discussed, embodiments of the subject technology enable delivery and organized presentation of media overlays for post-processed media content for use at a computing device (e.g., client device 102). In some examples, individual media overlays can be displayed during the post-processing phase along with previously captured media content (e.g., images and / or videos). As further described herein, when a user performs a particular touch input gesture (e.g., a swipe or drag gesture) within a user interface, a particular media overlay is selected and presented to the user.

[0182] A particular media overlay may be selected based on an order within a set of media overlays and based on an order of the set of media overlays. A particular media overlay may be selected based on an order among other media overlays within a specified classification (e.g., filter type, filter type, etc.), based on an order of other media overlays across other classifications, or a combination thereof. In embodiments, a set of media overlays may be grouped based on the same classification for presentation in a carousel interface. The remaining media overlays in a set of media overlays may also be selected for presentation in a carousel interface, as described below.

[0183] In some embodiments, a given electronic device (e.g., client device 102 and / or message server system 108) determines a ranking score for each media overlay according to a normal distribution. In some embodiments, a corresponding media overlay (e.g., filter) may be given a "boost" to increase the distribution by N standard deviations. In an example, a default score (e.g., priority) may be assigned to each media overlay based on a predetermined value (e.g., 500) within a predetermined value range (e.g., 0 to 1000). In an example, the ranking score may be considered an indicator of the relevance of each media overlay. Such an indicator of relevance (e.g., a relevance score or metric) may be based on various signals including, for example, geographic location, time or date, objects identified in the retrieved image data, use, etc. Based on such a signal, the ranking score may be adjusted to be higher or lower to indicate, respectively, whether the media overlay is considered to be more relevant or less relevant relative to the retrieved image data and / or a classification (e.g., type) associated with the media overlay.

[0184] In some embodiments, machine learning techniques can be used to determine the presentation of media overlays within a user interface. An electronic device (e.g., client device 102) can learn whether a media overlay belongs to one classification or another classification. An electronic device can utilize a sorting method to evaluate the relevance and priority of a media overlay with respect to a classification (e.g., name or description, numerical index, universal unique identifier (UUID), etc.) and order of a corresponding record of a media overlay. A record of a media overlay can be any appropriate data structure including, for example, a table, a list, an array, a vector, a file, etc. After adding a media overlay to a record, an electronic device can change or move the media overlay to change the order in which the media overlay appears in the presentation of the media overlay within the user interface. In an example, an electronic device can remove less relevant or lower priority media overlays from storage, for example, to increase storage available for user-generated content. An electronic device or server can preferentially deliver a more relevant or higher priority media overlay or a collection of media overlays through a network, for example, to improve the responsiveness of the user interface when certain media overlays are selected.

[0185] The set of media overlays may include any number of media overlays. However, it will be appreciated that in some embodiments, the number of media overlays provided for inclusion in the user interface is constrained based on predetermined limits to help reduce the use of computing resources (e.g., memory, display power, battery, etc.). The media overlays in the set may be arranged, sorted, prioritized, or otherwise organized according to one or more predetermined criteria (e.g., based on the media overlay type discussed further below). Each media overlay set (e.g., group) in a plurality of sets of media overlays may be arranged, sorted, prioritized, or otherwise organized according to one or more predetermined criteria (e.g., use, etc.). As discussed further below, each media overlay set may include any number of different media overlays organized based on media overlay type. In addition, media overlays may be organized, prioritized, or otherwise arranged in each set according to use, sorting, etc.

[0186] In some implementations, an electronic device (e.g., client device 102 and / or one or more components of message server system 108) selects a set of media overlays by comparing or matching classifications (e.g., the same media overlay type) associated with one or more media overlays in the set of media overlays.

[0187] In some implementations, each media overlay corresponding to the media overlay is associated with at least one classification (e.g., a category associated with a given media overlay). The classifications can correspond to various categories (e.g., classification labels or category indicators for the media overlays), which are then used to organize (e.g., group) the various media overlays into various sets (e.g., groups) for inclusion in a user interface for presenting (and interacting with) a given user of an electronic device. Fig.16 and Fig.17 As discussed, based at least in part on such groupings, a user may select, via a user interface, a plurality of media overlays to be applied to the media content during a post-processing stage. Figures 18 to 25 The various categories of media overlays are discussed in more detail in the examples in

[0066] .

[0188] In addition, in an implementation, whether a media overlay is selected for inclusion in a carousel depends on whether the media overlay applicable context includes information indicating that the media overlay is considered a post-capture media overlay (e.g., can be previewed or presented by an electronic device during post-processing for display on media content). Thus, the media overlay applicable context can be used to determine that the media overlay can be applied as a post-processing image operation in a post-capture phase relative to the media content. In another example, the context is determined based on a determination that the client device 102 and / or the messaging client application 104 has accessed media content for modification during a post-capture time period, which in turn initiates processing of a record for populating the media overlay, which is then used to select the media overlay for presentation in the carousel.

[0189] Fig.16 An example of a user interface including a carousel for selecting and stacking multiple media overlays to apply post-processing to media content (eg, images or videos) in a messaging client application 104 (or messaging system 100) is shown in accordance with some implementations.

[0190] In implementations of such user interfaces, selectable graphical items 1650 may be presented in a carousel arrangement in which a portion or subset of selectable graphical items 1650 is visible on a display screen of a given computing device (e.g., client device 102). As an example, a user may utilize various inputs to rotate the selectable graphical items onto and off the display screen in a manner corresponding to a carousel providing a cyclic view of the graphical items. Thus, the carousel arrangement provided in the user interface allows multiple graphical items to occupy a particular graphical area on the display screen.

[0191] As described herein, media overlays may be organized into respective groups for inclusion on a carousel arrangement, thereby enabling rotation through the media overlays in groups. Furthermore, media overlays may be selected for inclusion based on various signals including, for example, time, date, geographic location, metadata associated with the media content, etc. Similarly, media overlays may be grouped based on such signals. Furthermore, in an example, a score may be generated and assigned to each media overlay, and a subset of the media overlays may be selected for inclusion on a carousel arrangement of a user interface based at least in part on the respective scores. As described below, in Fig.16 In the carousel arrangement of the user interface example of FIG. 4 , corresponding media overlays are selected from different media overlay groups.

[0192] In the following discussion, selectable graphical items correspond to corresponding media overlays applied to media content as part of post-processing. During post-processing, media content has been previously captured and then retrieved from storage. As shown in user interface 1600, selectable graphical items 1650 corresponding to a carousel arrangement include selectable graphical items 1651 in a display screen of an electronic device (e.g., client device 102). Selectable graphical item 1650 is selected via a user's touch input.

[0193] exist Fig.16 In a second example, the user instead provides a different touch input corresponding to a swipe or drag gesture that enables scrolling through (e.g., provides navigation through the various graphical items presented in user interface 1600) selectable graphical item 1650, and in response to the swipe or drag gesture, a second selectable graphical item 1652 is displayed. As indicated in user interface 1600, second selectable graphical item 1652 belongs to the same media overlay group as selectable graphical item 1650.

[0194] exist Fig.16 In a third example of FIG. 1 , the user continues the swipe or drag gesture, which causes a third selectable graphical item 1654 to be displayed in user interface 1600. In this example, third selectable graphical item 1654 also belongs to the same media overlay group as the respective media overlays corresponding to selectable graphical item 1651 and second selectable graphical item 1652. To apply the media overlay corresponding to third selectable graphical item 1654, another input, such as another touch input (e.g., a tap or press), may be provided by the user.

[0195] In a third example, the user provides a subsequent input (e.g., a tap or press) to apply a media overlay corresponding to the third selectable graphical item 1654. In response, a media overlay 1656 is applied and provided for display in the user interface 1600 on the display screen. After applying the media overlay 1656, the user interface 1600 provides an indication that the third selectable graphical item 1654 was previously selected (e.g., by highlighting the third selectable graphical item 1654).

[0196] exist Fig.16In a fourth example of , the user then provides a subsequent swipe or drag gesture that further enables scrolling through selectable graphical items 1650. In an embodiment, due to the display area limitations of the carousel arrangement, the user interface displays the third selectable graphical item 1654 as a smaller version of itself while also causing other selectable graphical items to be hidden (or disappear) from the carousel arrangement. In response to the swipe or drag gesture, user interface 1600 displays a fourth selectable graphical item 1658, which, based on a graphical indication, belongs to a different media overlay group than the first group that includes the media overlay corresponding to the previously selected third selectable graphical item 1654. The user can continue the swipe or drag gesture (or, alternatively, start a new swipe or drag gesture) to scroll through other selectable graphical items for further selection of additional media overlays (e.g., in a different media overlay group).

[0197] Fig.17 Additional examples of user interfaces (e.g., carousels) for selecting and stacking multiple media overlays to apply post-processing to media content (e.g., images or videos) in the messaging client application 104 (or messaging system 100) are shown in accordance with some embodiments. Fig.16 The example provided Fig.17 For example, refer to Fig.16 discuss Fig.17 .

[0198] exist Fig.17 In the example shown, the above Fig.16 16 is a user interface 1600 described in detail for display on a display screen of an electronic device (eg, client device 102).

[0199] exist Fig.17 In the first example, from Fig.16 Continuing with the fourth example discussed in , selectable graphical item 1751 is included in selectable graphical item 1750, and selectable graphical item 1750 corresponds to the carousel arrangement as displayed in user interface 1600. As further shown, previously in Fig.16 The third optional graphic item 1654 and the fourth optional graphic item 1658 discussed in Fig.17 1750. In this example, the selectable graphical item 1751 based on the graphical indication belongs to the same media overlay group as the fourth selectable graphical item 1658 (not selected as described above). The user provides additional gesture input (e.g., a swipe or drag gesture) to scroll through other selectable graphical items 1750 corresponding to different media overlays.

[0200] exist Fig.17In a second example, the user continues the swipe or drag gesture, which causes selectable graphical item 1752 to be displayed in user interface 1600. Selectable graphical item 1752, based on the graphical indication, belongs to a different media overlay group than the media overlay group corresponding to selectable graphical item 1751 and fourth selectable graphical item 1658. In an embodiment, due to the display area limitation of the carousel arrangement, user interface 1600 displays selectable graphical item 1756, which represents the media overlay group including selectable graphical item 1751 and fourth selectable graphical item 1658. In addition, in the carousel arrangement, user interface 1600 includes selectable graphical item 1754, which represents the media overlay group including selectable graphical item 1651, second selectable graphical item 1652, and third selectable graphical item 1654 (which was previously selected to apply media overlay 1656).

[0201] In this second example, the user provides a subsequent input (e.g., a tap or press) to apply a media overlay corresponding to selectable graphical item 1752. To provide a layering (e.g., stacking) of different media overlays, media overlay 1758 is applied in response to the subsequent input, where media overlay 1758 is provided for display in user interface 1600 along with displayed media overlay 1656. After applying media overlay 1758, user interface 1600 provides an indication that selectable graphical item 1752 was previously selected (e.g., by highlighting selectable graphical item 1752).

[0202] The following discussion relates to examples of different types of media overlays according to some embodiments of the subject technology. In the examples, the media overlay type indicates a specific category that can be used to group associated media overlays. As described above, such groupings of media overlays can then be presented in a carousel arrangement interface, including selectable graphical items for selecting one or more media overlays, and the selected media overlays can be stacked or applied or layered in combination as part of post-processing media content. Based on the aforementioned sorting and / or selection techniques, in the above Fig.16 and Fig.17 Within the carousel interface discussed in , the example media overlays shown in the discussion of the following figures can be selected for inclusion in various groups.

[0203] Fig.18 An example of a media overlay that can be selected to be applied to media content during post-processing is shown. Fig.18 In the example of FIG. 1 , a collection of media overlays based on geographic location is shown, including media overlay 1810, media overlay 1820, media overlay 1830, and media overlay 1840. In at least some embodiments, such geographic location media overlays include static media overlays specific to a geographic location or dynamic media overlays that vary with location.

[0204] Fig.19 An example of a media overlay that can be selected to be applied to media content during post-processing is shown. Fig.19 , a collection of day-based media overlays is shown in the example of , including media overlay 1910, media overlay 1920, media overlay 1930, and media overlay 1940. In at least some embodiments, such day media overlays include media overlays that indicate a day (e.g., a specific day of the week, a holiday, etc.) or media overlays that are related to a specific current day.

[0205] Fig. 20 An example of a media overlay that can be selected to be applied to media content during post-processing is shown. Fig. 20 , including media overlay 2010, media overlay 2020, media overlay 2030, and media overlay 2040. In at least some embodiments, such event media overlays include: media overlays for specific events, media overlays for live story production, sports (e.g., sports teams, etc.) media overlays, and media overlays that fly (e.g., activate) to smaller areas and shorter time frames.

[0206] Fig.21 An example of a media overlay that can be selected to be applied to media content during post-processing is shown. Fig.21 In the example of , a collection of moment-based media overlays is shown, including media overlay 2110, media overlay 2120, media overlay 2130, and media overlay 2140. In at least some embodiments, such moment media overlays include: hyper-contextual media overlays for specific messages, media overlays for shorter time frames and / or specific purpose flights (e.g., activations), media overlays with advanced target conditions including friend / social network media overlays (e.g., friends are used as context), time-based (morning / evening), visual context-based, deep links (e.g., specifying a location in an app defined by a deep link), etc.

[0207] Fig. 22 An example of a media overlay that can be selected to be applied to media content during post-processing is shown. Fig. 22 In the example of FIG. 1 , a set of decorative-based media overlays is shown, including media overlay 2210, media overlay 2220, media overlay 2230, and media overlay 2240. In at least some embodiments, such decorative media overlays include non-message-based media overlays, where the primary purpose is decoration, there are no words or phrases, and can be flown at any time and / or anywhere, and in some cases can also be flown to meet holiday needs.

[0208] Fig.23 An example of a media overlay that can be selected to be applied to media content during post-processing is shown. Fig.23 In the example of , a collection of atmosphere-based media overlays is shown, including media overlay 2310, media overlay 2320, media overlay 2330, and media overlay 2340. In at least some embodiments, such ambient media overlays include media overlays with general, non-contextual expressions, including expression transcripts that are generally targeted to mood / atmosphere, and can fly in a specific place or geographic location at any time.

[0209] Fig.24 An example of a media overlay that can be selected to be applied to media content during post-processing is shown. Fig.24 , a collection of personal-based media overlays is shown in the example of , including media overlay 2410, media overlay 2420, media overlay 2430, and media overlay 2440. In at least some embodiments, such personal media overlays include media overlays personalized to the user, and may include emoticons (e.g., corresponding user icons or user avatars), user names, or be suitable for selfies (e.g., using frontal selfies).

[0210] Fig.25 An example of a media overlay that can be selected to be applied to media content during post-processing is shown. Fig.25 A collection of information template-based media overlays is shown in the example of , including media overlay 2510, media overlay 2520, media overlay 2530, and media overlay 2540. In at least some embodiments, such information template media overlays include media overlays with a combination of dynamic text as the primary content of the media overlay, with minimal or no direct message other than the dynamic text, and may have a decorative, minimal or no template background.

[0211] Fig.26 2 is a flowchart illustrating a method 2600 for generating a message based on at least one post-processing operation on image data according to certain example embodiments. The method 2600 may be embodied in computer-readable instructions for execution by one or more computer processors, such that the operations of the method 2600 may be performed by the message client application 104, particularly with respect to the above-described method 2600. Figure 6 The various components of the annotation system 206 described in the description may be implemented in part or in whole; therefore, the following references Figure 6 The method 2600 is described by way of example. However, it should be understood that at least some operations of the method 2600 may be deployed on various other hardware configurations, and the method 2600 is not intended to be limited to the message client application 104.

[0212] At operation 2602, the post-processing engine 620 retrieves first image data from a storage device. The first image data may be an image such as a photograph or video frame captured by an electronic device at a previous time. For example, as described above with respect to at least Figure 6 As described, the first image data is captured by the client device 102 at a previous time, stored to a storage device, and later retrieved by the post-processing engine 620 .

[0213] At operation 2604, the post-processing engine 620 receives first metadata corresponding to the selected image processing operation (e.g., media overlay). Such metadata may be stored on the client device 102 (or stored in the database 102) and / or generated by the client device 102 to enable post-processing of the media content (e.g., such as Figure 6 and Figure 7 For example, as previously described with respect to at least Figure 6 and Figure 7 As described, metadata or information corresponding to a media overlay to be applied to media content (e.g., first image data) is provided during post-processing (e.g., by client 102 and / or message server system 108) to enable the selected image processing operation to be performed. In addition, the selected image processing operation is displayed via a display of the client device (e.g., using at least Fig.11 , Fig.12 , Fig.16 and Fig.17 In an embodiment, the selected image processing operation corresponds to a media overlay that belongs to a specific media overlay group based on the category associated therewith.

[0214] At operation 2606, the post-processing engine 620 generates second image data based at least in part on the first metadata and the image processing operations performed on the received first image data. For example, as previously described, at least Fig.9A , Fig. 9B , Fig. 10A , Fig. 10B , Fig.16 and Fig.17 In the embodiment of the present invention, the client device 102 applies the image processing operation corresponding to the selected media overlay to the media content (e.g., to enable the applied media overlay to be presented on the display screen of the client device 102). The presentation engine 630 presents the second image data for display by the client device, the second image data including the first image corresponding to the first image data modified by the selected image processing operation (as previously described in at least Figure 6 described in ).

[0215] In another example, the post-processing engine 620 receives sensor data including an orientation of the client device 102 corresponding to the first image data, the sensor data including metadata associated with the first image data, wherein the second image data is also generated based on the sensor data, the received sensor data indicating the orientation of the client device 102. For example, using a specific tracking engine with the sensor data (e.g., provided by the tracking engine 610), the post-processing engine 620 detects the device orientation based on the sensor data, and then generates the second image data in a format that accommodates the device orientation and / or appropriately provides an enhancement of the first image data to thereby generate the second image data. In an embodiment, the post-processing engine 620 may store the second image data separately from the first image data.

[0216] In another example, the first image data includes a representation of the environment. The post-processing engine 620 performs a tracking process based on the first image data. The tracking process can be tracking a surface identified in the representation of the environment, tracking a face identified in the representation of the environment, tracking an object using a machine learning model, etc. In an example implementation, the post-processing engine 620 performs only one tracking process at a time.

[0217] At operation 2608, the post-processing engine 620 generates second metadata including information related to the image processing operation (e.g., media overlay). For example, the second metadata includes a first identifier associated with the first image data, a second identifier associated with the second image data, and / or a previously generated information about at least Figure 7 Other metadata described. In addition, in some embodiments, the second metadata may include the following information to indicate image processing that occurred during the post-processing stage: gesture information (e.g., slide direction), gesture sequence count (e.g., the number of slide sequences or series), tap count (e.g., the number of taps / touch inputs received), media overlay name, media overlay type (e.g., classification indicator), media overlay index, number of media overlays, media overlay score (e.g., relevance indicator), media overlay group, etc.

[0218] At operation 2610, the sharing engine 640 generates a message including the second metadata, the second image data, and the first image data. For example, such a message may include the previous Figure 7 Each of the aforementioned information described.

[0219] At operation 2612, the sharing engine 640 sends a message to a server (e.g., the message server system 108) for sharing with one or more other client devices. For example, the sharing engine 640 sends the message as previously described with respect to at least Figure 6 Such message described.

[0220] Fig. 27 2700 is a flowchart illustrating a method 2700 for generating a message based on a layering of post-processing operations on image data according to certain example embodiments. The method 2700 may be embodied in computer-readable instructions for execution by one or more computer processors, such that the operations of the method 2700 may be performed by the message client application 104, particularly with respect to the above-described method 2700. Figure 6 The various components of the annotation system 206 described in the description may be implemented in part or in whole; therefore, the following references Figure 6 The method 2700 is described by way of example. However, it should be understood that at least some operations of the method 2700 may be deployed on various other hardware configurations, and the method 2700 is not intended to be limited to the message client application 104.

[0221] At operation 2702, the post-processing engine 620 retrieves first image data from a storage device. Figure 6 In the depicted example, first image data is captured by the client device 102 at a previous time, stored to a storage device, and later retrieved by the post-processing engine 620 .

[0222] At operation 2704, post-processing engine 620 detects a selection of a first media overlay corresponding to a first image processing operation. Fig.11 , Fig.12 , Fig.16 and Fig.17 As described in , a user selects a media overlay via input (e.g., touch input, slide gesture, etc.). In addition, based on the category associated with each of the media overlays, the selected first media overlay is associated with a specific media overlay group.

[0223] At operation 2706, the post-processing engine 620 performs a first image processing operation corresponding to the first media overlay on the first image data to generate second image data. Fig.9A , Fig. 9B , Fig. 10A and Fig. 10B As described in , the selected media overlay is applied to the media content (e.g., to enable the applied media overlay to be presented on the display screen of the client device 102). In addition, as previously described in at least Fig.13 , Fig.14 and Fig.15 As described in , a first image processing operation is performed by a first layer of a graphics processing pipeline.

[0224] At operation 2708, post-processing engine 620 detects a selection of a second media overlay corresponding to a second image processing operation. Fig.16 and Fig.17 As described in, the user can select the second media overlay via input (e.g., touch input, slide gesture, etc.). In addition, based on the different categories associated with the second media overlays, the second media overlay corresponding to the second image processing operation is associated with the second group of second media overlays.

[0225] At operation 2710, the post-processing engine 620 performs a second image processing operation on the second image data to generate third image data. Fig.16 and Fig.17 As described in , a second media overlay may be applied to media content that has been previously processed (e.g., with a first media overlay) to provide a layering of media overlays to the media content. Fig.13 , Fig.14 and Fig.15 As described in , the second image processing operation is performed by the second layer of the graphics processing pipeline. In an embodiment, the second layer is performed after the first layer is performed.

[0226] At operation 2712, the sharing engine 640 generates a message including the third image data. Figure 6 , Figure 7 , Fig.13 , Fig.14 , Fig.15 , Fig.16 and Fig.17 As described in , multiple media overlays can be applied to media content to provide a composite image that includes at least a combination of the applied media content. The sharing engine 640 can send the generated message to a server (e.g., the message server system 108) to share with one or more other client devices.

[0227] Fig.28 2800 is a flowchart illustrating a method 2800 for providing a carousel interface for post-processing operations according to certain example embodiments. The method 2800 may be included in computer-readable instructions for execution by one or more computer processors, such that the operations of the method 2800 may be performed by the message client application 104, particularly with respect to the above-described Figure 6 The various components of the annotation system 206 described in the description may be executed in part or in whole; therefore, the following references Figure 6 The method 2800 is described by way of example. However, it should be understood that at least some operations of the method 2800 may be deployed on various other hardware configurations, and the method 2800 is not intended to be limited to the message client application 104.

[0228] At operation 2802, the post-processing engine 620 retrieves, via the client device, first image data previously captured by a client device (e.g., the client device 102) from a storage device. Figure 6 In the example described in , the first image data is captured by the client device 102 at a previous time, stored to a storage device, and later retrieved by the post-processing engine 620. In some implementations, the first image data is then displayed on a display screen of the client device 102.

[0229] At operation 2804, the post-processing engine 620 selects a set of media overlays in response to the display of the first image data. In an example, each respective media overlay is associated with a category indicator. The client device 102 matches the category indicators in the respective media overlays to determine one or more groupings of the media overlays based on the respective category indicators. As previously described in at least Fig.16 and Fig.17 As described in , the client device 102 selects one of the groups for the selected set of media overlays. In another example, the client device 102 selects multiple groups of media overlays.

[0230] In an embodiment, each corresponding media overlay is associated with a score that can correspond to a relevance index of each corresponding media overlay (e.g., a higher score indicates a higher relevance, and a lower score indicates a lower relevance). As previously described, the relevance index can be based on an initial default score or value, and then adjusted to be higher or lower based on one or more signals (e.g., geographic location, time or date, use, objects of interest in image data, etc.). The post-processing engine 620 sorts the media overlays based on the corresponding scores associated with each media overlay. In this example, the post-processing engine 620 selects a predetermined number of the highest ranked media overlays as a collection of media overlays. In another embodiment, the sorting is performed instead by a component of the message server system 108, and the collection of selected media overlays based on the determined sorting can then be provided to the post-processing engine 620 for processing in the manner described above. In an example where a single grouping of media overlays has been selected (e.g., based on a unique matching category indicator), the corresponding scores of each media overlay within the same single grouping can be sorted. In examples where multiple groupings of media overlays have been selected (eg, based on different category indicators), a ranking may be performed on the respective scores of each media overlay within each respective grouping of media overlays.

[0231] At operation 2806, post-processing engine 620 causes a carousel interface (e.g., user interface 1600) to be displayed that includes selectable graphical items. Each of the selectable graphical items corresponds to a respective media overlay in the set of media overlays. In embodiments, the selectable graphical items are provided in the carousel interface based at least in part on the aforementioned ranking of the media overlays, wherein the highest ranked media overlay is selected for display. Thus, it will be appreciated that each selectable graphical item in the carousel interface may correspond to one of the highest ranked media overlays. In addition, as described with respect to at least Fig.16 and Fig.17 As described, a carousel interface is presented on a display screen of the client device 102. As previously described, multiple groups of media overlays can be provided for display in a carousel interface, and corresponding media overlays from a particular group (e.g., based on a matching category indicator) are provided for display based on the aforementioned sorting (e.g., providing the highest sorted media overlay for display). In the example, the user interface 1600 provides a visual indicator that distinguishes the selected first selectable graphical item from other selectable graphical items in the plurality of selectable graphical items that are not selected. In addition, the selected first selectable graphical item is within a first display area, and the first display area is different from a second display area in which other selectable graphical items in the plurality of selectable graphical items that are not selected are presented.

[0232] At operation 2808, the post-processing engine 620 receives a selection of a first selectable graphical item from a plurality of selectable graphical items. Fig.16 and Fig.17 As described in , a particular media overlay corresponding to a particular selectable graphical item may be selected by a user via input (eg, touch input, etc.).

[0233] At operation 2810, the post-processing engine 620 applies a first media overlay corresponding to the selected first selectable graphical item to the first image data. Fig.16 and Fig.17 As described in , the selected media overlay may be applied to the media content (eg, such that the applied media overlay is presented on a display screen of the client device 102).

[0234] Fig.292 is a flow chart illustrating a method 2900 for determining a set of media overlays to populate a set of media overlays, which may be used by a client electronic device during a post-processing phase to modify media content, according to certain example implementations. The method 2900 may be embodied in computer-readable instructions for execution by one or more computer processors, such that the operations of the method 2900 may be performed in part or in whole by the message server system 108; therefore, the method 2900 is described below by way of example with reference to the message server system 108. However, it should be understood that at least some of the operations of the method 2900 may be deployed on various other hardware configurations, and the method 2900 is not intended to be limited to the message server system 108.

[0235] At operation 2902, the message server system 108 receives metadata corresponding to a media overlay. In an example, the metadata may be stored in a database 120 accessible via a database server 118. Alternatively, the metadata is stored by the client 102 (e.g., as part of a storage device associated with the message client application 104). The metadata includes information indicating that the media overlay is configured to be applied as an image processing operation during post-processing of the image data in a post-capture phase. For example, as previously described in at least Figure 6 and Figure 7 Such metadata associated with the media overlay may be used to implement post-processing operations of the message server system 108 as described in .

[0236] At operation 2904, the message server system 108 selects a media overlay indicated in the metadata corresponding to the media overlay. Fig.16 and Fig.17 As described, the message server system 108 determines that a media overlay is selected by determining whether the media overlay applicable context includes information indicating that the media overlay is considered to be a post-capture media overlay (e.g., can be previewed or rendered by an electronic device during post-processing for display on the media content). In one example, the message server system 108 determines the media overlay applicable context by detecting that the client device 102 and / or the message client application 104 has accessed the media content for modification in the post-capture time period, which in turn initiates an operation for populating a record of the media overlay, which is then used to select a media overlay for presentation in a user interface (e.g., user interface 1600).

[0237] At operation 2906, the message server system 108 populates the group of media overlays with at least the corresponding media overlays based at least in part on the category indicators associated with the corresponding media overlays. Fig.16 and Fig.17As described, the message server system 108 determines the number of media overlays to be included in the group of media overlays by determining that the group of media overlays includes a set of media overlays that is less than a specific number corresponding to a maximum limit (or upper limit) of media overlays to be included. In addition, the message server system 108 determines the number of media overlays to be included in the group of media overlays, and refuses to include a particular media overlay based at least in part on determining that the group of media overlays includes a set of media overlays that satisfies the number of media overlays.

[0238] At operation 2908, the message server system 108 sends second metadata including information related to the group of media overlays to the client device. For example, the second metadata includes information indicating that each media overlay in the media overlay group corresponds to the same media overlay type. Alternatively, the information indicates that only a subset of media overlays from the media overlay group corresponds to the same media overlay type. In addition, in the example, the message server system 108 sends the second metadata including information related to the media overlay group in response to the client electronic device accessing a specific selectable graphical item in the carousel interface (e.g., user interface 1600).

[0239] Fig.30 is a block diagram illustrating an example software architecture 3006 that may be used in conjunction with the various hardware architectures described herein. Fig.30 is a non-limiting example of a software architecture, and it should be understood that many other architectures may be implemented to facilitate the functionality described herein. Software architecture 3006 may be implemented in a system such as Fig.31 3100, which includes a processor 3104, a memory 3114, and an (input / output) I / O component 3118, etc. A representative hardware layer 3052 is shown and the representative hardware layer 1152 may represent, for example Fig.31 The machine 3100 of the embodiment of the present invention is shown in Figure 3100. The representative hardware layer 3052 includes a processing unit 3054 having associated executable instructions 3004. The executable instructions 3004 represent executable instructions of the software architecture 3006, including implementations of the methods, components, etc. described herein. The hardware layer 3052 also includes a memory and / or storage module memory / storage device 3056 also having executable instructions 3004. The hardware layer 3052 may also include other hardware 3058.

[0240] exist Fig.30In the example architecture of , software architecture 3006 can be conceptualized as a stack of layers, each of which provides specific functionality. For example, software architecture 3006 may include layers such as operating system 3002, library 3020, framework / middleware 3018, application 3016, and presentation layer 3014. In operation, application 3016 and / or other components within a layer may call API call 3008 through a software stack and receive a response in response to API call 3008. The layers shown are representative in nature, and not all software architectures have all layers. For example, some mobile operating systems or dedicated operating systems may not provide framework / middleware 3018, while other operating systems may provide such a layer. Other software architectures may include additional layers or different layers.

[0241] The operating system 3002 can manage hardware resources and provide public services. The operating system 3002 may include, for example, a kernel 3022, services 3024, and drivers 3026. The kernel 3022 may serve as an abstraction layer between the hardware and other software layers. For example, the kernel 3022 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. Services 3024 may provide other public services for other software layers. Drivers 3026 are responsible for controlling or interfacing with the underlying hardware. For example, depending on the hardware configuration, drivers 3026 include display drivers, camera drivers, drives, flash drives, serial communications drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0242] The library 3020 provides a common infrastructure used by the application 3016 and / or other components and / or layers. The library 3020 provides the following functions, which allow other software components to perform tasks in an easier way than directly interfacing with the functions of the underlying operating system 3002 (e.g., the core 3022, the service 3024, and / or the driver 3026). The library 3020 may include a system library 3044 (e.g., a C standard library), which may provide functions such as memory allocation functions, string operation functions, mathematical functions, etc. In addition, the library 3020 may include an API library 3046, such as a media library (e.g., a library that supports the presentation and manipulation of various media formats such as MPREG4, H.264, MP3, AAC, AMR, JPG, PNG), a graphics library (e.g., an OpenGL framework that can be used to present 2D and 3D graphics content on a display), a database library (e.g., SQLite that can provide various relational database functions), a web library (e.g., WebKit that can provide web browsing functions), etc. The library 3020 may also include various other libraries 3048 to provide numerous other APIs to the application 3016 and other software components / modules.

[0243] Framework / middleware 3018 (also sometimes referred to as middleware) provides a higher level common infrastructure that can be used by applications 3016 and / or other software components / modules. For example, framework / middleware 3018 can provide various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. Framework / middleware 3018 can provide a wide range of other APIs that can be used by applications 3016 and / or other software components / modules, some of which can be specific to a particular operating system 3002 or platform.

[0244] Applications 3016 include built-in applications 3038 and / or third-party applications 3040. Examples of representative built-in applications 3038 may include, but are not limited to, contact applications, browser applications, book reader applications, location applications, media applications, messaging applications, and / or game applications. Third-party applications 3040 may include applications that are developed by entities other than the vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) for developing applications and can be used on platforms such as IOS TM ANDROID TM , Mobile software running on the mobile operating system of the phone or other mobile operating systems. The third-party application 3040 can call API calls 3008 provided by the mobile operating system (e.g., operating system 3002) to facilitate the functions described herein.

[0245] Applications 3016 may create user interfaces to interact with users of the system using built-in operating system functionality (e.g., kernel 3022, services 3024, and / or drivers 3026), libraries 3020, and framework / middleware 3018. Alternatively or additionally, in some systems, interaction with the user may occur through a presentation layer, such as presentation layer 3014. In these systems, the application / component "logic" may be separated from aspects of the application / component that interact with the user.

[0246] Fig.31 3100 is a block diagram illustrating components of a machine 3100 capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any one or more of the methods discussed herein, according to some exemplary embodiments. Specifically, Fig.31 A schematic representation of a machine 3100 in the example form of a computer system is shown in which instructions 3110 (e.g., software, programs, applications, applet, app or other executable code) for causing the machine 1200 to perform any one or more of the methods discussed herein can be executed. Similarly, instructions 3110 can be used to implement the modules or components described herein. Instructions 3110 convert a general unprogrammed machine 3100 into a specific machine 3100 that is programmed to perform the functions described and shown in the described manner. In alternative embodiments, the machine 3100 operates as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 3100 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 3100 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, web appliances, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 3110 specifying actions to be taken by the machine 3100. In addition, while only a single machine 3100 is shown, the term "machine" should also be taken to include a collection of machines that individually or jointly execute instructions 3110 to perform any one or more of the methodologies discussed herein.

[0247] The machine 3100 may include a processor 3104, a memory / storage device 3106, and an I / O component 3118, which may be configured to communicate with each other, for example, via a bus 3102. The memory / storage device 3106 may include a memory 3114, such as a main memory or other memory storage device, and a storage unit 3116, both of which the processor 3104 can access, for example, via the bus 3102. The storage unit 3116 and the memory 3114 store instructions 3110 that implement any one or more of the methods or functions described herein. The instructions 3110 may also reside, completely or partially, within the memory 3114, within the storage unit 3116, within at least one of the processors 3104 (e.g., within a cache memory of a processor), or any suitable combination thereof during execution thereof by the machine 3100. Thus, memory 3114, storage unit 3116, and memory of processor 3104 are examples of machine-readable media.

[0248] I / O components 3118 may include a variety of components that receive input, provide output, generate output, transmit information, exchange information, capture measurements, etc. The specific I / O components 3118 included in a particular machine 3100 will depend on the type of machine. For example, a portable machine such as a mobile phone may include a touch input device or other such input mechanism, while a headless server machine may not include such a touch input device. It will be appreciated that I / O components 3118 may include Fig.31 Many other components not shown in the figure. The I / O components 3118 are grouped according to function only to simplify the following discussion, and the grouping is by no means limiting. In various example embodiments, the I / O components 3118 may include output components 3126 and input components 3128. The output components 3126 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), auditory components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The input components 3128 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing instruments), tactile input components (e.g., physical buttons, touch screens or other tactile input components that provide the location and / or force of touch or touch gestures), audio input components (e.g., microphones), etc.

[0249] In other example embodiments, the I / O component 3118 may include various other components such as a biometric component 3130, a motion component 3134, an environment component 3136, or a position component 3138. For example, the biometric component 3130 may include a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 3134 may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. The environment component 3136 may include, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an auditory sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects concentrations of hazardous gases for safety or measures pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. The location component 3138 may include a positioning sensor component (e.g., a GPS receiver component), an altitude sensor component (e.g., an altimeter or barometer from which the air pressure is detected and the altitude can be obtained), an orientation sensor component (e.g., a magnetometer), etc.

[0250] A variety of technologies may be used to achieve communication. I / O components 3118 may include communication components 3140 that are operable to couple machine 3100 to network 3132 or device 3120 via coupling 3124 and coupling 3122, respectively. For example, communication components 3140 may include a network interface component or other suitable device to interface with network 3132. In other examples, communication components 3140 may include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Components (e.g. Low power consumption), Device 3120 may be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).

[0251] In addition, the communication component 3140 can detect the identifier or can include a component that is capable of operating to detect the identifier. For example, the communication component 3140 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional bar codes such as universal product codes (UPC) bar codes, multi-dimensional bar codes such as quick response (QR) codes, Aztec codes, data matrix, data symbols (Dataglyph), MaxiCode, PDF417, Ultra Code, UCC RSS-2D bar codes and other optical codes) or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be obtained via the communication component 3140, such as location via Internet Protocol (IP) geolocation, ... Signal triangulation to obtain location, location obtained via detection of NFC beacon signals that can indicate a specific location, etc.

[0252] The following discussion refers to various terms or phrases that are referred to throughout the subject disclosure.

[0253] "Signal medium" refers to any intangible medium that is capable of storing, encoding or carrying instructions to be executed by a machine, and includes digital or analog communication signals or other intangible media to facilitate the communication of software or data. The term "signal medium" should be deemed to include any form of modulated data signals, carrier waves, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode the information in the signal. The terms "transmission medium" and "signal medium" mean the same medium and may be used interchangeably in this disclosure.

[0254] "Communications network" means one or more parts of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a part of the Internet, a part of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, A network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network may include a wireless network or a cellular network, and the coupling may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rate for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other data transmission technologies defined by various standard setting organizations, other long distance protocols, or other data transmission technologies.

[0255] A "processor" refers to any circuit or virtual circuit (a physical circuit simulated by logic executed on an actual processor) that manipulates data values ​​according to control signals (e.g., "commands," "opcodes," "machine codes," etc.) and produces corresponding output signals that are applied to operate a machine. For example, a processor may be a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0256] “Machine storage medium” refers to a single or multiple storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions, routines, and / or data. Thus, the above terms should be considered to include, but are not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage media, computer storage media, and / or device storage media include: non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms “machine storage medium,” “device storage medium,” and “computer storage medium” mean the same thing and are used interchangeably in this disclosure. The terms “machine storage medium,” “computer storage medium,” and “device storage medium” expressly exclude carrier waves, modulated data signals, and other such media, at least some of which are covered by the term “signal media.”

[0257] "Component" refers to a device, physical entity or logic with the following boundaries: the boundary is defined by a function or subroutine call, a branch point, an API or other technology provided for dividing or modularizing a specific processing or control function. A component can be combined with other components via its interface to perform machine processing. A component can be a packaged functional hardware unit designed for use with other components and is usually part of a program that performs a specific function of a related function. A component can constitute a software component (e.g., a code implemented on a machine-readable medium) or a hardware component. A "hardware component" is a tangible unit that can perform certain operations and can be configured or arranged in a certain physical manner. In various example embodiments, one or more computer systems (e.g., an independent computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a processor group) can be configured as a hardware component for performing certain operations as described herein by software (e.g., an application or an application part). Hardware components can also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component may include a dedicated circuit system or logic that is permanently configured to perform certain operations. The hardware component can be a special processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component can also include a programmable logic or circuit temporarily configured to perform certain operations by software. For example, the hardware component can include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a specific machine (or a specific component of a machine) that is uniquely customized to perform the configured function and is no longer a general-purpose processor. It will be understood that the decision to implement the hardware component mechanically, in a dedicated and permanently configured circuit system, or in a temporarily configured circuit system (e.g., configured by software) can be driven by cost and time considerations. Therefore, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, that is, an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in some way or perform certain operations described herein. Considering the implementation of the hardware component being temporarily configured (e.g., programmed), it is not necessary to configure or instantiate each of the hardware components at any time. For example, where a hardware component includes a general purpose processor that is configured by software to be a special purpose processor, the general purpose processor may be configured to be a different special purpose processor (e.g., including different hardware components) at different times. The software configures one or more specific processors accordingly to, for example, constitute a specific hardware component at one time, and to constitute different hardware components at different times. Hardware components may provide information to other hardware components and receive information from other hardware components. Therefore, the described hardware components may be considered to be communicatively coupled.In the case of multiple hardware components being present at the same time, communication can be achieved by signal transmission between or among two or more hardware components (e.g., through appropriate circuits and buses). In an embodiment in which multiple hardware components are configured or instantiated at different times, communication between such hardware components can be achieved, for example, by storing information in a memory structure accessed by multiple hardware components and retrieving information in the memory structure. For example, a hardware component can perform an operation, and the output of the operation is stored in a memory device to which it is communicatively coupled. Then, other hardware components can access the memory device at a subsequent time to retrieve the stored output and process it. The hardware component can also initiate communication with an input device or an output device, and can operate on resources (e.g., the collection of information). The various operations of the example methods described in this article can be performed at least in part by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform related operations. Whether it is temporarily configured or permanently configured, such a processor can constitute a processor-implemented component that operates to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refer to hardware components implemented using one or more processors. Similarly, the method described herein can be implemented at least in part by a processor, wherein one or more specific processors are examples of hardware. For example, at least some operations of the method can be performed by one or more processors 1004 or the parts implemented by the processor. In addition, one or more processors can also operate to support the execution of the related operations in the "cloud computing" environment or operate as "software as a service" (SaaS). For example, at least some operations in the operation can be performed by a computer group (as an example of a machine including a processor), wherein these operations can be accessed via a network (for example, the Internet) and via one or more appropriate interfaces (for example, API). The execution of some operations can be distributed between processors, not only resident in a single machine, but can be deployed on several machines. In some example embodiments, a processor or a part implemented by a processor can be located in a single geographical location (for example, in a home environment, an office environment or a server group). In other example embodiments, a processor or a part implemented by a processor can be distributed across several geographical locations.

[0258] "Carrier signal" refers to any intangible medium that can store, encode or carry instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions may be sent or received over a network using a transmission medium via a network interface device.

[0259] "Computer-readable media" refers to both machine storage media and transmission media. Thus, the term includes both storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure.

[0260] "Client device" refers to any machine that interfaces with a communication network to obtain resources from one or more server systems or other client devices. A client device may be, but is not limited to, a mobile phone, a desktop computer, a laptop computer, a portable digital assistant (PDA), a smart phone, a tablet computer, an ultrabook, a netbook, multiple laptop computers, a multiprocessor system, a microprocessor-based or programmable consumer electronics product, a game console, a set-top box, or any other communication device that a user may use to access a network. In this disclosure, a client device is also referred to as an "electronic device."

[0261] "Ephemeral messages" are messages that are accessible for a limited duration of time. Ephemeral messages can be text, images, videos, etc. The access time of an ephemeral message can be set by the sender of the message. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is ephemeral.

Claims

1. A method for generating a message, comprising: retrieving, by a client device from a storage device, first image data captured by the client device at a previous time; receiving first metadata corresponding to the selected image processing operation; generating, by the client device, second image data based on the first metadata and the image processing operation performed on the first image data; generating, by the client device, second metadata including information corresponding to the image processing operation, a first identifier associated with the first image data, and a second identifier associated with the second image data, wherein the second metadata further includes gesture information indicating a swipe direction, a gesture sequence count indicating a number of swipes, and a tap count indicating a number of touch inputs received; as well as A message is generated including the second metadata, the second image data, and the first image data.

2. The method according to claim 1, further comprising: The message is sent to a server for sharing with one or more other client devices.

3. The method according to claim 1, further comprising: Sensor data indicative of an orientation of the client device corresponding to the first image data is received, wherein generating second image data is further based on the orientation of the client device.

4. The method according to claim 1, further comprising: The second image data is stored separately from the first image data.

5. The method according to claim 1, wherein: The selected image processing operation is selected via a display of the client device using a slide gesture.

6. The method according to claim 1, wherein: The selected image processing operation corresponds to a media overlay operation that belongs to a particular group of media overlay operations based on a category associated with the media overlay operation.

7. The method according to claim 1, further comprising: The second image data is rendered for display by the client device, the second image data comprising a first image corresponding to the first image data modified by the selected image processing operation.

8. The method according to claim 1, wherein: The first image data comprises a representation of an environment.

9. The method according to claim 8, wherein: Generating the second image data further includes: Performing a tracking process based on the first image data, wherein the tracking process includes: tracking surfaces identified in the representation of the environment, tracking a face identified in the representation of the environment, or Track objects using machine learning models.

10. The method according to claim 9, wherein: Only one of the tracking processes is performed at a time.

11. A system for generating a message, comprising: processor, a memory including instructions that, when executed by the processor, cause the processor to perform operations comprising: retrieving, by a client device from a storage device, first image data captured by the client device at a previous time; receiving first metadata corresponding to the selected image processing operation; generating, by the client device, second image data based on the first metadata and the image processing operation performed on the first image data; generating, by the client device, second metadata including information related to the image processing operation, the second metadata including a first identifier associated with the first image data and a second identifier associated with the second image data, wherein the second metadata further includes gesture information indicating a swipe direction, a gesture sequence count indicating a number of swipes, and a tap count indicating a number of touch inputs received; and A message is generated including the second metadata, the second image data, and the first image data.

12. The system according to claim 11, wherein: The memory includes further instructions, the further instructions further causing the processor to perform further operations, the further operations including: The message is sent to a server for sharing with one or more other client devices.

13. The system according to claim 11, wherein: The memory includes further instructions, the further instructions further causing the processor to perform further operations, the further operations including: Sensor data including an orientation of the client device corresponding to the first image data is received, the sensor data including metadata associated with the first image data, wherein generating the second image data is also based on the sensor data, the received sensor data indicating the orientation of the client device.

14. The system according to claim 11, wherein: The memory includes further instructions, the further instructions further causing the processor to perform further operations, the further operations including: The second image data is stored separately from the first image data.

15. The system according to claim 11, wherein: The selected image processing operation is selected via a display of the client device using a slide gesture.

16. The system according to claim 11, wherein: The selected image processing operation corresponds to a media overlay operation that belongs to a particular group of media overlay operations based on a category associated with the media overlay operation.

17. The system of claim 11, wherein: The memory includes further instructions, the further instructions further causing the processor to perform further operations, the further operations including: The second image data is rendered for display by the client device, the second image data comprising a first image corresponding to the first image data modified by the selected image processing operation.

18. The system of claim 11, wherein: The first image data comprises a representation of an environment.

19. The system of claim 18, wherein: Generating the second image data further causes the processor to perform additional operations, the additional operations comprising: Performing a tracking process based on the first image data, wherein the tracking process includes: tracking surfaces identified in the representation of the environment, tracking a face identified in the representation of the environment, or Track objects using machine learning models.

20. A non-transitory computer-readable medium comprising instructions that, when executed by a computing device, cause the computing device to perform operations comprising: retrieving, by a client device from a storage device, first image data captured by the client device at a previous time; receiving first metadata corresponding to the selected image processing operation; generating, by the client device, second image data based on the first metadata and the image processing operation performed on the first image data; generating, by the client device, second metadata including information related to the image processing operation, the second metadata including a first identifier associated with the first image data and a second identifier associated with the second image data, wherein the second metadata further includes gesture information indicating a swipe direction, a gesture sequence count indicating a number of swipes, and a tap count indicating a number of touch inputs received; as well as A message is generated including the second metadata, the second image data, and the first image data.

Images