annotation-based engagement analysis

Abstract

A messaging system performs engagement analysis based on annotations associated with content items generated by users of the messaging system. The messaging system is configured to process a content item including an image to identify elements in the image, and determine an annotation for the image based on a condition that indicates when to associate an annotation of the annotations with an image of the images based on the elements in the image. The messaging system is further configured to associate the annotation with the content item in response to determining to associate the annotation with the image, thereby associating the annotation with the content item. The messaging system is further configured to determine an engagement score for the annotation based on user interactions with the content item associated with the annotation, and adjust the engagement score to determine a trend for the annotation, thereby generating an adjusted engagement score.

Description

[0001] CLAIM OF PRIORITY

[0002] This patent application claims priority to U.S. Application Serial No. 17 / 248,400, filed January 22, 2021, which claims the benefit of priority to U.S. Provisional Application Serial No. 63 / 132,916, filed December 31, 2020, which are incorporated by reference herein in their entireties. TECHNICAL FIELD

[0003] Embodiments of the present disclosure generally relate to a messaging system for engagement analysis based on annotations of content items. More specifically, but not by way of limitation, embodiments of the present disclosure relate to determining an engagement score based on a trend component, a seasonality component, and a remainder component, and generating new content based on the determined engagement score. BACKGROUND

[0004] Current messaging systems provide users with an opportunity to produce and post content, such as images and videos. This content is provided in the messaging system for consumption by other users. Users can produce a very large amount of content. For example, there can be millions of images and videos available for users to consume. It can be difficult or time consuming to find content that is currently popular based on annotations associated with the content. SUMMARY

[0005] According to a first aspect, there is provided a method for annotation based engagement analysis, comprising: processing a content item comprising an image to identify elements in the image; determining an annotation for the image based on a condition that indicates when to associate an annotation of the annotation with an image in the image based on elements in the image; associating the annotation with the content item in response to determining to associate the annotation with the image; determining an engagement score for the annotation based on user interactions with the content item associated with the annotation; and adjusting the engagement score to determine a trend for the annotation to generate an adjusted engagement score.

[0006] According to a second aspect, there is provided a system for engagement analysis based on annotations, comprising: one or more computer processors; and one or more computer-readable media having stored therein instructions that, when executed by the one or more computer processors, cause the system to perform operations comprising: processing a content item comprising an image to identify elements in the image; determining an annotation for the image based on a condition that indicates when to associate an annotation of the annotations with an image of the images based on elements in the image; associating the annotation with the content item in response to determining to associate the annotation with the image; determining an engagement score for the annotation based on user interactions with the content item associated with the annotation; and adjusting the engagement score to determine a trend for the annotation to generate an adjusted engagement score.

[0007] According to a third aspect, there is provided a non-transitory computer-readable storage medium having stored therein instructions for execution by one or more processors of an apparatus of a computer, the instructions to configure the one or more processors to: process a content item comprising an image to identify elements in the image; determine an annotation for the image based on a condition that indicates when to associate an annotation of the annotations with an image of the images based on elements in the image; associate the annotation with the content item in response to determining to associate the annotation with the image; determine an engagement score for the annotation based on user interactions with the content item associated with the annotation; and adjust the engagement score to determine a trend for the annotation to generate an adjusted engagement score. BRIEF DESCRIPTION OF DRAWINGS

[0008] In the drawings, like reference numerals can describe similar but not necessarily identical elements throughout the several views. Some embodiments are illustrated by way of example in the drawings and are described in detail below. In the drawings:

[0009] Figure 1 is a diagrammatic representation of a networked environment in which the present disclosure can be deployed according to some examples.

[0010] Figure 2 is a diagrammatic representation of a messaging system having both client-side functionality and server-side functionality according to some examples.

[0011] Figure 3 is a diagrammatic representation of a data structure maintained in a database according to some examples.

[0012] Figure 4is a graphical representation of a message according to some examples.

[0013] Figure 5 is a flowchart of an access restriction process according to some examples.

[0014] Figure 6 A system for engagement analysis based on visual tags is shown according to some embodiments.

[0015] Figure 7 A content item is shown according to some embodiments.

[0016] Figure 8 A content item is shown displayed on a mobile device.

[0017] Figure 9 A system for determining annotations of a content item is shown according to some embodiments.

[0018] Figure 10 A profile of a user is shown according to some embodiments.

[0019] Figure 11 A user metadata collection module is shown according to some embodiments.

[0020] Figure 12 A content consumption database is shown according to some embodiments.

[0021] Figure 13 A content consumption metric extraction module is shown according to some embodiments.

[0022] Figure 14 A determine engagement score module is shown according to some embodiments.

[0023] Figure 15 A graph of usage of annotations is shown according to some embodiments.

[0024] Figure 16 A graph of usage of annotations is shown according to some embodiments.

[0025] Figure 17 An adjust engagement score module is shown according to some embodiments.

[0026] Figure 18 Raw data of four annotations is shown according to some embodiments.

[0027] Figure 19 A plot of simple moving average (SMA) of annotations is shown according to some embodiments.

[0028] Figure 20 A plot of trend momentum (TM) of annotations is shown according to some embodiments.

[0029] Figure 21 A plot of momentum (M) is shown, according to some embodiments.

[0030] Figure 22 A plot of statistical analysis of Label 1 is shown, according to some embodiments.

[0031] Figure 23 A plot of static seasonality of Label 1 is shown, according to some embodiments.

[0032] Figure 24 A plot of seasonality normalized detrended of Label 1 is shown, according to some embodiments.

[0033] Figure 25 A plot of trend component of popularity of Label 1 is shown, according to some embodiments.

[0034] Figure 26 A plot of seasonality component of popularity of Label 1 is shown, according to some embodiments.

[0035] Figure 27 A plot of seasonality component of popularity of Label 1 on Thursday is shown, according to some embodiments.

[0036] Figure 28 A plot of denoised component of popularity of Label 1 is shown, according to some embodiments.

[0037] Figure 29 A plot of denoised component of popularity of Label 1 on Monday is shown, according to some embodiments.

[0038] Figure 30 A user interface module is shown, according to some embodiments.

[0039] Figure 31 A method 3100 for engagement analysis based on labeling is shown, according to some embodiments.

[0040] Figure 32 is a diagrammatic representation of a machine in the form of a computer system within which a set of instructions can be executed to cause the machine to perform any one or more of the methodologies discussed herein, according to some examples.

[0041] Figure 33 is a block diagram illustrating a software architecture, whereby an example can be implemented.

[0042] Figure 34 is a diagrammatic representation of a processing environment, according to some examples. DETAILED DESCRIPTION

[0043] The description that follows includes systems, methods, techniques, instruction sequences, and computing machine program products embodying illustrative implementations of the present disclosure. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide an understanding of various embodiments of the inventive subject matter. It will be evident, however, to those skilled in the art, that embodiments of the inventive subject matter can be practiced without some or all of these specific details. In general, well-known instruction instances, protocols, structures, and techniques have not been shown in detail.

[0044] A messaging system is disclosed that analyzes user participation of a messaging system with visual tags or annotations associated with content items generated and consumed by users of the messaging system. The messaging system analyzes the users, the content items, the generation of the content items, and the consumption of the content items to determine anomalous activity associated with annotations or visual tags associated with the content items.

[0045] Example annotations or visual tags include "bird," "cat," "dog," "fishing," "bowling," "indoor," "outdoor," and the like. The messaging system analyzes the content items and identifies objects or elements within images of the content items. Example objects include "boy," "bowling," "bowling alley," "shorts," and the like. The messaging system determines whether to associate annotations with the content items based on the objects or elements identified in the content items. For example, a content item can be an image taken at a bowling alley. The messaging system extracts the objects "boy," "bowling," "indoor," and "bowling alley." The messaging system then determines which annotations to associate with the content item. In some implementations, annotations are associated with conditions that indicate whether the annotation should be associated with the content item. For example, an annotation for "bowling" can have a condition that indicates that the annotation for "bowling" will be associated with the content item if the objects for "person," "bowling," and "indoor" are identified in the image of the content item. There can be many conditions that trigger annotations to be associated with a content item. A content item can be associated with many annotations.

[0046] The messaging system then generates an engagement score for the annotations that indicates the degree of interaction of users of the messaging system with content items associated with the annotations. Engagement can be measured using content consumption metrics. Example content consumption metrics for a content item include: views, view time, number of shares, number of screenshots, and number of shares. One problem is that it can be difficult to determine user engagement with annotations with so many content consumption metrics. The solution to this problem is to use total content consumption metrics, such as passion and popularity. For passion and popularity, there is a weight vector associated with each of the content consumption metrics, so that a single total engagement score can be determined, which can make it easier to analyze user engagement with annotations.

[0047] The messaging system adjusts the total engagement score and the engagement score. This adjustment makes it easier to determine whether activity associated with a hashtag is abnormally high or low. In some implementations, the messaging system determines a simple moving average. In some implementations, the messaging system determines a trend momentum. In some implementations, the messaging system adjusts the engagement score according to a dynamic seasonal adjustment or a dynamic seasonal adjustment. In some implementations, the messaging system adjusts the engagement score to determine a dynamic seasonality for one interaction of a plurality of interactions of a user, where the dynamic seasonality is determined according to an average of subtracting a trend component from a value of the interaction.

[0048] The messaging system monitors the adjusted engagement score and can perform actions when the adjusted engagement score is abnormally high or low for a hashtag. These actions include reporting the abnormal activity and generating enhanced content related to the hashtag that can be added to content items by users of the messaging system.

[0049] The messaging system takes various generated databases about content items and users and generates an overall database that deletes personal information of users to ensure privacy of the users. In some implementations, the messaging system deletes generated databases that expose private data of users. Some implementations improve the identification of abnormal activity by deleting seasonal fluctuations in engagement scores.

[0050] Some implementations provide a technical solution to the technical problem of identifying abnormal activity associated with a hashtag within a messaging system. Some implementations provide a technical solution to the technical problem of using user profile data of users of a messaging system while maintaining privacy of individual users.

[0051] Some implementations have the advantage of improving content consumption of users of a messaging system by recommending content to users that has abnormal activity. Some implementations improve availability of content on a messaging system by recommending content to users of the messaging system that is related to hashtags that have abnormal activity. Some implementations improve targeting of advertisements and likely prices charged by locking advertisements related to hashtags that have very abnormal activity and by eliminating seasonal fluctuations in engagement scores. Some implementations improve the environment in which users produce messages by generating modified content (e.g., stickers, titles, and songs that can be added to content items related to hashtags that have very abnormal activity) using abnormal activity associated with a hashtag.

[0052] Networking computing environment

[0053] Figure 1is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages and associated content) over a network. The messaging system 100 includes multiple instances of client devices 102, each hosting multiple applications including a messaging client 104. Each messaging client 104 is communicatively coupled to the messaging server system 108 and other instances of the messaging client 104 via network 106 (e.g., the Internet).

[0054] The messaging client 104 is capable of communicating and exchanging data with another messaging client 104 and the messaging server system 108 via the network 106. The data exchanged between messaging clients 104 and between a messaging client 104 and the messaging server system 108 includes functionality (e.g., commands to activate functionality) as well as payload data (e.g., textual, audio, video or other multimedia data).

[0055] The messaging server system 108 provides server-side functionality via the network 106 to particular messaging clients 104. While certain functions of the messaging system 100 are described herein as being performed by messaging clients 104 or by the messaging server system 108, the location of certain functionality within either the messaging client 104 or the messaging server system 108 can be a design choice. For example, it can be technically preferable to initially deploy certain technology and functionality within the messaging server system 108 but to later migrate this technology and functionality to the messaging client 104 where the client device 102 has sufficient processing capacity.

[0056] The messaging server system 108 supports various services and operations that are provided to the messaging client 104. Such operations include sending data to, receiving data from, and processing data generated by the messaging client 104. As an example, this data can include message content, client device information, geolocation information, media

[0057] Turning now specifically to the messaging server system 108, an Application Program Interface (API) server 110 is coupled to, and provides a programmatic interface to, the application server 112. The application server 112 is communicatively coupled to a database server 118, which facilitates access to the database 120 that stores, among other data, information associated with messages processed by the application server 112. Similarly, a web server 124 is coupled to the application server 112, and provides web-based interfaces to the application server 112. To this end, the web server 124 processes incoming network requests with the Hypertext Transfer Protocol (HTTP) and several related protocols.

[0058] The Application Program Interface (API) server 110 receives and transmits message data (e.g., commands and message payloads) between the client device 102 and the application server 112. Specifically, the Application Program Interface (API) server 110 provides a set of interfaces (e.g., routines and protocols) that the messaging client 104 can call to invoke the functionality of the application server 112. The Application Program Interface (API) server 110 exposes various functions supported by the application server 112, including: account registration; login functionality; sending messages from a particular messaging client 104 to another messaging client 104 via the application server 112; sending media files (e.g., images or videos) from the messaging client 104 to the messaging server 114 and for possible access by another messaging client 104; setting media data collections (e.g., stories); retrieving a user's list of friends for the client device 102; retrieving such collections; retrieving messages and content; adding and deleting entities (e.g., friends) in an entity graph (e.g., social graph); locating friends in a social graph; and opening application events (e.g., related to the messaging client 104).

[0059] The application server 112 hosts a number of server applications and subsystems, including, for example, a messaging server 114, an image processing server 116, and a social network server 122. The messaging server 114 implements a number of message processing technologies and functions, particularly with respect to the aggregation and other processing of content (e.g., textual and multimedia content) included in messages received from multiple instances of the messaging client 104. As will be described in further detail, textual and media content from multiple sources can be aggregated into collections of content (e.g., referred to as stories or galleries). These collections are then made available to the messaging client 104. Other processor and memory intensive processing of data can also be performed server-side by the messaging server 114, in view of the hardware requirements for such processing.

[0060] The application servers 112 also include an image processing server 116 that is dedicated to performing various image processing operations, typically with respect to images or videos within the payload of messages sent from or received at the messaging server 114.

[0061] The social network server 122 supports various social networking functions and services, and makes these functions and services available to the messaging server 114. To this end, the social network server 122 maintains and accesses an entity graph 306 (shown in Figure 3 FIG. 3) within the database 120. Examples of functions and services supported by the social network server 122 include identifying other users of the messaging system 100 with which a particular user has a relationship or is "following," as well as identifying interests and other entities of a particular user.

[0062] System Architecture

[0063] Figure 2 is a block diagram illustrating further details regarding the messaging system 100, according to some examples. In particular, the messaging system 100 is shown to include the messaging client 104 and the application servers 112. The messaging system 100 includes several subsystems that are supported on the client-side by the messaging client 104 and on the server-side by the application servers 112. These subsystems include, for example, a ephemeral timer system 202, a collection management system 204, a modification system 206, a map system 208, a game system 210, and a participation system 214.

[0064] The ephemeral timer system 202 is responsible for enforcing temporary or time-limited access to content by the messaging client 104 and the messaging server 114. The ephemeral timer system 202 includes several timers that selectively enable access (e.g., for presentation and display) of messages and associated content via the messaging client 104 based on a duration and display parameters associated with a message or collection of messages (e.g., a story). Additional details regarding the operation of the ephemeral timer system 202 are provided below.

[0065] The collection management system 204 is responsible for managing groups or collections of media (e.g., collections of text, images, video, and audio data). Collections of content (e.g., messages, including images, videos, text, and audio) can be organized into "event libraries" or "event stories." Such collections can be available for a specified period of time (e.g., a duration of an event to which the content relates). For example, content related to a concert can be made available as a "story" for the duration of the concert. The collection management system 204 can also be responsible for publishing icons to the user interface of the messaging client 104 that provide notification of the existence of a particular collection.

[0066] Additionally, the collection management system 204 includes a curation interface 212 that enables collection managers to manage and curate particular content collections. For example, the curation interface 212 enables event organizers to curate content collections related to particular events (e.g., to delete inappropriate content or redundant messages). Additionally, the collection management system 204 employs machine vision (or image recognition techniques) and content rules to automatically curate content collections. In some examples, users can be paid compensation for including user-generated content into a collection. In such cases, the collection management system 204 operates to automatically pay such users for using their content.

[0067] The augmentation system 206 provides various functionality that enables users to augment (e.g., annotate or otherwise modify or edit) media content associated with messages. For example, the augmentation system 206 provides functionality related to generating and publishing media overlays for messages to be processed by the messaging system 100. The augmentation system 206 is operable to provide media overlays or augmentations (e.g., image filters) to the messaging client 104 based on a geographic location of the client device 102. In another example, the augmentation system 206 is operable to supply media overlays to the messaging client 104 based on other information such as social network information of a user of the client device 102. The media overlays can include audio and visual content as well as visual effects. Examples of audio and visual content include pictures, text, logos, animations, and sound effects. Examples of visual effects include color overlays. The audio and visual content or visual effects can be applied to a media content item (e.g., a photo) at the client device 102. For example, a media overlay can include text or an image that can be overlaid on top of a photo taken by the client device 102. In another example, a media overlay includes a location identification (e.g., Venice Beach) overlay, a name of a live event, or a merchant name (e.g., Beach Coffee Shop) overlay. In another example, the augmentation system 206 uses a geographic location of the client device 102 to identify a media overlay that includes a name of a merchant at the geographic location of the client device 102. The media overlay can include other indicia associated with the merchant. The media overlays can be stored in the database 120 and accessed through the database server 118.

[0068] In some examples, the augmentation system 206 provides a user-based publishing platform that enables users to select a geographic location on a map and upload content associated with the selected geographic location. The user can also specify circumstances under which a particular media overlay should be provided to other users. The augmentation system 206 generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geographic location.

[0069] In other examples, the augmentation system 206 provides a merchant-based publishing platform that enables merchants to select particular media overlays associated with geographic locations via a bidding process. For example, the augmentation system 206 associates the media overlays of the highest bidding merchants with respective geographic locations for a predefined amount of time.

[0070] The map system 208 provides various geolocation functionality and supports the presentation of map-based media content and messages by the messaging client 104. For example, the map system 208 enables the display of user icons or avatars (e.g., stored in the profile data 308) on a map to indicate the current or past locations of a user's "friends" and media content (e.g., collections of messages including photos and videos) generated by such friends in the context of the map. For example, on a map interface of the messaging client 104, a message posted by a user from a particular geographic location to the messaging system 100 can be displayed within the context of that particular location on the map to a "friend" of the particular user. The user can also share his or her location and status information with other users of the messaging system 100 via the messaging client 104 (e.g., using an appropriate status avatar), where the location and status information is similarly displayed to selected users in the context of the map interface of the messaging client 104.

[0071] The game system 210 provides various gaming functionality within the context of the messaging client 104. The messaging client 104 provides a game interface that provides a list of available games that can be launched by a user within the context of the messaging client 104 and played with other users of the messaging system 100. The messaging system 100 also enables a particular user to invite other users to participate in playing a particular game by issuing invitations from the messaging client 104 to such other users. The messaging client 104 also supports voice messaging and text messaging (e.g., chat) within the context of a game, provides a leaderboard for the game, and also supports the provision of in-game rewards (e.g., coins and items).

[0072] The engagement system 214 provides various functionality related to determining engagement scores and supports providing engagement score-based suggestions to the messaging client 104. The engagement system 214 provides a system that helps generate additional modifications that the augmentation system 206 can provide to the messaging client 104. According to some implementations, the modifications can be referred to as augmentations. The engagement system 214 can monitor and determine statistics related to content items generated within the messaging system 100. The engagement system 214 can monitor activity of the collection management system 204, the augmentation system 206, the map system 208, and the game system 210, as well as other activity of the messaging client 104 and the application server 112, to determine that a tag associated with content has an unusual level of activity, either lower than expected or higher than expected. The engagement system 214 can generate suggestions to the messaging client 104 and / or the application server 112, such as which content to display or suggest to a user based on a tag associated with the content. The engagement system 214 can generate additional content or suggestions to create content associated with certain tags. The engagement system 214 can generate reports that can be used for marketing and sales that can determine which content to include advertisements and help determine the value of the advertisements. The content is selected based on the tags associated with the content and the level of activity of users engaging with the content associated with the tags. The engagement system 214 can analyze content and determine external events, such as the pandemic of the coronavirus, and generate new content to indicate the external event. Figure 6 An overview of the engagement system 214 is provided.

[0073] Data Architecture

[0074] Figure 3 FIG. 3 is a schematic diagram illustrating a data structure 300 that can be stored in the database 120 of the messaging server system 108, according to certain examples. While the contents of the database 120 are shown to include a number of tables, it will be appreciated that the data can be stored in other types of data structures, e.g., as an object-oriented database.

[0075] The database 120 includes message data stored within a messages table 302. For any particular message, the message data includes at least message sender data, message recipient (or receiver) data, and a payload. The message sender data and the message recipient data are described below with reference to the user profiles 306 and the user-to-user connections 308. Figure 4 Additional details regarding information that can be included in a message and included in the message data stored in the messages table 302 are described below.

[0076] The entity table 304 stores entity data and is linked to (e.g., references) the entity graph 306 and profile data 308. Entities for which records are maintained within the entity table 304 can include individuals, corporate entities, organizations, objects, places, events, and the like. Regardless of entity type, any entity for which the messaging server system 108 stores data can be an identified entity. Each entity is provided a unique identifier as well as an entity type identifier (not shown).

[0077] The entity graph 306 stores information about relationships and associations between entities. Such relationships can be interest-based or activity-based social relationships, professional relationships (e.g., working at a common company or organization), just as examples.

[0078] The profile data 308 stores various types of profile data about a particular entity. Based on privacy settings specified by the particular entity, the profile data 308 can be selectively used and presented to other users of the messaging system 100. In the case of an entity being a person, the profile data 308 includes, for example, a user name, phone number, address, settings (e.g., notification and privacy settings), and a user-selected avatar representation (or collection of such avatar representations). The particular user can then selectively include one or more of these avatar representations in the content of messages communicated via the messaging system 100 as well as on map interfaces displayed by the messaging client 104 to other users. The collection of avatar representations can include a "status avatar" that presents a pictorial representation of a status or activity that the user can select to be in communication at a particular time.

[0079] In the case of an entity being a group, the profile data 308 for the group can similarly include one or more avatar representations associated with the group in addition to the group name, members, and various settings (e.g., notifications) for the related group.

[0080] The database 120 also stores augmentation data, such as overlays or filters, in an augmentation table 310. The augmentation data is associated with and applied to videos (for which data is stored in a video table 314) and images (for which data is stored in an image table 316).

[0081] In one example, the filter is a cover that is displayed as overlaid on an image or video during presentation to the recipient user. The filter can be of various types, including a filter that is user-selected from a set of filters presented by the messaging client 104 to the sending user when the sending user is composing a message. Other types of filters include a geolocation filter (also referred to as a geo filter) that can be presented to the sending user based on a geolocation. For example, a geolocation filter that is specific to a nearby or special location can be presented by the messaging client 104 within the user interface based on geolocation information determined by a global positioning system (GPS) unit of the client device 102.

[0082] Another type of filter is a data filter that can be selectively presented by the messaging client 104 to the sending user based on other inputs or information collected by the client device 102 during the message creation process. Examples of data filters include a current temperature at a particular location, a current speed at which the sending user is traveling, a battery life of the client device 102, or a current time.

[0083] Other augmentation data that can be stored within the image table 316 includes augmented reality content items (e.g., corresponding to an application lens or augmented reality experience). The augmented reality content items can be real-time special effects and sounds that can be added to an image or video.

[0084] As described above, augmentation data includes augmented reality content items, overlays, image transformations, AR images, and similar items that involve modifications that can be applied to image data (e.g., video or images). This includes real-time modifications, which modify images as they are captured using device sensors (e.g., one or more cameras) of the client device 102 and then display the images on the screen of the client device 102 with the modifications. This also includes modifications to stored content, such as modifications to video clips in a library that can be modified. For example, in a client device 102 that has access to multiple augmented reality content items, a user can use a single video clip with multiple augmented reality content items to see how different augmented reality content items would modify the stored clip. For example, by selecting different augmented reality content items for the content, multiple augmented reality content items that apply different pseudo-random motion models can be applied to the same content. Similarly, real-time video capture can be used with the shown modifications to display how video images currently being captured by the sensors of the client device 102 would modify the captured data. Such data can simply be displayed on the screen without being stored in memory, or the content captured by the device sensors can be recorded and stored in memory with or without modifications (or both). In certain systems, a preview function can display how different augmented reality content items would be displayed simultaneously in different windows of a display. For example, this can enable multiple windows with different pseudo-random animations to be viewed on a display at the same time.

[0085] Accordingly, data using augmented reality content items and various systems or other such transformation systems that modify content using the data can involve detection of objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.) in video frames, tracking of the objects as they move around the field of view, enter and exit the field of view, and modifying or transforming the objects as they are tracked. In various implementations, different methods for implementing such transformations can be used. Some examples can involve generating a 3-dimensional mesh model of one or more objects, and using transformations and animated textures of the model within the video to implement the transformations. In other examples, tracking of points on the objects can be used to place images or textures (which can be two-dimensional or three-dimensional) at the tracked locations. In still further examples, neural network analysis of video frames can be used to place images, models, or textures in content (e.g., images or video frames). Accordingly, augmented reality content items refer both to images, models, and textures used to create transformations in content, and to additional modeling and analysis information needed to implement such transformations through object detection, tracking, and placement.

[0086] Real-time video processing can be performed with any type of video data (e.g., video streams, video files, etc.) saved in memory of any type of computerized system. For example, a user can load a video file and save it in the memory of a device, or can generate a video stream using sensors of the device. Moreover, any object can be processed using computer animation models, such as faces of people and parts of human bodies, animals, or non-living things such as chairs, cars, or other items.

[0087] In some examples, when a particular modification is selected with the content to be transformed, the elements of the object to be transformed are identified by the computing device and then detected and tracked if they exist in the frames of the video stream. The elements of the object are modified according to the modification request, thereby transforming the frames of the video stream. For different types of transformations, the transformation of the frames of the video stream can be performed by different methods. For example, for a transformation of the frames that mainly refers to changing the form of the elements of the object, feature points of each element of the object are computed (e.g., using an Active Shape Model (ASM) or other known methods). Then, a mesh based on the feature points is generated for each of the at least one element of the object. The mesh is used in the following stages of tracking the elements of the object in the video stream. During the tracking process, the mentioned mesh for each element is aligned with the position of each element. Then, additional points are generated on the mesh. A first set of points is generated for each element based on the modification request, and a second set of points is generated for each element based on the set of first points and the request of the modification. Then, the frames of the video stream can be transformed by modifying the elements of the object based on the first set of points and the second set of points and the mesh. In this method, the background of the modified object can also be changed or distorted by tracking and modifying the background.

[0088] In some examples, the transformation of some regions of the object using the elements of the object can be performed by computing feature points of each element of the object and generating a mesh based on the computed feature points. Points are generated on the mesh, and then various regions are generated based on the points. Then, the elements of the object are tracked by aligning the regions of each element with the position of each of the at least one element, and the properties of the regions can be modified based on the request of the modification, thereby transforming the frames of the video stream. Depending on the specific request of the modification, the properties of the mentioned regions can be transformed in different ways. Such modifications can involve changing the color of the regions, removing at least some of the regions from the frames of the video stream, including one or more new objects in the regions based on the request of the modification, and modifying or distorting the regions or the elements of the object. In various implementations, any combination of such modifications or other similar modifications can be used. For certain models to be animated, some of the feature points can be selected as control points for determining the entire state space of options for the animation of the model.

[0089] In some examples of computer animation models that use face detection to transform image data, a particular face detection algorithm (e.g., Viola-Jones) is used to detect faces on an image. Then, an Active Shape Model (ASM) algorithm is applied to the face region of the image to detect facial feature reference points.

[0090] In other examples, other methods and algorithms suitable for face detection can be used. For example, in some implementations, landmarks are used to locate features, which are distinguishable points that exist in most images under consideration. For example, for face landmarks, the position of the left eye pupil can be used. If the initial landmark is not identifiable (e.g., if a person has an eye patch), a secondary landmark can be used. Such a landmark identification process can be used for any such object. In some examples, a set of landmarks forms a shape. A shape can be represented as a vector using the coordinates of the points in the shape. One shape is aligned to another shape by a similarity transformation (allowing translation, scaling, and rotation) that minimizes the average Euclidean distance between the shape points. The average shape is the average of the aligned training shapes.

[0091] In some examples, a search for landmarks starts from an average shape that is aligned to the position and size of the face determined by a global face detector. Such a search then repeats the following steps: a tentative shape is suggested by adjusting the position of the shape points through template matching of the image texture around each point, and then the tentative shape is brought into conformity with the global shape model until convergence occurs. In some systems, individual template matching is unreliable, and the shape model pools the results of weak template matching to form a stronger overall classifier. The entire search is repeated at each level of an image pyramid from coarse resolution to fine resolution.

[0092] The transformation system can capture images or video streams on a client device (e.g., client device 102) and perform complex image manipulations locally on the client device 102 while maintaining a proper user experience, computation time, and power consumption. Complex image manipulations can include size and shape changes, emotion transformations (e.g., changing a face from frowning to smiling), state transformations (e.g., aging an object, reducing apparent age, changing gender), style transformations, application of graphical elements, and any other suitable image or video manipulations implemented by convolutional neural networks that have been configured to efficiently execute on the client device 102.

[0093] In some examples, a computer animation model for transforming image data can be used by a system in which a user can use a client device 102 having a neural network to capture an image or video stream of the user (e.g., a selfie) that operates as part of a messaging client 104 operating on the client device 102. A transform system operating within the messaging client 104 determines the presence of a face in the image or video stream and provides modification icons associated with computer animation models to transform the image data, or the computer animation models can be presented in association with the interfaces described herein. The modification icons include changes that can be the basis for modifying the user’s face in the image or video stream as part of a modification operation. Once a modification icon is selected, the transform system initiates a process to convert the image of the user to reflect the selected modification icon (e.g., generate a smiling face on the user). Once the image or video stream is captured and the specified modification is selected, the modified image or video stream can be presented in a graphical user interface displayed on the client device 102. The transform system can implement a complex convolutional neural network on a portion of the image or video stream to generate and apply the selected modification. That is, once a modification icon is selected, the user can capture an image or video stream and present the modification results in real-time or near real-time. Further, the modification can be persistent while a video stream is being captured and the selected modification icon remains toggled. Machine-taught neural networks can be used to implement such modifications.

[0094] A graphical user interface presenting modifications performed by the transform system can provide additional interaction options for the user. Such options can be based on the interface used to initiate the selection of a particular computer animation model and content capture (e.g., initiated from a content creator user interface). In various implementations, the modification can be persistent after the initial selection of a modification icon. The user can turn the modification on or off by tapping or otherwise selecting the face modified by the transform system and store it for later viewing or browsing to other areas of the imaging application. In cases where multiple faces are modified by the transform system, the user can turn the modification on or off globally by tapping or selecting a single face modified and displayed within the graphical user interface. In some implementations, individual faces in a group of multiple faces can be modified individually, or such modifications can be toggled individually by tapping or selecting individual faces or a series of individual faces displayed within the graphical user interface.

[0095] The story table 312 stores data regarding collections of messages and associated image, video, or audio data that are compiled into a collection (e.g., a story or gallery). The creation of a particular collection can be initiated by a particular user (e.g., each user for which a record is maintained in the entity table 304). A user can create a "personal story" in the form of a collection of content that has been created and sent / broadcast by that user. To this end, the user interface of the messaging client 104 can include a user-selectable icon to enable a sender user to add particular content to his or her personal story.

[0096] A collection can also constitute a "live story" that is a collection of content from multiple users that is created manually, automatically, or using a combination of manual and automatic techniques. For example, a "live story" can constitute a curated stream of content submitted by users from different locations and events. Users whose client devices are enabled for location services and are at a common location event at a particular time can be presented with an option to contribute content to a particular live story, e.g., via the user interface of the messaging client 104. A user can be identified with a live story by the messaging client 104 based on his or her location. The end result is a "live story" told from a community perspective.

[0097] Another type of collection of content is referred to as a "location story," which enables users whose client devices 102 are located within a particular geographic location (e.g., at a college or university campus) to contribute to a particular collection. In some examples, a secondary authentication can be required for contribution to a location story to verify that the end-user belongs to a particular organization or other entity (e.g., is a student in a university campus).

[0098] As mentioned above, the video table 314 stores video data that, in one example, is associated with messages for which records are maintained within the message table 302. Similarly, the image table 316 stores image data that is associated with messages for which message data is stored in the entity table 304. The entity table 304 can associate various augmentations from the augmentation table 310 with various images and videos stored in the image table 316 and the video table 314.

[0099] Referring to Figure 6 The database 120 can also store a tagging database 614, a content consumption database 618, and a user database 620 in the participation table 318.

[0100] Data Communication Architecture

[0101] Figure 4is a schematic diagram showing the structure of a message 400 generated by a messaging client 104 for transmission to another messaging client 104 or messaging server 114, according to some examples. The content of a particular message 400 is used to populate a message table 302 stored in a database 120, which can be accessed by a messaging server 114. Similarly, the content of a message 400 is stored in memory as "in-transit" or "in-flight" data for a client device 102 or application server 112. The message 400 is shown to include the following example components.

[0102] Message identifier 402: a unique identifier that identifies the message 400. Message text payload 404: text to be generated by a user via a user interface of a client device 102 and included in the message 400.

[0103] Message image payload 406: image data captured by a camera component of the client device 102 or retrieved from a memory component of the client device 102 and included in the message 400. Image data for a sent or received message 400 can be stored in an image table 316.

[0104] Message video payload 408: video data captured by a camera component or retrieved from a memory component of the client device 102 and included in the message 400. Video data for a sent or received message 400 can be stored in a video table 314.

[0105] Message audio payload 410: audio data captured by a microphone or retrieved from a memory component of the client device 102 and included in the message 400.

[0106] Message augmentation data 412: augmentation data (e.g., filters, stickers, or other annotations or augmentations) representing augmentations to be applied to the message image payload 406, message video payload 408, or message audio payload 410 of the message 400. Augmentation data for a sent or received message 400 can be stored in an augmentation table 310.

[0107] Message duration parameter 414: a parameter value indicating an amount of time in seconds that the content of the message (e.g., message image payload 406, message video payload 408, message audio payload 410) will be presented to or made accessible to a user via the messaging client 104.

[0108] Message geolocation parameter 416: Geolocation data (e.g., latitude and longitude coordinates) associated with the content payload of the message. Multiple message geolocation parameter 416 values can be included in the payload, each of which is associated with a content item included in the content (e.g., a particular image within the message image payload 406 or a particular video in the message video payload 408).

[0109] Message story identifier 418: An identifier value that identifies one or more content collections (e.g.,“stories” identified in the story table 312) with which a particular content item in the message image payload 406 of the message 400 is associated. For example, multiple images within the message image payload 406 can each be associated with multiple content collections using identifier values.

[0110] Message tags 420: Each message 400 can be tagged with multiple tags, each of which indicates a subject matter of content included in the message payload. For example, where a particular image included in the message image payload 406 depicts an animal (e.g., a lion), a tag value indicating the relevant animal can be included in the message tags 420. Tag values can be generated manually based on user input, or can be generated automatically using, for example, image recognition.

[0111] Message sender identifier 422: An identifier (e.g., a messaging system identifier, email address, or device identifier) that indicates a user of the client device 102 on which the message 400 was generated and from which the message 400 was sent.

[0112] Message recipient identifier 424: An identifier (e.g., a messaging system identifier, email address, or device identifier) that indicates a user of the client device 102 to which the message 400 is addressed.

[0113] The content (e.g., values) of the various components of the message 400 can be pointers to locations in tables within which the content data values are stored. For example, the image values in the message image payload 406 can be pointers to locations (or addresses of locations) within the image table 316. Similarly, the values within the message video payload 408 can point to data stored within the video table 314, the values stored within the message augmentations 412 can point to data stored in the augmentation table 310, the values stored within the message story identifier 418 can point to data stored in the story table 312, and the values stored within the message sender identifier 422 and the message recipient identifier 424 can point to user records stored within the entity table 304.

[0114] Although the flow diagrams can illustrate a process as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations can be re-arranged. A process is terminated when its operations are completed. A process can correspond to a method, a procedure, an algorithm, or the like. The operations of the method can be performed all or in part by one or more aspects of the system such as the systems described herein or any portion thereof such as a processor included in any system.

[0115] Time-based access restriction architecture

[0116] Figure 5 FIG. 5 is a schematic diagram illustrating an access restriction process 500 according to which access to content (e.g., a transient message 502 and an associated multimedia payload of data) or a collection of content (e.g., a group of transient messages 504) can be time-limited (e.g., such that it is transient).

[0117] The transient message 502 is shown as being associated with a message duration parameter 506, the value of which determines the amount of time that the messaging client 104 will display the transient message 502 to a receiving user of the transient message 502. In one example, depending on the amount of time specified by the sending user using the message duration parameter 506, the receiving user can view the transient message 502 for up to 10 seconds.

[0118] The message duration parameter 506 and the message recipient identifier 424 are shown as inputs to a message timer 512, which is responsible for determining the amount of time to show the transient message 502 to a particular receiving user identified by the message recipient identifier 424. In particular, the transient message 502 is shown to the relevant receiving user only for the time period determined by the value of the message duration parameter 506. The message timer 512 is shown as providing output to a more generalized transient timer system 202, which is responsible for the overall timing of the display of content (e.g., the transient message 502) to a receiving user.

[0119] Figure 5The ephemeral messages 502 shown in the middle are included within an ephemeral message group 504 (e.g., a collection of messages in a personal story or an event story). The ephemeral message group 504 has an associated group duration parameter 508, the value of which determines the duration for which the ephemeral message group 504 is presented and can be accessed by users of the messaging system 100. For example, the group duration parameter 508 can be the duration of a concert, where the ephemeral message group 504 is a collection of content related to that concert. Alternatively, the value of the group duration parameter 508 can be specified by a user (either the owning user or a curator user) when the setup and creation of the ephemeral message group 504 is performed.

[0120] Additionally, each ephemeral message 502 within the ephemeral message group 504 has an associated group engagement parameter 510, the value of which determines the duration for which the ephemeral message 502 will be accessible within the context of the ephemeral message group 504. Thus, a particular ephemeral message group 504 can "expire" and become inaccessible in the context of the ephemeral message group 504 before the ephemeral message group 504 itself expires according to the group duration parameter 508. The group duration parameter 508, the group engagement parameter 510, and the message recipient identifier 424 each provide input to a group timer 514, which is operable to first determine whether a particular ephemeral message 502 of the ephemeral message group 504 will be displayed to a particular receiving user, and if so, for how long. Note that the ephemeral message group 504 is also aware of the identity of the particular receiving user due to the message recipient identifier 424.

[0121] Thus, the group timer 514 is operable to control the overall lifespan of the associated ephemeral message group 504, as well as the individual ephemeral messages 502 included in the ephemeral message group 504. In one example, each ephemeral message 502 within the ephemeral message group 504 remains viewable and accessible for the time period specified by the group duration parameter 508. In another example, a certain ephemeral message 502 can expire within the context of the ephemeral message group 504 based on the group engagement parameter 510. Note that the message duration parameter 506 can still determine the duration for which a particular ephemeral message 502 is displayed to a receiving user, even within the context of the ephemeral message group 504. Thus, the message duration parameter 506 determines the duration for which a particular ephemeral message 502 is displayed to a receiving user, regardless of whether the receiving user is viewing that ephemeral message 502 within the context of the ephemeral message group 504 or not.

[0122] The short-lived timer system 202 can also operatively remove a specific short-lived message 502 from the short-lived message group 504 based on determining that the specific short-lived message 502 has exceeded its associated group participation parameter 510. For example, when the sending user has established a group participation parameter 510 for 24 hours from the date of publication, the short-lived timer system 202 will remove the relevant short-lived message 502 from the short-lived message group 504 after the specified twenty-four hours. The short-lived timer system 202 also operates to remove the short-lived message group 504 when the group participation parameter 510 for each short-lived message 502 within the short-lived message group 504 has expired, or when the short-lived message group 504 itself has expired according to the group duration parameter 508.

[0123] In certain use cases, the creator of a specific ephemeral message group 504 can specify an indefinite group duration parameter 508. In this case, the expiration of the group participation parameter 510 for the last remaining ephemeral message 502 within the ephemeral message group 504 will determine when the ephemeral message group 504 itself expires. In this case, adding a new ephemeral message 502 with a new group participation parameter 510 to the ephemeral message group 504 effectively extends the lifetime of the ephemeral message group 504 to a value equal to the group participation parameter 510.

[0124] In response to the short-lived timer system 202 determining that a short-lived message group 504 has expired (e.g., is no longer accessible), the short-lived timer system 202 communicates with the messaging system 100 (and, in particular, the messaging client 104) to cause the markers (e.g., icons) associated with the relevant short-lived message group 504 to no longer be displayed in the user interface of the messaging client 104. Similarly, when the short-lived timer system 202 determines that the message duration parameter 506 for a particular short-lived message 502 has expired, the short-lived timer system 202 causes the messaging client 104 to no longer display the markers (e.g., icons or text identifiers) associated with the short-lived message 502.

[0125] Participation analysis based on visual tags

[0126] Figure 6 A system 600 for participation analysis based on visual tags, according to some embodiments, is shown. User 604 is... Figure 1 The messaging system 100 generates and consumes content items 602 (606 and 608). For example, user 604 may be associated with client device 102, which includes a camera, and user 604 uses client device 102 to generate content item 602 (606). Other users 604 of the messaging system 100 may consume content item 602 (608) and generate their own content items 602 (606).

[0127] Combination Figure 7Discussion item 602. In conjunction Figure 10 Discussion user 604. Annotation extraction module 610 determines annotations to be associated with content item 602 and establishes annotation database 614. In conjunction Figure 9 Discussion annotations and annotation extraction module 610. User metadata collection module 613 establishes user database 620. In conjunction Figure 11 Discussion user metadata collection module 613.

[0128] Content consumption metrics extraction module 612 extracts content consumption metrics from users 604 that produce 606 and consume 608 content item 602 and establishes content consumption database 618. In conjunction Figure 13 Discussion content consumption metrics extraction module 612.

[0129] Participation score determination module 622 determines participation scores for annotations, and adjusted participation score determination module 624 determines adjusted participation scores for annotations. Activity detection module 626 monitors adjusted participation scores and detects adjusted participation scores for annotations with low, high, or abnormal activity. In conjunction Figure 14 Participation score determination module 622 is disclosed. In conjunction Figure 17 Adjusted participation score determination module 624 and activity detection module 626 are disclosed. User interface module 628 presents a user interface that presents results of activity detection module 626 and provides options for users to query system 600. In conjunction Figure 30 Discussion user interface module 628. In some implementations, system 600 operates based on real-time monitoring of production and consumption of content item 602, updating adjusted activity scores for annotations, and notifying users of low, high, or abnormal activity via user interface module 628.

[0130] Figure 7 Content item 602 is shown in accordance with some implementations. Content item 602 can include content-related items 702, topics 704, media types 706, images 708, videos 710, themes 712, titles 714, terms 716, stickers 718, annotations 720, modifications 722, and producers 724. Topics 704 can be themes determined to be related to content item 602. Topics 704, themes 712, and terms 716 can be selected by user 604 that produces 606 content item 602, or can be determined by system 600. Media types 706 indicate media types of content item 602. Media types 706 include images 708, videos 710, audio, etc. Titles 714 are one or more titles added to content item 602 by producer 724. Stickers 718 are modification items that can be added to content item 602. Annotations 720 are names or visual labels determined by system 900. Participation scores are determined by participation score determination module 622 for annotations 720, in conjunctionFigure 14 Discussion. Element 726 is an object, action, scene, person, place, event, etc. Element 726 is identified by content item analysis module 902 based on element descriptors 918 within image 708, video 710, audio, etc.

[0131] In some implementations, user 604 that generated 606 content item 602 can tag 720 an object in content item 602. Modification 722 indicates one or more modifications that have been added to content item 602 by augmentation system 206. Modification 722 can include title 714, sticker 718, etc. Producer 724 is the user 604 that generated 606 content item 602. In some implementations, producer 724 can be messaging system 100. For example, in some implementations, an external news story is generated by messaging system 100 and provided as content item 602. Content related items 702 can include additional items, such as an indication that user 604 has consumed content item 602 or a link to a copy of content item 602. In some implementations, there can be more than one field, such as topic 704, terms 716, tags 720, etc.

[0132] Figure 8 Content item 602 is shown displayed on mobile device 802. Mobile device 802 includes camera 814 for capturing image 708. Mobile device 802 is client device 102 as discussed in Figure 1 . Mobile device 802 is associated with user 604. Content item 602 is image 708 with modifications 722, such as sticker 718 and title 714, and song 810. Elements 726A, 726B, 726C, and 726D are portions of image 708 and are identified by content item analysis module 902. Element 726A is a first person or first boy. Element 726B is a second person or second boy. Element 726C is a bowling ball. Element 726D is a bowling lane. Other modifications can be added by user 604.

[0133] In some implementations, modifications 722 can be modifications added by messaging server system 108 and stored in such a way that modifications 722 are recognized by content item analysis module 902 as being associated with image 708. For example, content item 602 can have been generated by augmentation system 206 as part of message 400 shown in Figure 9 . In some implementations, augmentation system 206 enables user 604 to add modifications 722 to image 708, such as audio content such as song 810 or visual content such as sticker 718 or title 714, which can be stored in augmentation table 310. Figure 4

[0134] Figure 9 ​A system 900 for determining annotations 914 of a content item 602 is shown in accordance with some embodiments. The content item 602 is processed or analyzed by a content item analysis module 902 to determine elements 726 within the image 708, video 710, audio, etc. In some embodiments, the content item analysis module 902 is one or more neural networks trained to recognize elements 726. For example, one or more convolutional neural networks can be used to segment an image of the image 708 or video 710 and recognize different elements 726. In some embodiments, the content item analysis module 902 accesses a database of element descriptors 918, where the element descriptors 918 include a visual representation 920 and a name 922. The name 922 can be an annotation or name given to the element 726 in one or more human languages. The visual representation 920 can be a neural network that recognizes the element 726, or another representation such as a vector representation or computer model of the element 726. In some embodiments, the content item analysis module 902 determines the elements 726 based on the modifications 722. For example, the content item analysis module 902 can recognize the sticker 718 as a modification 722 from the augmentation system and determine that the sticker 718 should be classified as an element 726. The determination of whether to classify a modification 722 as an element 726 can depend on the element descriptors 918. For example, the element descriptors 918 can include an indication of the modification 722 as a visual representation 920 and a corresponding name 922 of the modification 722. In the image 708 of the exemplary content item 602 Figure 7 The title 714, song 810, and sticker 718 can all be determined by the content item analysis module 902 as elements 726 from the element descriptors 918 in the image 708 of the exemplary content item 602. The modification 722 can not be accessible to the content item analysis module 902 via the augmentation system, but can have been incorporated into the image 708. For example, the sticker 718 can be incorporated into the pixels of the image 708 rather than overlaid. The content item analysis module 902 can recognize the sticker 718 as a heart based on the element descriptors 918, where the visual representation 920 recognizes a heart in the image.

[0135] The content item analysis module 902 generates a list of elements 726 that are part of the image 708 or the video 710. The annotation extraction module 610 determines annotations 914 based on the elements 726, the content item 602, and the taxonomy of annotations 906. In some implementations, the annotations 914 can be referred to as visual labels when determined for the image 708. The taxonomy of annotations 906 includes graphemes of the annotations 914, the conditions 910, and the hierarchical position 912. The hierarchy 913 is a hierarchy of the annotations 914. The hierarchical position 912 indicates a position in the hierarchy 913. For example, the element 726B is identified as the annotation 914 of a boy, and the hierarchy 913 indicates that the annotation 914 of a boy has a parent of the annotation 914 of a person. The hierarchy 913 can indicate a level of abstraction associated with the annotation 914. For example, the annotation 914 of a boy can have a level of abstraction of 3, and it has a parent of the annotation 914 of a person that can have a level of abstraction of 2, and a parent of the person can have a level of abstraction of 1 of the annotation 914 of a living thing. In another example, the image 708 can be further analyzed by the content item analysis module 902 to include the element 726 of shorts as the element 726B of a boy. The annotation extraction module 610 determines that the element 726 of shorts is the annotation 914 of shorts with the name of shorts. The annotation 914 of shorts has a level of abstraction of 3, and the annotation 914 of a garment has a level of abstraction of 2. Other examples of elements 726 of the example image 708 are bowling ball, standing, interior, ceiling, friend, light, and so forth. In some implementations, the annotation extraction module 610 includes a parent or an ancestor of the annotation 914 within the hierarchy 913 as an annotation 914 of the content item 602.

[0136] The name 922 assigned to the element 726 can be used by the tag extraction module 610 to determine the tag 914. The conditions 910 indicate what conditions must be met for the content item 602 to be considered to have the tag 914. For example, the condition 910 for the tag 914 of shorts can be that the name 922 of the element 726 is shorts. The conditions 910 can include multiple conditions and can be based on any field of the content item 602, the element 726, and the tag 914 associated with the content item 602. In another example, the condition 910 for the tag 914 of funny can include a content item 602 with at least two people smiling. In another example, the condition 910 for the tag 914 of bowling can include a person bowling and the location being a bowling alley. The location can be determined via the content related item 702, which can include the geographic location where the content item 602 was captured. The conditions 910 can be satisfied by a tag 914 that is related in the hierarchy 913. For example, the condition 910 for the tag 914 of bowling can be a person at a location determined to be a bowling alley. The element 726 can be identified as a child of the tag 914, and then the tag 914 person is added as a parent of the tag 914 in the hierarchy 913. The condition 910 for the tag 914 of bowling can then be satisfied.

[0137] The tag extraction module 610 associates the content item 602 with the tags 914 that it is determined to have. The visual tag extraction module 610 builds a tag database 614 that is used by the engagement score module 622 to determine engagement scores for the tags 914.

[0138] Figure 10 A profile 1002 of the user 604 is shown in accordance with some embodiments. The profile 1002 includes age 1004, gender 1006, age group 1008, city 1010, language 1012, country 1014, device attributes 1016, content production 1018, produced content 1020, content consumption metrics 1022, consumed content 1024, viewing times 1026, and searches 1028.

[0139] Age 1004 is an indication of the age or age range of the user 604. Gender 1006 is an indication of the gender of the user 604. Age group 1008 is an indication of the age group of the user 604, e.g., 18-24, etc. City 1010 is an indication of the city in which the user 604 resides, works, or is associated with. Language 1012 is an indication of the human natural language used by the user 604 in the messaging system 100. Country 1014 is an indication of the country associated with the user 604, e.g., the country 1014 in which the user 604 resides. Device attributes 1016 are attributes of one or more devices used by the user 604, including the devices used by the user 604 to access the messaging system 100. Content production 1018 is data about content items 602 produced 606 by the user 604. This data can include an indication of the produced content 1020, which can include the topic 704 of the produced content 1020, the terms 716 of the produced content, which other users 604 the produced content 1020 was sent to, whether the produced content 1020 was made public, whether the produced content 1020 was part of a ephemeral message, etc., modifications used by the user 604, etc.

[0140] Content consumption metrics 1022 include data about content consumed 608 by the user 604. The content consumption metrics 1022 can include an indication of the consumed content 1024 and the time 1026 the content was viewed. This data can include interactions of the user 604 with the consumed content 1024, e.g., whether the user 604 reacted to the content item 602, viewed the content item 602 multiple times, liked the content item 602, and other content consumption metrics. Searches 1028 indicate searches 1028 performed by the user 604. In some implementations, there can be more than one field, e.g., city 1010, language 1012, etc.

[0141] Figure 11A user metadata collection module 613 is shown according to some embodiments. The user metadata collection module 613 uses information from the user 604 to generate a user database 620, where the user database 620 captures information about the user 604. In some embodiments, the user database 620 protects the privacy of the user 604. Some data associated with the user 604 is filtered out or not used in schema 1100. For example, to protect the privacy of the user 604, the username, address, account identifier, and the like are not used in schema 1100. The age of the user 604 is grouped into age groups 1106. In some embodiments, only when the number of times the user 604 interacts with the content item 602 reaches a threshold, such as 1, 5, or other number, is the user 604 added to the user database 620. The user metadata collection module 613 aggregates the frequency of interaction 1114 of the user 604 with the content 1104 based on content consumption metrics and date 1102, where the data 1102 can be a range, and the aggregation is based on one or more of: age group 1106, gender 1108, country 1110, state 1112, and city 1113.

[0142] Figure 12 A content consumption database 618 is shown according to some embodiments. The content consumption database 618 includes consumed content 1204, annotations 1206, media type 1208, images 1210, videos 1212, users 604, average viewing time 1216, screenshots 1218, number of views 1220, search log 1222, viewing date 1224, viewing time 1226, number of shares 1228, terms 1230. Other metrics can be used by the content consumption metrics extraction module 612 on the content consumption database 618. The content consumption database 618 can be organized in schema 1412. Figure 14 The consumed content 1204 is an indication of the content item 602. The annotations 1206 are one or more annotations 914 that the annotation extraction module 610 determines the content item 602 should have. The media type 1208 is an indication of the type of media of the content item 602. The images 1210 are images 914 that are determined by the image extraction module 610 to be in the content item 602. The videos 1212 are videos 914 that are determined by the video extraction module 610 to be in the content item 602. The users 604 are users 604 that interact with the content item 602. The average viewing time 1216 is an average of the time the users 604 spend viewing the content item 602. The screenshots 1218 are screenshots 914 that are determined by the screenshot extraction module 610 to be in the content item 602. The number of views 1220 is the number of times the content item 602 is viewed by the users 604. The search log 1222 is an indication of the search terms 1230 that are used to find the content item 602. The viewing date 1224 is an indication of the date the content item 602 is viewed by the users 604. The viewing time 1226 is an indication of the time the content item 602 is viewed by the users 604. The number of shares 1228 is the number of times the content item 602 is shared by the users 604. The terms 1230 are search terms 914 that are determined by the search log extraction module 616 to be used to find the content item 602. Figure 9 Figure 7 ​The terms 716 are the same or similar. Media type 1208 indicates the media type of the consumed content 1204, such as image 1210, video 1212, audio, search log, etc. User 604 indicates one or more users 604 that consumed the consumed content 1204. Average viewing time 1214 indicates the average time that the users 604 spent viewing the consumed content 1204. Screenshot 1218 indicates the number or average number of screenshots 1218 of the consumed content 1204. View count 1220 indicates the number of views of the consumed content 1204. Search log 1222 indicates the number of times the consumed content 1204 appeared in a search and / or the number of times the consumed content 1204 was selected from a search. Viewing date 1224 indicates the time window for the statistics about the consumed content 1204. Viewing time 1226 indicates one or more statistics about the one or more users 1214 viewing the consumed content 1204. Share count 1228 indicates statistics about the number of shares of the consumed content 1204 by the one or more users 1214. According to some embodiments, the content consumption database 618 includes table 1 metrics 1231. The table 1 metrics 1231 include the metrics disclosed in Table 1 below. It can be appreciated that the content consumption database 618 can include additional metrics about the consumed content 1204 by the one or more users 1214.

[0143] Figure 13 The content consumption metrics extraction module 612 is shown according to some embodiments. The content consumption metrics extraction module 612 builds the content consumption database 618 based on the generation 606, the consumption 608, the users 604, and the content items 602. For example, a user 604 can access a content item 602 and view the content item 602 on their mobile device. The content consumption metrics extraction module 612 updates the database entry for the consumed content 1204 that matches the content item 602 based on the user 1214's interaction with the content item 602 in the consumed content 1204, where the user 604 is added as a user 1214, the average viewing time 1216 is updated, the view count 1220 is updated, the viewing data 1224 is updated, the viewing time 1226 is updated, etc. Figure 12 The content consumption metrics extraction module 612 is shown according to some embodiments. The content consumption metrics extraction module 612 builds the content consumption database 618 based on the generation 606, the consumption 608, the users 604, and the content items 602. For example, a user 604 can access a content item 602 and view the content item 602 on their mobile device. The content consumption metrics extraction module 612 updates the database entry for the consumed content 1204 that matches the content item 602 based on the user 1214's interaction with the content item 602 in the consumed content 1204, where the user 604 is added as a user 1214, the average viewing time 1216 is updated, the view count 1220 is updated, the viewing data 1224 is updated, the viewing time 1226 is updated, etc.

[0144] are discussed in conjunction with one another Figure 14 to Figure 16 . Figure 14The determine engagement score module 622 is shown according to some embodiments. The determine engagement score module 622 uses statistics from the annotation database 614, the content consumption database 618, and the user database 620 to generate a metrics database 1410. The determine engagement score module 622 then uses the metrics database 1410 and the weight table 1402 to determine an engagement score 1408. The engagement score 1408 includes a total content consumption metric 1420 and content consumption metrics 1404, both of which are for the annotation 914.

[0145] According to some embodiments, the metrics database 1410 is organized according to a schema 1412. The schema 1412 includes a date 1414, a content item 1416, a content consumption metric 1404, and a metric value 1406. The date 1414 can be the same or similar to the date 1102 of the content consumption database 618. Figure 11 The content item 1416 can be an indication of the content item 602. The content consumption metric 1404 includes one or more of the fields disclosed in connection with Figure 12 The metric value 1406 is a value for the corresponding content consumption metric 1404.

[0146] Figure 15 A graph 1500 of usage of an annotation is shown according to some embodiments. In Figure 15 The variable value 1502 along the vertical axis and the date 1504 in weeks along the horizontal axis are shown in the graph 1500. The variable value 1502 is a normalized value of the content consumption metric 1404, including a screenshot 1506, a view time 1508, a view 1510, a number of shares 1512, and a save 1514. A share indicates that a user 604 shared the content item 602 with another user 604 of the messaging system 100, for example, in a transient message.

[0147] According to some embodiments, the graph 1500 is presented by the user interface module 628. The example annotation 914 is a bike with different content consumption metrics 1404. Even with a graph of only five content consumption metrics 1404, a user can have difficulty understanding the graph 1500. In some embodiments, the graph 1500 includes additional content consumption metrics 1404 from the content consumption database 618 and Table 1, which can make it difficult to understand how the content item 602 of the bike with the annotation 914 performed in terms of the user 604 creating and consuming the content item 602 of the bike with the annotation 914.

[0148] Table 1 shows the content consumption metrics 1404, with a weight 1418 for popularity, a weight 1418 for passion.

[0149]

[0150]

[0151] The determine engagement score module 622 uses the weight table 1402 and the metric database 1410 to determine the total content consumption metrics 1420. The weight table 1402 includes the content consumption metrics 1404 and the weights 1418. Table 1 is an example of the weight table 1402 for the two total content consumption metrics 1420 of popularity and passion. The total content consumption metrics 1420 can be easier for a person to understand the consumption of the content item 602 with the annotations 914. The weight of popularity and the weight of passion are the weights 1418 of the content consumption metrics 1404. In Table 1, the weight of popularity is 4 for views and the weight of passion is 0 for views.

[0152] In some embodiments, the determine engagement score module 622 is configured to determine the engagement scores 1408 such as popularity and passion according to equations (1) through (3). Equation (1): W = {w(l), w(2),..., w(n)}, where W is a vector of the weights 1418 and w(n) is the nth weight. An example of W is the column of the weights of popularity and passion in Table 1. Equation (2): M = {m(l), m(2),..., m(n)}, M is a vector of the values 1406 of the content consumption metrics 1404, where m(n) is the metric value 1406 of the nth content consumption metric 1404 of the annotations 914. Equation (3): Score = w(l)*m(l) + w(2)*m(2)... + w(n)*m(n), where the score can correspond to the popularity 1602 or the passion 1604 of the annotations 914 for a date or a time window. For the example of the bicycle as the annotations 914, each metric value 1406, m(n), of the nth content consumption metric 1404 of Table 1 is multiplied by the corresponding weight 1418 of the weight of popularity or the weight of passion to determine the value of the popularity 1602 or the passion 1604, respectively. Both the popularity 1602 and the passion 1604 are engagement scores 1408 of the type total content consumption metrics 1420 of the annotations 914.

[0153] Figure 16 A plot 1600 of the usage of the annotations 914 is shown according to some embodiments. The plot 1600 shows the normalized values of the content consumption metrics of popularity 1602 and passion 1604 of Table 1 for the annotations 914 during weeks 1 through 23. The annotations 914 are the same bicycle in Figure 15

[0154] Figure 17 ​The adjust engagement score module 624 is shown to adjust engagement scores 1408 according to some embodiments. The adjust engagement score module 624 obtains engagement scores 1408 and adjusts them to generate adjusted engagement scores 1704. The activity detection module 626 monitors the adjusted engagement scores 1704 and notifies a user if the adjusted engagement scores 1704 are abnormally low or high. For example, the determine engagement scores module 622 can continuously update engagement scores 1408, while the adjust engagement scores module 624 can continuously determine adjusted engagement scores 1704. The activity detection module 626 detects abnormal or irregular scores and can take action 1702 based on the abnormal or irregular scores. For example, the activity detection module 626 can detect a trend variable value 2504 of false 2210 in Figure 25 a trend variable value 2604 of false 2610 in Figure 26 a trend variable value 2704 of false 2710 in Figure 27 a trend variable value 2804 of false 2810 in Figure 28 a trend variable value 2904 of false 2910 in Figure 29 The activity detection module 626 can take action 1702 to notify the user that the television associated with the annotation is outside of a normal range. The activity detection module 626 can cause a content item 602 associated with the annotation to be generated by another module, or can prompt the user to generate an additional content item 602 associated with the annotation. The activity detection module 626 can take action 1702 to cause a modification 722 related to the annotation to be generated, which can be provided to the user 604 by the augmentation system 206.

[0155] Adjustment of engagement scores 1408 by the adjust engagement score module 624 to generate adjusted engagement scores 1704 is discussed in connection with Figure 18 to Figure 29

[0156] Figure 18 Raw data 1814 for four annotations is shown according to some embodiments. Figure 18 Content consumption 1802 along a vertical axis and dates 1804 by week along a horizontal axis are shown inThe content consumption 1802 indicates the number of content items 602 that were consumed 608 that had annotation 1 1806, annotation 2 1808, annotation 3 1810, or annotation 4 1812. These annotations are example annotations such as indoor, outdoor, cat, dog, bowling, sports, friends, running, and so on. There can be zero content consumption 1802 for an annotation on certain days or weeks. It can be difficult to explain the content consumption 1802 for an annotation due to the weekly variations, as indicated by the dates 1804 by week. The content consumption 1802 is an indication of engagement by the user 604 with the annotations.

[0157] In some implementations, content consumption 1802 can be represented with a time series as shown in Equation (4), which separates content consumption 1802 into a trend component, a seasonal component, and a remainder component, which can be the portion of content consumption 1802 that cannot be attributed to the trend component or the seasonal component.

[0158] Equation (4): Time 序列 = Trend 分量 + Seasonality 分量 + Remainder.

[0159] Figure 19 A plot 1900 of a labeled simple moving average (SMA) 1902 is shown, in accordance with some implementations. The adjustment engagement score module 624 determines the SMA using the same raw data 1814 as Figure 18 Equations (5) and (6), in accordance with some implementations. Equation (5): where d i is the ith date, m 归一化 (d i ) is the value of the content consumption metric 1404 or the value of the total content consumption metric 1420 for label l at d i , w is the time window size, and d k is the date for which the SMA is determined. Equation (5) applies when d k >= w, and Equation (6) applies when d k < w. Equation (6): The window size w used for plot 1900 is 14 days. Other window sizes, such as 7 days, 28 days, and so on, can also be used.

[0160] Figure 20 A plot 2000 of a labeled trend momentum (TM) 2002 is shown, in accordance with some implementations. The adjustment engagement score module 624 determines the labeled trend momentum 2002 using the same raw data 1814 as Figure 18 Equations (7), in accordance with some implementations. Equation (7): where w 短 is a short-term time window size, w 长 is a long-term time window size, and a is a weight, where a is assigned a value between 0 and 1. For plot 2000, the adjustment engagement score module 624 uses w 短 = 3 days, w 长 = 7 days, and a = 0.9. In some implementations, the adjustment engagement score is determined to be TM, where TM is based on subtracting a first SMA having a first window from a second SMA having a second window, where the second window is shorter than the first window.

[0161] Figure 21A plot 2100 of momentum (M) 2102 is shown, according to some embodiments. The adjusted engagement score module 624 determines the trended momentum 2002 for the raw data 1814 using Figure 18 Equation (7), based on where w is the time window size. For the plot 2100, the adjusted engagement score module 624 uses w = 7 days.

[0162] Figure 22 A plot 2200 of statistical analysis of the label 1 1806 is shown, according to some embodiments. The adjusted engagement score module 624 determines the daily statistical analysis 2202 and the monthly statistical analysis 2204 for the label 1 1806, based on the range of values, the standard deviation, and the percentiles, using the raw data 1814. Figure 18 Figure 22 The normalized detrended values 2206 along the vertical axis and the workdays 2208 or months 2210 along the horizontal axis are shown in

[0163] Figure 23 A plot 2300 of static seasonality of the label 1 1806 is shown, according to some embodiments. The adjusted engagement score module 624 determines the value of the static seasonality 2302 for the label 1 1806, based on Equation (9), using the raw data 1814. Figure 18 Equation (9): Detrended = Time 序列 - Trend 分量 . So according to Equation (9), the detrended is the time series without the trend component. Figure 23 The static seasonality 2302 along the vertical axis and the weekly dates 2304 along the horizontal axis are shown in

[0164] Figure 24 A plot 2400 of seasonally normalized detrended of the label 1 1806 is shown, according to some embodiments. Figure 24 The seasonally normalized detrended 2402 along the vertical axis and the weekly dates 2404 along the horizontal axis are shown in Figure 18 The adjusted engagement score module 624 determines the value of the seasonally normalized detrended 2402 for the label 1 1806, based on Equations (10) to (13), using the raw data 1814. Equation (10): where ψ(d i ) refers to the workday of the data d i , w is the time window size, and m is the content consumption metric 1404 or the total content consumption metric 1420. In Figure 24 , w = 28, and the seasonal period (SP) = 7, which is every workday. Equation (11): where s represents the seasonal ​

[0165] Equation (12): Then the seasonal component = Equation (13)

[0166] Equation (13):

[0167] Figure 25 to Figure 29 A time series of adjusted engagement scores 1704 of the aggregate content consumption metric 1420 that analyzes buzz is shown. Figure 25 A plot 2500 of a trend component 2502 of buzz according to some embodiments is shown. The adjusted engagement score module 624 determines the trend variable values 2504 of Buzz Label 1 1806 on a weekly basis 2506 according to Equation (4) of Buzz Label 1. According to some embodiments, the adjusted engagement score module 624 determines the trend variable values 2504 according to Equation (4) and Equation 7).

[0168] Equations (14) and (15) define the normal range and the trend range, respectively. Equation (14): Normal Range: Mean - Threshold * Standard Deviation < Trend Variable Score < Mean + Threshold * Standard Deviation. Equation (15): Trend Range: Variable Score < Mean - Threshold * Standard Deviation; or, Variable Score > Mean + Threshold * Standard Deviation. In some embodiments, instead of using the mean and the standard deviation, the normal and trend ranges are determined based on the 25th percentile (Ql) and the 75th percentile (Q3), for example: Normal Range: Ql - Threshold * IQR < Trend Variable Score < Q3 + Threshold * IQR, where IQR = Q3 - Ql. Equation (15): Trend Range: Variable Score < Ql - Threshold * IQR; or, Variable Score > Q3 + Threshold * IQR.

[0169] Figure 25 A negative two standard deviation (SDS) 2220 determined with the first part of Equation (15) is shown, where the threshold is two. A positive two standard deviation 2218 is determined with the second part of Equation (15), where the threshold is two. In some embodiments, the normal 2214 is the average of the trend variables (TVs) in the date range of the week 2506. It can be appreciated that the normal 2214 can be determined in other ways.

[0170] The legend 2508 indicates the categories of the TV values 2504 as false 2210 or true 2212. False 2210 indicates that the TV value 2504 is less than the negative two SDS 2220 or greater than the positive two SDS 2218, or that the TV value 2504 is in the normal range of Equation (14). True 2212 indicates that the TV value 2504 is less than the negative two SDS 2220 or that the TV value 2504 is greater than the positive two SDS 2218; or, that the TV value 2504 is in the trend range defined by Equation (15).

[0171] Figure 26 A graph 2600 showing the seasonal component 2602 of popularity according to some embodiments of reference 1 is shown. The participation score module 624 is adjusted based on equations (10) to (13) using... Figure 18 The original data 1814 determines the trend variable value 2604 of the week 2606 for label 1 1806. According to some implementations, the adjustment participation scoring module 624 determines the trend variable value 2604 based on equation (4). For example, the trend variable value 2604 can be determined from equation (7). Legend 2608 indicates that the category of TV value 2604 is false 2610 or true 2612. False 2610 indicates that TV value 2604 is less than negative two SDS2220 or greater than positive two SDS2218, or TV value 2604 is within the normal range of equation (14). True 2612 indicates that TV value 2604 is less than negative two SDS2220 or TV value 2604 is greater than positive two SDS2218; or, TV value 2604 is within the trend range defined by equation (15).

[0172] Figure 27 A graph 2700 shows the seasonal component of the popularity of Thursday 2702 according to some implementations of label 1. The participation score module 624 is adjusted based on equations (10) to (13) using Figure 18 The original data 1814 determines the trend variable value 2704 of the week 2706 labeled 11806. According to some implementations, the adjustment participation scoring module 624 determines the trend variable value 2704 based on equation (4). For example, the trend variable value 2604 can be determined from equation (7). Legend 2708 indicates that the category of TV value 2704 is false 2710 or true 2712. False 2710 indicates that TV value 2704 is less than negative two SDS2220 or greater than positive two SDS2218, or TV value 2704 is within the normal range of equation (14). True 2612 indicates that TV value 2704 is less than negative two SDS2220 or TV value 2704 is greater than positive two SDS2218; or, TV value 2704 is within the trend range defined by equation (15).

[0173] Figure 28 A graph 2800 of the denoised component 2802 of popularity (labeled 1) according to some embodiments is shown. The participating scoring module 624 is adjusted according to equation (16). Figure 18trend variable values 2804 for Label 1 1806 on a week 2806 basis using the raw data 1814. Equation (16): Raw - Residual = Trend + Seasonality. The legend 2808 indicates that the category for the television value 2804 is false 2810 or true 2812. False 2810 indicates that the television value 2804 is less than the negative two SDS 2220 or greater than the positive two SDS 2218, or the television value 2804 is within the normal range of equation (14). True 2812 indicates that the television value 2804 is less than the negative two SDS 2220 or the television value 2804 is greater than the positive two SDS 2218; or, the television value 2804 is within the trend range defined by equation (15).

[0174] Figure 29 A plot 2900 showing the denoised component of the week one 2902 popularity of Label 1 is shown in accordance with some embodiments. The engagement score module 624 adjusts the category for the television value 2904 to be false 2910 or true 2912 based on equation (16) using the raw data 1814. Figure 18 trend variable values 2904 for Label 1 1806 on a week 2906 basis using the raw data 1814. The legend 2908 indicates that the category for the television value 2904 is false 2910 or true 2912. False 2910 indicates that the television value 2904 is less than the negative two SDS 2220 or greater than the positive two SDS 2218, or the television value 2904 is within the normal range of equation (14). True 2912 indicates that the television value 2904 is less than the negative two SDS 2220 or the television value 2904 is greater than the positive two SDS 2218; or, the television value 2904 is within the trend range defined by equation (15).

[0175] Figure 30 A user interface module 628 is shown in accordance with some embodiments. The user interface module 628 presents a menu for a user to select values for filters 3002. The filters 3002 include a time window 3004, a label 3006, a content consumption metric 3008, and a classifier 3010. For example, to have the plot 1800 presented on a display, a user selects “raw data” as the classifier 3010; the user selects one or more of Label 1 1806, Label 2 1808, Label 3 1810, and Label 4 1812 as the label 3006; the user selects a content consumption metric of “total time spent” as the content consumption metric 3008. Figure 18

[0176] Figure 31 A method 3100 for label-based engagement analysis is shown in accordance with some embodiments. The method 3100 begins processing a content item at operation 3102. The operation 3102 can include processing a content item that includes an image to identify elements in the image. For example, Figure 9 The content item analysis module 902 processes the content item 602 that includes the image 708 and the video 710 to generate the elements 726 based on the element descriptors 918.​

[0177] The method 3100 continues at operation 3104 with determining a label for the image. In some implementations, operation 3104 includes determining a label for the image based on a condition that indicates when to associate a label in the label with an image in the image based on an element in the image. For example, Figure 9 The label extraction module 610 of the system 600 determines a label 914 for the content item 602 based on the element 726 identified in the content item 602.

[0178] The method 3100 continues at operation 3106 with associating the label with the content item. In some implementations, operation 3106 includes associating the label with the content item in response to determining to associate the label with the image. For example, continuing the example of operation 3104, the label extraction module 610 stores an association between the content item 602 and the label 914 in the label database 614.

[0179] The method 3100 continues at operation 3108 with determining an engagement score for the label. In some implementations, operation 3108 includes determining an engagement score for the label based on user interactions with content items associated with the label. For example, Figure 14 The determination engagement score module 622 of the system 600 determines an engagement score 1408.

[0180] The method 3100 continues at operation 3110 with adjusting the engagement score. In some implementations, operation 3110 includes adjusting the engagement score to determine a trend for the label, generating an adjusted engagement score. For example, the adjustment engagement score module 624 adjusts the engagement score 1408 and generates an adjusted engagement score 1704.

[0181] One or more operations of the method 3100 can be optional. The method 3100 can include one or more additional operations. The operations of the method 3100 can be performed in a different order. According to some implementations, the machine 3200, or a device of the machine 3200, is configured to perform the method 3100 and other methods disclosed herein.

[0182] Machine Architecture

[0183] Figure 32is a diagrammatic representation of a machine 3200 within which instructions 3208 (e.g., software, a program, an application, an applet, an app, or other executable code) for causing the machine 3200 to perform any of the methodologies discussed herein can be executed. For example, the instructions 3208 can cause the machine 3200 to execute any of the methodologies described herein. The instructions 3208 transform the general, non-programmed machine 3200 into a particular machine 3200 programmed to carry out the described and illustrated functions in the manner described. The machine 3200 can operate as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 3200 can operate in the capacity of a server machine or a client machine in server-client network environments, or as a peer machine in peer-to-peer (or distributed) network environments. The machine 3200 can comprise, but not be limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular telephone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing the instructions 3208, sequentially or otherwise, that specify actions to be taken by machine 3200. Further, while only a single machine 3200 is illustrated, the term “machine” shall also be taken to include a collection of machines that individually or jointly execute the instructions 3208 to perform any of the methodologies discussed herein. For example, the machine 3200 can include any of the client device 102 or a number of server devices forming a part of the messaging server system 108. In some examples, the machine 3200 can further include both a client and a server system, wherein certain operations of a particular method or algorithm are performed in the server side, and certain operations of the particular method or algorithm are performed in the client side.

[0184] The machine 3200 can include processors 3202, memory 3204, and input / output (I / O) components 3238, which can be configured to communicate with each other via a bus 3240. According to some examples, the processors 3202 can be referred to as computer processors. In examples, the processors 3202 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio-frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) can include, for example, a processor 3206 and a processor 3210 that execute instructions 3208. The term “processor” is intended to include multiple cores of a single processor, that can include two or more independent processors (sometimes called “cores”) that can execute instructions contemporaneously. Although FIG. 3 shows multiple processors 3202, the machine 3200 can include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof. Figure 32 The machine 3200 includes processors 3202, memory 3204, and input / output (I / O) components 3238, which can be configured to communicate with each other via a bus 3240. According to some examples, the processors 3202 can be referred to as computer processors. In examples, the processors 3202 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio-frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) can include, for example, a processor 3206 and a processor 3210 that execute instructions 3208. The term “processor” is intended to include multiple cores of a single processor, that can include two or more independent processors (sometimes called “cores”) that can execute instructions contemporaneously. Although FIG. 3 shows multiple processors 3202, the machine 3200 can include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0185] The memory 3204 includes main memory 3212, a static memory 3214, and storage unit 3216, which can be accessed via the bus 240 by the processors 3202. The main memory 3204, static memory 3214, and storage unit 3216 store the instructions 3208 that implement any one or more of the methodologies or functions described herein. The instructions 3208 can also reside completely, or a portion thereof, within the main memory 3212, within the static memory 3214, within the storage unit 3216, within at least one of the processors 3202 (e.g., within the processor’s cache memory), or any suitable combination thereof, during execution thereof by the machine 3200.

[0186] The I / O components 3238 can include various components to receive input, provide output, produce output, transmit information, exchange information, capture measurements, and so forth. The specific I / O components 3238 that are included in the particular machine will depend on the type of machine. For example, portable machines such as mobile phones will likely include a touch input device or other such input mechanisms, while a headless server machine will not. It will be appreciated that the I / O components 3238 can include Figure 32Many other components not shown in FIG. 31 can also be included. In various examples, the I / O components 3238 can include user output components 3224 and user input components 3226. The user output components 3224 can include visual components (e.g., a display such as a plasma display panel (PDP), a light-emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., speakers), haptic components (e.g., a vibratory motor, resistance mechanisms), other signal generators, and so forth. The user input components 3226 can include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing instruments), tactile input components (e.g., a physical button, a touch screen that provides location and force of touches or touch gestures, or other tactile input components), audio input components (e.g., a microphone), and the like.

[0187] In further examples, the I / O components 3238 can include biometric components 3228, motion components 3230, environmental components 3232, or position components 3234, among a wide array of other components. For example, the biometric components 3228 include components to detect expressions (e.g., hand expressions, facial expressions, vocal expressions, body gestures, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identify a person (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or electroencephalogram-based identification), and the like.

[0188] The environmental components 3232 include, for example, one or more cameras (with still and / or video capabilities), illumination sensors (e.g., photometers), temperature sensors (e.g., one or more thermometers that detect ambient temperature), humidity sensors, pressure sensors (e.g., barometers), acoustic sensors (e.g., one or more microphones that detect background noise), proximity sensors (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors to detect concentrations of hazardous gases for safety or to measure pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to a physical environment.

[0189] With respect to cameras, the client device 102 can have a camera system that includes, for example, a front-facing camera on a front surface of the client device 102 and a rear-facing camera on a rear surface of the client device 102. The front-facing camera can be used, for example, to capture still images and videos of a user of the client device 102 (e.g., “selfies”) that can then be enhanced with the above-described enhancement data (e.g., filters). The rear-facing camera can be used, for example, to capture still images and videos in a more traditional camera mode that are similarly enhanced using the enhancement data. In addition to front- and rear-facing cameras, the client device 102 can also include a 360° camera for capturing 360° photos and videos.

[0190] In addition, the camera system of the client device 102 can include dual rear-facing cameras (e.g., a primary camera and a depth-sensing camera), or even a triple, quadruple, or penta- rear-facing camera configuration on the front and rear sides of the client device 102. These multi-camera systems can include, for example, a wide-angle camera, an ultra-wide-angle camera, a telephoto camera, a macro camera, and a depth sensor.

[0191] The positioning component 3234 includes location sensor components (e.g., GPS receiver components), altitude sensor components (e.g., altimeters or barometers that detect atmospheric pressure from which altitude can be derived), orientation sensor components (e.g., magnetometers), and the like.

[0192] Communication can be implemented using a wide variety of technologies. The I / O component 3238 also includes a communication component 3236, which can be operable to couple the machine 3200 to a network 3220 or to devices 3222 via respective coupling or connection. For example, the communication component 236 can include a network interface component or another suitable device to interface with the network 3220. In further examples, the communication component 3236 can include a wired communication component, a wireless communication component, a cellular communication component, a Near Field Communication (NFC) component, Components (e.g., Low power consumption), Components and other communication components to provide communication via other modalities. The devices 3222 can be other machines or any of a wide variety of peripheral devices (e.g., peripheral devices coupled via a USB).

[0193] Moreover, the communication components 3236 can detect identifiers or include components operable to detect identifiers. For example, the communication components 3236 can include radio frequency identification (RFID) tag reader components, NFC smart tag detection components, optical reader components (e.g., an optical sensor to detect one-dimensional bar codes such as Universal Product Code (UPC) bar codes, multi-dimensional bar codes such as Quick Response (QR) codes, Aztec codes, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D bar codes, and other optical codes), or acoustic detection components (e.g., microphones to identify tagged audio signals). In addition, a variety of information can be derived via the communication components 3236, such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, location via cellular signal triangulation, location via the detection of NFC beacon signals that can indicate a particular location, and so forth.

[0194] The various memories (e.g., main memory 3212, static memory 3214, and memory shared by the processor 3202) and memory unit 3216 can store one or more sets of instructions and data structures (e.g., software) realized or used by any one or more of the methods or functions described herein. The instructions (e.g., instructions 3208) realized by the processor 3202, when executed, cause the various operations to implement the disclosed examples.

[0195] The instructions 3208 can be transmitted or received via the network 3220 using a transmission medium and any one of a number of well-known transfer protocols (e.g., Hyper Text Transfer Protocol (HTTP)). Similarly, the instructions 3208 can be transmitted or received using a transmission medium via the coupling (e.g., peer-to-peer coupling) to the devices 3222.

[0196] Software Architecture

[0197] Figure 33is a block diagram 3300 illustrating software architectures 3304, which can be installed on any one or more of the devices described herein. The software architectures 3304 are supported by hardware, such as machine 3302, which includes processors 3320, memory 3326, and I / O components 3338. In this example, the software architectures 3304 are conceptual ly illustrated as a stack of layers, where each layer provides specific functionality. The software architectures 3304 include layers such as an operating system 3312, libraries 3310, frameworks 3308, and applications 3306. Operationally, the applications 3306 and the framework 3308 modules can invoke API calls 3350 through the software stack and receive messages 3352 in response to the API calls 3350.

[0198] The operating system 3312 manages hardware resources and provides common services. The operating system 3312 includes, for example, a kernel 3314, services 3316, and drivers 3322. The kernel 3314 acts as an abstraction layer between the hardware and the other software layers. For example, the kernel 3314 provides memory management, processor management (e.g., scheduling), component management, networking, and security settings, among other functionality. The services 3316 can provide other common services that the applications 3306 and other software layers use. The drivers 3322 are responsible for controlling or interfacing with the underlying hardware, according to techniques described herein. For instance, the drivers 3322 can include display drivers, camera drivers, Bluetooth® drivers, flash or Low-Power drivers, flash memory drivers, serial communication drivers (e.g., USB drivers), audio drivers, power management drivers, and so forth.

[0199] The libraries 3310 provide a higher-level common infrastructure, including system C libraries 3318 (e.g., C standard library), as well as APIs 3324 (e.g., media libraries 3324, graphics libraries 3324, database libraries 3324, web libraries 3324, and so on), which can be used by the applications 3306 and other software layers. The middleware 3308 can provide a higher-level common infrastructure that can be used by the applications 3306 and other software layers. The middleware 3308 is examples of such infrastructure can include a database middleware layer 3330 that can provide various types of database management functions, an authentication middleware layer 3332 that can provide various types of authentication mechanisms, a network middleware layer 3334 that can provide various types of network connectivity functions, and so on.​

[0200] The framework 3308 provides common high-level facilities that are used by the applications 3306. For example, the framework 3308 provides various graphical user interface (GUI) functions, high-level resource management, and high-level positioning services. The framework 3308 can provide a broad spectrum of other APIs that can be used by the applications 3306, some of which can be specific to a particular operating system or platform.

[0201] In an example, the applications 3306 can include a home application 3336, a contacts application 3330, a browser application 3332, a book reader application 3334, a participation analytics application 3341, a location application 3342, a media application 3344, a messaging application 3346, a game application 3348, and a broad assortment of other applications such as a third-party application 3340. The participation analytics application 3341 can be the same as or similar to the participation analytics application 1341 described in connection with Figure 6 The application 3306 is a program that executes functions defined in the program. One or more of the applications 3306 can be created using a variety of programming languages such as object-oriented programming languages (e.g., Objective-C, Java, or C++) or procedural programming languages (e.g., C or assembly language) in a variety of ways. In a particular example, the third-party application 3340 (e.g., an application developed by an entity other than the vendor of the particular platform) can be a mobile software running on a mobile operating system such as the IOS TM or ANDROID TM software development kit (SDK) developed by the vendor of the particular platform. In this example, the third-party application 3340 can invoke the API calls 3350 provided by the operating system 3312 to, for example, facilitate the functionality described herein. TM TM In this example, the third-party application 3340 can invoke the API calls 3350 provided by the operating system 3312 to, for example, facilitate the functionality described herein.

[0202] Processing components

[0203] Turning now to Figure 34 , a schematic diagram of a processing environment 3400 is shown, which includes a processor 3402, a processor 3406, and a processor 3408 (e.g., a GPU, a CPU, or a combination thereof). The processor 3402 is shown coupled to a power supply 3404 and includes (permanently configured or temporarily instantiated) modules, namely, a determining participation score component 3410, an adjusting participation score component 3412, and an activity detection component 3414. Reference is made to Figure 14 and Figure 17 ​​determining engagement score component 3410 operable to generate engagement scores 1408; adjusting engagement score component 3412 operable to generate adjusted engagement scores 1704; and activity detection component 3414 operable to generate actions 1702. As shown, processor 3402 is communicatively coupled to both processor 3406 and processor 3408.

[0204] Glossary

[0205] “Carrier signal” refers to any intangible medium that is capable of storing, encoding, or carrying the instructions for execution by a machine and includes digital or analog communications signals or other intangible media to facilitate communication of such instructions. Instructions can be transmitted or received over a network via a network interface device utilizing a transmission medium.

[0206] “Client device” refers to any machine that interfaces to a communications network to obtain resources from one or more server systems or other client devices. A client device can be, but is not limited to, a mobile phone, desktop computer, laptop computer, portable digital assistants (PDAs), smart phones, tablet computers, ultrabooks, netbooks, laptops, multi-processor systems, microprocessor-based or programmable consumer electronics, game consoles, set-top boxes, or any other communication device that a user can use to access a network.

[0207] “Communication network” refers to one or more portions of a network that can be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. The network, other types of networks, or a combination of two or more such networks. For example, a network or a portion of a network can include a wireless network or a cellular network, and the coupling can be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile Communications (GSM) connection, or other types of cellular or wireless coupling. In this example, the coupling can enable any of a variety of types of data transfer techniques, such as Single Carrier Radio Transmission Technology (lxRTT), Evolution-Data Optimized (EVDO) technology, General Packet Radio Service (GPRS) technology, Enhanced Data rates for GSM Evolution (EDGE) technology, third Generation Partnership Project (3GPP) including 3G, fourth generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Worldwide Interoperability for Microwave Access (WiMAX), Long Term Evolution (LTE) standard, others defined by various standards setting organizations, other long range protocols, or other data transfer techniques.

[0208] A "component" refers to a device, physical entity or logic having boundaries defined by function or subroutine calls, branches points, APIs, or other technologies that provide for the partitioning or modularization of a particular processing or control function. Components can be combined via their interfaces to execute machine processes, such as those processes described herein. A component can be a packaged functional hardware unit designed for use with other components and can be a part of a program or piece of code that can be used to implement a specific function. Components can constitute either software components (e.g., code embodied on a machine-readable medium) or hardware components. A "hardware component" is a tangible unit capable of performing certain operations and can be configured or arranged in a certain physical manner. In various example embodiments, one or more computer systems (e.g., a standalone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a processor core) can be configured by software (e.g., an application or application portion) as a hardware component that operates to perform certain operations as described herein. A hardware component can also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component can include dedicated circuitry or logic that is permanently configured to perform certain operations. A hardware component can be a special-purpose processor, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). A hardware component can also include programmable logic or circuitry that is temporarily configured by software to perform certain operations. For example, a hardware component can include software executed by a general-purpose processor or other programmable processor. Once configured by such software, hardware components become specific machines (or specific components of a machine) uniquely tailored to perform the configured functions and are no longer general-purpose processors. It will be appreciated that theWhere multiple hardware components are present at the same time, communication can be achieved through signal transmission between or among two or more hardware components (e.g., through appropriate circuits and buses). Where multiple hardware components are configured or instantiated at different times, communication between such hardware components can for example be achieved through storage of information in a memory structure that is accessible to all multiple hardware components, and retrieval of that information by the multiple hardware components at different times. For example, one hardware component can perform an operation, and store the output of that operation in a memory device to which it is communicatively coupled. A further hardware component can then, at a later time, access the memory device to retrieve the stored output and process the stored output. The hardware components can also initiate communications with input or output devices, and can operate on resources (e.g., collection of information). The various operations of example methods described herein can be performed, at least partially, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors can constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, “processor-implemented component” refers to a hardware component implemented using one or more processors. Similarly, the methods described herein can be at least partially processor-implemented, with a particular processor or processors being an example of hardware. For example, at least some of the operations of a method can be performed by one or more processors 3202 or processor-implemented components. Moreover, the one or more processors can also operate to support performance of the relevant operations in a “cloud computing” environment or as a “software as a service” (SaaS). For example, at least some of the operations can be performed by a group of computers (as examples of machines including processors), with these operations being accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., an API). The performance of certain of the operations can be distributed among the processors, not only residing within a single machine, but deployed across a number of machines. In some example embodiments, the processors or processor-implemented components can be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other example embodiments, the processors or processor-implemented components can be distributed across a number of geographic locations.

[0209] “Computer-readable storage medium” refers to both machine-storage media and transmission media. Thus, the terms “computer-readable storage medium” and “computer-readable medium” are intended to include both storage devices / media and carrier waves / modulated data signals. The terms “machine-readable medium,” “computer-readable medium,” and “device-readable medium” mean the same thing and can be used interchangeably in this disclosure. The plural forms “computer-readable media” can be used to refer to more than one computer-readable medium.

[0210] A "ephemeral message" refers to a message that is accessible for a limited duration of time. An ephemeral message can be text, image, video, etc. The access time for an ephemeral message can be set by the message sender. Alternatively, the access time can be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is temporary.

[0211] "Machine-storage medium" refers to a single or multiple storage devices and media (e.g., centralized or distributed databases, and associated caches and servers) that store executable instructions, routines, and data. Thus, the term should be taken to include, but not be limited to, solid state memories, as well as optical and magnetic media, including memory internal or external to processors. Specific examples of machine-storage media, computer-storage media, and / or device-storage media include non-volatile memory, including by way of example semiconductor memory devices, e.g., erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine-storage medium," "device-storage medium," "computer-storage medium," and "device-storage medium" mean the same thing and can be used interchangeably in this disclosure. The terms "machine-storage medium," "computer-storage medium," and "device-storage medium" explicitly contemplate that the term "medium" is not limited to tangible, physical, or non-transitory machine-storage media.

[0212] "Non-transitory computer-readable storage medium" refers to a tangible medium that is capable of storing, encoding, or carrying the instructions for execution by a machine.

[0213] "Signal medium" refers to any intangible medium that is capable of storing, encoding, or carrying the instructions for execution by a machine, and includes digital or analog communications signals or other intangible media to facilitate communication of software or data. The term "signal medium" shall be taken to include any form of a modulated data signal, carrier wave, and so on. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The terms "transmission medium" and "signal medium" mean the same thing and can be used interchangeably in this disclosure.

Claims

1. A method for labeled engagement analysis, comprising: processing a content item comprising a plurality of images to identify elements in the images, wherein metadata is associated with the content item, and wherein the images comprise augmentations; determining a plurality of labels for the plurality of images based on a condition that indicates when to associate a label of the plurality of labels with an image of the plurality of images based on an element of the image, the metadata, and the augmentations, the plurality of labels having a hierarchical position; in response to determining to associate the label with the image, associating the label with the content item and associating an ancestor of the label within a hierarchical structure of labels with the content item; determining an engagement score for the label based on a user’s interaction with the content item associated with the label; and adjusting the engagement score to determine a trend for the label, thereby generating an adjusted engagement score.

2. The method of claim 1, further comprising: storing the user’s interaction with the content item.

3. The method of claim 1, further comprising: causing the adjusted engagement score to be displayed on a display of a computer.

4. The method of claim 1, wherein, The user’s interaction with the content item comprises a plurality of content consumption metrics, and wherein the method further comprises: determining a passion as the engagement score for the label based on multiplying a weight of a plurality of weights with a corresponding content consumption metric of the plurality of content consumption metrics.

5. The method of claim 4, wherein, The plurality of weights are a first plurality of weights, and wherein the method further comprises: determining a popularity as the engagement score for the label based on multiplying a weight of a second plurality of weights with a corresponding content consumption metric of the plurality of content consumption metrics.

6. The method of any one of claims 1-4, further comprising: adjusting the engagement score to determine a trend momentum (TM) based on subtracting a first simple moving average (SMA) having a first window from a second SMA having a second window, wherein the second window is shorter than the first window.

7. The method of any one of claims 1-4, further comprising: adjusting the engagement score to determine a dynamic seasonality of the user’s interaction based on an average of a trend component subtracted from a value of the interaction.

8. The method of any one of claims 1 to 4, wherein, The image is generated by a client device associated with the user.

9. The method of any one of claims 1 to 4, wherein, The elements comprise objects, scenes, and actions.

10. The method of any one of claims 1 to 4, wherein, The element of the elements is identified based on using a convolutional neural network trained to identify the element.

11. The method of any one of claims 1-4, further comprising: determining the user’s interaction based on a date range.

12. The method of claim 11, further comprising: determining whether an adjusted engagement score of the adjusted engagement scores is in a normal range or a trend range based on a first percentile and a third percentile of the adjusted engagement scores.

13. The method of claim 12, further comprising: In response to determining that the adjusted engagement score of the adjusted engagement scores is within the normal range, generating a content item associated with the annotation or generating a report indicating the annotation and the adjusted engagement score, and causing the report to be displayed by a computer.

14. A system for annotation-based engagement analysis, comprising: one or more computer processors; and one or more computer-readable media storing instructions that, when executed by the one or more computer processors, cause the system to perform operations comprising: processing a content item comprising a plurality of images to identify elements in the images, wherein metadata is associated with the content item, and wherein the images include augmentations; determining a plurality of annotations for the plurality of images based on conditions indicating when to associate an annotation of the plurality of annotations with an image of the plurality of images based on elements of the image, the metadata, and the augmentations, the plurality of annotations having a hierarchical position; in response to determining to associate the annotation with the image, associating the annotation with the content item and associating an ancestor of the annotation within a hierarchy of annotations with the content item; determining an engagement score for the annotation based on user interactions with the content item associated with the annotation; and adjusting the engagement score to determine a trend for the annotation, thereby generating an adjusted engagement score.

15. A non-transitory computer-readable storage medium storing instructions for execution by one or more processors of a device of a computer, the instructions to configure the one or more processors to: processing a content item comprising a plurality of images to identify elements in the images, wherein, metadata is associated with the content item, and wherein the images include augmentations; determining a plurality of annotations for the plurality of images based on conditions indicating when to associate an annotation of the plurality of annotations with an image of the plurality of images based on elements of the image, the metadata, and the augmentations, the plurality of annotations having a hierarchical position; in response to determining to associate the annotation with the image, associating the annotation with the content item and associating an ancestor of the annotation within a hierarchy of annotations with the content item; determining an engagement score for the annotation based on user interactions with the content item associated with the annotation; and adjusting the engagement score to determine a trend for the annotation, thereby generating an adjusted engagement score.

Images