Graph-based annotation of heterogeneous digital content items

Abstract

In some embodiments, techniques for graph-based labeling of digital content items include generating and storing a content graph for a digital content item received by an application system from a user system. The content graph can include nodes corresponding to digital content items with labels, un-labeled nodes corresponding to digital content items without labels, and edges indicating relationships between content items. Edge data for edges between an un-labeled node and adjacent nodes can be retrieved from the content graph. In response to a set of inputs including the retrieved edge data and embedding data associated with the un-labeled node, a machine learning model trained on labeled nodes and edges of the content graph can assign a label to the un-labeled node.

Description

Technical Field

[0001] This disclosure generally relates to the annotation of digital content items using machine learning models, and more specifically to graph-based annotation of heterogeneous digital content items using machine learning models to detect spam and / or content moderation policy triggers. Background Technology

[0002] Network-based application systems can distribute digital content items to a large number of users on a network. Typically, the digital content items distributed by the application system are generated by the system's users. The application system can use algorithms to determine the extent to which it distributes digital content items to its users or to limit the distribution of digital content items to its users. Attached Figure Description

[0003] This disclosure will be more fully understood from the detailed description given below and the accompanying drawings, which are based on various embodiments of this disclosure. However, the drawings should not be construed as limiting this disclosure to the specific embodiments, but are merely for illustrative and understanding purposes.

[0004] Figure 1 The illustration shows an exemplary computing system including a graph-based annotation system according to some embodiments of the present disclosure.

[0005] Figure 2A Examples of application systems including graph-based annotation systems according to some embodiments of this disclosure.

[0006] Figure 2B An example of a digital content item processor communicating with a graph-based annotation system according to some embodiments of the present disclosure is illustrated.

[0007] Figure 3 This is a flowchart of an exemplary method for labeling digital content items according to some embodiments of the present disclosure.

[0008] Figure 4A These are examples of machine learning models for labeling digital content items according to some embodiments of this disclosure.

[0009] Figure 4B1 , Figure 4B2 and Figure 4B3 The diagram illustrates the input to a machine learning model architecture for labeling digital content items according to some embodiments of the present disclosure.

[0010] Figure 4C These are examples of machine learning model architectures for generating embeddings for digital content items, according to some embodiments of this disclosure.

[0011] Figure 4DThis is another example of a machine learning model architecture for generating embeddings for digital content items, according to some embodiments of this disclosure.

[0012] Figure 5 This is a flowchart of an exemplary method for controlling the distribution of digital content items according to some embodiments of the present disclosure.

[0013] Figure 6 This is a block diagram of an exemplary computer system in which embodiments of the present disclosure can operate. Detailed Implementation

[0014] Not every digital content item received by an application system will be, or should be, distributed to all users of that system. The distribution of digital content items can be algorithmically controlled by the application system based on various criteria (e.g., rules, policies, and user preferences). In some application systems, users' ability to generate distributable digital content is largely unrestricted. The continued global expansion of smartphone use has led to an explosive growth in the generation of user-generated content in many different languages.

[0015] The availability of high-speed internet has facilitated the creation and web-based distribution of a massive amount of multimedia content, including digital music, videos, live streaming video, and live podcasts. One report concluded that since 2021, there have been 4.2 billion active social media users worldwide—more than half the world's population. Another report concluded that online media usage, including messaging, voice and video calls, multimedia downloads, photo uploads, work applications, and digital payment transactions, has seen nearly 50 million digital content items per internet minute since August 2020.

[0016] User-generated digital content items can be considered unstructured data because they often lack metadata, such as semantic tags, which application systems could otherwise use to algorithmically control the online distribution of such digital content items. Specifically, users often want their content items to "spread like a virus" online without being labeled as spam or unwanted content. In practice, some users may attempt to disguise the true nature of their content items to avoid detection by spam filters and content moderation strategies. When metadata is unavailable for digital content items, the application system needs to analyze the available characteristics of the digital content items and create or allocate metadata based on those characteristics.

[0017] Using, for example, a rules engine, an application system can compare content item tags with content distribution strategies (such as spam policies and content moderation policies). In some contexts, terms such as content distribution strategy, content moderation strategy, spam filter, and spam policy can be used interchangeably. Based on the results of these comparisons, the application system can constrain the distribution of digital content items tagged as different types of spam (e.g., inflammatory content, financial fraud, adult content, hate speech, unsuitable work, etc.) in different ways. On the other hand, the application system can also promote the distribution of digital content items tagged as, for example, high-quality, educational, or personal health.

[0018] As used herein, spam can refer to any category of unwanted digital content, including but not limited to various categories of digital content items that trigger content moderation policies. Spam categories can include negative labels such as inflammatory messages, financial fraud, adult content, hate speech, and workplace inappropriateness. Spam categories can also include positive labels such as high-quality, educational, health, personal, and privacy. Applicable spam categories may vary and depend on the specific application and / or the usage at hand.

[0019] Many application systems handle a wide range of content types, modalities, languages, and content distribution strategies. The terms "content type" and "modality" are used herein to refer to different attributes of content items that the system can use to categorize or classify digital content items. However, in some uses, the terms "modality" and "content type" may be used interchangeably; the definitions provided are for ease of discussion within the scope of this disclosure.

[0020] As used herein, a content type can refer to the means by which a digital content item exists within an application system. For example, a content type can correspond to an action taken by a user of the application system to publish or otherwise introduce a digital content item into the application system. Examples of content types include posts, comments, replies, and articles. The content type of a digital content item can be a signal of dependency on another digital content item within the application system. For example, a content item with the "comment" type has a dependency on at least one other content item.

[0021] An example of dependency is time dependency. Time dependency can be determined at least in part by the timestamps associated with content items. A timestamp is system-generated digital data that indicates a specific date and time according to the system clock. For example, a timestamp can indicate the date and time an article was uploaded to the application system, or the date and time a post, comment, or reply was created in the application system. Timestamps can indicate time dependencies because a comment on an article will have a timestamp that is later than the article's timestamp, and a reply to a comment will have a timestamp that is later than the comment's timestamp. A content item can have dependencies on multiple different other content items. For example, a comment on an article can also be a reply to another comment on the same article.

[0022] Alternatively, different digital content items can have different types of non-temporal relationships with other digital content items in the application system, including various types of semantic relationships and social graph-based relationships. Examples of semantic relationships include content items that have been assigned the same semantic tags, content items that have been identified as related to the same topic, and content items that have the same modality.

[0023] Examples of relationships based on social graphs include content items that have been published or shared by the same user in the application system, or content items that have interacted through an n-degree connection of a specific user who has published or shared the content item, where n is a positive integer. Information about the relationships between digital content items in the application system can be stored in fields or columns of a database, and / or indicated by keys associated with digital content item data records. In a graph database, digital content items can be represented as nodes, and relationships between content items can be stored as edges.

[0024] As used herein, a modality can refer to a specific digital form of a content item. Examples of modalities include text, long text, short text, images, original images, video (recorded or streamed), audio (recorded or streamed), multimedia, hashtags, emojis, animated GIFs, and OCR (Optical Character Recognition) text. The language of a written or spoken content item can also be stored as a modality. Therefore, a digital content item can have more than one modality. For example, a post can include short text, images, OCR text extracted from an image, language associated with the short text, and language associated with the OCR text.

[0025] Content type and modal information for digital content items can be stored as attribute data associated with the content item. For example, content type and modality can be stored in one or more fields or columns of a database. In a graph database, digital content items can be represented as nodes, and content type and modal information can be stored at those nodes.

[0026] Existing systems have used binary classifiers with limited success in generating semantic labels for digital content items, such as spam labels. Semantic labels predicted by binary classifiers can be used to predict the likelihood of a digital content item triggering a content distribution strategy within a narrowly defined problem domain. For example, if a binary classifier has been trained on inflammatory articles written in English, it might predict the likelihood that a particular article written in English will be labeled as inflammatory, but it would not be able to make such a prediction if the classifier has been trained on financial fraud articles written in Italian.

[0027] More generally, a binary classifier can predict the probability that input in a specific language Y (e.g., English, French, Mandarin, German) with a specific content type and / or modality X (e.g., text comments, photo posts, text articles, image comments, etc.) will be assigned a semantic label that triggers a specific policy Z. For example, semantic labels can trigger content distribution policies that cause the application system to distribute content items with the semantic labels differently from other content items that are not labeled in the same way, whether by promoting or restricting their distribution.

[0028] Due to the nature of binary classifiers, the total number of binary classifiers required for an application system to process all possible combinations of content item types, modalities, languages, and policies that the application system will encounter will increase very rapidly, thus requiring the training and maintenance of up to X*Y*Z binary classifiers.

[0029] Maintaining such a large number of classifiers involves a significant amount of human time for development and maintenance. Binary classifiers require a large number of manually labeled training examples to generate reliable predictions. Therefore, a significant amount of human time is needed for data labeling and label review. The need to support multiple different languages ​​further demands human resources proficient in those languages.

[0030] Furthermore, the review criteria (e.g., policies) used by application systems to control or approve the distribution of digital content items are not static. In the case of binary classifiers, frequent policy changes render historically labeled datasets unsuitable for ongoing training iterations, leading to a shortage of training data. This necessitates training the binary classifier for newer policies or in response to policy changes.

[0031] Applications that combine the distribution of digital content items with asynchronous commenting / conversation capabilities enable users to add comments, review content, and reply to earlier comments on posts. However, known binary classifiers classify individual content items in isolation; that is, articles and comments about articles are classified independently by the same or different binary classifiers. Therefore, binary classifiers are prone to misclassifying comments and other content items that depend on other content items because they lack access to the context provided by the article post and / or other comments within the thread.

[0032] For example, a binary classifier trained on inflammatory text might inaccurately label a comment such as "Let's take a road trip" as "clear," even if the classifier had already labeled the original post "What do you think about letting a driver run over thugs?" as inflammatory. This error occurs because, when censorship is blocked, "Let's take a road trip" does not contain any inflammatory words or phrases. As another example, the different content items in a multimodal context can be classified by different binary classifiers trained to handle different individual modalities, producing poor, uncertain, or inaccurate labels.

[0033] Correspondingly, when used for spam detection and / or content moderation, binary classifiers may suffer from poor recall, especially in asynchronous comment / dialogue scenarios and in multimodal, multilingual, and multitasking situations.

[0034] As a result of the technical challenges and shortcomings described above in existing systems and other ongoing developments, application systems that distribute large amounts of user-generated content to a large number of users require a scalable way to reliably generate metadata that the application system can use to determine whether user-generated digital content items trigger spam filters, content moderation policies, or other applicable content distribution standards.

[0035] As the volume of user-generated content on the internet continues to grow and expand into different types of digital media and languages, the standards used for distributing digital content items online are also constantly changing and evolving. Application systems need digital content item annotation technologies that can adapt to new types of content items, new content modalities, and content distribution standards that change over time.

[0036] Various aspects of this disclosure provide a digital content annotation system that addresses the aforementioned and other shortcomings. For example, various aspects of this disclosure use machine learning models to model digital content items and the relationships between them into a multi-content-type, multi-modal, multilingual, multi-task content graph, thereby enabling many different types of content types, usages, and relationships between content items to be reflected in the content graph, the annotation model trained on the content graph, and the content tags generated by the annotation model.

[0037] As indicated by the multi-content-type specification used in this paper: the model architecture of the disclosed digital content annotation system can include a single ensemble machine learning-based annotation model that can take digital content items with many different content types as input and assign labels to content items with multiple different content types. This contrasts with existing systems, which require multiple distinct binary classifiers trained separately for different content types, rather than a single ensemble model capable of handling many different combinations of content types.

[0038] Multimodal indication: The model architecture of the disclosed digital content annotation system can include a single ensemble annotation model capable of assigning labels to content items with multiple different modalities. This contrasts with existing systems, which require separate binary classifiers trained for each modality.

[0039] Multilingual Indication: The machine learning model architecture of the disclosed digital content annotation system can include a single ensemble annotation model capable of assigning labels to content items with text written in multiple different languages. This contrasts with existing systems, which require separate binary classifiers trained for different languages.

[0040] Multi-task indication: The machine learning model architecture of the disclosed digital content labeling system can include a single labeling model capable of assigning labels to content items for different spam detection and / or content moderation tasks. As used herein, "task" can refer to a system control application that applies policies to content items. An example of a task is a system process that determines whether a content item triggers a specific policy, such as a spam policy or a content moderation policy, even if the policy is initially implemented for different content types, modalities, languages, or combinations of content types, modalities, and / or languages. This contrasts with existing systems that require separately trained binary classifiers for different spam detection and content moderation tasks.

[0041] In a broad sense, aspects of this disclosure include graph-based annotation models capable of labeling digital content items across content types, modalities, languages, and tasks / strategies. Thus, aspects of this disclosure provide a graph-based model architecture capable of generating multiple types of label predictions for unlabeled content items with various types, modalities, and languages. Embodiments of the graph-based model respond to inputs extracted from a content graph, where nodes correspond to digital content items in an application system. These inputs can include: node attribute data reflecting characteristics of the corresponding digital content items, such as modality, content type, language, and pre-existing semantic tags (if any); and edge data reflecting relationships between digital content items represented by nodes in the content graph.

[0042] The disclosed graph-based model architecture enables the propagation of label predictions across edge-related heterogeneous content types, modalities, and languages, thereby avoiding the need for separate classifiers for each combination of content type, modality, language, and task / policy. Furthermore, the disclosed graph-based approach requires significantly fewer labels for classifier training, reducing the burden of training and maintenance. Moreover, the disclosed techniques enable digital content item labeling systems to improve recall and adapt to new content types, modalities, languages, and policies, and handle policy variations even with only a small number of training samples available.

[0043] Figure 1 An exemplary computing system 100 including a graph-based annotation system 150 is illustrated. Figure 1 In one embodiment, the computing system 100 includes a user system 110, a network 120, an application system 130, a digital storage system 140, and a graph-based annotation system 150.

[0044] User system 110 includes at least one computing device, such as a personal computing device, server, mobile computing device, or smart appliance. User system 110 includes at least one software application installed on the computing device or accessible via a network, including user interface 112. For example, user interface 112 may be or include the front end of application system 130, which may be implemented as a native application on the computing device or a web application launched in a web browser.

[0045] User interface 112 is any type of user interface described above. User interface 112 can be used to input, upload, or share digital content items, and / or view or otherwise perceive digital content items distributed by application system 130. For example, user interface 112 can include a graphical user interface, a haptic interface, and / or a conversational voice / conversation interface, which includes one or more mechanisms for viewing and manipulating digital content items.

[0046] Application system 130 is any type of application software system that includes or utilizes the functionality provided by graph-based annotation system 150. Examples of application system 130 include, but are not limited to: network-connected software (such as professional and / or general social media platforms) and systems based on or not based on network-connected software, such as digital content distribution services, general search engines, job search software, recruitment search software, sales assistance software, advertising software, learning and education software, or any combination of the foregoing. Figure 2A An exemplary embodiment of the application system 130 is shown below.

[0047] Data storage system 140 includes a data repository and / or data service that stores digital content items, data received, used, manipulated, and generated by application system 130, and data received, used, manipulated, and generated by graph-based annotation system 150. Data storage system 140 may include distributed data services. As used herein, a data service may refer to a physical geographic group of machines, a logical group of machines, or a single machine. For example, a data service may be a data center, a cluster, a set of clusters, or machines.

[0048] The data storage system 140's data repository can include one or more offline repositories and / or one or more nearline repositories. The data repository can be implemented using databases such as key-value repositories, relational databases, and / or graph databases. The nearline data repository can store real-time event streams, such as Kafka event streams. Data can be written to and read from the data repository using query techniques (e.g., SQL or NoSQL).

[0049] Graph databases use graph data structures that include several interconnected graph primitives to organize data. Examples of graph primitives include nodes, edges, and predicates, where nodes store data, edges establish relationships between two nodes, and predicates are semantic labels assigned to edges that define or describe the type of relationship that exists between the nodes connected by the edges.

[0050] Nodes in a graph database can correspond to tables, subsets of tables, or sets of rows and columns from different tables in a non-graph database. Another example of graph primitives is a specific part of a graph or subgraph, such as a particular pair of nodes plus an edge connecting those nodes. Graph query languages ​​can be used to construct and execute queries on graph databases, and to build, update, and manipulate components of graph databases. In the case of graph databases, query results include subsets of the graph stored in the graph database. For example, data records can be transformed from non-graph formats into graphs using graph nodes, edges, and edge labels (also known as predicates).

[0051] Data stored in the non-graph database portion of data storage system 140 can be transformed or converted into a graph format that can be stored in a graph database and queried using various types of graph queries (such as edge queries and path queries). An example of an edge query is a query to retrieve the number of first-degree connections between nodes. An example of a path query is a query to determine how many edges exist between two nodes, for example, to determine the degree of connectivity separating two nodes. Graph queries can be implemented using graph query languages ​​(such as datalog).

[0052] To transform data stored according to a non-graph data structure into a graph format, graph databases employ a set of graph primitives. Examples of graph primitives include rules based on edges in the graph (which express the non-graph database schemas) and composite keys specifying relationships between nodes, edges, and predicates in the graph corresponding to the non-graph data structure. For example, the graph primitives can include rules expressing relation schemas in the non-graph data structure. Rules based on graph-based entity-to-entity relationships a to b and b to c can link a, b, and c in the non-graph schema. As another example, composite keys can be used to express relationships between tables and attributes in a non-graph data structure, where attributes correspond to predicates in the graph schema and tables correspond to nodes in the graph schema.

[0053] Data storage system 140 can reside on at least one persistent and / or volatile storage device, which can reside on the same local network as at least one other device of computing system 100, and / or the storage device can reside in a network located remotely relative to at least one other device of computing system 100. Therefore, although depicted as being included in computing system 100, portions of data storage system 140 can be part of computing system 100, or can be accessed by computing system 100 via a network (such as network 120).

[0054] The graph-based annotation system 150 is capable of generating semantic tags for unstructured data, including but not limited to user-generated digital content items, such as spam tags. In some embodiments, the application system 130 includes at least a portion of the graph-based annotation system 150. (As in...) Figure 6 As shown, the graph-based annotation system 150 can be implemented as instructions stored in memory, and the processing device 602 can be configured to execute the instructions stored in memory to perform the operations described herein.

[0055] The disclosed technology can be described with reference to exemplary use cases of labeling user-generated digital content items for spam detection and / or content moderation policy violations. For example, the disclosed technology can be used to automatically apply spam labels to digital content items that are entered, uploaded, shared, or otherwise distributed on social graph applications (such as professional social networks or other social media).

[0056] The disclosed technologies are not limited to social graph applications, but can be used more generally to annotate unstructured digital content items. The disclosed technologies can be used in many different types of web-based applications that wish to monitor and / or audit the proliferation of unstructured and / or unannotated digital content items. For example, references will be made below. Figure 2A and Figure 2B Additional description of the graph-based annotation system 150 is provided.

[0057] Any one of user system 110, application system 130, data storage system 140, and graph-based annotation system 150 includes an interface embodied as computer programming code stored in computer memory, which, when run, enables a computing device to perform bidirectional communication with any other of user system 110, application system 130, data storage system 140, and graph-based annotation system 150 using communication coupling mechanisms 101, 103, 105, and 107. Examples of communication coupling mechanisms include network interfaces, inter-process communication (IPC) interfaces, and application interfaces (APIs).

[0058] The client portion of application system 130 can operate within user system 110, for example, as a plugin or desktop applet in the graphical user interface of a software application, or as a web browser executing user interface 112. In an embodiment, the web browser is capable of transmitting HTTP requests over a network (e.g., the Internet) in response to user input received through the user interface provided by the web application and displayed through the web browser. The server running application system 130 and / or the server portion of application system 130 is capable of receiving the input, performing at least one operation using the input, and returning output using an HTTP response, which the web browser receives and processes.

[0059] Other technologies that can be used to implement communication of data and instructions between user system 110 and any of user system 110, application system 130, data storage system 140, and graph-based annotation system 150 include: application programming interfaces (APIs), such as REST (Representative State Transfer) API and SOAP (Simple Object Access Protocol); scripting languages, such as JavaScript; markup languages, such as XML (Extensible Markup Language) and JSON (JavaScript Object Markup); and AJAX (Asynchronous JavaScript and XML).

[0060] Each of the user system 110, application system 130, data storage system 140, and graph-based annotation system 150 is implemented using at least one computing device that is communicatively coupled to an electronic communication network 120 via communication coupling mechanisms 101, 103, 105, and 107. Any one of the user system 110, application system 130, data storage system 140, and graph-based annotation system 150 can be bidirectionally communicatively coupled through network 120. User system 110, as well as one or more different user systems (not shown), can be bidirectionally communicatively coupled to application system 130.

[0061] A typical user of user system 110 can be an administrator or end user of application system 130 and / or graph-based annotation system 150. The administrator or end user can be a person or a computer program (such as a web bot) designed to mimic human use of application system 130. User system 110 is configured to communicate bidirectionally with any one of application system 130, data storage system 140, and graph-based annotation system 150 via network 120 using communication coupling mechanism 101. User system 110 has at least one address that identifies user system 110 to network 120 and / or application system 130; for example, an IP (Internet Protocol) address, device identifier, MAC (Media Access Control) address, session identifier, user account identifier, or any combination of the foregoing.

[0062] The features and functions of user system 110, application system 130, data storage system 140, and graph-based annotation system 150 are implemented using computer software, hardware, or a combination of software and hardware, and can include combinations of automatic functions, data structures, and digital data schematically shown in the accompanying drawings. For ease of discussion, user system 110, application system 130, data storage system 140, and graph-based annotation system 150 are... Figure 1 The illustrated system, services, and data repositories (or their functions) are shown as separate elements, but this illustration is not intended to imply that these elements must be separate. The illustrated system, services, and data repositories (or their functions) can be partitioned across any number of physical systems (including a single physical computer system), and they can communicate with each other in any suitable manner.

[0063] Network 120 can be implemented based on any medium or mechanism that facilitates the exchange of data, signals, and / or instructions between various components of computing system 100. For example, data and instructions can be represented as signals, where each signal comprises a series of bits, and the bit values ​​correspond to specified charge levels that can traverse network 120 and be received and processed by devices on network 120. Examples of network 120 include, but are not limited to: local area network (LAN), wide area network (WAN), Ethernet network, the Internet, at least one terrestrial, satellite, or wireless link, or any combination of any number of different network and / or communication links.

[0064] Figure 2A This is an exemplary embodiment of an application system 130 including a graph-based annotation system 150, according to some embodiments of this disclosure. Figure 2A In the application system 130, there are functions 202, digital content item processor 204, entity data 206, activity data 208, content item data 210, social graph 212, and graph-based annotation system 150.

[0065] Function 202 generally includes user-facing and backend functions that enable data manipulation and communication between the user and application system 130. Entities in application system 130 are logical constructs linked to the address of physical user system 110. User system 110 can be associated with more than one entity in application system 130. For example, physical user system 110 can be associated with multiple different logical account identifiers. Examples of entity types include user, company, organization, job, and content item.

[0066] The data manipulation and communication performed by the user system 110 in the application system 130 can be described with reference to entities associated with the user system 110. The user-oriented functions of the application system 130 include functions exposed to users of the application system 130 through a user interface. These user-oriented functions include source entity functions 214 and destination entity functions 216. A source entity can refer to an entity that generates content items or otherwise provides content items to the application system 130. Therefore, source entity functions 214 can include, for example, user interface features and functions that enable users to view, create, upload, share, forward, reply to, and save digital content items in the application system 130, to view, add, edit, and delete comments and reply to comments about digital content items, and / or to view, send, and receive messages including digital content items.

[0067] Examples of digital content items include posts, comments, and replies to comments. Examples of posts include articles and / or links to articles, such as URLs (Uniform Resource Locators). Posts can include text and / or multimedia content items, such as images, videos, and audio, whether previously recorded or streamed. Comments can refer to any type of content item linked to a post, such as comments and replies to comments. Comments can include text and multimedia content items, such as images, videos, and audio, whether previously recorded or streamed. Text can have any length and can include metadata such as hashtags and URLs.

[0068] A destination entity can refer to an entity that is the recipient of a digital content item initially generated by a source entity in application system 130. Therefore, destination entity functionality 216 can include, for example, user interface features and functions that allow a user to view and manipulate digital content items received by the destination entity in search results, feeds, recommendations, notifications, or messages generated by application system 130. A destination entity can also be a source entity. For example, a destination entity can share a digital content item with other entities after viewing it in a feed. A source entity can also be a destination entity. For example, a source entity can receive (e.g., by clicking a push notification) comments made by another entity about a post initially published by the source entity.

[0069] In application entity 130, source entity function 214 and destination entity function 216 are implemented via Internet communication technology. For example, source entity function 214, which enables the sharing of digital content items in application system 130, includes sending and receiving network messages between the user system sharing the digital content items and application system 130. Destination entity function 216, which enables the searching, viewing, and manipulating of digital content items in application system 130, includes sending and receiving network messages between the user system viewing and / or manipulating the digital content items and application system 130. In some contexts, network messages can refer to requests. Similarly, source entity function 214 and destination entity function 216 can be asynchronous.

[0070] The back-end portion of function 202 includes computer operations that support the front-end functions, such as data manipulation and communication. The back-end functions can include: reading data such as entity data 206, activity data 208, content item data 210, and / or social graph 212 from a data repository; performing calculations and / or transformations on the data; and writing the data and / or the results of the calculations and / or transformations to the data repository.

[0071] The backend functionality may include computer execution of algorithms that determine the extent to which digital content items can spread within application system 130. The backend functionality may include performing queries against one or more data repositories. For example, the backend functionality may include performing queries to determine the nth degree connection (where n is a positive integer) of entities in social graph 212 and queries against one or more data repositories to retrieve search results. The backend functionality may include executing algorithms to select and retrieve digital content items used to populate search results, feeds, notifications, messages, push notifications, or recommendations. The backend functionality may include performing edge queries and / or path queries against a graph database. The backend functionality may include transforming or converting data stored in a non-graph data structure into a graph.

[0072] Algorithms executed as part of the backend functionality can include, for example, rule engines, heuristics, and / or machine learning algorithms trained using one or more training datasets. The backend functionality can be executed by a server computer or a server network, and / or portions of the backend functionality can be implemented on a client device (e.g., user system 110). The frontend functionality can be executed by a client device (e.g., user system 110), and / or portions of the frontend functionality can be implemented on a server computer or a server network.

[0073] Entity data 206 is a data repository in data storage system 140 that stores entity data records. Entity data records include entity identifiers. Entity data records can include or relate to entity type data and / or entity profile data. Entity type data indicates the type of entity associated with the entity identifier, such as an individual, organization, job, or content. Entity profile data includes attribute data associated with the entity identifier, such as name, location, or title. Entity data 206 can be stored in a non-graph database and can also be transformed into a graph stored in a graph database. For example, portions of the entity data (such as attribute data) can be stored in tables in a non-graph database and nodes in a graph database.

[0074] Activity data 208 is a data repository in data storage system 140 that stores activity data records. Activity data records include activity identifiers, entity identifiers, and data / timestamps. Activity data records can include or relate to activity type data. Activity type data indicates the type of activity associated with the entity identifier and the front-end function. Examples of activity type data include views, clicks, comments, shares, likes, and follows. Activity data 208 can be captured as a real-time or near-real-time data stream and stored, for example, in the near-line portion of data storage system 140.

[0075] Content item data 210 is a data repository in data storage system 140 that stores digital content item data records. Digital content item data records can include content item identifiers, entity identifiers, and data / timestamps. Content item data records can include or relate to content type data. Content type data indicates the content type and / or modality associated with the content item identifier. Examples of content type and / or modality data include text, video, image, audio, stream, article, comment, and metadata. Content item data 210 can include or relate to the content item identifier of another digital content item in application system 130. Content item data 210 can include tag data generated by graph-based annotation system 150.

[0076] Content item data 210 can be stored in the non-graph database portion of the data storage system 140. Using graph primitives, portions of the content item data 210 can be transformed into a graph and stored in the graph database portion of the data storage system 140. For example, portions of the content item data, such as attribute data including content type data, modal data, language data, tag data, and relationship data between content items, can be stored in tables in the non-graph database and / or nodes and edges in the graph database. The content graph generated by the graph-based annotation system 150 can be stored in the graph database portion of the content item data 210.

[0077] Social graph 212 is a data repository for storing entities and relationships between entities in data storage system 140, represented by a graph-based representation. In social graph 212, entities are represented as nodes, and relationships between entities are represented as edges. Nodes contain node identifiers and node type data. Node type data can correspond to entity types. Edges contain node identifier data for the two nodes connected by the edge, as well as edge types. Edge types correspond to different types of relationships between entities. Examples of relationships between entities include contact, like, follow, share, and comment. An edge can connect two different node types. For example, an edge can represent a relationship between an individual entity and a digital content item, or between an individual entity and an organization, or between an organization and a job. Social graph 212 can include content graphs generated by graph-based annotation system 150.

[0078] Entity data 206, activity data 208, content item data 210, and social graph B12 are each bidirectionally digitally communicated with function 202 via communication couplers 220, 222, 224, 226, and 228, respectively.

[0079] The digital content item processor 204 communicates bidirectionally with the function 202 via communication coupling 218. The function 202 can issue calls or send data and / or instructions to the digital content item processor 204 via communication coupling 218. The digital content item processor 204 can issue calls or send data and / or instructions to the function 202 via communication coupling 218.

[0080] For example, in response to function 202 receiving or processing a digital content item identified by a digital content item identifier in application system 130, function 202 can issue a call to digital content item processor 204 containing the digital content item identifier as a variable or parameter. In response to the call from function 202, digital content item processor 204 can return to function 202 a semantic tag, such as a spam tag, for the digital content item corresponding to the digital content item identifier included in the call. Digital content item processor 204 can include, as described below, as in... Figure 2B Other functions shown.

[0081] The digital content item processor 204 engages in bidirectional digital communication with the graph-based annotation system 150 via communication coupling 228. The digital content item processor 204 can issue calls or send data and / or instructions to the graph-based annotation system 150 via communication coupling 228. The graph-based annotation system 150 can issue calls or send data and / or instructions to the digital content item processor 204 via communication coupling 228. For example, in response to a call from function 202, the digital content item processor 204 can issue a call to the graph-based annotation system 150 via communication coupling 228 containing a digital content identifier as a variable or parameter. In response to the call from the digital content item processor 204, the graph-based annotation system 150 can return a semantic label to the digital content item processor 204 for the digital content item corresponding to the digital content item identifier contained in the call. The graph-based annotation system 150 can include features described below, such as... Figure 2B The functions shown are as follows.

[0082] Figure 2B The illustration shows an exemplary flow 244 of operation of an embodiment of a digital content item processor 204 communicating with a graph-based annotation system 150 according to some embodiments of the present disclosure. Parts of flow 244 can be implemented as online operations, while other parts of flow 244 can be implemented as offline or near-line operations. For example, operations performed by the digital content item processor 204 and the annotator 252 can be implemented as online operations, while operations performed by the graph generator 254, the embedder 256, and / or the model trainer 258 can be implemented as offline operations.

[0083] exist Figure 2B In the example, the digital content item processor 204 receives incoming digital communication from user system 110 (user 1) via communication coupling 230, and / or receives incoming digital communication from function 202 via communication coupling 232. In response to the incoming digital communication, the digital content item processor 204 sends outgoing digital communication to zero or more user systems 110 (users 0...N, where N is a positive integer) and / or function 202 via communication coupling 242.

[0084] Examples of digital communications that the digital content item processor 204 can receive from the user system 110 (user 1) are signals containing digital content items, digital content item identifiers, digital content item identifiers and entity identifiers, or commands containing digital content item identifiers and / or entity identifiers. Examples of commands that the digital content item processor 204 can receive from the user system 110 (user 1) include uploading, publishing, commenting, and sharing.

[0085] Examples of digital communications that the digital content item processor 204 can receive from function 202 include digital content items, digital content item identifiers, digital content item identifiers and entity identifiers, or calls that include digital content item identifiers and / or entity identifiers. Examples of functions 202 that can issue calls to the digital content item processor 204 include feed services, search engines, and recommender systems.

[0086] For example, before the feed service includes the digital content item in the user's feed, it can submit the digital content item to the digital content item processor 204. For example, the feed service can use tags generated by the graph-based annotation system 150 to determine whether to include the digital content item in the user's feed.

[0087] Before including digital content items in a user's search results, the search engine can submit the digital content items to the digital content item processor 204. For example, the search engine can use tags generated by a graph-based labeling system 150 to determine whether to include the digital content items in the search results.

[0088] Before the recommender system sends a recommendation notification including the digital content item to the user, the recommender system can submit the digital content item to the digital content item processor 204. For example, a search engine can use tags generated by a graph-based labeling system 150 to determine whether to include the digital content item in the recommendation notification.

[0089] Examples of outgoing digital communications that the digital content item processor 204 can send via communication coupling 242 to zero or more user systems 110 include network messages containing or relating to digital content items that will be included in feeds, search results, or recommendation notifications of one or more user systems 110.

[0090] exist Figure 2B In the example, the digital content item processor 204 includes an evaluator 234 and a distributor 240. The evaluator 234 examines communications received from the user system 110 and / or function 202 to obtain semantic tags and / or other metadata that can indicate that the digital content item identified in the communications has contained semantic tags that can be used to control the distribution of the digital content item in the application system 130.

[0091] If evaluator 234 determines that the digital content item identified in the communication already contains a semantic tag that can be used to control the distribution of the digital content item in application system 130, and the tag prediction model 260 is in prediction mode, then evaluator 234 forwards the communication to distributor 240 via communication coupling 238. If evaluator 234 determines that the digital content item identified in the communication already contains a semantic tag that can be used to control the distribution of the digital content item in application system 130, and the tag prediction model 260 is in training mode, then evaluator 234 forwards the communication to graph-based labeling system 150 via bidirectional communication coupling 236. To determine whether the tag prediction model 260 is in training mode or prediction mode, evaluator 234 can check the value of a flag set for the tag prediction model 260 by, for example, model trainer 258.

[0092] If the evaluator 234 determines that the digital content item identified in the communication does not contain a semantic tag that can be used to control the distribution of the digital content item in the application system 130, then the evaluator 234 forwards the communication to the graph-based annotation system 150 via bidirectional communication coupling 236.

[0093] In response to a communication received from evaluator 234, graph-based annotation system 150 generates semantic tags that can be used to control the distribution of digital content items identified in the communication within application system 130. Graph-based annotation system 150 returns the generated tags to evaluator 234. In response to receiving tags from graph-based annotation system 150, evaluator 234 forwards the received tags to distributor 240.

[0094] In response to receiving a tag from evaluator 234, distributor 240 compares the tag with a stored policy or a set of policies (such as a set of spam policies and / or content moderation policies). To this end, distributor 240 can determine whether the tag matches a tag associated with a predetermined policy or any policy in the set of policies. If the tag matches a policy, then distributor 240 iterates through the policies that match the tag to determine the content distribution instruction associated with the policy having the tag. If the tag does not match a policy, then distributor 240 distributes the digital content item to up to N user systems within application system 130 via communication coupling 242, where N is a positive integer.

[0095] If the tag matches the policy, the distributor 240 executes the content distribution instruction associated with the tag through the matched policy. Thus, the distributor 240 distributes the digital content item associated with the tag to zero or fewer than N user systems 110 (users 0…N) within the application system 130 according to the content distribution instruction.

[0096] If the content distribution instruction indicates that the distribution of the digital content item is unrestricted based on the tag, then the distributor 240 can distribute the digital content item to more than zero and up to N user systems. If the content distribution instruction indicates that the distribution of the digital content item is restricted, then the distributor 240 can distribute the digital content item to zero or fewer than N user systems.

[0097] The strategy can be defined relative to the social graph 212. For example, the strategy can include content distribution instructions that restrict the distribution of digital content items tagged "personal" to the first-degree connections of users associated with the digital content item, or restrict the distribution of digital content items tagged "political" to users who follow entities that generate political posts. As another example, the strategy can include content distribution instructions that route digital content items labeled "unsuitable for work" or "fraud" to another destination, such as user system 110 that has been designated as a honeypot, spam trap, or quarantine zone.

[0098] Examples of strategies include user-spam policies (e.g., unsubscribe, unfollow, subscribe, follow) and content moderation policies. Strategies can be derived based on entity data 206, activity data 208, and / or social graph 212. For example, a strategy can include content distribution instructions that restrict the distribution of digital content items labeled “machine learning” to entities with the job title “software engineer,” or restrict the distribution of digital content items labeled “baseball” to entities that have previously viewed or liked baseball-related digital content items.

[0099] The strategy and associated content distribution directives can be implemented, for example, using Boolean logic or lookup operations (which iterate through data structures that map tags to content distribution directives).

[0100] exist Figure 2B In the example, the graph-based annotation system 150 includes an annotator 252, a graph generator 254, an embedder 256, a model trainer 258, and a label prediction model 260. The graph generator 254, the embedder 256, the model trainer 258, and the label prediction model 260 are communicatively coupled to the annotator 252 via bidirectional communication couplings 262, 264, 266, and 268, respectively.

[0101] The annotator 252 communicates bidirectionally with the evaluator 234. For example, the annotator 252 receives digital content item communication from the evaluator 234, obtains the input required for the label prediction model 260 to generate label predictions, and provides the labels predicted by the label prediction model 260 in response to the received digital content items to the evaluator 234. As an alternative implementation, the label prediction model 260 can directly communicate the label prediction data to the evaluator 234. In this case, the annotator 252 assembles the input required for the label prediction model 260 to generate label predictions, but does not communicate the label predictions to the evaluator 234.

[0102] Upon receiving digital content item communication from evaluator 234, annotator 252 obtains the set of inputs for label prediction model 260 to generate label prediction outputs. This set of inputs includes a content graph generated by grapher 254 for the digital content item and an embedding generated by embedder 256. If label prediction model 260 is in training mode, this set of inputs also includes labels determined by model trainer 258.

[0103] The annotator 252 receives digital communications identifying digital content items and associated entity identifiers from the evaluator 234. In response to receiving the communications identifying digital content items, the annotator 252 forwards the digital content item identifier data to the mapper 254 via communication coupling 262 and to the embedder 256 via communication coupling 264. If the label prediction model 260 is in training mode, the annotator 252 also forwards the digital content identifier data to the model trainer 258.

[0104] Several alternatives exist for implementing the grapher 254. In one implementation, an offline process is run periodically to generate a complete content graph or a set of content graphs representing all content items in the application system. Then, when the grapher 254 receives a digital content item identifier from the annotator 252, the grapher 254 extracts a subgraph of the larger, previously created and stored content graph, wherein the subgraph is specific to the digital content item identifier received from the annotator 252. In this example, the operations that the grapher 254 needs to perform upon receiving communication from the annotator 252 may include a simple set of queries on the previously created content graph.

[0105] In another embodiment, the graph generator 254 generates a content graph "on the fly" upon receiving a digital content item identifier from the annotator 252. In this example, the graph generator 254 queries a non-graph database to obtain data records of digital content items identified by the digital content item identifier received from the annotator 252, extracts attribute and relation data from the non-graph data records, and uses graph primitives to transform the extracted data from a non-graph form into a graph. In this example, the graph generator 254 creates content graphs specific to content items as needed.

[0106] In any case, the content graph created by the graph generator 254 in response to communication from the annotator 252 can be a relatively small graph, which includes the nodes for the requested content items as well as adjacent nodes and edges. In other words, for the purpose of responding to communication from the annotator 252, the graph generator 254 may not need to generate a complete content graph of all content items in the application system.

[0107] Therefore, in response to receiving digital content item identifier data from annotator 252, graph generator 254 generates a content graph for the digital content item. To this end, graph generator 254 can locate the node corresponding to the digital content item in an existing content graph stored, for example, in content item data 210, and extract a subgraph from the content graph containing the node corresponding to the digital content item. For example, the subgraph generated by graph generator 254 can include the node corresponding to the digital content item identified in the digital communication from evaluator 234, as well as adjacent nodes (e.g., nodes that have an edge to or "jump away from" the node corresponding to the digital content item). Graph generator 254 can extract data from the content graph, for example, by issuing a query against a graph database storing the content graph and retrieving the query results from the graph database.

[0108] Alternatively, in some implementations, the content item data is stored in a non-graph data structure, such as a key-value store or a relational database (e.g., the content item data 210). In these implementations, the graph generator 254 is capable of issuing queries to extract relational data for the digital content items from the data structure, where, for example, the relational data may be determined based on key values. Using, for example, graph primitives, the graph generator 254 transforms the non-graph node relational data for the digital content items into a content graph, in which the digital content items are represented as nodes, related digital content items are represented as adjacent nodes, and the edges between the digital content items and adjacent nodes are labeled using relation type data.

[0109] Examples of relational data types between content items or nodes include time dependencies, such as article-comment, comment-reply, and comment-comment. Other examples of relational data types include semantic relationships and social graph-based relationships. For example, when content items have the same content type, modality, language, or tag, graph generator 254 can create edges between corresponding nodes.

[0110] As another example, grapher 254 is capable of creating edges between nodes representing heterogeneous content items. For instance, if there is a dependency between the comment and the text article (e.g., the comment was submitted in a thread associated with the article), grapher 254 can create an edge between the text article and the image comment. To further illustrate, grapher 254 can create an edge, for example, between a video comment in a thread associated with a post and another post containing a video, even if there is no dependency between the video comment and the video post, provided that grapher 254 determines there is some other reason for creating an edge between the video comment and the video post; for example, both the video comment and the video post depict a dancing elephant.

[0111] The digital content items selected by graph generator 254 for creating a content graph, or the content subgraph selected by graph generator 254, may include, as appropriate, some digital content items that have already been labeled with semantic tags (which distributor 240 can use to control the distribution of digital content items within application system 130). Content items may have been previously labeled through, for example, previous operations using a graph-based labeling system, through a pre-existing binary classifier, or through a previous manual labeling process. However, in many use cases, only a small number of content items have been labeled. If a content item has been previously labeled, then the indication of the label for that content item can be stored in the attributes of the node.

[0112] Examples of situations where a small number of labeled content items may exist include, for instance, after a policy change or the introduction of a new type of content item (e.g., detecting a new value for a content type attribute or modality attribute—for example, detecting "podcast" as a modality when it was previously considered a modality on other content items). Therefore, the content graph generated by grapher 254 tends to have very few labels because the number of unlabeled nodes is greater than the number of labeled nodes relative to a specific task / policy. For example, the ratio of labeled to unlabeled nodes could be 1:M, where M is a positive integer greater than 1 by X, where X can be a real number representing an offset or a multiple.

[0113] As an example, suppose a new spam policy is implemented in the application system. To initialize the new spam policy, a relatively small number of content items are manually identified as triggering the new spam policy and manually labeled with new spam tags. The graph-based labeling system 150 uses this initial set of verified / basic fact labels for the new spam policy and the content graph generated by the graph generator 254 to determine whether to apply the new spam tags to other unlabeled content items in the application system.

[0114] For example, when mapper 254 receives communication from labeler 252 indicating that an unlabeled content item has been identified, the graph created by mapper 254 for the identified unlabeled content item may or may not contain nodes representing any content item among those that have already been manually labeled with the new spam tag. This set of inputs to the tag prediction model 260, as described below, includes the graph data generated by mapper 254 for the identified unlabeled content item. Therefore, the presence or absence of any labeled nodes in the graph data for the unlabeled content item is reflected in the input to the tag prediction model 260, and thus in the tag prediction output by the tag prediction model 260 for that particular unlabeled content item.

[0115] The content graph generated by the graph generator 254 can include nodes corresponding to heterogeneous digital content items. Heterogeneous digital content items can refer to two or more digital content items with different content types, modalities, and / or languages. For example, articles and comments can be heterogeneous because the text in a comment is typically much shorter than that of an article. Images and text can be heterogeneous because image data (e.g., pixel data) differs from alphanumeric text. Two different text items can be heterogeneous if they are written in different languages.

[0116] The content graph generated by graph generator 254 can contain edges that express dependencies or other types of relationships between heterogeneous content types. For example, a pair of nodes can represent a text article and an image comment, respectively, and the edge between the nodes of the pair can indicate the dependency relationship between the text article and the image; for example, the image is a comment on the text article.

[0117] The graph generator 254 can determine dependencies by, for example, comparing the date / timestamps of digital content items, such that digital content items with later timestamps can have dependencies on digital content items with earlier timestamps. Alternatively or additionally, when digital content items are stored in a non-graph data structure and are therefore available for extraction by the graph generator 254 from the non-graph data structure, dependencies (such as post-comment and comment-reply) can be made explicit.

[0118] The plotter 254 returns the generated content map to the labeler 252 or, alternatively, directly to the label prediction model 260 or the model trainer 258.

[0119] In response to receiving digital content item identifier data from annotator 252, embedder 256 generates an embedding for the identified digital content item. The embedding can refer to a semantic representation of the digital content item generated by comparing the values ​​of its features with the values ​​of corresponding features of other digital content items. The embedding can be implemented as a vector, wherein each dimension of the vector contains values ​​corresponding to different features of the digital content item that facilitate the determination of its semantic meaning. The features of the digital content item can include raw features extracted from the digital content item, such as pixels and n-grams, and / or computed features, such as counts, probabilities, averages, and the mean of the raw features.

[0120] To generate embeddings, embedder 256 can apply known embedding techniques, such as BERT (Bidirectional Encoder Representation from Transformer), Inception, or NetVLAD (Local Aggregate Descriptor Vector Neural Network), to one or more portions of the digital content item to generate the embeddings. Alternatively or additionally, embedder 256 can apply a universal language model (ULM), such as TULR-v3 (Turing Universal Language Representation), to the digital content item to generate embeddings, particularly when the digital content item is a multimodal and / or multilingual content item. Alternatively or additionally, embedder 256 can apply the following reference to the digital content item... Figure 4C or Figure 4D One or more of the described embedding techniques are used to generate the embedding. Embedder 256 returns the generated embedding to labeler 252 or, alternatively, directly to label prediction model 260 or model trainer 258.

[0121] When the label prediction model 260 is in training mode, the annotator 252 invokes the model trainer 258. In training mode, in response to receiving a digital content item identifier from the annotator 252, the model trainer 258 obtains a known semantic label for the digital content item (e.g., the digital content item has been previously labeled during an earlier training round or by another annotation system), obtains an embedding for the digital content item from the embedder 256, and obtains a content graph for the digital content item from the grapher 254. Collectively, the known labels for the digital content item, the embeddings for the digital content item, and the content graph for the digital content item constitute training data instances for the untrained or partially trained label prediction model 260.

[0122] To obtain known labels for training data, model trainer 258 can, for example, query content item data 210 stored in a non-graph database or traverse an existing content graph to obtain previously labeled nodes (in the case of a previously created and stored content graph). Model trainer 258 applies an untrained or partially trained label prediction model 260 to the training data to produce a trained version of label prediction model 260.

[0123] Applying a model to training data can involve executing a machine learning algorithm on a specified set of inputs, including the training data, to generate a set of outputs corresponding to that set of inputs. Therefore, an untrained model can refer to a machine learning algorithm or a set of machine learning algorithms implemented via computer code that are independent of any dataset.

[0124] On the other hand, a trained model can refer to the relationship between the inputs to those groups and the outputs produced by applying the machine learning algorithm to those inputs, and this relationship will be reflected in the values ​​of the machine learning algorithm parameters. That is, applying the machine learning algorithm to the training data will iteratively adjust the values ​​of the machine learning parameters until parameter values ​​that will produce statistically reliable outputs (e.g., predictions or classifications) are found. A loss function can be used to calculate model error (e.g., a comparison of model-generated values ​​with verified or baseline fact values) to determine whether the model is producing reliable outputs or whether parameter values ​​need to be adjusted. The choice of machine learning algorithm, loss function, and associated parameter values ​​can depend on the requirements of the specific application system; for example, the type of output desired and the nature of the inputs.

[0125] Machine learning algorithms can refer to a single algorithm applied to a single set of inputs, multiple iterations of the same algorithm for different inputs, or a combination of different algorithms applied to different inputs. For example, in a neural network, a node corresponds to an algorithm executed for one or more inputs to that node to produce one or more outputs. If each node in a set of nodes executes the same algorithm for different inputs from the same set of inputs, then that set of nodes can be called a layer of the neural network. The outputs of a neural network layer can constitute the input to another layer of the neural network. A neural network can include an input layer that receives and manipulates one or more raw inputs and transmits the outputs to one or more hidden layers, and an output layer that receives and manipulates the outputs produced by the one or more hidden layers to produce the final output.

[0126] The label prediction model 260 is a machine learning model with a training mode and a prediction mode. In the training mode, as described above, the model trainer 258 applies training data to the untrained or partially trained label prediction model 260. Partial training can refer to a model trained using only a small training dataset, or it can refer to a model that has been trained using training data but the training data is now outdated or needs to be refreshed or updated.

[0127] In prediction mode, label prediction model 260 generates label predictions in response to sets of inputs, including a content map generated by mapper 254 for a given digital content item and an embedding generated by embedder 256. Labeler 252 forwards the label predictions generated by label prediction model 260 to evaluator 234 via communication coupling 236, or alternatively directly to distributor 240 via a different communication coupling.

[0128] The following will refer to Figure 4A as well as Figure 4B1 , Figure 4B2 and Figure 4B3 An exemplary embodiment of the label prediction model 260 is described.

[0129] Figure 3 This is a flowchart of an exemplary method 300 for annotating digital content items using a processing device according to some embodiments of the present disclosure. Method 300 can be performed by processing logic that can include hardware (e.g., processing device, circuit system, dedicated logic, programmable logic, microcode, device hardware, integrated circuit, etc.), software (e.g., instructions that run or execute on the processing device), or a combination thereof. In some embodiments, method 300 is performed by… Figure 1 The graph-based annotation system 150 is used to perform this.

[0130] Although illustrated in a specific sequence or order, the order of processes may be modified unless otherwise indicated. Therefore, the illustrated embodiments should be understood as examples only, and the illustrated processes may be executed in different orders, and some processes may be executed in parallel. Furthermore, one or more processes may be omitted in various embodiments. Therefore, not all processes are required in every embodiment. Other process flows are also possible.

[0131] At operation 302, the processing device determines a sparsely labeled set of digital content items that can be distributed by an application system to multiple user systems over a network based on a social graph, wherein the sparsely labeled set of digital content items includes unlabeled digital content items. Operation 302 can be performed by the processing device querying a content item database to obtain a set of heterogeneous digital content items, wherein each of the heterogeneous digital content items has at least one type of relationship with the unlabeled digital content items.

[0132] At method 300, process flows 302, 304, and 310 can be executed in parallel with and / or independently of process flows 302, 306, 308, and 310. That is, the process of determining embedded data can be completed without a content graph.

[0133] At operation 304, the processing device determines the embedded data for the unlabeled digital content item. To this end, operation 304 can include the above reference. Figure 2B This describes part of the functionality of the embedder 256. Operation 304 is capable of applying a cross-modal model to the content item to produce a cross-modal embedding. (As described below...) Figure 4C and Figure 4D An example is shown that can be used by operation 304 to generate a cross-modal model with cross-modal embedding.

[0134] Alternatively, operation 304 can apply an entity activity model to the data associated with entities represented in the social graph to produce entity activity embeddings and / or entity activity content item embeddings. (As described below...) Figure 4C and Figure 4D An example of an entity activity model is shown that can be used by operation 304 to generate entity embeddings and / or entity activity content item embeddings. Alternatively, operation 304 can machine learn the embedding data of the content items corresponding to the unlabeled nodes through the embedding layer of the model. For example, the output of embedder 256 can be fully connected to the input of label prediction model 260.

[0135] At operation 306, the processing device generates a content graph that includes the set of heterogeneous digital content items, comprising the unlabeled digital content items. For this purpose, operation 306 can include the above reference. Figure 2B This describes a portion of the functionality of the graph generator 254. Operation 306 can include converting non-graph query results into a graph representation of a set of heterogeneous numeric content items and the relationships between them. Alternatively, operation 306 can include traversing an existing content graph and extracting a subset of the existing content graph that contains the unlabeled numeric content items.

[0136] The content graph generated by operation 306 includes nodes corresponding to content items in the group of heterogeneous digital content items and edges indicating the relationships between content items in the group of heterogeneous digital content items, wherein the nodes include more unlabeled nodes than labeled nodes.

[0137] At operation 308, the processing device extracts neighbor data for the unlabeled content item from the content graph generated by operation 306. The amount of neighbor data extracted for the unlabeled content item can be configured based on the requirements of a particular design or implementation. In some embodiments, the neighbor data includes all node attribute data and edge relationship data of nodes that have an N-degree connection with the node corresponding to the unlabeled content item, where N is a positive integer and N = 1. In other embodiments, N can be any positive integer.

[0138] Neighbor data can include edge data between unlabeled nodes and their neighboring nodes. Neighbor data can include node attribute data, such as modal data for the unlabeled node and different modal data for the neighboring nodes. The modal data can indicate that the unlabeled node corresponds to content with a certain modality, for example, text, image, video, or audio, and the different modal data can indicate that the neighboring nodes correspond to content with different modalities, for example, text, image, video, or audio. That is, the unlabeled node and the neighboring node can have different modalities; for example, the modality of the unlabeled node can be video, while the modality of the neighboring node is text, or the modality of the unlabeled node can be text, while the modality of the neighboring node is live streaming audio.

[0139] Neighbor data can include node attribute data, such as content type data and / or modal data, from the unlabeled nodes and the neighboring nodes in the content graph. Neighbor data can include content type or modal data for the unlabeled nodes and different content types or modalities for the neighboring nodes. For example, neighbor data can indicate that the unlabeled node corresponds to an article, and the neighboring node corresponds to a comment.

[0140] Alternatively or additionally, the adjacent data can indicate that the language data for the unlabeled node differs from the language data for adjacent nodes in the content graph. Additionally or alternatively, the node attribute data for any node (e.g., the unlabeled node or the adjacent node) can indicate that the content item associated with that node contains text written or spoken in multiple different languages.

[0141] At operation 310, the processing device determines whether the labeled model is in training mode or prediction mode. To do this, the processing device can check the value of a flag set by the model trainer 258. If the processing device determines that the labeled model is in training mode, then the processing device proceeds to operation 312. If the processing device determines that the labeled model is in prediction mode, then the processing device proceeds to operation 316. The following description... Figure 4A and Figure 4B2 An example of a labeled model is shown.

[0142] At operation 312, the processing device obtains label training data for a subgraph of the content graph. That is, the content graph generated at operation 306 is sparsely labeled, such that only a subset of the content graph contains labeled nodes. Operation 312 can obtain training data using, for example, a model trainer 258, as described above. At operation 314, the processing device applies an untrained or partially trained labeling model to a set of inputs including the embedded data, neighboring data, and the label training data obtained in operation 312. As a result of operation 314, one or more parameter values ​​of the labeling model can be adjusted to, for example, improve the predictive accuracy of the labeling model.

[0143] At operation 316, the processing device applies the trained annotation model to a set of inputs including the embedded data generated in operation 304 and the neighboring data extracted from the content graph in operation 308. The neighboring data included in this set of inputs can include modal data for the unlabeled node and different modal data for the neighboring node. The neighboring data included in this set of inputs can include content type data for the unlabeled node and different content type data for the neighboring node. The neighboring data included in this set of inputs can include language data for the unlabeled node and different language data for the neighboring node.

[0144] At operation 318, the processing device uses the output of the annotation model to generate tag data for the digital content item. Operation 318 may include providing the tag data to the application system. Operation 318 may include generating tags for the content item based on the output of the annotation model in response to the set of inputs, assigning tags to the content item, and / or tagging the content item using tags from a set of predefined tags (such as a set of spam tags). Operation 318 may include providing the tags to the application system.

[0145] Figure 4A This is an example of a model architecture 400 for labeling digital content items according to some embodiments of the present disclosure. Figure 4AThe model architecture can respond to a set of model inputs 402. This set of model inputs 402 includes a set of heterogeneous graph nodes 412, where each node 412 contains attributes of a digital content item. Examples of attributes include modal data and content type data. In one embodiment, the set of heterogeneous graph nodes includes nodes that are separated from each other by at most N edges (Nth degree connection), where N is a positive integer and N = 2. In other embodiments, N is any positive integer.

[0146] The set of heterogeneous graph nodes 412 includes a subset of nodes of the first type and another subset of nodes of the second type, wherein, for example, different node types indicate that the nodes represent different digital content items with different content types, modalities, and / or languages. For example, the set of heterogeneous graph nodes 412 is obtained by querying a pre-existing content graph or by generating a content graph, for example, by converting content item data stored according to one or more non-graph data structures into a graph form.

[0147] In some embodiments, the set of model inputs 402 includes content embedding data X for each of the nodes in the set of heterogeneous graph nodes 412. Embeddings for multimodal, multilingual content representing heterogeneous content types and / or modalities can be generated using embedding services or combinations of embedding services such as Universal Language Model (ULM) for text, Inception for images, and NetVLAD for video. Examples of heterogeneous content types and / or modalities include, for example, feed posts—text updates, feed posts—images, comments—text, comments—videos, articles, etc.

[0148] To generate text embeddings, ULM is a deep learning language model based on a large-scale transformer architecture that has learned billions of parameters from text datasets from the Internet (e.g., Wikipedia). An example of a ULM that can be used to generate embedding data X for the text of each of the 412 nodes is the Turing Universal Language Representation (TULR-V3), available from Microsoft. The ULM embedding model can be fine-tuned for specific domain use cases. Other natural language processing (NLP) models can be used; however, the advantage of ULM is that it supports multilingual NLP tasks and can be used across a wide range of applications. In some embodiments, the output of the embedding model is fully fed into the input of the label prediction model architecture.

[0149] To generate embeddings for the image and video portions of the content item represented by node 412, Inception and NetVLAD can be used respectively. Therefore, node 412, representing a multimodal content item, can have multiple different embeddings generated through different embedding services. The following will refer to... Figure 4C and Figure 4DDescribe examples of techniques that can be used to combine these different embeddings.

[0150] Embedsions of content items can be computed independently of the content graph. For example, embeddings can be computed for content items that happen to be represented by nodes in the graph, before the content graph is generated and without requiring a graph structure. For example, data about content items can be provided to the embedding service from a non-graph database, or a portion of the original content item can be provided as input to the embedding service. Therefore, content embeddings can be generated independently of the content graph and independently of any edge relationships with other content items in the content graph. For example, the embedding service can be invoked by a process different from the generation of the content graph.

[0151] The model input 402 contains edge relationship data 414. Edge relationship data indicates the edge relationships between nodes in the heterogeneous graph nodes 412, stored as an adjacency matrix A. Edge relationship data 414 can include one-to-one relationships and one-to-many relationships. An example of a one-to-one relationship is a feed post to a comment about that feed post and a reply to that comment. An example of a one-to-many relationship is a feed post to two different comments. Edge relationship data 414 can be obtained, for example, by querying the content graph to retrieve the edge data.

[0152] When the labeled model is in training mode, the input 402 of this model set contains a partial label Y. ^ The partial label indicates that the input 402 for this set of models is sparsely labeled. For example, in Figure 4A In the dataset, there are three type 1 nodes and four type 2 nodes. One type 1 node from the three type 1 nodes is labeled, and two type 2 nodes from the four type 2 nodes are labeled. Therefore, in... Figure 4A In the example, 50% or fewer nodes are labeled. The threshold for considering a set of nodes as sparsely labeled is variable, depending on the requirements of a particular design or implementation. Similarly, while the described technique is well-suited for sparsely labeled datasets, it is not limited to use with sparsely labeled datasets.

[0153] Model architecture 404 is a neural network architecture based on graph transformers, which uses a label propagation algorithm to predict the labels of unlabeled nodes by iteratively performing label propagation steps. Unlike binary classifiers, graph transformers are able to capture relationships between content items (e.g., dialogue context and / or (e.g.) relationships between post-comment, article-comment, and comment-comment content item pairs).

[0154] Model architecture 404 can be implemented as a neural network having separate layers for each modality and / or content type. For example, model architecture 404 can have layers for articles and layers for comments. Each of the modality and / or content type-specific layers can include multiple layers. The modality and / or content type-specific layers are connected; for example, the output of the article layer constitutes the input of the comment layer, or the output of the comment layer forms the input of the article layer. Thus, different modality and / or content type-specific layers can share information therein to propagate tags across modalities and / or content types.

[0155] Modality and content type data can each be reflected in the attributes of a node. Therefore, a content item with two different content types, such as an article and a comment or a comment and a reply, can have an edge between them regardless of the modality of each content item. For example, an article can have a text modality but can share an edge with a comment that has an image modality.

[0156] In the operation, for the starting node K and one or more Q-neighbor (or adjacent) nodes, model architecture 404 is applied to the inputs X, A, and Y. ^ When querying the content graph, the starting (or source) node K is designated as the key, and the neighboring nodes Q become the values. Neighbor vectors can be generated for all neighboring nodes Q (e.g., all n-degree connections of K, where n can be any positive integer). Nodes K and Q can be referred to as the source node and destination node, respectively, or collectively as an edge pair.

[0157] The additional input V is an aggregation of the edge relationship data between node K and each neighbor node Q across all Q neighbor nodes. This aggregation can be calculated as, for example, the mean or average of the edge relationship data across all neighbor nodes Q of K.

[0158] If the source node is a labeled node and the destination node is an unlabeled node, then model architecture 404 can generate a label prediction for the destination node; for example, the label of the source node also applies to the probability of the destination node. If the source node is an unlabeled node and the destination node is a labeled node, then model architecture 404 will generate a label prediction for the source node; for example, the label of the destination node also applies to the probability of the source node.

[0159] Model architecture 404 includes a multi-head attention mechanism, comprising a linear transformation layer 428, a scaled dot product attention layer 426, a multiplication layer 424, and a connection layer 422. The multi-head attention mechanism can run the attention mechanism several times in parallel. The linear transformation layer 428 applies a linear transformation operation, such as a scalar function, to the input.

[0160] The scaled dot-product attention layer 426 can transform the input, which is a vector of numbers, into a vector of probabilities, where the probability of each value is scaled relative to the relative scale of each value in the vector. For example, the scaled dot-product attention layer 426 can compute the dot product of the input and scale it to a value of 1, where h is a positive integer that can refer to the number of hidden states between the input and output layers.

[0161] Multiplication layer 424 applies a multiplication function to the input. Connection layer 422 connects the individual outputs of multiplication layer 424 to produce a final output representing the probability that the label applies to the unlabeled node.

[0162] Model architecture 404 produces model output 406, which is mathematically given as P(Yu|X,A,Y), or it can be given input X (content embedding), A (adjacency matrix including content relation edge data), and previously observed / predicted label Y. ^ The probability of the unlabeled content Yu (if in training mode). Therefore, the model output can include a set of labeled prediction data 432 for the unlabeled nodes of the input X.

[0163] exist Figure 4A In the example, label prediction data 423 shows the labels predicted by model architecture 404 based on partially labeled dataset 416: shaded (newly labeled) nodes in label prediction dataset 432 correspond to unshaded (unlabeled) nodes in partially labeled dataset 416, and unshaded nodes in label prediction dataset 432 correspond to shaded nodes in dataset 416. In other words, dataset 416 and label prediction dataset 432 are the same dataset, except that label prediction dataset 432 contains labels for all nodes in the dataset, while partially labeled dataset 416 only contains labels for a subset of the nodes in the dataset.

[0164] Unlike binary classifiers, the label propagation algorithm can capture labels across relevant / connected content types. This allows the labeling model to utilize partial label information to, for example, automatically propagate labels from already labeled article posts to unlabeled comments about the same article post, or automatically propagate labels from labeled comments to the previously unlabeled main article post. Therefore, the labeling model can be trained with far fewer labels compared to binary classifiers.

[0165] Unlike binary classifiers, model architecture 404 is able to train and predict using a single model for different content types and captures dialogue context as well as interdependencies between the same and different content types by modeling these dependencies as content graphs.

[0166] Figure 4B1 , Figure 4B2 and Figure 4B3 The illustration shows the input to a model architecture for labeling digital content items according to some embodiments of the present disclosure.

[0167] Figure 4B1 Section 440 illustrates data extracted from a content graph containing heterogeneous graph nodes 412. Embedding X can be computed for each node of the content graph using process 441. In process 441, exemplary input nodes are heterogeneous, even if they have the same modality, i.e., text. For example, a type 1 node could correspond to a comment, while a type 2 node could correspond to an article, or vice versa.

[0168] In process 441, a transformer-based neural network is applied to each input node to generate an embedding for each input node. For example, the transformer-based neural network generates an embedding X1 for a type 1 node and an embedding X2 for a type 2 node. The transformer-based neural network can be used for multiple different node types, or different embedding mechanisms can be applied to different node types. The node embeddings generated by the transformer-based neural network are inputs to the labeled model (e.g., model architecture 404).

[0169] The transformer-based neural network allows heterogeneous content types to be input into the annotation model. Examples of transformer-based neural network frameworks that can be used to generate content embeddings for text (e.g., articles and comments) include Turing ULRv3 and mBERT. The content embeddings generated by process 441 can be represented as multi-dimensional vectors. For example, mBERT produces embeddings as 768-dimensional vectors, and TULRv3 produces embeddings as 1024-dimensional vectors. For image input, Inception v3 produces embeddings as 2048-dimensional vectors, and for video, NetVLAD produces embeddings as 1024-dimensional vectors.

[0170] Process 441 can be executed by embedder 256.

[0171] Figure 4B2 Part 442 illustrates the adjacency matrix A, which includes edge relation data 414 describing the relationships between heterogeneous graph nodes 412. The edge relation data 414 and the adjacency matrix A more generally capture a dialogue thread structure that includes multi-level dependencies between heterogeneous content items. In the illustrated embodiment, the maximum adjacency degree is N degrees, where N is a positive integer up to and including 2. In other embodiments, N can be any positive integer.

[0172] Figure 4B2Part 450 illustrates the computation of edge relationships between heterogeneous content items (e.g., nodes with different content types, languages, or modalities). These edge relationships can be computed using a preorder tree traversal with timestamps as keys. For example, at time t0, an article A of type 1 is published to a feed, which is represented as node 443 in the content graph. At time t1, a comment B of type 2 is published about article A, which is represented as node 445 in the content graph. Based on a comparison of time t1 and time t0 (e.g., time t1 is after time t0), an edge 444 is created from article A → comment B, or from node 443 to node 445 in the content graph.

[0173] At time t2, a comment C of type 2 on article A is published, which is represented as node 447 in the content graph. Based on the comparison of time t2 with times t0 and t1 (e.g., time t2 is after times t0 and t1), an edge 446 is created from article A to comment C, or from node 443 to node 447 in the content graph.

[0174] At time t3, comment D of type 2 is published as a reply to comment B, which is represented as node 449 in the content graph. Based on the comparison of time t3 with times t0, t1, and t2 (e.g., time t3 is after times t0, t1, and t2), edge 448 is created from comment C to comment D, or from node 447 to node 449 in the content graph.

[0175] although Figure 4B2 Only two levels of content item dependencies are illustrated, but the enumeration of edges is not limited to two levels. Similarly, although... Figure 4B2 Only two content types are illustrated, but any combination of nodes in the content graph can be heterogeneous in any number of ways. For example, node 445 can be associated with content items containing different modalities (e.g., images instead of text), and edge relationships will be calculated in a similar manner.

[0176] Figure 4B3 The illustration shows an example 454 of determining training data for a labeled model, which can be implemented as a model trainer 258. When the model is in training mode, the input nodes include nodes labeled with verified or underlying fact labels, thus reflecting examples of content items in the already labeled training dataset. Since the training dataset includes only a portion of the labeled nodes—for example, less than half of the nodes are labeled—the training dataset contains partial labels from a partially labeled dataset 416, as shown in... Figure 4B3 As shown in section 452.

[0177] To obtain training data, the model trainer traverses the content graph and, at each node, checks whether the node has been labeled. If the node has been labeled, it is added to a subset of the content graph that contains labeled nodes. If the node has not been labeled, it is added to a different subset of the content graph that contains unlabeled nodes.

[0178] The training data includes heterogeneous nodes, for example, some labeled nodes for type 1 content items and other labeled nodes for type 2 content items. Unlabeled nodes can also be heterogeneous, for example, including both type 1 and type 2 content items. The labeling model uses the labeled nodes for label propagation.

[0179] Once the embeddings and content item dependencies (graph adjacency) have been created as described above, label propagation and relation-based label learning / prediction are computed in the graph transformer core of model architecture 401. For example, the labeling model computes the probability that the label of an labeled node applies to the unlabeled node for each unlabeled node.

[0180] Figure 4C These are examples of generating model architectures for embedding digital content items according to some embodiments of this disclosure. The model architecture (such as in...) can be... Figure 4C The cross-modal embeddings generated (as shown) can be used as described above. Figure 4A and Figure 4B1 The embedded input X in this way, Figure 4A and Figure 4B2 The tag prediction model architecture can be extended to include multimodal embeddings for multimodal content items.

[0181] The input 462 to the cross-modal model 460 can include content items with multiple different modalities, including text, images, videos, comments, derived metadata (e.g., text generated by optical character recognition or speech recognition), and sparse metadata (e.g., tic-tac-toe tags, topics). A single content item can include multiple different modalities. Input 462 can also include entity-related embeddings, such as neighbor embeddings and entity activity embeddings.

[0182] The cross-modal model 460 treats content item classification as a shared cross-modal task, where semantics can be learned in a joint manner. Therefore, for multimodal content items, the different modalities are initially separated and individual embeddings are created for each different modal part of the content item, and then these individual embeddings are fused together to create a unified or cross-modal embedding for the multimodal content item.

[0183] The cross-modal model 460 utilizes an intermediate gating fusion strategy to combine semantics from multiple different modalities to generate a cross-modal embedding 472. The cross-modal embedding 472 can be used as... Figure 4A and Figure 4B1 The input embedding X. Therefore, if a digital content item contains multiple different modalities, then Figure 4A and Figure 4B1 The input embedding X in the model does not necessarily have to reflect only a single modality (e.g., text, image, or video). In other words, using a cross-modal model 460 to generate cross-modal embeddings 472 enables... Figure 4A and Figure 4B2 The model architecture is capable of predicting labels for multimodal content items.

[0184] As shown on the right side of the figure, the cross-modal content embedding 472 can be a machine learning endpoint ending with a specific set of loss functions (e.g., focal loss for hard-to-distinguish negative samples) and a prediction component. The cross-modal model 460 includes per-modal shared layers, including an embedding service 464 and a fully connected layer 466. Examples of embedding services include BERT, Inception, NetVLAD, transformer-based embedders, and lookup services. The cross-modal model 460 also includes a prediction component 476, which is a multi-label prediction component with a single head.

[0185] In response to the cross-modal embedding 472, the prediction component 476 outputs the probability that the target label 480 is applicable to the content item based on the cross-modal embedding 472 of the content item. A portion 478 of the cross-modal model 460 can be trained independently or can be jointly trained, for example, by participating in a relevant supervised task.

[0186] Depending on the specific application system required, target tags can be roughly defined and / or finely defined. Examples of roughly defined tags are SPAM and CLEAR. Examples of finely defined tags include different types of spam content, such as financial fraud or NSFW, and different types of non-spam content, such as educational videos or healthy lifestyles.

[0187] While generating or fine-tuning the user activity embedding 468, the cross-modal content embedding 472 can be frozen, as shown on the left side of the figure. In other words, the cross-modal content embedding 472 and the user activity embedding 468 can be created separately using different inputs and then merged using, for example, a join. The user activity data used to generate the user activity embedding 468 can be obtained, for example, by querying entity data 206, activity data 208, and / or the social graph 212. For example, activity data 208 can include indications of which content items have been posted, commented on, viewed, and shared by a user in application system 130 and when. The user activity data and user profile data can be represented in the social graph 212.

[0188] Cross-modal content embedding 472 can be published for use by downstream applications. For example, cross-modal content embedding 472 can be consumed by feed sorting systems and / or content moderation systems. The advantage of cross-modal content embedding 472 is that downstream systems do not need to check the modality of content items, but can call cross-modal model 460 for content items regardless of the modality, even if the content item contains multiple modalities.

[0189] The cross-modal model 460 can be implemented as a backend service, such as a database or cloud service that stores the multimodal content embeddings 472 for querying and retrieval by other downstream systems. For example, downstream systems (such as news feeds) can have their own content classifiers and can use the cross-modal content embeddings 472 as input to their classifiers. The cross-modal model 460 can improve system resource utilization by replacing 50+ individual binary classifiers used for spam detection and content moderation.

[0190] The cross-modal content embedding 472 can be merged or integrated (e.g., by linking) with the generated or refined user activity embedding 468 to produce a user content embedding 470. User content embedding 470 can combine information that is typically or historically independent but related. That is, user content embedding 470 can create previously non-existent connections between social graph 212 and activity data 208.

[0191] User activity confirmation 468 is generated based on pre-existing embeddings, including neighbor embeddings and activity embeddings. Activity embeddings are generated based on user activities in their session feeds (e.g., clicks, likes, and shares), which are determined using activity data 208. Neighbor embeddings create similar activity embeddings for the user's neighbors (e.g., first-degree connections), which are determined using a social graph 212.

[0192] In response to user content embedding 470, prediction component 474 can generate and output the probability that a user of the application system will participate in a specific type of activity within the application system, given content items assigned to a specific target label 480. For example, prediction component 474 can adjust the output of prediction component 476 by weights, where the weight values ​​correspond to user activity embedding 468. Examples of the output of prediction component 474 include the probability that a user will view spam, the probability that a user will share spam, the probability that a user will view content items not labeled as spam, and the probability that a user will share content items not labeled as spam.

[0193] Figure 4D This is another example of a model architecture 490 for generating embeddings for digital content items according to some embodiments of this disclosure.

[0194] Figure 4D Model architecture 490 and Figure 4C Similarly, this model architecture is an example of a unified architecture for content annotation that utilizes shared layers across all modalities. Instead of a multi-label prediction component with a single head, the prediction component of model architecture 490 uses auxiliary losses from individual modality-specific heads, as exemplified by the concat / linear summation component 492, the fully connected layer 494, and the task layer 496. For example, the concat / linear summation component 492 combines single-modal embeddings generated for each input (sparse metadata, activity-derived metadata, comments, text articles, images, videos), and the output of the concat / linear summation component 492 is provided to a modality-specific classification task (e.g., text task, image task, video task), and can also be used to create a combined user content embedding 470, through which the user embedding and content embedding are jointly annotated.

[0195] Figure 5 This is a flowchart of an exemplary method 500 for controlling the distribution of digital content items according to some embodiments of the present disclosure.

[0196] Method 500 may be executed by processing logic including hardware (e.g., processing device, circuit system, dedicated logic, programmable logic, microcode, device hardware, integrated circuit, etc.), software (e.g., instructions that run or execute on the processing device), or a combination thereof. In some embodiments, a portion of method 500 is performed by... Figure 1The application system 130 and / or the graph-based annotation system 150 are executed. Although shown in a specific sequence or order, the order of the processes may be modified unless otherwise indicated. Therefore, the illustrated embodiments should be understood as examples only, and the illustrated processes may be executed in different orders, and some processes may be executed in parallel. Furthermore, one or more processes may be omitted in various embodiments. Therefore, not all processes are required in every embodiment. Other process flows are also possible.

[0197] In operation 502, the processing device receives digital content item data that can be distributed to users by an application system. Operation 502 may include performing the functions of the digital content item processor 204 described above. Operation 502 may include receiving content items from an application system that distributes digital content to user systems over a network, and forwarding the received content items and an address identifying the origin of the received content items on the network to a tagging system that generates tags for the digital content.

[0198] In operation 504, the processing device determines a graph-based model input for the digital content item. Operation 504 can include performing the functions of the graph generator 254 and embedder 256 described above. For example, operation 504 can include generating a content graph that includes nodes corresponding to content items in a set of digital content items that can be distributed by the application system and include the content items received at operation 502, as well as edges indicating relationships between the content items in that set of digital content items that include the received content items.

[0199] At operation 506, the processing device applies the labeling model to the graph-based model input determined in operation 504. Operation 506 can include the functionality of performing the label prediction model 260 described above. For example, operation 506 can include extracting edge data from the content graph for edges between nodes corresponding to the received content items and their neighboring nodes, and applying a model trained on the labeled nodes of the content graph to a set of inputs, which includes the edge data and embedding data for the received content items. Figure 4A and Figure 4B2 An example of a labeled model is shown in the figure.

[0200] At operation 508, the processing device determines whether the output of the annotation model in operation 506 triggers a digital content item distribution strategy. Operation 504 may include performing the functions of the graph-based annotation system 150 and / or distributor 240 described above. If the processing device determines that the annotation model output does not trigger a digital content distribution strategy, then the processing device proceeds to operation 510. If the processing device determines that the annotation model output triggers a digital content distribution strategy, then the processing device proceeds to operation 512.

[0201] At operation 510, the processing device labels the digital content item using zero or more tags associated with the output of the annotation model. At operation 512, the processing device labels the digital content item using one or more tags based on the output of the annotation model. Labeling the digital content item using one or more tags can include storing tag values ​​in a data record associated with the digital content item in the content item data 210.

[0202] At operation 514, the processing device controls the distribution of the digital content item based on the one or more tags used to label the digital content item in operation 512 and the triggered content distribution policy. Operation 514 may include forwarding the tag to an application system. Operation 514 may include, at the application system, distributing the received content item to multiple other addresses on the network based on the tag. Operation 514 may include, at the application system, determining that the received content item triggered a policy based on the tag. Operation 514 may include, at the application system, controlling the distribution of the received content item on the network according to the triggered policy. The tag may correspond to a content category, such as spam, inappropriate content, personal, adult content, inflammatory, hate speech, illegal, prohibited, or other categories of easily detectable unwanted content.

[0203] Figure 6 An exemplary machine, computer system 600, is illustrated, within which a set of instructions can be executed, causing the machine to perform any or more of the methods discussed herein. In some embodiments, computer system 600 may correspond to a networked computer system (e.g., Figure 1 A component of a computer system 100), which includes and is coupled to or utilizes a machine to execute an operating system, thereby performing operations corresponding to... Figure 1The operation of the graph-based annotation system 150. This machine can be connected to (e.g., networked to) other machines in a local area network (LAN), intranet, extranet, and / or the Internet. This machine can function as a server or client machine in a client-server network environment, or as a peer in a peer-to-peer (or distributed) network environment, or as a server or client machine in a cloud computing infrastructure or environment.

[0204] The machine can be a personal computer (PC), smartphone, tablet PC, set-top box (STB), personal digital assistant (PDA), cellular phone, web appliance, server, or any machine capable of executing a set of instructions (sequential instructions or other instructions) that dictate the actions taken by the machine. Furthermore, although a single machine is illustrated, the term "machine" should be understood to include any set of machines that individually or collectively execute a set (or more sets) of instructions to perform any or more of the methods discussed herein.

[0205] An exemplary computer system 600 includes a processing device 602, a main memory 604 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) (such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM)), a memory 606 (e.g., flash memory, static random access memory (SRAM), etc.), an input / output system 610, and a data storage system 640, which communicate with each other via a bus 630.

[0206] Processing device 602 identifies one or more general-purpose processing devices, such as microprocessors or central processing units. More specifically, the processing device may be a Complex Instruction Set Computing (CISC) microprocessor, a Reduced Instruction Set Computing (RISC) microprocessor, a Very Long Instruction Word (VLIW) microprocessor, or a processor implementing other instruction sets or combinations of instruction sets. Processing device 602 may also be one or more special-purpose processing devices, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), or network processors. Processing device 602 is configured to execute instructions 612 to perform the operations and steps discussed herein.

[0207] Computer system 600 may also include a network interface device 608 that communicates via network 620. Network interface device 608 can provide dual-path data communication coupling to the network. For example, network interface device 608 may be an Integrated Services Digital Network (ISDN) card, a wired modem, a satellite modem, or a modem providing data communication connectivity to a corresponding type of telephone line. As another example, network interface device 608 may be a Local Area Network (LAN) card to provide data communication connectivity to a compatible LAN. A wireless link may also be implemented. In any such implementation, network interface device 608 can transmit and receive electrical, electromagnetic, or optical signals carrying digital data streams representing various types of information.

[0208] A network link can provide data communication through at least one network leading to other data devices. For example, a network link can provide a connection to a worldwide packet data communication network commonly referred to as the “Internet” through, for example, a local network leading to a host computer or a data device operated by an Internet Service Provider (ISP). The local network and the Internet use electrical, electromagnetic, or optical signals carrying digital data to and from the computer system 600.

[0209] Computer system 600 can send messages and receive data (including program code) via a network and network interface device 608. In an Internet example, the server can transmit requested code for an application via the Internet 628 and network interface device 608. The received code can be executed by processing device 602 upon receipt and / or can be stored in data storage system 640 or other non-volatile storage device for later execution.

[0210] Input / output system 610 may include output devices, such as a display (e.g., a liquid crystal display (LCD) or touchscreen display) for displaying information to a computer user, a speaker, a haptic device, or other forms of output device. Input / output system 610 may include input devices, such as alphanumeric keys and other keys configured to convey information and command selections to processing device 602. Alternatively or additionally, input devices may include cursor controls, such as a mouse, trackball, or arrow keys, to convey directional information and command selections to processing device 602 and control cursor movement on the display. Alternatively or additionally, input devices may include microphones, sensors, or sensor arrays to convey sensed information to processing device 602. For example, sensed information may include voice commands, audio signals, geographic location information, and / or digital images.

[0211] Data storage system 640 may include machine-readable storage medium 642 (also known as computer-readable medium) storing one or more sets of instructions 644 or software embodying one or more of the methods or functions described herein. Instructions 644 may also reside wholly or at least partially in main memory 604 and / or processing device 602 during execution by computer system 600, which also constitute machine-readable storage media.

[0212] In one embodiment, instruction 626 includes implementing a component corresponding to a digital content item annotation (e.g., Figure 1 The instructions for the function of the graph-based annotation system 150. Although the machine-readable storage medium 642 is shown as a single medium in the exemplary embodiment, the term "machine-readable storage medium" should be understood to include a single medium or multiple media storing one or more sets of instructions. "Machine-readable storage medium" should also be understood to include any medium capable of storing or encoding a set of instructions that will be executed by a machine and cause said machine to perform one or more of the methods of this disclosure. The term "machine-readable storage medium" should be understood accordingly to include, but is not limited to, solid-state memory, optical media, and magnetic media.

[0213] Some parts of the detailed description above are presented in terms of algorithms and symbolic representations of operations performed on data bits in computer memory. These algorithmic descriptions and representations are the way that those skilled in the art of data processing communicate the essence of their work to others in the field. An algorithm here generally refers to a self-contained sequence of operations leading to a desired result. Operations are those that require physical manipulation of physical quantities. Typically, though not always necessary, these quantities take the form of electrical or magnetic signals that can be manipulated by storage, combination, comparison, and other means. Sometimes (primarily due to common usage) it proves convenient to refer to these signals as bits, values, elements, symbols, characters, items, numbers, etc.

[0214] However, it should be remembered that all these and similar terms are associated with appropriate physical quantities and are merely convenient labels applied to those quantities. This disclosure may relate to the actions and processes of a computer system or similar electronic computing device, which manipulate and transform data representing physical (electronic) quantities within the registers and memory of the computer system into other data representing physical quantities similarly represented within the memory or registers of the computer system or other such information storage systems.

[0215] This disclosure also relates to an apparatus for performing the operations described herein. Such an apparatus may be specifically constructed for its intended purpose, or it may include a general-purpose computer that will be selectively activated or reconfigured by a computer program stored therein. For example, in response to a processor of a computer system or other data processing system (e.g., computing system 100) executing a computer program (e.g., a sequence of instructions) contained in memory or other non-transitory machine-readable storage medium, the computer system or other data processing system may implement the computer implementation methods described above, including... Figure 2B Methods 300 and 500 are shown. Such a computer program can be stored in a computer-readable storage medium, such as, but not limited to, any type of disk (including floppy disk, optical disk, CD-ROM and magneto-optical disk), read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic card or optical card, or any type of medium suitable for storing electronic instructions, each of which is coupled to a computer system bus.

[0216] The algorithms and displays described herein are not inherently related to any particular computer or other device. Various general-purpose systems can be used with the program according to the teachings of the present invention, or it can be demonstrated that more specialized devices can be conveniently constructed to perform the method. The structures for a wide variety of such systems will become apparent from the explanations in the description below. Furthermore, this disclosure is not described with reference to any particular programming language. It should be understood that the teachings of this disclosure described herein can be implemented using a wide variety of programming languages.

[0217] This disclosure may be provided as a computer program product or software, which may include a machine-readable medium having instructions stored thereon, which can be used to program a computer system (or other electronic device) to perform processes according to this disclosure. Machine-readable media include any mechanism for storing information in a machine-readable (e.g., computer-readable) form. In some embodiments, machine-readable (e.g., computer-readable) media include machine-readable (e.g., computer-readable) storage media, such as read-only memory (ROM), random access memory (RAM), disk storage media, optical storage media, flash memory components, etc.

[0218] Illustrative examples of the techniques disclosed herein will be provided below. Embodiments of the techniques may include any of the examples described below or a combination thereof.

[0219] In Example 1, a method includes generating and storing a content graph for a set of digital content items received by an application system from a user system. The application system is capable of distributing digital content to the user system over a network based on a social graph and a set of spam tags. The content graph includes labeled nodes corresponding to subsets of the set of digital content items that have spam tags, unlabeled nodes corresponding to subsets of the set of digital content items that do not have spam tags, and edges indicating relationships between the subsets of the set of digital content items. The content graph includes more unlabeled nodes than labeled nodes. The method also includes querying the content graph to retrieve edge data for edges between unlabeled nodes and adjacent nodes. The unlabeled nodes correspond to digital content items that do not have spam tags. The adjacent nodes correspond to different digital content items. The edge data indicates the relationships between the unlabeled nodes and the adjacent nodes. The method also includes assigning tags from the set of spam tags to the unlabeled nodes by a machine learning model trained on the labeled nodes and edges of the content graph, in response to a set of inputs including the retrieved edge data and embedded data associated with the unlabeled nodes.

[0220] Example 2 includes the topic of Example 1, and further includes: querying the content graph to retrieve modal data for the unlabeled node and different modal data for the neighboring nodes, and in response to the set of inputs including the modal data for the unlabeled node and the different modal data for the neighboring nodes, the machine learning model determines the label for the unlabeled node. Example 3 includes the topic of Example 1 or Example 2, and further includes: querying the content graph to retrieve content type data for the unlabeled node and different content type data for the neighboring nodes, and in response to the set of inputs including the content type data and the different content type data, the machine learning model determines the label for the unlabeled node. Example 4 includes the topic of any of Examples 1-3, and further includes: generating a cross-modal embedding for the digital content item from a cross-modal model trained for digital content items with different modalities, and in response to the set of inputs including the cross-modal embedding, the machine learning model determines the label for the unlabeled node. Example 5 includes the topic of any of Examples 1-4, and further includes: generating an entity activity-content item embedding by an entity activity model trained on entity activity data extracted from the application system in response to activity data of entities associated with the digital content item and represented in the social graph; and determining the label for the unlabeled node by the machine learning model in response to the set of inputs including the entity activity-content item embedding. Example 6 includes the topic of any of Examples 1-5, and further includes: machine learning the embedding data associated with the unlabeled node by an embedding layer of the machine learning model; and determining the label for the unlabeled node by the machine learning model in response to the set of inputs including the machined embedding data.

[0221] In Example 7, a system includes at least one processor and at least one computer memory operatively coupled to the at least one processor. The at least one computer memory includes instructions that, when executed by the at least one processor, enable the at least one processor to: generate a content graph for a set of heterogeneous digital content items received by an application system distributing digital content to multiple user systems via a network; the content graph includes nodes having labeled nodes corresponding to content items in the set of heterogeneous digital content items that have spam tags and unlabeled nodes corresponding to content items in the set of heterogeneous digital content items that do not have spam tags, and edge data indicating relationships between the content items among the nodes; and train a neural network model for a set of inputs including the content graph to produce a trained neural network model.

[0222] Example 8 includes the subject of Example 7, wherein the at least one computer memory further includes instructions that, when executed by the at least one processor, enable the at least one processor to extract content type data for a node from the content graph, extract different content type data for adjacent nodes from the content graph, and train the neural network model for the set of inputs including the content type data and the different content type data. Example 9 includes the subject of Example 8, wherein the content type data indicates that the node corresponds to the following: a first type is a post, article, comment, or reply, and the different content type data indicates that the adjacent node corresponds to a second type different from the first type. Example 10 includes the subject of any of Examples 7-9, wherein the at least one computer memory further includes instructions that, when executed by the at least one processor, enable the at least one processor to extract modal data for a node from the content graph, extract different modal data for adjacent nodes from the content graph, and train the neural network model for the set of inputs including the modal data and the different modal data. Example 11 includes the topic of any of the examples 7-10, wherein the modal data indicates that the node corresponds to content having a first modality, which is text, image, video, or audio; and the different modal data indicates that the adjacent node corresponds to a second modality, which is different from the first modality. Example 12 includes the topic of any of the examples 7-11, wherein the at least one computer memory further includes instructions that, when executed by the at least one processor, enable the at least one processor to apply a cross-modal model to content items in the set of heterogeneous digital content items to generate cross-modal embeddings, and to train the neural network model for the set of inputs including the cross-modal embeddings. Example 13 includes the topic of any of the examples 7-12, wherein the at least one computer memory further includes instructions that, when executed by the at least one processor, enable the at least one processor to apply an entity activity model to data associated with entities represented in a social graph to generate entity activity embeddings, and to train the neural network model for the set of inputs including the entity activity embeddings. Example 14 includes the subject matter of any of the examples in Examples 7-13, wherein the at least one computer memory further includes instructions that, when executed by the at least one processor, enable the at least one processor to machine learn embedded data through an embedding layer of the neural network model and use the machined embedded data as input to another layer of the neural network model.

[0223] In Example 15, a method includes: receiving a content item at an application system that controls the distribution of digital content across multiple user systems on a network based on a set of spam labels; forwarding the received content item to a labeling system that generates labels for the digital content; at the labeling system, (i) generating and storing a content graph including nodes corresponding to content items in a set of digital content items that can be distributed by the application system, including the received content item, and edges indicating relationships between the content items in the set of digital content items including the received content item; (ii) querying the content graph to retrieve edge data for the edges between the nodes corresponding to the received content item and adjacent nodes; and (iii) in response to a set of inputs including the edge data and embedded data for the received content item, assigning labels from the set of spam labels to the received content item at a machine learning model trained on labeled nodes and edges of the content graph.

[0224] Example 16 includes the topic of Example 15, and further includes forwarding a tag assigned to a received content item to the application system at the tagging system. Example 17 includes the topic of Example 15 or Example 16, and further includes distributing the received content item to the plurality of user systems on the network based on the tag at the application system. Example 18 includes the topic of any of Examples 15-17, and further includes determining at the application system that a policy has been triggered by the received content item based on the tag. Example 19 includes the topic of any of Examples 15-18, and further includes controlling the distribution of the received content item on the network based on the triggered policy at the application system. Example 20 includes the topic of any of Examples 15-20, wherein the node corresponding to the received content item and the adjacent nodes are heterogeneous.

[0225] In the foregoing description, embodiments of the present disclosure have been described with reference to specific exemplary embodiments. It will be apparent that various modifications can be made to the exemplary embodiments of the present disclosure without departing from the broader spirit and scope of the embodiments set forth in the following claims. Accordingly, the description and drawings should be considered in an illustrative sense rather than a limiting sense.

Claims

1. A method comprising: For a set of digital content items received by the application system from the user system, a content graph is generated and stored, wherein: The application system can distribute digital content to the user system via the network based on a social graph and a set of spam tags; and The content graph includes labeled nodes corresponding to the subset of digital content items with spam tags, unlabeled nodes corresponding to the subset of digital content items without spam tags, and edges between the nodes in the content graph. Query the content graph to retrieve edge data for edges between unlabeled nodes and their adjacent nodes, where: The unlabeled nodes correspond to digital content items that do not have spam tags; The adjacent nodes correspond to different digital content items; and The edge data indicates the relationship between the unlabeled node and its neighboring nodes; and In response to a set of inputs including the retrieved edge data and the embedded data associated with the unlabeled node, a machine learning model assigns a label from the set of spam labels to the unlabeled node, wherein: The embedded data includes the output of the cross-modal embedding model; The output of the cross-modal embedding model is connected to the input of the machine learning model; The machine learning model includes a graph transformer-based neural network with connected modality-specific layers; The graph-transformer-based neural network is trained to determine the relationships between content items based on the labeled nodes and edges of the content graph; and The labels are assigned to unlabeled nodes by modality-specific layers of a trained graph transformer-based neural network, which performs graph-based label prediction propagation from labeled nodes to unlabeled nodes of different modalities based on edge relationships between the nodes in the content graph.

2. The method according to claim 1, further comprising: The content graph is queried to retrieve modal data for the unlabeled node and different modal data for the adjacent node, and in response to the set of inputs including the modal data for the unlabeled node and the different modal data for the adjacent node, the label for the unlabeled node is determined by the machine learning model.

3. The method according to claim 1, further comprising: The content graph is queried to retrieve content type data for the unlabeled node and different content type data for the adjacent node, and in response to the set of inputs including the content type data and the different content type data, the machine learning model determines the label for the unlabeled node.

4. The method according to claim 1, further comprising: The cross-modal embedding model, trained for digital content items of different modalities, generates cross-modal embeddings for the digital content items, and in response to the set of inputs including the cross-modal embeddings, the machine learning model determines the labels for the unlabeled nodes.

5. The method according to claim 1, further comprising: In response to activity data of entities associated with the digital content item and represented in the social graph, an entity activity-content item embedding is generated by an entity activity model trained on entity activity data extracted from the application system, and in response to the set of inputs including the entity activity-content item embedding, the machine learning model determines the label for the unlabeled node.

6. The method according to claim 1, further comprising: The embedding data associated with the unlabeled node is machine-learned by the embedding layer of the machine learning model, wherein the embedding layer of the machine learning model includes the cross-modal embedding model, and the label for the unlabeled node is determined by the machine learning model in response to the set of inputs including the machine-learned embedding data.

7. A system comprising: At least one processor; At least one computer memory operatively coupled to the at least one processor; The at least one computer memory includes instructions that, when executed by the at least one processor, enable the at least one processor to: For a set of digital content items received by the application system from the user system, a content graph is generated and stored, wherein: The application system can distribute digital content to the user system via the network based on a social graph and a set of spam tags; and The content graph includes labeled nodes corresponding to the subset of digital content items with spam tags, unlabeled nodes corresponding to the subset of digital content items without spam tags, and edges between the nodes in the content graph. Query the content graph to retrieve edge data for edges between unlabeled nodes and their adjacent nodes, where: The unlabeled nodes correspond to digital content items that do not have spam tags; The adjacent nodes correspond to different digital content items; and The edge data indicates the relationship between the unlabeled node and its neighboring nodes; and In response to a set of inputs including the retrieved edge data and the embedded data associated with the unlabeled node, a machine learning model assigns a label from the set of spam labels to the unlabeled node, wherein: The embedded data includes the output of the cross-modal embedding model; The output of the cross-modal embedding model is connected to the input of the machine learning model; The machine learning model includes a graph transformer-based neural network with connected modality-specific layers; The graph-transformer-based neural network is trained to determine the relationships between content items based on the labeled nodes and edges of the content graph; and The labels are assigned to unlabeled nodes by modality-specific layers of a trained graph transformer-based neural network, which performs graph-based label prediction propagation from labeled nodes to unlabeled nodes of different modalities based on edge relationships between the nodes in the content graph.

8. The system according to claim 7, wherein, The at least one computer memory further includes instructions that, when executed by the at least one processor, enable the at least one processor to: query the content graph to retrieve modal data for the unlabeled node and different modal data for the adjacent node, and, in response to the set of inputs including the modal data for the unlabeled node and the different modal data for the adjacent node, determine the label for the unlabeled node by the machine learning model.

9. The system according to claim 7, wherein, The at least one computer memory further includes instructions that, when executed by the at least one processor, enable the at least one processor to: query the content graph to retrieve content type data for the unlabeled node and different content type data for the adjacent node; and, in response to the set of inputs including the content type data and the different content type data, determine the label for the unlabeled node by the machine learning model.

10. The system according to claim 7, wherein, The at least one computer memory further includes instructions that, when executed by the at least one processor, enable the at least one processor to: generate a cross-modal embedding for the digital content item from the cross-modal embedding model trained for digital content items of different modalities, and, in response to the set of inputs including the cross-modal embedding, determine the label for the unlabeled node by the machine learning model.

11. The system according to claim 7, wherein, The at least one computer memory further includes instructions that, when executed by the at least one processor, enable the at least one processor to: generate an entity activity-content item embedding by an entity activity model trained on entity activity data extracted from the application system, in response to activity data of entities associated with the digital content item and represented in the social graph; and determine the label for the unlabeled node by the machine learning model, in response to the set of inputs including the entity activity-content item embedding.

12. The system according to claim 7, wherein, The at least one computer memory further includes instructions that, when executed by the at least one processor, enable the at least one processor to: machine learn the embedded data associated with the unlabeled node by the embedding layer of the machine learning model, wherein the embedding layer of the machine learning model includes the cross-modal embedding model, and, in response to the set of inputs including the machined embedded data, determine the label for the unlabeled node by the machine learning model.

13. A non-transitory machine-readable medium comprising instructions that, when executed by a processor, cause the processor to: For a set of digital content items received by the application system from the user system, a content graph is generated and stored, wherein: The application system is able to distribute digital content to the user system via the network based on a social graph and a set of spam tags; and The content graph includes labeled nodes corresponding to the subset of digital content items with spam tags, unlabeled nodes corresponding to the subset of digital content items without spam tags, and edges between the nodes in the content graph. Query the content graph to retrieve edge data for edges between unlabeled nodes and their adjacent nodes, where: The unlabeled nodes correspond to digital content items that do not have spam tags; The adjacent nodes correspond to different digital content items; and The edge data indicates the relationship between the unlabeled node and its neighboring nodes; and In response to a set of inputs including the retrieved edge data and the embedded data associated with the unlabeled node, a machine learning model assigns a label from the set of spam labels to the unlabeled node, wherein: The embedded data includes the output of the cross-modal embedding model; The output of the cross-modal embedding model is connected to the input of the machine learning model; The machine learning model includes a graph transformer-based neural network with connected modality-specific layers; The graph-transformer-based neural network is trained to determine the relationships between content items based on the labeled nodes and edges of the content graph; and The labels are assigned to unlabeled nodes by modality-specific layers of a trained graph transformer-based neural network, which performs graph-based label prediction propagation from labeled nodes to unlabeled nodes of different modalities based on edge relationships between the nodes in the content graph.

14. The non-transitory machine-readable medium according to claim 13, wherein, When executed by the processor, the instruction also causes the processor to: query the content graph to retrieve modal data for the unlabeled node and different modal data for the adjacent node, and, in response to the set of inputs including the modal data for the unlabeled node and the different modal data for the adjacent node, determine the label for the unlabeled node by the machine learning model.

15. The non-transitory machine-readable medium according to claim 13, wherein, When the instruction is executed by the processor, the processor also causes the processor to: query the content graph to retrieve content type data for the unlabeled node and different content type data for the adjacent node, and, in response to the set of inputs including the content type data and the different content type data, determine the label for the unlabeled node by the machine learning model.

16. The non-transitory machine-readable medium according to claim 13, wherein, When executed by the processor, the instructions also cause the processor to: generate a cross-modal embedding for the digital content item from the cross-modal embedding model trained for digital content items of different modalities, and, in response to the set of inputs including the cross-modal embedding, determine the label for the unlabeled node by the machine learning model.

17. The non-transitory machine-readable medium according to claim 13, wherein, When executed by the processor, the instructions also cause the processor to: generate an entity activity-content item embedding from an entity activity model trained on entity activity data extracted from the application system, in response to activity data of entities associated with the digital content item and represented in the social graph; and determine the label for the unlabeled node by the machine learning model in response to the set of inputs including the entity activity-content item embedding.

18. The non-transitory machine-readable medium according to claim 13, wherein, When executed by the processor, the instruction also causes the processor to: machine learn the embedded data associated with the unlabeled node by the embedding layer of the machine learning model, wherein the embedding layer of the machine learning model includes the cross-modal embedding model, and, in response to the set of inputs including the machined embedded data, determine the label for the unlabeled node by the machine learning model.

Images