Generative collaboration message suggestions

Abstract

Embodiments of the disclosed techniques include configuring a first machine learning model to generate and output suggested message content based on a first correlation between message content and message acceptance data, where the first machine learning model includes a first encoder-decoder model architecture; configuring a second machine learning model to generate and output message evaluation data based on a second correlation between the message content and the message acceptance data, where the second machine learning model comprises a second encoder-decoder model architecture; coupling an output of the first machine learning model to an input of the second machine learning model; and coupling an output of the second machine learning model to an input of the first machine learning model.

Description

TECHNICAL FIELD

[0001] The technical field to which this disclosure pertains is the generation of digital content such as electronic messages. Another technical field to which this disclosure pertains is automated content generation using artificial intelligence. COPYRIGHT NOTICE

[0002] This Patent Document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction of the Patent Document or the patent disclosures contained herein for the purpose and extent of 37 C.F.R. § 1. 14(b)(2) as it appears in the Office’s publicly available records, but, otherwise, reserves all copyright rights whatsoever. BACKGROUND

[0003] A messaging system is a computer system that uses a computer network to transmit electronic messages between or among user accounts using computing devices. For example, an electronic message is composed by a sender at a computing device of the sender using an account of the sender. The sender identifies one or more intended recipients of the message, and the messaging system transmits the message to the account of the one or more intended recipients.

[0004] There are many different types of messaging systems. Email is a form of electronic messaging. Instant messaging, text messaging, direct messaging, in-application messaging, mobile messaging, multimedia messaging, and push notifications are forms of messaging systems that generally have fewer features than email and are often useful for short, asynchronous, or real-time conversations between or among users.

[0005] In-application messaging differs from email and text messaging in that a sender can attempt to send email and text messages to any recipient as long as the recipient’s handle or address is known, whereas the intended recipients of in-application messaging are limited to the user base of the application.

[0006] Some forms of in-application messaging are public. For example, some application software systems, such as social networking services and some asynchronous messaging systems, allow their users to message one another in a way that makes the messages viewable by other users of those systems. Direct messaging systems provide a non-public mode of electronic communication between or among users of an application software system. In direct messaging, only the sender and recipient can see the messages exchanged between them.

[0007] Some social network-based applications further restrict direct messaging. For example, in some applications, the ability to send a direct message is limited to the sender’s connections. That is, a sender can only indicate message recipients with whom the sender is connected through a social network. If the sender’s user account satisfies one or more applicable criteria, the application can grant the sender broader access to a larger group of potential direct message recipients. For example, if the sender has not had any policy violations while using the application, or if the sender’s role qualifies the sender for broader access (e.g., if the sender is a premium user or a recruiter), the sender can be granted broader access.

[0008] Internet-based software applications can have millions or billions of users worldwide. In these cases, a messaging system facilitates the distribution of electronic messages among or between very large numbers of application users. For example, a messaging system can distribute millions of messages per day to hundreds of millions of user devices worldwide. The messages can include various different forms of digital content, including text, image, audio, and / or video content. BRIEF DESCRIPTION OF DRAWINGS

[0009] The present disclosure will become more fully understood from the detailed description given herein below and from the accompanying drawings, wherein:-

[0010] Figure 1 is a flow diagram of an example method for automated message suggestion generation using components of a generative message suggestion system in accordance with some embodiments of the present disclosure.

[0011] Figure 2 is a timing diagram illustrating an example of communication between a message generation interface and a generative message suggestion system in accordance with some embodiments of the present disclosure.

[0012] Figure 3A 、 Figure 3B 、 Figure 3C 、 Figure 3D FIGS. 3E, 3F, 3G, and 3H illustrate examples of at least one flow of screen captures of user interface screens configured to create an electronic message based on at least one AI-generated message suggestion in accordance with some embodiments of the present disclosure.

[0013] Figure 4 illustrates an example of a screen capture of a user interface screen configured to compose a message based on at least one AI-generated message suggestion in accordance with some embodiments of the present disclosure.

[0014] Figure 5is a block diagram of a computing system including a generative message suggestion system according to some embodiments of the present disclosure.

[0015] Figure 6 is an example of an entity graph according to some embodiments of the present disclosure.

[0016] Figure 7 is a flowchart of an example method for automated message suggestion generation using components of a generative message suggestion system according to some embodiments of the present disclosure.

[0017] Figure 8 is a flowchart of an example method for automated message suggestion generation using components of a generative message suggestion system according to some embodiments of the present disclosure.

[0018] Figure 9 is a flowchart of an example method for configuring a generator model for automated message suggestion generation using components of a generator model subsystem according to some embodiments of the present disclosure.

[0019] Figure 10 is a flowchart of an example method for configuring a scoring model for automated message suggestion generation using components of a scoring model subsystem according to some embodiments of the present disclosure.

[0020] Figure 11 is a flowchart of an example method for automated message suggestion generation according to some embodiments of the present disclosure.

[0021] Figure 12 is a flowchart of an example method for automated message suggestion generation according to some embodiments of the present disclosure.

[0022] Figure 13 is a block diagram of an example computer system including components of a generative message suggestion system according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0023] When a sender of an electronic message sends a message to an intended recipient, the message can be at least temporarily stored in the intended recipient's account's message inbox, but the sender does not guarantee that the message will be accepted by the intended recipient. Thus, as used herein, an intended recipient can refer to a user to whom a sender has sent an electronic message but who has not yet accepted the message, while a recipient can refer to an intended recipient who has accepted the message. A recipient or intended recipient can refer to one or more users who are the target of a sender's message. For example, a sender can address their message to one or more intended recipients, and some or all of the intended recipients can accept the message.

[0024] Acceptance as used herein can refer to any action or series of actions by an intended recipient that indicates acceptance of a message, including clicking or tapping on a message icon related to a message, opening a message, viewing a message, reading a message, replying to a message, and the like. Conversely, actions that indicate non-acceptance of a message include explicitly rejecting a message, rejecting receipt of a message, ignoring a message, deleting a message, moving a message to a "junk" folder, reporting a message as spam, or allowing a message to remain unopened or unread for an extended period of time.

[0025] Acceptance data indicating whether a message by a sender has been accepted can be logged and tracked, for example, by a logging service or analytics component of a messaging system. In some implementations, acceptance can refer to the occurrence of a particular user interface event. For example, acceptance can occur when a recipient of a message takes a particular action, such as clicking on a "Yes, interested" button on a messaging interface to explicitly indicate that the recipient is interested in a topic or opportunity represented by the message. Alternatively or additionally, acceptance can occur implicitly when a recipient of a message responds to the message with any typed free text that is interpreted by the system as an acceptance. For example, an artificial intelligence model trained to classify clicks and / or sequences of keystrokes as accepted or not accepted (e.g., a binary classifier or other machine learning model) can be used to detect implicit acceptance.

[0026] The acceptance data can be used to compute an acceptance rate for a sender. Acceptance rate as used herein can refer to a computation based on historical data reflecting a comparison of the number of messages by a sender that have been accepted within a particular time period to the total number of messages sent by the sender within that same time period. Acceptance rates can be computed for individual senders or for groups of senders as a whole, where the groups of senders have at least one common characteristic. Acceptance rates can also be computed for various combinations of recipients and intended recipients. For example, a sender can have different acceptance rates for different groups of recipients, where the groups of recipients have at least one common characteristic. A sender can use acceptance rates to measure the success of their previous messaging efforts.

[0027] Historical acceptance rate data can be used to compute an acceptance probability. Acceptance probability as used herein can refer to a computed (e.g., probabilistic or statistical) likelihood that a particular message created and sent by a particular sender will be accepted by a particular intended message recipient. Acceptance probabilities can be used, for example, to predict the likelihood that a message being composed by a sender will be accepted by an intended recipient before the message is sent.

[0028] It can be challenging for many message senders to compose messages with a high probability of acceptance. It can also be very time consuming to find the right words and organize them in an engaging way based on the needs, interests, or preferences of a particular intended recipient. Furthermore, many message senders can not know the message composition and / or message writing techniques that can lead to a high probability of acceptance. The limitations of conventional user input devices can exacerbate these barriers to effective message composition. For example, smaller form factor touch-based keyboards and touch screens make typing prone to error, and conventional auto-correct mechanisms are often inaccurate. Speech-based input mechanisms can facilitate input of message text by enabling senders to input message content using their speech, but conventional speech-to-text and speech-to-image techniques still produce transcriptions that require manual correction using a touch screen. Because of these and other limitations of conventional input devices, senders often need to perform labor-intensive review and revision of the messages they create before the messages are ready to be sent to the intended recipient. However, message senders often can not have the time or inclination to perform such review and revision, resulting in suboptimal acceptance rates of the messages they send.

[0029] Because of these and other problems, a technical challenge is for messaging systems to optimize acceptance probability while also optimizing efficiency of the message creation process.

[0030] Conventional attempts to improve the message creation process have provided generalized message templates. Although effective, these standardized templates are impersonal and have low acceptance rates. A conventional alternative to generic templates is for senders to manually write personalized messages or manually modify templates for each individual intended recipient. Manually writing personalized messages or manually modifying templates has resulted in higher acceptance rates, but the message creation process is not efficient. For high-volume senders, the use of generic templates is scalable, but results in suboptimal acceptance rates. On the other hand, the manual personalization approach results in improved acceptance rates, but is not scalable.

[0031] Accordingly, a technical challenge is to develop an automated process for generating message suggestions that provides an efficient user experience for message senders, is scalable, and provides message suggestions that improve the acceptance probability of the sender, particularly with respect to the sender's intended recipients.

[0032] Another technical challenge for messaging systems is to effectively determine the most relevant content for a particular recipient and focus the recipient's communication on that most relevant content. This is a continuing challenge as the amount of content available from multiple different sources continues to increase, making it increasingly challenging to extract information for the purpose of generating a message. Embodiments of the disclosed technology can facilitate and improve the process of identifying the most relevant content for a particular user based on the user's intent or goal, and then effectively creating a message corresponding to the identified most relevant content.

[0033] Another technical challenge is how to machine generate digital content including images, videos, and / or audio. Yet another technical challenge is how to reduce the burden of user input when creating a message. Yet another technical challenge is to scale the machine generation of message suggestions to large user groups (e.g., hundreds of thousands to millions or more users) without necessarily increasing the size of the generative message suggestion system. A further technical challenge is to improve the efficiency of message suggestion distribution over a network, including adapting generative message suggestions to a variety of different hardware platforms, screen sizes, and device types. An additional technical challenge is to provide a generative message system that is robust to latency issues.

[0034] To address these and other technical challenges of conventional message creation systems, the disclosed approach utilizes artificial intelligence technology including generative models to facilitate a generative, collaborative process that can result in the creation of messages with improved acceptance probability. As described in greater detail below, the present disclosure provides an artificial intelligence model architecture configured to recursively machine generate message suggestions using a generator model, score messages (including messages containing machine generated message suggestions) using a scoring model, and tune the generator model based on the output of the scoring model.

[0035] Examples of message suggestions that can be machine generated using the disclosed technology include suggestions for attribute values to be included in a message composed by a sender, insights about which particular attribute values the sender should emphasize or not emphasize when composing a message, and suggested samples of machine generated message content, e.g., examples of content that can form or be included in the body of a message. Message suggestions are customized based on available information about the sender and / or the intended recipient, in some implementations obtained using a dynamically updated entity graph. The sender can further develop or modify the machine generated suggested message to further customize the message for the particular intended recipient.

[0036] In contrast to conventional approaches for facilitating message creation, the disclosed technology is both generative and collaborative in that these aspects can effectively produce message suggestions that are automatically tailored based on up-to-date available information about the sender, the subject matter of the message, and / or the intended recipient. This approach facilitates the message composition process and improves the likelihood that the sender will produce a message that will be accepted by the intended recipient, while reducing the need for the sender to engage in lengthy interactions with cumbersome input mechanisms, thereby reducing the time from message creation to transmission to the intended recipient. The disclosed approach also enables a large amount of information to be filtered or curated to the most relevant information for a particular purpose, such as responding to a particular audience, message, intent, purpose, or goal of the sender user or the recipient user.

[0037] As described in greater detail below, embodiments of a generative message suggestion system include one or more of the following components: an input data collection subsystem, a data anonymizer subsystem, a training data formulation subsystem, a generator model subsystem, a scoring model subsystem, a message suggestion generation subsystem, a message generation interface, a message distribution service, a pre-sending feedback subsystem, and a post-sending feedback subsystem.

[0038] The input data collection subsystem can collect and output input data associated with a message sender, an intended message recipient, and / or one or more other entities associated with a message to be created, such as a company that can be hiring or posting a job. The input data collected by the input data collection subsystem can include a historical collection of messages previously created and sent by one or more message senders and associated acceptance data, which can be anonymized and used to train machine learning models of the generative message suggestion system. The data anonymizer subsystem can identify personally identifiable information (PII) in the input data and mask the PII so that it is not used for model training or other downstream processes. The training data formulation subsystem creates a training data set for machine learning models of the generative message suggestion system based on the anonymized input data.

[0039] The generator model subsystem includes a generator model that includes a machine learning model trained to generate message suggestions based on a training data set formulated by the training data formulation subsystem. The scoring model subsystem includes a scoring model that includes a machine learning model trained to score messages based on a training data set formulated by the training data formulation subsystem, the training data set including messages containing suggestions output by the generator model and messages created by a sender without suggestions provided by the generator model.

[0040] The message suggestion generation subsystem applies a trained generator model to a sender user, a message, a recipient user, or a particular explicit or implicit intent, purpose, or goal of an audience. The explicit or implicit intent, purpose, or goal can be represented by attribute data selected or input, for example, by the sender user, for example, at inference time, and one or more message suggestions are machine-generated and output based on the attribute data. One or more of the machine-generated message suggestions are presented to the sender via, for example, a message generation interface. In response to one or more of the presented message suggestions, the sender can create a message or modify the machine-generated suggested message content. The message distribution service can distribute the sender’s message to the intended recipient.

[0041] The message generation interface can communicate pre-send feedback to a pre-send feedback subsystem. The message distribution service can communicate post-send feedback to the post-send feedback subsystem. The pre-send feedback subsystem and the post-send feedback subsystem can each generate outputs that can act as proxies for expected outputs to the message suggestion generation subsystem or as labels or scores for actual outputs to the message suggestion generation subsystem. Pre-send feedback and / or post-send feedback can be used to measure the quality of machine-generated message suggestions output by the message suggestion generation subsystem, for example, in terms of acceptance probability, and to improve subsequent outputs of the message suggestion generation subsystem. For example, some or all of the feedback generated by the post-send feedback subsystem and / or the outputs of the pre-send feedback subsystem are returned to the generator model subsystem to tune the generator model based on the feedback. Additionally or alternatively, feedback generated by the pre-send feedback subsystem and / or the post-send feedback subsystem is provided to the scoring model to tune the scoring model based on the feedback. As a result of these and other aspects of the described generative message suggestion system, at least some of the message suggestions produced by the generative message suggestion system can facilitate efficient creation of electronic messages with improved acceptance probability while minimizing the laborious task of typing on a small form factor device or viewing and modifying content.

[0042] The disclosed generative message suggestion system components are configured in a manner that makes personalized message suggestion generation scalable. For example, previous attempts to facilitate personalized message creation have not been successful because they are not scalable due to the amount of human effort required to manually customize message content. In contrast, the disclosed technology includes scalable machine learning model architectures because, for example, the generator model and the scoring model are connected in a closed loop. As such, the generator model can be configured to simultaneously generate multiple different or alternative message suggestions so that a user can select from among the message suggestions. When multiple different message suggestions are machine generated for each sender at the same time, the number of potential message suggestions can quickly scale to, for example, accommodate highly active users. In environments where latency can be an issue, to reduce latency, these message suggestions can be stored in a message suggestion library for future use, reuse, or modification, and reuse on an individualized basis with each particular sender. For example, when a set of message suggestions is machine generated for a particular sender, the unused or unselected message suggestions can be stored in a real-time data store or near-line data store, for example, so that the message suggestions can be suggested or modified in real-time at any time when the particular sender initiates a subsequent message creation process.

[0043] Particular aspects of the disclosed technology are described in the context of generative models that output written pieces (i.e., natural language text). However, the disclosed technology is not limited to use in conjunction with text output. For example, aspects of the disclosed technology can be used to generate message suggestions in non-textual forms that include machine-generated output, such as digital images, videos, and / or audio.

[0044] Particular aspects of the disclosed technology are described in the context of non-public electronic messages that are distributed via a user network, a user connection network, or an application software system, such as a direct messaging feature of a social networking service. However, aspects of the disclosed technology are not limited to direct messaging or social networking services, but can be used to improve the generation of customized electronic messages with other types of software applications. Any network-based application software system can act as a user network or application software system to which the disclosed technology can be applied. For example, news, entertainment, and e-commerce applications installed on mobile devices, enterprise systems, messaging systems, search engines, workflow management systems, collaboration tools, and social graph-based applications can all act as application software systems that can use the disclosed technology.

[0045] The present disclosure will become more fully understood from the detailed description given herein below, taken in conjunction with the accompanying figures. The detailed description is explained with reference to the figures. The detailed description, given together with the figures, is to explain and understand, and should not be considered as limiting the present disclosure to the specific embodiments described.

[0046] In the accompanying drawings and the following description, reference may be made to components that have the same name but different reference numerals in different drawings. The use of different reference numerals in different drawings indicates that components with the same name may represent the same embodiment or different embodiments of the same component. For example, in some embodiments, components with the same name but different reference numerals in different drawings may have the same or similar functions, such that the description of one component with respect to one drawing can be applied to other components with the same name.

[0047] Similarly, the components shown and described in conjunction with some embodiments in the accompanying drawings and the following description can be used with or incorporated into other embodiments. For example, components shown in a particular drawing are not limited to use in conjunction with the embodiments described in the drawing, but can be used with or incorporated into other embodiments, including those shown in other drawings.

[0048] Figure 1 This is a flowchart of an example method for automated message suggestion generation using components of a generative message suggestion system, according to some embodiments of this disclosure. The method is executed by processing logic, which includes hardware (e.g., processing device, circuitry, dedicated logic, programmable logic, microcode, device hardware, integrated circuits, etc.), software (e.g., instructions that run or execute on the processing device), or a combination thereof. In some embodiments, the method is performed by… Figure 5 The generative message suggestion system 580 is executed by components, in some embodiments including Figure 5 The possibility shown is Figure 1 Components not specifically shown. Although shown in a particular sequence or order, the order of processes can be modified unless otherwise stated. Therefore, the illustrated embodiments should be understood as examples only, and the illustrated processes can be executed in different orders, and some processes can be executed in parallel. In addition, at least one process can be omitted in various embodiments. Therefore, not all processes are required in every embodiment. Other process flows are also possible.

[0049] exist Figure 1 The example shows a computing system 100 including a generative message suggestion system 108. Figure 1 The generative message recommendation system 108 includes a data anonymizer 110, a model input specifyer 112, a generator model 114, a scoring model 116, and a model output evaluation interface 122. Figure 1In the example of FIG. 1, the components of the generative message suggestion system 108 are implemented using an application server or server cluster that can include a secure environment (e.g., a secure enclave, an encryption system, etc.) for processing of message data. The generative message suggestion system 108 is in bi-directional communication with the message generation interface 126 via a network. In some embodiments, the message generation interface 126 includes a front-end user interface functionality that is considered part of the generative message suggestion system 108. From time to time, messages created at the message generation interface 126 are transmitted to the message reception interface 140.

[0050] As indicated in Figure 1 In some implementations, the components of the computing system 100 are distributed across multiple different computing devices (e.g., one or more client devices, application servers, web servers, and / or database servers) that are connected via a network. In other implementations, at least some of the components of the computing system 100 are implemented on a single computing device, such as a client device. For example, in some implementations, some or all of the generative message suggestion system 108 is implemented directly on a user’s client device, thereby avoiding the need to communicate over a network, such as the Internet, with a server.

[0051] To create and operate the various parts of the generative message suggestion system 108, input data can be collected from a number of different sources. The input data 106 can include message activity data 106a, profile data 106b, and entity connection data 106c. The input data 106 can be provided to the generative message suggestion system 108 from potentially a variety of different data sources, including user interfaces, databases, and other types of data stores, including online, real-time, and / or offline data sources. In Figure 1 In the example of FIG. 1, the message activity data 106a is received via one or more user devices or systems, such as portable user devices like smartphones, wearable devices, tablet computers, or laptop computers; the profile data 106b is received via one or more web servers; and the entity connection data 106c is received via one or more database servers; however, any different type of input data 106 can be received by the generative message suggestion system 108 via any type of electronic machine, device, or system.

[0052] Examples of message activity data 106a include messages that have previously been created by senders to intended recipients via messaging systems or application software systems that operate messaging systems, such as social networking services that operate direct messaging functionality. For messages of a particular sender, the message activity data 106a includes data indicating how the intended recipient responded to the message, e.g., whether the sender’s message was accepted, rejected, or ignored by the intended recipient.

[0053] For particular messages, the message activity data 106a can also include interaction data associated with the sender and / or the intended recipient. For example, the message activity data 106a can include a history of interactions between the sender and the intended recipient, a history of interactions of the sender with other entities on the social network (e.g., other users, content items, job postings, company pages, skill pages, etc.), and / or a history of interactions of the intended recipient with other entities on the social network (e.g., other users, content items, job postings, company pages, skill pages, etc.). Examples of interactions include creating documents, messages, posts, articles, images, video files, audio files, multimedia files, digital reactions (e.g., likes, comments, shares, etc.), requests (e.g., follow requests, connection requests, etc.), search history, and transaction history (e.g., online submission of job applications, e-commerce transactions, etc.).

[0054] The message activity data 106a can include image or video content. The message activity data 106a containing text, image, audio, and / or video content can be user-created or machine-generated, e.g., by a generative model. The message activity data 106a can be obtained by the generative message suggestion system 108 via a user interface such as the message generation interface 126 and / or retrieved from one or more data stores such as searchable databases storing historical information about the use of the messaging system or application software systems operating the messaging system. The message activity data 106a can include structured data such as data input by a user into an online form that enforces one or more input rules constraining the values and / or format of the input and / or unstructured data such as natural language text, audio, or transcription.

[0055] Examples of the profile data 106b include user experience, interests, areas of expertise, education history, job titles, skills, job history, etc. The profile data 106b can be obtained by the generative message suggestion system 108 by, e.g., querying one or more data stores storing entity profile data for the messaging system or application software systems operating the messaging system.

[0056] Examples of entity connection data 106c include data extracted from entity graph 103 and / or knowledge graph 105. Entity graph 103 includes entity profile data arranged according to a connection graph, e.g., a graph of connections and relationships between users of a user connection network and between users and other entities. For example, entity graph 103 represents entities as nodes and relationships between entities as edges between nodes. In some implementations, entity graph 103 includes cross-application knowledge graph 105. Cross-application knowledge graph 105 is a subset of entity graph 103 or a superset of entity graph 103 (e.g., a combination of multiple entity graphs) that links data from a user connection network with data from other application software systems, such as a user connection network or a search engine. Examples of entity graphs or cross-application knowledge graphs are shown in Figure 6

[0057] An entity, as used herein, can refer to a user of the messaging system or an application software system that operates the messaging system, or another type of entity, such as a company, organization, or institution, or a digital content item, such as an article, post, comment, share, or job posting. For example, in a user connection network, an entity can include or reference a web page that a user of the user connection network can interact with, where the web page is configured to display a digital content item, such as an article, post, message, profile of another user, or profile data related to a company, organization, institution, or job posting. In some implementations of entity graph 103, 112, an activity is represented as an entity. An activity, as used herein, can refer to a network activity, such as a digital communication between a computing device and a system. Examples of network activities include initiating a session with an application software system by, e.g., logging into an application, initiating a page load to load a web page into a browser, creating, editing, sending, receiving, viewing, and interacting with a message, uploading, downloading, creating, and sharing a digital content item on a network, inputting or performing a search query, and performing a social action, such as connecting or following another user, adding a comment to an article or post on a network, and / or inputting a social reaction.

[0058] Entity connection data 106c is extracted from an application software system that operates entity graph 103 and knowledge graph 105, e.g., by traversing entity graph 103 or knowledge graph 105, e.g., by executing one or more queries on one or more data stores that store data associated with nodes and edges of entity graph 103 or knowledge graph 105.

[0059] As indicated by the legend in Figure 1 Multiple different operational flows are shown for generative message suggestion system 108, including a training or tuning flow, a feedback flow, and an online flow, as indicated by the legend in

[0060] ​The training or tuning process configures the generator model 114 to include correlations between message elements and acceptance rates. For example, the generator model 114 includes correlations between attribute data, such as entity data, and message acceptance rates. In some implementations, the historical message activity data 106a is anonymized using an automated process that does not involve human review of the message activity data 106a, and the anonymized data is used as training data. In other implementations, training data is synthesized using, for example, a large language model.

[0061] In implementations where historical message activity data 106a is used to formulate training data, the data anonymizer 110 anonymizes the message activity data 106a before it is used for training or shared with other components of the generative message suggestion system 108. In the training or tuning process, the data anonymizer 110 pre-processes potentially sensitive input data 106 (e.g., message content and / or metadata) so that it can be used as training data for machine learning models of the generative message suggestion system 108. The data anonymizer 110 identifies personally identifiable information (PII) in the input data 106 and replaces or masks the PII with non-PII labels using a delexicalization process based on, for example, named entity recognition techniques. The data anonymizer 110 creates anonymized input data 111 based on the received input data 106.

[0062] In some implementations, the data anonymizer 110 includes a private named entity recognition (NER) model. The private NER model redacts any personally identifiable information (PII) from message data, such as names, titles, phone numbers, and email addresses, and delexicalizes any PII in the message data. Delexicalization includes a process of replacing PII values with corresponding semantic labels. For example, “John Adams” is replaced with [NAME]. As another example, an attribute identified in message text (e.g., software engineer) is replaced with a word that only references the type of attribute (e.g., TITLE replaces software engineer). The NER labels provide information about which entities exist in the message data without revealing the PII to ensure that inferences are not made based on such information. In some implementations, the delexicalization process is only applied to PII. Non-PII entity data, such as skills, is not replaced with NER labels by the data anonymizer 110. Including the original data values for non-PII entity data improves the ability of the generator model to learn correlations between unlabelled entity data and other parts of the input data. In some implementations, the data anonymizer 110 uses string matching or string search techniques to identify entities and attributes in message text and / or other parts of the input data 106.

[0063] To illustrate the operation of the data anonymizer 110, Table 1 below shows an example of message data before and after processing by the data anonymizer 110. Table 1. Data anonymization example.

[0064] In Table 1, the first column shows an example of message text before delexicalization by the data anonymizer 110, e.g., an example of the input data 106, and the second column shows the same example after delexicalization, e.g., an example of the anonymized input data 111.

[0065] The data anonymizer 110 outputs the anonymized input data 111 for processing by the model input formulator 112. In a training or tuning process, the model input formulator 112 formulates training data for the machine learning models of the generative message suggestion system 108 (e.g., the generator model 114 and the scoring model 116) based on the anonymized input data 111 and / or based on synthetic training examples of message text.

[0066] In some implementations, the model input formulator 112 uses heuristics to map labels added to the input data 106 by the NER model to standardized attribute labels or tokens. The standardized attribute labels are used to create an input context for the message, which can be used to train or tune the generator model 114. For example, labels output by the NER model labels can include generic labels such as [NAME_1], [NAME_2]. These labels can be further mapped to, e.g., [SENDER_NAME] and [RECIPIENT_NAME] to enhance model training, such that as a result, the generator model 114 can distinguish between the two different types of names.

[0067] After mapping the NER labels to standardized labels using heuristics, the model input formulator 112 uses a machine learning classifier, such as a few-shot classifier, to extract parts and entities from the message text and include these extracted items in the input context. The resulting input context provides a summary or outline of the message text, including its structure and semantic content. Table 2 below shows an example of an input context for the example of message text shown in Table 1. Table 2. Example of input context.

[0068] In Table 2, bracketed text indicates message parts determined using the machine learning classifier, all-caps text indicates standardized labels, and remaining text indicates untokenized portions of the message text. As shown in Table 2, the input context includes part information for each line of the message text, followed by the relevant attributes.

[0069] The input context preserves the order of the parts as in the original message text. During model training, the input context provides the generator model with a message structure definition that the model can use to generate message suggestions with the same or similar message structure. Thus, for example, generator model 114 can be trained to generate message content with a message structure that is the same as or similar to other messages that a particular sender has historically created, thereby providing a high degree of customization to the machine-generated message suggestions, such as sender-side personalization. As another example, model input staging 112 can link the input context to the corresponding message text so that when the scoring model 116 scores the message text, the input context is associated with the obtained score. In this way, different input contexts (e.g., message structure and / or semantic content) can be associated with the probability of acceptance.

[0070] To facilitate the prediction of acceptance probabilities, model input specifier 112 extracts historical acceptance data associated with historical message examples from input data 106. For example, model input specifier 112 determines, based on message activity data 106a, whether each example of historical message text is included in a message that was accepted, rejected, or ignored by the expected recipient. Examples of instances of training data that can be specified by model input specifier 112 include anonymized message metadata (e.g., anonymized sender and receiver data), anonymized message text (e.g., the delexicalized body of the message), input context, and corresponding acceptance data (e.g., a value of 0 indicating that the message was accepted by the recipient, and a value of 1 indicating that the message was rejected or ignored by the expected recipient).

[0071] The model input specifyer 112 creates and output training datasets for each of the generator model 114 and the scoring model 116, including generator model training data 113 and scoring model training data 115. Generator model training data 113 and scoring model training data 115 can include different training instances. For example, positive training examples (e.g., examples where the message is accepted by the receiver) can be included in both generator model training data 113 and scoring model training data 115, while negative training examples (e.g., examples where the message is rejected or ignored by the expected receiver) can be included only in scoring model training data 115 and not in generator model training data 113.

[0072] exist Figure 1 In the examples, both generator model 114 and scoring model 116 are configured using encoder-decoder model architectures, such as those including a bidirectional encoder and an autoregressive decoder. Other implementations can use different model architectures, as referenced in this paper. Figure 7 As described.

[0073] In some implementations, the generator model 114 includes an instance of a text-based encoder-decoder model that accepts a string as input and returns a single string as output. More specifically, the generator model 114 includes a self-recursive model (e.g., an order model that generates an output word based on the words it has already generated until it reaches a special end word). During training of the generator model 114, the input string includes an instance of the generator model training data 113.

[0074] The generator model 114 can be trained and tuned using one or more training and tuning processes. In some implementations, the generator model 114 includes a pre-trained language model that can generate text recursively. The generator model 114 model can be tuned using, for example, task-agnostic and task-specific training on a domain-specific dataset.

[0075] In some implementations, the generator model 114 is tuned based on prefix language modeling (PLM) to improve the model’s ability to generate suggested message content and understand the relevance between domain-specific entities. For example, in the domain of job postings, the PLM can be used to improve the generator model 114’s ability to understand the relevance between job titles and skills, such as a machine learning engineer working at a particular company will likely have the skill in a particular programming language or platform commonly used at that company. In some implementations, the downstream task (e.g., the task of assigning the model to complete an incomplete message) is pre-trained using a PLM with a text fill denoising objective method by which the model learns to generate missing tokens by replacing spans of input text with a single sentinel token.

[0076] In some implementations, supervised fine-tuning of the generator model 114 for domain adaptation and task adaptation is performed based on input context and delexicalized message data to configure the generator model 114 to generate customized suggested message content. This fine-tuning is performed using, for example, a seq2seq training process in which the input sequence to the model includes attribute data for the sender and attribute data for entities of the subject (e.g., a job) that the message as a sender wants to create, and the target sequence is a delexicalized message. To fine-tune the seq2seq model, the input context and the output sequence are used, where the output sequence is a delexicalized message and the input context includes message parts and attributes (e.g., message structure and semantic content).

[0077] During inference or suggestion generation by the generator model 114, the model input includes the attribute data 128, and the model output includes the machine-generated message suggestion 130. For example, the sender preferences, the intended recipient preferences, and the feature values are encoded into the input string, and the generator model 114 performs conditional content generation to output suggested message content based on the encoded preferences.

[0078] Personalization of the suggested message content for the sender, the subject of the message, and / or the intended recipient is achieved by conditioning the content generation of the model on the intended recipient data (e.g., the profile data 106b of the intended recipient), the subject of the message (e.g., the job posting), the match between the subject of the message (e.g., the job requirements) and the background of the intended recipient (e.g., the profile data 106b), and the sender’s preferences (e.g., based on the sender’s profile information 106b and / or message activity data 106a). In this way, the generator model 114 is configured to generate message suggestions that are personalized for the particular domain and intended recipient, including suggested message content that is personalized for the particular sender (e.g., matching the sender’s preferred style, structure, and tone).

[0079] A match or matching, as used herein, can refer to an exact match or an approximate match, e.g., a match based on a computation of similarity between two data segments. An example of a similarity computation is cosine similarity. Other schemes that can be used to determine similarity between or among data segments include clustering algorithms (e.g., k-means clustering), binary classifiers trained to determine whether two items in a pair are similar or dissimilar, and neural network-based vectorization techniques such as WORD2VEC. In some implementations, generative language models are used to determine similarity of data segments.

[0080] In some implementations, the generator model 114 incorporates a flexible sampling method into its decoding strategy to diversify the message suggestions output by the generator model 114 (e.g., so that the same or similar message suggestions are not repeatedly presented to the same sender). Sampling methods like kernel sampling or top-k sampling can result in different outputs for the same input because they randomly select tokens from the set of high-probability tokens generated by the language model, rather than always taking the token with the highest probability. This results in multiple but different sequences for the same input, resulting in different outputs. To balance diversity and factual correctness in these sampling methods, a sampling threshold or k value is selected to ensure that the generated output is grammatically and factually correct, while still allowing some diversity in the generated text.

[0081] The trained or tuned generator model 114 is made accessible by the message generation interface 126, e.g., as a cloud-based hosting service for model inference. The hosting environment for the generator model 114 is configured to keep latency low to support online services for message suggestions in real-time.

[0082] The scoring model 116 is configured to generate and output a score for a message, including messages created by senders, without using message suggestions that include or are based on message suggestions generated by the generator model 114 and messages, where the score indicates a likelihood of message acceptance or acceptance probability.

[0083] In some implementations, the scoring model 116 includes an encoder-decoder model architecture similar to the generator model 114 but trained differently. For example, in some implementations, the generator model 114 and the scoring model 116 are different instances of an encoder-decoder model. In some implementations, the scoring model 116 includes a transformer model. In some implementations, the scoring model 116 performs binary classification to output a probability of message acceptance for a given message suggestion generated by the generator model 114.

[0084] In some implementations, the scoring model 116 is trained on a large corpus of anonymized historical messages and their corresponding labels (e.g., acceptance data) to predict the probability of message acceptance and to identify key phrases and / or sentences that impact message acceptance. In some implementations, the scoring model includes a hierarchical attention model that first learns to encode each sentence in a message text and then further encodes the sequence of these sentence encodings into a final representation, followed by a classifier head that predicts the probability of acceptance of the message based on the final representation. In some implementations, the scoring model 116 is trained in this way to identify key phrases within a message, e.g., attributes or snippets of text from a message that are key determinants of the probability of acceptance.

[0085] In Figure 1In the example of FIG. 1, the output of the generator model 114 (e.g., generator model output 118) is connected to the scoring model 116, and the output of the scoring model 116 (e.g., scoring model feedback 120) is connected to the generator model 114. In this way, the scoring model 116 can be used to predict the probability of acceptance of a message created based on the generator model output 118 by an intended recipient and / or to identify key phrases in the message suggestion text. This information can be utilized to further tune the generator model 114. For example, if the scorer model 116 predicts a low chance of message acceptance, a message suggestion specific to the particular phrase that is contributing to the low probability of acceptance can be generated by the generator model 114 and presented to the sender. Similarly, if the scorer model predicts a high probability of acceptance of a message written by the sender without any prior message suggestion, the generator model 114 can not be used, as the high probability of acceptance indicates that a message suggestion can not be needed. In this way, the scoring model 116 can be used to optimize the use of the generator model 114 only for those scenarios where the sender’s message or the generator model output 118 has a low probability of acceptance.

[0086] The generative message suggestion system 108 is enabled to access the trained or tuned scoring model 116, e.g., as a cloud-based hosted service. The hosting environment for the scoring model 116 is configured for online use, such that the output from the scorer model can be used as additional input to the generator model 114 for message suggestion generation. For example, if the score output by the scoring model 116 for a particular message suggestion generated by the generator model 114 is below a threshold, the generator model 114 can generate another message suggestion taking into account the score.

[0087] Alternatively or additionally, the output of the scoring model 116 (e.g., scoring model feedback 120) can be surfaced to the sender via the message generation interface 126 along with the corresponding message suggestion 130, or even while the sender is composing a message via the message generation interface 126. For example, the message generation interface 126 can display the changes in predicted probability of acceptance in real-time as the sender composes and edits a message.

[0088] Before the message suggestions 130 are returned to the message generation interface 126, the model output evaluation interface 122 performs post-processing on the generator model output 118 and / or the scoring model feedback 120. The model output evaluation interface 122 can include automated evaluation processes and / or human review processes. For example, the model output evaluation interface 122 filters the generator model output 118 for inappropriate language or irrelevant information. Alternatively or additionally, the generator model output 118 can be evaluated using heuristics and / or metrics for coherence, coverage, diversity, etc. As another example, human evaluation can be used to review and filter the generator model output 118 for subjective aspects such as hallucination and engagement. The results of the automated and / or human evaluation can be formulated as evaluation feedback 123, 124 and passed back to the generator model 114 and / or the scoring model 116 for further tuning of these models.

[0089] The message generation interface 126 includes a front-end component through which a sender can interact with the generative message suggestion system 108 at the sender’s electronic device. In some cases, the message generation interface 126 receives the message suggestions 130 from the generative message suggestion system 108 and presents the message suggestions 130 to the sender along with the scoring model feedback 120 in the context of message composition for an intended recipient. The message generation interface 126 passes the attribute data 128 to the generative message suggestion system 108 upon sender initiation. For example, the sender selects, via the message generation interface 126, particular attributes (e.g., title, skills, employing company, location) to be included in the message, and the message generation interface 126 passes the attribute data 128 selected by the sender to the generative message suggestion system 108.

[0090] In response to the message suggestions 130 and / or the scoring model feedback 120, the message generation interface 126 can generate pre-send feedback 132, such as sender interaction with the message suggestions 130, and pass the pre-send feedback 132 to the generative message suggestion system 108 for the purpose of improving the generator model 114 and / or the scoring model 116.

[0091] In response to the message suggestions 130 and / or the scoring model feedback 120, the message generation interface 126 can initiate sending of an AI-assisted message 134 based on the message suggestions 130 to the intended recipient. The intended recipient can receive and process the AI-assisted message 134 at a message reception interface 140 at the intended recipient’s electronic device. The message reception interface 140 can generate post-send feedback 136, such as intended recipient interaction with the AI-assisted message 134 via the message reception interface 140, and pass the post-send feedback 136 to the generative message suggestion system 108 for the purpose of improving the generator model 114 and / or the scoring model 116.

[0092] For purposes of illustration, the examples shown in Figure 1 and the accompanying description are provided above. The present disclosure is not limited to the described examples. Additional or alternative details and implementations are described herein.

[0093] Figure 2 is a timing diagram showing an example of communication between a message generation interface and a generative message suggestion system according to some embodiments of the present disclosure.

[0094] In Figure 2 , the communications denoted by labeled arrows occur in a temporal sequence, e.g., the attribute suggestion (1) communication from the generative message suggestion system 108 occurs at a first time instance, and the attribute data (1) communication from the message generation interface 126 occurs at a second time instance after the first time instance.

[0095] In Figure 2 , the communications between the components shown in include, e.g., network communications and / or on-device communications. For example, all or portions of the message generation interface 126, the generative message suggestion system 108, and the message reception interface 140 can be implemented on a single device or across multiple devices. For example, embodiments can generate and suggest messages on a user’s client device based on attribute data (where the attribute data can be obtained from the user’s device and / or one or more other devices (e.g., servers or databases)), and the user’s client device can also receive message suggestions and / or suggestions of attribute data from, e.g., an external database that stores historical data related to correlations between attributes and message acceptance (e.g., statistics about characteristics of messages related to successful acceptance).

[0096] In Figure 2 , the generative message suggestion system 108 initiates the interaction flow with the message generation interface 126 by transmitting the attribute suggestion (1) to the message generation interface 126. The attribute suggestion (1) includes, e.g., suggested attributes related to high probability of acceptance if those attributes are included in a message. In response to the attribute suggestion (1), the message generation interface 126 generates the attribute data (1), e.g., by receiving and processing a sender’s selection from the attribute suggestion (1), and transmits the attribute data (1) to the generative message suggestion system 108.

[0097] In response to the attribute data (1), the generative message suggestion system 108 uses the techniques described herein to machine-generate a generative message draft (1) based on the attribute data (1) and transmit the generative message draft (1) to the message generation interface 126. For example, the generative message draft (1) includes a draft of the body of a message (e.g., natural language text) that includes the attribute data (1) and has been generated based on the output of a generator model, such as the generator model 114.

[0098] Based on the output of a scoring model, such as the scoring model 116, the generative message suggestion system 108 generates and transmits a message personalization suggestion (1) to the message generation interface 126. For example, the message personalization suggestion (1) includes one or more message suggestions that are highly relevant to the acceptance probability based on attribute data and / or preferences of the sender, the intended recipient, or the subject matter of the message.

[0099] In response to the message personalization suggestion (1), the message generation interface 126 generates attribute data (2). For example, the message generation interface 126 receives and processes a selection from the message personalization suggestion (1) by the sender and transmits the attribute data (2) to the generative message suggestion system 108.

[0100] In response to the attribute data (2), the generative message suggestion system 108 machine-generates a generative message draft (2) based on the attribute data (2) and transmits the generative message draft (2) to the message generation interface 126. For example, the generative message draft (2) includes a modified (e.g., reworded or rephrased) version of the generative message draft (1) or an alternative message draft that has been machine-generated by a generator model, such as the generator model 114, based on the attribute data (2). For example, the generative message draft (2) can have a different tone, structure, or writing style than the generative message draft (1), or can include references to the attribute data (2) in a different order than the attributes mentioned in the generative message draft (1). As an example, an attribute mentioned in the first paragraph of the generative message draft (1) can not be mentioned until the second paragraph of the generative message draft (2), or vice versa.

[0101] In response to the generative message draft (2), the message generation interface 126 generates and sends pre-send feedback (1) to the generative message suggestion system 108. For example, the message generation interface 126 receives and processes an interaction by the sender with the generative message draft (2), including an interaction indicating that the sender approves a message based on the generative message draft (2) for sending to the intended recipient. In addition to the sender’s approval signal, the pre-send feedback (1) can also include the body of the message approved by the sender.

[0102] In response to the pre-send feedback (1), the message generation interface 126 formulates an artificial intelligence (AI) assisted message (1) and causes the AI assisted message (1) to be transmitted at the device of the intended recipient to the message reception interface 146 (via a message distribution service, such as the message distribution service 724, herein referred to as the message distribution service 724). Figure 7 described but not shown in Figure 2 formulates an AI assisted message (1) and causes the AI assisted message (1) to be transmitted at the device of the intended recipient to the message reception interface 146.

[0103] In response to the AI assisted message (1), the message reception interface 140 generates a response to the AI assisted message (1) and transmits the response to the AI assisted message (1) to the message generation interface 126 (via the message distribution service, not shown in Figure 2 For example, the message reception interface 140 receives an interaction from the intended recipient indicating that the AI assisted message (1) has been accepted and read and transmits a notification of the message acceptance to the message generation interface 126 of the sender. Based on the response to the AI assisted message (1), the message reception interface 140 formulates post-send feedback (1) and transmits the post-send feedback (1) to the generative message suggestion system 108. For example, the message reception interface 140 formulates the post-send feedback (1) by joining the response to the AI assisted message (1) with the AI assisted message (1).

[0104] For illustrative purposes, the examples shown in Figure 2 and the accompanying description are provided. The present disclosure is not limited to the described examples. Additional or alternative details and implementations are described herein.

[0105] Figure 3A , Figure 3B , Figure 3C , Figure 3D , FIGS. 3E, 3F, 3G, and 3H illustrate examples of at least one flow of screen captures of user interface screens including being configured to create electronic messages based on at least one AI generated message suggestion, in accordance with some embodiments of the present disclosure. Figure 3A , Figure 3B , Figure 3C , Figure 3D , FIGS. 3E, 3F, 3G, and 3H illustrate user interface flows or sequences of user interface views that can be presented to a message sender to assist the sender by machine generating and outputting one or more customized message suggestions. Figure 3A , Figure 3B , Figure 3C , Figure 3D , FIGS. 3E, 3F, 3G, and 3H each illustrate examples of user interface screens that can be used to facilitate message composition using the automated message suggestion generation techniques described herein.

[0106] In some implementations, in response to the pre-send feedback (1), the message generation interface 126 formulates an AI assisted message (1) and causes the AI assisted message (1) to be transmitted at the device of the intended recipient to the message reception interface 146.Figure 3A , Figure 3B , Figure 3C , Figure 3D One or more of the user interfaces shown in Figures 3E, 3F, 3G, and 3H display predictive data associated with machine-generated message suggestions. For example, some implementations generate and display rating values, such as rankings or percentages adjacent to or related to message suggestions or attributes, where these rating values ​​indicate to the sending user the likelihood that the selection of a message suggestion or attribute will lead to successful message acceptance by the recipient user. As another example, while a user is drafting a message and before it is sent to the intended recipient, some implementations generate and display similar rating values, such as rankings or percentages adjacent to or relative to messages the user has already created, where these rating values ​​indicate to the sending user the likelihood that the user's message will lead to successful message acceptance by the recipient user.

[0107] exist Figure 3A , Figure 3B , Figure 3C , Figure 3D Figures 3E, 3F, 3G, 3H and Figure 4 In the user interface shown, for the purposes of this disclosure, specific data that will typically be displayed has been anonymized. In the live example, the actual data will be displayed instead of the anonymized version. For example, the text “JobTitle” will be replaced with the actual job title (e.g., Software Engineer), and “FirstName LastName” will be replaced with the user’s actual name.

[0108] exist Figure 3A , Figure 3B , Figure 3C , Figure 3D Figures 3E, 3F, 3G, 3H and Figure 4 The user interface shown is presented by an application software system, such as a user network and / or messaging system, to a user who wants to create and send messages to a intended recipient via the network. In some implementations, the user interface is implemented as a webpage, for example, stored on a server or in a cache on the user's device, and then loaded onto the user's device's display via the user device sending a page load request to the server. The icons, selections, and arrangement of the elements shown in the user interface are copyright 2023 LinkedIn Corporation, all rights reserved.

[0109] The graphical user interface control elements (e.g., fields, boxes, buttons, etc.) shown in the screen captures are implemented via software used to construct the user interface screens. Although the screen captures illustrate examples of user interface screens, e.g., visual displays of numbers, e.g., in online forms or web pages, the present disclosure is not limited to online form or web page implementations, visual displays, or graphical user interfaces. In other implementations, for example, an automated chat bot is used to fill in a form, where the chat bot requests that the user input requested information via a conversation, natural language dialogue, or message-based format using text and / or spoken audio received via a microphone embedded in the computing device.

[0110] Figure 3A FIGURE 1 illustrates an example of a screen capture of a user interface for viewing user profile data 102 for an intended message recipient, according to some embodiments of the present disclosure. Figure 3A The user interface 100 enables a sender to initiate creation and sending of a message to an intended recipient whose profile data is displayed in the user interface 100.

[0111] In the example of FIGURE 1, message sending statistics 106 are presented. For example, in the field of job recruitment, the sender can be a recruiter, and the message sending statistics 106 assist the recruiter in tracking the status of recruitment-related communications with respect to a job prospect. The user interface 100 includes an interactive control element 104, e.g., a message initiation mechanism. User selection of the message initiation mechanism results in a transition from the user interface 100 to the user interface 108 shown in FIGURE 2. Figure 3A In the example of FIGURE 1, message sending statistics 106 are presented. For example, in the field of job recruitment, the sender can be a recruiter, and the message sending statistics 106 assist the recruiter in tracking the status of recruitment-related communications with respect to a job prospect. The user interface 100 includes an interactive control element 104, e.g., a message initiation mechanism. User selection of the message initiation mechanism results in a transition from the user interface 100 to the user interface 108 shown in FIGURE 2.

[0112] Figure 3B FIGURE 3 illustrates an example of a screen capture of a user interface for facilitating composition of a message by a sender with respect to an intended recipient whose profile data is shown in the user interface 300, according to some embodiments of the present disclosure. Figure 3B The user interface 300 includes an inactive window 312 that continues to show the profile data of the intended recipient in the background while the sender is composing the message, and an active window 314 that allows the sender to compose the message to the intended recipient while still viewing the profile data of the intended recipient in the background.

[0113] The active window 314 includes a search input mechanism 315, an address input mechanism 316, a subject input mechanism 318, and a message input mechanism 319. The search input mechanism 315 enables the sender to input one or more search criteria to search for a predefined message template. The address input mechanism 316 enables the sender to select or input a name or address of an intended recipient. The subject input mechanism 318 enables the sender to input a subject of the message. The message input mechanism 319 enables the sender to input message content (e.g., a body of the message) to the intended recipient by, for example, typing, copying and pasting, editing, or otherwise composing the message content.

[0114] The active window 314 also includes a draft personalized message mechanism 317. The draft personalized message mechanism 317 enables the sender to utilize the generative message suggestion techniques described herein to assist in composing a message. In Figure 3B the example, a user selection of the draft personalized message mechanism 317 results in a transition from the user interface 310 to the user interface 330 shown in Figure 3D

[0115] In Figure 3C the user interface 320 is shown. The user interface 320 includes an inactive window 322 and an active window 324, the inactive window 322 persistently displays profile data for an intended recipient of the sender. A selection of the “(i)” mechanism adjacent to the draft personalized message mechanism 317 in the user interface 310 results in a transition from the user interface 310 to the user interface 320. In response to the selection of the “(i)” mechanism in the user interface 310, the active window 324 displays a notification 326. The notification 326 notifies the sender that a selection of the draft personalized message mechanism 317 invokes artificial intelligence (AI)-based message creation assistance.

[0116] In Figure 3D the user interface 330 is shown. The user interface 330 includes an inactive window 332 and an active window 334, the inactive window 332 persistently displays profile data for an intended recipient of the sender. In response to a user selection of the draft personalized message mechanism 317, the generative message suggestion system begins drafting personalized message suggestions using the techniques described herein, and the active window 334 displays one or more indications that the generative message suggestion system has been invoked.

[0117] ​In FIG. 3E, a user interface 340 is shown. The user interface 340 includes an inactive window 342 and an active window 344, the inactive window 342 persistently displaying profile data for the intended recipient of the sender. The active window 344 is populated in response to completion of the generative message suggestion system generating a message suggestion in response to user selection of the draft personalized message mechanism 317. The active window 344 includes a message header 345, an interactive notification 348, a message body 346, and a save mechanism 349.

[0118] In the example of FIG. 3E, the message header 345 and the message body 346 contain text that has been machine-generated by the generative message suggestion system using the techniques described herein. The generator model of the generative message suggestion system generates and customizes the message header 345 and the message body 346 based on the profile data of the intended recipient, the profile data of the sender, and / or information about the subject of the message (e.g., the job opportunity), in addition to the output of the scoring model of the generative message suggestion system that includes the acceptance probability data.

[0119] The interactive notification 348 alerts the sender that the message header 345 and the message body 346 contain AI-generated content. The interactive notification 348 includes a selectable element (e.g., “further personalize this message”). The save mechanism 349 enables the sender to store the message body 346 as a message template for future reuse.

[0120] The selectable element of the interactive notification 348, if selected by the sender, causes a transition to a user interface 350 of FIG. 3F. In the example of FIG. 3F, the user interface 350 includes an inactive window 352 and an active window 354, the inactive window 352 persistently displaying profile data for the intended recipient of the sender. The active window 354 displays a previously generated message body 356, a plurality of additional personalization suggestions 358, 360, 364, 368, and a re-draft mechanism 370 in response to user selection of the selectable element of the interactive notification 348.

[0121] The additional personalization suggestions 358 include suggested attributes related to the sender. These suggested attributes can be obtained, for example, by traversing an entity graph as described herein. In the example of FIG. 3F, the additional personalization suggestions 358 are inactive because the generative message suggestion system requires these attributes for message generation and the sender cannot deselect them.

[0122] Further personalization suggestions 360 include suggested attributes related to the subject matter of the message (e.g., professional attributes). These suggested attributes can be obtained, for example, by traversing an entity graph as described herein. In the example of FIG. 3F, further personalization suggestions 360 include some attributes, such as attribute 362, that are inactive because the generative message suggestion system requires these attributes for message generation and the sender is unable to deselect them. Further personalization suggestions 360 include other attributes, such as skills, location, workplace type, compensation, and employment type, that are active so that the sender is able to select them for inclusion in the subsequent further personalized machine-generated message.

[0123] Further personalization suggestions 364 include suggested attributes related to the intended recipient. These suggested attributes can be obtained, for example, by traversing an entity graph as described herein. In the example of FIG. 3F, further personalization suggestions 364 include some attributes, such as experience attribute, that are inactive because the generative message suggestion system requires these attributes for message generation and the sender is unable to deselect them. Further personalization suggestions 364 include other attributes, such as open job attribute 366, that are active so that the sender is able to select them for inclusion in the subsequent further personalized machine-generated message.

[0124] Further personalization suggestions 368 include suggested attributes related to entities associated with the subject matter of the message, for example, attributes related to the company that employs the position described by attribute 362. These suggested attributes can be obtained, for example, by traversing an entity graph as described herein. In the example of FIG. 3F, further personalization suggestions 368 include attributes that are displayed in a selected mode, such as about the company attribute, because the attribute has been selected but can be deselected by the sender. For example, the about the company attribute has toggle capability so that it can be alternately selected or deselected by the sender. Attributes that are not currently selected, such as attribute 366, can be implemented in a similar manner with toggle capability.

[0125] In FIG. 3F, user selection of the re-draft mechanism 370 causes a transition to user interface 380 of FIG. 3G. In FIG. 3G, user interface 380 includes inactive window 382 that continues to display profile data for the intended recipient of the sender and active window 384. In active window 384, the previously machine-generated message 386 is still displayed, but active window 384 is updated to show the sender’s revised attribute selections for further personalization. For example, location attribute 388 is displayed as selected in active window 384, whereas the same attribute is displayed as not selected in FIG. 3F.

[0126] In FIG. 3G, user selection of the re-drafting mechanism causes a transition to user interface 390 of FIG. 3H. In FIG. 3H, user interface 390 includes an inactive window 392 and an active window 394, with inactive window 392 persistently displaying profile data for the intended recipient of the sender. In active window 394, a machine-generated redrafted version of message body 396 is displayed along with a status of further personalization attribute options. In the example of FIG. 3G, the opening paragraph of the message has been revised (e.g., reworded or rephrased) by the generative message suggestion system using techniques described herein to, for example, include more personalized information about the sender as an introduction. Additionally, the second paragraph has been revised by the generative message suggestion system using techniques described herein. For example, message body 396 has been redrafted according to the sender’s preferred message tone, style, and structure (e.g., based on the input context).

[0127] For illustrative purposes, examples are provided in Figure 3A , Figure 3B , Figure 3C , Figure 3D FIGS. 3E, 3F, 3G, 3H, and the accompanying description above. For example, although examples are illustrated as user interface screens for larger form factors such as desktop or laptop devices, the user interfaces can be configured for other forms of electronic devices such as smartphones, tablet computers, and wearable devices. The present disclosure is not limited to the described examples. Additional or alternative details and implementations are described herein.

[0128] Figure 4 FIGS. 3E, 3F, 3G, 3H, and the accompanying description above. For example, although examples are illustrated as user interface screens for larger form factors such as desktop or laptop devices, the user interfaces can be configured for other forms of electronic devices such as smartphones, tablet computers, and wearable devices. The present disclosure is not limited to the described examples. Additional or alternative details and implementations are described herein.

[0129] In Figure 4In the example shown in FIG. 4, the user interface 400 illustrates an example of a compose message window 402. The compose message window 402 includes a sample message 406 (e.g., a message template) and a message insight pane 404. The message insight pane 404 includes a plurality of customized message suggestions 408 that have been machine-generated by the generative message suggestion system using the techniques described herein. For example, the message suggestions 408 include insights 409, 410, 412, 414, 416, 418. As an example, the insight 409 is personalized to the intended recipient based on recent activity data of the intended recipient in the application software system (e.g., activities such as a recent job search). Similarly, the insights 410, 412, 414, 416 are personalized to the intended recipient based on profile data of the intended recipient (e.g., current profile data and / or recent profile updates). The insight 418 is personalized to the intended recipient based on recent activity data of the intended recipient in the application software system (e.g., recent viewing of a profile page of a company, following of an employee of the company, and / or liking of a post by an employee of the company or a post about the company, etc.).

[0130] For purposes of illustration, the examples shown in FIG. 4 and the accompanying description are provided. For example, although the examples are illustrated as user interface screens for larger form factor user interfaces such as desktop or laptop devices, the user interfaces can be configured for other forms of electronic devices such as smartphones, tablet computers, and wearable devices. The present disclosure is not limited to the described examples. Additional or alternative details and implementations are described herein. Figure 4

[0131] Figure 5 is a block diagram of a computing system including a generative message suggestion system in accordance with some embodiments of the present disclosure.

[0132] In the example shown in FIG. 4, the user interface 400 illustrates an example of a compose message window 402. The compose message window 402 includes a sample message 406 (e.g., a message template) and a message insight pane 404. The message insight pane 404 includes a plurality of customized message suggestions 408 that have been machine-generated by the generative message suggestion system using the techniques described herein. For example, the message suggestions 408 include insights 409, 410, 412, 414, 416, 418. As an example, the insight 409 is personalized to the intended recipient based on recent activity data of the intended recipient in the application software system (e.g., activities such as a recent job search). Similarly, the insights 410, 412, 414, 416 are personalized to the intended recipient based on profile data of the intended recipient (e.g., current profile data and / or recent profile updates). The insight 418 is personalized to the intended recipient based on recent activity data of the intended recipient in the application software system (e.g., recent viewing of a profile page of a company, following of an employee of the company, and / or liking of a post by an employee of the company or a post about the company, etc.). Figure 5 In the example shown in FIG. 4, the user interface 400 illustrates an example of a compose message window 402. The compose message window 402 includes a sample message 406 (e.g., a message template) and a message insight pane 404. The message insight pane 404 includes a plurality of customized message suggestions 408 that have been machine-generated by the generative message suggestion system using the techniques described herein. For example, the message suggestions 408 include insights 409, 410, 412, 414, 416, 418. As an example, the insight 409 is personalized to the intended recipient based on recent activity data of the intended recipient in the application software system (e.g., activities such as a recent job search). Similarly, the insights 410, 412, 414, 416 are personalized to the intended recipient based on profile data of the intended recipient (e.g., current profile data and / or recent profile updates). The insight 418 is personalized to the intended recipient based on recent activity data of the intended recipient in the application software system (e.g., recent viewing of a profile page of a company, following of an employee of the company, and / or liking of a post by an employee of the company or a post about the company, etc.). Figure 5 ​In some implementations, all or part of the generative message suggestion system 580 can be implemented directly on the user system 510 (e.g., the user's client device). In other words, both the user system 510 and the generative message suggestion system 580 can be implemented on the same computing device.

[0133] Components of the computing system 500, including the generative message suggestion system 580, are described in greater detail below.

[0134] The user system 510 includes at least one computing device, such as a personal computing device, a server, a mobile computing device, or a smart device, and at least one software application executable by the at least one computing device, such as an operating system or a front end of an online system. Many different user systems 510 can be connected to the network 520 at the same time or at different times. Different user systems 510 can contain similar components as described in connection with the user system 510 illustrated. For example, many different end users of the computing system 500 can interact with many different instances of the application software system 530 through their respective user systems 510 at the same time or at different times.

[0135] The user system 510 includes a user interface 512. The user interface 512 is installed on the user system 510 or is accessible by the user system 510 over the network 520. Embodiments of the user interface 512 include a message generation interface 514 and / or a message reception interface 515. The message generation interface 514 enables sender users of the application software system 530 to create, edit, and send messages to other users, view and act on message suggestions, and perform other interactions with the application software system 530 that are associated with the creation, sending, and handling of messages created and sent between or among users. The message reception interface 515 enables intended recipients and recipients of messages sent by sender users of the application software system 530 to receive, accept, reject, ignore, view, read, respond to, and handle those messages, and perform other interactions with the application software system 530 that are associated with the receiving, viewing, reading, responding, and handling of messages created and sent between or among users. In some implementations, the message generation interface 514 and the message reception interface 515 are part of the same interface, e.g., a messaging interface that enables messages to be created, sent, and received. For example, in some implementations, the message generation interface 514 and the message reception interface 515 are part of a front end of a messaging system portion of the application software system 530.

[0136] The message generation interface 514 and the message reception interface 515 each include, for example, a graphical display screen that includes graphical user interface elements, such as at least one input box or other input mechanism and at least one slot. A slot, as used herein, refers to a space on a graphical display, such as a web page or mobile device screen, into which digital content, such as message suggestions and messages, can be loaded for display to a user. The location and size of particular graphical user interface elements on a screen are specified using, for example, a markup language such as HTML (HyperText Markup Language). On a typical display screen, graphical user interface elements are defined by two-dimensional coordinates. In other implementations, such as virtual reality or augmented reality implementations, three-dimensional coordinate systems can be used to define slots. Examples of user interface screens that can be included in the message generation interface 514 and / or the message reception interface 515 are shown in the screen captures illustrated in the accompanying drawings and described herein.

[0137] The user interface 512 can be used to input data, create, edit, send, view, receive, and process messages. In some implementations, the user interface 512 enables a user to upload, download, receive, send, or share other types of digital content items, including posts, articles, comments, and shares, to initiate user interface events, and view or otherwise perceive output, such as data and / or digital content produced by the application software system 530, the generative message suggestion system 580, and / or the message distribution service 538. For example, the user interface 512 can include a graphical user interface (GUI), a conversational speech / voice interface, a virtual reality, augmented reality, or mixed reality interface, and / or a haptic interface. The user interface 512 includes mechanisms for logging into the application software system 530, clicking or tapping GUI user input control elements, and interacting with the message generation interface 514 and digital content items, such as messages and machine-generated message suggestions. Examples of the user interface 512 include a web browser, a command line interface, and a mobile application front end. As used herein, the user interface 512 can include an application programming interface (API).

[0138] In Figure 5In the example of FIG. 5, user interface 512 includes message generation interface 514 and message reception interface 515. Message generation interface 514 and message reception interface 515 each include a front-end user interface component of generative message suggestion system 580, application software system 530, or a messaging component of application software system 530. For ease of discussion, message generation interface 514 and message reception interface 515 are shown as components of user interface 512, but access to message generation interface 514 and / or message reception interface 515 can each be limited to particular user systems 510. For example, in some implementations, access to message generation interface 514 and message reception interface 515 is limited to registered users of generative message suggestion system 580 or application software system 530, or to users that have been designated as message senders by generative message suggestion system 580 or application software system 530.

[0139] Network 520 includes an electronic communications network. Network 520 can be implemented on any medium or mechanism that provides for the exchange of digital data, signals, and / or instructions between the various components providing computing system 500. Examples of network 520 include, but are not limited to, a local area network (LAN), a wide area network (WAN), the Internet, or at least one terrestrial, satellite, or wireless link, or a combination of any number of different networks and / or communication links.

[0140] Application software system 530 includes any type of application software system that provides or enables, through user interface 512, the creation, upload, and / or distribution of at least one form of digital content, including machine-generated message suggestions and messages, between or among user systems such as user systems 510. In some implementations, portions of generative message suggestion system 580 are components of application software system 530. Components of application software system 530 can include entity graph 532 and / or knowledge graph 534, user connection network 536, message distribution service 538, and search engine 540.

[0141] In Figure 5 the example of FIG. 5, application software system 530 includes entity graph 532 and / or knowledge graph 534. Entity graph 532 and / or knowledge graph 534 include data organized according to a graph-based data structure that can be traversed via queries and / or indexes to determine relationships between entities. Examples of entity graphs are shown in Figure 6 FIG. 6, as described herein. For example, as described in more detail with reference to Figure 6 Entity graph 532 and / or knowledge graph 534 can be used to compute various types of affinity scores, similarity measurements, and / or statistics between, among, or related to entities, as described in more detail below.

[0142] The entity graphs 532, 512 include graph-based representations of data stored in the data storage system 550, as described herein. For example, the entity graphs 532, 512 represent entities, such as users, organizations, and content items, such as posts, articles, comments, and shares, as nodes of a graph. The entity graphs 532, 512 represent relationships (also referred to as mappings or links) between or among entities as edges or combinations of edges between nodes of the graph. In some implementations, mappings between different pieces of data used by the application software system 530 are represented by one or more entity graphs. In some implementations, the edges, mappings, or links indicate online interactions or activities related to the entities connected by the edges, mappings, or links. For example, if a recipient is expected to accept a message from a sender, an edge connecting the sender entity to the recipient entity in the entity graph can be created, where the edge can be labeled with a label such as "message accepted."

[0143] Parts of the entity graphs 532, 512 can be automatically regenerated or updated from time to time based on changes and updates to stored data (e.g., updates to entity data and / or activity data). As such, the entity graphs 532, 512 can refer to the entire system-wide entity graph or only a portion of the system-wide graph. For example, the entity graphs 532, 512 can refer to a subset of the system-wide graph, where the subset is related to a particular user or group of users of the application software system 530.

[0144] In some implementations, the knowledge graph 534 is a subset or superset of the entity graphs 532. For example, in some implementations, the knowledge graph 534 includes multiple different entity graphs 532 joined by edges. For example, the knowledge graph 534 can join entity graphs 532 that have been created across multiple different databases or across different software products. In some implementations, the entity nodes of the knowledge graph 534 represent concepts, such as product surfaces, verticals, or application domains. In some implementations, the knowledge graph 534 includes a platform that extracts and stores different concepts that can be used to establish links between data across multiple different software applications. Examples of concepts include topics, industries, and skills. The knowledge graph 534 can be used to generate and derive content and entity-level embeddings that can be used to discover or reason new interrelationships between entities and / or concepts, which can then be used to identify related entities. Like other parts of the entity graphs 532, the knowledge graph 534 can be used to compute various types of affinity scores, similarity measures, and / or statistical correlations between or among entities and / or concepts.

[0145] Knowledge graph 534 comprises a graph-based representation of the data stored in the data storage system 550 described herein. Knowledge graph 534 represents relationships between entities or concepts as edges or combinations of edges between nodes in the graph, also referred to as links or mappings. In some implementations, mappings between different data segments used by application software system 530 or across multiple different application software systems are represented by knowledge graph 534.

[0146] User connection network 536 includes, for example, social networking services, specialized social networking software, and / or other applications based on social graphs. Messaging distribution service 538 includes, for example, messaging systems, such as those enabling message creation and public or non-public exchange, or peer-to-peer messaging systems between users of application software system 530. Search engine 540 includes a search engine that enables users of application software system 530 to input and execute search queries on user connection network 536 and / or entity graph 532, knowledge graph 534. Application software system 530 may include online systems that provide social networking services, general-purpose search engines, specialized search engines, messaging systems, content distribution platforms, e-commerce software, enterprise software, or any combination of the foregoing or other types of software.

[0147] The front-end portion of application software system 530 can operate within user system 510, for example, as a plugin or component in a graphical user interface of a web application or mobile software application, or as a web browser executing user interface 512. In an embodiment, the mobile application or web browser of user system 510 can transmit network communication (such as an HTTP request) over network 520 in response to user input received through a user interface (such as user interface 512) provided by the web application, mobile application, or web browser. The server running application software system 530 can receive input from the web application, mobile application, or browser executing user interface 512, perform at least one operation using the input, and return output to user interface 512 using network communication such as an HTTP response, which the web application, mobile application, or browser receives and processes at user system 510.

[0148] exist Figure 6 In the example, application software system 530 includes a message distribution service 538. Message distribution service 538 can include data storage services, such as a web server, which stores messages and / or message suggestions generated by generative message recommendation system 580, and uses network 520 to transmit messages created based on message suggestions generated by generative message recommendation system 580 from message creators / senders to intended recipients.

[0149] In some embodiments, the message distribution service 538 processes requests from, for example, the application software system 530 and distributes messages and message suggestions generated by the generative message suggestion system 580 to the user system 510 in response to the requests. The requests include, for example, web messages such as HTTP (HyperText Transfer Protocol) requests to transfer data from an application front-end to a back-end of the application, or from a back-end to a front-end of the application, or more generally, requests to transfer data between two different devices or systems, such as data transfers between a server and a user system. The requests are formulated, for example, by a browser or mobile application at a user device in conjunction with user interface events such as logins, clicks on graphical user interface elements, or page loads. In some implementations, the message distribution service 538 is part of the application software system 530 or the generative message suggestion system 580. In other implementations, the message distribution service 538 interfaces with the application software system 530 and / or the generative message suggestion system 580, for example, via one or more application programming interfaces (APIs).

[0150] In Figure 6 In the example of FIG. 5, the application software system 530 includes a search engine 540. The search engine 540 is a software system designed to search and retrieve information by executing queries on data stores such as databases, connected networks, and / or graphs. The queries are designed to find information that matches specified criteria such as keywords and phrases. For example, the search engine 540 is used to retrieve data 534 by executing queries on various data stores of the data store system 550 or by traversing the entity graph 532.

[0151] The generative message suggestion system 580 automatically generates user- and / or group-specific message suggestions using one or more machine learning models based on input received via the message generation interface 514 and / or other data sources. In some implementations, the generative message suggestion system 580 generates message suggestions based on various forms of input data, including user-selected and / or machine-suggested attribute data, and formulates one or more user-specific model inputs to a generator model based on the input data. The generator model outputs one or more message suggestions based on the one or more model inputs. The generative message suggestion system 580 sends one or more of the machine-generated message suggestions to the message generation interface 514 for display to a message sender user. In various embodiments, additional or alternative features and functionality of the generative message suggestion system described herein are included in the generative message suggestion system 580.

[0152] The event logging service 570 captures and logs network activity data generated during operation of the application software system 530, including user interface events generated in real-time at the user system 510 via the user interface 512, and formulates the user interface events into data streams that can be consumed by, for example, a stream processing system. Examples of network activity data include page loads, clicks on messages or graphical user interface control elements, creation, editing, sending, and viewing of messages, and social action data such as likes, shares, comments, and social reactions (e.g., “insightful,” “curious,” etc.). For example, when a user of the application software system 530 clicks on a user interface element (such as a message, link) or user interface control element (such as a view, comment, share, or reaction button), or uploads a file, or creates a message, loads a web page, or scrolls through a feed, etc. via the user system 510, the event logging service 570 fires an event to capture an identifier such as a session identifier, an event type, a date / time stamp of when the user interface event occurred, and possibly other information about the user interface event such as the impression portal and / or impression channel involved in the user interface event. Examples of impression portals and channels include, for example, device type, operating system, and software platform, e.g., web or mobile platform.

[0153] For example, when a sender user creates a message or reacts to a received message based on a message suggestion generated by the generative message suggestion system 580, the event logging service 570 stores corresponding event data in a log. The event logging service 570 generates a data stream that includes a record of real-time event data for each user interface event that has occurred. The event data logged by the event logging service 570 can be pre-processed and anonymized as needed so that it can be used, for example, to generate affinity scores, similarity measurements, and / or to formulate training data for artificial intelligence models.

[0154] The data storage system 550 includes data stores and / or data services that store digital data received, used, manipulated, and produced by the application software system 530 and / or the generative message suggestion system 580, including message suggestions, messages, message metadata, attribute data, activity data, machine learning model training data, machine learning model parameters, and machine learning model inputs and outputs such as machine-generated content and machine-generated score data.

[0155] In Figure 5In the example of FIG. 5, data storage system 550 includes attribute data store 552, activity data store 554, score data store 556, message data store 558, and training data store 560. Attribute data store 552 stores data related to users and other entities used by generative message suggestion system 580, such as profile data, for example, to generate message suggestions and / or compute statistics, similarity measures, or scores.

[0156] Score data store 556 stores scores and related metadata generated and output by scoring models of generative message suggestion system 580, which can be used by generator models of generative message suggestion system 580 to generate message suggestions or to tune the generator models. Message data store 558 stores messages and / or machine-generated message suggestions, related metadata, and related data, such as human-edited versions of machine-generated message suggestions, generated by generator models of generative message suggestion system 580. Training data store 560 stores data that can be used or generated by training data formulator of generative message suggestion system 580, which can be used to train or tune generator models and / or scoring models of generative message suggestion system 580. For example, portions of training data store 560 can include pre- send feedback data and / or post-send feedback data.

[0157] In some embodiments, data storage system 550 includes multiple different types of data stores and / or distributed data services. As used herein, a data service can refer to a physical, geographic grouping of machines, a logical grouping of machines, or a single machine. For example, a data service can be a data center, a cluster, a group of clusters, or a machine. Data stores of data storage system 550 can be configured to store data produced by real-time and / or offline (e.g., batch) data processing. Data stores configured for real-time data processing can be referred to as real-time data stores. Data stores configured for offline or batch data processing can be referred to as offline data stores. Data stores can be implemented using databases such as key-value stores, relational databases, and / or graph databases. Data can be written to and read from data stores using query technology (e.g., SQL or NoSQL).

[0158] A key-value database or key-value store is a non-relational database that organizes and stores data records as key-value pairs. A key uniquely identifies a data record, i.e., the value associated with the key. The value associated with a given key can be, for example, a single data value, a list of data values, or another key-value pair. For example, the value associated with a key can be the data identified by the key or a pointer to that data. A relational database defines data structures as tables or collections of tables in which data is stored in rows and columns, where each column of the table corresponds to a data field. Relational databases use keys to create relationships between data stored in different tables, and the keys can be used to join data stored in different tables. A graph database organizes data using graph data structures that include a plurality of interconnected graph primitives. Examples of graph primitives include nodes, edges, and predicates, where nodes store data, edges create relationships between two nodes, and predicates are assigned to edges. Predicates define or describe the type of relationship that exists between nodes connected by an edge.

[0159] The data storage system 550 resides on at least one persistent and / or volatile storage device, which can reside in the same local network as at least one other device of the computing system 500 and / or in a network remote with respect to at least one other device of the computing system 500. Thus, although depicted as included in the computing system 500, portions of the data storage system 550 can be part of the computing system 500 or accessible by the computing system 500 over a network, such as the network 520.

[0160] Although not specifically shown, it should be understood that any of the user system 510, the application software system 530, the generative message suggestion system 580, the data storage system 550, and the incident record service 570 include interfaces embodied as computer program code stored in computer memory that, when executed, enable a computing device to communicate bi-directionally with any other of the user system 510, the application software system 530, the generative message suggestion system 580, the data storage system 550, or the incident record service 570 using a communication coupling mechanism. Examples of communication coupling mechanisms include network interfaces, inter-process communication (IPC) interfaces, and application program interfaces (APIs).

[0161] Each of user system 510, application software system 530, generative message suggestion system 580, data store system 550, and event logging service 570 is implemented using at least one computing device communicatively coupled to electronic communication network 520. Any of user system 510, application software system 530, generative message suggestion system 580, data store system 550, and event logging service 570 can be communicatively coupled bidirectionally by network 520. User system 510, as well as other different user systems (not shown), can be communicatively coupled bidirectionally to application software system 530 and / or generative message suggestion system 580.

[0162] A typical user of user system 510 can be an administrator or an end user of application software system 530 or generative message suggestion system 580. User system 510 is configured to communicate bidirectionally with any of application software system 530, generative message suggestion system 580, data store system 550, and event logging service 570 over network 520.

[0163] Terms such as components, systems, and models, as used in this document, refer to computer-implemented structures, e.g., combinations of software and hardware, such as computer programming logic, data, and / or data structures implemented in circuitry, stored in memory, and / or executed by one or more hardware processors.

[0164] The features and functionality of user system 510, application software system 530, generative message suggestion system 580, data store system 550, and event logging service 570 are implemented using computer software, hardware, or software and hardware, and can include combinations of automated functionality, data structures, and digital data represented schematically in the figures. For ease of discussion, user system 510, application software system 530, generative message suggestion system 580, data store system 550, and event logging service 570 are shown in Figure 5 as separate elements, but the illustrations do not imply that the elements need be separate unless otherwise described. The illustrated systems, services, and data stores of each of user system 510, application software system 530, generative message suggestion system 580, data store system 550, and event logging service 570 (or the functionality thereof) can be divided among any number of physical systems, including a single physical computer system, and can communicate with each other in any suitable manner.

[0165] In Figure 13In some implementations, the message generation interface 514, the message reception interface 515, and portions of the generative message suggestion system 580 are collectively referred to as a generative message suggestion system 1350 for ease of discussion. The message generation interface 514, the message reception interface 515, and the generative message suggestion system 580 need not all be implemented on the same computing device, in the same memory, or loaded into the same memory at the same time. For example, access to any of the message generation interface 514, the message reception interface 515, and the generative message suggestion system 580 can be limited to different, mutually exclusive sets of user systems. For example, in some implementations, a separate personalized version of the generative message suggestion system 580 (such as user-specific versions of generator models and scoring models) is created for each user of the generative message suggestion system 580, such that data is not shared between or among the separate personalized versions of the system. Additionally, while the message generation interface 514 and the message reception interface 515 can generally be implemented on user systems, the generative message suggestion system 580 can generally be implemented on a server computer or server group. Further details regarding the operation of the generative message suggestion system 1350 are described herein.

[0166] Figure 6 is an example of an entity graph according to some embodiments of the present disclosure. According to some embodiments of the present disclosure, the entity graph 600 can be used by an application software system, for example, to support a user connection network. The entity graph 600 can be used (e.g., queried or traversed) to obtain or generate input data for use in formulating model inputs for generator models and / or scoring models of a generative message suggestion system.

[0167] An entity graph includes nodes, edges, and data associated with nodes and / or edges, such as labels, weights, or scores. Nodes can be weighted based on, for example, edge counts or other types of calculations, and edges can be weighted based on, for example, affinity, relationship, activity, similarity, or commonality between nodes connected by an edge, such as common attribute values (e.g., two users have the same job title or employer, or two users are degree connected in a user connection network. n

[0168] A graphing mechanism is used to create, update, and maintain the entity graph. In some implementations, the graphing mechanism is a component of a database architecture used to implement the entity graph 600. For example, the graphing mechanism can be a component of the data storage system 550 and / or the application software system 530 shown in FIG. 6, and the entity graph created by the graphing mechanism can be stored in one or more data stores of the data storage system 550. Figure 5

[0169] ​​Entity graph 600 is dynamic (e.g., continuously updated) because it is updated in response to the occurrence of interactions between entities in an online system (e.g., a user connection network) and / or the computation of new relationships between or within the nodes of the graph. These updates are achieved through real-time data ingestion and storage techniques, or through offline data extraction, computation, and storage techniques, or a combination of real-time and offline techniques. For example, entity graph 600 is updated in response to user updates to user profiles, user connections with other users, and user creation of new content items such as messages, posts, articles, comments, and shares. As another example, entity graph 600 is updated when new computations are performed, such as when new relationships between nodes are inferred based on statistical correlation or machine learning models.

[0170] Entity graph 600 includes a knowledge graph containing cross-application links. For example, message activity data obtained from a messaging system can be linked to entities in the entity graph.

[0171] exist Figure 6 In the example, entity graph 600 includes entity nodes that represent entities, such as content item nodes (e.g., post U21, article 1), user nodes (e.g., user 1, user 2, user 3, user 4), and job node (e.g., job 1, job 2). Entity graph 600 also includes attribute nodes that represent the attributes of entities (e.g., profile data, topic data). Examples of attribute nodes include title nodes (e.g., title U1, title A1), company nodes (e.g., company 1), topic nodes (topic 1, topic 2), and skill nodes (e.g., skill A1, skill U11, skill U31, skill U41).

[0172] Entity graph 600 also includes edges. These edges individually and / or collectively represent various types of relationships between or among nodes. Data can be linked to both nodes and edges. For example, when stored in a data store, each node is assigned a unique node identifier, and each edge is assigned a unique edge identifier. The edge identifier can be, for example, a combination of the node identifier of the node connected by the edge and a timestamp indicating the date and time the edge was created. For example, in graph 600, an edge between user nodes can represent an online social connection between users represented by nodes, such as a "friend" or "follower" connection between connected nodes. As an example, in entity graph 600, user 3 is a first-level connection of user 1 via a connection edge between user 3 and user 1, while user 2 is a second-level connection of user 3, although user 1 and user 2 have a different type of connection than user 3, namely, "followers".

[0173] In the entity graph 600, edges can represent activities involving the entities represented by the nodes connected by the edges. For example, a POSTED edge between the user 2 node and the post U21 node indicates that the user represented by the user 2 node posted the digital content item represented by the post U21 node to the application software system (e.g., as a job posting posted to a user connection network). As another example, a SHARED edge between the user 1 node and the post U21 node indicates that the user represented by the user 1 node shared the content item represented by the post U21 node. Similarly, a CLICKED edge between the user 3 node and the article 1 node indicates that the user represented by the user 3 node clicked on the article represented by the article 1 node, and a LIKED edge between the user 3 node and the comment U1 node indicates that the user represented by the user 3 node liked the content item represented by the comment U1 node.

[0174] In some implementations, combinations of nodes and edges are used to compute various scores, and those scores are used by various components of the generated message suggestion system 580, e.g., to generate message suggestions, to score message suggestions, and / or to rank feedback. For example, a score measuring the affinity of the user represented by the user 1 node for the job represented by the job 2 node can be computed using the following paths: p1 , which includes a sequence of edges between the nodes user 1, post U21, and job 2; and / or path p2, which includes a sequence of edges between the nodes user 1, comment U1, and job 2; and / or path p3 , which includes a sequence of edges between the nodes user 1, user 2, post U21, and job 2; and / or path p4 , which includes a sequence of edges between the nodes user 1, user 3, job 1, company 2, and job 2. Path p1 , p2 , p3 , p4 And / or any one or more of the other paths through the graph 600 can be used to compute scores representing affinity, relationship, or statistical correlation between different nodes. For example, based on relative edge counts, a user-job affinity score computed between user U1 and job 2 can be higher than a user-job affinity score computed between user U4 and job 2. Similarly, a user-skill affinity score computed between user 3 and skill U31 can be higher than a user-skill affinity score computed between user 3 and skill U11. As another example, a job-skill affinity score computed between job 1 and skill U31 can be higher than a job-skill affinity score computed between job 1 and skill U41.

[0175] For purposes of illustration, the following are provided: Figure 6The present disclosure is not limited to the described examples.

[0176] Figure 7 is a flow diagram of an example method for automated message suggestion generation using components of a generative message suggestion system in accordance with some embodiments of the present disclosure.

[0177] Method 700 is performed by processing logic that comprises hardware (e.g., processing device, circuitry, dedicated logic, programmable logic, microcode, hardware of a device, integrated circuit, etc.), software (e.g., instructions run or executed on a processing device), or a combination thereof. In some embodiments, method 700 is performed by one or more components of generative message suggestion system 108 of FIG. 1 or generative message suggestion system 580 of FIG. 5. Figure 1 Figure 5 The order in which the processes are illustrated is not meant to be limiting. Unless otherwise noted, the order of the processes can be modified. Thus, the illustrated embodiments should be understood only as examples, the processes illustrated can be performed in a different order, and some processes can be performed in parallel. Additionally, at least one process can be omitted in various embodiments. Thus, not all processes are required in every embodiment. Other process flows are possible.

[0178] In the example of FIG. 7, generative message suggestion system 740 includes input data collection subsystem 702, data anonymizer subsystem 705, training data formulation subsystem 706, generator model subsystem 710, scoring model subsystem 711, message suggestion generation subsystem 714, message generation interface 718, pre- transmission feedback subsystem 720, message distribution service 724, and post-transmission feedback subsystem 728. Other implementations of generative message suggestion system 740 include some or all of the components shown in FIG. 7 and / or other components. In some implementations, one or more components of generative message suggestion system 740 include functionality described herein with reference to generative message suggestion system 108 of FIG. 1. Figure 7 Figure 7 In the example of FIG. 7, generative message suggestion system 740 includes input data collection subsystem 702, data anonymizer subsystem 705, training data formulation subsystem 706, generator model subsystem 710, scoring model subsystem 711, message suggestion generation subsystem 714, message generation interface 718, pre- transmission feedback subsystem 720, message distribution service 724, and post-transmission feedback subsystem 728. Other implementations of generative message suggestion system 740 include some or all of the components shown in FIG. 7 and / or other components. In some implementations, one or more components of generative message suggestion system 740 include functionality described herein with reference to generative message suggestion system 108 of FIG. 1. Figure 1

[0179] In some implementations, method 700 includes both online and offline processes. For example, machine learning model training and / or tuning can be performed offline, while use of trained models can be performed online in response to user interactions with message generation interface 718. Alternatively or additionally, model tuning can be performed in response to online use of the generative message suggestion system. For example, in online operation, message suggestions output by a pre-trained generator model of generator model subsystem 710 can be input to a pre-trained scoring model of scoring model subsystem 711, and the corresponding output of pre-trained scoring model subsystem 711 can be used to tune the generator model.​​​

[0180] The input data collection subsystem 702 includes one or more computer programs or routines that collect input data from one or more data sources, such as activity logs, message data stores, profile data stores, and entity graphs of application software systems. In this document, for example, reference is made to Figure 1 Examples of input data are described. To collect input data, the input data collection subsystem 702 executes queries against one or more databases or data stores, including real-time data stores, and / or interfaces with stream processing or event logging services, such as the logging service 570, to obtain real-time input or updates. The input data collection subsystem 702 outputs input data 704 for use by the data anonymizer subsystem 705.

[0181] The data anonymizer subsystem 705 includes one or more computer programs or routines that remove potentially sensitive information from the input data 704 before the input data 704 can be used by other components of the generative message suggestion system 740. In some implementations, such as the implementation described in this document with reference to Figure 1 The data anonymizer subsystem 705 performs a named entity recognition process on portions of the input data 704. The output of the named entity recognition includes attribute labels that replace potentially sensitive attribute data, such as entity names, geographic locations, and user profile data, such as job titles and experience. The input data collection subsystem 702 and the data anonymizer subsystem 705 can be logically and / or physically isolated from other components of the generative message suggestion system 740 to prevent leakage of potentially sensitive data. The data anonymizer subsystem 705 outputs anonymized input data 708 for use by the training data formulation subsystem 706.

[0182] The training data formulation subsystem 706 includes one or more computer programs or routines that formulate training data for machine learning models of the generative message suggestion system 740. The training data formulated by the training data formulation subsystem 706 can be sender-specific. For example, the training data formulation subsystem 706 can produce a set of sender-specific training data such that the model training process produces a sender-specific version of the machine learning model of the generative message suggestion system 740 for each message creator / sender.

[0183] The training data formulation subsystem 706 generates and outputs generator model training data 707 and scoring model training data 709. In some implementations, the generator model training data 707 and the scoring model training data 709 contain different sets of training data. For example, in some implementations, the training data formulation subsystem 706 formulates negative examples of training data that are not included in the generator model training data 707 but are included in the scoring model training data 709. The training data formulation subsystem 706 can also formulate positive examples of training data that are included in both the generator model training data 707 and the scoring model training data 709.

[0184] Examples or instances of training data formulated by the training data formulation subsystem 706 include historical examples of message content created by a particular sender and sent to a particular intended recipient, as well as a label indicating whether the message containing the message text snippet was accepted, rejected, or ignored by the intended recipient. For example, an instance of training data includes a message text snippet that has previously been sent by a sender to an intended recipient and a data value indicating whether the message containing the message text snippet was accepted, rejected, or ignored by the intended recipient. The data value can be, for example, a numerical value in a set of valid values, such as [1, 0, 1], where -1 indicates that the message was rejected, 0 indicates that the message was ignored, and 1 indicates that the message was accepted. Alternatively, other methods for representing message acceptance data can be used, such as a canonical text label.

[0185] An instance of training data is considered a positive example if the intended recipient accepted the message, and an instance of training data is considered a negative example if the intended recipient rejected or ignored the message. In the training instances, attribute data or metadata that can typically be used to identify the particular sender and / or intended recipient is excluded from the training instances because the data has already been anonymized by the data anonymizer subsystem 705.

[0186] The training data formulation subsystem 706 outputs the generator model training data 707 to the generator model subsystem 710 and the scoring model training data 709 to the scoring model subsystem 711.

[0187] The generator model subsystem 710 includes one or more computer programs or routines that apply a generator model to generator model training data 707 during model training or tuning to produce a trained or tuned generator model based on the generator model training data 707. When the trained or tuned generator model is brought online, the trained or tuned generator model generates message suggestions 712 in response to attribute data 717 received by the message suggestion generation subsystem 714 via the message generation interface 718. During online operation, the trained or tuned generator model also communicates message suggestions 712 to and receives message suggestion scores 713 from the scoring model subsystem 711. In some implementations, the online execution of the trained or tuned generator model subsystem 710 is initiated by an API call from the message suggestion generation subsystem 714, and the communication between the generator model subsystem 710 and the scoring model subsystem 711 is implemented via API calls.

[0188] The scoring model subsystem 711 includes one or more computer programs or routines that apply a scoring model to scoring model training data 709 during model training or tuning to produce a trained or tuned scoring model based on the scoring model training data 709. When the trained or tuned scoring model is brought online, the trained or tuned scoring model generates and outputs message suggestion scores 713 in response to messages created via the message generation interface and / or message suggestions 712 received from the generator model subsystem 710. In some implementations, the online execution of the trained or tuned scoring model subsystem 711 is initiated by an API call from the message suggestion generation subsystem 714 or the generator model subsystem 710.

[0189] In some implementations, both the generator model subsystem 710 and the scoring model subsystem 711 are implemented using an encoder-decoder model architecture. In other implementations, the generator model subsystem 710 and the scoring model subsystem 711 are implemented using one or more different model architectures. For example, in some implementations, only a decoder or transformer-based architecture is used in one or both of the generator model subsystem 710 and the scoring model subsystem 711. In other implementations, an instance of a generative model, such as a large language model, is configured to generate and output message content in the generator model subsystem 710, while another instance of the generative model is configured to generate and output message suggestion scores in the scoring model subsystem 711. In yet other implementations, a different model architecture is used for the scoring model than the model architecture used for the generator model.

[0190] Generative models use artificial intelligence techniques to machine-generate new digital content based on model inputs and pre-existing data that the model has been trained with. Discriminative models, however, are based on the conditional probability P(y|x), i.e., the probability of an output y given an input x (e.g., is the photo of a dog?). Generative models capture the joint probability P(x, y), i.e., the likelihood of x and y occurring together (e.g., given the photo of a dog and an unknown person, the likelihood that the person is Sam, the dog’s owner?). Discriminative models are trained to discriminate between different classes of outputs given an input. For example, a discriminative model can be trained to discriminate between different classes of photos given a photo. The classes can be defined by the presence or absence of certain features in the photo. For example, the classes can be defined by the presence or absence of a dog in the photo. Discriminative models are trained to output a probability that a given photo belongs to a given class. Discriminative models are trained to discriminate between different classes of outputs given an input. For example, a discriminative model can be trained to discriminate between different classes of photos given a photo. The classes can be defined by the presence or absence of certain features in the photo. For example, the classes can be defined by the presence or absence of a dog in the photo. Discriminative models are trained to output a probability that a given photo belongs to a given class.

[0191] Generative language models are a particular type of generative model that generate new text in response to model inputs. The model inputs include a task description, also referred to as a prompt. The task description can include instructions and / or examples of digital content. The task description can be in the form of natural language text, such as a question or statement, and can include content in non-textual forms, such as digital images and / or digital audio. In some implementations, an input layer of the generative language model converts the task description into an embedding or set of embeddings. In other implementations, one embedding or multiple embeddings are generated by a preprocessor based on the task description and then input to the generative language model.

[0192] Given a task description, a generative model can generate a set of task description-output pairs, where each pair contains a different output. In some implementations, the generative model assigns a score to each of the generated task description-output pairs. The output in a given task description-output pair contains text that is generated by the model itself, rather than being provided as input to the model.

[0193] The score associated with a given task description-output pair by the model represents the probabilistic or statistical likelihood that there is a relationship between the output and the corresponding task description in the task description-output pair. For example, given an image of an animal and an unknown person, a generative model can generate the following task description-output pairs and associated scores: [What is the picture?; This is a picture of a dog playing with a little boy by a lake; 0.9], [What is the picture? This is a picture of a dog walking with an old woman on the beach; 0.1]. The higher score of 0.9 indicates a higher likelihood that the picture shows a dog playing with a little boy by a lake rather than a dog walking with an old woman on the beach. The score for a given task description-output pair depends on the way the generative model has been trained and the data used to perform the model training. The generative model can rank the task description-output pairs by score and output only the one or more pairs with the top k k ​is a positive integer that represents a desired number of pairs to return for a particular design or implementation of the generative model. For example, the model can discard lower scoring pairs and only output the highest scoring pair as its final output.

[0194] In some implementations, one or more of the generator model subsystem 710 and the scoring model subsystem 711 are implemented using a graph neural network. For example, in one model instance, a modified version of a bidirectional encoder representation with a transformer (BERT) neural network is specifically configured to generate and output message suggestions, and in another instance, to generate and output message suggestion scores. In some implementations, the modified BERT is trained with self-supervision, e.g., by masking some portions of the input data so that the BERT learns to predict the masked data. During scoring, an entity is masked that is associated with a portion of the input data, and the model generates an output based on the input data at the location of the masked entity.

[0195] In some implementations, one or more of the generator model subsystem 710 and the scoring model subsystem 711 are constructed using a neural network-based machine learning model architecture. In some implementations, the neural network-based architecture includes one or more input layers that receive a model input, generate one or more embeddings based on the model input, and pass the one or more embeddings to one or more other layers of the neural network. In other implementations, the one or more embeddings are generated based on a model input by a preprocessor, the embeddings are input to the neural network model, and the neural network model generates an output, e.g., a message suggestion or a message suggestion score, based on the embeddings.

[0196] In some implementations, the neural network-based machine learning model architecture includes one or more self-attention layers that allow the model to assign different weights to different parts of the model input. Alternatively or additionally, the neural network architecture includes feedforward layers and residual connections that allow the model to machine-learn complex data patterns that include relationships between different parts of the model input in multiple different contexts. In some implementations, the neural network-based machine learning model architecture is constructed using a transformer-based architecture that includes self-attention layers, feedforward layers, and residual connections between the layers. The exact number and arrangement of each type of layer and the hyperparameter values used to configure the model are determined based on the requirements of a particular design or implementation of the generative message suggestion system.

[0197] In some examples, the neural network-based machine learning model architecture includes or is based on one or more generative transformer models, one or more generative pre-trained transformer (GPT) models, one or more bidirectional encoder representations from transformers (BERT) models, one or more large language models (LLMs), one or more XLNet models, and / or one or more other natural language processing (NL) models. In some examples, the neural network-based machine learning model architecture includes or is based on one or more predictive text neural models capable of receiving text input and generating one or more outputs based on processing the text with one or more neural network models. Examples of predictive neural models include, but are not limited to, generative pre-trained transformer (GPT), BERT, and / or recurrent neural networks (RNNs). In some examples, one or more types of neural network-based machine learning model architecture include or are based on one or more multi-modal neural networks capable of outputting different modalities (e.g., text, images, sound, etc.) individually and / or in combination based on text input. Thus, in some examples, multi-modal neural networks implemented in the generative message suggestion system are capable of outputting digital content that includes a combination of two or more of text, images, video, or audio.

[0198] In some implementations, one or more of the models of the generator model subsystem 710 and / or the scoring model subsystem 711 are trained on large datasets of digital content, such as natural language text, images, video, audio files, or multi-modal datasets. For example, training samples of digital content, such as natural language text extracted from publicly available data sources, are used to train one or more generative models of the generative message suggestion system. The size and composition of the datasets used to train one or more models of the generator model subsystem 710 and the scoring model subsystem 711 can vary depending on the requirements of the particular design or implementation of the generative message suggestion system. In some implementations, one or more of the datasets used to train one or more models of the generator model subsystem 710 and the scoring model subsystem 711 include hundreds of thousands to millions or more distinct training samples.

[0199] In some embodiments, the one or more models of one or more of the generator model subsystem 710 and the scoring model subsystem 711 include multiple generative models trained on different sized datasets. For example, the message suggestion generation system can include a comprehensive but low capacity generative model trained on a large dataset and used to generate message suggestion examples, and the same generative model can also include a less comprehensive but high capacity model trained on a smaller dataset, where the high capacity model is used to generate output based on examples obtained from the low capacity model. In some implementations, reinforcement learning is used to further improve the output of one or more of the generator model subsystem 710 and the scoring model subsystem 711. In reinforcement learning, true examples of desired model output are paired with respective inputs, and these input example pairs are used to train or fine-tune one or more of the generator model subsystem 710 and the scoring model subsystem 711.

[0200] In online operation, the message suggestion generation subsystem 714 includes one or more computer programs or routines (e.g., APIs) that receive attribute data 717 from the message generation interface 718 and output one or more message suggestions 716 in response to the received attribute data 717. In response to the attribute data 717, the message suggestion generation subsystem 714 interfaces with the trained or tuned generator model subsystem 710 to obtain one or more message suggestions 712 produced by the trained or tuned generator model subsystem 710, and interfaces with the trained or tuned scoring model subsystem 711 to obtain one or more message suggestion scores 713 produced by the trained or tuned generator model subsystem 710. The one or more message suggestion scores 713 correspond to the message suggestions 712 obtained from the generator model subsystem 710. For example, a message suggestion score 713 is generated and output by the scoring model subsystem 711 for each message suggestion 712 generated and output by the generator model subsystem 710.

[0201] The message suggestions 712 include, for example, machine-generated content that a message sender can use to create a message that is customized for a particular intended recipient, e.g., message text, an outline, or a summary. Examples of message suggestions include insights, such as useful hints that a message sender can consider when the sender is creating a message. Other examples of message suggestions include suggested attributes that a message sender can include in a message to improve the probability of acceptance by a particular intended recipient. Other examples of message suggestions include machine-generated message suggestion content (e.g., text and / or other content) to be included in the body of a message to improve the probability of acceptance by a particular intended recipient. Message suggestions can include one or more different forms of content, e.g., text, audio, video, a combination of text and images or video, etc.

[0202] The message suggestion generation subsystem 714 selects one or more message suggestions from the message suggestions 712 based on the corresponding message suggestion scores 713. For example, the message suggestion generation subsystem 714 orders or ranks the message suggestions 712 in order based on the message suggestion scores 713 (e.g., from highest score to lowest score), and selects the top k message suggestions from the ordered or ranked list of message suggestions 712, where k is a positive integer whose value can be configured based on requirements or design of a particular implementation of the generative message suggestion system.

[0203] In response to the attribute data 717, the message suggestion generation subsystem 714 outputs one or more selected message suggestions 716 to a message generation interface 718. The message generation interface 718 presents the message suggestion(s) 716 to the message sender user, for example, via a graphical user interface. In response to the message suggestion(s) 716, the message generation interface 718 can receive a pre-send signal 719, such as a message drafted by the sender, edits to the attribute data 717, a selection of a message suggestion, edits to a message suggestion, a request to regenerate a message suggestion, or other forms of user reactions to the message suggestion(s) 716 that occur before the sender sends the message to the intended recipient. The message generation interface 716 sends the received pre-send signal 719 to the message suggestion generation subsystem 714, which can use the received pre-send signal 719 to generate a new or modified version of the message suggestion(s) 716.

[0204] The message generation interface 718 can also provide the pre-send signal 719 to a pre- send feedback subsystem 720. After one or more iterations of user interaction with the message suggestion(s) 716, the message generation interface 718 can output a message 722. In an example, the message 722 includes a message body created based at least in part on a machine-generated message suggestion and message metadata (e.g., sender identifier, intended recipient identifier).

[0205] The pre-send feedback subsystem 720 includes one or more computer programs or routines that obtain the pre-send signal 719 related to the message suggestion 716 produced by the message suggestion generation subsystem 714 and formulate pre-send feedback 721, for example, by mapping the pre-send signal 719 to a corresponding message or message suggestion 716 and retuning the pre-send feedback 721 to the training data formulation subsystem 706 for tuning one or more models of the generator model subsystem 710 and / or the scoring model subsystem 711.

[0206] The message distribution service 724 includes one or more computer programs or routines that formulate a transmittable version of the message 722 created based on the message suggestion 716 and cause the transmittable version of the message 722 to be distributed to the intended recipient, e.g., via a network such as the user connection network. In some implementations, execution of the message distribution service 724 is initiated by an API call from the generative message suggestion system 740. Transmitting a message as described herein includes transmitting a message created by a sender based on a machine-generated message suggestion from the sender's user account to the intended recipient's user account in an online system such as the application software system 530 over a network.

[0207] The message distribution service 724 can interface with, for example, the logging service 570 to collect and log post-sending signals 726 received from the intended recipient's user account. The post-sending signals 726 include data indicating whether the sender's message was delivered to the intended recipient and whether and how the intended recipient reacted to the sender's message. For example, the post-sending signals include data values indicating whether the intended recipient accepted, rejected, or ignored the sender's message.

[0208] The post-sending feedback subsystem 728 includes one or more computer programs or routines that receive and track the intended recipient's post-sending feedback 729 related to the sender's message 722. In some implementations, the post-sending feedback subsystem 728 returns the post-sending feedback 729 to the training data formulation subsystem 706 for use in tuning one or more models of the generator model subsystem 710 and / or the scoring model subsystem 711.

[0209] To generate the pre-sending feedback 721 or the post-sending feedback 729, references to the message suggestions 716 used to create the message 722 are saved in a data store, e.g., by the logging service 570. For example, each message suggestion 712 is assigned a unique message suggestion identifier by the generator model subsystem 710. The message suggestion identifier associated with the message suggestion 716 is passed to the message generation interface 718 by the message suggestion generation subsystem 714.

[0210] When the sender interacts with the message suggestion 716, the message generation interface 718 links the pre-send signal 719 with the corresponding message suggestion identifier, and potentially also with the attribute data 717 that produced the message suggestion 716. When the sender sends the message 722 based on the message suggestion 716, the message generation interface 718 generates a unique message identifier for the message 722 and links it with the corresponding message suggestion identifier, and causes the message suggestion identifier-message identifier pair to be persistently stored in a data store accessible to the message distribution service 724. When the message distribution service 724 receives the post-send signal 726 related to the message 722, the message distribution service 724 links the post-send signal 726 with the message suggestion identifier-message identifier pair, and passes the linked data to the post- send feedback subsystem 728.

[0211] For purposes of illustration, examples shown in Figure 7 and the above description appended thereto are provided. The present disclosure is not limited to the described examples.

[0212] Figure 8 is a flow diagram of an example method for automated message suggestion generation using components of a generative message suggestion system in accordance with some embodiments of the present disclosure.

[0213] The method 800 is performed by processing logic that comprises hardware (e.g., processing device, circuitry, dedicated logic, programmable logic, microcode, hardware of a device, integrated circuit, etc.), software (e.g., instructions run or executed on a processing device), or a combination thereof. In some embodiments, the method 800 is performed by one or more components of the generative message suggestion system 108 of Figure 1 or the generative message suggestion system 580 of Figure 5 Although illustrated in a particular order or sequence, unless otherwise specified, the order or sequence can be modified. Thus, the illustrated embodiments should be understood only as examples, and that the illustrated processes can be performed in a different order, and that some processes can be performed in parallel. Additionally, at least one process can be omitted in various embodiments. Thus, not all processes are required in every embodiment. Other process flows are possible.

[0214] Figure 8 An example of an online flow is shown, in which the generator model subsystem 810 and the corresponding scoring model subsystem (in Figure 8The model (not shown) is pre-trained based on input data 804 or tuned at action (1) based on score data 806 and / or feedback data 808 (e.g., pre- or post- send feedback data). At action (2), the generator model subsystem 810 machine generates one or more machine-generated message suggestions 812 and sends them to the message suggestion selection subsystem 814, which can be implemented as a component of the message suggestion generation subsystem 714. For example, the generator model subsystem 810 machine generates a list of suggested attributes based on data extracted from the sender’s profile and / or data extracted from the intended recipient’s profile. Action (2) can be triggered by, for example, the sender loading a page of the message generation interface 818 at the sender’s device while also viewing a profile page of the intended recipient. At action (3), the message suggestion selection subsystem 814 sends one or more selected message suggestions 817 to the message generation interface 818. For example, the message suggestion selection subsystem 814 sends a subset of the list of suggested attributes output by the generator model subsystem 810 to the message generation interface 818 as selected message suggestions 817 based on the sender’s profile data.

[0215] At action (4), the message generation interface 818 sends attribute data 819 to the generator model subsystem 810. For example, the message generation interface 818 presents the suggested attributes received from the message suggestion selection subsystem 814 to the sender, and the sender selects attribute data 819.

[0216] At action (5), the generator model subsystem 810 machine generates a second set of message suggestions 812 based on the attribute data 819 and outputs them to the message suggestion selection subsystem 814. For example, the generator model subsystem 810 machine generates and outputs insights based on the attribute data 819. At action (6), the message suggestion selection subsystem 814 sends a subset of the second set of message suggestions 812 (e.g., the top-ranked insights) to the message generation interface 818 as selected message suggestions 817.

[0217] At action (7), the message generation interface 818 sends a second set of attribute data 819 to the generator model subsystem 810. For example, the message generation interface 818 presents the top-ranked insights (based on the output of the scoring model) to the sender, and the sender revises the selected attribute data based on one or more of the insights, e.g., to add or delete one or more attributes from the attribute data 819.

[0218] At act (8), the generator model subsystem 810 machine generates a third set of message suggestions 812 based on the second set of attribute data 819 and sends the third set of message suggestions 812 to the message suggestion selection subsystem 814. For example, the generator model subsystem 810 machine generates and outputs one or more examples of suggested message content based on the revised list of attributes received from the message generation interface 818.

[0219] At act (9), the message suggestion selection subsystem 814 selects from the third set of message suggestions 812 and provides the selected message suggestion from the third set of message suggestions to the message generation interface 818. For example, the message suggestion selection subsystem 814 sends the highest ranked (based on the output of the scoring model) example of suggested message content to the message generation interface 818.

[0220] At act (10), the message generation interface 818 generates and sends pre- send feedback 821 to the generator model subsystem 810. For example, the message generation interface 818 presents one or more of the highest ranked examples of suggested message content to the sender and the sender requests that the message suggestions be re-generated based on a different set of attribute data or using a different tone or style.

[0221] At act (11), the generator model subsystem 810 machine generates and outputs a fourth set of message suggestions 812 to the message suggestion selection subsystem 814. For example, the generator model subsystem 810 machine generates and outputs a second set of examples of suggested message content based on the pre-send feedback 619.

[0222] At act (12), the message suggestion selection subsystem 814 selects one or more message suggestions from the fourth set of message suggestions 812 and sends the selected suggestions to the message generation interface 818. For example, the message suggestion selection subsystem 814 sends the second set of examples of suggested message content generated based on the pre-send feedback 619 to the message generation interface 818.

[0223] At act (13), the message generation interface 818 sends a message 822 including at least one of the machine generated message suggestions to a message distribution service 824. For example, the sender reviews the second set of examples of suggested message content, including one of the examples for a message to an intended recipient, and initiates sending of the message to the intended recipient.

[0224] At action (14), message distribution service 824 generates post-send feedback 825 and sends it to generator model subsystem 810. For example, message distribution service 824 receives user interface event data from the login session of the intended recipient to the message receiving interface, wherein the user interface event data indicates that the intended recipient has accepted the message 822. The message acceptance data is configured as post-send feedback 825 and is used to tune one or more models of generator model subsystem 810 and / or rating model subsystem.

[0225] For illustrative purposes, the above provides examples of... Figure 8 The examples shown and the description attached above. This disclosure is not limited to the examples described.

[0226] Figure 9 This is a flowchart of an example method for configuring a generator model for automated message suggestion generation using components of a generator model subsystem, according to some embodiments of this disclosure.

[0227] Method 900 is executed by processing logic, which includes hardware (e.g., processing device, circuitry, special-purpose logic, programmable logic, microcode, device hardware, integrated circuits, etc.), software (e.g., instructions that run or execute on the processing device), or a combination thereof. In some embodiments, method 900 is performed by... Figure 1 Generative message suggestion system 108 or Figure 5 Generative message suggestion systems 580 (such as in Figure 7 The messages shown suggest that one or more components of the generation subsystem 714 be executed as described herein. Although shown in a particular order or sequence, the order of processes may be modified unless otherwise stated. Therefore, the illustrated embodiments should be understood as examples only, and the illustrated processes can be executed in different orders, and some processes may be executed in parallel. In addition, at least one process may be omitted in various embodiments. Therefore, not all processes are required in every embodiment. Other process flows are also possible.

[0228] exist Figure 9 In this context, the generator model subsystem 914 includes one or more computer programs or routines that train or tune the generator model 906 for message suggestion generation tasks, such as configuring the generator model 906 to machine-generate and output message suggestions. In some implementations, the execution of the generator model subsystem 914, or more specifically the generator model 906, is initiated by API calls from, for example, a generative message suggestion system 580.

[0229] exist Figure 9In the example of FIG. 9, the generator model subsystem 914 includes a model trainer 902, a generative model 906, and a feedback processor 910 operatively coupled together in a closed loop. The model trainer 902 receives score data 916 from the scoring model subsystem described herein and / or feedback data 918 produced from a previous iteration of the generative model 906. Message-score-feedback data 912 is generated by the feedback processor 910 in response to the output of a previous iteration of the generative model 906.

[0230] To create the message-score-feedback data 912, in some implementations, the feedback processor 910 computes a score, such as a reward score, based on the score data 916 and / or the feedback data 918 related to a particular message or message suggestion. For example, given an instance of a message suggestion 908, the feedback processor 910 computes a reward score for the message suggestion 908 by applying a reinforcement learning model to the feedback 916, 918 associated with the message suggestion 908.

[0231] In some implementations, the generative model 906 is pre-trained on a large corpus (e.g., millions of training examples) and can be retrained or tuned for a particular application or domain. For example, a user-specific version of the generative model 906 can be created by tuning the generative model 906 based on historical message activity data of a particular sender.

[0232] The model trainer 902 creates training data based on the message-score-feedback data 912 received from the feedback processor 910. The training data created by the model trainer 902 (e.g., training message-score pairs 904) is used to train or tune the generative model 906 using, for example, supervised machine learning or semi-supervised machine learning. An example of the training data includes ground truth data for a given message-score pair, where the ground truth data includes, for example, a reward score, classification, or label generated by the feedback processor 910 in communication with one or more feedback subsystems, such as the pre-sent feedback subsystem 720 or the post-sent feedback subsystem 728. For example, the ground truth data includes historical message acceptance data. In a training or fine-tuning mode, the generative model 906 is applied to the training message-score pairs 904 and one or more model parameters of the generative model 906 are updated based on the training or fine-tuning. Alternatively or additionally, the architecture of the generative model 906 can be redesigned based on new instances of the training data or based on a new application or domain. In an operational mode, the generative model 906 generates message suggestions in response to model inputs. The message suggestions 908 generated by the generative model 906 are processed by the feedback processor 910 to create the message-score-feedback data 912.

[0233] In some implementations, the feedback processor 910 includes a reinforcement learning component, such as a reinforcement learning model, that machine learns a reward function based on feedback associated with message suggestions. For example, given a message suggestion 908, the feedback processor 910 receives or identifies feedback 916, 918 related to the message suggestion 908. The feedback can include pre-sent feedback and / or post-sent feedback received from one or more other components of the generative message suggestion system. The feedback processor 910 applies a reward function to the received or identified feedback to generate a reward score for the corresponding message suggestion based on the feedback associated with the message suggestion. The reward score is incorporated into message-score-feedback data 912, which is then used to train or tune the generator model 906 using, for example, supervised or semi-supervised machine learning. For purposes of illustration, the example shown in Figure 9 The present disclosure is not limited to the described examples.

[0234] Figure 10 is a flowchart of an example method for configuring a scoring model for automated message suggestion generation using components of a scoring model subsystem in accordance with some embodiments of the present disclosure.

[0235] The method 1000 is performed by processing logic that comprises hardware (e.g., processing device, circuitry, dedicated logic, programmable logic, microcode, hardware of a device, integrated circuit, etc.), software (e.g., instructions run or executed on a processing device), or a combination thereof. In some embodiments, the method 1000 is performed by one or more components of a generative message suggestion system 108, such as the message suggestion generation subsystem 714 shown in Figure 1 The method 1000 is performed by processing logic that comprises hardware (e.g., processing device, circuitry, dedicated logic, programmable logic, microcode, hardware of a device, integrated circuit, etc.), software (e.g., instructions run or executed on a processing device), or a combination thereof. In some embodiments, the method 1000 is performed by one or more components of a generative message suggestion system 108, such as the message suggestion generation subsystem 714 shown in Figure 5 The method 1000 is performed by processing logic that comprises hardware (e.g., processing device, circuitry, dedicated logic, programmable logic, microcode, hardware of a device, integrated circuit, etc.), software (e.g., instructions run or executed on a processing device), or a combination thereof. In some embodiments, the method 1000 is performed by one or more components of a generative message suggestion system 108, such as the message suggestion generation subsystem 714 shown in Figure 7 The method 1000 is performed by processing logic that comprises hardware (e.g., processing device, circuitry, dedicated logic, programmable logic, microcode, hardware of a device, integrated circuit, etc.), software (e.g., instructions run or executed on a processing device), or a combination thereof. In some embodiments, the method 1000 is performed by one or more components of a generative message suggestion system 108, such as the message suggestion generation subsystem 714 shown in

[0236] In Figure 10In some implementations, the scoring model subsystem 1014 includes one or more computer programs or routines that train or tune the scoring model 1006 for the message suggestion scoring task, e.g., to configure the scoring model 1006 to machine-generate and output a message suggestion score that indicates a probability of acceptance for a message suggestion generated by the generator model subsystem. In some implementations, execution of the scoring model subsystem 1014, or more specifically the scoring model 1006, is initiated by an API call from, e.g., the generative message suggestion system 580.

[0237] In some implementations, the scoring model subsystem 1014 includes one or more computer programs or routines that train or tune the scoring model 1006 for the message suggestion scoring task, e.g., to configure the scoring model 1006 to machine-generate and output a message suggestion score that indicates a probability of acceptance for a message suggestion generated by the generator model subsystem. In some implementations, execution of the scoring model subsystem 1014, or more specifically the scoring model 1006, is initiated by an API call from, e.g., the generative message suggestion system 580. Figure 10 In some implementations, the scoring model subsystem 1014 includes one or more computer programs or routines that train or tune the scoring model 1006 for the message suggestion scoring task, e.g., to configure the scoring model 1006 to machine-generate and output a message suggestion score that indicates a probability of acceptance for a message suggestion generated by the generator model subsystem. In some implementations, execution of the scoring model subsystem 1014, or more specifically the scoring model 1006, is initiated by an API call from, e.g., the generative message suggestion system 580.

[0238] To create the message-score-feedback data 1012, in some implementations, the feedback processor 1010 computes a score, such as a reward score, based on the feedback 1018 associated with a particular message-score pair. For example, given a message-score pair 1008, the feedback processor 1010 computes a score for the message-score pair 1008 by applying a reinforcement learning model to the feedback 1018 associated with the message-score pair.

[0239] In some implementations, the scoring model 1006 is pre-trained on a large corpus (e.g., millions of training examples) and can be retrained or fine-tuned for a particular user, application, or domain. The model trainer 1002 creates training data based on message-score-feedback data 1012 received from the feedback processor 1010. The training data created by the model trainer 1002 (e.g., training message-score pairs 1004) is used to train or fine-tune the scoring model 1006 using, for example, supervised or semi-supervised machine learning. Examples of training data include ground truth data for a given message-score pair, where the ground truth data includes, for example, a reward score, classification, or label generated by the feedback processor 1010 in communication with one or more feedback subsystems such as the pre-send feedback subsystem 720 or the post-send feedback subsystem 728. In a training or fine-tuning mode, the scoring model 1006 is applied to the training message-score pairs 1004 and one or more model parameters of the scoring model 1006 are updated based on the training or fine-tuning. Alternatively or additionally, the architecture of the scoring model 1006 can be redesigned based on new examples of training data or based on a new user, application, or domain. In an operational mode, the scoring model 1006 generates a score in response to a message suggestion produced by the generator model. The message-score pairs 1008 generated by the scoring model 1006 are processed by the feedback processor 1010 to create message-score-feedback data 1012 when the feedback processor 1010 receives feedback related to a respective message-score pair 1008.

[0240] In some implementations, the feedback processor 1010 includes a reinforcement learning component, such as a reinforcement learning model, that machine learns a reward function based on feedback associated with a message-score pair. For example, given a message-score pair 1008, the feedback processor 1010 receives or identifies feedback related to the message-score pair 1008. The feedback can include pre-send feedback and / or post-send feedback received from one or more other components of the generative message suggestion system. The feedback processor 1010 applies a reward function to the received or identified feedback to generate a reward score for the corresponding message-score pair based on the feedback associated with the message-score pair. The reward score is incorporated into the message-score-feedback data 1012, which is then used to train or fine-tune the scoring model 1006 using, for example, supervised or semi-supervised machine learning. The examples shown in FIG. 10 and the description that follows are provided for illustrative purposes. Figure 10 The present disclosure is not limited to the described examples.

[0241] Figure 11 is a flowchart of an example method for automated message suggestion generation according to some embodiments of the present disclosure.

[0242] Method 1100 is performed by processing logic that comprises hardware (e.g., processing device, circuitry, dedicated logic, programmable logic, microcode, hardware of a device, integrated circuit, etc.), software (e.g., instructions run or executed on a processing device), or a combination thereof. In some embodiments, method 1100 is performed by one or more components of the generative message suggestion system 108 of Figure 1 or the generative message suggestion system 580 of Figure 5 In some implementations, portions of method 1100 are performed by one or more components of the generative message suggestion system 108 described herein and / or shown in Figure 1 and / or Figure 5 Although illustrated in a particular order or sequence, unless otherwise specified, the order or sequence can be modified. Thus, the illustrated embodiments should be understood only as examples, and the illustrated processes can be performed in a different order, and some processes can be performed in parallel. Certain processes can be omitted, in various embodiments. Thus, not all processes are required in every embodiment. Other process flows are possible.

[0243] At operation 1102, the processing device configures a first machine learning model to generate and output suggested message content based on a first correlation between message content and message acceptance data. In some implementations, the first machine learning model comprises a first encoder-decoder model architecture.

[0244] In some implementations, the processing device trains the first machine learning model based on first training data. The first training data comprises positive examples of message acceptance data. In some implementations, the processing device formulates instances of the first training data to include message content, sender metadata associated with the message content, recipient metadata associated with the message content, and an acceptance label associated with the recipient metadata. The acceptance label comprises an indicator of (i) acceptance of a message comprising the message content by a recipient, (ii) rejection of the message by the recipient, or (iii) no response to the message by the recipient.

[0245] In some implementations, the processing device anonymizes at least one of the message content, the sender metadata, or the recipient metadata, and uses the at least one of the message content, the sender metadata, or the recipient metadata to formulate instances of the first training data.

[0246] In some implementations, the processing device determines, for an instance of the suggested message content, a model input to which to apply the first machine learning model to generate the instance of the suggested message content, determines a difference between the instance of the suggested message content and the model input, and tunes the first machine learning model based on the difference between the instance of the suggested message content and the model input.

[0247] At operation 1104, the processing device configures the second machine learning model to generate and output message evaluation data based on a second correlation between message content and message acceptance data. In some implementations, the second machine learning model comprises a second encoder-decoder model architecture.

[0248] In some implementations, the processing device trains the second machine learning model based on the first training data and second training data. The second training data comprises negative examples of message acceptance data.

[0249] At operation 1106, the processing device couples an output of the first machine learning model to an input of the second machine learning model. In some implementations, the processing device inputs the suggested message content output by the first machine learning model to the second machine learning model.

[0250] At operation 1108, the processing device couples an output of the second machine learning model to an input of the first machine learning model. In some implementations, the processing device inputs the message evaluation data output by the second machine learning model to the first machine learning model.

[0251] In some implementations, the processing device receives, via the message generation interface, pre-transmission feedback data related to the suggested message content, and tunes at least one of the first machine learning model or the second machine learning model based on the received pre-transmission feedback data. In some implementations, the pre-transmission feedback data is based on at least one interaction of an intended message sender with the message generation interface in response to presentation of the suggested message content by the message generation interface prior to transmission of a message comprising the suggested message content by the intended message sender to at least one recipient.

[0252] In some implementations, the processing device receives, via the message reception interface, post-transmission feedback data related to the suggested message content, and tunes at least one of the first machine learning model or the second machine learning model based on the received post-transmission feedback data. In some implementations, the post-transmission feedback data is based on at least one interaction of an intended message recipient with the message reception interface in response to presentation of a message comprising the suggested message content by the message reception interface to the intended message recipient.

[0253] In some implementations, the suggested message content generated and output by the first machine learning model includes any of a video, audio, and / or one or more digital images.

[0254] In some implementations, the processing device presents the generative message suggestion to the user at a messaging interface and receives user input in response to the generative message suggestion, where the user input includes any of one or more modifications to the generative message suggestion, one or more requests for a new generative message suggestion, an action to incorporate the generative message suggestion into a new segment of digital content and cause the new segment of digital content to be distributed in a network of users, e.g., via a social networking service.

[0255] In some implementations, the technical problem of scalability is addressed by the processing device selecting a generative message suggestion from a set of generative message suggestions, where the set of generative message suggestions is at least an order of magnitude smaller in size than the number of users of the messaging system.

[0256] In some implementations, the technical problem of efficient distribution of message suggestions is addressed by the processing device converting the generative message suggestion from a first size to a second size prior to distribution, where the second size is more efficient than the first size for presentation to a user or for distribution in a network of users.

[0257] In some implementations, the processing device configures the generative message suggestion based on one or more interaction parameters of the sending user; for example, the one or more interaction parameters are input to a machine learning model to cause the machine learning model to formulate the generative message suggestion to be appropriate for presentation at end user devices having different screen sizes or resolutions and / or different device capabilities, to facilitate facilitating interaction between the user and the message suggestion, resulting in improved message creation and distribution.

[0258] In some implementations, the technical problem of processing delay is addressed by the processing device detecting an increase in delay to output a generative message suggestion, and in response to detecting the increase in delay, performing one or more of the following actions: reducing the number of model inputs; or using a machine learning model having a reduced size; or reducing the size of the message suggestion (e.g., reducing a maximum text length or byte size).

[0259] For the purposes of illustration, the examples shown in Figure 11 and the above accompanying description are provided. The present disclosure is not limited to the described examples.

[0260] Figure 12 is a flowchart of an example method for automated message suggestion generation according to some embodiments of the present disclosure.

[0261] Method 1200 is executed by processing logic, which includes hardware (e.g., processing device, circuitry, special-purpose logic, programmable logic, microcode, device hardware, integrated circuits, etc.), software (e.g., instructions that run or execute on the processing device), or a combination thereof. In some embodiments, method 1200 is performed by... Figure 1 Generative message suggestion system 108 or Figure 5 The generative message suggestion system 580 is executed by one or more components. For example, in some implementations, part of method 1200 is performed by the components described herein. Figure 1 and / or Figure 5 The generative message suggestion system illustrated executes one or more components. Although shown in a specific order or sequence, the order of processes can be modified unless otherwise stated. Therefore, the illustrated embodiments should be understood as examples only, and the illustrated processes can be executed in different orders, and some processes can be executed in parallel. Furthermore, at least one process may be omitted in various embodiments. Therefore, not all processes are required in every embodiment. Other process flows are also possible.

[0262] At operation 1202, the processing device receives first message attribute data via a message generation interface. In some implementations, the processing device determines links between a first entity and a second entity based on a social graph, wherein at least one of the first entity or the second entity represents an intended message recipient in the social graph, and determines the first message attribute data based on the links.

[0263] At operation 1204, the processing device inputs the first message attribute data into a first machine learning model. The first machine learning model is configured to generate and output suggested message content based on a first correlation between the message content and the received message data. In some implementations, the first machine learning model includes a first encoder-decoder model architecture.

[0264] At operation 1206, the processing device generates a first set of message content suggestions based on first message attribute data using a first machine learning model. At operation 1208, the processing device selects at least one message content suggestion from the first set of message content suggestions using the first machine learning model based on message evaluation data received by the first machine learning model from a second machine learning model. In some implementations, the second machine learning model includes a second encoder-decoder model architecture.

[0265] At operation 1210, the processing device, in response to the presentation of at least one selected message content suggestion at the message generation interface, receives feedback data related to the at least one selected message content suggestion via the message generation interface.

[0266] In some implementations, the feedback data is based on at least one interaction by the intended message sender with the message generation interface in response to presentation of the at least one message content suggestion by the message generation interface prior to sending, by the intended message sender, a message including the at least one message content suggestion to at least one intended message recipient. In some implementations, the feedback data is based on at least one interaction by the intended message recipient with the message reception interface in response to presentation of the message including the at least one message content suggestion by the message reception interface to the intended message recipient. In some implementations, the processing device tunes the second machine learning model based on the feedback data.

[0267] At operation 1212, the processing device tunes the first machine learning model based on the feedback data. At operation 1214, the processing device generates, by the tuned first machine learning model, a second set of message content suggestions based on the first message attribute data. In some implementations, the second set of message content suggestions includes at least one of the following: a restated version of a message content suggestion of the first set of message content suggestions, a reworded version of a message content suggestion, or an alternative version of a message content suggestion. In some implementations, the processing device receives second message attribute data via the message generation interface and, based on the second message attribute data, generates, by the first machine learning model, at least one of the following: a restated version of a message content suggestion of the first set of message content suggestions, a reworded version of a message content suggestion, or an alternative version of a message content suggestion. In some implementations, the processing device outputs, by the second machine learning model, estimated recipient acceptance data associated with a message content suggestion and presents the estimated recipient acceptance data to the intended message sender via the message generation interface.

[0268] In some implementations, the suggested message content generated and output by the first machine learning model includes any of the following: a video, an audio, and / or one or more digital images. In some implementations, the processing device presents the suggested message content to the user at the messaging interface and receives user input in response to the generative message suggestion, where the user input includes any of the following: one or more modifications to the generative message suggestion, one or more requests for new generative message suggestions, an action to incorporate the generative message suggestion into a new segment of digital content and cause the new segment of digital content to be distributed in a network of users, e.g., via a social networking service.

[0269] In some implementations, the technical problem of scalability is addressed by the processing device selecting a generative message suggestion from a set of generative message suggestions, where the size of the set of generative message suggestions is at least one order of magnitude smaller than the number of users of the messaging system.

[0270] In some implementations, the technical problem of efficient distribution of message suggestions is addressed by the processing device converting the generative message suggestion from a first size to a second size prior to distribution, where the second size is more efficient than the first size for distribution in a network of users.

[0271] In some implementations, the processing device configures the generative message suggestion based on one or more interaction parameters of the sending user; for example, the one or more interaction parameters are input to a machine learning model to cause the machine learning model to formulate the generative message suggestion to be suitable for presentation at end user devices having different screen sizes or resolutions and / or different device capabilities in order to facilitate interaction between the user and the message suggestion, resulting in improved message creation and distribution.

[0272] In some implementations, the technical problem of processing delay is addressed by the processing device detecting an increase in delay in outputting the generative message suggestion, and in response to detecting the increase in delay, performing one or more of the following actions: reducing a number of model inputs; or using a machine learning model having a reduced size; or reducing a size of the message suggestion (e.g., reducing a maximum text length or byte size).

[0273] For purposes of illustration, examples shown in Figure 12 and the above description attached thereto are provided. The present disclosure is not limited to the described examples.

[0274] Figure 13 is a block diagram of an example computer system including components of a generative message suggestion system in accordance with some embodiments of the present disclosure. In Figure 13 , an example machine is shown that is within which a set of instructions for causing the machine to perform any one or more of the methodologies discussed herein can be executed. In some embodiments, the computer system 1300 can correspond to components of a computing system 100 (e.g., as Figure 1 or a computer system 500 of Figure 5 that includes, is coupled to, or utilizes a machine to execute an operating system to perform operations corresponding to one or more components of a generative message suggestion system 108 of Figure 1 or a generative message suggestion system 580 of Figure 5 For example, when the computing system is executing a portion of the generative message suggestion system 580, the computer system 1300 corresponds to a portion of the computing system 500.

[0275] The machine is connected (e.g., networked) to other machines in a Local Area Network (LAN), an intranet, an extranet, and / or the Internet. The machine can operate in the capacity of a server or a client machine in client-server network environment, as a peer machine in peer-to-peer (or distributed) network environment, or as a server or a client machine in a cloud computing infrastructure or environment.

[0276] The machine is a personal computer (PC), a smart phone, a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, a web appliance, a wearable device, a server, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while a single machine is illustrated, the term "machine" includes any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any of the methodologies discussed herein.

[0277] Example computer system 1300 includes a processing device 1302, a main memory 1304 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM) or Rambus DRAM (RDRAM), etc.), a memory 1303 (e.g., flash memory, static random access memory (SRAM), etc.), an input / output system 1310, and a data storage system 1340, which communicate with each other via a bus 1330.

[0278] Processing device 1302 represents at least one general-purpose processing device such as a microprocessor, central processing unit, or the like. More particularly, the processing device can be complex instruction set computing (CISC) microprocessor, reduced instruction set computing (RISC) microprocessor, very long instruction word (VLIW) microprocessor, or processor implementing other instruction sets, or processors implementing a combination of instruction sets. Processing device 1302 can also be at least one special-purpose processing device such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. Processing device 1302 is configured to execute instructions 1312 for performing the operations and steps discussed herein.

[0279] In Figure 13When the computer system 1300 is executing those portions of the generative message suggestion system 580, the generative message suggestion system 1350 represents the portions of the generative message suggestion system 580. When those portions of the generative message suggestion system 1350 are being executed by the processing device 1302, the instructions 1312 include the portions of the generative message suggestion system 1350. Thus, the generative message suggestion system 1350 is shown in dashed line as part of the instructions 1312 to illustrate that at times the portions of the generative message suggestion system 1350 are being executed by the processing device 1302. For example, when at least some portions of the generative message suggestion system 1350 are embodied in instructions that cause the processing device 1302 to perform the method(s) described herein, some of the instructions can be read from the main memory 1304 and / or the data storage system 1340 into the processing device 1302 (e.g., into an internal cache or other memory). However, it is not required that all of the generative message suggestion system 1350 be included in the instructions 1312 at the same time, and portions of the generative message suggestion system 1350 are stored in at least one other component of the computer system 1300 at other times, for example, when at least a portion of the generative message suggestion system 1350 is not being executed by the processing device 1302.

[0280] The computer system 1300 also includes a network interface device 1308 to communicate over the network 1320. The network interface device 1308 provides a two-way data communication coupling to the network. For example, the network interface device 1308 can be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, the network interface device 1308 can be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links can also be implemented. In any such implementation, the network interface device 1308 can send and receive electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

[0281] The network link can provide data communication through at least one network to other data devices. For example, the network link can provide a connection to a worldwide packet data communication network, commonly referred to as the "Internet", through a local network to a host computer or data equipment operated by an Internet service provider (ISP). The local network and the Internet use electrical, electromagnetic or optical signals that carry digital data streams.

[0282] Computer system 1300 can send messages and receive data, including program code, through the network(s) and network interface device 1308. In the Internet example, a server might transmit a requested code for an application program through the Internet, and network interface device 1308. The received code can be executed by processor 1302 as it is received, and / or stored in data storage system 1340, or other non-volatile storage for later execution.

[0283] Input / output system 1310 includes output devices such as a display (e.g., a liquid crystal display (LCD) or touch screen display to show information to a computer user), or speakers, haptic devices, or another form of output device. Input / output system 1310 can include input devices, e.g., alphanumeric keyboards and other keys configured for communicating information and command selections to processor 1302. Alternatively or additionally, input devices can include cursor control devices, such as a mouse, scroll wheel, or cursor direction keys for communicating direction information and command selections to processor 1302, and for controlling cursor movement on the display. Alternatively or additionally, input devices can include microphones, sensors, or arrays of sensors for communicating sensed information to processor 1302. For example, sensed information can include voice commands, audio signals, geographic location information, and / or digital images.

[0284] Data storage system 1340 includes a machine-readable storage medium 1342 (also known as a computer-readable medium) on which is stored at least one set of instructions 1344 or software embodying any of the methodologies or functions described herein. The instructions 1344 might also reside completely, or at least partially, within the main memory 1304 and / or within the processor 1302 during execution thereof by the computer system 1300, the main memory 1304 and the processor 1302 also constituting machine-readable storage media.

[0285] In one embodiment, instructions 1344 include instructions to implement functionality corresponding to a generative message suggestion system, such as generative message suggestion system 108 of Figure 1 or generative message suggestion system 580 of Figure 5 .

[0286] Dotted lines represent optional message flows Figure 13The instructions 1312, 1313, and 1344 collectively embody the generative message suggestion system. In one example, portions of the generative message suggestion system are embodied in instructions 1344, instructions 1344 are read into main memory 1304 as instructions 1313, and portions of instructions 1313 are read into processing device 1302 as instructions 1312 for execution. In another example, some portions of the generative message suggestion system are embodied in instructions 1344, other portions are embodied in instructions 1313, and other portions are embodied in instructions 1312.

[0287] Although the machine -readable storage medium 1342 is shown in an example embodiment to be a single medium, the term "machine-readable storage medium" should be taken to include a single medium or multiple media that store the Figure 13 The examples shown in FIG. 13 and the above accompanying description are provided for illustrative purposes only. The present disclosure is not limited to the described examples.

[0288] Some portions of the preceding detailed descriptions have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, is considered to be a self- consistent sequence of operations leading to a desired result. The operations are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0289] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. The present disclosure can refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage systems.

[0290] The present disclosure also relates to an apparatus for performing the operations herein. This apparatus can be specially constructed for the intended purposes, or it can include a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. For example, a computer system or other data processing system (such as computing system 100 or computing system 500) can perform the computer-implemented methods described above in response to its processor executing a computer program (e.g., an instruction sequence) contained in a memory or other non-transitory machine-readable storage medium. Such a computer program can be stored in a computer readable storage medium, such as, but not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions, each coupled to a computer system bus.

[0291] The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general purpose systems can be used with programs in accordance with the teachings herein, or it can prove convenient to construct a more specialized apparatus to perform the method. The structure for a variety of these systems will appear in the description below. In addition, the present disclosure is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings of the disclosure as described herein.

[0292] The present disclosure can be provided as a computer program product, or software, that can include a machine-readable medium having stored thereon instructions, which can be used to program a computer system (or other electronic devices) to perform a process according to the present disclosure. A machine-readable medium includes any mechanism for storing information in a form accessible by a machine (e.g., a computer). In some embodiments, a machine-readable (e.g., computer-readable) medium includes a machine (e.g., a computer) readable storage medium such as read-only memory ("ROM"), random access memory ("RAM"), magnetic disk storage media, optical storage media, flash memory components, etc.

[0293] Illustrative examples of the technologies disclosed herein are provided below. An embodiment of the technology can include any one or combination of the examples described herein or any combination of any portions of the examples described herein.

[0294] In Example 1, a method comprising: configuring a first machine learning model to generate and output suggested message content based on a first correlation between message content and message acceptance data, wherein the first machine learning model comprises a first encoder-decoder model architecture; configuring a second machine learning model to generate and output message evaluation data based on a second correlation between the message content and the message acceptance data, wherein the second machine learning model comprises a second encoder-decoder model architecture; coupling an output of the first machine learning model to an input of the second machine learning model; and coupling an output of the second machine learning model to an input of the first machine learning model.

[0295] Example 2 includes the subject matter of Example 1, further including inputting the proposed message content output by the first machine learning model to the second machine learning model. Example 3 includes the subject matter of Example 2, further including inputting the message evaluation data output by the second machine learning model to the first machine learning model. Example 4 includes the subject matter of any of Examples 1-3, further including training the first machine learning model based on first training data, wherein the first training data includes positive examples of the message acceptance data. Example 5 includes the subject matter of Example 4, further including training the second machine learning model based on the first training data and second training data, wherein the second training data includes negative examples of the message acceptance data. Example 6 includes the subject matter of Example 4, further including formulating instances of the first training data to include message content, sender metadata associated with the message content, recipient metadata associated with the message content, and an acceptance label associated with the recipient metadata, wherein the acceptance label includes an indicator of (i) acceptance of a message including the message content by a recipient from a sender, (ii) rejection of the message by the recipient, or (iii) no response to the message by the recipient. Example 7 includes the subject matter of Example 6, further including anonymizing at least one of the message content, the sender metadata, or the recipient metadata; and formulating instances of the first training data using the anonymized at least one of the message content, the sender metadata, or the recipient metadata. Example 8 includes the subject matter of any of Examples 1-7, further including receiving pre- send feedback data related to the proposed message content via a message generation interface; and tuning at least one of the first machine learning model or the second machine learning model based on the received pre-send feedback data. Example 9 includes the subject matter of Example 8, wherein the pre-send feedback data is based on at least one interaction of an intended message sender with the message generation interface in response to presentation of the proposed message content by the message generation interface prior to sending of a message including the proposed message content by the intended message sender to at least one recipient. Example 10 includes the subject matter of any of Examples 1-9, further including receiving post- send feedback data related to the proposed message content via a message reception interface; and tuning at least one of the first machine learning model or the second machine learning model based on the received post-send feedback data. Example 11 includes the subject matter of any of Examples 1-10, wherein the post-send feedback data is based on at least one interaction of an intended message recipient with the message reception interface in response to presentation of a message including the proposed message content by the message reception interface to the intended message recipient.Example 12 includes the subject matter of any one of Examples 1-11, further comprising, for an instance of suggested message content, determining a model input to which to apply the first machine learning model to generate the instance of suggested message content; determining a difference between the instance of suggested message content and the model input; and tuning the first machine learning model based on the difference between the instance of suggested message content and the model input.

[0296] Example 13 includes a method comprising: any one or more steps, operations, or processes shown in any of the accompanying drawings or described in the specification, and performed by any one or more of the systems, devices, components, subsystems, or elements shown in any of the accompanying drawings or described in the specification.

[0297] Example 14 includes a system comprising: at least one processor; and at least one memory coupled to the at least one processor; wherein the at least one memory comprises instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising any one or more of Examples 1-13. Example 15 includes a non-transitory computer-readable medium comprising at least one memory that can be coupled to at least one processor, wherein the at least one memory comprises instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising any one or more of Examples 1-13.

[0298] In Example 21, a method comprising: receiving first message attribute data via a message generation interface; inputting the first message attribute data to a first machine learning model, wherein the first machine learning model is configured to generate and output suggested message content based on a first correlation between message content and message acceptance data; generating, by the first machine learning model, a first set of message content suggestions based on the first message attribute data; selecting, by the first machine learning model, at least one message content suggestion from the first set of message content suggestions based on message evaluation data received by the first machine learning model from a second machine learning model; receiving, via the message generation interface, feedback data related to the selected at least one message content suggestion in response to a presentation of the selected at least one message content suggestion at the message generation interface; tuning the first machine learning model based on the feedback data; and generating, by the tuned first machine learning model, a second set of message content suggestions based on the first message attribute data.

[0299] Example 22 includes the subject matter of Example 21, wherein the second set of message content suggestions includes at least one of: a restated version of a message content suggestion of the first set of message content suggestions, a rewording of the message content suggestion, or an alternative version of the message content suggestion. Example 23 includes the subject matter of Example 22, further comprising: receiving, via the message generation interface, second message attribute data; and based on the second message attribute data, generating, by the first machine learning model, at least one of: a restated version of a message content suggestion of the first set of message content suggestions, a rewording of the message content suggestion, or an alternative version of the message content suggestion. Example 24 includes the subject matter of any of Examples 21-23, further comprising: outputting, by the second machine learning model, estimated recipient acceptance data associated with the message content suggestion; and presenting, via the message generation interface, the estimated recipient acceptance data to the intended message sender. Example 25 includes the subject matter of any of Examples 21-24, further comprising: determining, based on a social graph, a link between a first entity and a second entity, wherein at least one of the first entity or the second entity represents an intended message recipient in the social graph; and determining the first message attribute data based on the link. Example 26 includes the subject matter of any of Examples 21-25, further comprising: tuning the second machine learning model based on the feedback data. Example 27 includes the subject matter of any of Examples 21-26, wherein the feedback data is based on at least one interaction of an intended message sender with the message generation interface in response to a presentation of the at least one message content suggestion by the message generation interface prior to a sending of a message by the intended message sender to at least one intended message recipient that includes the at least one message content suggestion. Example 28 includes the subject matter of any of Examples 21-27, wherein the feedback data is based on at least one interaction of an intended message recipient with the message reception interface in response to a presentation of a message by the message reception interface to the intended message recipient that includes the at least one message content suggestion. Example 29 includes the subject matter of any of Examples 21-28, wherein the first machine learning model comprises a first encoder-decoder model architecture. Example 30 includes the subject matter of Example 29, wherein the second machine learning model comprises a second encoder-decoder model architecture.

[0300] Example 31 includes a method comprising any one or more of the steps, operations, or processes shown in any of the accompanying drawings, and described in the specification, and performed by any one or more of the systems, devices, components, subsystems, or elements shown in any of the accompanying drawings, and described in the specification. Example 32 includes a system comprising at least one processor; and at least one memory coupled to the at least one processor; wherein the at least one memory comprises instructions which, when executed by the at least one processor, cause the at least one processor to perform operations comprising any one or more of Examples 21-31. Example 33 includes a non-transitory computer-readable medium comprising at least one memory that can be coupled to at least one processor, wherein the at least one memory comprises instructions which, when executed by the at least one processor, cause the at least one processor to perform operations comprising any one or more of claims 21-31.

[0301] Example 41 includes the subject matter of any of the preceding Examples, wherein the suggested message content (e.g., generative message suggestion) generated and output by the first machine learning model comprises any one of a video, audio, and / or one or more digital images.

[0302] Example 42 includes the subject matter of any of the preceding Examples, wherein the processing device presents the generative message suggestion to the user at a messaging interface and receives user input in response to the generative message suggestion, wherein the user input comprises any one of one or more modifications to the generative message suggestion, one or more requests for a new generative message suggestion, an action to incorporate the generative message suggestion into a new segment of digital content and cause the new segment of digital content to be distributed in a network of users, e.g., via a social networking service.

[0303] Example 43 includes the subject matter of any of the preceding Examples, wherein the processing device selects the generative message suggestion from a set of generative message suggestions, wherein a size of the set of generative message suggestions is at least one order of magnitude smaller than a number of users of the messaging system.

[0304] Example 44 includes the subject matter of any of the preceding Examples, wherein the processing device converts the generative message suggestion from a first size to a second size prior to distribution, wherein the second size is more efficient than the first size for presentation to the user or for distribution in the network of users.

[0305] Example 45 includes the subject matter of any of the preceding examples, wherein the processing device configures the generative message suggestion based on one or more interaction parameters of the sending user; for example, the one or more interaction parameters are input to a machine learning model to cause the machine learning model to formulate the generative message suggestion to be suitable for presentation at an end-user device having a different screen size or resolution and / or different device capabilities, so as to facilitate interaction between the user and the message suggestion, resulting in improved message creation and distribution.

[0306] Example 46 includes the subject matter of any of the preceding examples, wherein the processing device detects an increase in latency to output a generative message suggestion, and in response to detecting the increase in latency, performs one or more of: reducing a number of the model inputs; or using a machine learning model having a reduced size; or reducing a size of the message suggestion (e.g., reducing a maximum text length or byte size).

[0307] In the foregoing specification, embodiments of the disclosure have been described with reference to specific examples thereof. It is evident, however, that various modifications can be made thereto without departing from the broader spirit and scope of the disclosure as set forth in the appended claims. The specification and drawings are, accordingly, to be regarded in an illustrative sense rather than a restrictive sense.

Claims

1. A method comprising: receiving (1202), via a message generation interface, first message attribute data; inputting (1204) the first message attribute data to a first machine learning model, wherein the first machine learning model is configured to generate and output suggested message content using a first correlation between message content and message acceptance data; generating (1206), by the first machine learning model, a first set of message content suggestions using the first message attribute data; selecting (1208), by the first machine learning model, at least one message content suggestion from the first set of message content suggestions using message evaluation data received by the first machine learning model from a second machine learning model; receiving (1210), via the message generation interface, feedback data related to the selected at least one message content suggestion in response to a presentation of the selected at least one message content suggestion at the message generation interface; tuning (1212) the first machine learning model using the feedback data; and generating (1214), by the tuned first machine learning model, a second set of message content suggestions using the first message attribute data.

2. The method of claim 1, wherein, the second set of message content suggestions includes at least one of a reworded version of a message content suggestion in the first set of message content suggestions, a rephrased version of the message content suggestion, or an alternative version of the message content suggestion.

3. The method of claim 2, further comprising: receiving second message attribute data via the message generation interface; and generating, by the first machine learning model, using the second message attribute data, at least one of a reworded version of a message content suggestion in the first set of message content suggestions, a rephrased version of the message content suggestion, or an alternative version of the message content suggestion.

4. The method of claim 1, further comprising: outputting, by the second machine learning model, estimated recipient acceptance data associated with the at least one message content suggestion; and presenting, via the message generation interface, the estimated recipient acceptance data to an intended message sender.

5. The method of claim 1, further comprising: determining a link between a first entity and a second entity from a social graph, wherein at least one of the first entity or the second entity represents an intended message recipient in the social graph; and determining the first message attribute data using the link.

6. The method of claim 1, further comprising: tuning the second machine learning model using the feedback data.

7. The method of claim 1, wherein, the feedback data is from at least one interaction of an intended message sender with the message generation interface in response to a presentation of the at least one message content suggestion by the message generation interface prior to a sending of a message including the at least one message content suggestion by the intended message sender to at least one intended message recipient.

8. The method of claim 1, wherein, the feedback data is from at least one interaction of an intended message recipient with the message reception interface in response to presentation of a message including the at least one message content suggestion to the intended message recipient by the message reception interface.

9. The method of claim 1, wherein, the first machine learning model comprises a first encoder-decoder model architecture.

10. The method of claim 9, wherein, the second machine learning model comprises a second encoder-decoder model architecture.

11. A system comprising: at least one processor; and at least one memory coupled to the at least one processor, wherein the at least one memory comprises instructions that, when executed by the at least one processor, cause the at least one processor to perform at least one operation comprising: receiving (1202), via a message generation interface, first message attribute data; inputting (1204) the first message attribute data to a first machine learning model, wherein the first machine learning model is configured to generate and output message content suggestions using a first correlation between message content and message acceptance data; generating (1206), by the first machine learning model, a first set of message content suggestions using the first message attribute data; selecting (1208), by the first machine learning model, at least one message content suggestion from the first set of message content suggestions using message evaluation data received by the first machine learning model from a second machine learning model; receiving (1210), via the message generation interface, feedback data related to the selected at least one message content suggestion in response to presentation of the selected at least one message content suggestion at the message generation interface; tuning (1212) the first machine learning model using the feedback data; and generating (1214), by the tuned first machine learning model, a second set of message content suggestions using the first message attribute data.

12. The system of claim 11, wherein, the second set of message content suggestions comprises at least one of the following: a reworded version of a message content suggestion in the first set of message content suggestions, a rephrased version of the message content suggestion, or an alternative version of the message content suggestion; and the instructions, when executed by the at least one processor, cause the at least one processor to perform at least one operation further comprising: receiving second message attribute data via the message generation interface; and generating, by the first machine learning model, at least one of the following using the second message attribute data: a reworded version of a message content suggestion in the first set of message content suggestions, a rephrased version of the message content suggestion, or an alternative version of the message content suggestion.

13. The system of claim 11, wherein, the instructions, when executed by the at least one processor, cause the at least one processor to perform at least one operation further comprising: outputting, by the second machine learning model, estimated recipient acceptance data associated with the at least one message content suggestion; and presenting the estimated recipient acceptance data to an intended message sender via the message generation interface. ​ 14. The system of claim 11, wherein, The instructions, when executed by the at least one processor, cause the at least one processor to perform at least one operation further comprising: determining, from a social graph, a link between a first entity and a second entity, wherein at least one of the first entity or the second entity represents an intended message recipient in the social graph; and using the link to determine the first message property data.

15. The system of claim 11, wherein, The first machine learning model comprises a first encoder-decoder model architecture and the second machine learning model comprises a second encoder-decoder model architecture.

16. At least one non-transitory computer-readable medium comprising at least one memory able to be coupled to at least one processor, wherein, The at least one memory comprises instructions that, when executed by the at least one processor, cause the at least one processor to perform at least one operation comprising: receiving (1202), via a message generation interface, first message property data; inputting (1204) the first message property data to a first machine learning model, wherein the first machine learning model is configured to generate and output suggested message content using a first correlation between message content and message acceptance data; generating (1206), by the first machine learning model, a first set of message content suggestions using the first message property data; selecting (1208), by the first machine learning model, at least one message content suggestion from the first set of message content suggestions using message evaluation data received by the first machine learning model from a second machine learning model; receiving (1210), via the message generation interface, feedback data related to the selected at least one message content suggestion in response to a presentation of the selected at least one message content suggestion at the message generation interface; tuning (1212) the first machine learning model using the feedback data; and generating (1214), by the tuned first machine learning model, a second set of message content suggestions using the first message property data.

17. The at least one non-transitory computer readable medium of claim 16, wherein, The second set of message content suggestions comprises at least one of: a reworded version of a message content suggestion in the first set of message content suggestions, a rephrased version of the message content suggestion, or an alternative version of the message content suggestion; and The instructions, when executed by the at least one processor, cause the at least one processor to perform at least one operation further comprising: receiving second message property data via the message generation interface; and generating, by the first machine learning model, using the second message property data, at least one of: a reworded version of a message content suggestion in the first set of message content suggestions, a rephrased version of the message content suggestion, or an alternative version of the message content suggestion.

18. The at least one non-transitory computer readable medium of claim 16, wherein, The instructions, when executed by the at least one processor, cause the at least one processor to perform at least one operation further comprising: outputting, by the second machine learning model, estimated recipient acceptance data associated with the at least one message content suggestion; and presenting, via the message generation interface, the estimated recipient acceptance data to an intended message sender.

19. The at least one non-transitory computer readable medium of claim 16, wherein, The instructions, when executed by the at least one processor, cause the at least one processor to perform at least one operation further comprising: determining a link between a first entity and a second entity from a social graph, wherein at least one of the first entity or the second entity represents an intended message recipient in the social graph; and using the link to determine the first message property data.

20. The at least one non-transitory computer readable medium of claim 16, wherein, The first machine learning model comprises a first encoder-decoder model architecture and the second machine learning model comprises a second encoder-decoder model architecture. The first machine learning model comprises a first encoder-decoder model architecture and the second machine learning model comprises a second encoder-decoder model architecture.

Images