Predicting marketing outcomes using comparative learning

Abstract

Techniques for predicting marketing outcomes using comparative learning are disclosed, including: acquiring historical marketing information; obtaining historical opening rates associated with the historical marketing messages; generating potential spatial representations respectively associated with the historical marketing messages based on the historical marketing messages; training a first machine learning model to map the content of the marketing message to the corresponding potential spatial representation of the marketing message based on the potential spatial representation and the corresponding content of the historical marketing message; based at least on the potential spatial characterization and the historical opening rate, a second machine learning model is trained to map the potential spatial characterization of the marketing message to a predicted opening rate of the marketing message.

Description

Technical Field

[0001] The present disclosure relates to predicting marketing results. Specifically, the present disclosure relates to predicting marketing results using machine learning. Background Art

[0002] Generally, it is difficult to predict how successful a particular marketing campaign will be. For example, given a particular marketing email message, it is difficult to predict what the open rate of that message will be. One approach is to perform natural language processing (NLP) on the subject lines of the email messages and attempt to correlate the semantics of each subject line with its open rate. However, the semantics of email subject lines rarely correlate with their corresponding open rates. Therefore, typical approaches that rely on techniques such as regression modeling tend to be ineffective for predicting marketing results.

[0003] Methods that rely on human judgment (e.g., hiring marketing experts to analyze previous campaigns and craft subject lines for new campaigns) tend to be ineffective for similar reasons; the semantics of email subject lines (i.e., those parts that are most easily discerned by human readers) rarely correlate with their corresponding open rates. NLP, regression modeling, etc. tend to suffer similar results for other kinds of campaign prediction, such as predicting click-through rates for a given call to action (e.g., text- and / or image-based calls to action), progression rates at different stages of a marketing funnel, etc.

[0004] A method for predicting marketing outcomes is described in U.S. patent application Ser. No. 17 / 100,525, filed on November 20, 2020, and entitled “Techniques for Selecting Content to Include in User Communications,” the entire contents of which are incorporated herein by reference. There, user-facing content (e.g., email subject lines) is used to generate a vocabulary of “token sets,” where each token set can be associated with a performance parameter representing the impact that the token set has on a target outcome (e.g., open rate). By focusing on specific phrases, this approach overcomes some of the problems associated with semantic analysis while still relying on analysis of user-facing content, i.e., specific phrases that the intended recipient expects to see (e.g., single-word phrases, two-word phrases, three-word phrases, etc.).

[0005] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The embodiments are illustrated by way of example and not by way of limitation in the figures of the accompanying drawings. References to "one" or "an" embodiment in the present disclosure do not necessarily refer to the same embodiment, and mean at least one. In the figures:

[0007] Figure 1 shows a block diagram illustrating an example of a system according to one or more embodiments;

[0008] Figure 2 illustrates an example set of operations for predicting marketing outcomes using contrastive learning in accordance with one or more embodiments;

[0009] Figure 3 An example set of operations for generating a latent space representation is illustrated in accordance with one or more embodiments.

[0010] Figure 4 An example set of operations for training a contents-to-LSR model is illustrated in accordance with one or more embodiments.

[0011] Figure 5A-Figure 5B illustrates an example of using contrastive learning to predict marketing outcomes according to one or more embodiments; and

[0012] Figure 6 A block diagram illustrating a computer system in accordance with one or more embodiments is shown. DETAILED DESCRIPTION

[0013] In the following description, for the purpose of explanation and in order to provide a thorough understanding, many specific details are set forth. One or more embodiments may be practiced without these specific details. Features described in one embodiment may be combined with features described in different embodiments. In some examples, well-known structures and devices are described in reference block diagram form to avoid unnecessarily obscuring the present invention.

[0014] The following list is for reference only and should not be construed as limiting the scope of one or more embodiments.

[0015] 1. General Overview

[0016] 2. Example System

[0017] 2.1. System Components

[0018] 2.2. Marketing platform

[0019] 2.3. Data Storage

[0020] 2.4. User Interface

[0021] 2.5. Tenants

[0022] Machine Learning

[0023] 3. Use Contrastive Learning to Predict Marketing Outcomes

[0024] 3.1. Process Overview

[0025] 3.2. Generate LSR for historical marketing materials

[0026] 3.3. Training content to LSR model

[0027] 4. Example Embodiments

[0028] 5. Sample Subject Line

[0029] 6. Content Distribution

[0030] 7. Computer Networks and Cloud Networks

[0031] 8. Microservice Applications

[0032] 8.1. Trigger

[0033] 8.2. Actions

[0034] 9. Hardware Overview

[0035] 10. Miscellaneous; Extensions

[0036] 1. General Overview

[0037] One or more embodiments predict marketing results of marketing materials without relying on attempts to correlate semantic content (e.g., email subject lines) with marketing results (e.g., open rates). One or more embodiments use contrastive learning to train a machine learning model to generate a latent space representation of the marketing materials based on historical marketing results that are independent of the content of the corresponding historical marketing materials. One or more embodiments also train a machine learning model to predict marketing results based on the latent space representation. The two machine learning models can together predict marketing results without relying on highly fallible semantic analysis techniques. Therefore, when (a) the relationship between semantic content and actual marketing results is known to be highly variable and unreliable, and (b) the correlation between marketing materials and their corresponding results depends on non-content attributes and is therefore not easily discernible by humans, one or more embodiments provide a technical solution to the technical problem of predicting marketing results.

[0038] In an embodiment, predicting marketing results allows marketing strategies to be based on consistent, quantifiable metrics rather than subjective decisions. In response to predicting marketing results for one or more sets of marketing materials, the system can present (e.g., in a user interface) a marketing recommendation. For example, the system can recommend publishing a particular set of marketing materials over another set of marketing materials based on the predicted marketing results corresponding to the particular set of marketing materials and the other set of marketing materials. Alternatively or additionally, in response to comparing the predicted marketing results, the system can perform one or more actions without user input, such as publishing the marketing materials with the best predicted results.

[0039] One or more embodiments described in the specification and / or recited in the claims may not be included in this general summary.

[0040] 2. Example System

[0041] 2.1. System Components

[0042] Figure 1 An example of a system 100 is shown in FIG. Figure 1 As shown in FIG. 1 , system 100 includes a marketing platform 102, a data warehouse 106, an interface 108, one or more tenants 110, and their respective components. In one or more embodiments, system 100 may include: Figure 1 More or fewer components than those shown in the illustrations. Figure 1 The components illustrated in may be local or remote from each other. Figure 1 The components illustrated in the figure may be implemented in software and / or hardware. Each component may be distributed across multiple applications and / or machines. Multiple components may be combined into one application and / or machine. Operations described with respect to one component may instead be performed by another component. Additional embodiments and / or examples related to computer networks are described below in the section entitled "Computer Networks and Cloud Networks."

[0043] In one or more embodiments, one or more components of system 100 are implemented on one or more digital devices. The term "digital device" generally refers to any hardware device including a processor. A digital device may refer to a physical device that executes an application or a virtual machine. Examples of digital devices include computers, tablet computers, laptops, desktops, netbooks, servers, web servers, network policy servers, proxy servers, general machines, specific function hardware devices, hardware routers, hardware switches, hardware firewalls, hardware firewalls, hardware network address translators (NATs), hardware load balancers, mainframes, televisions, content receivers, set-top boxes, printers, mobile handheld terminals, smart phones, personal digital assistants ("PDAs"), wireless receivers and / or transmitters, base stations, communication management equipment, routers, switches, controllers, access points, and / or client devices.

[0044] 2.2. Marketing platform

[0045] In general, the marketing platform 102 refers to hardware and / or software configured to perform operations associated with digital marketing services (e.g., email marketing, social media marketing, SMS marketing, etc.). Such services may include, for example: generating marketing materials; publishing / delivering marketing materials; tracking the results of specific marketing materials (e.g., end-user click-through rates / counts, open rates / counts, view rates / counts, etc.); analyzing marketing materials for opportunities to improve results (e.g., increase click-through rates, open rates, and / or view rates / counts); etc. In particular, the marketing platform 102 is configured to perform operations for predicting marketing results using contrastive learning, examples of which are described herein.

[0046] In an embodiment, a latent space representation (LSR) generator 104 is configured to generate an LSR of historical marketing materials 120. An LSR is a representation of the historical marketing materials 120 in a latent space, where materials that are most similar according to one or more specified characteristics are located closest to each other in the latent space. For example, an LSR may be an N-dimensional vector, each vector representing a subset of the underlying historical marketing materials 120 (e.g., a particular email, social media post, or text message), where the vectors representing the most similar materials are located closest to each other in the N-dimensional latent space. The LSR generator 104 may be configured to generate an LSR, where proximity in the latent space is associated with corresponding marketing results 126 of the historical marketing materials 120. For example, if the historical marketing materials 120 include emails, the marketing results 126 may include corresponding open rates, and the LSRs associated with the most similar open rates may be located closest to each other in the latent space. An example of an operation for generating an LSR of historical marketing materials 120 will be described in further detail below.

[0047] In an embodiment, the machine learning platform 112 is configured to train and / or apply one or more machine learning models using one or more machine learning algorithms 114. Specifically, the (one or more) machine learning algorithms 114 may be configured to perform one or more operations described herein for training the content to LSR model 116 and / or the LSR to results model 118. The content to LSR model 116 is a machine learning model configured to generate an LSR of marketing materials based on the content of the marketing materials (e.g., an email subject line). The LSR to results model 118 is a machine learning model configured to generate predicted marketing results of the marketing materials based on the corresponding LSR of the marketing materials. Examples of operations for training and using the content to LSR model 116 and / or the LSR to results model 118 will be described in further detail below.

[0048] 2.3. Data Storage

[0049] In one or more embodiments, data warehouse 106 is any type of storage unit and / or device for storing data (e.g., a file system, a database, a collection of tables, or any other storage mechanism). In an embodiment, data warehouse 106 is configured to store historical marketing materials 120. Historical marketing materials 120 may include examples of previous marketing materials that have been presented to potential customers. For example, historical marketing materials 120 may include emails, text messages, social media ads, web pages, multimedia files, and / or another type of marketing material that has been presented to potential customers.

[0050] The content 122 includes target-oriented attributes of the historical marketing material 120. For example, the content 122 may include text (if any) presented to potential consumers and / or multimedia content included in the marketing material. Specifically, the content 122 may include an email subject line, which is typically the only content visible to the recipient before the email is opened. The non-content attributes 124 include non-target-oriented attributes of the historical marketing material 120. For example, the non-content attributes 124 may include metadata embedded in the historical marketing material 120 or otherwise associated with the historical marketing material 120. Specifically, the non-content attributes 124 include marketing results 126, which indicate how well the corresponding historical marketing material 120 performed according to one or more performance indicators (e.g., open rate, click-through rate, progress rate of different stages of the marketing funnel, etc.).

[0051] In an embodiment, the data warehouse 106 is configured to store untagged marketing materials 128. Untagged marketing materials 128 are marketing materials that have not been tagged with marketing results. Marketing results corresponding to the untagged marketing materials 128 may be unknown (e.g., if the marketing materials are not yet live), unreliable, and / or otherwise unavailable (e.g., if a particular publishing platform does not expose an interface for obtaining performance metrics). As an example, the untagged marketing materials 128 may include prospective marketing materials that are being evaluated prior to publishing. The untagged marketing materials 128 may include materials that are already live and / or materials that are not yet live. The untagged marketing materials 128 may include different versions of a proposed marketing campaign (e.g., an email campaign), where the marketing results predicted for each version can help decide which version to publish.

[0052] The data warehouse 106 may include a plurality of different storage units and / or devices. The plurality of different storage units and / or devices may or may not be of the same type or located at the same physical site. The data warehouse 106 may be implemented or executed on the same computing system as the marketing platform 102 and / or on a computing system separate from the marketing platform 102. The data warehouse 106 may be communicatively coupled to one or more other components of the system 100 via a direct connection or via a network. Information describing the historical marketing material 120 and / or the untagged marketing material 128 may be implemented across any of the components of the system 100. However, for purposes of clarity and explanation, this information is shown within the data warehouse 106.

[0053] 2.4. User Interface

[0054] In one or more embodiments, interface 108 refers to hardware and / or software configured to facilitate communication between a user and marketing platform 102. Interface 108 presents user interface elements and receives input via user interface elements. Examples of interfaces include graphical user interfaces (GUIs), command line interfaces (CLIs), tactile interfaces, and voice command interfaces. Examples of user interface elements include check boxes, radio buttons, drop-down lists, list boxes, buttons, switches, text fields, date and time selectors, command lines, sliders, pages, and forms. Different components of interface 108 can be specified in different languages. For example, the behavior of user interface elements can be specified in dynamic programming languages ​​such as JavaScript. The content of user interface elements can be specified in markup languages ​​such as hypertext markup language (HTML) or XML user interface language (XUL). The layout of user interface elements can be specified in style sheet languages ​​such as cascading style sheets (CSS). Alternatively, interface 108 can be specified in one or more other languages ​​such as Java, Python, C, or C++.

[0055] 2.5. Tenants

[0056] In one or more embodiments, tenant 110 is a company, organization, enterprise, or other entity that accesses a shared computing resource, such as marketing platform 102. System 100 may include multiple tenants 110 that are independent of each other, such that the business or operations of one tenant are separate from the business or operations of another tenant. Some examples of multi-tenant architectures according to one or more embodiments are described in further detail below.

[0057] Machine Learning

[0058] In one or more embodiments, the machine learning algorithm 114 is an algorithm that can be iterated to use a set of training data (e.g., historical marketing materials 120) to learn a target model (e.g., a content to LSR model 116 and / or an LSR to results model 118) that best maps a set of input variables to one or more output variables. The training data includes a data set and an associated label. The data set is associated with the input variables of the target model. The associated label is associated with the (one or more) output variables of the target model. For example, a label associated with a data set in the historical marketing material 120 can indicate one or more corresponding marketing results 126. The training data can be updated based on, for example, feedback on the accuracy of the current target model. The updated training data can be fed back into the machine learning algorithm 114, which can then update the target model.

[0059] The machine learning algorithm 114 may generate a target model such that the target model best fits the dataset of training data to the labels of the training data. Specifically, the machine learning algorithm 114 may generate a target model such that when the target model is applied to the dataset of training data, the maximum number of results determined by the target model matches the labels of the training data (e.g., historical marketing materials 120). For example:

[0060] 1. The machine learning algorithm 114 may generate a content-to-LSR model 116 such that when applied to content 122 (e.g., email subject lines) of historical marketing material 120, the content-to-LSR model 116 outputs an LSR that matches the LSR generated by the LSR generator 104 for the corresponding material.

[0061] 2. The machine learning algorithm 114 may generate an LSR-to-results model 118 such that when applied to the LSR generated by the LSR generator 104 and / or the content-to-LSR model 116, the LSR-to-results model 118 outputs predicted marketing results that match the marketing results 126 of the corresponding material.

[0062] Generate different target models based on different machine learning algorithms and / or different training datasets.

[0063] The machine learning algorithm 114 may include supervised and / or unsupervised components. Various types of algorithms may be used, such as linear regression, logistic regression, linear discriminant analysis, classification and regression trees, naive Bayes, k-nearest neighbors, learning vector quantization, support vector machines, bagging and random forests, boosting, back propagation, and / or clustering.

[0064] In an embodiment, the machine learning algorithm 114 is configured to perform contrastive learning. Typically, contrastive learning trains a machine learning model to identify similarities in unlabeled data by comparing (or "contrasting") data points to each other. One or more embodiments use contrastive learning to identify similarities between historical marketing materials 120 without relying on semantic analysis of target-oriented content 122 (such as email subject lines). The techniques of contrastive learning are discussed in further detail below.

[0065] 3. Use Contrastive Learning to Predict Marketing Outcomes

[0066] 3.1 Process Overview

[0067] Figure 2 Illustrated is an example set of operations for predicting marketing outcomes using contrastive learning in accordance with one or more embodiments. Figure 2 One or more operations illustrated in the figure may be modified, rearranged or omitted altogether. Figure 2 The particular sequence of operations illustrated in the drawings should not be construed as limiting the scope of one or more embodiments.

[0068] In order to train one or more machine learning models, the system can obtain historical marketing materials (operation 202). As described above, the marketing materials include marketing content and non-content attributes. The non-content attributes can include marketing results associated with the historical marketing materials.

[0069] The system can generate LSRs for historical marketing materials (operation 204). Generating the LSRs can be performed in various ways. In an embodiment, the LSRs are vectors representing the corresponding marketing materials. The system can generate the LSRs based on non-content attributes without considering the corresponding content of the marketing materials (e.g., without considering the email subject line). For example, the system can generate the LSRs so that the LSRs for marketing materials with the most similar marketing results are closest to each other in latent space, even if the target-oriented content of those marketing materials is very different (i.e., the content would not be considered a close match if compared using semantic analysis techniques). The following is about Figure 3 Some operational examples for generating LSRs are described in further detail.

[0070] The system can train a machine learning model to generate LSRs based on the corresponding content of the marketing materials (operation 206), i.e., a content-to-LSR machine learning model (also referred to herein as a "C2LSR model" for ease of discussion). Specifically, given the content of historical marketing materials (e.g., email subject lines) in the absence of corresponding marketing results, the system trains the C2LSR model to generate LSRs that match those generated as described above with respect to FIG. 204. Thus, the C2LSR is trained to generate LSRs that tend to be closest to each other in latent space when the predicted marketing results are most similar—even if (1) the earlier generated LSRs were generated without considering the corresponding content of the marketing materials, and (2) the C2LSR model was trained without considering the corresponding marketing results of the marketing materials. The following describes the C2LSR model. Figure 4 Some operational examples for training the C2LSR model are described in further detail.

[0071] The system can train a machine learning model to generate predicted marketing results based on the LSRs associated with the corresponding marketing materials (operation 208), i.e., an LSR to results machine learning model (also referred to herein as a "LSR2O model" for ease of discussion). Specifically, with the LSRs associated with historical marketing materials, the system trains the LSR2O model to generate predicted marketing results that match those predicted marketing results known for the corresponding marketing materials. Thus, the LSR2O model is trained to generate predicted marketing results for these marketing materials given only the LSRs associated with the marketing materials. In an embodiment, the LSR2O model is a dense neural network comprising one or two hidden layers. As discussed herein, the neural network can be configured to generalize from known content (e.g., subject lines) to all possible content (e.g., possible subject lines) by training the model on known content. A model trained on LSRs for known content can be configured to predict marketing results for any LSR (whether known or unknown).

[0072] In an embodiment, the system obtains untagged marketing materials (operation 210). For example, the untagged marketing materials may be stored as drafts in a marketing platform. To predict marketing results for the untagged marketing materials, the system may apply a C2LSR model to the untagged marketing materials to obtain corresponding LSRs (operation 212). The system may then apply an LSR2O model to the LSRs to obtain predicted marketing results (operation 214). In both operations, the system may not perform any form of semantic analysis on the content of the untagged marketing materials; the machine learning model may generate predictions without regard to the specific semantic structure or content of the content.

[0073] 3.2. Generate LSR for historical marketing materials

[0074] Figure 3 Illustrated is an example set of operations for generating an LSR for historical marketing material in accordance with one or more embodiments. Figure 3 One or more operations illustrated in the figure may be modified, rearranged or omitted altogether. Figure 3 The particular sequence of operations illustrated in the drawings should not be construed as limiting the scope of one or more embodiments.

[0075] In an embodiment, Figure 3 The operations illustrated in the figure correspond to Figure 2 Operation 204. The LSR generated according to these operations can be a vector representing the corresponding marketing material. The system can generate the LSR based on non-content attributes without considering the corresponding content of the marketing material. For example, the system can generate the LSR so that the LSRs of marketing materials with the most similar marketing results are closest to each other in the latent space, even if the target-oriented content (e.g., email subject lines) of those marketing materials are very different.

[0076] In an embodiment, the system selects pairs of historical marketing materials that are associated with similar marketing results (operation 302). For example, if the historical marketing materials are emails previously sent to prospective customers and / or existing customers, the system may select pairs of emails that have similar open rates. The system may pair marketing materials that have the same results at a certain level of precision (e.g., emails with the same open rates within one decimal point or another degree of precision). Alternatively or additionally, the system may pair marketing materials that have results that are within a certain range of each other (e.g., emails with open rates that are within five percent or another predetermined range of each other). More than two emails may meet the pairing criteria, and the system may select multiple permutations of the marketing material pairs accordingly.

[0077] As described above, the LSR may be a vector. The system may generate a vector representing the marketing material (operation 304). For example, the system may generate a vector of numbers corresponding to content and / or non-content attributes of the marketing material.

[0078] Given a pair of vectors corresponding to a pair of marketing materials, the system can calculate the dot product of the vector pair (e.g., the dot product can be scaled as described below) (operation 306). The system can then determine whether the dot product satisfies a threshold condition for generating an LSR (operation 308). The goal is to obtain vectors that are very similar when the corresponding marketing results are very similar. Therefore, the threshold condition can be a function of the difference in results for the two materials. For example, the system can use a function that scales the dot product so that the target value is close to one (or another predetermined value) when the difference in results is small and the target value is close to zero (or another predetermined value) when the difference in results is large. Therefore, the threshold condition may require, for example, that the scaled dot product is a specific value, or within a certain range of values ​​on a scale of zero to one, proportional to the difference between the two marketing results.

[0079] If the threshold condition is not met, the system may adjust the vectorization formula (operation 310). For example, in a method using a neural network, the system may adjust the vectorization formula using back propagation while training the neural network.

[0080] The system can then generate new vectors for the marketing material pairs using the adjusted formula (operation 304), calculate the scaled dot product of the new vectors (operation 306), and determine whether the new result meets the threshold condition (operation 308). This process can be iterated until the scaled dot product meets the threshold condition, at which point the system can use the vectors as LSRs for the historical marketing materials. The system can use these LSRs to train a C2OSR model as described herein. For example, in an approach using a neural network, the neural network can continue to be trained until the constraints for all LSR pairs are satisfied as much as possible.

[0081] In an embodiment, the system uses machine learning to generate LSRs for historical marketing materials. The machine learning can follow a similar approach to the above operation, i.e., iterating vector pairs and adjusting the vectorization formula until the vectors meet the threshold criteria. The system can determine that the machine learning is complete when all vector pairs corresponding to the historical marketing materials or a threshold percentage of vector pairs meet the threshold criteria.

[0082] 3.3. Training content to LSR model

[0083] Figure 4 An example set of operations for training a C2LSR model is illustrated in accordance with one or more embodiments. Figure 4 One or more operations illustrated in the figure may be modified, rearranged or omitted altogether. Figure 4 The particular sequence of operations shown in the drawings should not be construed as limiting the scope of one or more embodiments.

[0084] In an embodiment, the system obtains historical marketing materials and associated LSRs (operation 402). The LSRs may be LSRs generated by the LSR generator described above. Based on the content of the historical marketing materials and the associated LSRs, the system trains a machine learning model to generate LSRs based on the content of the marketing materials (operation 404). An example of such training is described below.

[0085] As described above, the LSR may be a vector. In an embodiment, the system converts the content of the historical marketing material (e.g., an email subject line) into a vector having the same dimensions as the LSR (operation 406). For example, the system may generate a vector of numbers corresponding to the content of the marketing material. The system may calculate the dot product of the content vector and the corresponding LSR (the dot product may be scaled as described below) (operation 408), and determine whether the scaled dot product satisfies a predetermined threshold condition (operation 410). Specifically, given the purpose of generating an LSR that matches those previously generated, the target value may be the dot product of the LSR with itself, where the scaling layer accounts for the diversity between the content vector and the LSR. Therefore, the threshold condition may require that the scaled dot product is the same as the dot product of the LSR with itself or is within a threshold range of the dot product of the LSR with itself.

[0086] If the threshold condition is not met, the system can adjust the vectorization formula (operation 412). The system can then generate a new vector for the content of the historical marketing material using the adjusted formula (operation 406), calculate the scaled dot product of the new vector and the corresponding LSR (operation 408), and determine whether the new result meets the threshold condition (operation 410). This process can be iterated until the scaled dot product meets the threshold condition, at which point the system can use the trained model to generate an LSR for the unlabeled marketing material (operation 414).

[0087] 4. Example Embodiments

[0088] For the purpose of clarity, detailed examples are described below. The components and / or operations described below should be understood as a specific example that may not be applicable to certain embodiments. Therefore, the components and / or operations described below should not be interpreted as limiting the scope of any one of the claims.

[0089] Specifically, Figure 5A-Figure 5B An example of using contrastive learning to predict marketing results according to one or more embodiments is illustrated. Figure 5A-Figure 5B The operations described are performed by a marketing platform, which for ease of discussion will also be referred to simply as the “platform.”

[0090] exist Figure 5A-Figure 5BIn the example of , the platform is configured to predict the open rates of marketing emails. The platform obtains historical email data 500, including email content 504 (specifically, in this example, email subject lines) and non-content attributes (e.g., metadata associated with the email but generally not visible to recipients). The historical email data 500 is associated with emails that have been sent, and historical open rates 506 for the historical email data are available. That is, for each of the emails represented in the historical email data 500, the open rates 506 include a corresponding open rate based on the actual historical performance of the email.

[0091] Based on the non-content attributes 502, the platform generates corresponding LSRs 508. Based on the LSRs 508 and the corresponding email content 504, the platform trains the content to the LSR model 510. Based on the LSRs 508 and the corresponding open rates, the platform also trains the LSRs to the results model (operation 512). Note that once the LSRs 508 are available, the platform can train the C2LSR model 510 and the LSR2O model 512 in parallel; neither is dependent on the other for training.

[0092] Once trained, the platform can use the C2LSR model 510 and the LSR2O model 512 to predict marketing results for untagged emails, as described herein. Figure 5B As shown in , the platform can obtain a draft email 514 including email content 518 and non-content attributes 516. The platform can apply the C2LSR model 510 to the email content 518 (e.g., subject line) to obtain an LSR 520 corresponding to the draft email 514. The platform can then apply the LSR2O model 512 to the LSR 520 to obtain a predicted open rate 522 for the draft email 514.

[0093] 5. Sample Subject Line

[0094] As discussed above, the system can be configured to generate LSRs of historical marketing materials that are close to each other in the latent space based on their corresponding historical representations, even if their contents are very different. In an illustrative example, the historical email data includes emails with the following subject lines:

[0095] 1. remind ACME manufacturing February forward looking (remind ACME manufacturing February outlook)

[0096] 2.ACME trademark sign coyote the first in region

[0097] 3. December anvil update and season's greetings

[0098] 4. Roadrunner trap weekly update October week 3 (Roadrunner trap weekly update October week 3)

[0099] 5. Roadrunner bait benefit of a single active ingredient

[0100] 6. Plateau and tundra coyote update for region

[0101] 7. Test anvil drop training

[0102] 8.Thank you for your subscription

[0103] 9. Your cartoon club grant entry has been successful

[0104] 10. Urgent notice anvil warehouse closed for holidays (urgent notice anvil warehouse closed for holidays)

[0105] 11. Roadrunner trap data weekly update for 2018 (2018 roadrunner trap data weekly update)

[0106] 12. Roadrunner trap data weekly update for 2019 (2019 roadrunner trap data weekly update)

[0107] In this example: emails 1 through 5 might all have 93% open rates, even though they are semantically very different; emails 6 through 9 might all have 100% open rates, even though they are semantically very different; emails 10 and 11 might both have 60% open rates, even though they are semantically very different and each has some semantic similarity to an email with a much higher open rate; and email 12 might have only a 35% open rate, even though it is semantically very similar to an email with a much higher open rate.

[0108] An approach that relies on semantic analysis may find high similarity between emails 4, 11, and 12, even though they have very different open rate differentials. Similarly, an approach that relies on semantic analysis may find little similarity between emails 1 and 3, because they have no words in common and are not grammatically similar, even though they both have 93% open rates. One or more embodiments described herein predict open rates based on underlying similarities that such approaches typically fail to consider.

[0109] 6. Content Distribution

[0110] In an embodiment, predicting marketing results allows for marketing strategies based on consistent, quantifiable metrics rather than subjective decisions. In response to predicting marketing results for one or more sets of marketing materials, the system can present (e.g., in a user interface) marketing recommendations. For example, the system can recommend publishing a particular set of marketing materials over another set of marketing materials based on the predicted marketing results corresponding to the particular set of marketing materials and the other set of marketing materials. Alternatively or additionally, in response to comparing the predicted marketing results, the system can perform one or more actions without user input, such as publishing the marketing materials with the best predicted results.

[0111] One or more embodiments may identify and / or generate additional content to include in marketing materials. For example, given an email with a high predicted open rate, the system may identify and / or generate additional content (e.g., one or more images, videos, audio, etc.) to include in the body of the email. The system may be configured to identify and / or generate additional content without requiring the user to manually search and filter available content. For example, technology for generating images used in content distribution is described in U.S. patent application Ser. No. 17 / 017,486, filed on September 10, 2020, entitled “Scalable Architecture for Automatic Generation of Content Distribution Images,” the entire contents of which are incorporated herein by reference.

[0112] 7. Computer Networks and Cloud Networks

[0113] In one or more embodiments, a computer network provides connectivity between a set of nodes. The nodes can be local to each other and / or remote from each other. The nodes are connected by a set of links. Examples of links include coaxial cables, unshielded twisted pairs, copper cables, optical fibers, and virtual links.

[0114] A subset of nodes implements a computer network. Examples of such nodes include switches, routers, firewalls, and network address translators (NATs). Another subset of nodes uses a computer network. Such nodes (also referred to as "hosts") can execute client processes and / or server processes. Client processes request computing services (such as the execution of a specific application and / or the storage of a specific amount of data). Server processes respond by, for example, executing the requested service and / or returning corresponding data.

[0115] A computer network may be a physical network, comprising physical nodes connected by physical links. A physical node is any digital device. A physical node may be a hardware device of a specific function, such as a hardware switch, a hardware router, a hardware firewall, or a hardware NAT. Additionally or alternatively, a physical node may be a general-purpose machine configured to execute various virtual machines and / or applications that execute corresponding functions. A physical link is a physical medium that connects two or more physical nodes. Examples of links include coaxial cables, unshielded twisted pair cables, copper cables, and optical fibers.

[0116] A computer network may be an overlay network. An overlay network is a logical network implemented on top of another network (such as a physical network). Each node in the overlay network corresponds to a corresponding node in the underlying network. Therefore, each node in the overlay network is associated with an overlay address (to address the overlay node) and an underlying address (to address the underlying node that implements the overlay node). Overlay nodes may be digital devices and / or software processes (such as virtual machines, application instances, or threads). The links connecting overlay nodes are implemented as tunnels through the underlying network. The overlay nodes at either end of the tunnel treat the underlying multi-hop path between them as a single logical link. Tunnels are performed by encapsulation and decapsulation.

[0117] The client may be local to the computer network and / or remote from the computer network. The client may access the computer network through other computer networks (such as a private network or the Internet). The client may communicate the request to the computer network using a communication protocol such as the Hypertext Transfer Protocol (HTTP). The request is communicated through an interface such as a client interface (e.g., a web browser), a program interface, or an application programming interface (API).

[0118] In one or more embodiments, a computer network provides connectivity between clients and network resources. Network resources include hardware and / or software configured to execute server processes. Examples of network resources include processors, data storage, virtual machines, containers, and / or software applications. Network resources are shared between multiple clients. Clients request computing services from a computer network independently of each other. Network resources are dynamically assigned to requests and / or clients on an on-demand basis. Network resources assigned to each request and / or client can be scaled up or down based on, for example, (a) computing services requested by a specific client, (b) aggregated computing services requested by a specific tenant, and / or (c) aggregated computing services requested by a computer network. Such a computer network may be referred to as a "cloud network".

[0119] In one or more embodiments, a service provider provides a cloud network to one or more end users. Various service models can be implemented through a cloud network, including but not limited to software as a service (SaaS), platform as a service (PaaS), and infrastructure as a service (IaaS). In SaaS, a service provider provides end users with the ability to use the service provider's applications executed on network resources. In PaaS, a service provider provides end users with the ability to deploy customized applications to network resources. Customized applications can be created using programming languages, libraries, services, and tools supported by the service provider. In IaaS, a service provider provides end users with the ability to supply processing, storage, network, and other basic computing resources provided by network resources. Any arbitrary application, including an operating system, can be deployed on network resources.

[0120] Computer networks can implement various deployments, including but not limited to private clouds, public clouds, and / or hybrid clouds. In a private cloud, network resources are supplied for exclusive use by a specific group of one or more entities (the term "entity" used herein refers to a company, organization, individual, or other entity). Network resources can be located locally at the site of a specific entity group and / or away from the site of a specific entity group. In a public cloud, cloud resources are supplied for multiple entities (also referred to as "tenants" or "customers") that are independent of each other. A computer network and its network resources are accessed by clients corresponding to different tenants. Such a computer network can be referred to as a "multi-tenant computer network". Several tenants can use the same specific network resource at different times and / or at the same time. Network resources can be located locally at the site of a tenant and / or away from the site of a tenant. In a hybrid cloud, a computer network includes a private cloud and a public cloud. The interface between a private cloud and a public cloud allows data and application portability. Data stored at a private cloud and data stored at a public cloud can be exchanged through an interface. Applications implemented at a private cloud and applications implemented at a public cloud may have dependencies on each other. Calls from an application at the private cloud to an application at the public cloud (and vice versa) may be performed through the interface.

[0121] In one or more embodiments, the tenants of a multi-tenant computer network are independent of each other. For example, the business or operation of one tenant can be separated from the business or operation of another tenant. Different tenants can require different network requirements for the computer network. Examples of network requirements include processing speed, data storage capacity, security requirements, performance requirements, throughput requirements, latency requirements, elasticity requirements, quality of service (QoS) requirements, tenant isolation and / or consistency. The same computer network may need to implement different network requirements required by different tenants.

[0122] In a multi-tenant computer network, tenant isolation can be implemented to ensure that applications and / or data of different tenants are not shared with each other. Various tenant isolation methods can be used. Each tenant can be associated with a tenant identifier (ID). Each network resource of a multi-tenant computer network can be marked with a tenant ID. A tenant can be allowed to access a specific network resource only if the tenant and the specific network resource are associated with the same tenant ID.

[0123] For example, each application implemented by a computer network can be marked with a tenant ID, and the tenant can be allowed to access a specific application only when the tenant and the specific application are associated with the same tenant ID. Each data structure and / or data set stored by a computer network can be marked with a tenant ID, and the tenant can be allowed to access a specific data structure and / or data set only when the tenant and the specific data structure and / or data set are associated with the same tenant ID. Each database implemented by a computer network can be marked with a tenant ID, and the tenant can be allowed to access the data of a specific database only when the tenant and the specific database are associated with the same tenant ID. Each entry in a database implemented by a multi-tenant computer network can be marked with a tenant ID, and the tenant can be allowed to access a specific entry only when the tenant and the specific entry are associated with the same tenant ID. However, a database can be shared by multiple tenants.

[0124] In one or more embodiments, the subscription list indicates which tenants are authorized to access which network resources. For each network resource, a list of tenant IDs of tenants authorized to access the network resource may be stored. A tenant may be allowed to access a specific network resource only if the tenant ID of the tenant is included in the subscription list corresponding to the specific network resource.

[0125] In one or more embodiments, network resources corresponding to different tenants (such as digital devices, virtual machines, application instances, and threads) are isolated from tenant-specific overlay networks maintained by a multi-tenant computer network. As an example, packets from any source device in a tenant overlay network may only be transmitted to other devices in the same tenant overlay network. Encapsulation tunnels can be used to prohibit any transmission from a source device on a tenant overlay network to devices in other tenant overlay networks. Specifically, a packet received from a source device can be encapsulated in an outer packet. The outer packet is transmitted from a first encapsulation tunnel endpoint (communicating with a source device in the tenant overlay network) to a second encapsulation tunnel endpoint (communicating with a destination device in the tenant overlay network). The second encapsulation tunnel endpoint decapsulates the outer packet to obtain the original packet transmitted by the source device. The original packet is transmitted from the second encapsulation tunnel endpoint to a destination device in the same specific overlay network.

[0126] 8. Microservice Applications

[0127] In one or more embodiments, the technology described herein is implemented in a microservices architecture. Microservices in this context refer to software logic designed to be independently deployable, with endpoints that can be logically coupled to other microservices to build a variety of applications. Applications built using microservices are different from monolithic applications that are designed as a single fixed unit and typically include a single logical executable file. Using microservice applications, different microservices can be independently deployed as separate executable files. Microservices can communicate using Hypertext Transfer Protocol (HTTP) messages and / or via application programming interface (API) endpoints according to other communication protocols. Microservices can be managed and updated separately, written in different languages, and executed independently of other microservices.

[0128] Microservices provide flexibility for managing and building applications. Different applications can be built by connecting different sets of microservices without changing the source code of the microservices. Therefore, microservices act as logical building blocks that can be arranged in a variety of ways to build different applications. Microservices can provide monitoring services that notify the microservice manager (such as If-This-Then-That (IFTTT), Zapier, or Oracle Self-Service Automation (OSSA)) when a trigger event in a set of trigger events exposed to the microservice manager occurs. Microservices exposed to applications can alternatively or additionally provide action services that perform actions in applications based on data received from the microservice manager (which can be controlled and configured by the microservice manager by passing values, connecting actions to other triggers, and / or data passed from other actions in the microservice manager). Microservice triggers and / or actions can be linked together to form recipes for actions that occur in optional different applications that are not otherwise aware of or have no control or dependency on each other. These managed applications can be authenticated or plugged into the microservice manager, for example, using application credentials supplied by a user of the manager, without requiring re-authentication each time the managed application is used alone or in combination with other applications.

[0129] Microservices can be connected via a GUI. For example, microservices can be displayed as logical blocks within a window, frame, or other element of a GUI. Users can drag and drop microservices into a GUI area for building applications. Users can connect the output of one microservice to the input of another microservice using directed arrows or any other GUI elements. The application builder can run validation tests to confirm that the output and input are compatible (e.g., by checking data types, size limits, etc.).

[0130] 8.1. Trigger

[0131] According to one or more embodiments, the above techniques can be encapsulated into a microservice. In other words, a microservice can trigger a notification based on the above techniques (entering the microservice manager for optional use by other plug-in applications, referred to herein as a "target" microservice) and / or can be represented as a GUI block and connected to one or more other microservices. The trigger condition can include an absolute or relative threshold of a value, and / or an absolute or relative threshold of the amount or duration of data to be analyzed, so that whenever a plug-in microservice application detects that a threshold has been exceeded, a trigger to the microservice manager occurs. For example, when a microservice application detects that a value has exceeded a trigger threshold, a user can request a trigger to enter the microservice manager.

[0132] The trigger, when satisfied, may output data for consumption by the target microservice. Alternatively or additionally, when satisfied, the trigger may output a binary value indicating that the trigger condition has been satisfied, and / or may output a field name or other contextual information that the trigger condition has been satisfied. Additionally or alternatively, the target microservice may be connected to one or more other microservices so that the alert is input to the other microservices. The other microservices may perform responsive actions based on the above techniques, including but not limited to deploying additional resources, adjusting system configuration, and / or generating a GUI.

[0133] 8.2. Actions

[0134] The plug-in microservice application can expose an action to the microservice manager. The exposed action can receive as input the identification of data or data objects or the location of data that causes the data to be moved to the data cloud.

[0135] The exposed operation can receive as input a request to increase or decrease an existing alert threshold. The input can identify an existing in-application alert threshold and whether to increase, decrease, or delete the threshold. The input can request the microservice application to create a new in-application alert threshold. In-application alerts can trigger alerts to the user when logging into the application, or can trigger alerts to the user using a default or user-selected alert mechanism available within the microservice application itself rather than through other applications plugged into the microservice manager.

[0136] A microservice application can generate and provide output based on input that identifies, locates, or provides historical data and defines the extent or scope of the requested output. When triggered, the action causes the microservice application to provide, store, or display output, such as a data model or aggregate data describing a data model.

[0137] 9. Hardware Overview

[0138] According to one embodiment, the technology described herein is implemented by one or more special-purpose computing devices. (One or more) special-purpose computing devices can be hard-wired to perform the technology, and / or can include digital electronic devices (such as one or more application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or network processing units (NPUs)) that are permanently programmed to perform the technology, or can include one or more general-purpose hardware processors that are programmed to perform the technology according to program instructions in firmware, memory, other storage devices, or combinations thereof. Such special-purpose computing devices can also combine customized hard-wired logic, ASICs, FPGAs, or NPUs with customized programming to complete the technology. The special-purpose computing device can be a desktop computer system, a portable computer system, a handheld device, a networking device, or any other device that incorporates hard-wiring and / or program logic to implement the technology.

[0139] For example, Figure 6 6 is a block diagram illustrating a computer system 600 upon which one or more embodiments of the present invention may be implemented. Computer system 600 includes a bus 602 or other communication mechanism for communicating information, and a hardware processor 604 coupled to bus 602 for processing information. Hardware processor 604 may be, for example, a general purpose microprocessor.

[0140] The computer system 600 also includes a main memory 606, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus 602 to store information and instructions to be executed by the processor 604. The main memory 606 may also be used to store temporary variables or other intermediate information during execution of instructions to be executed by the processor 604. When such instructions are stored in a non-transitory storage medium accessible to the processor 604, the computer system 600 is presented as a special-purpose machine customized to perform the operations specified in the instructions.

[0141] Computer system 600 also includes a read only memory (ROM) 608 or other static storage device coupled to bus 602 for storing static information and instructions for processor 604. A storage device 610, such as a magnetic or optical disk, is provided and coupled to bus 602 for storing information and instructions.

[0142] The computer system 600 may be coupled to a display 612, such as a cathode ray tube (CRT), via the bus 602 to display information to a computer user. An input device 614, including alphanumeric and other keys, is coupled to the bus 602 to communicate information and command selections to the processor 604. Another type of user input device is a cursor control 616, such as a mouse, trackball, or cursor direction keys, for communicating directional information and command selections to the processor 604 and for controlling cursor movement on the display 612. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), which allows the device to be positioned in a specified plane.

[0143] Computer system 600 may implement the techniques described herein using customized hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic that is combined with computer system 600 to make or program computer system 600 a special purpose machine. In one embodiment, the techniques herein are performed by computer system 600 in response to processor 604 executing one or more sequences of one or more instructions contained in main memory 606. Such instructions may be read into main memory 606 from another storage medium, such as storage device 610. Execution of the sequences of instructions contained in main memory 606 causes processor 604 to perform the process steps described herein. Alternatively, hardwired circuitry may be used in place of or in combination with software instructions.

[0144] The term "storage medium" as used herein refers to any non-temporary medium that stores instructions and / or data that cause a machine to operate in a particular manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical disks or magnetic disks, such as storage device 610. Volatile media include dynamic memory, such as main memory 606. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, tapes or any other magnetic data storage media, read-only compact disks (CD-ROMs), any other optical data storage media, any physical media with hole patterns, RAM, PROM and EPROM, FLASH-EPROM, NVRAM, any other memory chip or tape cassette, content addressable memory (CAM) and ternary content addressable memory (TCAM).

[0145] Storage media are distinct from but can be used in conjunction with transmission media. Transmission media participate in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wire, and optical fiber, including the wires of bus 602. Transmission media can also take the form of acoustic or light waves, such as those generated during radio frequency (RF) and infrared data communications.

[0146] Various forms of media may be involved in carrying one or more sequences of one or more instructions to the processor 604 for execution. For example, the instructions may initially be carried on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions over a telephone line or other communication medium using a modem. A modem local to the computer system 600 may receive the data over a telephone line or other communication medium and convert the data into an infrared signal using an infrared transmitter. An infrared detector may receive the data carried in the infrared signal and appropriate circuitry may place the data on the bus 602. The bus 602 carries the data to the main memory 606, from which the processor 604 retrieves and executes the instructions. The instructions received by the main memory 606 may optionally be stored on the storage device 610 before or after execution by the processor 604.

[0147] Computer system 600 also includes a communication interface 618 coupled to bus 602. Communication interface 618 provides bidirectional data communication coupled to a network link 620 connected to a local network 622. For example, communication interface 618 can be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem that provides a data communication connection to a corresponding type of telephone line. As another example, communication interface 618 can be a local area network (LAN) card that is configured to provide a data communication connection to a compatible LAN. Wireless links can also be implemented. In any such implementation, communication interface 618 sends and receives electrical signals, electromagnetic signals, or optical signals that carry digital data streams representing various types of information.

[0148] The network link 620 typically provides data communication to other data devices through one or more networks. For example, the network link 620 can provide a connection to a host computer 624 or to data equipment operated by an Internet Service Provider (ISP) 626 through a local network 622. The ISP 626 in turn provides data communication services through the global packet data communication network now commonly referred to as the "Internet" 628. Both the local network 622 and the Internet 628 use electrical, electromagnetic, or optical signals that carry digital data streams. The signals through the various networks and the signals on the network link 620 and through the communication interface 618, which carry the digital data to and from the computer system 600, are example forms of transmission media.

[0149] Computer system 600 can send messages and receive data, including program code, through the network(s), network link 620, and communication interface 618. In the Internet example, server 630 can transmit the requested code for an application program through Internet 628, ISP 626, local network 622, and communication interface 618.

[0150] The received code may be executed by processor 604 as it is received, and / or stored in storage device 610 or other non-volatile storage for later execution.

[0151] 10. Miscellaneous; Extensions

[0152] Embodiments are directed to a system having one or more devices including a hardware processor and configured to perform any of the operations described herein and / or recited in any of the following claims.

[0153] In one or more embodiments, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more hardware processors, cause performance of any of the operations described herein and / or recited in any of the claims.

[0154] Any combination of the features and functions described herein may be used according to one or more embodiments. In the foregoing description, embodiments have been described with reference to many specific details that may vary from implementation to implementation. Therefore, the description and drawings are to be considered in an illustrative rather than a restrictive sense. The sole and exclusive indication of the scope of the invention, and what the applicant intends to be the scope of the invention, is the literal and equivalent range of the set of claims claimed for protection in this application, in the specific form in which such claims are claimed, including any subsequent corrections.

Claims

1. One or more non-transitory machine-readable media storing instructions that, when executed by one or more processors, cause execution of operations including: Get multiple historical marketing messages; Obtaining a plurality of historical open rates respectively associated with the plurality of historical marketing messages; Based on the plurality of historical marketing messages, generating a plurality of latent space representations respectively associated with the plurality of historical marketing messages; training a first machine learning model to map content of marketing messages to corresponding latent space representations of the marketing messages based on the plurality of latent space representations and corresponding content of the plurality of historical marketing messages; Based at least on the plurality of latent space representations and the plurality of historical open rates, a second machine learning model is trained to map the latent space representations of marketing messages to predicted open rates of the marketing messages. 2 . The one or more non-transitory machine-readable media of claim 1 , wherein generating the plurality of latent space representations is not based on the corresponding contents of the plurality of historical marketing messages.

3. The one or more non-transitory machine-readable media of claim 1, the operations further comprising: Applying the first machine learning model to a specific marketing message to obtain a latent space representation of the specific marketing message; Applying the second machine learning model to the latent space representation of the specific marketing message to obtain a predicted open rate for the specific marketing message.

4. The one or more non-transitory machine-readable media of claim 1 , wherein training the first machine learning model to map content of a marketing message to a corresponding latent space representation of the marketing message comprises: calculating the square of the magnitude of a particular latent space representation from the plurality of latent space representations; In various eras of machine learning: converting corresponding contents of the plurality of historical marketing messages into vectors having the same dimension as the specific latent space representation; Calculating a dot product of the vector and the particular latent space representation; Wherein during the plurality of epochs, the machine learning targets convergence of the dot product with respect to the square of the magnitude of the particular latent space representation.

5. The one or more non-transitory machine-readable media of claim 1 , wherein generating the plurality of latent space representations comprises: A third machine learning model is applied to the plurality of historical open rates respectively associated with the plurality of historical marketing messages, the third machine learning model being configured to minimize a distance between latent space representations of marketing messages having similar open rates.

6. The one or more non-transitory machine-readable media of claim 5, wherein applying the third machine learning model to the plurality of historical open rates comprises: A plurality of permutations of historical marketing message pairs from the plurality of historical marketing messages are provided as input to the third machine learning model.

7. One or more non-transitory machine-readable media as described in claim 1, wherein training the first machine learning model and training the second machine learning model are performed at least partially in parallel.

8. The one or more non-transitory machine-readable media of claim 1, wherein generating the plurality of latent space representations comprises vectorizing the plurality of historical marketing messages.

9. One or more non-transitory machine-readable media as described in claim 1, wherein the first machine learning model and the second machine learning model comprise corresponding layers in a single neural network.

10. The one or more non-transitory machine-readable media of claim 1, wherein the plurality of historical marketing messages comprises a plurality of historical marketing emails.

11. One or more non-transitory machine-readable media storing instructions that, when executed by one or more processors, cause operations including: Access to multiple historical marketing materials; Acquire a plurality of historical marketing results respectively associated with the plurality of historical marketing materials; Based on the plurality of historical marketing results, generating a plurality of latent space representations respectively associated with the plurality of historical marketing materials; training a first machine learning model to map content of marketing materials to corresponding latent space representations of the marketing materials based on the plurality of latent space representations and corresponding content of the plurality of historical marketing materials; Based at least on the plurality of latent space representations and the plurality of historical marketing results, a second machine learning model is trained to map the latent space representations of the marketing material to predicted marketing results for the marketing material.

12. The one or more non-transitory machine-readable media of claim 11, wherein generating the plurality of latent space representations is not based on the corresponding contents of the plurality of historical marketing materials.

13. The one or more non-transitory machine-readable media of claim 11, the operations further comprising: Applying the first machine learning model to specific marketing materials to obtain a latent space representation of the specific marketing materials; Applying the second machine learning model to the latent space representation of the specific marketing material to obtain predicted marketing results for the specific marketing material.

14. The one or more non-transitory machine-readable media of claim 11, wherein training the first machine learning model to map content of marketing material to a corresponding latent space representation of the marketing material comprises: calculating the square of the magnitude of a particular latent space representation from the plurality of latent space representations; In various eras of machine learning: converting corresponding contents in the plurality of historical marketing materials into vectors having the same dimension as the specific latent space representation; Calculating a dot product of the vector and the particular latent space representation; Wherein during the plurality of epochs, the machine learning targets convergence of the dot product with respect to the square of the magnitude of the particular latent space representation.

15. The one or more non-transitory machine-readable media of claim 11, wherein generating the plurality of latent space representations comprises: A third machine learning model is applied to the plurality of historical marketing results respectively associated with the plurality of historical marketing materials, the third machine learning model being configured to minimize a distance between latent space representations of marketing materials having similar marketing results.

16. The one or more non-transitory machine-readable media of claim 15, wherein applying the third machine learning model to the plurality of historical marketing results comprises: A plurality of permutations of pairs of historical marketing materials from the plurality of historical marketing materials are provided as input to the third machine learning model.

17. One or more non-transitory machine-readable media as described in claim 11, wherein training the first machine learning model and training the second machine learning model are performed at least partially in parallel.

18. The one or more non-transitory machine-readable media of claim 1, wherein generating the plurality of latent space representations comprises vectorizing the plurality of historical marketing materials.

19. One or more non-transitory machine-readable media as described in claim 1, wherein the first machine learning model and the second machine learning model comprise corresponding layers in a single neural network.

20. The one or more non-transitory machine-readable media of claim 11, wherein the predicted marketing results of the marketing material include a predicted open rate for an email.

21. A method comprising the operations of any one of claims 1 to 20.

22. A system comprising a hardware processor and configured to perform the operations of any one of claims 1 to 20.

23. A system comprising means for performing the operations of any one of claims 1 to 20.

Images