Generating and utilizing channel allocation models for resource allocation recommendations

Abstract

An example embodiment of the present disclosure provides an example method that includes obtaining data associated with a media channel. An example method includes inputting data into a machine learning model. An example method includes estimating a structure of a causal graph through a machine learning model. The example method includes applying a second machine learning model to the causal graph to estimate one or more parameters associated with the causal graph. An example method includes determining a resource allocation for a media channel based on a causal graph.

Description

Technical Field

[0001] The present disclosure generally relates to systems and methods for determining direct and indirect relationships of media channels for use in determining future resource allocations. Background Art

[0002] The computing device may perform data processing and run machine learning models. Users may participate in various online and offline activities, which may result in information being exposed to the user. Subsequent activities of the user may be influenced by prior activities and information exposure. Summary of the invention

[0003] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0004] In one example aspect, the present disclosure provides an example system for generating and utilizing a channel allocation model to make resource allocation recommendations, the example system comprising one or more processors and one or more memory devices, the one or more memory devices storing instructions, the instructions executable to cause the one or more processors to perform operations. In some implementations, the one or more memory devices may include one or more temporary or non-temporary computer-readable media, the one or more temporary or non-temporary computer-readable media storing instructions, the instructions executable to cause the one or more processors to perform operations. In the example system, the operation may include obtaining data associated with a plurality of media channels. In the example system, the operation may include inputting the data into a machine learning model. In the example system, the operation may include estimating the structure of a causal graph through a machine learning model. In the example system, the operation may include applying a second machine learning model to the causal graph to estimate one or more parameters associated with the causal graph. In the example system, the operation may include determining resource allocation to a plurality of media channels based on the causal graph.

[0005] In some embodiments of the example system, the causal graph comprises a Bayesian belief network, and the second machine learning model comprises a kernel-based machine learning model.

[0006] In some embodiments of the example system, estimating the structure of the causal graph by the machine learning model includes performing structural learning to generate a causal graph including a plurality of nodes and a plurality of edges.

[0007] In some embodiments of the example system, structural learning includes applying a machine learning model for feature selection. In some embodiments of the example system, structural learning includes performing bootstrap aggregation for updating the causal graph.

[0008] In some embodiments of the example system, estimating the structure of the causal graph includes performing parameter learning to determine (i) a size and (ii) a direction of each respective edge of the plurality of edges.

[0009] In some embodiments of the example system, performing parameter learning includes performing parameter regularization.

[0010] In some embodiments of the example system, the causal graph includes a directed acyclic graph representing conditional probabilities between nodes.

[0011] In some embodiments of the example system, operations may include performing the verification method by transmitting data including instructions that cause a graphical representation of the causal graph to be displayed via a graphical user interface of a device associated with a user. In some embodiments of the example system, operations may include obtaining data indicating user input of accepting the graphical representation of the causal graph.

[0012] In some embodiments of the example system, determining resource allocation to the plurality of media channels based on the causal graph includes determining a relationship between a first node, a second node, and a third node. In some embodiments of the example system, determining resource allocation to the plurality of media channels based on the causal graph includes determining an optimal budget allocation for a first media channel associated with the first node, a second media channel associated with the second node, and a third media channel associated with the third node based on the relationship between the first node, the second node, and the third node.

[0013] In some embodiments of the example system, operations include: determining an optimal budget allocation for a first media channel, a second media channel, and a third media channel includes simulating data indicating a plurality of budget allocation scenarios, wherein each budget allocation scenario indicates a budget amount allocated to each of the first media channel, the second media channel, and the third media channel. In some embodiments of the example system, operations include: determining an optimal budget allocation for the first media channel, the second media channel, and the third media channel includes obtaining user-inputted data indicating a target parameter. In some embodiments of the example system, operations include: determining an optimal budget allocation for the first media channel, the second media channel, and the third media channel includes, for each respective budget allocation scenario in the plurality of budget allocation scenarios, determining a respective parameter of a respective budget allocation scenario in the plurality of budget allocation scenarios. In some embodiments of the example system, operations include: determining an optimal budget allocation for the first media channel, the second media channel, and the third media channel includes comparing each respective parameter to the target parameter. In some embodiments of the example system, operations include: determining an optimal budget allocation for the first media channel, the second media channel, and the third media channel includes determining that a first respective parameter associated with a first budget allocation scenario is within a threshold of the target parameter. In some embodiments of the example system, operations include determining an optimal budget allocation for the first media channel, the second media channel, and the third media channel includes selecting a first budget allocation scenario as the optimal budget allocation scenario in response to determining that the first corresponding parameter is within a threshold of a target parameter.

[0014] In some embodiments of the example system, the target parameter and the corresponding parameter associated with each corresponding budget allocation scenario indicate a percentage reduction in cost per sale.

[0015] In some embodiments of the example system, the cost per sale is determined by calculating media spend for a corresponding media channel divided by the number of unit sales driven by the corresponding media channel.

[0016] In some embodiments of the example system, the target parameters and corresponding parameters associated with each corresponding budget allocation scenario indicate attributions associated with the effectiveness of the media channel and promotion usage on the target action.

[0017] In some embodiments of the example system, attribution for a respective media channel is determined by calculating the number of unit sales driven by the respective media channel divided by the total sales.

[0018] In some embodiments of the example system, the target parameter and the corresponding parameter associated with each corresponding budget allocation scenario is an overall total media spend budget.

[0019] In some embodiments of the example system, the operations include generating data indicative of a graphical representation of a causal graph. In some embodiments of the example system, the operations include transmitting data including instructions that, when executed, cause the graphical representation of the causal graph to be rendered via a graphical user interface of a user device.

[0020] In some embodiments of the example system, operations include generating data indicating a graphical representation of the determined resource allocation to the plurality of media channels. In some embodiments of the example system, operations include transmitting data including instructions that, when executed, cause the graphical representation of the resource allocation to the plurality of media channels to be rendered via a graphical user interface of a user device.

[0021] In some embodiments of the example system, data associated with a plurality of media channels includes outcome data, predictor variables, and control variables.

[0022] In an example aspect, the present disclosure provides an example computer-implemented method. The example method includes obtaining data associated with a plurality of media channels. The example method includes inputting the data into a machine learning model. The example method includes estimating a structure of a causal graph by the machine learning model. The example method includes applying a second machine learning model to the causal graph to estimate one or more parameters associated with the causal graph. The example method includes determining a resource allocation to the plurality of media channels based on the causal graph.

[0023] In some embodiments of the example method, the causal graph includes a Bayesian belief network and the second machine learning model includes a kernel-based machine learning model.

[0024] In some embodiments of the example methods, estimating the structure of the causal graph by the machine learning model includes performing structural learning to generate a causal graph including a plurality of nodes and a plurality of edges.

[0025] In some embodiments of the example methods, structure learning includes applying a machine learning model for feature selection. In some embodiments of the example methods, structure learning includes performing self-service aggregation for updating the causal graph.

[0026] In some embodiments of the example methods, estimating the structure of the causal graph includes performing parameter learning to determine (i) a size and (ii) a direction of each respective edge of the plurality of edges.

[0027] In some embodiments of the example methods, performing parameter learning includes performing parameter regularization.

[0028] In some embodiments of the example methods, the causal graph comprises a directed acyclic graph representing conditional probabilities between nodes.

[0029] In some embodiments of the example method, the method includes performing the verification method by transmitting data including instructions that cause a graphical representation of the causal graph to be displayed via a graphical user interface of a device associated with a user. In some embodiments of the example method, the method includes obtaining data indicating user input of accepting the graphical representation of the causal graph.

[0030] In some embodiments of the example methods, determining resource allocation to the plurality of media channels based on the causal graph includes determining a relationship between a first node, a second node, and a third node. In some embodiments of the example methods, determining resource allocation to the plurality of media channels based on the causal graph includes determining an optimal budget allocation for a first media channel associated with the first node, a second media channel associated with the second node, and a third media channel associated with the third node based on the relationship between the first node, the second node, and the third node.

[0031] In some embodiments of the example method, the method includes: determining an optimal budget allocation for a first media channel, a second media channel, and a third media channel includes simulating data indicating a plurality of budget allocation scenarios, wherein each budget allocation scenario indicates a budget amount allocated to each of the first media channel, the second media channel, and the third media channel. In some embodiments of the example method, the method includes: determining an optimal budget allocation for the first media channel, the second media channel, and the third media channel includes obtaining user-inputted data indicating a target parameter. In some embodiments of the example method, the method includes: determining an optimal budget allocation for the first media channel, the second media channel, and the third media channel includes, for each respective budget allocation scenario in the plurality of budget allocation scenarios, determining a respective parameter of a respective budget allocation scenario in the plurality of budget allocation scenarios. In some embodiments of the example method, the method includes: determining an optimal budget allocation for the first media channel, the second media channel, and the third media channel includes comparing each respective parameter to the target parameter. In some embodiments of the example method, the method includes: determining an optimal budget allocation for the first media channel, the second media channel, and the third media channel includes determining that a first respective parameter associated with a first budget allocation scenario is within a threshold of the target parameter. In some embodiments of the example method, the method includes determining an optimal budget allocation for the first media channel, the second media channel, and the third media channel includes selecting a first budget allocation scenario as the optimal budget allocation scenario in response to determining that the first corresponding parameter is within a threshold of the target parameter.

[0032] In some embodiments of the example method, the target parameter and the corresponding parameter associated with each corresponding budget allocation scenario indicate a percentage reduction in cost per sale.

[0033] In some embodiments of the example method, the cost per sale is determined by calculating the media spend of the respective media channel divided by the number of unit sales driven by the respective media channel.

[0034] In some embodiments of the example method, the target parameter and the corresponding parameter associated with each corresponding budget allocation scenario indicate attribution associated with the effectiveness of the media channel and promotion usage on the target action.

[0035] In some embodiments of the example method, attribution for a respective media channel is determined by calculating the number of unit sales driven by the respective media channel divided by the total sales.

[0036] In some embodiments of the example method, the target parameter and the corresponding parameter associated with each corresponding budget allocation scenario is an overall total media spend budget.

[0037] In some embodiments of the example method, the method includes generating data indicative of a graphical representation of a causal graph. In some embodiments of the example method, the method includes transmitting data including instructions that, when executed, cause the graphical representation of the causal graph to be rendered via a graphical user interface of a user device.

[0038] In some embodiments of the example method, the method includes generating data indicating a graphical representation of the determined resource allocation to the plurality of media channels. In some embodiments of the example method, the method includes transmitting data including instructions that, when executed, cause the graphical representation of the resource allocation to the plurality of media channels to be rendered via a graphical user interface of a user device.

[0039] In some embodiments of the example methods, data associated with a plurality of media channels includes outcome data, predictor variables, and control variables.

[0040] In an example aspect, the present disclosure provides an example transient or non-transitory computer-readable medium, which is embodied in a computer-readable storage device and stores instructions that, when executed by a processor, cause the processor to perform operations. In the example transient or non-transitory computer-readable medium, the operations include obtaining data associated with multiple media channels. In the example transient or non-transitory computer-readable medium, the operations include inputting the data into a machine learning model. In the example transient or non-transitory computer-readable medium, the operations include estimating the structure of a causal graph through a machine learning model. In the example transient or non-transitory computer-readable medium, the operations include applying a second machine learning model to the causal graph to estimate one or more parameters associated with the causal graph. In the example transient or non-transitory computer-readable medium, the operations include determining resource allocations to multiple media channels based on the causal graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] A detailed discussion of the embodiments for those skilled in the art is set forth in this specification with reference to the accompanying drawings, in which:

[0042] Figure 1 An example flow chart depicting an example process for determining direct and indirect relationships of media channels for use in determining future resource allocations according to an example embodiment of the present disclosure is depicted.

[0043] Figure 2 An example flow chart depicting an example process for determining direct and indirect relationships of media channels for use in determining future resource allocations according to an example embodiment of the present disclosure is depicted.

[0044] Figure 3 Depicted is a block diagram of an example causal graph according to an example embodiment of the present disclosure.

[0045] FIG. 4A to FIG. 4B Depicted is a flow chart of an example method for performing determination of direct and indirect relationships of media channels for use in determining future resource allocations in accordance with an example embodiment of the present disclosure.

[0046] Figure 5 An example user interface is depicted according to an example embodiment of the present disclosure.

[0047] Figure 6 Depicted is a block diagram of an example computing system that performs determination of direct and indirect relationships of media channels for use in determining future resource allocations according to an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0048] In general, the present disclosure relates to a channel allocation model for determining direct and indirect relationships of media channels for determining future resource allocations. The channel allocation model includes a machine learning method for fitting multidimensional data for multiple media channel problems. The modeling method can include a structural learning step for identifying the underlying structure and interrelationships of variables, and a parameter learning step for updating the quantitative portion of a causal graph (e.g., a Bayesian network), and can quantify the magnitude of the effect of each variable on the outcome. The model can be used to predict outcomes based on allocations to various media channels, and can be used to optimize resource allocations to corresponding channels.

[0049] Data from various media channels is complex and available in different forms. Analyzing this data to determine the efficiency of resource allocation is important for planning future resource allocation. The present disclosure utilizes mathematically formalized probabilistic graphical models with the assistance of machine learning and algorithm optimization to identify direct and indirect relationships between various media channels. This can allow automated causal reasoning and prediction.

[0050] Bayesian methods can be combined with nonparametric methods to achieve compact computation by decomposing the joint distribution into local conditional distributions of each variable given its parents (e.g., Markov blankets). This provides increased computational efficiency compared to commonly used Markov Chain Monte Carlo (MCMC) sampling methods, which are usually computationally expensive.

[0051] The present method also includes determining the structure of the network (e.g., which nodes are connected to which nodes and which connections are stronger) rather than strictly determining the prediction of input-output bindings as performed by Hidden Markov Models (MM), Naive Bayes, and Neural Networks (NN). This allows the generation of presentations for display that include nonlinearities, synergies, and other interactions between media channels (or other media variables).

[0052] Traditionally, previous channel allocation methods have viewed media channels (and their corresponding performance) in isolation. However, the impact on the target actions performed (e.g., conversion rate, sales volume, etc.) caused by user exposure to content items via various media forms based on resource allocation is relevant. In addition, the impact of user exposure in various media channels on conversion rate caused by resource allocation is usually not a bell curve normal distribution, but usually includes an increase in resource allocation (e.g., computing resources, network resources, bandwidth resources, expenditures) for various events (e.g., seasonality, PR events). Although there are current methods for transforming data and allowing missing value filling methods, these methods will result in changes in data structures and data sets, making the data structures and data sets more difficult to interpret or use in various applications. In addition, due to the complexity of existing models, a technical problem associated with current methods is the overfitting problem.

[0053] The present disclosure provides a technical solution to the technical problem by constructing a set of machine-learned modeling capabilities that are robust to data assumptions and easily scalable. The output of these modeling capabilities can be used to more efficiently allocate resources to improve desired target results (e.g., conversion rates, increases in data indicating user selections or interactions with one or more interactive digital components, more efficient allocation of limited user interface display space, etc.). The present disclosure focuses the solution on the following key indicators: the effectiveness of content item display in increasing target actions (e.g., attribution, effectiveness) and the efficiency of specific media channels in increasing target actions (e.g., resource allocation efficiency). The method allows for the acceleration of the modeling process of media channel performance using machine learning, and allows for near real-time learning of the impact of resource allocation on the target actions performed.

[0054] The example aspects of the present disclosure can provide a variety of technical effects and benefits. For example, the example aspects relate to improving user access to networked resources related to the user or user task, or otherwise promoting the user's intention when operating a computing system. For example, by determining the appropriate resource allocation for presenting content items to the user (e.g., via various electronic surfaces, various media channels, etc.) to provide selectable input elements to be filled via the user interface of the user computing device, the computing device can provide a more efficient user-machine interface to complete the task and perform actions that otherwise may require more complex or indirect input sequences. For example, the user input element filled in the user interface can be directly linked to the network resources related to the project desired by the user, rather than needing to access the first network resource that provides the option index, select the supplier's options, scroll through the various projects from the supplier and finally select the desired project. By learning to predict the input elements filled in the user interface based on the probability associated with achieving the user's task or goal, a more direct and efficient user interface can be provided according to the system and method of the present disclosure for completing the specific tasks (e.g., target actions, conversion rates) that the user is using the computing device to perform. For example, in this way, the computing resources (e.g., computing cycles, memory resources, electrical resources, etc.) used to render multiple different interfaces to achieve a given task can be reduced. In addition, the user-machine interface can be improved by providing a more efficient and direct user interface flow for completing a given task. Therefore, the present disclosure can provide a more efficient allocation of computing resources to prevent redundant data transmission or communicating irrelevant input element items or content items to a computing device associated with a user.

[0055] Additionally, the present disclosure may provide increased user privacy based on determining a global performance value for a particular media channel rather than determining performance at an individual user level. Thus, the solution may provide analysis of aggregated data rather than data associated with a particular user or user device.

[0056] The systems and methods described herein can be executed at any time interval. For example, the systems and methods can be executed daily, weekly, monthly, yearly, etc. Benefits of the present disclosure include the ability to generate a causal graph (e.g., in the form of a directed acyclic graph (DAG)) representing the relationships between various media channels, providing insights faster than traditional methods even with fine-grained (e.g., daily) data. This allows for the generation of forecasts and recommendations for marketing campaigns with less market history and smaller spend amounts in a computationally inexpensive manner. The improvements associated with the systems and methods discussed herein can be further understood with reference to the accompanying drawings.

[0057] Figure 1An example flow 100 associated with an implementation of the channel allocation model described herein is depicted. The data flow 100 can be performed by a computing system (e.g., computing system 600). The data flow 100 can be performed in a computing system (e.g., computing system 600) by a single device or across multiple devices. The data flow 100 can include a data preparation phase 105. The data flow 100 can include a causal graph generation phase 110, an attribution and return on advertising spend (ROAS) calculation phase 125, and a resource allocation determination phase 140. Each phase can include sub-phases. For example, the causal graph generation phase 110 can include a structure learning phase 115 and a parameter learning phase 120.

[0058] The data preparation stage 105 may include obtaining data from multiple sources. For example, the source may include multiple media channels, aggregated data, etc. The prepared data may include unit sales (or activation rates), daily active users (DAU) (or other application usage data), media impressions, expenditures by media channels (e.g., weekly expenditures by designated market area (DMA)), non-media data (e.g., price discounts, retail equipment, store promotions, competitive expenditures), retail marketing data, promotional data, data associated with historical resource allocation (e.g., media expenditures for one or more months), daily data at a geographic level (e.g., zip code, city, or DMA level) (e.g., resource allocation, sales, conversion rates for various media channels), weekly data at a geographic level (e.g., zip code, city, or DMA level) (e.g., resource allocation, sales, conversion rates for various media channels), or different media weights across different time periods (e.g., when resources are not allocated to media channels to correspond to expected benchmarks). Media channels may include, for example, digital media, search, television, and sales, etc. The system may input the obtained data into one or more machine learning models and obtain output.

[0059] In some implementations, the obtained data may be in a variety of forms. The data preparation stage 105 may include a normalization method to translate the data into a form that can be input into one or more machine-learned models or statistical models as described herein.

[0060] In the causal graph generation phase 110, the system can obtain the prepared data as input. For example, the first machine learning model can be associated with the structure learning phase 115, and the second machine learning model can be associated with the parameter learning phase 120. In some implementations, the structure learning phase 115 can include obtaining an output including a plurality of features that affect the result (e.g., conversion rate). In some implementations, the features can include media channels. The causal graph can include a plurality of nodes and a plurality of edges. The causal graph can be in the form of a Bayesian belief network.

[0061] In some implementations, the parameter learning phase 120 can include obtaining an output including edges (e.g., with direction and magnitude) representing the relationships between the multiple features determined during the structure learning phase 115. The output can indicate direct and indirect relationships between marketing, non-marketing, and business performance variables to allow for automated causal reasoning and prediction in other stages of the data flow 100. Figure 2 The structure learning phase 115 and the parameter learning phase 120 are described in further detail (eg, the structure learning phase 215 and the parameter learning phase 220 ).

[0062] The attribution and ROAS calculation stage 125 may include a prediction stage 130 and an accuracy stage 135. The attribution and ROAS calculation stage 125 may include performing a prediction of how various resource allocations to the respective media channels will affect performance or another relevant metric. For example, the metric may include a key performance indicator (KPI), cost per unit sold, attribution of total sales to the respective media channels, etc. The KPI may include sales volume, offline (e.g., store) activation rate, app usage rate, or foot traffic to a retail location. Figure 2 The attribution and ROAS calculation stage 125 is described in further detail (eg, the attribution and ROAS calculation stage 225 ).

[0063] Additionally or alternatively, the accuracy of the predicted Bayesian belief network can be analyzed to help update a causal graph representing the relationships between various features. For example, the causal graph can be in the form of a directed acyclic graph (DAG) representing the relationships between various features. The DAG can include nodes representing various variables (e.g., media channels, features, etc.) and edges representing the relationships between various variables (e.g., media channels, features, etc.). Edges can have a direction and a size. The direction of an edge can indicate the direction of a causal or correlation relationship between two nodes. The size of an edge can indicate the strength of a causal or correlation relationship between two nodes.

[0064] For example, a causal graph (or graphical model) can represent a set of random variables and their conditional dependencies via a DAG. The graphical model structure combined with associated parameters can define a joint probability distribution over a set of nodes (e.g., variables, features, channels). Machine-learned conditional dependencies can be determined, which can help evaluate both qualitative and quantitative measures associated with the input data.

[0065] Based on the metrics determined in the attribution and ROAS calculation stage 125, the data flow 100 may include a resource allocation determination stage 140. The system may employ a variety of methods to determine how to allocate future resources. For example, resources may include computing resources, financial resources, etc.

[0066] Figure 2 Describes the Figure 1 An example data flow 200 of finer-grained detail. Data flow 200 may be performed by a computing system (e.g., computing system 600). Data flow 200 may be performed in a computing system (e.g., computing system 600) by a single device or across multiple devices. Data flow 200 may include a data preparation stage 205, a causal graph generation stage 210, an attribution and ROAS calculation stage 225, and a resource allocation determination stage 240. Data preparation stage 205 and resource allocation determination stage 240 may generally correspond to data preparation stage 105 and resource allocation determination stage 140, respectively.

[0067] The causal graph generation phase 210 may include a structure learning phase 215 and a parameter learning phase 220. The causal graph generation phase 210 may include using one or more machine learning or open source platforms that provide fast processing of data and models.

[0068] In some implementations, the causal graph generation phase 210 may be a Bayesian belief network generation phase. The structure learning phase 215 may include feature selection 210A and belief updating 210B. In the structure learning phase 215, the underlying structure and relationships between variables may be determined. The relationships may include causal relationships or correlation relationships.

[0069] Feature selection 210A may include determining one or more features (e.g., variables, channels) to be included in the structure. The system may determine the links between the corresponding variables. In some implementations, a Tabu Search learning algorithm may be used to determine the relationships between the variables. For example, the variables may indicate media channels. In some implementations, the variables may include paid media channels, competitors, external factors, search channels, audiences, brand effects, and sales. The data stream may generate a structure including the variables and their corresponding relationships. Figure 3 A depiction of an example structure is described. In some implementations, variables can have direct and indirect effects on other variables. For example, a paid media channel can have a direct effect on sales, and have an additional indirect effect on sales. For example, a paid media channel can have a direct effect on a search channel, and the search channel can have a direct effect on sales and a brand effect channel. The brand effect channel can have a direct effect on sales. Therefore, a paid media channel can have an indirect effect on sales via a direct effect on the search channel, external factors, and competitors.

[0070] Although these connections can be described in words by analyzing the prepared data and / or the relationships can be represented by formulas representing conditional probabilities, the present disclosure provides for generating a causal graph that can provide a visualization of the connections between variables. This can take complex data and convert it into a form that can provide easy-to-understand visual materials for users. In addition, resource allocation determination can be performed using conditional probabilities and complex data analysis performed by various models herein (e.g., in the resource allocation determination stage 240). And this can provide technical benefits, including optimization of the display of data via user devices with limited display space.

[0071] The belief update 210B may include updating the relationship between the corresponding variables based on the additional processing. For example, the belief update 210B may include using one or more machine learning techniques to identify variables of interest (e.g., important variables). For example, a machine learning model may analyze data including a large number of variables (tens, hundreds, etc.). A causal graph including all variables may produce an unreadable graph that does not provide valuable insights. The present disclosure provides systems and methods for determining the most relevant variables to be included in a causal graph to provide a display to a user and to be utilized in the resource allocation determination phase 240.

[0072] Machine learning techniques may include, for example, random forests. Random forests may be meta-estimators that aggregate into a single ensemble model through model voting or averaging. In some implementations, random forests may outperform the output of individual decision trees. For example, decision trees may be computationally expensive to train and may be at risk of overfitting and finding a local optimum.

[0073] In some implementations, the final causal graph (e.g., the final network represented as a DAG) can be an ensemble network (e.g., model averaging) achieved through self-service aggregation or bagging of hundreds of self-service networks. By generating an ensemble network, a robust network with high stability can be learned.

[0074] In additional or alternative implementations, variable importance can be calculated by random forests for feature selection and to assist in favoring or penalizing the inclusion of specific relationships in a causal graph (e.g., a network represented as a DAG). In some implementations, the system can obtain a final network by performing a bootstrap aggregation of more than 100 networks or more to aid in statistical robustness. The bootstrap aggregation can additionally aid in confidence in the learned network so that important features can be robust to perturbations of observations.

[0075] The parameter learning phase 220 may include custom fitting 220A and parameter regularization 220B. The parameter learning phase 220 may include updating the quantitative portion of the Bayesian network and quantifying the magnitude of the effect of each variable on the outcome.

[0076] In some implementations, the variables may be continuous variables. For continuous variables, the system may utilize a score-based hill climbing algorithm to determine the maximum score improvement (e.g., goodness of fit) at each step. In some implementations, the mixed media marketing data may be discrete. A discrete Bayesian network for categorical data may be learned by a model designed to process discrete data.

[0077] In some implementations, the media marketing data may include continuous variables. A Gaussian Bayesian network may be learned by a model designed to process continuous data. Maximum likelihood estimates and Bayesian posterior estimates may be available. For example, the computing system may determine the posterior probability to reduce the dimensionality. In some implementations, the computing system may consider only local distributions in the Bayesian belief network when determining the posterior probability.

[0078] Custom fit 220A may include estimating robust coefficients through custom fit. Custom fit may include, for example, penalized regression, elastic net regression, etc.

[0079] Parameter regularization 220B can include more robust parameter estimation without requiring a user to manually search for different interaction orders between variables. Robust parameter estimation can also be performed without relying on linear or additivity assumptions. For example, parameter regularization 220B can be performed using a kernel-based regularized least squares (KRLS) model. This can provide technical benefits. For example, KRLS allows minimization of overfitting, reduction of the impact of bad leverage points (e.g., outliers far from the regression line), and the ability to perform regularization on small sample sizes.

[0080] Causal graphs (e.g., probabilistic graphs) can be learned in the form of DAGs as described herein. DAGs can be determined and driven empirically by data (and analysis thereof) without being affected by human bias in data analysis. The structure of a DAG can correspond to a set of conditional dependency statements for variables in a model. In some implementations, variables are not connected (e.g., there is no edge between two nodes). This lack of connectivity can indicate a conditional independence statement. Conditional independence can indicate that a change in one variable does not result in a change in an independent variable.

[0081] exist Figure 3 An example causal graph (e.g., learned network, DAG, Bayesian belief network) is depicted in . The various connections indicate relationships between variables that have direct effects on other variables. For example, Figure 3An example graphical representation 300 (e.g., a visual representation) is depicted, which can be provided for display via a user interface of a user device. For example, graphical representation 300 can include multiple nodes representing multiple variables. The variables can include variable 305, variable 310, variable 315, variable 320, variable 325, variable 330, and variable 335. Multiple nodes can be connected via multiple edges. Multiple edges can be divided into groups (e.g., Markov groups) that represent which variables (e.g., nodes) have a direct impact on other variables and which variables operate independently of each other. For example, edge groups can include edge 340, edge 345, edge 350, and edge 355.

[0082] For example, edge 340 can be associated with a variable that has a direct relationship with variable 315. For example, variable 315 can represent a search channel, variable 305 can represent a paid media channel, and variable 320 can represent a variable for an external factor. Edge 340 depicts a direct relationship between a paid media channel (e.g., variable 305) and a search channel (e.g., variable 315) and a direct relationship between an external factor variable (e.g., variable 320) and a search channel (e.g., variable 315). For example, edge 340 depicts a Markov blanket indicating variables that have a direct relationship with variable 315.

[0083] Edge 345 may be associated with a variable that has a direct relationship with variable 330. For example, variable 330 may represent a brand effect variable, and variable 325 may represent an audience variable. Edge 345 depicts a direct relationship between an external factor variable (e.g., variable 320) and a brand effect variable (e.g., variable 330), and a direct relationship between an audience variable (e.g., variable 325) and a brand effect variable (e.g., variable 330). For example, edge 345 depicts a Markov blanket indicating variables that have a direct relationship with variable 330.

[0084] Edge 350 may be associated with variables that have a direct relationship with variable 335. For example, variable 335 may represent a conversion rate variable (e.g., sales, clicks, etc.). For example, variable 310 may represent a competitor's variable. Edge 350 may represent a direct relationship between a paid media channel (e.g., variable 305) and a conversion rate variable (e.g., variable 335), a direct relationship between an external factor variable (e.g., variable 320) and a conversion rate variable (e.g., variable 335), a direct relationship between a competitor variable (e.g., variable 310) and a sales volume variable (e.g., variable 335), a direct relationship between an audience variable (e.g., variable 325) and a sales volume variable (e.g., variable 335), and a direct relationship between a brand effect variable (e.g., variable 330) and a sales volume variable (e.g., variable 335). For example, edge 350 depicts a Markov blanket indicating variables that have a direct relationship with variable 335.

[0085] In some implementations, a causal graph (e.g., a DAG) can represent a decomposition of a distribution into conditional probabilities of variables that depend on parent variables. The structure of the causal graph (e.g., a DAG) can be specified in a format that reflects the decomposition into local probabilities. The structure of the causal graph (e.g., a DAG) can be defined by a joint probability distribution, and the order of the variables can be irrelevant.

[0086] While existing machine learning tools are often black-box, causal graphs (e.g., DAGs, belief networks) can allow users to see visual representations of how covariates relate to each other. Therefore, belief networks can explore what the underlying structure is (e.g., which variables are connected to which variables and which connections are stronger). The conditional dependencies between various nodes can be represented by the following formula:

[0087] Formula 1:

[0088] Formula 2:

[0089] Formula 3:

[0090] Formula 1 indicates that the probability of A given B is equal to the probability of B given A multiplied by (the probability of A divided by the probability of B). A, B, and C may represent various nodes (eg, variables, media channels, etc.).

[0091] Formula 2 represents the probability of A given B and C. The probability of A given B and C is equal to the probability of A and C multiplied by (the probability of B given A and C divided by the probability of A given C).

[0092] Formula 3 represents the nodes x1, ... and x nThe probability of nodes x1, ... and x n The probability is equal to that for all nodes i=1 to i=n at a given node x i The parent of (y i ) in the case of node x i These formulas can be used to determine the conditional dependencies between various nodes and their associated variables.

[0093] Back to Figure 3 , example probabilities of nodes can be represented using Formulas 1 to 3. Thus, the probability of variable 315 given variable 305 and variable 320 can be equal to the probability of variables 315 and 305 multiplied by (the probability of variable 320 given variable 315 and variable 305 divided by the probability of variable 315 given variable 320).

[0094] Although edges represent relationships between variables, it is important to note that relationships do not always represent causality. Relationships can often be concise (e.g., having limited indirect effects) and can allow for easier identification of important relationships (e.g., based on size). In some cases, the edges of a causal graph (e.g., a DAG) can represent causal semantics (e.g., direct causal influence). In some implementations, a PC algorithm can be implemented to test persistent associations between nodes in a causal graph (e.g., a DAG) in the absence of data from a causal experiment. This can be conditional on all subsets of other variables. In some implementations, a fast causal inference PC algorithm can be used to additionally account for the effects of latent variables. Additionally or alternatively, the method may include running a matched market test to confirm the validity and direction of the inferred causal effects of various media channels (and other variables) on conversion rates (e.g., sales).

[0095] In some embodiments, a causal graph (e.g., DAG) may be provided for display via a user device. In some implementations, a computing system may obtain data indicating that a user accepts the causal graph (e.g., DAG). For example, a user may review the graph and confirm that the arrows are consistent with a business concept. For example, a particular direction of a link may be selected based on market intuition in media marketing practices.

[0096] In some implementations, review of a causal graph (e.g., DAG) can be automatically performed by the system. For example, random forest results can be used to ensure that certain variables are linked (e.g., whitelisted) or prevent the connection of certain variables (e.g., blacklisted). For example, some variables can be automatically linked based on the random forest results of the corresponding variables.

[0097] The causal graph (e.g., DAG) can be filtered to increase the interpretability and simplicity of the graph. For example, the results of the structure and parameter learning phases may be noisy due to data limitations.

[0098] The statistical significance of the quartiles and average marginal effects of the marginal effects can be determined. The marginal effects can indicate heterogeneity or interactive effects. For example, the point-by-point marginal effects of each variable at each data point can be examined to determine where an adjustment in resource allocation (e.g., an adjustment in media spending) has a positive effect on one or more measured effects (e.g., has a positive effect on driving sales), and what levels of resource allocation have a negative coefficient (e.g., no positive effect) due to under-allocating resources or wasting too many impressions due to over-spending.

[0099] Back to Figure 2 , the causal graph (e.g., DAG) generated in the causal graph generation phase 210 can be used by the system to perform attribution and ROAS calculations during the attribution and ROAS calculation phase 225. The attribution and ROAS calculation phase 225 can include a prediction phase 230 and an accuracy phase 235. The prediction phase 230 can include response curve generation 232. The accuracy phase 235 can include mean absolute error (MAE), root mean square error (RSME), residual graphs 235A, confidence intervals 235B, and cross validation 235C.

[0100] For example, response curve generation 232 may include generating a plurality of curves including the predicted performance (e.g., Figure 5 ). In some implementations, the response curve may include the simulated performance (e.g., simulated performance 510). For example, the response curve may be a graphical depiction of the predicted performance, including media expenditures on the x-axis and cost per sale on the y-axis. Simulated portfolio performance curves may be generated for portfolios of various sizes. For example, curves for 50 simulated portfolios and curves for 200 simulated portfolios may be generated and displayed. In addition, the simulated portfolios may be aggregated and smoothed to generate a smoothed aggregate curve to be displayed. The system may perform one or more calculations to determine a "media-driven sales" curve. The media-driven sales curve may display data indicating low returns, medium returns, and high returns for sales driven by media. The data driving this visualization may be used to optimize target parameters. Target parameters may include total media expenditures across available media channels, cost per sale, and the like.

[0101] Utilizing machine learning models can allow for measuring the performance of all campaigns and channels across product areas, how different media channels and campaigns interact with each other, and how media channels and campaigns work together. This allows for improvements over existing approaches, which typically consider media channels (and their associated performance) in isolation. Thus, using the systems and methods described herein, learning from the underlying data structures and the interrelationships between various channels can be used to determine resource allocations. For example, determining resource allocations can include optimizing a budget for future media channel expenditures. In some implementations, the method can include generating hundreds of planning scenarios (e.g., such as Figure 3 Through simulation, the method can determine low-return, medium-return, and high-return points to achieve better future resource allocation to meet specific optimization goals. For example, the optimization goal can be a goal associated with a target parameter, a specific feature, or different budget levels.

[0102] The prediction stage 230 and accuracy stage 235 of attribution and ROAS may be used in a resource allocation determination stage 240 to determine how to allocate future resources. Resource allocation may include, for example, the distribution or allocation of computing resources or financial resources.

[0103] A causal graph (e.g., a DAG) can be used for a variety of purposes. In some implementations, a graph can be provided for display via a user device. The display can be used to help the user understand the impact and interrelationships of various variables (e.g., media channels, features). This relationship can indicate multiple conditional probabilities of various variables (e.g., media channels, graph nodes, features). In some implementations, the system can extract information about the conditional probabilities of various features. In some implementations, a graph can be provided for display to the user via a display on a user device. In some implementations, the user can provide input via the user device. For example, a computing system can obtain data indicating user input. For example, the user input can indicate acceptance of the graph. In some implementations, the user input can indicate rejection of part (or all) of the graph.

[0104] FIG. 4A to FIG. 4B A flow chart depicts an example method 400 for determining direct and indirect relationships of media channels for use in determining future resource allocations. FIG. 4A to FIG. 4B The steps are depicted in a specific order for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the order or arrangement specifically described. The various steps of the method 400 may be omitted, rearranged, combined or adapted in various ways without departing from the scope of the present disclosure.

[0105] At (402), method 400 may include obtaining data associated with a plurality of media channels. For example, a computing system (e.g., computing system 600) may obtain data associated with a plurality of media channels. As described herein, data associated with a plurality of media channels includes outcome data, predictor variables, and control variables. For example, the data may include unit sales (or activation rates), daily active users (DAU) (or other application usage data), media impressions, expenditures by media channel (e.g., weekly expenditures by designated market area (DMA)), non-media data (e.g., price discounts, retail equipment, store promotions, competitive expenditures), retail marketing data, promotional data, data associated with historical resource allocation (e.g., media expenditures for one or more months), daily data at a geographic level (e.g., zip code, city, or DMA level) (e.g., resource allocations, sales, conversion rates for various media channels), weekly data at a geographic level (e.g., zip code, city, or DMA level) (e.g., resource allocations, sales, conversion rates for various media channels), or different media weights across different time periods (e.g., when resources are not allocated to media channels to correspond to an expected benchmark). Media channels may include, for example, digital media, search, television, and sales.

[0106] At (404), method 400 may include inputting data into a machine learning model. For example, a computing system (e.g., computing system 600) may input data into a machine learning model. As described herein, the machine learning model may be a single machine learning model or an integration of machine learning models. In some implementations, the integration of machine learning models may include models of the same type. In some implementations, the integration of machine learning models may include a diverse arrangement of model types.

[0107] At (406), method 400 may include estimating the structure of a causal graph. For example, a computing system (e.g., computing system 600) may estimate the structure of the causal graph. As described herein, the causal graph may be estimated by a machine learning model. For example, the causal graph may include a Bayesian belief network. In some implementations, the causal graph may include a directed acyclic graph.

[0108] In some implementations, estimating the structure of the causal graph may include performing structural learning to generate a causal graph including a plurality of nodes and edges. The structural learning may include applying a machine learning model for feature selection. The structural learning may include performing self-help aggregation for updating the causal graph.

[0109] In some embodiments, estimating the structure of the causal graph includes performing parameter learning to determine (i) the size and (ii) the direction of each respective edge in the plurality of edges. Performing parameter learning may include performing parameter regularization. The causal graph may include a directed acyclic graph representing conditional probabilities between nodes.

[0110] At (408), method 400 may include applying a second machine learning model to the causal graph to estimate one or more parameters associated with the causal graph. For example, a computing system (e.g., computing system 600) may apply the second machine learning model to the causal graph to estimate one or more parameters associated with the causal graph. In some implementations, the second machine learning model may be a kernel-based machine learning model. For example, the kernel-based machine learning model may be a kernel-based regularized least squares regression model.

[0111] At (410), method 400 may include generating data indicative of a graphical representation of a causal graph.For example, a computing system (eg, computing system 600) may generate data indicative of a graphical representation of a causal graph.

[0112] At (412), method 400 may include executing a verification method. For example, a computing system (eg, computing system 600) may execute the verification method. Figure 4B , executing the verification method may include step 412A and step 412B.

[0113] At (412A), the verification method may be performed by transmitting data including instructions that cause a graphical representation of the causal graph to be displayed via a graphical user interface of a device associated with a user. For example, a computing system (e.g., computing system 600) may transmit data including instructions that cause a graphical representation of the causal graph to be rendered via a graphical user interface of a device associated with a user (e.g., Figure 5 For example, method 400 may include transmitting data including instructions that cause a graphical representation of a cause-effect graph to be displayed via a graphical user interface of a device associated with a user.

[0114] At (412B), the verification method may be performed by obtaining data indicating user input to accept the graphical representation of the causal graph. For example, a computing system (e.g., computing system 600) may obtain data indicating user input to accept the graphical representation of the causal graph. For example, method 400 may include obtaining data indicating user input to accept the graphical representation of the causal graph.

[0115] At (414), method 400 may include determining resource allocations to multiple media channels based on a causal graph. For example, a computing system (e.g., computing system 600) may determine resource allocations to multiple media channels based on a causal graph. As described herein, determining resource allocations to multiple media channels based on a causal graph may include determining a relationship between a first node, a second node, and a third node. Based on the relationship between the first node, the second node, and the third node, determining resource allocations to multiple media channels based on the causal graph may include determining an optimal budget allocation for a first media channel associated with the first node, a second media channel associated with the second node, and a third media channel associated with the third node. This may be performed for any number of nodes.

[0116] Determining the optimal budget allocation for the first media channel, the second media channel, and the third media channel may include simulating data indicating a plurality of budget allocation scenarios (e.g., Figure 5 ), wherein each budget allocation scenario indicates a budget amount allocated to each of the first media channel, the second media channel, and the third media channel. Determining the optimal budget allocation for the first media channel, the second media channel, and the third media channel may include obtaining user-input data indicating a target parameter. Determining the optimal budget allocation for the first media channel, the second media channel, and the third media channel may include, for each corresponding budget allocation scenario in a plurality of budget allocation scenarios, determining a corresponding parameter of a corresponding budget allocation scenario in a plurality of budget allocation scenarios. Determining the optimal budget allocation for the first media channel, the second media channel, and the third media channel may include comparing each corresponding parameter with a target parameter. Determining the optimal budget allocation for the first media channel, the second media channel, and the third media channel may include determining that a first corresponding parameter associated with the first budget allocation scenario is within a threshold of the target parameter. Determining the optimal budget allocation for the first media channel, the second media channel, and the third media channel may include selecting the first budget allocation scenario as the optimal budget allocation scenario in response to determining that the first corresponding parameter is within the threshold of the target parameter. For example, the target parameter and the corresponding parameter may indicate a percentage reduction in cost per sale. In some implementations, the target parameter and the corresponding parameter may indicate an overall total media expenditure budget.

[0117] In some implementations, the target parameter and the corresponding parameter associated with each respective budget allocation scenario indicate a percentage reduction in cost per sale. In some implementations, the cost per sale is determined by calculating the media expenditure for the respective media channel divided by the unit sales driven by the respective media channel. In some implementations, the target parameter and the corresponding parameter associated with each respective budget allocation scenario indicate an attribution associated with the effectiveness of the media channel and promotional use for the target action. In some implementations, the attribution for the respective media channel is determined by calculating the unit sales driven by the respective media channel divided by the overall sales.

[0118] At (416), method 400 may include generating data indicating a graphical representation of the determined resource allocation for the plurality of media channels. For example, a computing system (e.g., computing system 600) may generate data indicating a graphical representation of the determined resource allocation for the plurality of media channels. Figure 5 A graphical representation of the determined resource allocation for a plurality of media channels is described.

[0119] At (418), method 400 may include transmitting data including instructions that, when executed, cause a graphical representation of resource allocations to a plurality of media channels to be rendered via a graphical user interface of a user device (e.g., as Figure 5 For example, a computing system (eg, computing system 600) may transmit data including instructions that, when executed, cause a graphical representation of resource allocations to multiple media channels to be rendered via a graphical user interface of a user device.

[0120] As described herein, systems and methods may provide for transmission of data including instructions that cause display of various data to be rendered via a user interface of one or more devices associated with a user. Figure 5 An example graphical user interface 500 is depicted. The graphical user interface 500 may represent a dashboard provided to a user associated with an online service (e.g., for generating, supplying, or analyzing the performance of content items displayed to third parties). The graphical user interface 500 may include a visualization of a cause-effect diagram 505, a visualization of a simulated performance 510, and a recommended resource allocation 515.

[0121] As described herein, a computing system (e.g., computing system 600) can transmit data including instructions. A client device can receive the instructions and execute the instructions. Execution of the instructions can cause causal graph 505, simulated performance 510, or recommended resource allocation 515 to be displayed via a user interface (e.g., graphical user interface 500) of a device associated with a user.

[0122] Figure 5An example user interface rendering is depicted for example purposes only and is not intended to limit the manner in which the information may be rendered via a user interface on a device associated with a user. In some implementations, the display may include an audio display, a visual display, or a combination of an audio and visual display. In some implementations, the graphical user interface 500 may include one or more selectable components to obtain data indicative of user input (e.g., such as Figure 6 described in ).

[0123] Figure 6 A block diagram of an example computing system 600 is depicted that performs determination of direct and indirect relationships of media channels for determining future resource allocations according to an example embodiment of the present disclosure. Computing system 600 includes a user computing system 602, a server computing system 630, and a training computing system 650 communicatively coupled via a network 680.

[0124] The user computing system 602 may include a user device. The user device may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0125] The user computing system 602 includes one or more processors 612 and memory 614. The one or more processors 612 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or multiple processors that are operably connected. The memory 614 can include one or more temporary or non-temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc. and combinations thereof. The memory 614 can store data 616 and instructions 618 executed by the processor 612 to cause the user computing system 602 to perform operations.

[0126] In some implementations, the user computing system 602 may store or include one or more machine-learned models 620. For example, the machine-learned model 620 may be or may otherwise include various machine-learned models, such as a Bayesian belief network, kernel-based regularized least squares regression. A Bayesian belief network may include a probabilistic graphical model that includes nodes and directed edges and learns from data. Kernel-based regularized least squares regression may include nonlinear regression estimation. In some implementations, the machine-learned model 620 may be or may otherwise include a neural network (e.g., a deep neural network) or other types of machine-learned models, including nonlinear models and / or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks. Some example machine-learned models may utilize attention mechanisms, such as self-attention. For example, some example machine-learned models may include a multi-head self-attention model (e.g., a transformer model). Reference Figure 1 and Figure 2 An example machine-learned model 620 is discussed.

[0127] In some implementations, one or more machine-learned models 620 may be received from the server computing system 630 via the network 680, stored in the user computing device memory 614, and then used or otherwise implemented by the one or more processors 612. In some implementations, the user computing system 602 may implement multiple parallel instances of a single machine-learned model 620 (e.g., to perform parallel learning across multiple instances of causal graph generation).

[0128] More particularly, the overall model may include a set of machine-learned modeling capabilities that are robust to data assumptions and easily scalable. The machine-learned modeling capabilities may determine the structure of a network (e.g., a Bayesian belief network) and utilize the network to make resource allocation determinations. The output of the modeling capabilities may be used to more efficiently allocate resources to improve desired target outcomes. The machine-learned models may be used to accelerate the modeling process of media channel performance and allow for near real-time learning of the impact of resource allocation on target actions performed.

[0129] Additionally or alternatively, one or more machine-learned models 640 may be included in or otherwise stored and implemented by a server computing system 630 that communicates with the user computing system 602 according to a client-server relationship. For example, the machine-learned models 640 may be implemented by the server computing system 630 as part of a web service (e.g., a marketing service). Thus, one or more models 620 may be stored and implemented at the user computing system 602, and / or one or more models 640 may be stored and implemented at the server computing system 630.

[0130] The user computing system 602 may also include one or more user input components 622 that receive user input. For example, the user input component 622 may be a touch-sensitive component (e.g., a touch-sensitive display screen or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0131] The user computing system 602 may include one or more user interfaces 624. For example, the user interface 624 may include a graphical user interface, an audio user interface, a command line interface, a menu-driven user interface, a touch user interface, a voice user interface, a form-based user interface, or a natural language user interface.

[0132] The server computing system 630 includes one or more processors 632 and memory 634. The one or more processors 632 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or multiple processors that are operably connected. The memory 634 can include one or more temporary or non-temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc. and combinations thereof. The memory 634 can store data 636 and instructions 638 that are executed by the processor 632 to cause the server computing system 630 to perform operations.

[0133] In some implementations, the server computing system 630 includes or is otherwise implemented by one or more server computing devices. Where the server computing system 630 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0134] As described above, the server computing system 630 may store or otherwise include one or more machine-learned models 640. For example, the model 640 may be or may otherwise include various machine-learned models. The machine-learned model 640 may include, for example, one or more structural learning models 640A, one or more parameter learning models 640B, or one or more resource allocation models 640C. Example machine-learned models may be or may otherwise include various machine-learned models, such as Bayesian belief networks, kernel-based regularized least squares regression. The Bayesian belief network may include a probabilistic graph model that includes nodes and directed edges and learns from data. Kernel-based regularized least squares regression may include nonlinear regression estimation. In some implementations, the machine-learned model 640 may be or may otherwise include a neural network or other multi-layer nonlinear model. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine-learned models may utilize attention mechanisms, such as self-attention. For example, some example machine-learned models may include multi-head self-attention models (e.g., transformer models). Reference Figure 1 and Figure 2 Example model 640 is discussed.

[0135] User computing system 602 and / or server computing system 630 may train models 620 and / or 640 via interaction with training computing system 650 communicatively coupled via network 680. Training computing system 650 may be separate from server computing system 630 or may be part of server computing system 630.

[0136] The training computing system 650 includes one or more processors 652 and memory 654. The one or more processors 652 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or multiple processors operably connected. The memory 654 can include one or more temporary or non-temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc. and combinations thereof. The memory 654 can store data 656 and instructions 658 executed by the processor 652 to cause the training computing system 650 to perform operations. In some implementations, the training computing system 650 includes one or more server computing devices or is otherwise implemented by one or more server computing devices.

[0137] The training computing system 650 may include a model trainer 660 that trains the machine-learned models 620 and / or 640 stored at the user computing system 602 and / or the server computing system 630 using various training or learning techniques, such as, for example, back propagation of errors. For example, a loss function may be back propagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update parameters over multiple training iterations.

[0138] In some implementations, performing back propagation of errors may include performing truncated back propagation through time. The model trainer 660 may perform a variety of generalization techniques (eg, weight decay, dropout, etc.) to improve the generalization capabilities of the model being trained.

[0139] In particular, the model trainer 660 can train the machine-learned models 620 and / or 640 based on a set of training data 662. The training data 662 can include, for example, data associated with historical resource allocations (e.g., one or more months of media spending), daily data at a geographic level (e.g., zip code, city, or DMA level) (e.g., resource allocations, sales, conversion rates for various media channels), weekly data at a geographic level (e.g., zip code, city, or DMA level) (e.g., resource allocations, sales, conversion rates for various media channels), or different media weights across different time periods (e.g., when resources are not allocated to media channels to correspond to expected benchmarks). Media channels can include, for example, digital media, search, television, and sales, etc.

[0140] In some implementations, if the user has provided consent, the training examples can be provided by the user computing system 602. Thus, in such implementations, the model 620 provided to the user computing system 602 can be trained by the training computing system 650 based on user-specific data received from the user computing system 602. In some cases, this process can be referred to as personalizing the model.

[0141] Model trainer 660 includes computer logic for providing the desired functionality. Model trainer 660 can be implemented in hardware, firmware, and / or software that controls a general purpose processor. For example, in some implementations, model trainer 660 includes a program file stored on a storage device, loaded into a memory, and executed by one or more processors. In other implementations, model trainer 660 includes one or more sets of computer executable instructions stored in a tangible computer readable storage medium such as RAM, a hard disk, or an optical or magnetic medium.

[0142] The network 680 can be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. In general, communications over the network 680 can be conducted via any type of wired and / or wireless connection using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0143] The machine-learned models described in this specification can be used in a variety of tasks, applications, and / or use cases.

[0144] In some implementations, the input of the machine-learned model of the present disclosure may be statistical data. The statistical data may be, represent, or otherwise include data calculated and / or computed from some other data source. The machine-learned model may process the statistical data to generate an output. As an example, the machine-learned model may process the statistical data to generate an identification output. As another example, the machine-learned model may process the statistical data to generate a prediction output. As another example, the machine-learned model may process the statistical data to generate a classification output. As another example, the machine-learned model may process the statistical data to generate a segmentation output. As another example, the machine-learned model may process the statistical data to generate a visualization output. As another example, the machine-learned model may process the statistical data to generate a diagnostic output.

[0145] In some implementations, the input of the machine-learned model of the present disclosure may be text or natural language data. The machine-learned model may process text or natural language data to generate an output. As an example, the machine-learned model may process natural language data to generate a language encoding output. As another example, the machine-learned model may process text or natural language data to generate a potential text embedding output. As another example, the machine-learned model may process text or natural language data to generate a translation output. As another example, the machine-learned model may process text or natural language data to generate a classification output. As another example, the machine-learned model may process text or natural language data to generate a text segmentation output. As another example, the machine-learned model may process text or natural language data to generate a semantic intent output. As another example, the machine-learned model may process text or natural language data to generate an amplified text or natural language output (e.g., text or natural language data with higher quality than the input text or natural language, etc.). As another example, the machine-learned model may process text or natural language data to generate a predicted output.

[0146] In some implementations, the input of the machine-learned model of the present disclosure may be sensor data. The machine-learned model may process the sensor data to generate an output. As an example, the machine-learned model may process the sensor data to generate a recognition output. As another example, the machine-learned model may process the sensor data to generate a prediction output. As another example, the machine-learned model may process the sensor data to generate a classification output. As another example, the machine-learned model may process the sensor data to generate a segmentation output. As another example, the machine-learned model may process the sensor data to generate a visualization output. As another example, the machine-learned model may process the sensor data to generate a diagnostic output. As another example, the machine-learned model may process the sensor data to generate a detection output.

[0147] Figure 6 An example computing system that can be used to implement the present disclosure is shown. Other computing systems can also be used. For example, in some implementations, the user computing system 602 can include a model trainer 660 and training data 662. In such implementations, the model 620 can be both trained and used locally at the user computing system 602. In some of such implementations, the user computing system 602 can implement the model trainer 660 to personalize the model 620 based on user-specific data.

[0148] The techniques discussed herein refer to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for a variety of possible configurations, combinations, and partitioning of tasks and functionality between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0149] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided by way of explanation, rather than limitation of the present disclosure. Those skilled in the art may easily produce changes, variations, and equivalents to such embodiments after understanding the foregoing. Therefore, the present disclosure does not exclude such modifications, variations, and / or additions to the subject matter that would be readily apparent to those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment may be used together with another embodiment to produce yet another embodiment. Therefore, the present disclosure is intended to cover such changes, variations, and equivalents.

[0150] The steps depicted or described are merely illustrative and may be omitted, combined, and / or performed in an order different from that depicted or described; the numbering of the depicted steps is merely for ease of reference and does not imply that any particular order is necessary or preferred.

[0151] The functions and / or steps described herein may be embodied in computer-usable data or computer-executable instructions executed by one or more computers or other devices to perform one or more functions described herein. Typically, such data and / or instructions include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific data types when executed by one or more processors in a computer and / or other data processing device. Computer-executable instructions may be stored on computer-readable media such as hard disks, optical disks, removable storage media, solid-state memory, read-only memory (ROM), random access memory (RAM), etc. As will be understood, the functionality of such instructions may be combined and / or distributed as needed. In addition, the functionality may be embodied in firmware and / or hardware equivalents such as integrated circuits, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. Specific data structures may be used to more effectively implement one or more aspects of the present disclosure, and such data structures are contemplated to be within the scope of computer-executable instructions and / or computer-usable data described herein.

[0152] Although not required, it will be appreciated by those of ordinary skill in the art that the various aspects described herein may be embodied as methods, systems, devices, and / or one or more computer-readable media storing computer-executable instructions. Thus, the various aspects may take the form of all-hardware embodiments, all-software embodiments, all-firmware embodiments, and / or embodiments combining software, hardware, and / or firmware aspects in any combination.

[0153] As described herein, various methods and actions may operate across one or more computing devices and / or networks. Functionality may be distributed in any manner, or may be located in a single computing device (eg, server, client computer, user device, etc.).

[0154] Various aspects of the present disclosure have been described with respect to illustrative embodiments thereof. Numerous other embodiments, modifications and / or variations within the scope and spirit of the appended claims may occur to one of ordinary skill in the art upon review of the present disclosure. For example, one of ordinary skill in the art may appreciate that the steps depicted and / or described may be performed in an order different from that described, and / or one or more of the steps shown may be optional and / or combined. Any and all features in the appended claims may be combined and / or rearranged in any possible manner.

[0155] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided by way of explanation, rather than limitation of the present disclosure. Those skilled in the art may easily produce changes, variations and / or equivalents to such embodiments after understanding the foregoing. Therefore, the present disclosure does not exclude such modifications, variations and / or additions to the subject matter that would be readily apparent to those of ordinary skill in the art. For example, a feature shown and / or described as part of one embodiment may be used together with another embodiment to produce yet another embodiment. Therefore, the present disclosure is intended to cover such changes, variations and / or equivalents.

Claims

1. A computer-implemented method comprising: Obtain data associated with multiple media channels; inputting the data into a machine learning model; estimating the structure of the causal graph by the machine learning model; applying a second machine learning model to the causal graph to estimate one or more parameters associated with the causal graph; as well as An allocation of resources to the plurality of media channels is determined based on the cause-effect graph.

2. The method of claim 1, wherein the causal graph comprises a Bayesian belief network and the second machine learning model comprises a kernel-based machine learning model.

3. The method of any preceding claim, wherein estimating the structure of the causal graph by the machine learning model comprises: Structural learning is performed to generate a causal graph including a plurality of nodes and a plurality of edges.

4. The method of claim 3, wherein the structure learning comprises: Apply machine learning models for feature selection; as well as Self-service aggregation is performed for updating the causal graph.

5. The method of claim 3 or 4, wherein estimating the structure of the causal graph comprises: Parameter learning is performed to determine (i) a size and (ii) a direction of each respective edge of the plurality of edges.

6. The method of claim 5, wherein performing the parameter learning comprises: Perform parameter regularization.

7. The method of any one of claims 3 to 6, wherein the causal graph comprises a directed acyclic graph representing conditional probabilities between the nodes.

8. A method as claimed in any preceding claim, comprising performing the verification method by: transmitting data including instructions that cause a graphical representation of the cause-effect diagram to be displayed via a graphical user interface of a device associated with a user; and Data indicative of acceptance of user input of the graphical representation of the cause-effect diagram is obtained.

9. The method of any preceding claim, wherein determining the resource allocation to the plurality of media channels based on the cause-effect graph comprises: determining a relationship between the first node, the second node, and the third node; as well as Based on the relationship between the first node, the second node, and the third node, optimal budget allocations for a first media channel associated with the first node, a second media channel associated with the second node, and a third media channel associated with the third node are determined.

10. The method of claim 9, wherein determining the optimal budget allocation for the first media channel, the second media channel, and the third media channel comprises: simulating data indicative of a plurality of budget allocation scenarios, wherein each budget allocation scenario indicates an amount of budget allocated to each of the first media channel, the second media channel, and the third media channel; obtaining user input data indicative of a target parameter; For each corresponding budget allocation scenario in the multiple budget allocation scenarios, determining corresponding parameters of the corresponding budget allocation scenario in the multiple budget allocation scenarios; comparing each corresponding parameter to the target parameter; determining that a first corresponding parameter associated with a first budget allocation scenario is within a threshold of the target parameter; and In response to determining that the first corresponding parameter is within the threshold of the target parameter, the first budget allocation scenario is selected as an optimal budget allocation scenario.

11. The method of any one of claims 9 to 10, wherein the target parameter and the respective parameter associated with each respective budget allocation scenario indicate a percentage reduction in cost per sale.

12. The method of claim 11, wherein the cost per sale is determined by calculating the media expenditure of the corresponding media channel divided by the unit sales driven by the corresponding media channel.

13. The method of any one of claims 10 to 12, wherein the target parameter and the respective parameter associated with each respective budget allocation scenario indicate attribution associated with the effectiveness of media channels and promotion usage on a target action.

14. The method of claim 13, wherein the attribution for the respective media channel is determined by calculating unit sales driven by the respective media channel divided by overall sales.

15. The method of any one of claims 10 to 14, wherein the target parameter and the corresponding parameter associated with each corresponding budget allocation scenario is an overall total media expenditure budget.

16. A method as claimed in any preceding claim, comprising: generating data indicative of a graphical representation of the cause-effect graph; as well as Data including instructions are transmitted that, when executed, cause the graphical representation of the causal graph to be rendered via a graphical user interface of a user device.

17. A method as claimed in any preceding claim, comprising: generating data indicating a graphical representation of the determined resource allocations to the plurality of media channels; as well as Data is transmitted including instructions that, when executed, cause the graphical representation of the resource allocations to the plurality of media channels to be rendered via a graphical user interface of a user device.

18. A method as claimed in any preceding claim, wherein the data associated with the plurality of media channels comprises outcome data, predictor variables and control variables.

19. A computing system comprising: one or more processors; as well as One or more computer-readable media storing instructions executable to cause the one or more processors to perform operations including the method of any one of claims 1 to 18.

20. One or more computer-readable media storing instructions executable by one or more processors to perform operations comprising the method of any one of claims 1 to 18.

Images