Deep Causal Learning for E-Commerce Content Generation and Optimization

Abstract

A system for optimizing business objectives of e-commerce content may include a memory and a processor coupled to the memory. The processor may receive one or more hypotheses for multivariate comparison of content. The content may be provided to a user of the e-commerce system. The processor may iteratively generate self-organizing experimental units (SOEUs) based on the one or more hypotheses. The processor may inject the SOEUs into an online system to generate quantified inferences about the content. The processor may identify at least one confidence interval within the quantified inference in response to injecting the SOEUs. The processor may iteratively modify the SOEUs based on the at least one confidence interval to identify at least one causal interaction of e-commerce content in the system. Other methods and apparatus are described.

Description

[Technical Field]

[0001] The present invention relates to determining the effectiveness of e-commerce content and optimizing content delivery to enhance business goals, and more particularly to performing these operations simultaneously. [Background technology]

[0002] E-commerce is a rapidly growing retail channel. Vendors can tailor how their products are marketed to consumers by varying the content that is presented to consumers as they browse and transact on e-commerce sites. By varying such content, vendors can influence consumer responses and further glean insights into how this may affect transactions for the corresponding service or product. Effective management of presented content, understanding consumer responses to the content, and its continuous optimization are key components for vendors to maximize their e-commerce business goals (i.e., enhancing sales and / or profits). Summary of the Invention

[0003] Disclosed herein are systems, devices, software, and methods for optimizing e-commerce content to maximize business goals.

[0004] In one embodiment, a system for optimizing business objectives of e-commerce content is described, the system comprising: a memory; and a processor coupled to the memory, the processor configured to (a) receive one or more hypotheses for randomized multivariate comparisons of content provided to users of the system; (b) iteratively generate Self-Organizing Experimental Units (SOEUs) based on the one or more hypotheses; (c) inject the SOEUs into the system to generate quantified inferences about the content; (d) in response to the injection of the SOEUs, identify at least one confidence interval within the quantified inference; and (e) iteratively revise the SOEUs based on the at least one confidence interval to identify at least one causal interaction of e-commerce content in the system.

[0005] In another embodiment, a computer-implemented method for optimizing business objectives of e-commerce content is described, the method including: receiving one or more hypotheses for multivariate comparison of content, the content including content to be provided to users of a system; iteratively generating self-organizing experimental units (SOEUs) based on the one or more hypotheses; injecting the SOEUs into the system to generate quantified inferences about the content; identifying at least one confidence interval within the quantified inference in response to injecting the SOEUs; and iteratively revising the SOEUs based on the at least one confidence interval to identify at least one causal interaction of e-commerce content in the system.

[0006] These and other aspects will become apparent from the following detailed description. However, this broad summary should not be construed to limit the subject matter that can be claimed, regardless of whether such subject matter is presented in the claims originally filed in the application or in claims amended or otherwise presented during prosecution. [Brief explanation of the drawings]

[0007] The drawings are not necessarily drawn to scale, and like reference numerals in the drawings may describe similar components in different views. Like reference numerals with different letter suffixes may represent different instances of similar components. Some embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings.

[0008] [Figure 1] FIG. 1 illustrates a system for e-commerce content creation and optimization, according to various embodiments.

[0009] [Figure 2] FIG. 2 is a block diagram of software modules and core processes for a system according to various embodiments.

[0010] [Figure 3] 1 is a flowchart of a computer-implemented method for e-commerce content generation and optimization, according to various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0011] With regard to the following glossary of defined terms, these definitions shall apply throughout this application, unless a different definition is provided in the claims or elsewhere in this specification.

[0012] Glossary Certain terms are used throughout this specification and claims, most of which are well known but may require some explanation. As used in this specification and the accompanying embodiments, it should be understood that:

[0013] The singular forms "a," "an," and "the" include plural referents unless the content clearly dictates otherwise. As used in this specification and the accompanying embodiments, the term "or" is generally utilized in its sense including "and / or" unless the content clearly dictates otherwise.

[0014] The terms independent variable (IV) and extraneous variable (EV) are generally used to refer to variables that are manipulated by the user and variables that are not controlled by the user. Independent variables can be discrete or continuous. Extraneous variables are typically continuous.

[0015] The term "level" used in conjunction with experimental units is generally used as a feature status or choice of independent variable (IV). For example, if two levels of a feature are defined, the first level means that the feature is active in the experimental unit, and the second level would define the feature as inactive. An additional state or status may then be defined that simply defines whether the IV is active or inactive.

[0016] The term "repeatedly" is used generally to refer to something that is done with or without a particular order. As an example, a process can perform a set of steps in a specified order, continuously or repeatedly (e.g., if a process includes steps 1-5, the process performs steps 1, 2, 3, 4, 5 in that order, or in the reverse order, i.e., steps 5, 4, 3, 2, 1), or the steps can be performed randomly or non-sequentially (e.g., 1, 3, 5, 4, 2, or any combination thereof).

[0017] "Interchangeable" or "interchangeability" is generally deployed to mean statistically equivalent in terms of the results of content allocation.

[0018] The terms "causal" or "causal relationship / interaction / inference" indicate, positively or negatively, that the presence, absence, variation, or modification of particular content impacts or influences other content and its ability to affect user interaction (i.e., purchasing a particular product).

[0019] "Positivity" is generally defined to mean a probability of occurrence or selection greater than or equal to zero or non-zero.

[0020] The term "confounders" includes Hawthorne effects, order / carryover effects, demand characteristics, extraneous variables, and / or other factors that may vary systematically with levels of the independent variables.

[0021] The recitation of numerical ranges by endpoints includes all numbers subsumed within that range (eg, 1 to 5 includes 1, 1.5, 2, 2.75, 3, 3.8, 4, and 5).

[0022] Unless otherwise indicated, all numbers expressing quantities or ingredients, measurements of properties, and the like used in the specification and embodiments are to be understood as being modified in all instances by the term "about." Accordingly, unless otherwise indicated, the numerical parameters set forth in the foregoing specification and accompanying recitations of embodiments may vary depending upon the desired properties sought to be obtained by one of ordinary skill in the art utilizing the teachings of the present disclosure. At the very least, each numerical parameter should at least be construed in light of the number of reported significant digits and by applying ordinary rounding techniques; however, this is not intended to limit the application of the doctrine of equivalents to the scope of the claimed embodiments.

[0023] Various exemplary embodiments of the present disclosure will now be described with specific reference to the drawings. Various modifications and variations may be made to the exemplary embodiments of the present disclosure without departing from the spirit and scope of the present disclosure. Therefore, it should be understood that the embodiments of the present disclosure are not limited to the exemplary embodiments described below, but are instead governed by the limitations set forth in the claims and any equivalents thereof.

[0024] Generally, humans and many machine learning implementations make decisions under conditions of probabilistic uncertainty. Recognizing patterns, inferences, or connections within a dataset through passive observation is difficult without introducing conscious or unconscious bias or unregulated assumptions. Datasets can pose additional challenges because they can introduce 1) selection or sampling bias, 2) confounding variables, and 3) a lack of directional evidence. Controlled or adaptive experiments aim to eliminate bias by introducing aspects of randomization, blocking, and balancing, but progress remains hindered by the vast amount of prior knowledge required to produce tangible results (i.e., ensuring high internal and external validity) and the inflexible constraints imposed by real-world decisions. Adaptive experiments perform one or more steps sequentially, often requiring previous steps to finish before subsequent steps can be addressed. The technology described herein overcomes passive observation and adaptive experimentation by transforming controlled or adaptive experiments into a discontinuous process that iteratively analyzes and optimizes data through self-organizing experiments. The self-organizing process intelligently exploits natural variability in decision timing, sequence, and parameters to automatically calculate and reliably infer causal relationships. Advantages of self-organizing adaptive learning systems and methods over existing adaptive experimentation techniques include their ability to operate with poor-quality inputs, where conditions or interactions are initially unknown, incomplete, or hypothetical estimates and are learned over time. Another advantage of adaptive learning systems and methods is their robustness to erroneous assumptions, including the effects of time, the duration over which content conditions need or can be analyzed, and external factors (e.g., consumer trends or trends, seasonal fluctuations, natural or man-made disasters, etc.). The iterative exploitation of spatiotemporally discontinuous causal relationships is another advantage over existing systems because the location of content and its comparative impact on the sequence of other content is critical to understanding and optimizing the most effective content for e-commerce systems.

[0025] The system and method provides fully automated operational control and integrated multi-objective optimization while providing real-time understanding and quantification of causality. The behavior of the self-organizing system and method is robust and scalable, and works effectively for complex real-world systems, including systems subject to deviations in space-time relationships and product diversity (i.e., e-commerce systems).

[0026] Modern e-commerce systems allow vendors to influence different types of consumer behavior based on the interactive content elements displayed. One challenge experienced in e-commerce systems is demonstrating measurable consumer responses to content. Some vendors focus on consumer responses, which are easy to measure and understand and may include click-through rates or survey / questionnaire responses. For example, consumer "clicks" are a type of consumer response that analyzes which products, images, or links consumers view or interact with on an e-commerce site. These are interest measures that may or may not result in an actual product sale. Typically, consumer responses, which are easy to understand and measure, may not accurately reflect parameters that provide strategic direction to vendors, such as sales, revenue, and profits. For example, consumers may click because an image caught their attention, even though they have no intention of purchasing the corresponding product. In this example, a vendor seeking only to optimize the number of "clicks" on an e-commerce site selling products may miss an opportunity to select content that directly enhances sales and profits. "Clicks" are variables, and consumer behavior varies in what they may represent and how they are interpreted as conversions (i.e., indicating that a particular piece of content influenced sales). For example, if a consumer already knows they want to purchase a product from an e-commerce site, they will source it with a single click. Another consumer may actively browse multiple e-commerce sites one or more times on the same day or over several days or weeks before actually purchasing the product. A system intended to maximize correlation needs to understand which content directly leads to sales, and when this is confirmed.

[0027] In many e-commerce systems, displayed or interactable content options are manually selected to address business goals (i.e., increasing sales and profits), which is costly and time / labor-intensive. Such manual selection becomes increasingly difficult for e-commerce sites that manage multiple products. Furthermore, optimizing sales for individual products can gain market share from competitors, but it can also lead to in-house competition for similar products from a single vendor. For example, a vendor may sell multiple furnace filters with many different options and profit margins. The vendor would prefer those furnace filters to be purchased over competitor brands, but at the same time, the vendor would prefer the high-margin furnace filters to be purchased over lower-margin furnace filters. By only optimizing sales, the vendor may miss opportunities to optimize profits or revenue. Typically, each product is managed individually, and its interactions with other products are not considered. Another advantage of self-organizing adaptive learning systems and methods is the ability to evaluate and address in-house competition for products, and more generally, product portfolio optimization.

[0028] Embodiments include methods and systems for optimizing business goals on e-commerce platforms. System inputs can include candidate content elements (e.g., snippets of text and / or images) and constraints on how and why content can be combined and presented to consumers (e.g., a 200-character limit on product names or descriptions). Inputs can also include initial assumptions, e.g., related to business goals, historical context and prior discovery / learning, time lag between content viewing and purchase decisions, and systemic constraints. Systems according to embodiments can specify protocols for assembling content elements. Methods according to some embodiments can identify causal relationships between provided content elements and purchasing behavior while optimizing revenue and profits. The system can be configured for any objective goal represented by human behavior. As described in more detail below, causality is measured by calculating the statistical significance of the presence (versus absence) of a content element on or within a group of self-organizing experimental units. Evaluating statistical significance is achieved by calculating a confidence interval, thereby quantifying the expected value of the content element's effect and the uncertainty surrounding it (and representing a measure or degree of inference). Calculating unbiased confidence intervals in this case is relatively straightforward due to random sampling / randomization. The interpretation and adaptive use of confidence intervals to automatically understand and exploit the specific effects of content inclusion, placement, and duration, as well as deep learning-like self-organized comparisons with other content (to remove confounding effects of covariates), advantageously differentiate this system and method from the limitations of current solutions. The calculation of one or more confidence intervals quantifies both the expected effect and the range around it (i.e., quantification of best- and worst-case scenarios), enabling risk-adjusted optimization. Methods and systems according to embodiments can identify and adjust for erroneous inputs (e.g., erroneous assumptions) that would confound cause-and-effect knowledge and limit optimization results, as well as monitor and exploit changes in the causal relationship between content and consumer behavior.

[0029] 1 illustrates a system 100 for e-commerce content generation and optimization, according to various embodiments. The system 100 includes a memory 102 and a processor 104 coupled to the memory 102. The processor 104 can receive input from a user interface 110 that includes one or more hypotheses 106 for multivariate comparison of content. The hypotheses 106 can also be retrieved from the memory 102. The input can further include content elements, which can be stored in or accessed from the memory 102. As previously described herein, the content is provided to and optimized on the e-commerce system 114 to maximize business goals.

[0030] The processor 104 and memory 102 may be part of a user system 116, which includes a user interface 110 for inputting hypotheses. As an example, the user system 116 may be a mobile device (e.g., a smartphone, laptop, etc.) or a fixed device (i.e., a desktop computer) running an application on the device or in a cloud environment that displays the user interface 110 and connects to the e-commerce system 114 via a wired or wireless network. In another embodiment, the processor 104 and memory 102 may operate on an e-commerce user system 118. The e-commerce user system 118 will receive input from a user interface 110 running on a mobile device or a fixed device running an application on the device or in a cloud environment. The hypotheses 106, including the content elements, will be stored and processed directly on the e-commerce user system 118. The user system 116 and the e-commerce user system 118 can also operate simultaneously, meaning that data is stored and processed interchangeably between them.

[0031] The processor 104 can iteratively generate self-organizing experimental units (SOEUs) 112 based on one or more hypotheses 106. The SOEUs 112 (described in more detail later in this specification with respect to FIG. 3 and associated tables) quantify inferences within and between content.

[0032] At least one SOEU 112 may include a duration for which the SOEU 112 is active within a system (e.g., the electronic commerce system 114). The processor 104 may generate multiple SOEUs 112 with randomly selected durations based on a uniform distribution, a Poisson distribution, a Gaussian distribution, a binomial distribution, or any distribution supported by a bounded or unbounded interval. In one embodiment, the duration may be the longest duration among all generated SOEUs, and all intermediate durations will be recorded simultaneously. The processor 104 may then select the duration that maximizes statistical significance among all recorded durations. The processor 104 may also dynamically modify (i.e., increase or decrease) the potential duration between SOEUs 112 until the carryover effect of the SOEU 112 on subsequent SOEUs 112 weakens or is completely eliminated, meaning that the carryover effect is fully reversible. The processor 104 may increase or decrease the duration of at least one SOEU 112 based on quantified inference or in adaptation to a positive or negative outcome of a causal evaluation (i.e., an assessment of external validity by comparing utilization to a baseline, where the baseline may be the average of all possible content options as defined in more detail with respect to FIG. 2).

[0033] The e-commerce system 114 may include an online shopping or product sales portal, a website, or a mobile application. The e-commerce system 114 may be, for example, an enterprise content management system, optimized for business-to-business (B2B) purposes, or a direct-to-consumer private or public portal (e.g., Amazon, Target, Home Depot, Walmart, etc.) that displays and transacts products. Intranet or Internet search engines (e.g., Google, Yahoo, Bing, etc.) are also included, as consumers / users utilize them to explore products, compare prices, and read customer reviews. Each SOEU 112 may represent a single product or may represent variations in content specific to a single product. The processor 104 may group the SOEUs 112 into blocks or clusters based on quantified inferences of variations in content effectiveness between experimental groups. The quantified inferences are based on characteristics of the content contained in each SOEU as well as characteristics across experimental groups, such as product, time period, and geographic location. The processor 104 can identify separate causal interactions for each cluster and select optimal content for each cluster based on the separate causal inference for each cluster.

[0034] Once generated, the processor 104 may continuously inject the SOEUs 112 into the electronic commerce system 114 and iteratively modify the SOEUs 112 according to the methods and criteria described below with respect to FIG. 3 to identify at least one causal interaction of content within the electronic commerce system 114. The processor 104 may assign content to the SOEUs 112 initially uniformly and then iteratively more non-uniformly in proportion to the amount of evidence of relative expected utility as quantified by a confidence interval. The processor 104 may generate at least one group of SOEUs 112 based on a uniform probability distribution of inclusion experimental units associated with at least one hypothesis 106 using a process defined as described below.

[0035] The assumptions 106 may include goals for the electronic commerce system 114. Goals may include performance metrics for which the system performs risk-adjusted optimization. Examples include, but are not limited to, revenue, maximum or minimum sales, gross margin, profit margin, cost of goods sold (COGS), inventory control / levels, price, transportation / shipping costs, market share, or combinations thereof.

[0036] The hypotheses 106 may include content elements that identify product attributes or specific details, such as, but not limited to, product name, description, purpose, dimensions, price, or combinations thereof.

[0037] The assumptions 106 can include time constraints or specific constraints on the content. Time constraints include the time and duration that the content is to be active, inactive (i.e., only appropriate at certain times of the day or year), or displayed within the system. Constraints on the content can include the presence or absence of product images or videos, standardization of product brand names or descriptions, whitespace or blank text, duplicate text, use of symbols, maximum number of characters allowed, or a combination thereof.

[0038] The hypotheses 106 can be defined initially and then iteratively updated, either manually or automatically, as additional information becomes available or as the system analyzes and optimizes causal inference.

[0039] The user interface 110 is a web or application-based portal that users access to enter hypotheses 106 for the system. The user interface 110 may be presented as a graphical user window on a monitor or smartphone display. The user enters hypotheses 106 using a keyboard or virtual keyboard on the device used to access the system.

[0040] Components of system 100 can operate on a fixed device (e.g., a desktop computer or server) and / or a mobile device (e.g., a smartphone) while connected to the e-commerce system via a local, group, or cloud-based network. One or more components of system 100 can also operate on the fixed device and / or the mobile device after connection and instructions are received by the e-commerce system 114.

[0041] FIG. 2 is a block diagram of software modules and self-organizing core processes for the e-commerce content generation and optimization system 100 for execution by the processor 104.

[0042] The software modules and self-organizing process include an objective goal module 202, a content element module 204, a normative data module 206, a max / min time reached data module 208, and a content constraint module 210. The objective goal module 202, the content element module 204, the normative data module 206, the max / min time reached data module 208, and the content constraint module 210 can provide enough structure to begin generating the SOEU 112 (FIG. 1) without requiring exhaustive, specific details or precision.

[0043] A human supervisor or artificial intelligence (AI) agent 211 can adjust content elements and content constraints before, during, or after performance of the method, or whenever it is reasonable to do so. For example, if the system and method are operating at the maximum of boundary conditions (defined by constraints) and if the impact of effects has not yet plateaued. In some embodiments, the processor 104 may provide (e.g., on a display) an indication of potential actions to be taken by the human supervisor or AI agent. Feedback or updates to assumptions or goals may also be received manually or automatically from a human processor or AI (i.e., customer reviews or trends received by social media sites).

[0044] The processor 104 may further prompt or enable the user to provide an ongoing prioritized list or queue of candidate content options. When this queue is provided to the processor 104, the processor 104 may rationally introduce new options when doing so will not adversely affect optimality. Similarly, content options may be removed when the processor 104 detects that they have little or no benefit, and a human operator may be prompted to review and remove them.

[0045] The processor 104 can also adjust for the fact that the cost of changing the content may be non-zero. The cost of changing the content may be part of the objective goal and utility measured by the processor 104, identifying a resource allocation optimization problem where cost (usually known) is balanced against perceived potential value (not yet quantified).

[0046] The objective goal module 202 receives, stores, displays, and modifies one or more conversion performance metrics that the system is to optimize. These goals range from simple metrics (e.g., sales, revenue, gross margin, cost of goods sold, etc.) to weighted combinations of multiple metrics (e.g., factoring in complex cost drivers, supply chain concerns, inventory availability, etc.) or other functional transformations. The metrics and their corresponding user-assigned weights (i.e., importance values), if specified, are combined into a multi-objective utility function. The user-assigned weights can be expressed as numbers or percentages. In some embodiments, weight values are non-negative and non-zero and may be less than, equal to, or greater than 1, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, or 99%. In other embodiments, the weight values may be numeric and may be less than, equal to, or greater than 1, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, or 99. The multi-objective utility function may be modified or refined whenever (or wherever) business goals change (i.e., aggressive market entry to maximize profits).

[0047] The content element module 204 receives, stores, displays, and modifies user-provided content selections, including the complete array of possible content combinatorial search space. Content elements are concrete instances of text, images, videos, etc. that define a service or product in technical or marketing language. Further examples of content elements include customer reviews of products obtained on an e-commerce system or other web page or site, payment for products or services, use of financial incentives (i.e., discounts), and inventory levels / management. Note that content elements can be granular to control (for example) phrases / words, image elements, etc. Content elements may be manually entered or updated via a user interface (i.e., user interface 110 of FIG. 1 ), or automatically pasted, imported, copied, or uploaded into the system from another program application or platform (e.g., MICROSOFT, LinkedIn, Pinterest, Facebook, Amazon.com, other social media sites, etc.) using natural language processing, sentiment analysis, generative adversarial networks, etc. Importantly, content elements can be updated (eg, added and / or removed) without affecting what the system has already learned.

[0048] Normative data module 206 receives, stores, modifies, and represents past or historical conversion performance metrics (corresponding to defined objective goals) that describe the performance of e-commerce products prior to implementation of the system for a group of services or products. This data may optionally be used to calibrate variations in the system and its initial decisions. The data also includes previous discoveries or inferences learned by users or the system during previous implementations. Normative data may be entered manually via a user interface (i.e., user interface 110 of FIG. 1 ) or automatically imported, copied, or uploaded into the system from another program or platform (e.g., ORACLE, MICROSOFT, TURBOTAX, SAP, etc.).

[0049] The Max / Min Time Reached Data module 208 receives, stores, modifies, and represents initial estimates of the maximum and minimum ranges over which the causal effects of content variations on actions / decisions spread and decay throughout the e-commerce system. Decay in this case refers to the amount of time between the deactivation of an experiment unit and the activation of another experiment unit. It refers to the amount of time it takes for the results of a particular content assignment to clear the system (i.e., become undetectable). There may also be a system- or user-defined period (in hours or percentages of time) between when an experiment is active and when it is inactive. This module is used to define the initial search space and generate orthogonal self-organizing experiment units.

[0050] The content constraint module 210 contains a set of content rules provided by the user or the e-commerce system that limit the overall combinatorial search space of possibilities. The content constraint module 210 receives, stores, modifies, and represents user- or system-defined constraints. These constraints include user-defined or e-commerce system-specified rules and deterministic models that define content boundaries (or limits). Constraints may be "soft," meaning the system adheres to the rules until evidence is provided that the assumptions defining the boundaries are false, or "hard," meaning the system adheres to the rules without consideration of deviations or other evidence (i.e., never violates them). Constraints include, but are not limited to, where content can be applied within the e-commerce platform (e.g., product name vs. detailed product description), constraints on multiplicity and co-occurrence (e.g., content options that cannot be used together if content can be repeated), and constraints dictated by the e-commerce platform (e.g., maximum character length for product names). Constraints can be updated during implementation as inferences are quantified to explore the impact on utility at or near the boundaries. Content elements and constraints provide opportunities for human agents to manage risk versus reward by limiting or expanding the range of options for the system.

[0051] The objective goal module 202, content element module 204, normative data module 206, max / min time reach data module 208, and content constraint module 210 are used by core algorithmic methods and processes 212 to generate content specification protocols 214 that define the real-world content to apply at any given time. The core algorithmic methods and processes 212 may be initialized by humans, other machine learning methods (e.g., to initialize correlation inference), other statistical methods (e.g., to define initial sampling probability distributions for experiment units and content elements), or a combination thereof. The core algorithmic methods and processes 212 include an experiment unit generation process 216, a treatment allocation process 218, an exploration / exploitation management process 220, a baseline monitoring process 222, a data inclusion window management process 224, and an experiment unit clustering process 226.

[0052] Based on input received from core modules 202, 204, 206, 208, and 210, experimental unit generation process 216 identifies statistically equivalent spatiotemporal units (i.e., when experimental conditions are equivalent and when unit durations are Pareto-optimal to minimize carryover effects while maximizing statistical power). An ideal experimental unit is characterized by a minimum spatial / temporal extent that prevents carryover effects from degrading the causal knowledge generated. In one embodiment, ideal experimental units can be identified by systematically exploring the spatial / temporal extent of experimental units to find an optimal unit size that corresponds to an average effect size that is within a 95% confidence interval (p=0.05) from the asymptotic mean effect over a large spatiotemporal extent. Experimental unit generation process 216 identifies interchangeable experimental units (i.e., forms clusters of interchangeable experimental units) and optimizes their spatial and temporal characteristics within each cluster by minimizing carryover effects while maximizing statistical power (i.e., the number of EUs). Examples of generating and running experimental units, selecting and using independent and dependent variables, and assigning space / time conditions are described, for example, in commonly owned U.S. Pat. No. 9,947,018 (Brooks et al.) and U.S. Patent Application Publication No. 2016 / 0350796 (Arsenault et al.).

[0053] The treatment allocation process 218 provides controlled random assignment of content elements (e.g., randomization without replacement, counterbalancing, and blocking) to experimental units with assignment frequencies that follow a uniform or predefined probability distribution (i.e., historical or normal behavior) until utility variations are detected, explored, and exploited. Within each cluster of interchangeable experimental units, the assignment of independent variable (IV) levels can follow full factorial, partial factorial, blocking, or Latin square designs, which allow for multiple blocking factors. Independent variables (IVs) are assigned so that the relative frequencies of assignments match those specified by the exploration / exploitation management process (described below). Blocking involves balancing assignments among extraneous factors (i.e., confounds), while clustering involves separating assignments by confounder. Whether blocking or clustering is chosen depends on the strength of the covariates and statistical power (i.e., starting clustering only after sufficient SOEUs have accumulated). Blocking and clustering can coexist when the number of external factors is large and both are an essential part of the "self-organization" process.

[0054] Carryover effects of content assignment within an experimental unit are dynamically and adaptively controlled. Carryover effects mean that the influence of one content assignment contaminates the measured effect of the next treatment. To eliminate carryover effects, the duration of the treatment assignment needs to match the time to reach the maximum / minimum of the effect. For example, if min=0 and max=4, the optimum may be a duration of 4 and a frequency of 1 / 8 (using the last 4 days of an 8-day period). In another example, if min=4 and max=4, the optimum may be a duration of 1 and a frequency of 1. The optimum may also depend on whether the effect is sustained (i.e., stable over time within the duration of the experiment) or transient (i.e., changing over time within the duration of the experiment).

[0055] The exploration / exploitation management process 220 analyzes the overlap between confidence intervals (CIs) through probability matching, rational choice theory, or other techniques to explore the frequency at which smaller overlaps between CIs result in more frequent use of the level associated with the highest utility. For each experimental unit, the system must decide whether to allocate experiments to make the most probabilistically optimal decision or to improve the precision of the probability estimate (i.e., CI). The system can vary the aggressiveness of the exploitation allocation and, under experimental control, find (i.e., explore) the aggressiveness that maximizes utility (including regret minimization) compared to the exploration allocation determined through baseline monitoring, where the baseline is defined as the average of all levels. The system monitors the difference between exploitation and exploration and provides an objective regret measure. Regret is the expected decrease in utility / reward from initiating an exploration process instead of optimizing with an exploitation process. When the costs (including opportunity costs) of performing treatments are heterogeneous across independent variable levels, Bonferroni-corrected confidence intervals (or inferences) are calculated such that more evidence is required to exploit more costly treatments.

[0056] The baseline monitoring process 222 continuously analyzes the baseline in real time through periodic random assignments to provide an unbiased measure of utility improvement. The baseline can be assigned depending on which metric is desired to quantify its value, and its default state may be assigned to exploration or exploitation. In addition to the assigned experimental units as described above, the system continuously determines, through statistical power analysis, the number of baseline experimental units required to monitor performance differences between these baseline trials and treatment assignments. The baseline experimental units are randomly sampled according to normative operating range data. The difference between the baseline trials and the exploration / exploitation trials provides an unbiased measure of the utility of internal parameters (including clustering, data inclusion window, and exploration / exploitation aggressiveness) and allows for objective adjustment of such parameters. The baseline trials also ensure that the entire search space defined by the constraints is explored.

[0057] The data inclusion window (DIW) management process 224 uses factorial ANOVA or other methods (i.e., normality tests) on the experimental unit duration to analyze the impact of time variation on the stability of the strength and direction of the interaction between the selected independent variables and the utility function, and therefore on the degree to which the data represents the current state of the e-commerce system for real-time decision support. For each independent variable, the data inclusion window (DIW) management process 224 identifies a Pareto-optimal data inclusion window that maximizes both the experimental power (across all experimental unit clusters and the entire decision search space) and the statistical significance of the causal effect. This allows the process to prevent overfitting the data and maintain high responsiveness to dynamic changes in the underlying system structure. Confidence intervals are calculated across the Pareto-optimal data inclusion window, resulting in a trade-off between precision (narrow confidence intervals) and accuracy as conditions change over time. The DIW may be initially user-defined based on input constraints. Generally, the system operates assuming instability (i.e., not 100% stable) and dynamically adapts.

[0058] The experimental unit clustering process 226 conditionally optimizes SOEU injection and content allocation based on external factors outside the experimental control, providing unbiased or unbiased evidence for causal interactions. Clustering is used to manage the dimensionality of the system by learning how to conditionally allocate independent variable levels based on factor interactions between experimental unit effects and attributes that cannot be manipulated by the system (e.g., seasonal or weather effects, content demand, e-commerce site placement, etc.). The dimensionality / granularity of the system (i.e., the number of clusters) is always commensurate with the amount of data available. Thus, there is no limit to the number of external factors that can or should be considered. External factors with large effects are identified and clustered first, while other factors are controlled for by blocking. The more that is known about the characteristics of the experimental units, the more effective the process will be in removing confounds and effect modifiers. Confounds are typically addressed by randomization, and effect modifiers are removed by clustering. Initial assumptions include which characteristics should be considered based on a priori knowledge or evidence that they are actually important. Assumptions can be added or removed over time as needed. Adding additional characteristics does not necessarily increase dimensionality, as they are ignored until evidence supports the need for clustering. The addition of additional characteristics is achieved by pooling experimental units into clusters where the within-cluster similarity and between-cluster differences in the effects of the independent variables on utility are maximized. The number of clusters is optimized using two related mechanisms: 1) techniques including factorial ANOVA, independence tests, and conditional inference trees are used to find factors that explain the greatest amount of between-cluster variation, and stepwise statistical power analysis is used to select the number of factors that result in clusters with sufficient statistical power to find exploitable effects; and 2) the clustering decision is brought under experimental control by continuously testing them and using baseline monitoring to objectively explore and exploit their effects on utility.

[0059] Table 1 shows how each of the core algorithmic methods and processes 212 (FIG. 2) can operate in phases once the e-commerce system is implemented. The phases are defined as initiation, exploration / exploitation, cluster initiation, and continuous cluster optimization. The initiation phase occurs as soon as the assumptions 106 (FIG. 1) are entered and defined. The system begins analyzing the data contained in the objective goals, content elements, normative data, maximum / minimum time-to-reach data, and content constraint modules (202, 204, 206, 208, and 210 in FIG. 2) to define variables and experimental unit widths. The exploration / exploitation phase iteratively evaluates the data using statistical probability matching, adjusts experimental unit durations to explore the search space definition, and determines cluster assignments. The cluster initiation phase actively analyzes one or more assigned clusters and their potential impact on the iteratively calculated confidence intervals. The continuous cluster optimization phase calculates cluster variability and identifies causal inferences between confidence intervals. [Table 1]

[0060] The point of sale business data module (POS data) 228 receives, stores, and accesses data regarding customer transactions, including payments for products or services, use of financial incentives (i.e., discounts), inventory levels, and supply chain management. Information uploaded and used in the point of sale (POS) business data module 228 can provide additional context for generating SOEUs, iterating, and identifying causal inferences. POS data can be received daily, weekly, monthly, yearly, etc., and its receipt is primarily based on the structure and requirements of the e-commerce site.

[0061] The causal knowledge module 230 systematically executes the core algorithmic methods and processes 212 (previously defined) to calculate confidence around the relative effects of different content allocations, representing the expected value of the effect and the uncertainty around that expected value for a multi-objective optimization function while minimizing confounds from external or internal factors, exploring / exploiting causal inference, and optimizing operations based on the initially defined or refined objectives. Confidence intervals are calculated in the causal knowledge module 230 for each independent variable level or dependent variable level, or for combinations of independent variable levels. Confidence intervals are calculated by taking the difference in the mean effect when a variable is activated and deactivated over the data inclusion window, which provides an estimate of the causal effect. Illustratively, in some embodiments, confidence intervals for each duration can be calculated simultaneously or sequentially if the data inclusion window satisfies a normality test (i.e., a Shapiro-Wilk test) with a maximum p-value (i.e., 0.05) for each duration. Alternatively, the duration with the greatest statistical power (or alternatively, the smallest t-test p-value) for each data inclusion window may be selected. There may be a specific data inclusion window for each variable and cluster (i.e., the data inclusion windows may all be the same or different). Process execution does not need to be continuous; it is advantageously operated independently as frequently as necessary to improve optimization capabilities. The incremental value of learning for exploitation (i.e., how many more values are probabilistically available to incorporate) is continually evaluated, including the potential effect of adding, editing, or deleting independent variables (i.e., expanding the search space). Causal inference requires 1) interchangeability between experimental units, meaning they can be exchanged at any time during the analysis and the results will not change; 2) independence between experimental units (i.e., no carryover effects); 3) consistency in treatment assignment and administration; 4) reversibility of effects; and 5) positivity in selection.

[0062] The continuous optimization module 232 invokes processes for identifying, monitoring, and improving by further refining the effectiveness of probability matching during the experiment unit clustering process 226 and the exploration / exploitation management process 220 .

[0063] Figure 3 is a flowchart of a computer-implemented method 300 for content generation and optimization, according to various embodiments. The operations of method 300 may be performed by elements of system 100 or by elements of Figure 2, and references will be made to elements within system 100 or Figure 2. The steps outlined in Figure 3 and computer-implemented method 300 may be performed simultaneously, may be performed in a different order, or may include steps not specifically identified.

[0064] Method 300 will be described with an illustrative example, in which a vendor wishes to optimize sales for two products offered on an e-commerce site, designated PR01 and PR02.

[0065] Referring to FIG. 3 and using the example scenario outlined above, a method 300 for content generation and optimization begins at operation 302, when the processor 104 (FIG. 1) receives one or more hypotheses for a randomized multivariate comparison of content provided by a vendor to the e-commerce system 114 (FIG. 1). The hypotheses include, for example, descriptive content and constraints on that content, such as those provided by the content elements 204 and content constraint module 210. The constraints include time constraints (e.g., provided by the max / min time reach data module 208), or constraints on content type, or other constraints, or a combination thereof. The hypotheses include, for example, goals of the e-commerce system 114, such as those received by the objective goal module 202.

[0066] In this example, the objective goal (managed by the objective goal module 202 (FIG. 2)) involved optimizing sales for two products and was entered into the user system 116 via the user interface 110 (FIG. 1). The content elements (managed by the content element module 204 (FIG. 2)) included, for example, product names and identified descriptive features. The normative data (managed by the historical conversion data module 206 (FIG. 2)) included reported and collected historical sales data for the two products. The max / min time-to-reach data (managed by the min / max time-to-reach data module 208 (FIG. 2)) included data regarding how quickly consumers purchased the product after being exposed to product content. For example, for PR01, 95% of consumers are likely to purchase the product within 1 to 3 days of exposure to that content on the e-commerce site. A summary of the assumptions is shown in Table 2. One constraint defined and limited the number of alphanumeric characters allowed for the descriptive features. [Table 2]

[0067] The vendor then provided content options that best conveyed or expressed information about the product name or descriptive features that were likely to generate interest and lead to sales. Example content options for two products are shown in Table 3. <Blank> indicates that no text was provided as an option or that no content option was defined. The variables (Name 1, Name 2, A1, A2, B1, B2, and C1) represent any alphanumeric text that specifies the feature (e.g., "durable," "superior performance," or "available in multiple colors"). Some feature options were similar between the two products, while others were different. For example, the option for Feature C was the same for both products, and the options for Feature A and Feature B were different. [Table 3]

[0068] Method 300 continues at operation 304, where processor 104 iteratively generates SOEUs 112 that quantify inferences between content based on one or more assumptions. In the illustrative example, SOEUs 112 comprise iteratively generating and repeating core algorithmic methods and processes 212 (FIG. 2).

[0069] The experimental unit generation process 216 (FIG. 2) assigns variables and randomizes content options to begin analyzing their effects on the e-commerce system. Table 4 shows the variable assignments based on the assumptions and content options for this example. EV represents the extraneous variables. IV represents the independent variables. RV represents the response variables (e.g., level-dependent within the independent variables) to the content assignments. [Table 4]

[0070] In some embodiments, the processor 104 can generate experiments with randomly selected durations based on a specific statistical distribution. Several factors influence or lead to the selection of a statistical distribution, generally involving a trade-off between efficiency and computational time. The statistical distribution can be uniform if prior knowledge does not indicate that one duration is better than another; it can be distributed around historical estimates, typically, or any distribution supported by a bounded or unbounded interval. Speed and accuracy are important in the analysis. Computing high-quality causal inferences can take longer. As previously mentioned, statistical distributions include uniform, Poisson, Gaussian, binomial, or any distribution supported by a bounded or unbounded interval. In this example, a uniform distribution was selected without loss of generality. Table 5 shows an example of randomized experimental units generated with double-blind randomization without replacement. The duration is defined as the length of time the experimental unit remained active within the e-commerce system, with T1, T2, and T3 representing different time intervals. The randomized experimental units created a content specification protocol 214 (Figure 2) that the e-commerce system would execute to quantify causal inference. Content probability distributions were initially based on historical data and / or constraints (if any, otherwise uniform), and over time based on what was discovered through exploration / exploitation management. Product probability distributions were also based on blocking and clustering over time. [Table 5]

[0071] The treatment assignment process 218 (FIG. 2) defined the baseline as the average of all combinations of variables and assigned a portion of the generated experimental units to the baseline. The baseline monitoring process 222 (FIG. 2) assigned the baseline to explore and continued to refine the baseline definition (i.e., exploration frequency) as the method 300 continued to operate. The evaluation was based on the SOEU definition, and initially one block and one cluster were assigned.

[0072] Method 300 continues with operation 306, where processor 104 continuously injects self-organizing experiment units (SOEUs) into electronic commerce system 114 to generate quantified inferences about the content. Processor 104 injected the experiment units by following instructions contained in content specification protocol 214 (FIG. 2). Once the experiment units were injected into the electronic commerce system, the SOEUs were started and executed. Once the experiment units were finished, the next available unexecuted experiment unit (i.e., assigned to a different block) was started. POS data 228, collected as a result of executing the SOEUs on electronic commerce system 114, is received by core algorithm process 212, and sales differentials, confidence intervals, and causal interactions are calculated by causal knowledge process 230 (FIG. 2). Method 300 continues at operation 308, where processor 104 identifies one or more confidence intervals between the injected SOEUs. Once the experimental unit is complete, the confidence intervals are iteratively calculated, representing the inferences the experiment made about the sales figures for the two products. For each SOEU, the resulting sales figures for the two products were calculated by processor 104. Table 6 shows how the two SOEUs generated a response variable representing the resulting sales figures for one of the two products (RS1 or RS2). Note: The response variable calculation is performed for all SOEUs, but is limited to only two to simplify this example. [Table 6]

[0073] The difference between the response variables of the two products at different levels was calculated. Note that in this example, only IV4 meets the requirement for a one-level difference. The difference (Δ) was calculated as |RS2 - RS1|. Most commonly, differences are calculated between adjacent levels (e.g., "on" vs. "off," or "level 1" vs. "level 2") between "similar" (i.e., interchangeable) experimental units. The difference can also be calculated as one level vs. the average of all other levels (if more than one). Confidence intervals (CIs) were then calculated for the mean and standard deviation of the sampling distribution (see Equation 1). Here, μ represents the mean and σ represents the standard deviation. A coefficient of 1.96 defined a 95% confidence interval.

number

[0074] Method 300 continues at operation 310, where processor 104 iteratively refines the SOEUs based on at least one confidence interval to identify at least one causal interaction of the contents in the system. An exploration / exploitation management process (FIG. 2) identified variation among the calculated confidence intervals to ascertain which levels have higher utility than other levels. An experimental unit clustering process 226 explored and identified variation within the confidence intervals for external variables and identified effect modifiers. A continuous optimization process 232 (FIG. 2) further refined the cluster assignments by performing statistical analysis (e.g., ANOVA) to favorably identify clusters. This was performed by aggregating differences between response variables across all levels and performing time-series evaluations. Once this clustering was performed, the calculated differences were cluster-specific and no longer representative of effects among all SOEUs.

[0075] If no relationship is found between separate SOEUs 112 and sales fluctuations (or other parameters), the above operations can continue indefinitely. However, if an underlying causal relationship exists, the processor 104 will identify causal interactions of content within the e-commerce system 114. The benefit of optimizing the duration of SOEUs 112 is that it moderates the time interval for duration effects on consumer responses and purchasing patterns. If the duration of SOEUs 112 is too short, consumer effects from SOEUs 112 will carry over after the product is switched to the next SOEU 112, violating the requirement for independent causal inference. This contaminates the attribution of sales fluctuations to product content and weakens the detection of the effect. On the other hand, if the SOEU 112 duration is too long, the effect is clear, but the system 100 wastes statistical power by failing to maximize the number of SOEUs it can run over time. Therefore, to optimize the SOEUs 112, the processor 104 adaptively modifies the duration of at least one SOEU 112 until the carryover effect of the SOEU 112 on subsequent SOEUs 112 is reduced. In some embodiments, the processor 104 can perform probability matching on the SOEU 112 durations so that the processor 104 can try longer / shorter durations and verify that the SOEU 112 durations are appropriately moderated. If the durations remain stable, the clusters will become smaller, providing continued opportunities for increasing intra-cluster homogeneity and inter-cluster heterogeneity. At this point, we determined that for each IV, each cluster, and each level pair difference (time series), we need to test for normality and modify the data inclusion window to ensure unbiased / unbiased confidence intervals that represent true causal interactions. The data inclusion window management process 224 (FIG. 2) manages and updates the normality tests. The duration (eg, T1, T2, or T2) and content variable level are updated by the processor 104 by modifying existing assumptions to obtain new SOEUs, defined as act 310 in the method 300.As SOEUs expire and are regenerated, new content specification protocols are generated by processor 104 and submitted to e-commerce system 114. Causal knowledge process 230 (FIG. 2) repeated the analysis, resulting in more accurate confidence intervals and identification of causal interactions within each cluster, effectively determining which content options had the greatest impact on sales of the two products. In addition to the above-described embodiments, the following aspects will be noted. (Appendix 1) 1. A system for optimizing business objectives of e-commerce content, comprising: Memory and a processor, coupled to the memory, receiving one or more hypotheses for a randomized multivariate comparison of content provided to users of the system; iteratively generating self-organizing experimental units (SOEUs) based on the one or more hypotheses; injecting the SOEUs into the system to generate quantified inferences about the content; identifying at least one confidence interval within the quantified inference in response to the injection of the SOEU; a processor configured to iteratively revise the SOEU based on the at least one confidence interval to identify at least one causal interaction of the e-commerce content in the system. (Appendix 2) 10. The system of claim 1, wherein the baseline monitoring determines a number of previously injected SOEUs used to identify at least one confidence interval. (Appendix 3) 10. The system of claim 1, wherein the assumptions include constraints on the content. (Appendix 4) 4. The system of claim 3, wherein the constraints include at least one temporal constraint. (Appendix 5) a user input device, wherein the processor: receiving user input including updated content; generating a subsequent SOEU based on the updated content; The system according to any one of appendices 1 to 4, further comprising: (Appendix 6) 6. The system of any one of claims 1 to 5, wherein the assumptions include an objective goal of the system. (Appendix 7) 7. The system of claim 6, wherein the objective goals include at least one of sales, profit margin, market share, or inventory management. (Appendix 8) 8. The system of claim 7, wherein the objective goal represents a weighted combination of two of sales, profit margin, or inventory management. (Appendix 9) 9. The system of any one of claims 1 to 8, wherein at least one SOEU includes a duration for which the SOEU is active within the system. (Appendix 10) 10. The system of claim 9, wherein the processor is further configured to generate a plurality of SOEUs having durations randomly selected based on a probability distribution. (Appendix 11) 11. The system of any one of claims 1 to 10, wherein the processor is further configured to adaptively modify the duration of at least one SOEU until the carryover effect of the SOEU on subsequent SOEUs is reduced. (Appendix 12) the processor: assigning one or more processes to said SOEU; Identifying distinct causal interactions based on the one or more processes; 12. The system of any one of claims 1 to 11, further configured to select optimal content for the one or more processes based on the distinct causal interactions. (Appendix 13) 13. The system of claim 12, wherein the one or more processes are assigned based on blocking, clustering, or any combination thereof. (Appendix 14) 14. The system of any one of claims 1 to 13, wherein the processor is further configured to assign at least one content option for one or more SOEUs based on utilizing variation in the calculated confidence intervals. (Appendix 15) 15. The system of claim 14, wherein the aggressiveness of the variability exploitation is determined through baseline monitoring. (Appendix 16) further comprising a user display; 16. The system of any one of claims 1 to 15, wherein the processor is further configured to provide a representation of at least one causal interaction of the content on the user display. (Appendix 17) 1. A computer-implemented method for optimizing business objectives of e-commerce content, comprising: receiving one or more hypotheses for a multivariate comparison of content, said content including content provided to users of the system; iteratively generating self-organizing experimental units (SOEUs) based on the one or more hypotheses; injecting the SOEUs into the system to generate quantified inferences about the content; identifying at least one confidence interval within the quantified inference in response to the injection of the SOEU; and iteratively revising the SOEU based on the at least one confidence interval to identify at least one causal interaction of the e-commerce content in the system. (Appendix 18) 18. The method of claim 17, wherein the baseline monitoring determines the number of previously injected SOEUs used to identify at least one confidence interval. (Appendix 19) 18. The method of claim 17, wherein the assumptions include constraints on the content. (Appendix 20) 20. The method of claim 19, wherein the constraints include at least one temporal constraint. (Appendix 21) receiving user input including updated content; generating a subsequent SOEU based on the updated content; 21. The method according to any one of appendices 17 to 20, further comprising: (Appendix 22) 22. The method of any one of claims 17 to 21, wherein the assumptions include objective goals of the system. (Appendix 23) 23. The method of claim 22, wherein the objective goal includes at least one of sales, profit margin, market share, or inventory control. (Appendix 24) 24. The method of claim 23, wherein the objective goal represents a weighted combination of two of sales, profit margin, or inventory control. (Appendix 25) 25. The method of any one of claims 17 to 24, wherein at least one SOEU includes a duration for which the SOEU is active in the system. (Appendix 26) 26. The method of claim 25, further comprising generating a plurality of SOEUs having durations randomly selected based on a probability distribution. (Appendix 27) 27. The method of any one of appendixes 17 to 26, further comprising adaptively modifying a data inclusion window of at least one SOEU until the carryover effect of said SOEU on subsequent SOEUs is reduced. (Appendix 28) assigning one or more processes to the SOEU; identifying distinct causal interactions based on the one or more processes; selecting optimal content for the one or more processes based on the distinct causal interactions; 28. The method of any one of appendices 17 to 27, further comprising: (Appendix 29) 29. The method of claim 28, wherein the one or more processes are assigned based on blocking, clustering, or any combination thereof. (Appendix 30) 30. The method of any one of claims 16 to 29, further comprising allocating at least one content option for one or more SOEUs based on utilizing variation in the calculated confidence intervals. (Appendix 31) 31. The method of claim 30, wherein the aggressiveness of variability utilization is determined through baseline monitoring. (Appendix 32) 32. The method of any one of claims 17-31, further comprising displaying a representation of at least one causal interaction of the content.

Claims

1. 1. A system having a memory and a processor coupled to the memory, the processor receiving from the memory one or more hypotheses regarding content to be provided to users of the system; the processor iteratively generates Self-Organizing Experimental Units (SOEUs) based on the one or more hypotheses; the processor injects the SOEUs into the system to generate quantified inferences about the content; The system, wherein the processor identifies at least one causal interaction of the e-commerce content based on the generated quantified inference about the content.

2. The system of claim 1 , wherein the baseline monitoring determines a number of previously injected SOEUs used to identify at least one confidence interval.

3. The system of claim 1 , wherein the assumptions include constraints on the content.

4. The system of claim 3 , wherein the constraints include at least one time constraint.

5. a user input device, wherein the processor: receiving user input including updated content; generating a subsequent SOEU based on the updated content; The system of any one of claims 1 to 4 further comprising:

6. 6. The system of claim 1, wherein the processor is further configured to adaptively modify the duration of at least one SOEU until a carryover effect of the SOEU on subsequent SOEUs is reduced.

7. the processor: assigning one or more processes to said SOEU; Identifying distinct causal interactions based on the one or more processes; The system of any one of claims 1 to 6, further configured to select optimal content for the one or more processes based on the distinct causal interactions.

8. The system of any one of claims 1 to 7, wherein the processor is further configured to assign at least one content option for one or more SOEUs based on exploiting variations in confidence intervals.

9. further comprising a user display; The system of any one of claims 1 to 8, wherein the processor is further configured to provide, on the user display, a representation of at least one causal interaction of the content.

10. 1. A computer-implemented method having a memory and a processor coupled to the memory, comprising: receiving, by the processor, from the memory, one or more hypotheses regarding content to be provided to users of the system; the processor iteratively generating self-organizing experimental units (SOEUs) based on the one or more hypotheses; the processor injecting the SOEUs into the system to generate quantified inferences about the content; the processor identifying at least one causal interaction of the e-commerce content based on the generated quantified inference about the content.

11. 11. The method of claim 10, wherein the baseline monitoring determines a number of previously injected SOEUs used to identify at least one confidence interval.

12. The method of claim 10 , wherein the assumptions include constraints on the content.

13. The method of claim 12 , wherein the constraints include at least one time constraint.

14. receiving user input including updated content; generating a subsequent SOEU based on the updated content; The method of any one of claims 10 to 13, further comprising:

15. The method of any one of claims 10 to 14, further comprising the step of assigning at least one content option for one or more SOEUs based on exploiting the variation in the confidence intervals.

16. The method of any one of claims 10 to 15, further comprising displaying a representation of at least one causal interaction of said content.

Images