Multidimensional testing for marketing analytics

Abstract

The systems and methods of the present invention provide automated scheduling, monitoring, and evaluation of repeated, alternating sets of A / B tests arranged in a multidimensional pattern (hypercube). This provides a robust system and method for computationally repeated A / B tests while allowing for easy tuning of statistical power and other parameters. The statistical power of the systems and methods can be increased by increasing the number of A / B test repetitions over time. Additionally, the systems and methods allow for axial (hypercube) evaluation of tests that can compare the impact of creative content on different variables.

Description

[Background technology]

[0001] A / B testing is a testing method to test the effect of a certain variable on a certain indicator. For example, you can test whether (A) playing an audio ad (variable) leads to an increase in sales (indicator) compared to (B) not playing an ad. Classical A / B testing only considers a single variable, while A / B / n testing or multivariate testing considers many variables and is particularly concerned with efficient testing regimes.

[0002] A / B testing requires that the underlying populations on which variable A or variable B is tested are comparable, otherwise the test results may be skewed. For example, if an advertisement is run in store A and not in store B, but store B generally sells more of the product, the benefit of the advertisement may go undetected because store A's sales remain lower than store B's despite the advertisement.

[0003] Instead of testing A and B in two different stores, one could record weekly sales in a single store and compare the results of one week with advertising (A) with one week without advertising (B). However, this remains problematic, as general trends in sales (declining or increasing over time) can skew the test results in the same way as a store split would.

[0004] Thus, traditionally, A / B testing is conducted on a large number of randomly selected users, stores, or generally "subjects", to mitigate potential distortions of the test results due to uneven populations in the A and B split, and the statistical significance of the test is based on the difference between the population mean and their standard deviation.

[0005] The reference to prior art in this specification is not an acknowledgement or indication that the prior art forms part of the common general knowledge. Summary of the Invention

[0006] The system and method of the present invention provides for automated scheduling, monitoring, and evaluation of repeated, alternating sets of A / B tests arranged in a multidimensional pattern (hypercube). This provides a robust system and method for computationally repeated A / B testing while allowing for easy adjustment of statistical power and other parameters. The number of iterations of the system and A / B tests can be increased over time to increase the statistical power of the method. Additionally, the system and method allows for axis-wise (hypercube) evaluation of tests that can compare the impact of creative content on different variables.

[0007] According to the present invention, by employing a repeated alternating pattern of A / B testing, the impact of a creative (such as an audio or visual advertisement or piece of music) on a business metric (such as sales volume, sales variance, revenue, basket size, customer numbers or customer retention) can be measured with a given confidence for a small number of different variations of a variable (e.g., store or location).

[0008] The systems and methods of the present invention are more robust to confounding external factors such as weather, holidays, or other events that affect the selected metrics, compared to traditional A / B testing on a small number of variants.

[0009] The statistical power of the testing system of the present invention can be easily and immediately increased by extending the test time (increasing the number of A / B test iterations). This is in contrast to the conventional approach of increasing statistical power by increasing the number of stores participating in the test. However, running an A / B test in more stores is usually more expensive than running an A / B test for longer, especially if the A / B test requires the installation of necessary hardware, such as an audio system.

[0010] Throughout this specification (including the claims which follow), unless the context otherwise requires, the word "comprise" and variations such as "comprises" and "comprising" are understood to imply the inclusion of stated integers or steps or groups of integers or steps but not the exclusion of other integers or steps or groups of integers or steps. [Brief description of the drawings]

[0011] [Figure 1] Figure 1 is a simple diagram to explain A / B testing. [Diagram 2] FIG. 2 is a simple diagram showing how the probabilities in the diagram of FIG. 1 are determined. [Diagram 3] Figure 3 is a simple diagram showing an A / B test over four time periods, say four weeks. [Figure 4] Figure 4 is a simple table showing an A / B test for four variables (e.g., location / store) over five periods, e.g., five weeks. [Diagram 5] Figure 5 is a diagram showing the effect of A / B test combinations across three variables (in this case, product, store, and week). [Figure 6] FIG. 6 is a diagram illustrating components of an exemplary system in accordance with the present invention. [Figure 7] FIG. 7 is a flow chart illustrating an exemplary process flow of an exemplary system in accordance with the present invention. [Figure 8] FIG. 8 illustrates an exemplary report generated in accordance with one embodiment of the present invention. [Figure 9] FIG. 9 illustrates an exemplary checkerboard pattern report generated in accordance with one embodiment of the present invention. [Figure 10] FIG. 10 shows an example 90% confidence interval for a product across multiple stores created in accordance with one embodiment of the present invention testing sales volume induced by a given advertisement. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] It is convenient to describe the invention herein with reference to a particularly preferred embodiment. It should be understood, however, that the invention is applicable to a wide variety of embodiments, and that other configurations and arrangements are contemplated within the scope of the invention. Also, various modifications, alterations, variations, or additions to the configurations and arrangements described herein are deemed to be within the scope of the invention.

[0013] The present invention provides a computer-implemented marketing analytics system including: a networked server that renders an interface, the interface including a plurality of selectable variables, a plurality of selectable items of creative content, and a plurality of selectable indicators; a test generator that receives one or more inputs from the server related to the selectable variables, the selectable creative content, and / or the selectable indicators and builds an n-dimensional data structure; each selectable geographic location including an audio and / or visual device and an information gathering device; a scheduler that receives the data structure and retrieves data related to the selected creative content from a database, the scheduler in communication with the audio and / or visual devices to control the audio and / or visual devices to present the creative content in accordance with the data structure; and a tracker that receives information from the information gathering devices and records indicators of interest over time in a results database.

[0014] The present invention also provides a computer-implemented marketing analytics system including: a networked server that renders an interface, the interface including a plurality of selectable variables including one or more of time, geographic location, and product, a plurality of selectable items of creative content, and a plurality of selectable indicators; a test generator that receives one or more inputs from the server related to the selectable variables, the selectable creative content, and / or the selectable indicators and builds an n-dimensional data structure; each selectable geographic location including an audio and / or visual device and an information gathering device; a scheduler that receives the data structure and retrieves data related to the selected creative content from a database, the scheduler in communication with the audio and / or visual devices to control the audio and / or visual devices to present the creative content in accordance with the data structure; and a tracker that receives information from the information gathering devices and records indicators of interest over time in a results database.

[0015] The information collection device may be, for example, one or more of a POS management system that monitors POS indicators, sensors that measure one or more indicators that may be proxy indicators (i.e., information that indicates the value of another indicator when the other indicator is more difficult to measure directly).

[0016] The present invention also relates to a computer implemented marketing analytics system, comprising: a web server that renders a web interface, the web interface including a geographic location, one or more products, a plurality of selectable variables, a plurality of selectable items of creative content, and a plurality of selectable indicators; a test generator that receives one or more inputs from the web server related to the selectable variables, the selectable creative content, and / or the selectable indicators and builds an n-dimensional data structure; each selectable geographic location including audio and / or visual devices, a POS system that monitors the POS, and optionally one or more sensors that measure one or more proxy indicators such as noise in a store or foot traffic; a scheduler that receives the data structure and retrieves data related to the selected creative content from a database, the scheduler in communication with the audio and / or visual devices to control the audio and / or visual devices to present the creative content in accordance with the data structure; a tracker that receives the POS data (and optional sensor data) and records indicators of interest over time in a results database; A marketing analysis system is provided, the system including: an evaluator service that retrieves the results stored in the result database and the A / B test schedule specified by the data structure, and calculates a report; and a reporter that makes the report available to a user.

[0017] The present invention also relates to a computer implemented marketing analytics system, comprising: a web server that renders a web interface, the web interface including a plurality of selectable variables including one or more of time, geographic location, and product, a plurality of selectable items of creative content including one or more of audio advertisements, visual advertisements, and music, and a plurality of selectable metrics including one or more of sales volume, sales variance, revenue, basket size, number of customers, and customer retention; a test generator that receives from the web server one or more inputs related to the selectable variables, the selectable creative content, and / or the selectable metrics, and builds an n-dimensional data structure consisting of a hypercube of repeated, alternating A / B tests to execute; A / B testing refers to playing or not playing selected creative content at selected times and / or locations for a given product, where the axes of the hypercube are user selected variables such as store, time period (e.g., week, etc.) and / or product, and the cells of the hypercube contain a list of advertisements to play or one or more boolean flags of True or False indicating whether to play / display an item of creative content for a particular week, store and / or product; each selectable location including an audio and / or visual device, a POS system that monitors one or more of unit sales, sales variance, sales revenue, basket size, customer numbers, or customer retention, and optionally one or more sensors that measure one or more proxy metrics such as in-store noise or foot traffic; a scheduler that receives the hypercube data and retrieves data related to the selected creative content from a database of music and audio advertisements, the scheduler communicating with the audio and / or visual device to control the audio and / or visual device to present the creative content in accordance with the hypercube data;If a cell for a particular time, store, and product contains the value true then the creative is played / displayed, otherwise it is not played / displayed; a tracker that receives POS data (and optional sensor data) and records metrics of interest over time in a results database; an evaluator service that takes the results stored in the results database and the A / B test schedule specified by the hypercube and calculates reports; and a reporter that makes the reports available to users via one or more of a web interface, email, hardcopy, or SMS.

[0018] The present invention is also a method of marketing analysis, including receiving one or more inputs from a user via a web interface regarding one or more selectable variables, selectable items of creative content, and selectable metrics; constructing an n-dimensional data structure including a hypercube of repeated, alternating A / B tests to perform an A / B test; performing the A / B test by presenting the creative content according to the data structure; receiving, recording, and tracking data related to one or more metrics associated with the data structure; and generating a report for presentation to a user.

[0019] The present invention is also a computer-implemented method of marketing analytics comprising: rendering an interface from a networked server, the interface including a plurality of selectable variables including one or more of time, geographic location, and product, a plurality of selectable items of creative content, and a plurality of selectable indicators; receiving one or more inputs from the server associated with the selectable variables, the selectable creative content, and / or the selectable indicators and constructing an n-dimensional data structure; each selectable geographic location including an audio and / or visual device and an information gathering device; receiving the data structure and retrieving data related to the selected creative content from a database, the scheduler communicating with the audio and / or visual devices to control the audio and / or visual devices to present the creative content in accordance with the data structure; and receiving information from the information gathering devices and recording indicators of interest over time in a results database.

[0020] The marketing analysis system of the present invention may be used in a variety of contexts, such as to improve understanding of a selected marketing and sales environment, resulting in a better understanding of various social and psychological factors, e.g., to improve sales and profitability.

[0021] The server may render any type of interface suitable for a user to receive and input information relevant to the analysis at hand. The server must be able to communicate with other elements of the system, for example over a network. In some preferred embodiments, the network is a global communications network such as the Internet. In some preferred embodiments, the server is a web server and the interface is a web interface.

[0022] The variables selectable through the interface may be of any type suitable for marketing analysis: for example, they may relate to cost, price, time, location, geographic location, product, product category, product positioning, target audience, people, packaging, etc.

[0023] The items of creative content may be of any type suitable for influencing marketing results. In some preferred embodiments, they include audio or visual marketing or advertising or music. They may also include other treatments of the environment, such as, for example, music, video entertainment, pre-recorded sounds, smells (e.g., to stimulate hunger), tactile offerings, etc. Creative content may be stored and deployed remotely or at the geographic location of interest. For example, audio or video content may be stored electronically on a server and transmitted to the location on demand, or may be stored locally and played when instructed. Other sensory content that cannot be transmitted digitally (such as smell or tactile content) may be stored locally but triggered to be displayed remotely.

[0024] The metrics may be of any suitable type that provides information to marketing analysis in accordance with the present invention, for example, they may include sales, sales volume, sales growth, sales variance, revenue, profits, leads, customer value, customer lifetime value, return on investment, customer acquisition cost, sales team response time, traffic at a location, net promoter score, basket size, spend per customer or type of customer, number of customers, customer retention rate, etc.

[0025] The test generator receives one or more inputs from the server related to selectable variables, selectable creative content, and / or selectable metrics, and builds an n-dimensional data structure consisting of a hypercube of iterative, alternating A / B tests to run, where the A / B test is to play or not play selected content at a selected time and / or location for a given product, and the axes of the hypercube are the user selected variables as described above. In some embodiments, the cells of the hypercube contain one or more of a list of content options to play or display, or a Boolean flag, e.g., True or False, indicating whether an item of content will be played / displayed for a particular time, store and / or product, etc.

[0026] Each of the geographic locations is equipped with a device for displaying or playing the selected content. The device may be of any suitable type for playing or displaying the content of interest, such as, for example, an audio speaker, a video display, a smell generator, or a tactile display.

[0027] The information collection device may be of any type suitable for collecting information meaningful for marketing analysis. For example, it may include a POS system (to collect sales indicators as described above) or one or more sensors for collecting locally sensed data, for example as proxy indicators. The sensors may be of any type suitable for collecting information useful for marketing analysis that may be used, for example, as proxy indicators. These may include, for example, microphones for sensing noise and foot traffic in the store, vibration sensors for sensing foot traffic, video cameras for analyzing video footage, counting store patrons and analyzing their behavior.

[0028] The scheduler receives the hypercube data and retrieves data related to the selected content from the content database. The scheduler communicates with the content device to control that device to present the content according to the hypercube data. The scheduler can communicate with the content device in any manner suitable to facilitate the present invention. It can be wireless or physical, it can be a network such as a global communication network or a local area network, or it can be directly via a wireless protocol method such as Bluetooth, Zigbee, etc.

[0029] In some preferred embodiments, if the cells of the cube for a particular time, store, and / or product contain the value "True", the content is played / displayed, otherwise it is not played / displayed.

[0030] The tracker receives data (such as POS data or sensor data) and records indicators of interest over time in a results database. The results database can be stored anywhere that enables the invention to work. For example, it can be on the server, on another server, or at a selected location.

[0031] The evaluator service takes the results stored in the results database and the A / B test schedule specified by the hypercube and computes a report, which may be of any type and may be presented in any manner suitable to provide the user with useful information about the analysis.

[0032] The reporter provides the report to the user, which may be done in any manner suitable for making the report and its information available to the user, for example, via one or more of a computer interface, email, SMS, hard copy, voice message, etc.

[0033] In the following description, certain preferred embodiments are set forth.

[0034] Referring to Figure 1, a fundamental problem with A / B testing is that confounding factors can easily skew the test results. For example, consider an A / B test that compares the average sales volume of a product in week 1 (w1) and week 2 (w2) when an audio ad is played (B) and when no ad is played (A).

[0035] As depicted in Figure 2, there are essentially two possibilities: either sales increase in week 2 or they do not. If we ignore the case where sales are exactly the same in both weeks, which is a negligible probability for mass-produced products, the probability of the two outcomes is 50% as the formula shows. TIFF2024538805000002.tif2694

[0036] As a result, if the impact of advertising on weekly sales is low compared to many other confounding factors such as weather, seasons, holidays, covid, other events that change sales, e.g. sporting events such as football matches, etc., the test results are largely random and it is nearly impossible to detect an effect of advertising.

[0037] In accordance with the present invention, this problem can be overcome by the systems and methods of the present invention, which allow for carefully constructed and repeated A / B testing over multiple periods of time (such as several weeks) using an alternating A / B testing pattern, such as that shown in FIG. 3, which shows a four-week testing pattern.

[0038] In this case, the probability of observing such a pattern by chance is only 12.5, compared to 50% for a single A / B test, making it much more likely to detect the effect of the advertising campaign as shown by the formula. TIFF2024538805000003.tif25107

[0039] Note that by increasing the number of iterations, you can significantly reduce the probability of observing an alternating pattern by chance. This is beneficial because it is relatively easy to extend the duration of a test, whereas adding additional store, product, or other version variables to improve statistical power is time-consuming and costly (which may require installing additional hardware such as audio systems and / or adding additional creative).

[0040] This novel approach of alternating A / B testing can be extended to multiple variations, resulting in, for example, a two-dimensional alternating grid (checkerboard) for time periods (e.g., weeks) and locations (e.g., stores) as shown in FIG. 4, or a three-dimensional alternating cube for time periods (e.g., weeks), locations (e.g., stores), and products as shown in FIG. 5, or in general an n-dimensional hypercube.

[0041] It should be noted that it is also considered within the scope of the present invention to repeat A / B tests, but in a randomized, non-repeated pattern. Such embodiments may be computationally more complex and difficult to interpret in some circumstances. In some embodiments, this can be improved by, for example, performing a balancing process to ensure that the number of each of the A and B alternatives is equal.

[0042] A diagram illustrating components of an exemplary system according to the present invention for iterative A / B testing of audio ads is shown in Figure 6. In this example, a web interface (601) running on a web server displays a user interface where a user (e.g., a store manager) selects which variables (610), such as week, store, product, etc., to run an A / B test on. The user further selects creative content (602), such as audio ads and / or ad displays and / or music, to play from a database of available creatives (615), and metrics (603) to evaluate, such as sales volume, sales variance, revenue, basket size, number of customers, or customer retention.

[0043] The web server sends the selected variables (e.g., as a list) to a test generator (612), which is a service that may, but need not, run on a computer system separate from the web server. The test generator configures an n-dimensional data structure, or hypercube (613), for performing repeated, alternating A / B tests of playing or not playing selected creative content (e.g., music), showing or not showing electronic displays, and / or playing or not playing audio ads, in a particular store, at a given time, and for a given product. The axes of the hypercube are user-selected variables, such as store, week, and / or product. The cells of the hypercube are Boolean flags (True or False) that indicate whether or not a creative will be played / displayed for a particular week, store, and / or product. Alternatively, for example, the cells could also contain a list of ads to play.

[0044] The hypercube data is fed to a scheduler (614), which retrieves selected creative content from a database of music, display text, and audio ads (615). (Note that the stores and dates are parts (axes) of the hypercube, which is itself generated by a test generator that receives this information via a web server / web interface.) The scheduler controls the in-store audio equipment (e.g., Qbit) or displays (e.g., video screens or electronic boards) (616) via an Internet connection (e.g., via a Rest API) to play creative content at the scheduled week / time, store, and / or product specified in the hypercube's axes and cells. Specifically, in one implementation, if the cell for a particular week, store, and / or product contains the value "true," the creative is played / displayed, otherwise it is not played / displayed. The creative content may be transmitted electronically to the store's audio / display devices, for example, or the content may be preloaded onto devices installed in the store, for example.

[0045] The brick-and-mortar store (617) may be equipped with audio equipment (616) (players, speakers, controllers, e.g., Qsic Qbit devices) for playing audio content, point-of-sale (POS) systems (618) for monitoring key information such as sales volume, sales variance, revenue, basket size, customer numbers or customer retention, and additional sensors (619) for measuring other proxy metrics such as in-store noise (AVA) and foot traffic (XY-Sense). Sensors may also provide proxy measurements of the metrics for which they were originally intended. For example, ambient noise detection signals (AVA) can be correlated with in-store occupancy, and XY-Sense can estimate foot traffic.

[0046] POS (Point of Sale) (618) and sensor (619) data is sent to a Tracker (620), which records the metrics of interest over time in a results database (611), for example a SQL database running on a computer system, while a Scheduler (614), again a service running on a computer, executes the A / B tests specified by the hypercube (613).

[0047] The Evaluator Service (621) reads the results stored in the Results Database (611) and the A / B test schedule specified by the hypercube (613). It computes a report (604) that is made available to the user, which can be displayed to the user through a web interface or sent through electronic means such as email or SMS to a mobile device, for example.

[0048] In this exemplary embodiment, the report shows the statistical significance (confidence interval) of the impact of the creative content (ad campaign) on the selected metrics for selected products and / or stores (Figure 10). In certain scenarios, e.g., when point-of-sale data (POS) is provided, the evaluator can estimate, for example, the increase in revenue associated with or caused by the creative.

[0049] Additionally, the report can also include statistical significance along different axes of the hypercube, allowing for independent assessment of the impact of advertising and / or music on a store or product (Figure 10).

[0050] FIG. 7 is a flow chart showing an exemplary process flow of the system according to the present invention. A user selects parameters such as variables (701), creative content (702), and metrics (703) on a web interface (711). A test generator (712) then creates a hypercube (713) that specifies a schedule of A / B tests to be executed by a scheduler (714). The scheduler (714) runs the A / B tests until the test is completed. In each loop, the scheduled creative content (702) is played on the in-store audio system (716) and the selected metrics (703) are measured by the tracker (710), for example, based on either POS (718) or sensor (719) data, or optionally both. Once the test is completed (the scheduled A / B test is completed), an evaluator (720) generates a report (704), which can be displayed to the user via the web interface (701) as described above.

[0051] 8 shows an example of a report generated in accordance with one embodiment of the present invention. The report shows daily sales (dashed line) of muffins (a product) due to an A / B test of audio ads scheduled over a four week period. The area labeled "Ads Active" indicates the weeks the ads were played. The average weekly sales (bold line) show an increase when the ads were played.

[0052] Figure 9 shows the checkerboard pattern for a set of products (milk, sandwich, muffin, slurpee, hot pastry, newspaper, hot drink, loaf) and weeks (weeks 32-37). The grey scale indicates the normalized sales volume of those products.

[0053] Figure 10 shows 90% confidence intervals for products across multiple stores. They include the average increase in revenue or sales volume due to advertising, and the lower and upper limits of the confidence interval describe the worst-case and best-case scenarios for expected advertising revenue. These confidence intervals can be provided as part of the report.

Claims

1. A marketing analysis system implemented on a computer, comprising: A network-connected server that renders an interface, the interface comprising: - A plurality of selectable variables; - A plurality of selectable creative content items; - A plurality of selectable metrics A server; A test generator that receives from the server one or more inputs related to the plurality of selectable variables, the plurality of selectable creative content, and / or the plurality of selectable metrics, and constructs an n-dimensional data structure; Each selectable geographical location, comprising an audio and / or visual device and an information collection device; A scheduler that receives the data structure, retrieves data related to the selected creative content from a database, and communicates with the audio and / or visual device to control the audio and / or visual device to present the creative content according to the data structure; A tracker that receives information from the information collection device and records metrics of interest in a results database over time; A marketing analysis system.

2. A marketing analysis system implemented on a computer, comprising: A network-connected server that renders an interface, the interface comprising: - A plurality of selectable variables including one or more of time, geographical location, and product; - A plurality of selectable creative content items; - A plurality of selectable metrics A server; A test generator that receives from the server one or more inputs related to the plurality of selectable variables, the plurality of selectable creative content, and / or the plurality of selectable metrics, and constructs an n-dimensional data structure; Each selectable geographical location, comprising an audio and / or visual device and an information collection device; A scheduler that receives the data structure and retrieves data related to the selected creative content from a database, and communicates with the audio and / or visual device to control the audio and / or visual device to present the creative content according to the data structure. A tracker that receives information from the information collection device and records the metrics of interest in the result database over time. A marketing analysis system including the above.

3. The marketing analysis system according to claim 1, wherein the information collection device is any one or more of a POS system that monitors POS metrics and sensors that measure one or more proxy metrics.

4. A marketing analysis system implemented on a computer, A web server that renders a web interface, and the web interface includes - One or more selectable variables including time, geographical location, and product, - Items of a plurality of selectable creative content, - A plurality of selectable metrics A web server including the above. A test generator that receives from the web server one or more inputs related to the plurality of selectable variables, the plurality of selectable creative content, and / or the plurality of selectable metrics, and constructs an n-dimensional data structure. Each selectable geographical location includes - An audio and / or visual device, - A POS system that monitors POS, - One or more optional sensors that measure one or more proxy metrics such as in-store noise and foot traffic Each selectable geographical location including the above. A scheduler that receives the data structure and retrieves data related to the selected creative content from a database, and communicates with the audio and / or visual device to control the audio and / or visual device to present the creative content according to the data structure. A tracker that receives POS data (and optional sensor data) and records the metrics of interest in the result database over time. An evaluator service that retrieves the results stored in the result database and the A / B test schedule specified by the data structure, and calculates a report, A reporter that makes the report available to the user, A marketing analysis system including the above.

5. A marketing analysis system implemented on a computer, A web server that renders a web interface, where the web interface includes A plurality of selectable variables including one or more of time, geographical location, and product, A plurality of selectable creative content items including one or more of audio advertisements, visual advertisements, and music, A plurality of selectable metrics including one or more of sales volume, sales variation, sales revenue, basket size, number of customers, and customer retention rate, A web server including the above, A test generator that receives one or more inputs related to the plurality of selectable variables, the plurality of selectable creative content, and / or the plurality of selectable metrics from the web server and constructs an n-dimensional data structure consisting of a hypercube of repeated alternating A / B tests for execution, An A / B test is to play or not play the selected creative content at the selected time and / or location for a given product. The axes of the hypercube are variables selected by the user, such as store, week, and / or product. The cells of the hypercube include a list of advertisements to play or one or more Boolean flags of True or False indicating whether to play / display an item of creative content for a particular week, store, and / or product. Each selectable location includes ・ An audio and / or visual device, ・ A POS system that monitors one or more of the number of sales, sales variation, sales revenue, basket size, number of customers, or customer retention rate, ・ One or more optional sensors that measure one or more proxy metrics such as in-store noise and foot traffic, Each selectable location including the above A scheduler that receives data of the hypercube and obtains data related to the selected creative content from databases of music and audio advertisements, and communicates with the audio and / or visual device to control the audio and / or visual device to present the creative content according to the data of the hypercube. If the cell of a specific time, store, and product contains the value true, the creative is played / displayed; otherwise, it is not. A tracker that receives POS data (and optional sensor data) and records the indicators of interest in the result database over time. An evaluator service that obtains the results stored in the result database and the A / B test schedule specified by the hypercube, and calculates a report. A reporter that makes the report available to the user via one or more of a web interface, email, or SMS. A marketing analysis system including the above. Receiving from a user, via a web interface, one or more inputs regarding one or more selectable variables, selectable items of creative content, and selectable indicators. Constructing an n-dimensional data structure including a hypercube of repeated alternating A / B tests to perform an A / B test. Performing an A / B test by presenting the creative content according to the data structure. Receiving, recording, and tracking data related to one or more indicators related to the data structure. Generating a report for presentation to the user. A marketing analysis method including the above. Rendering an interface from a network-connected server, where the interface includes: - One or more selectable variables including one or more of time, geographical location, and product. - Items of a plurality of selectable creative content. - A plurality of selectable indicators and receiving, from the server, one or more inputs associated with the one or more selectable variables, the plurality of selectable creative content, and / or the plurality of selectable indicators, and constructing an n-dimensional data structure. ​ ​ ​ Each selectable geographical location includes an audio and / or visual device and an information collection device, receives the data structure, obtains data related to the selected creative content from a database, and the scheduler communicates with the audio and / or visual device to control the audio and / or visual device to present the creative content according to the data structure, receives information from the information collection device, and records an indicator of interest in a result database over time, A computer-implemented marketing analysis method comprising:

8. The marketing analysis method according to claim 7, wherein the information collection device is any one or more of a POS system that monitors POS indicators and sensors that measure one or more proxy indicators.

9. The marketing analysis system according to any one of claims 1 to 5, wherein the creative content is either electronically transmitted to an audio / display device at the selectable location or, for example, pre-loaded onto a device at the location.

10. The marketing analysis method according to any one of claims 6 to 8, wherein the creative content is either electronically transmitted to an audio / display device at the selectable location or, for example, pre-loaded onto a device at the location.

Images