Automatic determination of parameter values

Abstract

To automatically determine parameter values that control or affect provision of content by a content platform.SOLUTION: In one aspect, evaluation points are identified for a parameter. Each evaluation point includes an evaluated parameter value of the parameter and a metric value of a metric corresponding to the provision of digital components by a content platform. A first model is generated using a set of evaluation points. A second model is generated based on the first model and an acquisition function that is based on mean values and confidence intervals of the first model and a configurable exploration weight that controls a priority of exploration for evaluating the parameter. A next parameter value to evaluate is determined from the second model and the content platform is configured to use the next parameter value to provide digital components.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates generally to data processing and automatically determining parameter values ​​that control or affect the provision of content by a content platform. [Background technology]

[0002] A video streamed to a user may include one or more digital components that are typically overlaid on top of the original video stream. The overlaid content may be presented to the user in a rectangular region that overlays a portion of the original video screen. The digital components may also include in-stream content that is played before, during, or after the original video stream. The presentation of the video and digital components may be controlled by a content platform (e.g., a video system) based on parameter values ​​of various parameters, such as the time interval between presentation of multiple digital components presented with the video, the frequency or likelihood of selecting different types of digital components, or other types of parameters.

[0003] As used throughout this document, the phrase "digital component" refers to a discrete unit of digital content or information (e.g., a video clip, an audio clip, a multimedia clip, an image, text, or another unit of content). A digital component can be stored electronically in a physical memory device as a single file or in a group of files, and can take the form of a video file, an audio file, a multimedia file, an image file, or a text file. For example, a digital component can be content that is intended to supplement the content of a video or other resource. More specifically, a digital component can include digital content related to the resource content (e.g., the digital component can relate to the same or otherwise related topic / content relating to the video). The provision of a digital component can thus supplement, and generally enhance, the web page or application content. Summary of the Invention [Means for solving the problem]

[0004] In general, one innovative aspect of the subject matter described herein includes operations of performing a plurality of iterations to identify a parameter value for a parameter based on which a content platform controls provision of a digital component with video content, where performing each iteration in the plurality of iterations includes identifying a set of evaluation points for the parameter, where each evaluation point includes an evaluation parameter value for the parameter and a metric value for a metric corresponding to provision of the digital component by the content platform, where the metric value for the evaluation point is determined from data generated by the content platform using the evaluation parameter value for the evaluation point to provide the digital component; generating a first model using the set of evaluation points; generating a mean and confidence interval for the first model; and generating a second model based on the first model and an acquisition function, where the acquisition function includes a mean value for the first model, a confidence interval for the first model, and a metric value for the second model. The present invention may be embodied in a method including: determining a next parameter value to be evaluated from the second model based on a confidence interval and a configurable search weight that controls the priority of exploration to evaluate the parameters; configuring the content platform to use the next parameter value to provide the digital component with the video content; and determining a next metric value based on data resulting from the content platform using the next parameter value to provide the digital component; determining a particular parameter value from among the parameter values ​​and corresponding metric values ​​of the parameters determined during the multiple iterations that either results in a highest metric value or satisfies a particular threshold; and configuring the content platform to use the particular parameter value to control or select the digital component to be provided with the video content during production. Other embodiments of this aspect include corresponding methods, apparatus, and computer programs encoded on a computer storage device and configured to perform the operations of the method. These and other embodiments may each optionally include one or more of the following features.

[0005] Certain embodiments of the subject matter described herein can be implemented to achieve one or more of the following advantages: The techniques described herein provide a parameter tuning system that can automatically tune parameters of another system, such as, for example, a content platform. A content platform may have, for example, hundreds or even thousands of parameters (e.g., parameters that control time intervals between presentations of digital components, parameters that represent the possibility of selecting a certain type of digital component). Manual tuning of each parameter may not be feasible. Thus, automatic tuning of parameters by a parameter tuning system can enable tuning of a larger number of parameters than is possible with manual tuning techniques. A content platform may have access to limited resources, such as data storage, processor time, administrator time, real-time experiment data, and network bandwidth, to name a few. Limited resources mean that the content platform can perform a limited number of experiments to determine the effects (e.g., metric values) that may result from providing content while the content platform is configured with a particular parameter value. Thus, a brute force approach of evaluating every possible parameter value is not feasible. Hundreds of parameters each have multiple parameter value options, and a combinatorial explosion may occur that would require the evaluation of an infeasible number of parameter value options in a brute force approach. That is, the content platform's limited resources for experimentation will be exhausted before all possible parameter values ​​have been evaluated. Other approaches, such as random evaluation of parameter values, can be inefficient because parameter values ​​that do not result in the desired metric value may continue to be selected for evaluation despite not being good parameter value candidates. For example, random selection of parameter values ​​may result in parameter values ​​that are close together being selected, but waste resources because evaluating similar parameters generally produces little value with respect to determining better parameter values.As used herein, a better parameter value is a parameter value that leads to a better metric value being derived from data generated by the content platform while the parameter value is configured compared to a worse metric value being derived from data generated by the content platform while configured with a different parameter value. A better metric value is a metric value (compared to other metric values) that is more preferred or more desirable by the provider of the content platform and / or an entity using the content platform. A better metric value may correspond to a goal of the content platform or an entity using the content platform. Entity goals are described in more detail below. A better metric value can be a value that is achieved when the content platform is configured with a particular parameter value compared to a worse metric value that is achieved when the content platform is configured with a different parameter value.

[0006] In contrast, the parameter tuning system described herein can select parameter values ​​to be evaluated by automatically and iteratively selecting parameter values ​​based on an acquisition function that identifies, for example, the parameter values ​​with the highest predicted metric values ​​(relative to metric values ​​corresponding to other parameter values) and / or the parameter values ​​with the highest potential in terms of being included in a yet unexplored parameter value space (relative to other parameter values). The selection process used by the parameter tuning system can lead to better parameter values ​​being selected for evaluation (and implementation) more quickly than can be achieved from brute force or random methods. By selecting better parameter values ​​more quickly, less experimentation needs to be done to determine better parameter values, thus reducing resources consumed by the content platform from experimenting to evaluate selected parameters. In this manner, selection of parameter values ​​by the parameter tuning system can lead to faster configuration of the content platform with better parameter values ​​and improved resource efficiency (relative to other methods, such as brute force or random methods).

[0007] In addition to saving resources through less experimentation, once a content platform has been configured using parameter values ​​determined, e.g., by a parameter tuning system, using the techniques described above (described further throughout this specification), configuring the content platform with better parameter values ​​can also lead to resource reductions or savings for the content platform. For example, example metrics may include a video abandonment rate indicating the rate at which users abandon a video and a view-through rate indicating the rate at which users watch a digital component presented with the video. The parameter tuning system can automatically determine new parameter values ​​for one or more parameters (e.g., a time interval parameter for a digital component) using the selection process described above (described further throughout this specification) that, once configured into the content platform, leads to the generation of better video abandonment rate and view-through rate metrics. For some metrics, better metric values ​​correlate to more efficient utilization. For example, for video abandonment rate and digital component view-through rate metrics, resource expenditures by the content platform may be more efficient after the content platform is configured with new parameter values ​​selected / determined (e.g., by a parameter tuning system) to lead to a better video abandonment rate or a better digital component view-through rate. In this example, efficient resource expenditure is achieved because the video content and digital components provided are more likely to be consumed compared to the previous period when the new parameter values ​​were configured. In other words, the processing cycles, network bandwidth and other resources expended in providing the video content and digital components are more resource efficient because the likelihood that a client device will receive the video content and digital components without a user consuming the video content and digital components is reduced.

[0008] As another example, the parameter tuning system, once configured into a content platform, can select better parameter values ​​that lead to the generation of better resource-specific metric values, such as for a latency-specific metric, such as the latency of providing video content or digital components to a user device. For example, the parameter tuning system can determine parameter values ​​that, if implemented, would lead to a lower latency-specific metric value as compared to a period prior to implementation of the parameter value. As another example, the parameter tuning system, once configured into a content platform, can select better parameter values ​​that lead to the generation of a certain metric value, such as a number of total digital component interactions in a particular period that can be achieved by providing fewer digital components as compared to the same number of total digital component interactions previously achieved in a previous period in which the content platform was configured using the previous parameter value. For example, the parameter tuning system, once configured into a content platform, can select parameter values ​​that lead to the selection by the content platform of digital components that are interacted with at a higher rate as compared to digital components selected by the content platform in a previous period in which the content platform was configured with other parameter values. As another example, the parameter tuning system, once configured into a content platform, can select better parameter values ​​that lead to better metric values ​​for various metrics (e.g., the total number of digital component interactions in a period of time) using the same amount of resources previously consumed in a previous period in which the content platform was configured using the previous parameter values.

[0009] The parameter tuning platform can automatically determine other parameter values ​​that lead to improved resource utilization compared to other parameter values ​​not automatically determined by the parameter tuning platform. For example, the parameter tuning platform can determine parameter values ​​for various parameters that can limit the number of digital components distributed by the content platform to a given user. Limiting the distribution of digital components by the content platform for various reasons can lead to, for example, less use of processor resources and less utilization of network resources. Example parameters whose values ​​the parameter tuning platform can automatically determine and that the content platform can use to limit the number of digital components distributed include 1) a frequency cap parameter that limits the total number of digital components that can be distributed by the content platform to each user in a particular time period, and 2) a maximum number of repetitions parameter that limits the number of times the same digital component can be distributed by the content platform to the same user in a particular time period.

[0010] As another example, the parameter tuning system can automatically determine values ​​for various parameters that a content platform can use to control the number of digital components that participate in a digital component selection auction conducted by the content platform. Limiting the number of digital components that participate in an auction conducted by the content platform can conserve processing resources compared to limiting the number of auction participants without using parameters.Example parameters for which the parameter tuning platform can automatically determine values ​​and that the content platform can use to limit the number of digital components participating in a content auction include: 1) various parameters each controlling a maximum number of digital components of a particular format (e.g., skippable format, non-skippable format) that can participate in a content selection auction conducted by the content platform; 2) various parameters each controlling a maximum number of video digital components having a particular video length range (e.g., 0-1 minute, 1-5 minutes, 5-10 minutes, over 10 minutes) that can participate in a digital component selection auction conducted by the content platform (e.g., the content platform can use these parameters to allow fewer videos of longer lengths and more videos of shorter lengths to participate in the auction); 3) various parameters each controlling a maximum number of video digital components having a particular video length range (e.g., 0-1 minute, 1-5 minutes, 5-10 minutes, over 10 minutes) that can participate in a digital component selection auction conducted by the content platform (e.g., the content platform can use these parameters to allow fewer videos of longer lengths and more videos of shorter lengths to participate in the auction); the content platform may use these parameters to allow a smaller number of auction participants to select digital components for users in a first country compared to a larger number of auction participants to select digital components for users in a second country); and 3) various parameters that each control a maximum number of video digital components having a particular creative quality (e.g., high quality, medium quality, low quality as determined by the content platform's quality analyzer) that may participate in the digital component selection auction conducted by the content platform (e.g., the content platform may use these parameters to allow fewer videos of lower quality and more videos of higher quality to participate in the auction).

[0011] The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram of an example environment for automatically determining parameter values. [Figure 2A] 13 is an example of a graph in which evaluation points are plotted against parameters. [Figure 2B] 1 is an example graph illustrating a function of truth versus a parameter; [Figure 3A] 1 is a graph illustrating mean values ​​and confidence intervals. [Figure 3B] 1 is a graph illustrating an acquisition function. [Figure 4] 1 illustrates example pseudo-code for automatically determining parameter values. [Diagram 5] FIG. 1 is a flow diagram of an example process for automatically determining parameter values. [Figure 6] FIG. 1 is a block diagram of an example computer system that can be used to perform the described operations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] As summarized below and described throughout the specification, the techniques described herein enable automated determination of parameter values ​​that control or affect the provision of content by a content platform. In some implementations, the techniques described herein provide a parameter tuning system that can determine parameter values ​​to be provided to a content platform (e.g., for the provision of content by the content platform) or some other type of system. The content platform parameters can, for example, control or affect the provision of content by the content platform. The parameter tuning system can evaluate experimental results from experiments in which different parameter values ​​are configured for one or more content platform parameters and run on the content platform. The parameter tuning system can iteratively select new parameter values ​​to evaluate based on the experimental results. As discussed above, iterative selection of parameters by the parameter tuning system can lead to better parameter values ​​being selected more quickly than can be achieved using other approaches. A better parameter value is a parameter value that leads to the generation of a better metric value from data generated by the content platform when the content platform is configured using the better parameter value compared to a worse metric value derived from data generated by the content platform when the content platform was configured using the previous parameter value.

[0014] More specifically, during a current iteration of evaluating content platform parameters, the parameter tuning system can identify a set of evaluation points for parameters corresponding to a previous experiment, each evaluation point including an evaluation parameter value and a metric value, the metric value being determined from data generated when the content platform uses the evaluation parameter values ​​to provide the digital component during the previous experiment.

[0015] For example, an example parameter may be a time interval amount between presentations of a digital component during a video viewing session. Example parameter values ​​for this parameter may be 10 seconds, 30 seconds, etc. An example metric is a video abandonment rate indicating how often a user abandons a video viewing session. The parameter tuning system (or experimentation system) may conduct a first experiment on the time interval amount parameter using a parameter value of 30 seconds. The content platform may track user interaction data including start and stop times of the video viewing session during the first experiment. The parameter tuning system (or content platform) may generate a video abandonment rate metric from the user interaction data generated during the first experiment. For example, an example video abandonment rate metric value that may be generated from the user interaction data from the first experiment may be 10%. That is, the first experiment may indicate that when the time interval amount parameter is 30 seconds, 10 percent of users abandon the video early (i.e., before reaching the end of the video). Thus, a rating score for the time interval amount parameter for the first experiment includes a parameter value of 30 seconds and a metric value of 10%. The parameter tuning system may perform a second experiment, for example, arriving at another evaluation point for the time interval quantity parameter with a parameter value of 10 seconds and a metric value of 15%. That is, the second experiment may show that a time interval parameter of 10 seconds causes 15 percent of users to abandon the video early.

[0016] After the parameter tuning system identifies a set of evaluation points for parameters for a previous experiment, the parameter tuning system can generate a first model using the set of evaluation points. For example, the parameter tuning system can generate a first model, such as a Gaussian model, that fits the evaluation points. The parameter tuning system can generate a mean and a confidence interval for the first model. The parameter tuning system can then generate a second model based on the first model using an acquisition function based on the mean of the first model, the confidence interval of the first model, and a configurable search weight that controls the priority of the search to evaluate the parameters. The search is described in more detail below.

[0017] The parameter tuning system can determine at least one next parameter value to evaluate from the second model. For example, the parameter tuning system can determine the next parameter value that leads to the highest gain function value generated from the second model (or, in some cases, the parameter tuning system can select a predetermined number of parameter values ​​having the highest gain function value). Once the parameter tuning system selects one or more parameter values ​​to evaluate, the parameter tuning system can configure the content platform to use the next parameter value when providing the digital component during the next experiment. The parameter tuning system (or the experimentation system) can perform the next experiment. The parameter tuning system can determine the next metric value based on data resulting from the content platform using the next parameter value to provide the digital component during the next experiment.

[0018] The parameter tuning system can evaluate the next metric value to determine whether the next metric value is an improved metric value compared to a previous metric value corresponding to a previous experiment conducted when the content platform was configured using the previously evaluated parameter value. In some cases, the parameter tuning system can determine to select the evaluated parameter value evaluated in the next experiment for non-experimental (e.g., production) use in the content platform. For example, the parameter tuning system can determine that the next metric value is greater than a predetermined threshold (e.g., the predetermined threshold can be a desired or target metric value). As another example, the parameter tuning system can be configured to perform a predetermined number of experiments, and the most recent experiment may have been the last experiment of the predetermined number of experiments. After the parameter tuning system determines to select the evaluation parameter value for non-experimental use, the content platform can be configured to use the evaluation parameter value to control or select, during production, a digital component to be provided along with video content provided by the content platform.

[0019] As another example, the parameter tuning system may decide to perform another experiment. For example, the parameter tuning system may determine that the next metric value is less than or equal to a predetermined threshold, or the parameter tuning system may determine that at least one more experiment should be performed before the parameter tuning system performs a predetermined number of experiments. In such a case, the parameter tuning system may create an updated set of rating points by adding additional rating points to the set of existing rating points. Each added rating point includes a parameter value evaluated during the last experiment and a corresponding metric value derived from data generated by the content platform during the last experiment. The updated set of rating points may be used for new experiments (e.g., to create a first model), as described above.

[0020] During each experimental iteration, the parameter tuning system can configure an exploration weight to control the priority of exploration during the current experiment. Prioritizing exploration can result in selecting a next parameter value to evaluate that has a higher confidence interval, which can correspond to a parameter value range that has not been explored as compared to other parameter value ranges. The exploration approach can be compared to an exploitation approach. The exploitation approach can correspond to continuing to explore parameter value ranges that have high predicted metric values ​​compared to other parameter value ranges. Prioritizing exploitation can be achieved by decreasing the exploration weight, which can result in selecting a next parameter value to evaluate that has a higher average value, for example. In some cases, the parameter tuning system prioritizes exploration in early experiments and exploitation in later experiments. For example, because the number of evaluation points increases as the number of iterations increases, the first model may represent a more accurate fit of the current set of evaluation points in the later iterations, and thus a more accurate prediction of the metric value can be generated using the first model in the later iterations compared to the previous iteration. Because more accurate predictions of the metric value are made in the later iterations, the parameter tuning system can prioritize exploitation in the later iterations. Additionally, the parameter tuning system may prioritize exploration in early iterations since there is a larger range of unexplored parameter values ​​in early iterations compared to later iterations. The parameter tuning system may use other techniques to balance the exploration / exploitation tradeoff. Although exploration weights are described, in some implementations the exploration / exploitation tradeoff is achieved by similarly or additionally configuring exploitation weights. These features, as well as additional features and benefits, are further described in more detail below with reference to Figures 1-7.

[0021] With respect to the descriptions throughout this document, a user may be provided with controls that allow the user to select whether and when the systems, programs, or features described herein may enable collection of user information (e.g., information about the user's social network, social actions or activities, occupation, user preferences, or the user's current location), as well as whether the user is sent content or communications from the server. In addition, some data may be processed in one or more ways before it is stored or used, such that personally identifiable information is removed. For example, the user's identity may be processed such that personally identifiable information cannot be determined for the user, or the user's geographic location may be generalized to the extent that location information is available (such as to the city, zip code, or state level) such that the user's specific location cannot be determined. In this way, the user may have controls over what information is collected about the user, how that information is used, and what information is provided to the user.

[0022] FIG. 1 is a block diagram of an example environment 100 for automatically determining parameter values. The example environment 100 includes a network 104. The network 104 may include a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. The network 104 may also include any type of wired and / or wireless network, satellite network, cable network, Wi-Fi network, mobile communication network (e.g., 3G, 4G, etc.), or any combination thereof. The network 104 may utilize communication protocols including packet-based and / or datagram-based protocols such as Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), or other types of protocols. The network 104 may further include several devices that facilitate network communication and / or form a hardware base for the network, such as switches, routers, gateways, access points, firewalls, base stations, repeaters, or combinations thereof.

[0023] The network 104 connects the client devices 102, the content platforms 106, the content providers 108, and the parameter tuning system 110. The example environment 100 can include many different content platforms 106, client devices 102, and content providers 108.

[0024] A content platform 106 is a computing platform (such as a network server or another data processing device described with reference to FIG. 7) that enables distribution of content. Example content platforms 106 include search engines, social media platforms, video sharing platforms, news platforms, data aggregator platforms, or other content sharing platforms. Each content platform 106 may be operated by a content platform service provider. Each of the components of the content platform 106 is a software component that includes instructions that are executed by a processing entity, such as a processor.

[0025] The content platform 106 can publish and make available its own content. For example, the content platform 106 can be a news platform and publish its own news articles. The content platform 106 may display content (e.g., digital components) provided by one or more content providers 108 that are not part of the content platform 106. In the above example, the news platform may display third-party content provided by one or more content providers 108. As another example, the content platform 106 may not publish its own content, but rather be a data aggregator platform that aggregates and displays third-party content provided by different content providers 108.

[0026] In some implementations, the content platform 106 may store certain information about the client device (e.g., device preference information, content consumption information, etc.). Such user information may be used by the content platform to, for example, tailor content provided to the client device 102 or to enable immediate access to particular content frequently accessed by the client device 102. In some implementations, the content platform 106 may not store such device information on the platform, however, the content platform 106 may still provide such information for storage on a particular server (separate from the content platform). The content platform 106 (also referred to herein as the content platform / server 106 or simply a server) thus refers to the content platform that stores such device information or the server (separate from the content platform) that stores such device information.

[0027] In some implementations, the content platform 106 is a video service that allows users to watch streamed video content. The video streamed to the user may include one or more digital components (e.g., provided by the content provider 108) that are overlaid on top of the original video stream. For example, to provide the digital components to the viewer of the video stream and to improve the amount of content delivered within the viewing range for a given video streaming bandwidth, it would generally be desirable to provide the overlaid content on the underlying video stream. In addition to or in the alternative to a video streaming scenario, the content platform 106 may include a video processor that processes the video file to modify it to include the overlaid content, and the processed video file is provided to the client device 102 with the overlaid content for display on the client device 102. As another example and as an alternative to overlaid content, the content platform 106 may provide a digital component (rather than overlaid on top of the video stream) that is presented to the user at scheduled breakpoints in the video stream.

[0028] The content platform 106 may maintain a user activity log 112 that includes anonymized information corresponding to user activity with respect to content (e.g., digital components) provided by the content platform 106 and content provided by the content providers 108. For example, for video content, the user activity log 112 may include information regarding start and stop times of video viewing sessions, presentation of the digital components with the video content, user interactions with the presented digital components, and other user activity information.

[0029] The metric generator 114 of the content platform 106 can generate various types of metrics from the data in the user activity log 112. The metrics generated by the metric generator 114 can be stored in the metric database 116. Example metrics can include metrics related to latency or responsiveness, as described in more detail below. Other example metrics generated by the metric generator 114 can include click-through rates indicating the rate of interaction with a digital component, view-through rates indicating the percentage of users who watch a particular video digital component to completion, abandonment rates indicating the percentage of users who abandon a video content item, revenue metrics indicating revenue derived from presenting a digital component, return on investment metrics, number of content and digital component presentations, and other types of metrics.

[0030] Some metrics generated by the metric generator 114 may correspond to particular goals of a user or entity using the content platform 106 and / or the user's or entity's satisfaction with the content platform 106. For example, the content platform 106 and the content providers 108 may receive revenue from the presentation of digital components with content provided by the content platform 106. Thus, metrics such as revenue metrics, number of digital components presented, etc. may be of interest to and correspond to the goals of the content platform 106 or content provider 108. The content providers 108 providing the digital components may have other goals, such as return on investment, and therefore may be interested in other metrics, such as interaction rates with the presented digital components.

[0031] As another example, some metrics generated by the metric generator 114 may correspond to user satisfaction with the content platform 106 by users of client devices 102 consuming content provided by the content platform 106. For example, some metrics may correspond to user perceptions of content provided by the content platform 106 (and / or user perceptions of the content platform 106 itself). For example, for video content, a video abandonment metric for a video presented in addition to a digital component may correlate to user perceptions of the presented digital components, the number and spacing of the presented digital components, etc. Thus, trade-offs may occur between different goals for different entities. For example, a video creator may want to receive more revenue by having more digital components scheduled for presentation with their video, but presenting more digital components may decrease user satisfaction with the video and cause the user to abandon the video early (thus displacing the playback of some of the scheduled digital components). Additionally, presenting more digital components may not necessarily increase the long-term return on investment for the provider of the digital components, as presenting more digital components may decrease the interaction rate for each digital component, either because the user becomes overwhelmed or confused by too many digital components, or because of an overall decrease in the relevance of the digital components to the presented resource content. Other metrics that may correspond to user satisfaction or user goals may include overall system latency or responsiveness. For example, as latency for playback of content or digital components increases, user satisfaction with the content platform 106 generally decreases.

[0032] In some implementations, the metric generator 114 can generate a composite metric that combines multiple different metrics. For example, the composite metric can be a combination of different metrics, each representing a goal of a different entity that uses the content platform 106. For example, the composite metric can be based on a combination of sub-metrics including a user acceptance metric (e.g., video abandonment rate), a revenue-related metric (e.g., the number of digital components presented with the video), and a digital component interaction rate metric. The metric generator 114 can generate the composite metric in some cases by scaling (e.g., multiplying) each sub-metric value by a corresponding sub-metric weight (e.g., a value ranging between 0 and 1) and then aggregating (e.g., adding) the scaled sub-metric values.

[0033] The provision of content by the content platform 106 can be controlled by various parameters 118. The parameters 118 can be configured by a parameter setter 120, which can be an automated process and / or can include a user interface component for receiving parameter values ​​from an administrator. The parameters 118 can include, for example, one or more parameters that can be used by the content platform 106 to control the time interval between presentation of multiple digital components presented with a video. Other parameters can be used by the content platform 106 to control the frequency or likelihood of selecting different types of digital components (e.g., digital components having a certain type of content) for a certain type of video. In general, the content platform 106 can include hundreds of different types of parameters.

[0034] The parameter setter 120 can set a given parameter to a particular parameter value, which can affect the provision of content by the content platform 106, as described above. The provision of content by the content platform 106 according to various parameter values ​​can affect user activity and therefore, in turn, the above-mentioned goal-related metrics generated by the metric generator 114. For example, if the time interval parameter is reduced by the parameter setter 120 from 30 seconds to 10 seconds (e.g., so that a new digital component is displayed every 10 seconds while a user is watching a video), short-term revenues may increase for the video creator due to the increased number of presentations of digital components, but users may become annoyed from, for example, repeated interruptions to video content playback by presentations of digital components, and as a result, spend less time viewing content on the content platform 106, thereby decreasing long-term revenues for the content platform 106 and the content provider 108.

[0035] Another example parameter for the content platform 106 is a user cost penalty parameter whose value represents, for example, the magnitude of long-term revenue cost reduction that may occur if the digital component is selected for playback at a point in a video by the content platform 106. The user cost penalty parameter value can be used by the content platform 106 in the auction used to select the digital component for playback with the video. The user cost penalty parameter value can be used by the content platform 106 to balance other factors, such as expected interaction with the digital component, expected short-term revenue, etc. An administrator of the content platform may not initially know what value to assign to the user cost penalty parameter. As described in more detail below, the parameter tuning system 110 can be used to automatically determine the value to use for the user cost penalty parameter (as well as other parameters).

[0036] Although the parameter setter 120 may change parameter values ​​that may ultimately have an impact on various metrics, the administrator of the content platform 106 may not know which parameter values ​​to specify for a parameter to achieve a respective metric value (relative to other determined metric values ​​for the parameter or relative to a predefined threshold). For example, there may be an unknown landscape for parameters for a given metric. The landscape for parameters for a metric may be complex and may represent an unknown function with high degrees of freedom and nonlinearity, so that it may not be possible to directly calculate a predicted metric value for any given parameter value without knowing the function. Thus, there may be no direct way for the parameter setter 120 to calculate parameter values ​​that achieve a better metric value relative to other determined metric values ​​for the parameter or relative to a predefined threshold.

[0037] The content platform 106 can learn the impact on the metrics in response to the parameter setter 120 changing the parameter values ​​by using the experimentation system 122. The experimentation system 122 can configure various experiments 124. The experiments 124 can include information identifying one or more values ​​to be evaluated during the experiment for one or more evaluation parameters. For example, a current value for a time interval parameter controlling the interval between digital components may be 30 seconds. The experiment 124 can specify that a different parameter value, such as 10 seconds, should be evaluated during the experiment. The content platform 106 can use the experimentation system 122 to split live traffic between productions (e.g., non-experimental) and experimental traffic 126. For example, the content platform 106 can use the experimentation system 122 such that a first portion (e.g., 99%) of the content requests are handled by the content platform 106 according to the current (e.g., production) parameter values, and a second portion (e.g., 1%) of the content requests are handled by the content platform 106 as experimental traffic 126 according to the information identified in the experiment 124. The parameter setter 120 can, for example, set parameter values ​​during the experiment 124 for the second portion of the content request according to the experiment 124 information.

[0038] The parameter setter 120's modification of parameter values ​​during an experiment can affect the content platform 106's provision of content (e.g., the video as a whole or digital components provided with / in the video). For example, more or less of a certain type of digital component may be selected by the content platform 106, or a different number of digital components may be selected by the content platform 106. The user of the client device 102 may react to the affected provision of content by the content platform 106. For example, the user may interact with the digital components more or less, or in a different manner (e.g., by clicking / selecting or spending more time watching). As another example, the user may, for example, interact differently with the video content with which the digital components are presented. For example, a user receiving content according to an experiment may tend to abandon the video content less or more frequently, or may abandon the video content closer or further from the end of the video. The content platform 106 can track and store these and other types of user interactions during the experiment as experiment activity 128.

[0039] The metric generator 114 can generate experiment metrics 130 from the experimental activity 128. The experiment metrics 130 can include information indicating which parameter values ​​were used for the experiment 124. The experiment system 122 can compare the experimental metrics 130 generated by the metric generator 114 from the experimental activity 128 to corresponding non-experimental metrics in the metric database 116 generated by the metric generator 114 from user activity from non-experimental traffic to determine whether at least some of the experimental metrics 130 are better than the corresponding non-experimental metrics. If the experimental metrics 130 are better than the corresponding non-experimental metrics, the parameter setter 120 can determine the parameter values ​​used during the experiment and set the corresponding parameters to those parameter values ​​for future (e.g., non-experimental) content offerings by the content platform 106.

[0040] As mentioned above, conducting experiments with the experimentation system 122 can be expensive in terms of resources. The resources available to the content platform 106 for experimentation (e.g., processing time, data storage, administrator time, network bandwidth, real-time content requests for the experiment) may be limited. Therefore, a brute force approach of trying all possible parameter values ​​for every parameter is not feasible. Other approaches, such as randomly selecting parameter values, are generally ineffective because parameter values ​​that are poor choices for achieving the desired metric value may continue to be selected for evaluation.

[0041] As an alternative to brute force, random or other techniques for selecting parameter values, parameter tuning system 110 can automatically determine parameter values ​​by performing a parameter tuning process on the parameters. Each of the components of parameter tuning system 110 is a software component that includes instructions that are executed by a processing entity, such as a processor. Although parameter tuning system 110 is described as tuning parameters for content platform 106, parameter tuning system 110 can tune parameters for other types of systems. Although illustrated as separate from content platform 106, in some implementations some or all of the components of parameter tuning system 110 may be included in content platform 106.

[0042] Driver 131 of parameter tuning system 110 may be used to control the parameter tuning process. For example, an administrator may use driver 131 to start a parameter tuning process or to stop a parameter tuning process. As another example, driver 131 may be used to perform automatic parameter tuning on a parameter. For example, driver 131 may perform a parameter tuning process on a parameter automatically and / or periodically (e.g., weekly, monthly).

[0043] In response to a request or decision to initiate a parameter tuning process, driver 131 can invoke parameter value selector 132. Parameter value selector 132 can perform multiple parameter tuning iterations when tuning parameters. During each parameter tuning iteration, parameter value selector 132 can automatically select one or more next parameter values ​​to evaluate in a next experiment based on data from past experiments. As described in more detail below, parameter value selector 132 can be configured to perform a predetermined number of parameter tuning iterations during the parameter tuning process, or parameter value selector 132 can stop the parameter tuning process in response to determining that an experimental metric value generated from the experimental data for the evaluated parameter value satisfies (e.g., meets or exceeds) a threshold metric value.

[0044] In more detail, the parameter value selector 132 can use as input evaluation scores 134 corresponding to past experiments. As described in more detail below, the initial set of evaluation scores 134 can correspond to experiments performed by the experimentation system 122 using random parameter values. The other evaluation scores 134 used by the parameter value selector 132 can correspond to experiments previously performed by the experimentation system 122 using parameter values ​​previously selected by the parameter value selector 132 in earlier iterations of the parameter tuning process. Each evaluation score 134 includes an experiment metric value from the previous experiment and parameter values ​​used by the content platform 106 to provide the content during the previous experiment. The evaluator 135 can receive or access the experiment data (e.g., experiment metrics 130) generated by the metric generator 114 from the content platform 106, for example, to generate the evaluation scores 134. Evaluation scores are illustrated and described in more detail below with respect to FIG. 2A.

[0045] To generate the initial evaluation points, the parameter value selector 132 can randomly generate a predetermined number of random parameter values ​​for the parameters. For each random parameter value, the driver 131 can use the experiment file editor 136 to create an experiment file 138 that includes the random parameter values ​​to be evaluated. The parameter tuning system 110 can provide the experiment file 138 to the experiment system 122. The experiment system 122 can use the experiment file 138 to conduct a random parameter value experiment during which content is provided by the content platform 106 according to the random parameter values ​​(e.g., as experiment traffic 126). As described above, during the random parameter value experiment, the parameter setter 120 can set the parameters to the random parameter values, and the content platform 106 can track the experiment activity 128 that occurs in response to the experiment traffic 126. The metric generator 114 can generate experiment metrics 130 based on the experiment activity 128, and the experiment metrics 130 can be provided by the content platform 106 to the evaluator 135. The evaluator 135 can set initial evaluation points 134 for the parameters using the random parameter values ​​and the received metrics generated from the random parameter value experiments.

[0046] During a parameter tuning iteration, the parameter value selector 132 obtains a current set of evaluation points 134 (e.g., initial evaluation points for the first iteration or evaluation points for a subsequent iteration after the first iteration). The parameter value selector 132 uses the obtained evaluation points 134 to generate a first model 140. The first model 140 can be a probabilistic model of the evaluation points 134 that fits the evaluation points 134. The first model 140 can be, for example, a Gaussian model that can generate predictions for metric values ​​for unevaluated parameter values. The parameter value selector 132 can determine a mean 142 and a confidence interval 144 from the first model 140. The confidence interval can correspond to, for example, a standard deviation value. The first model 140, the mean 142, and the confidence interval 144 are described in more detail below with respect to FIG. 3A and FIG. 4.

[0047] After parameter value selector 132 generates first model 140, parameter value selector 132 can generate second model 146 using mean values ​​142 and confidence intervals 144 derived from first model 140. For example, second model 146 can be based on an acquisition function based on a combination of mean values ​​142, confidence intervals 144, and search weights that weight confidence intervals 144. While first model 140 represents predictive metric values ​​for parameter values ​​based on fitting existing evaluation points, second model 146 can represent potentials for unexplored parameter values.

[0048] The parameter value selector 132 can assign larger search weights in early parameter tuning iterations and smaller search weights in later parameter tuning iterations. The parameter value selector 132 can determine the next parameter or parameters to evaluate based on the second model 146. The parameter value selector 132 can, for example, select the parameter value whose acquisition function produces the highest acquisition function value. The second model 146, the acquisition function, and exploration versus exploitation are described in more detail below with respect to FIGs. 3B and 4.

[0049] For each parameter value selected by the parameter value selector 132, the driver 131 can use the experiment file editor 136 to create an experiment file 138 that includes the selected parameter values. The parameter tuning system 110 can provide the experiment file 138 to the experiment system 122, which can then conduct an experiment in which content is provided by the content platform 106 (e.g., as the experiment traffic 126) according to the selected parameter values. As described above, the parameter setter 120 can set the parameters to the selected parameter values, and the content platform 106 can track the experiment activity 128 that occurs in response to the experiment traffic 126. The metric generator 114 can generate experiment metrics 130 based on the experiment activity 128, and the experiment metrics 130 can be provided to the evaluator 135. The experiment metrics 130 can include the goal-related metrics described above. As described above, the goal-related metrics can be composite metrics that represent a combination of different goals of different entities using the content platform 106.

[0050] In some implementations, the evaluator 135 determines whether the received experimental metric for the selected parameter value satisfies a threshold metric value. For example, the parameter tuning process can be configured to stop when the metric value generated by the metric generator 114 from the experiment performed using the selected parameter value satisfies (e.g., meets or exceeds) a threshold metric value (e.g., a predetermined, good or desired metric value). If the received experimental metric for the selected parameter value satisfies the threshold metric value, the parameter tuning system 110 can determine that the parameter tuning process is complete for the parameter. On the other hand, if the selected parameter value does not satisfy (e.g., is smaller than) the threshold metric value, the parameter tuning process can continue to be iterated through additional parameter tuning iterations. As another example, the parameter tuning system 110 can determine that the parameter tuning process is complete for a parameter when a predetermined number of parameter tuning iterations have been performed for the parameter.

[0051] In response to determining that the parameter tuning process is complete for a parameter, the parameter tuning system 110 can instruct the content platform 106 to use the evaluated parameter value previously evaluated by the parameter value selector 132 for the non-experimental traffic. The parameter value selector 132 can, for example, select the previously evaluated parameter value that led to the best-evaluated metric value. For example, the parameter setter 120 can set the parameter to the selected previously evaluated parameter value, and the content platform 106 can provide the content in response to a subsequent request for the content according to the parameter value. For example, the content platform 106 can use the selected previously evaluated parameter value to control or select digital components to be provided along with the video content during production.

[0052] If the parameter tuning system 110 determines that a stopping point has not been reached for the parameter tuning process for the parameter, the evaluator 135 can generate a new evaluation score 134 for the parameter using the parameter value selected by the parameter value selector 132 in the previous iteration and the corresponding metric value generated by the metric generator 114 from experiments conducted using the parameter value selected by the parameter value selector 132 in the previous iteration to generate an updated evaluation score 134 (e.g., the new evaluation score can be added to the existing set of evaluation scores 134). The parameter value selector 132 can use the updated evaluation score 134 for the parameter to perform the next parameter tuning iteration for the parameter.

[0053] Parameter tuning system 110 may perform different parameter tuning processes for different parameters. Each parameter tuning process for a parameter performed by parameter tuning system 110 may involve tuning the parameter with respect to a given metric. Parameter tuning system 110 may perform different parameter tuning processes for different metrics. Parameter tuning system 110 may perform different parameter tuning processes in parallel or sequentially. Additional structural and operational aspects of these components of parameter tuning system 110 and the parameter tuning processes are described below with reference to FIGS. 2A-6.

[0054] 2A is an example graph 200 in which evaluation points are plotted against a parameter. The X-axis 202 corresponds to the value of the parameter. The parameter may be a setting that can take real values ​​between 0 and 1, for example. The Y-axis 204 corresponds to the value of a metric generated by the metric generator 114, for example, when the content platform 106 provided content based on a particular parameter value for the parameter during an experiment. The evaluation points 206, 208, 210, 212, and 214 are observation points that each include an evaluation parameter value and a corresponding metric value. For example, as exemplified by the evaluation points 206, 208, 210, 212, and 214, when the parameter had values ​​of 0.8, 0.36, 0.4, 0.78, and 0.95, the corresponding metric values ​​generated by the metric generator 114 were 0.28, 0.55, 0.44, 0.61, and 0.44, respectively. Scores 206, 208, 210, 212 and 214 may be included, for example, in scores 134 described above with respect to FIG.

[0055] Evaluation point 212, having a parameter value of 0.78 and a metric value of 0.61, corresponds to the highest metric value among evaluation points 206, 208, 210, 212, and 214. Although the parameter value of 0.78 at evaluation point 212 may be the best parameter value for the parameter among the observation points, other, yet to be evaluated, parameter values ​​may be configured for the parameter by parameter setter 120, which may lead to a better corresponding metric value being generated by metric generator 114. However, as noted above, the landscape of parameter values, or the "true function" for the parameter, may not be known by parameter tuning system 110 after observing only the current observation point. The true function for the parameter may be a function that reflects the actual metric values ​​for various possible parameter values. The true function (if known) may be able to output a metric value given a particular parameter value for the parameter. Since parameter tuning system 110 does not know the true function, parameter tuning system 110 may perform a parameter tuning process for the parameter.

[0056] 2B is an example graph 250 illustrating a true function for a parameter. X-axis 252, Y-axis 254, and evaluation points 256, 258, 260, 262, and 264 correspond to X-axis 202, Y-axis 204, and evaluation points 206, 208, 210, 212, and 214, respectively, in FIG. 2A. A true function line 266 illustrates the true function, or landscape of metric values ​​given different parameter values. As previously mentioned, the parameter tuning system 110 does not know the true function. Rather, the parameter tuning system 110 currently knows evaluation points 256, 258, 260, 262, and 264 (which correspond to evaluation points 206, 208, 210, 212, and 214 in FIG. 2A). As mentioned above, evaluation point 262 (corresponding to evaluation point 212 in FIG. 2A ) has the highest observed metric value (e.g., 0.61), but an unevaluated parameter value of 0.23, as illustrated by point 268 on the true function line 266, configured by parameter setter 120 would lead to a higher metric value (e.g., 0.80) being generated by metric generator 114. Parameter value selector 132 is configured to perform a parameter tuning process to find parameter values ​​better than those of the observed evaluation point, such as the parameter value of 0.23 at point 268.

[0057] Figure 3A is a graph 300 illustrating means and confidence intervals. X-axis 302, Y-axis 304 and evaluation points 306, 308, 310, 312 and 314 correspond to X-axis 202, Y-axis 204 and evaluation points 206, 208, 210, 212 and 214, respectively, in Figure 2A. True function line 316 corresponds to true function line 266 in Figure 2B.

[0058] As described above with respect to FIG. 1, the parameter value selector 132 can generate a first model using the evaluation points 306, 308, 310, 312, and 314. The first model (which can be the first model 140) can be, for example, a Gaussian model that fits the evaluation points 306, 308, 310, 312, and 314. The first model is illustrated on the graph 300 as a Gaussian line 318. The Gaussian model can include observed points (e.g., evaluation points 306, 308, 310, 312, and 314) and predictions (e.g., mean value 142) for points between the observed points. For example, the predicted point 315 can include a parameter value of 0.27 and a predicted metric value of 0.64.

[0059] In some implementations, the first model is an aggregate model that combines multiple other models, where each of the multiple other models uses a different approach to fit the evaluation points 306, 308, 310, 312, and 314. When the first model is an aggregate model, the first model can include a median mean value determined as the average predicted metric value of the multiple other models.

[0060] Shaded regions on graph 300, such as shaded region 320, represent confidence intervals (e.g., confidence interval 144) that parameter value selector 132 calculates from the first model. As illustrated in graph 300, the confidence intervals at and near evaluation points 306, 308, 310, 312, and 314 are low because evaluation points 306, 308, 310, 312, and 314 are known or observed points rather than corresponding to predicted metric values. Larger confidence intervals exist for other portions of Gaussian line 318, such as within ranges of parameter values ​​that have fewer observed points and therefore are less explored than other parameter value ranges. For example, vertical lines 322, 324, 326, and 328 illustrate larger confidence intervals than other ranges of graph 300.

[0061] As discussed above, parameter value selector 132 can select the next parameter value to be evaluated based on a second model (e.g., second model 146) that uses a combination of the mean of the first model and the confidence interval of the first model. Parameter value selector 132 can be used, for example, to generate an acquisition function to generate the second model, as illustrated below with respect to FIG. 3B. As described in more detail below with respect to FIG. 4, the acquisition function can have the general form shown below in equation (1): f(x)=Exploration_Weight*Confidence_Interval(x)+Mean(x) (1)

[0062] The search weights can be set by an administrator and / or by the parameter value selector 132 to control the priority or bias used for the confidence interval values ​​relative to the mean in the acquisition function. A higher search weight can result in a higher priority being given by the parameter value selector 132 to searches (e.g., searches in parameter value ranges not yet searched). A lower search weight can result in a lower priority of searches for the parameter value selector 132, and thus a higher priority of exploitation for the parameter value selector 132. Exploitation can refer to the parameter value selector 132 selecting parameter values ​​that the first model predicts will lead to the highest metric value. As noted above, the parameter value selector 132 can initially prioritize exploration (e.g., by configuring a higher search weight) when the number of evaluated parameter values ​​is smaller, and then prioritize exploitation (e.g., by configuring a lower search weight) as the number of evaluated parameter values ​​increases. In summary, the parameter value selector 132 may initially prioritize searching an unexplored parameter value range, and then search for parameter values ​​that are predicted to have the highest metric values, such as parameter values ​​close to the evaluation parameter value that has resulted in the highest metric value so far.

[0063] Given that the gain function uses a combination of a confidence interval value and a mean value, the highest gain function value generated by parameter value selector 132 using the activation function may be a parameter value that has a relatively large confidence interval, a relatively large mean value, or a combination of a relatively large confidence interval and mean value, relative to the other parameter values. For example, predicted point 315 with a parameter value of 0.27 has a maximum prediction metric value of 0.64 among the parameter values ​​and a larger confidence interval (e.g., represented by the height of vertical line 322) than the other parameter values. Thus, parameter value selector 132 may select the parameter value 0.27 of predicted point 315 as the next parameter to evaluate, as described in more detail below with respect to FIG. 3B.

[0064] In some cases, the parameter value selector 132 selects one next parameter to be evaluated for each next experiment. In other cases, the parameter value selector 132 may select more than one next parameter value to be evaluated for the next experiment. Thus, the parameter value selector 132 may select another parameter value other than the parameter value 0.27 of the prediction point 315. For example, the prediction point 332 with a parameter value of 0.58 has a prediction metric value of 0.35, which is lower than the prediction metric value of 0.64 for the prediction point 315. However, the confidence interval (e.g., represented by the height of the vertical line 324) for the parameter value 0.58 of the prediction point 332 is larger than the confidence interval for the prediction point 315. Depending on the value of the search weight, the parameter value 0.58 of the prediction point may be selected as the next parameter to be evaluated. A relatively high confidence interval for the parameter value 0.58 of the prediction point 332 may represent a high potential (e.g., a high potential for the parameter value selector 132 to find a best or better parameter value compared to the existing evaluation parameter values). The relatively high potential of parameter value 0.58 may lead to parameter value 0.58 being selected as the next parameter value to be evaluated by parameter value selector 132, even though the prediction metric value for parameter value 0.58 is lower than the other prediction points. As another example, parameter values ​​of 0.68 and 0.85 for prediction points 334 and 336, respectively, may be selected as the next parameter value to be evaluated by parameter value selector 132 if the combination of the weighted confidence intervals and mean values ​​for those prediction points leads to the highest obtaining function value (or within a predetermined number of the highest obtaining function values).

[0065] FIG. 3B is a graph 350 illustrating an acquisition function. X-axis 352 corresponds to the parameter value. Y-axis 354 corresponds to the value of the acquisition function given a particular parameter value. An acquisition function line 355 is plotted on graph 350. An acquisition function is a function that can output a value used by a parameter value selector to select the next parameter value to evaluate (e.g., an acquisition function can generate a value that instructs parameter value selector 132 how to search the parameter value space during the parameter tuning process). For example, parameter value selector 132 can select the next parameter to evaluate by determining the parameter value having the highest acquisition function value. For example, parameter value selector 132 can determine that point 356 on acquisition function line 355 has the highest acquisition function value of 0.043 at a parameter value of 0.27. Point 356 is, for example, at the highest peak of acquisition function line 355. Point 356 corresponds to predicted point 315 described above with respect to FIG. 3A. Although the parameter value of 0.27 for point 356 may be selected by parameter value selector 132 based on a combination of a confidence value (e.g., exploration) and an average value (e.g., exploitation), the parameter value of 0.27 may be selected by parameter value selector 132 based primarily on the average value. As such, the parameter value of 0.27 may be considered to be primarily an "exploit" parameter value.

[0066] The parameter value selector 132 can select other parameter values ​​to evaluate based on the acquisition function. For example, the parameter value selector 132 can select parameter values ​​of 0.68 and 0.85 for points 358 and 360, respectively. Points 358 and 360 are each at other peaks of the acquisition function line 355. Points 358 and 360 correspond to predicted points 334 and 336, respectively, in FIG. 3A. Although the parameter value selector 132 can select the parameter values ​​0.68 and 0.85 based on a combination of confidence values ​​(e.g., exploration) and average values ​​(e.g., exploitation), the parameter values ​​of 0.68 and 0.85 may be selected by the parameter value selector 132 primarily based on the confidence values. Thus, the parameter values ​​of 0.68 and 0.85 may be considered to be primarily "exploration" parameter values. The average of the parameter values ​​0.68 and 0.85 may be influenced by the average of nearby known evaluation points 312 (e.g., where "nearby" means within a threshold distance of the known evaluation point 312). The selection of the parameter values ​​0.68 and 0.85 by the parameter value selector 132 may be thought of as further searching the range around the known evaluation point 312.

[0067] 4 illustrates example pseudocode 400 for automatically determining parameter values. In some implementations, code corresponding to pseudocode 400 may be executed by any suitable data processing device, including, for example, parameter value selector 132 and experiment system 122 described above with respect to FIG.

[0068] In line 1, parameter value selector 132 sets variable m to the number of trials per round, which may represent how many parameter values ​​are evaluated each round.

[0069] In line 2, parameter value selector 132 sets the variable k to the number of experimental rounds.

[0070] On line 3, parameter value selector 132 sets the gamma variable to the search bias value. The gamma variable corresponds to the search weight described above with respect to Figure 3B.

[0071] On line 4, parameter value selector 132 sets the data_points array to be an empty array.

[0072] On line 5, parameter value selector 132 sets the new_trials array to be an empty array.

[0073] On line 6, parameter value selector 132 configures the first iterative structure to repeat a total of m times (eg, one iteration for each trial).

[0074] On line 7, during a given iteration of the first repeating structure, parameter value selector 132 generates a random parameter value x[i].

[0075] On line 8, the parameter value selector 132 adds the random parameter value to the new_trials array.

[0076] At line 9, the experiment system 122 runs experiments using the random parameter values ​​in the new_trials array to generate metric values ​​y1, y2, ..., y m Generate.

[0077] On line 10, parameter value selector 132 selects the evaluation points (x1,y1), (x2,y2), ... (x m ,y m ) and add the evaluation points to the data_points array.

[0078] At line 11, parameter value selector 132 configures the second iterative structure to repeat a total of k-1 times. For example, as described above with respect to Figure 1, parameter value selector 132 can be configured to perform a predetermined number of parameter tuning iterations during the parameter tuning process. The value k at line 11 corresponds to the predetermined number of parameter tuning iterations.

[0079] On line 12, parameter value selector 132 creates a first model (eg, first model 140) that fits the evaluation points in the data_points array.

[0080] On line 13, parameter value selector 132 resets the new_trials array to be an empty array.

[0081] On line 14, parameter value selector 132 sets the new_trials array to be an empty array, clones the scores in the data_points array into it, and stores the cloned scores in a temporary array.

[0082] On line 15, parameter value selector 132 clones the first model and stores the cloned first model in a temporary object.

[0083] On line 16, parameter value selector 132 configures the third iterative structure to repeat a total of m times (eg, one iteration for each trial).

[0084] At line 17, based on the first model (e.g., the fitted Gaussian model created at line 12), parameter value selector 132 determines a function value using an upper confidence limit (ucb) function that corresponds to the acquisition function illustrated in Figure 3B. Parameter value selector 132 calculates the ucb function value using the mean of the Gaussian model, the standard deviation of the Gaussian model (e.g., the confidence), and a gamma parameter. The gamma parameter is a parameter that parameter value selector 132 can configure to control the priority level for exploration (vs. exploitation), as described above with respect to Figure 3B and below with respect to line 24.

[0085] At line 18, parameter value selector 132 determines the parameter value x[i] that has the maximum ucb function value (e.g., the maximum gain function value). For example, as illustrated in FIG. 3B, point 356 corresponds to the maximum gain function value.

[0086] On line 19, the parameter value selector 132 adds the parameter value x[i] with the maximum acquisition function value to the new_trials array (e.g., equivalent to including the maximum acquisition function value in the new experiment set up for the next parameter tuning iteration).

[0087] On line 20, parameter value selector 132 adds a new point to the temporary model, where the new point includes the X value x[i] of the parameter value with the highest obtaining function value and the Y value of the highest obtaining function value.

[0088] At line 21, parameter value selector 132 creates an adjusted first model by performing a fitting operation to fit the point to a temporary model (e.g., where the temporary model includes the new point with the highest acquisition function value). Including the new point in the fitting operation may lead to a confidence value for x[i] becoming zero in a later iteration of the third iterative structure (e.g., at line 17), which may cause parameter value selector 132 to select other parameter values ​​for evaluation that have higher (e.g., non-zero) confidence values. That is, higher confidence values ​​may lead to higher acquisition function values, according to the ucb function at line 17. As an example, parameter value selector 132 may select points 358 and 360 based in part on the non-zero confidence values ​​of points 358 and 360.

[0089] At line 22, the experiment system 122 runs experiments using the parameter values ​​in the new_trials array to generate metric values ​​y1, y2, ..., y m Generate.

[0090] On line 23, parameter value selector 132 selects the evaluation points (x1,y1), (x2,y2), ... (x m ,y m ) and add the evaluation points to the data_points array.

[0091] At line 24, the parameter value selector 132 optionally adjusts the gamma variable to correspond to different search weights. As described above with respect to FIG. 3B, the search weights can be higher in early iterations and lower in later search (so that search is prioritized by the parameter value selector 132 in early iterations and exploitation is prioritized by the parameter value selector 132 in later iterations). The parameter value selector 132 can decrease the value of the gamma parameter during each iteration of the second iteration structure starting at line 11. The parameter value selector 132 can decrease the gamma parameter to zero during the final iteration of the second iteration structure, so that in the final iteration, the parameter value selector 132 determines the ucb value using only the mean value of the Gaussian model, not the standard deviation value of the Gaussian model. After the gamma value is adjusted during a given iteration, the parameter value selector 132 can perform the next iteration, unless the last iteration has already been performed.

[0092] 5 is a flow diagram of an example process 500 for automatically determining parameter values. Operations of process 500 are described below as being performed by components of the systems described and depicted in FIGS. 1-4. Operations of process 500 are described below for illustrative purposes only. Operations of process 500 may be performed by any suitable device or system, such as any suitable data processing apparatus. Operations of process 500 may also be implemented as instructions stored on a computer-readable medium, which may be non-transitory. Execution of the instructions causes one or more data processing apparatuses to perform the operations of process 500.

[0093] The parameter tuning system 110 performs (at 502) multiple iterations to identify parameter values ​​for parameters based on which the content platform controls the provision of the digital component that accompanies the video content. In each iteration among the multiple iterations, the parameter tuning system performs operations 504-516, each of which is described below.

[0094] The parameter tuning system 110 identifies (at 504) a set of rating points for the parameters. For example, as described above with reference to FIGS. 1-4, the parameter value selector 132 identifies rating points 134. Each rating point includes an evaluation parameter value for the parameter and a metric value for a metric corresponding to provision of the digital component by the content platform 106. The metric generator 114 determines the metric value of the rating point from data generated by the content platform 106 using the evaluation parameter value for the rating point to provide the digital component during previous experimentation.

[0095] The parameter tuning system 110 generates (at 506) a first model using the set of evaluation points. For example, as described above with reference to FIGS. 1-4, the parameter value selector 132 can generate a first model 140 that fits the current set of evaluation points. The first model can be a Gaussian model that fits the current set of evaluation points. As another example and as described above with respect to FIG. 3A, the first model can be an aggregate model of multiple other models, where each other model uses a different approach to fit the current set of evaluation points. The first model can include an average predicted metric value that is determined based on an average of the respective predicted metric values ​​generated by each other model for each parameter value.

[0096] The parameter tuning system 110 generates (at 508) mean values ​​and confidence intervals for the first model. For example, as described above with reference to FIGS. 1-4, the parameter value selector 132 can generate mean values ​​142 and confidence intervals 144. The mean values ​​for points other than the current set of evaluation points correspond to predicted metric values ​​generated by the first model. The mean values ​​for the current set of evaluation points are known metric values ​​rather than predicted metric values. Thus, the confidence value for the current set of evaluation points is zero (e.g., the parameter tuning system can have complete confidence in the metric values ​​for the current set of evaluation points because the metric values ​​are known for the current set of evaluation points). The confidence value for points other than the current set of evaluation points is non-zero, and is higher the further a given point is from the known current evaluation points. That is, the confidence value for a given point represents the parameter tuning system's confidence in the predicted metric values ​​for that point, with higher confidence values ​​representing less confidence of the parameter tuning system in the predicted metric values.

[0097] The parameter tuning system 110 generates (at 510) a second model based on the first model using an acquisition function. The acquisition function is based on the mean value of the first model, the confidence interval of the first model, and a configurable search weight that controls the priority of the search to evaluate the parameters. For example, the parameter value selector 132 can generate the second model 146 as described above with reference to FIGS. 1-4. The parameter tuning system 110 can use the search weight to control the priority of searching new parameter values ​​when determining the next parameter value to evaluate from the second model. The search weight can correspond to the weight of the confidence interval in the acquisition function. The search weight can be higher in earlier parameter evaluation iterations and lower in later parameter evaluation iterations.

[0098] Parameter tuning system 110 determines (at 512) the next parameter value to evaluate from the second model. For example, as described above with reference to FIGS. 1-4, parameter value selector 132 can determine the next parameter value to evaluate based on the highest winning function value generated from second model 146. Parameter value selector 132 can determine more than one next parameter value to evaluate. For example, parameter value selector 132 can determine the parameter value that corresponds to a top predetermined number of the highest winning function values.

[0099] The parameter tuning system 110 configures (at 514) the content platform 106 to use the next parameter value to provide the digital component along with the video content. For example, as described above with reference to Figures 1-4, the parameter tuning system 110 can provide the experiment file 138 to the experiment system 122 to configure the experiment system 122 to perform an experiment based on the next parameter value.

[0100] The parameter tuning system 110 determines (at 516) the next metric value based on data resulting from the content platform 106 using the next parameter value to provide the digital component. For example, as described above with reference to Figures 1-4, the evaluator 135 can receive the experiment metrics 130 from the content platform 106.

[0101] The parameter tuning system 110 determines (at 518) from among the parameter values ​​and corresponding metric values ​​for the parameters determined during the multiple iterations, a particular parameter value that results in a highest metric value or satisfies a particular threshold. In some implementations, and as described above with reference to Figures 1-4, the parameter tuning system 110 selects from among the various parameter values ​​for the parameter that results in a highest metric value (relative to other metric values ​​determined for other parameter values). Alternatively, and as also described above with reference to Figures 1-4, the parameter tuning system 110 selects from among the various parameter values ​​for the parameter that a parameter value that satisfies (e.g., meets or exceeds) a threshold value for the parameter.

[0102] In some implementations, multiple iterations are determined based on a metric value from the current iteration satisfying (eg, meeting or exceeding) a threshold or a maximum number of iterations being reached.

[0103] The parameter tuning system 110 configures (at 520) the content platform to use the particular parameter values ​​(determined at 518) to control or select, during production, digital components that are provided along with the video content.

[0104] 6 is a block diagram of an example computer system 600 that can be used to perform the operations described above. The system 600 includes a processor 610, a memory 620, a storage device 630, and an input / output device 640. Each of the components 610, 620, 630, and 640 can be interconnected, for example, using a system bus 650. The processor 610 can process instructions for execution within the system 600. In some implementations, the processor 610 is a single-threaded processor. In other implementations, the processor 610 is a multi-threaded processor. The processor 610 can process instructions stored in the memory 620 or in the storage device 630.

[0105] The memory 620 stores information within the system 600. In one embodiment, the memory 620 is a computer-readable medium. In some implementations, the memory 620 is a volatile memory unit. In other implementations, the memory 620 is a non-volatile memory unit.

[0106] The storage device 630 is capable of providing mass storage for the system 600. In some implementations, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 may include, for example, a hard disk device, an optical disk device, a storage device shared over a network by multiple computing devices (e.g., a cloud storage device), or some other mass storage device.

[0107] The I / O device 640 provides input and output operations to the system 600. In some implementations, the I / O device 640 may include one or more of a network interface device, such as an Ethernet card, a serial communication device, such as an RS-232 port, and / or a wireless interface device, such as an 802.11 card. In another implementation, the I / O device may include a driver device configured to receive input data and send output data to the peripheral device 660, such as a keyboard, a printer, and a display device. However, other implementations may be used, such as mobile computing devices, mobile communication devices, set-top boxes, television client devices, and the like.

[0108] Although an example processing system is described in FIG. 6, example implementations of the objects and functional operations described herein can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, or in any combination or combinations thereof, including the structures disclosed herein and their structural equivalents.

[0109] The embodiments of the subject matter and operations described herein can be implemented in digital electronic circuitry, in computer software, firmware or hardware, including the structures disclosed herein and their structural equivalents, or in one or more combinations thereof. The embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium (or media) for execution by or to control the operation of a data processing device. Alternatively, or in addition, the program instructions can be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical or electromagnetic signal, that is generated to encode information for transmission to an appropriate receiver device for execution by the data processing device. The computer storage medium can be or be included in a computer readable storage device, a computer readable storage substrate, a random or serial access memory array or device, or one or more combinations thereof. Moreover, although the computer storage medium is not a propagating signal, the computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium may be, or may be contained in, one or more separate physical components or media (eg, multiple CDs, disks or other storage devices).

[0110] The operations described herein may be implemented as operations performed by a data processing apparatus on data stored in one or more computer-readable storage devices or received from other sources.

[0111] The term "data processing apparatus" encompasses all kinds of apparatus, devices and machines for processing data, including, by way of example, programmable processors, computers, systems on chips, or a combination or combinations thereof. The apparatus may include special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program, such as code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, a cross-platform execution environment, a virtual machine, or one or more combinations thereof. The apparatus and execution environment may implement a variety of different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures.

[0112] A computer program (also known as a program, software, software application, script or code) can be written in any type of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple linked files (e.g., files that store one or more modules, subprograms or portions of code). A computer program can be deployed and executed on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0113] The processes and logic flows described herein may be performed by one or more programmable processors that execute one or more computer programs to perform operations on input data and generate output. The processes and logic flows may also be performed by, and the apparatus may be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0114] Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors. In general, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for performing actions according to instructions, and one or more memory devices for storing instructions and data. In general, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks, for receiving data therefrom or transferring data thereto, or both. However, a computer need not have such devices. Moreover, a computer can be incorporated into another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name a few. Suitable devices for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM and flash memory devices, magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or integrated with special purpose logic circuitry.

[0115] To provide for interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user, for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback or tactile feedback, and input from the user can be received in any form, including acoustic, speech or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to the web browser in response to a request received from the web browser on the user's client device.

[0116] Embodiments of the subject matter described herein can be implemented in a computing system that includes a back-end component, e.g., as a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks ("LANs") and wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad-hoc peer-to-peer networks).

[0117] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data (e.g., HTML pages) to a client device (e.g., for the purposes of displaying the data to a user interacting with the client device and receiving user input from the user). Data generated at the client device (e.g., a result of user interaction) can be received at the server from the client device.

[0118] Although the present specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in the present specification in the context of separate embodiments can also be implemented in a single embodiment in combination. Conversely, various features described in the context of a single embodiment can also be implemented separately in multiple embodiments or in any suitable subcombination. Moreover, although features may be described above as acting in a certain combination and may initially be claimed as such, one or more features from a claimed combination may in some cases be deleted from the combination, and the claimed combination may be a subcombination or a variation of the subcombination.

[0119] Similarly, although operations are depicted in the figures in a particular order, this should not be understood as requiring that such operations be performed in the particular order or sequence shown, or that all of the illustrated operations be performed, to achieve desirable results. In some situations, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described may generally be integrated together in a single software product or implemented into multiple software products.

[0120] Thus, specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order or sequence shown to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0121] 100 Environment 102 client devices 104 Network 106 Content Platform 108 Content Provider 110 Parameter Tuning System 112 User Activity Log 114 Metric Generator 116 Metrics Database 118 Parameters 120 Parameter setting device 122 Experimental System 124 Experiment 126 Experimental Traffic 128 Experimental Activities 130 Experimental Metrics 131 Drivers 132 Parameter Value Selector 134 points 135 Evaluator 136 Experiment File Editor 138 Experiment File 140 First Model 142 Average 144 Confidence Intervals 146 Second Model 200 Graphs 202 X-axis 204 Y axis 206, 208, 210, 212, 214 Ratings 250 graphs 252 X-axis 254 Y axis 256, 258, 260, 262, 264 rating points 266 True Function Lines 268 points 300 Graphs 302 X-axis 304 Y axis 306, 308, 310, 312, 314 Rating points 315, 332, 334, 336 predicted points 316 True Function Lines 318 Gaussian Line 320 Shaded area 322, 324, 326, 328 Vertical lines 350 Graphs 352 X-axis 354 Y axis 355 Acquisition Function Line 356, 358, 360 points 400 Pseudocode 600 Computer Systems 610 Processor 620 Memory 630 Storage Devices 640 Input / Output Devices 650 System Bus 660 Peripheral Devices

Claims

1. A computer-implemented method comprising: performing a plurality of iterations to identify parameter values ​​for parameters controlling provision of a digital component with the video content by the content platform, wherein each iteration in the plurality of iterations comprises: identifying a set of rating points for the parameters, each rating point including a rating parameter value for the parameter and a metric value of a metric corresponding to the provision by the content platform of a digital component using the rating parameter value; generating a first model using the first model with the set of evaluation points; generating a mean and a confidence interval for the first model; generating a second model based on the first model and an acquisition function, the acquisition function being based on a mean and a confidence interval of the first model; determining next parameter values ​​to be evaluated from the second model; configuring the content platform to provide a digital component along with the video content using the next parameter value; determining a next metric value based on data resulting from the content platform providing a digital component using the next parameter value; determining selected parameter values ​​from among the parameter values ​​and corresponding metric values ​​of the parameters determined during the multiple iterations; configuring said content platform using said selected parameter values ​​to control or select digital components to be provided with said video content during production; 4. A computer-implemented method comprising:

2. The computer-implemented method of claim 1, wherein the acquisition function is based on a configurable search weight that controls the priority of searches for evaluating the parameters.

3. The computer-implemented method of claim 1, wherein the selected parameter value is the parameter value that results in the highest metric value or satisfies a particular threshold.

4. The computer-implemented method of claim 1, further comprising a step of determining at least one other parameter to be evaluated from the second model other than the next parameter value.

5. The computer-implemented method of claim 1, wherein generating the first model comprises generating the first model as a model that fits a current set of evaluation points.

6. The computer-implemented method of claim 2, wherein the configurable search weight controls the priority of searching for new parameter values ​​when determining the next parameter value to be evaluated from the second model.

7. The computer-implemented method of claim 2, wherein the configurable search weights correspond to weights of confidence intervals in the acquisition function.

8. The computer-implemented method of claim 2, wherein the configurable search weights are larger in earlier parameter evaluation iterations and smaller in later parameter evaluation iterations.

9. The computer-implemented method of claim 1, wherein each evaluation point in the initial set of evaluation points includes a randomly generated parameter value.

10. The computer-implemented method of claim 1, wherein determining the next parameter value to evaluate from the second model includes determining the parameter value having the corresponding highest acquisition function value.

11. A method of implementing a method for a computer-implemented program comprising: one or more memory devices for storing instructions; one or more data processing apparatus interacting with said one or more memory devices and configured to, when said instructions are executed, perform the method according to any one of claims 1 to 10; A system comprising:

12. A computer program comprising instructions which, when executed by one or more data processing apparatus, cause the one or more data processing apparatus to perform a method according to any one of claims 1 to 10.

Images