Reducing prediction errors of machine learning systems

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for reducing prediction errors in machine learning systems. In one aspect, a method includes creating a set of computer readable instructions that cause a computing device to perform operations including: obtaining a predicted outcome generated by a machine learning system; obtaining an actual outcome resulting from an allocation based on the predicted outcome; determining a loss attributable to an error of the machine learning system, where the loss is determined based on (i) a difference between the predicted outcome and the actual outcome, and (ii) a fraction of an actual outcome in which the predicted outcome is less than the actual outcome; and adjusting subsequent predictions generated by the machine learning model based on the loss attributable to the machine learning model. The created instructions are transmitted to a computing device.

Description

Attorney Docket No. t>6113-0787WO1REDUCING PREDICTION ERRORS OF MACHINE LEARNING SYSTEMSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to Israel Application No. 315293, filed on August 28th, 2024. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application.BACKGROUND

[0002] This specification relates to data processing, and reducing prediction errors of machine learning systems. Machine learning systems are used to generate predictions, which are used, in turn to take real-world actions. As such, errors in machine learning predictions can lead to unfavorable and / or unwanted real-world outcomes. The techniques discussed herein are provided to reduce the prediction errors of machine learning systems.SUMMARY

[0003] In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of creating a set of computer readable instructions that, upon execution by a computing device, cause the computing device to perform operations including: for each given allocation among a set of different allocations of different limited resources: obtaining a predicted outcome generated by a machine learning system; obtaining an actual outcome resulting from submission of an indication of priority for allocation of a limited resource based on the predicted outcome; determining, based on the predicted outcome and the actual outcome, a loss attributable to an error of a set of predicted outcomes generated by the machine learning system, wherein the loss is determined based on (i) a difference between the predicted outcome and the actual outcome for a first given allocation of a first limited resource for which the predicted outcome is greater than the actual outcome, and (ii) a fraction of the actual outcome for a second given allocation of a second limited resource for which the predicted outcome is less than the actual outcome; and adjusting subsequent predictions generated by the machine learning model based on the loss attributable to the error of the set of predicted outcomes generated by the machine learning model; and transmitting one or more of (i) the set ofAttorney Docket No. t>6113-0787WO1 computer readable instructions or (ii) a location of the set of computer readable instructions to the computing device.

[0004] Other embodiments of this aspect include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.

[0005] These and other embodiments can each optionally include one or more of the following features. Determining the loss attributable to the error of the set of predicated outcomes generated by the machine learning system can further include determining the error based on a difference between a first predicted outcome and a baseline outcome for predicted outcomes less than the baseline outcome.

[0006] The instructions can cause the computing device to perform operations further including incorporating, into the machine learning system, one or more terms that cause the machine learning system to generate a set of predicted outcomes that have a log-normal distribution.

[0007] One or more terms can cause a median actual outcome to increase sub-linearly relative to the actual outcome.

[0008] The indication of priority can be a value representing a level of priority of obtaining a particular given allocation of a limited resource, and wherein the indication of priority is one of (i) the predicted outcome or (ii) an output of a function applied to the predicted outcome.

[0009] In general, another aspect of the subject matter described in this specification can be embodied in methods that include the actions of: for each given allocation among a set of different allocations of different limited resources: obtaining a predicted outcome generated by a machine learning system; obtaining an actual outcome resulting from submission of an indication of priority for allocation of a limited resource based on the predicted outcome; determining, based on the predicted outcome and the actual outcome, a loss attributable to an error of a set of predicted outcomes generated by the machine learning system, wherein the loss is determined based on (i) a difference between the predicted outcome and the actual outcome for a first given allocation of a first limited resource for which the predicted outcome is greater than the actual outcome, and (ii) a fraction of the actual outcome for a second given allocation of a second limited resource for which the predicted outcome is lessAttorney Docket No. t>6113-0787WO1 than the actual outcome; and adjusting subsequent predictions generated by the machine learning model based on the loss attributable to the error of the set of predicted outcomes generated by the machine learning model; and transmitting one or more of (i) the set of computer readable instructions or (ii) a location of the set of computer readable instructions to the computing device.

[0010] Other embodiments of this aspect include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.

[0011] These and other embodiments can each optionally include one or more of the following features. Determining the loss attributable to the error of the set of predicated outcomes generated by the machine learning system can further include determining the error based on a difference between a first predicted outcome and a baseline outcome for predicted outcomes less than the baseline outcome.

[0012] Methods can include incorporating, into the machine learning system, one or more terms that cause the machine learning system to generate a set of predicted outcomes that have a log-normal distribution.

[0013] One or more terms can cause a median actual outcome to increase sub-linearly relative to the actual outcome.

[0014] The indication of priority can be a value representing a level of priority of obtaining a particular given allocation of a limited resource, and wherein the indication of priority is one of (i) the predicted outcome or (ii) an output of a function applied to the predicted outcome.

[0015] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. For example, the present subject matter can account for uncertainty or potential errors in machine learning systems that focus on maximizing effectiveness (e.g., actual outcomes or return on spend), which typically do not adequately account for these factors.

[0016] Instead of using standard loss functions which may not directly align with the goal of maximizing effectiveness (e.g., actual outcome or return) while still accounting for uncertainty or potential errors, the present subject matter uses a specialized loss function / setAttorney Docket No. t>6113-0787WO1 of loss functions, thereby resulting in a system that is more accurate and capable of accounting for prediction errors, while also maximizing effectiveness.

[0017] The present subject matter also reduces the error of machine learning systems by considering regression toward the mean that can occur, and is ignored, in standard machine learning systems. For example, extreme predictions (e g., very high or very low relative to a specified reference value) tend toward the average, which can reduce the accuracy and effectiveness of model predictions, whereas existing systems do not necessarily directly address this issue.

[0018] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0019] FIG. l is a block diagram of an example environment in which machine learning systems can be implemented.

[0020] FIG. 2 is a block diagram illustrating interactions related to improving the performance of a machine learning system.

[0021] FIG 3 is a flow chart of an example process for reducing prediction errors of machine learning systems.

[0022] FIG. 4 is a block diagram of an example computer system that can be used to perform operations described above.

[0023] Like reference numbers and designations in the varloPs drawings indicate like elements.DETAILED DESCRIPTION

[0024] This specification discusses improvements to machine learning models that reduce and / or minimizes the real-world impacts caused by prediction errors in the machine learning model. In this way, the techniques discussed herein result in the actions performed based on the output of the machine learning models to be more accurate and more effective than actions performed based on the output of machine learning models that do not use the techniques discussed herein. More specifically, the techniques discussed herein canAttorney Docket No. t>6113-0787WO1 increase / maximize the benefit of actions taken based on the outputs of the machine learning model by considering the difference between predicted and actual values, penalizing the model for excessively generating predictions that lead to no action being taken, and incorporating the concept of regression towards the mean by using a log-normal distribution of model outputs that are used to invoke actions based on the output of the models.

[0025] In practice, the techniques used herein can be used to optimize a machine learning model for resource allocation, where the output of the model can be used to determine which entity is allocated the resource. For example, the output of the model can represent a level of priority for allocating the resource to a given entity and / or a relative importance of the given entity being allocated the resource. In a specific example, assume that the resource being allocated is limited in nature, such that assignment of the resource (e.g., a cloud computing resource - such as a computing unit or memory block, manufacturing resources, energy resources, physical space, or electron! c / virtual space) precludes assignment of the resource to another entity. In this scenario, error in the predictions / outputs of the machine learning model can lead to misallocations that result in the resource being assigned to one entity having less actual priority for the resource than another entity that was not allocated the resource because of an error in the model. This misallocation can lead to inefficient and ineffective system that creates sub-optimal situations and / or leads to system failures. The techniques discussed herein improve the machine learning model system to reduce / avoid the real-world negative impact that can result from not accounting for machine learning model error.

[0026] FIG. 1 is a block diagram of an example environment 100 in which machine learning systems (e.g., models) can be implemented. The example environment 100 includes a network 102, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. The network 102 connects electronic document servers 104, user devices 106, digital component servers 108, and a service apparatus 110. The example environment 100 may include many different electronic document servers 104, user devices 106, and digital component servers 108.

[0027] A client device 106 is an electronic device capable of requesting and receiving online resources over the network 102. Example client devices 106 include personal computers, gaming devices, mobile communication devices, tablet devices, digital assistantAttorney Docket No. t>6113-0787WO1 devices, augmented reality devices, virtual reality devices, wearable devices, and other devices that can send and receive data over the network 102. A client device 106 typically includes a user application, such as a web browser, to facilitate the sending and receiving of data over the network 102, but native applications (other than browsers) executed by the client device 106 can also facilitate the sending and receiving of data over the network 102.

[0028] A gaming device is a device that enables a user to engage in gaming applications, for example, in which the user has control over one or more characters, avatars, or other rendered content presented in the gaming application. A gaming device typically includes a computer processor, a memory device, and a controller interface (either physical or visually rendered) that enables user control over content rendered by the gaming application. The gaming device can store and execute the gaming application locally, or execute a gaming application that is at least partly stored and / or served by a cloud server (e.g., online gaming applications). Similarly, the gaming device can interface with a gaming server that executes the gaming application and “streams” the gaming application to the gaming device. The gaming device may be a tablet device, mobile telecommunications device, a computer, or another device that performs other functions beyond executing the gaming application.

[0029] Digital assistant devices include devices that have a microphone and a speaker. Digital assistant devices are generally capable of receiving input by way of voice, and respond with content using audible feedback, and can present other audible information. In some situations, digital assistant devices also include a visual display or are in communication with a visual display (e.g., by way of a wireless or wired connection).Feedback or other information can also be provided visually when a visual display is present. In some situations, digital assistant devices can also control other devices, such as lights, locks, cameras, climate control devices, alarm systems, and other devices that are registered with the digital assistant device.

[0030] As illustrated, the client device 106 is presenting an electronic document 150. An electronic document is data that presents a set of content at a client device 106. Examples of electronic documents include webpages, word processing documents, portable document format (PDF) documents, images, videos, search results pages, and feed sources. Native applications (e.g., “apps” and / or gaming applications), such as applications installed on mobile, tablet, or desktop computing devices are also examples of electronic documents.Attorney Docket No. t>6113-0787WO1Electronic documents can be provided to client devices 106 by electronic document servers 104 (“Electronic Doc Servers”).

[0031] For example, the electronic document servers 104 can include servers that host publisher websites. In this example, the client device 106 can initiate a request for a given publisher webpage, and the electronic server 104 that hosts the given publisher webpage can respond to the request by sending machine executable instructions that initiate presentation of the given webpage at the client device 106.

[0032] In another example, the electronic document servers 104 can include app servers from which client devices 106 can download apps. In this example, the client device 106 can download files required to install an app at the client device 106, and then execute the downloaded app locally (i.e., on the client device). Alternatively, or additionally, the client device 106 can initiate a request to execute the app, which is transmitted to a cloud server. In response to receiving the request, the cloud server can execute the application and stream a user interface of the application to the client device 106 so that the client device 106 does not have to execute the app itself. Rather, the client device 106 can present the user interface generated by the cloud server’s execution of the app, and communicate any user interactions with the user interface back to the cloud server for processing.

[0033] Electronic documents can include a variety of content. For example, an electronic document 150 can include native content 152 that is within the electronic document 150 itself and / or does not change over time. Electronic documents can also include dynamic content that may change over time or on a per-request basis. For example, a publisher of a given electronic document (e.g., electronic document 150) can maintain a data source that is used to populate portions of the electronic document. In this example, the given electronic document can include a script, such as the script 154, that causes the client device 106 to request content (e.g., a digital component) from the data source when the given electronic document is processed (e.g., rendered or executed) by a client device 106 (or a cloud server). The client device 106 (or cloud server) integrates the content (e.g., digital component) obtained from the data source into the given electronic document to create a composite electronic document including the content obtained from the data source.

[0034] In some situations, a given electronic document (e.g., electronic document 150) can include a digital component script (e.g., script 154) that references the service apparatusAttorney Docket No. t>6113-0787WO1110, or a particular service provided by the service apparatus 110. In these situations, the digital component script is executed by the client device 106 when the given electronic document is processed by the client device 106. Execution of the digital component script configures the client device 106 to generate a request for digital components 112 (referred to as a “component request”), which is transmitted over the network 102 to the service apparatus 110. For example, the digital component script can enable the client device 106 to generate a packetized data request including a header and payload data. The component request 112 can include event data specifying features such as a name (or network location) of a server from which the digital component is being requested, a name (or network location) of the requesting device (e.g., the client device 106), and / or information that the service apparatus 110 can use to select one or more digital components, or other content, provided in response to the request. The component request 112 is transmitted, by the client device 106, over the network 102 (e.g., a telecommunications network) to a server of the service apparatus 110.

[0035] The component request 112 can include event data specifying other event features, such as the electronic document being requested and characteristics of locations of the electronic document at which digital component can be presented. For example, event data specifying a reference (e.g., URL) to an electronic document (e.g., webpage) in which the digital component will be presented, available locations of the electronic documents that are available to present digital components, sizes of the available locations, and / or media types that are eligible for presentation in the locations can be provided to the service apparatus 110. Similarly, event data specifying keywords associated with the electronic document (“document keywords”) or entities (e.g., people, places, or things) that are referenced by the electronic document can also be included in the component request 112 (e.g., as payload data) and provided to the service apparatus 110 to facilitate identification of digital components that are eligible for presentation with the electronic document. The event data can also include a search query that was submitted from the client device 106 to obtain a search results page.

[0036] Component requests 112 can also include event data related to other information, such as information that a user of the client device has provided, geographic information indicating a state or region from which the component request was submitted, or otherAttorney Docket No. t>6113-0787WO1 information that provides context for the environment in which the digital component will be displayed (e.g., a time of day of the component request, a day of the week of the component request, a type of device at which the digital component will be displayed, such as a mobile device or tablet device). Component requests 112 can be transmitted, for example, over a packetized network, and the component requests 112 themselves can be formatted as packetized data having a header and payload data. The header can specify a destination of the packet and the payload data can include any of the information discussed above.

[0037] The service apparatus 110 chooses digital components (e.g., third-party content, such as video fdes, audio files, images, text, gaming content, augmented reality content, and combinations thereof, which can all take the form of advertising content or non-advertising content) that will be presented with the given electronic document (e.g., at a location specified by the script 154) in response to receiving the component request 112 and / or using information included in the component request 112.

[0038] In some implementations, a digital component is selected in less than a second to avoid errors that could be caused by delayed selection of the digital component. For example, delays in providing digital components in response to a component request 112 can result in page load errors at the client device 106 or cause portions of the electronic document to remain unpopulated even after other portions of the electronic document are presented at the client device 106.

[0039] Also, as the delay in providing the digital component to the client device 106 increases, it is more likely that the electronic document will no longer be presented at the client device 106 when the digital component is delivered to the client device 106, thereby negatively impacting a user's experience with the electronic document. Further, delays in providing the digital component can result in a failed delivery of the digital component, for example, if the electronic document is no longer presented at the client device 106 when the digital component is provided.

[0040] In some implementations, the service apparatus 110 is implemented in a distributed computing system that includes, for example, a server and a set of multiple computing devices 114 that are interconnected and identify and distribute digital component in response to requests 112. The set of multiple computing devices 114 operate together to identify a set of digital components that are eligible to be presented in the electronicAttorney Docket No. t>6113-0787WO1 document from among a corpus of millions of available digital components (DCl-x). The millions of available digital components can be indexed, for example, in a digital component database 116. Each digital component index entry can reference the corresponding digital component and / or include distribution parameters (DPl-DPx) that contribute to (e.g., trigger, condition, or limit) the distribution / transmission of the corresponding digital component. For example, the distribution parameters can contribute to (e.g., trigger) the transmission of a digital component by requiring that a component request include at least one criterion that matches (e.g., either exactly or with some pre-specified level of similarity) one of the distribution parameters of the digital component.

[0041] In some implementations, the distribution parameters for a particular digital component can include distribution keywords / topics / categories that must be matched (e.g., by electronic documents, document keywords, or terms specified in the component request 112) in order for the digital component to be eligible for presentation. Additionally, or alternatively, the distribution parameters can include embeddings that can use various different dimensions of data, such as website details and / or consumption details (e.g., page viewport, user scrolling speed, or other information about the consumption of data). The distribution parameters can also require that the component request 112 include information specifying a particular geographic region (e.g., country or state) and / or information specifying that the component request 112 originated at a particular type of client device (e g., mobile device or tablet device) in order for the digital component to be eligible for presentation. The distribution parameters can also specify an eligibility value (e g., ranking score, or some other specified value) that is used for evaluating the eligibility of the digital component for distribution / transmission (e.g., among other available digital components).

[0042] The identification of the eligible digital component can be segmented into multiple tasks 117a-l 17c that are then assigned among computing devices within the set of multiple computing devices 114. For example, different computing devices in the set 114 can each analyze a different portion of the digital component database 116 to identify various digital components having distribution parameters that match information included in the component request 112. In some implementations, each given computing device in the set 114 can analyze a different data dimension (or set of dimensions) and pass (e.g., transmit) results (Res 1-Res 3) 118a- 118c of the analysis back to the service apparatus 110. ForAttorney Docket No. t>6113-0787WO1 example, the results 1 18a-l 18c provided by each of the computing devices in the set 114 may identify a subset of digital components that are eligible for distribution in response to the component request and / or a subset of the digital component that have certain distribution parameters. The identification of the subset of digital components can include, for example, comparing the event data to the distribution parameters, and identifying the subset of digital components having distribution parameters that match at least some features of the event data.

[0043] The service apparatus 110 aggregates the results 118a- 118c received from the set of multiple computing devices 114 and uses information associated with the aggregated results to select one or more digital components that will be provided in response to the request 112. For example, the service apparatus 110 can select a set of winning digital components (one or more digital components) based on the outcome of one or more content evaluation processes, as discussed below. In turn, the service apparatus 110 can generate and transmit, over the network 102, reply data 120 (e.g., digital data representing a reply) that enable the client device 106 to integrate the set of winning digital components into the given electronic document, such that the set of winning digital components (e.g., winning third- party content) and the content of the electronic document are presented together at a display of the client device 106.

[0044] In some implementations, the client device 106 executes instructions included in the reply data 120, which configures and enables the client device 106 to obtain the set of winning digital components from one or more digital component servers 108. For example, the instructions in the reply data 120 can include a network location (e.g., a Uniform Resource Locator (URL)) and a script that causes the client device 106 to transmit a server request (SR) 121 to the digital component server 108 to obtain a given winning digital component from the digital component server 108. In response to the request, the digital component server 108 will identify the given winning digital component specified in the server request 121 (e.g., within a database storing multiple digital components) and transmit, to the client device 106, digital component data (DC Data) 122 that presents the given winning digital component in the electronic document at the client device 106.

[0045] When the client device 106 receives the digital component data 122, the client device will render the digital component (e.g., third-party content), and present the digitalAttorney Docket No. t>6113-0787WO1 component at a location specified by, or assigned to, the script 154. For example, the script 154 can create a walled garden environment, such as a frame, that is presented within, e.g., beside, the native content 152 of the electronic document 150. In some implementations, the digital component is overlay ed over (or adjacent to) a portion of the native content 152 of the electronic document 150, and the service apparatus 110 can specify the presentation location within the electronic document 150 in the reply 120. For example, when the native content 152 includes video content, the service apparatus 110 can specify a location or object within the scene depicted in the video content over which the digital component is to be presented. The location specified by, or assigned to, the script 154 is considered a limited resource that is allocated / assigned to / reserved for presentation of the digital component. For example, because only one digital component can be presented in a specific display area at any given time, the allocation of the specific display area to one digital component precludes the presentation of another digital component at that specific display area.

[0046] In some implementations, the environment 100 includes a machine learning (“ML”) system 160 that generates predictions used, at least in part, to select the digital component that will be presented at the client device 106. For example, the ML system 160 can accept, as input, the event data discussed above and / or historical data corresponding to actions (e.g., browsing history or item acquisitions) of a user over some period of time. The predictions output by the ML model 160 can be used, for example, to rank the relative importance of allocation of a limited resource, such as the display space being assigned for presentation of the digital component, to each entity. For example, a higher value output by the ML system 160 can indicate a higher priority or importance of allocating the display space to a given digital component than a lower output value.

[0047] In operation, the ML system 160 can generate / output a prediction based on information input to the ML system 160. The prediction generated / output by the ML system 160 can then be used directly as the indication of priority / importance discussed above, or can be adjusted by the ML system 160 (or another system operated by a digital component provider) to arrive at a final indication of priority / importance. In either case, the service apparatus 110 can receive the indication of priority / importance from the ML system 160, or another system operated by the digital component provider, and use that information toAttorney Docket No. t>6113-0787WO1 determine whether a digital component of that digital component provider will be selected for presentation at the client device in response to a particular digital component request.

[0048] For example, the service apparatus 110 can receive indications of priority / importance from multiple different digital component providers, and select the winning digital component based on those received indications of priority / importance. More specifically, the digital component provider that submitted the highest indication of priority / importance can be selected as the winner, and the digital component provided by the provider can be selected as the winning digital component. As part of that selection, the winning digital component is allocated the presentation space within a particular presentation of the electronic document 150, thereby preventing other digital components from being allocated that presentation space within the particular presentation of the electronic document 150. In some situations, the service apparatus 110 can modify the indications of priority / importance received from the digital component provider based on one or more of the event data discussed above, historical data corresponding to the digital component provider and / or digital component, or other information available to the service apparatus 110. However, the service apparatus 110 can also use the indications of priority / importance as received from the ML system 160 or otherwise received from the digital component provider.

[0049] Although a single ML system 160 is shown in FIG. 1, multiple ML systems 160 can be implemented in the environment 100. For example, each digital component provider (e.g., third-party content creator) can implement a different ML system 160 to generate predictions the digital component provider will use to indicate its level of priority to obtain allocation of a specific allocation of a presentation space, or allocation of another limited resource, such as cloud computing resources. The training of each ML system 160 can be performed, for example, by each of the different digital component providers based on information that is known to the digital component provider, but not the service apparatus 110 or other entities (e.g., servers) within the environment 100. As such, the service apparatus 110 and the other entities in the environment are not privy to the actual priority or importance of the limited resource to the digital component provider, or to what extent errors by the various ML systems 160 are contributing to sub-optimal allocations of limited resources. As such, the service apparatus 110 is unable to directly make adjustments to theAttorney Docket No. t>6113-0787WO1ML models 160 implemented by digital component providers to improve / optimize the selection of digital components based on the predictions and / or allocation of limited resources (e.g., display spaces) to digital component providers.

[0050] To overcome these limitations and improve / optimize the allocation of limited resources when various ones of the ML systems 160 generate erroneous predictions, leading to erroneous indications of priority / importance upon which the selection of a winning digital component will be selected, the service apparatus 110 can generate and / or distribute an error detection tool that is made available to the digital component providers. The error detection tool can be implemented, for example, as instructions (e.g., a script or program) that, upon execution, cause one or more data processing apparatus to quantify the errors of ML systems 160, which can then be used to adjust the outputs of the ML systems 160 and / or retrain / update the ML systems 160 to reduce the errors generated by the ML systems 160. By providing the error detection tool to the digital component providers, the error detection tool can be implemented as part of their data management (e.g., customer relationship management) systems, and operate on real-time data and / or proprietary data that is not available to the service apparatus 110. In this way, the error detection tool can improve the overall accuracy, efficiency, and effectiveness of the entire resource allocation techniques implemented in the environment without any of the digital component providers revealing their proprietary data because each of the ML systems 160 will be improved in a similar manner and, therefore, be providing more accurate indications of priority / importance of allocation, which are used by the service apparatus 110 to make the final allocation of a limited resource.

[0051] FIG. 2 is a block diagram 200 illustrating interactions related to improving the performance of a machine learning system. The block diagram includes the service apparatus 110, a client device 106, a digital component provider 202, and the machine learning system 160. The service apparatus 110 is in communication with the digital component provider 202 and the client device 106. The digital component provider 202 is in communication with (or includes) the ML system 160, and is also in communication with the client device 106, as discussed in more detail below. The digital component provider 202 can include a computer or computing system that is configured to perform operations defined by a set of code, as discussed in more detail below.Attorney Docket No. t>6113-0787WO1

[0052] The service apparatus 1 10 can transmit resource allocation data (“RAD”) 204 to the digital component provider 202 announcing the upcoming allocation of a limited resource (e.g., cloud computing resource, display area, or another limited resource). For example, when the client device 106 renders a web page or a page of a native application, the client device 106 can submit a request 112 to the service apparatus 110, as discussed with respect to FIG. 1. Upon receipt of the request, the service apparatus 110 can generate the RAD 204 to include details about the limited resource that will be allocated. For example, the RAD 204 can include any event data or other data available to the service apparatus 110, as discussed with respect to FIG. 1. The RAD 204 can also include data requesting that the digital component provider 202 submit an indication of priority 206 (“IoP”) to the service apparatus 110. The IoP 206 is a value / score representing a level of priority of obtaining a particular given allocation of a limited resource. In other words, the IoP 206 is data informing the service apparatus 110 how important obtaining the limited resource being allocated by the service apparatus 110 is to the digital component provider 202.

[0053] The exact manner in which the digital component provider arrives at the IoP 206 is not critical, as both entities understand that the higher the IoP 206, the more important obtaining the limited resource is to the digital component provider 202. However, as discussed above and further below, errors in the computation of the IoP 206 by the digital component provider 202 results in a less efficient allocation of limited resources among digital component providers that are interested in obtaining the limited resource.

[0054] For example, for each limited resource being allocated, the service apparatus 110 will transmit the RAD 204 to multiple different digital component providers 202. In this example, each digital component provider 202 will generate their own IoP 206, and provide that to the service apparatus 110. The service apparatus 110 can rank these loPs 206 according to their magnitude (e.g., from largest to smallest), and identify the highest IoP 206 for allocation of the limited resource. More specifically, the digital component provider 202 that submitted the highest IoP 206 will be allocated the limited resource, thereby precluding other digital component providers 202 from obtaining that instance of the limited resource. As such, when errors in the computation of the IoP 206 of one or more digital component providers 202 has resulted in a different ranking of the loPs 206, the limited resource will be allocated to the “incorrect” digital component provider 202 resulting in a failed / inefficientAttorney Docket No. t>6113-0787WO1 allocation of the limited resource. This is detrimental to the digital component providers 202 as well as the service apparatus 110 because the allocation is sub-optimal, for example, by resulting in a misallocation of resources or wasted resources.

[0055] In some implementations, the digital component provider 202 (as well as other digital component providers who received the RAD 204) may use the ML system 160 to generate a prediction 208 used to arrive at the loP 206. For example, the digital component provider 202 can provide an input 210 to the ML system 160, which processes the input 210, and generates an output including the prediction 208. The input 210 provided to the ML system 106 by the digital component provider 202 can include any of the information included in the RAD 204, as well as any information the digital component provider 202 may have that is not included in the RAD 204 (e.g., is unknown to the service apparatus 110), such as historical interactions between the digital component provider 202 and the client device 106 (or a user logged in to an account registered with the digital component provider).

[0056] For example, in situations where the loP 206 of the digital component provider 202 (e.g., a given digital component provider) is identified by the service apparatus 110 as the highest loP 206 among all loPs received from various digital component providers, the service apparatus 110 can select a digital component provided by that digital component provider 202 as the digital component identified in the reply data 120 transmitted to the client device 106, as discussed with reference to FIG. 1. In this example, the client device 106 may visit a domain (e.g., second level Internet domain) of the digital component provider 202, and perform various interactions with the domain. During these interactions, the digital component provider can acquire interaction data 212 indicative of the interactions of the client device 106 and / or user of the client device 106 over time. The digital component provider 202 can use this information to determine an actual outcome of distribution of the digital component to the client device 106. In turn, the digital component provider is in a position to use the actual outcome to evaluate whether the prediction 208 and / or the loP 206 accurately reflect the actual outcome. In contrast, the service apparatus 110 often does not have access to the interaction data 212, and therefore may not be able to accurately determine the actual outcome and / or whether the loP 206 provided by the digital component provider accurately reflects the actual outcome.Attorney Docket No. t>6113-0787WO1

[0057] In a specific example, the loP 206 can be a value the digital component provider places on distributing the digital component to the client device 106 based on the RAD 204 received from the service apparatus 110. That value can be represented as a bid the digital component provider 202 submits to the service apparatus 110 in an attempt to present a digital component in a slot of a web page or application interface being presented at the client device 106. In some situations, the loP 206 can simply be the prediction 208 generated by the ML system 160. In other words, the ML system 160 can be trained to generate the loP 206 (e.g., bid) directly. In some situations, the loP 206 may be the result of some function applied to the prediction 208. For example, the ML system 160 may be trained to generate the prediction 208 (e.g., a prediction of the value of the actual outcome) based on one set of attributes, and a scaling factor (.e.g., 1.5* prediction) can be applied to the prediction 208 to adjust the prediction 208 to the loP 206 based on the context (e g., time of day, day of week, etc.) of the opportunity to present the digital component that the ML system 160 may not be trained to consider.

[0058] Continuing with this example, the loP 206 is then provided to the service apparatus 110, which will either allocate or not allocate the available slot (e.g., presentation area) of the web page or application interface to the digital component provider 202. In either case, the actual outcome (e.g., actual value) derived by the digital component provider 202 can be determined, and used to determine the error of the ML system 160 and / or loss to the digital component provider 202 based on the allocation or lack of allocation of the available slot (e.g. presentation of the digital component), as discussed in more detail below, which can be used to adjust subsequent predictions of the ML system 160.

[0059] The description of this example is also applicable to allocation of cloud computing resources, or other limited resources, where the digital component provider 202 can similarly use the ML system 160 to generate a prediction 208 of the importance (the value) of acquiring the cloud computing resource is to the digital component provider 202. The loP 206 resulting from that prediction can then be used by the service apparatus 110 (or another system) to determine whether the cloud computing resource is allocated to the digital component provider 202, and the actual result of allocating, or not allocating, that cloud computing resource (or another limited resource) to the digital component provider 202 canAttorney Docket No. t>6113-0787WO1 be determined, and used to similarly determine the value / loss to the digital component provider by being allocated / not allocated the cloud computing resource.

[0060] As discussed above, the service apparatus 110 may not have access to the interaction data 212, and may not be able to determine the actual outcome, value, or loss for a digital component provider 202 as it relates to allocations of limited resources. However, the service apparatus 110 can be configured to generate a script, code, or another set of instructions that can cause the digital component provider (or another computing device) to perform operations that use information available to the digital component provider 202 to detect the error between the prediction 208 and / or loP 206 and the actual outcome determined based on the interaction data 212.

[0061] In some implementations, the service apparatus 110 generates a set of code that defines an error detection tool (“EDT”) 220, and transmits the EDT 220 to the digital component provider 202. In some implementations, the service apparatus 110 can be transmit a link to a location at which the EDT 220 is stored so that the digital component provider 202 can download the EDT 220 to their computing device.

[0062] When the digital component provider 202 executes the EDT 220, the EDT 220 collects, for each given allocation of multiple allocations of limited resources, the predicted outcome 222 generated based on the prediction 208 of the ML system 160 as well as the actual outcome 224 derived from, or expressly provided by, the interaction data 212. For example, with respect to a given allocation of a limited resource, the predicted outcome 222 can be the expected value of an action by the user of the client device 106 that will result from the digital component provider 202 interacting with the client device 106 based on allocation of the limited resource to the digital component provider 202. The actual outcome 224 can be the value obtained by the digital component provider 202 resulting from allocation of the limited resource by the service apparatus 110. For example, if the digital component provider 202 was allocated the limited resource, which resulted in value transferring to the digital component provider 202 based on the interaction data 212, the value transferred to the digital component provider 202 can be the actual outcome. In situations where no value transferred to the digital component provider 202, either (i) because the limited resource was not allocated to the digital component provider 202 or (ii) because none of the interactions between the client device 106 and the digital componentAttorney Docket No. t>6113-0787WO1 provider 202 resulted in a transfer of value to the digital component provider 202, the actual outcome can be zero.

[0063] In any event, the EDT 220 can cause the digital component provider 202 to perform error detection 226 based on a difference between the predicted outcome 222 and the actual outcome 224. The difference and / or error between the predicted outcome 222 and the actual outcome 224 can be determined for example using a subtraction operation (e.g., actual outcome-predicted outcome). In some situations, more complex operations can be performed to determine the error. For example, the operations performed by the EDT 220 can include adjustments for different scenarios and / or can be different depending on whether the predicted outcome 222 was higher or lower than the actual outcome.

[0064] In some implementations, the error detection operations can take the form of loss functions that include one or more of the following:where, y actual is the actual outcome 224; yPred is one of the prediction 208 or the loP 206; tEFF is the target effectiveness / return the digital component provider specifies for allocation of a limited resource (e.g., allocation of a presentation space in a display in which a digital component of the provider is presented). The tEFF can represent a target ratio of the loP 206 and the actual outcome 224 (e.g., loP / actual outcome). In certain implementations, the tEFF can be a target return on spend or another return achieved based on the loP; andZeroloP value is an adjustment that penalizes the ML system 160 if it gives too many zero values for the loP.

[0065] Assumptions for implementing the loss function above include:Attorney Docket No. t>6113-0787WO1If the loP 206 value is too low (e.g., below a specified threshold and / or a specified fraction of (yactuai*(l / tEFF)), the digital component provider 202 will not be allocated the limited resource (e.g., the digital component of the digital component provider will not be displayed in the presentation space of a user interface).- If the loP 206 value is equal to or higher than (yactuai*(l / tEFF)), the digital component provider 202 will be allocated the limited resource.

[0066] The result of the loss function discussed above can be the output value of the error detection 226, which can also be referred to as the loss attributable to the error of the predicted outcome generated by the machine learning model.

[0067] The EDT 220 can cause the digital component provider 202 to perform model adjustment 228. In some implementations, the model adjustment 228 includes adjusting subsequent predictions generated by the ML system 160 based on the output value of the error detection 226 (e.g., the loss attributable to the error of the predicted outcome generated by the machine learning model). For example, the digital component provider 202 can adjust parameters of the ML system 160 to reduce the output value of the error detection 226 using standard machine learning adjustment techniques. The adjustment of the ML system 106 will result in subsequent predictions generated by the ML system 160 to be adjusted (e.g., different than) predications that would have been generated if the adjustment to the ML system 106 had not been performed.

[0068] In some implementations, the model adjustment 228 can include one or more other adjustments to the ML system 160. For example, an adjustment can be made to account for regression towards the mean. In statistical modeling, regression towards the mean refers to the phenomenon where extreme observations (either very high or very low) on one measurement tend to be closer to the average on a subsequent measurement. When applied to the scenarios discussed herein, this means that ML system 160 tends to estimate extreme actual values y actual or values (y actual* (1 / tEFF)) (e.g., values that are more than a specified amount higher or lower than one or more reference values, such as a distribution median, as defined by the digital component provider or among values in a distribution) closer to the average (e.g., mean) value. This means that in simulations, the loP 206 would rarely be allocated the limited resource for situations where the y actual or (yactuai*(l / tEFF) isAttorney Docket No. t>6113-0787WO1 very high, because the ML system 160 would be very unlikely to produce very high predictions. On the other hand, the loP 206 would almost always be allocated the limited resource where y actual and (yactuai*(l / tEFF) has a value of zero, because the ML system 160 (which may be a regression model) is very unlikely to produce a prediction 208 having a zero value.

[0069] To create a more realistic ML system 160 that accounts for regression towards the mean, we adjust the assumption about the allocation probability. Instead of being binary, the allocation probability becomes a continuous value that depends on the magnitude of the loP 206 relative to a certain value (denoted as "Value" below). Specifically, for a given ratio (C), the probability of being allocated the limited resource with an loP 206 of (C*Value) should not be the same across all values of y actual or (yactuai*(l / tEFF), but should increase for higher values of yactuai or (yactuai*(l / tEFF). Formally, this can be expressed as:P Allocation ! oil = C * Value1, Value1> P(P(Allocation\IoU— C * Value2, Value2if (Value , > Valued

[0070] This reflects the intuition that for allocations of a limited resource that are going to result in a greater transfer of value to the digital component provider 202, a proportionally equivalent prediction 208 / IoP206 is more likely to result in allocation of the limited resource to the digital component provider.

[0071] To achieve this dynamic, the distribution of loPs 206 for each value of yactuaior (yactuai*(l / tEFF) originates from a log-normal distribution, where the median of the distribution (and thus, indirectly, the mean) scales sub-linearly with the actual value of yactuai or (yactuai*(l / tEFF) (e.g., the value of the user of the client device 106 to the digital component provider 202). The log-normal distribution is used because it naturally produces values that are always positive, skewed to the right (e.g., higher magnitude, and can model the range of predictions 208 / IoPs 206 in a realistic manner.

[0072] In some implementations, adjustments to of y actual can be made to improve allocations of the limited resources. For example, a scaling factor (a) and a transformation function can be used to transform yactuai before taking the logarithm to set the distribution’sAttorney Docket No. t>6113-0787WO1 median. This transformation results in the median of the bidding distribution increasing more slowly than the yactuai increases, such as the sub-linear scaling: allocation yactuai, loU, o, a) CDFijOg^orinai(IoU | log(

[0073] This leads to a distribution where the median (and thus the likelihood of an loP resulting in allocation of the limited resource at a certain ratio of yaetuai increases, but not linearly with the yactuai itself

[0074] FIG 3 is a flow chart of an example process 300 for reducing prediction errors of machine learning systems. Operations of the process 300 can be performed by one or more computing device or other data processing apparatus. For example, operations of the process 300 can be performed by the service apparatus 110 and / or the digital component provider. Operations of the process can be implemented as instructions stored on a non-transitory computer readable medium, where execution of the instructions cause one or more data processing apparatus to perform operation of the process 300.

[0075] In some implementations, the service apparatus 110 and / or another data processing apparatus, can create the instructions that cause performance of the operations of the process 300. For example, the service apparatus 110 can create a set of computer readable instructions including instructions that, upon execution by a computing device, cause the computing device to perform operations of the process 300. The set of computer readable instructions can be created, for example, by obtaining the instructions that cause performance of the operations of the process 300 and either customizing the instructions for use by a given digital component provider, or adding additional data that enables delivery of the instructions to the given digital component provider (e.g., including the instructions in a file package formatted for delivery and / or use by the given digital component provided. For example, the instructions can be modified to incorporate digital component provider specific information, such that the instructions are ready for use by the digital component provided upon receipt. The service apparatus 110 can then transmit one or more of (i) the set of computer readable instructions or (ii) a location of the set of computer readable instructions to a computing device of a digital component provider, thereby enabling the digital component provider to invoke the error detection tool in their system.Attorney Docket No. t>6113-0787WO1

[0076] A predicted outcome generated by a machine learning system is obtained (302). In some implementations, the predicted outcome is obtained for each given allocation among a set of different allocations of different limited resources. For example, as discussed with reference to FIG. 2, the digital component provider may use the ML system to generate a prediction used to arrive at the loP, which is an example of a predicted outcome. For example, the digital component provider can provide an input to the ML system, which processes the input, and generates a prediction as an output. The input provided to the ML system by the digital component provider can include any of the information included in the RAD, as well as any information the digital component provider may have that is not included in the RAD (e.g., is unknown to the service apparatus), such as historical interactions between the digital component provider and the client device (or a user logged in to an account registered with the digital component provider).

[0077] The predicted outcome can be a value the digital component provider places on distributing the digital component to the client device based on the RAD received from the service apparatus. That value can be represented as a bid the digital component provider submits to the service apparatus in an attempt to present a digital component in a slot of a web page, application interface, or another electronic document being presented at the client device. In some situations, the predicted outcome can simply be the prediction generated by the ML system. In other words, the ML system can be trained to generate the predicted output (e.g., loP, bid) directly. In some situations, the predicted output may be the result of some function applied to the prediction output by the ML system. For example, the ML system may be trained to generate the prediction (e.g., a prediction of the value of the actual outcome) based on one set of attributes, and a scaling factor (.e.g., 1.5*prediction) can be applied to the prediction to adjust the prediction to the predicted outcome based on the context (e.g., time of day, day of week, etc.) of the opportunity to present the digital component that the ML system may not be trained to consider.

[0078] An actual outcome is obtained (304). In some implementations, the actual outcome is an outcome resulting from submission of an indication of priority for allocation of a limited resource based on the predicted outcome. In some implementations, the indication of priority is a value representing a level of priority of obtaining a particular given allocationAttorney Docket No. t>6113-0787WO1 of a limited resource. The indication of urgence can be, for example, one of (i) the predicted outcome or (ii) an output of a function applied to the predicted outcome

[0079] In some implementations, the actual outcome can be the value obtained by the digital component provider resulting from allocation of the limited resource by the service apparatus. For example, if the digital component provider was allocated the limited resource, which resulted in value transferring to the digital component provider based on the interaction data, the value transferred to the digital component provider can be the actual outcome. In a specific example, assume that the limited resource is a presentation slot on a web page, native application, or another electronic document. In this example, the allocation of the limited resource to the digital component provider enables the digital component provider to present a digital component in the presentation space when the web page, native application, or another electronic document is presented at the client device. In this example, the value transferred to the digital component provider can result from a user of the client device visiting the domain of the digital component provider and making a purchase, or otherwise taking an action to which the digital component provider has placed a value. As such, the value attributed to the action by the digital component provider can be the actual outcome.

[0080] In situations where no value transferred to the digital component provider, either (i) because the limited resource was not allocated to the digital component provider or (ii) because none of the interactions between the client device and the digital component provider resulted in a transfer of value to the digital component provider, the actual outcome can be zero.

[0081] A loss is determined based on the predicted outcome and the actual outcome (306). In some implementations, the determined loss is attributable to an error of a set of predicted outcomes generated by the machine learning system. The loss can be determined based on (i) a difference between the predicted outcome and the actual outcome for a first given allocation of a first limited resource for which the predicted outcome is greater than the actual outcome, and (iii) a fraction of the actual outcome for a second given allocation of a second limited resource for which the predicted outcome is less than the actual outcome.

[0082] In some implementations, the loss can be determined, at least in part, by determining the error based on a difference between a first predicted outcome and a baselineAttorney Docket No. t>6113-0787WO1 outcome for predicted outcomes less than the baseline outcome. The error detection operations can take the form of loss functions that include one or more of the following:where, yactuai is the actual outcome; yPred is one of the prediction or the loP;- tEFF is the target effectiveness / return the digital component provider specifies for allocation of a limited resource (e.g., allocation of a presentation space in a display in which a digital component of the provider is presented). The tEFF can represent a target ratio of the loP and the actual outcome (e.g., loP / actual outcome). In certain implementations, the can be a target return on spend or another return achieved based on the loP; andZeroloP value is an adjustment that penalizes the ML system if it gives too many zero values for the loP.Assumptions for implementing the loss function above include:If the loP value is too low (e.g., below a specified threshold and / or a specified fraction of (yactuai*(l / tEFF)), the digital component provider will not be allocated the limited resource (e.g., the digital component of the digital component provider will not be displayed in the presentation space of a user interface).If the loP value is equal to or higher than (yactuai*(l / tEFF)), the digital component provider will be allocated the limited resource.

[0083] In some implementations, tEFF can represent a target ratio of the minimum loP required to obtain allocation of the limited resource and the actual outcome. For example, in a situation where the limited resource is allocated to the digital component provider thatAttorney Docket No. t>6113-0787WO1 submitted the highest loP, the minimum loP can be the value of the second highest loP submitted by another digital component provider.

[0084] In some implementation, the magnitude of ZeroloP can be set to zero, but in other implementations, the magnitude of ZeroloP can be set to a non-zero positive number to account for situations in which the model is designed to output a non-zero value when the prediction for a given allocation of a limited resource is zero.

[0085] The result of the loss function discussed above can be considered the loss attributable to the error of the predicted outcome generated by the machine learning system.

[0086] Subsequent predictions generated by the machine learning model are adjusted based on the loss attributable to the error of the set of predicted outcomes generated by the machine learning model (308). The instructions can cause parameters of the ML system to be adjusted in a manner that reduces the output value of the error detection using standard machine learning adjustment techniques. The adjustment of the ML system will result in subsequent predictions generated by the ML system to be adjusted (e.g., different than) predications that would have been generated if the adjustment to the ML system had not been performed.

[0087] In some implementations, the model adjustment can include one or more other adjustments to the ML system. For example, an adjustment can be made to account for regression towards the mean, as discussed above with reference to FIG. 2. To create a more realistic ML system that accounts for regression towards the mean, the allocation probability is implemented as a continuous value that depends on the magnitude of the loP relative to a certain value (denoted as "Value" below). Specifically, for a given ratio (C), the probability of being allocated the limited resource with an loP of (C* Value) should not be the same across all values of yactuai or (y actual* (1 / tEFF), but should increase for higher values of yactuai or (yactuai*( 1 / tEFF). Formally, this can be expressed as:P Allocation\IoU = C * Value-^.Value-i > P(P( llocation\loU = C * Value2, Value2if (V aluet> Valued

[0088] One or more terms are incorporated into the machine learning system (310). In some implementations, the one or more terms cause the machine learning system to generateAttorney Docket No. t>6113-0787WO1 a set of predicted outcomes that have a log-normal distribution. The one or more terms can cause a median actual outcome to increase sub-linearly relative to the actual outcome.

[0089] For example, the distribution of loPs for each value of y actual or (yactuai*(l / tEFF) has a log-normal distribution, where the median of the distribution scales sub-linearly with the actual value of yactuai or (yactuai*(l / tEFF) (e.g., the value of the user of the client device to the digital component provider). The log-normal distribution is used because it naturally produces values that are always positive, skewed to the right (e.g., higher magnitude, and can model the range of predictions / loPs in a realistic manner.

[0090] In some implementations, adjustments to of yactuai can be made to improve allocations of the limited resources. For example, a scaling factor (a) and a transformation function can be used to transform yactuai before taking the logarithm to set the distribution’s median. This transformation results in the median of the bidding distribution increasing more slowly than the yactuai increases, such as the sub-linear scaling:

[0091] This leads to a distribution where the median (and thus the likelihood of an loP resulting in allocation of the limited resource at a certain ratio of yactuai increases, but not linearly with the yactuai itself.

[0092] FIG. 4 is a block diagram of an example computer system 400 that can be used to perform operations described above. The system 400 includes a processor 410, a memory 420, a storage device 430, and an input / output device 440. Each of the components 410, 420, 430, and 440 can be interconnected, for example, using a system bus 450. The processor 410 is capable of processing instructions for execution within the system 400. In one implementation, the processor 410 is a single-threaded processor. In another implementation, the processor 410 is a multi -threaded processor. The processor 410 is capable of processing instructions stored in the memory 420 or on the storage device 430.

[0093] The memory 420 stores information within the system 400. In one implementation, the memory 420 is a computer-readable medium. In one implementation, the memory 420 is a volatile memory unit. In another implementation, the memory 420 is a non-volatile memory unit.Attorney Docket No. t>6113-0787WO1

[0094] The storage device 430 is capable of providing mass storage for the system 400. In one implementation, the storage device 430 is a computer-readable medium. In various different implementations, the storage device 430 can include, for example, a hard disk device, an optical disk device, a storage device that is shared over a network by multiple computing devices (e.g., a cloud storage device), or some other large capacity storage device.

[0095] The input / output device 440 provides input / output operations for the system 400. In one implementation, the input / output device 440 can include one or more of a network interface device, e.g., an Ethernet card, a serial communication device, e.g., and RS-232 port, and / or a wireless interface device, e.g., and 802.11 card. In another implementation, the input / output device can include driver devices configured to receive input data and send output data to other devices, e.g., keyboard, printer, display, and other peripheral devices 460. Other implementations, however, can also be used, such as mobile computing devices, mobile communication devices, set-top box television client devices, etc.

[0096] Although an example processing system has been described in FIG. 4, implementations of the subject matter and the functional operations described in this specification can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.

[0097] An electronic document (which for brevity will simply be referred to as a document) may, but need not, correspond to a file. A document may be stored in a portion of a file that holds other documents, in a single file dedicated to the document in question, or in multiple coordinated files.

[0098] Embodiments of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on computer storage medium for execution by, or to control the operation of, data processing apparatus. Alternatively, or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, orAttorney Docket No. t>6113-0787WO1 electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially-generated propagated signal. The computer storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0099] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0100] The term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations, of the foregoing The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures.

[0101] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinatedAttorney Docket No. t>6113-0787WO1 files (e.g., files that store one or more modules, sub-programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0102] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0103] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random-access memory or both. The essential elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0104] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displayingAttorney Docket No. t>6113-0787WO1 information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s client device in response to requests received from the web browser.

[0105] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network.Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), an inter-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0106] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data (e.g., an HTML page) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., a result of the user interaction) can be received from the client device at the server.

[0107] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, but rather as descriptions of features specific to particular embodiments of particularAttorney Docket No. t>6113-0787WO1 inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0108] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0109] Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

[0110] What is claimed is:

Claims

Attorney Docket No. t>6113-0787WO1CLAIMS1. A method performed by data processing apparatus, the method comprising: creating a set of computer readable instructions that, upon execution by a computing device, cause the computing device to perform operations comprising: for each given allocation among a set of different allocations of different limited resources: obtaining a predicted outcome generated by a machine learning system; obtaining an actual outcome resulting from submission of an indication of priority for allocation of a limited resource based on the predicted outcome; determining, based on the predicted outcome and the actual outcome, a loss attributable to an error of a set of predicted outcomes generated by the machine learning system, wherein the loss is determined based on (i) a difference between the predicted outcome and the actual outcome for a first given allocation of a first limited resource for which the predicted outcome is greater than the actual outcome, and (ii) a fraction of the actual outcome for a second given allocation of a second limited resource for which the predicted outcome is less than the actual outcome; and adjusting subsequent predictions generated by the machine learning model based on the loss attributable to the error of the set of predicted outcomes generated by the machine learning model; and transmitting one or more of (i) the set of computer readable instructions or (ii) a location of the set of computer readable instructions to the computing device.

2. The method of claim 1, wherein determining the loss attributable to the error of the set of predicated outcomes generated by the machine learning system further comprises determining the error based on a difference between a first predicted outcome and a baseline outcome for predicted outcomes less than the baseline outcome.

3. The method of claim 1, wherein the instructions cause the computing device to perform operations further comprising incorporating, into the machine learning system, one or more terms that cause the machine learning system to generate a set of predicted outcomes that have a log-normal distribution.Attorney Docket No. t>6113-0787WO14. The method of claim 3, wherein one or more terms cause a median actual outcome to increase sub-linearly relative to the actual outcome.

5. The method of claim 1, wherein the indication of priority is a value representing a level of priority of obtaining a particular given allocation of a limited resource, and wherein the indication of priority is one of (i) the predicted outcome or (ii) an output of a function applied to the predicted outcome.

6. A method performed by a computing device, the method comprising: for each given allocation among a set of different allocations of different limited resources: obtaining a predicted outcome generated by a machine learning system; obtaining an actual outcome resulting from submission of an indication of priority for allocation of a limited resource based on the predicted outcome; determining, based on the predicted outcome and the actual outcome, a loss attributable to an error of a set of predicted outcomes generated by the machine learning system, wherein the loss is determined based on (i) a difference between the predicted outcome and the actual outcome for a first given allocation of a first limited resource for which the predicted outcome is greater than the actual outcome, and (iii) a fraction of the actual outcome for a second given allocation of a second limited resource for which the predicted outcome is less than the actual outcome; and adjusting subsequent predictions generated by the machine learning model based on the loss attributable to the error of the set of predicted outcomes generated by the machine learning model; and transmitting one or more of (i) the set of computer readable instructions or (ii) a location of the set of computer readable instructions to the computing device.

7. The method of claim 6, wherein determining the loss attributable to the error of the set of predicated outcomes generated by the machine learning system further comprises determining the error based on a difference between a first predicted outcome and a baseline outcome for predicted outcomes less than the baseline outcome.Attorney Docket No. t>6113-0787WO18. The method of claim 6, further comprising incorporating, into the machine learning system, one or more terms that cause the machine learning system to generate a set of predicted outcomes that have a log-normal distribution.

9. The method of claim 8, wherein one or more terms cause a median actual outcome to increase sub-linearly relative to the actual outcome.

10. The method of claim 6, wherein the indication of priority is a value representing a level of priority of obtaining a particular given allocation of a limited resource, and wherein the indication of priority is one of (i) the predicted outcome or (ii) an output of a function applied to the predicted outcome.Attorney Docket No. t>6113-0787WO111. A system comprising: a memory device; and one or more data processing apparatus configured to access the memory device and execute instructions that, upon execution, cause the one or more data processing apparatus to perform operations comprising: for each given allocation among a set of different allocations of different limited resources: obtaining a predicted outcome generated by a machine learning system; obtaining an actual outcome resulting from submission of an indication of priority for allocation of a limited resource based on the predicted outcome; determining, based on the predicted outcome and the actual outcome, a loss attributable to an error of a set of predicted outcomes generated by the machine learning system, wherein the loss is determined based on (i) a difference between the predicted outcome and the actual outcome for a first given allocation of a first limited resource for which the predicted outcome is greater than the actual outcome, and (iii) a fraction of the actual outcome for a second given allocation of a second limited resource for which the predicted outcome is less than the actual outcome; and adjusting subsequent predictions generated by the machine learning model based on the loss attributable to the error of the set of predicted outcomes generated by the machine learning model; and transmitting one or more of (i) the set of computer readable instructions or (ii) a location of the set of computer readable instructions to the computing device.

12. The system of claim 11, wherein determining the loss attributable to the error of the set of predicated outcomes generated by the machine learning system further comprises determining the error based on a difference between a first predicted outcome and a baseline outcome for predicted outcomes less than the baseline outcome.Attorney Docket No. t>6113-0787WO113. The system of claim 11 , wherein the instructions cause the one or more data processing apparatus to perform operations further comprising incorporating, into the machine learning system, one or more terms that cause the machine learning system to generate a set of predicted outcomes that have a log-normal distribution.

14. The system of claim 13, wherein one or more terms cause a median actual outcome to increase sub-linearly relative to the actual outcome.

15. The system of claim 11, wherein the indication of priority is a value representing a level of priority of obtaining a particular given allocation of a limited resource, and wherein the indication of priority is one of (i) the predicted outcome or (ii) an output of a function applied to the predicted outcome.Attorney Docket No. t>6113-0787WO116. A non-transitory computer readable medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising: for each given allocation among a set of different allocations of different limited resources: obtaining a predicted outcome generated by a machine learning system; obtaining an actual outcome resulting from submission of an indication of priority for allocation of a limited resource based on the predicted outcome; determining, based on the predicted outcome and the actual outcome, a loss attributable to an error of a set of predicted outcomes generated by the machine learning system, wherein the loss is determined based on (i) a difference between the predicted outcome and the actual outcome for a first given allocation of a first limited resource for which the predicted outcome is greater than the actual outcome, and (iii) a fraction of the actual outcome for a second given allocation of a second limited resource for which the predicted outcome is less than the actual outcome; and adjusting subsequent predictions generated by the machine learning model based on the loss attributable to the error of the set of predicted outcomes generated by the machine learning model; and transmitting one or more of (i) the set of computer readable instructions or (ii) a location of the set of computer readable instructions to the computing device.

17. The non-transitory computer readable medium of claim 16, wherein determining the loss attributable to the error of the set of predicated outcomes generated by the machine learning system further comprises determining the error based on a difference between a first predicted outcome and a baseline outcome for predicted outcomes less than the baseline outcome.

18. The non-transitory computer readable medium of claim 16, wherein the instructions cause the one or more data processing apparatus to perform operations further comprising incorporating, into the machine learning system, one or more terms that cause the machine learning system to generate a set of predicted outcomes that have a log-normal distribution.Attorney Docket No. t>6113-0787WO119. The non-transitory computer readable medium of claim 18, wherein one or more terms cause a median actual outcome to increase sub-linearly relative to the actual outcome.

20. The non-transitory computer readable medium of claim 16, wherein the indication of priority is a value representing a level of priority of obtaining a particular given allocation of a limited resource, and wherein the indication of priority is one of (i) the predicted outcome or (ii) an output of a function applied to the predicted outcome.

Images