Efficient use of computing resources to respond to content requests

By extracting the characteristics of content requests and inputting them into the priority sorting model, adjusting the threshold value based on the target loss percentage and actual results, the problems of wasted computing resources and inaccurate priority prediction in the prior art are solved, and efficient content request processing is achieved and resource management.

CN112154433BActive Publication Date: 2024-11-26GOOGLE LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980004540.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-04-26
Publication Date
2024-11-26
Estimated Expiration
2039-04-26

AI Technical Summary

Technical Problem

When processing content requests, the prior art cannot effectively distinguish high-priority and low-priority requests, resulting in waste of computing resources, and the inability to accurately predict the priority of content requests, affecting the request processing efficiency.

Method used

By extracting features from content requests, inputting them into the priority sorting model, generating priority values, and adjusting the specified threshold based on the target loss percentage and actual results, controlling access to computing resources, and dynamically updating the priority sorting model to improve prediction accuracy .

Benefits of technology

Effectively use computing resources, improve request processing efficiency, ensure that high-priority requests obtain sufficient resources, reduce the allocation of computing resources for low-priority requests, and improve the priority sorting accuracy of content requests.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112154433B_ABST
    Figure CN112154433B_ABST
Patent Text Reader

Abstract

Methods, systems, and apparatus, including a computer program encoded on a computer storage medium, for efficiently using computing resources in response to content requests. The method includes using a prioritization model and a specified threshold that specifies a maximum allowable negative outcome for a specified content provider to determine whether a received content request is a low priority request. The method also includes throttling access to the computing resource in response to the low priority request while providing access to the computing resource for other content requests. The method also includes periodically updating the prioritization model and the specified threshold based on data from a new set of content requests.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This specification relates to data processing and efficient use of computing resources in response to content requests. Background Art

[0002] Content providers regularly receive a large number of content requests and provide content in response to these requests. Processing and responding to these requests may require a large amount of computing resources. Summary of the invention

[0003] In general, an innovative aspect of the subject matter described in this specification can be embodied in a method including the following operations: receiving a content request from a client device by a data processing device; extracting features from the content request by the data processing device; inputting the features into a prioritization model trained using a previous set of content requests by the data processing device; obtaining a priority value of the content request by the data processing device from the prioritization model based on the features input to the model; determining a specified threshold based on a target loss percentage and the results of a previous set of content requests; and throttling access to computing resources for responding to requests by the data processing device based on the specified threshold and the priority value of the request, including: providing access to computing resources for responding to requests when the prioritization model outputs a priority value that satisfies the specified threshold; and denying access to computing resources when the prioritization model outputs a priority value that fails to meet the specified threshold. Other embodiments of this aspect include corresponding systems, devices, apparatuses, and computer programs configured to perform the actions of the method. The computer program (e.g., instructions) can be encoded on a computer storage device. These and other embodiments can each optionally include one or more of the following features.

[0004] The method may include identifying a subsequent set of content requests received after receiving a content request as hold requests; providing access to computing resources required to respond to the hold requests regardless of any priority value of the hold requests; monitoring actual results of content provided in response to the hold requests; and updating a prioritization model based on the actual results.

[0005] Updating the prioritization model may include obtaining a priority value for content provided in response to the hold request; comparing the priority value to an actual result; and adjusting the prioritization model based on a difference between the priority value and the actual result.

[0006] The method may include adjusting the prioritization model based on a relationship between the target loss percentage and a percentage of discarded content requests that are denied access to the computing resource.

[0007] The method may include obtaining, by a data processing device, a priority value for a content request from a model based on features input to the model, including using a prioritization model to determine a probability that a content request generates a positive result based on results of requests in a previous set of content requests; and using the prioritization model to determine an expected size of a positive result for a content request based on positive results generated by previous content requests; and combining, by the data processing device, the probability that a content request generates a positive result with the expected size of a positive result to generate a priority value.

[0008] Determining a specified threshold based on a target loss percentage and results of a previous set of content requests may include obtaining results for each request in the previous set of content requests; aggregating the results for each request in the previous set of content requests to generate an aggregate result for the previous set of content requests; and scaling the aggregate result by the target loss percentage to generate the specified threshold.

[0009] The method may include updating the specified threshold based on the target loss percentage and the result of the hold request.

[0010] Updating a specified threshold based on a target loss percentage and results of a hold request may include obtaining a result of each request in the hold request; aggregating the results of each request in the hold request to generate an aggregate result of the hold request; and scaling the aggregate result by the target loss percentage to generate the specified threshold.

[0011] As described above, processing and responding to requests may require a large amount of computing resources, but no specific results are guaranteed. For example, some of these requests will bring positive results to the content provider, while others will not. A positive result can be any response to a request that results in a result that is favorable to the content provider, such as a user interacting with the content provided by the content provider. For example, a user can interact with the content by selecting the content with an input device, downloading the content, hovering over the content with a cursor, or viewing the content on an output device. In contrast, some content requests may not result in the user interacting with the content provided by the content provider. Therefore, there are some requests that do not bring positive results to the content provider, but require a large amount of computing resources. In either case, the processing and response to the request consumes limited computing resources, which are then not available for processing other requests, or are allocated to other tasks (for example, making the process completely independent of the request processing). This document discusses various technologies and systems that can improve the efficiency of request processing, for example, by limiting the allocation of computing resources to those requests that are identified as likely to result in positive results, thereby releasing computing resources that would otherwise be allocated to process all requests assigned to other tasks.

[0012] Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. For example, the innovations described in this specification facilitate the efficient use of computing resources to respond to content requests. Specifically, the innovations described in this specification achieve operational efficiency by reserving computing resources rather than spending them on content requests that do not bring positive results to content providers. As a result, these computing resources can be used for high priority requests, i.e., content requests that bring positive results to content providers.

[0013] As another example, the innovation described in this specification promotes more accurate prioritization of content requests through more accurate predictive modeling. Traditional models are typically static, i.e., they are not updated with new data sets. Due to the stale and static data on which these models are based, they may incorrectly classify certain high-priority requests as low-priority requests. The modeling techniques described in this specification achieve higher accuracy in request classification by utilizing a dynamic model that is regularly updated with a new set of content requests called hold requests. As part of training and updating the model, these hold requests are provided with access to the necessary computing resources, regardless of whether they include features that may cause them to be classified as low-priority requests, for example. By providing these hold requests with the necessary computing resources, their actual results can be determined, and then the model is updated based on the difference between the actual results and the priority values ​​generated by the model. Therefore, the updated model generates more accurate priority values ​​for future content requests. In other words, future content requests can be more accurately classified as high-priority requests or low-priority requests.

[0014] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram of an example environment in which content is distributed for presentation.

[0016] Figure 2 is a flow diagram of an example process for determining whether to provide access to computing resources required to respond to a content request.

[0017] Figure 3 is a flow chart of an example process for updating a prioritization model.

[0018] Figure 4 is a flow chart of an example process for determining a specified threshold.

[0019] Figure 5 is a block diagram of an example computer system.

[0020] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0021] Content providers regularly receive and respond to a large number of content requests.Processing and responding to these requests may require a large amount of computing resources.In some cases, content providers receive content requests that require a large amount of computing resources, but will not bring positive results to content providers.This request is referred to as a low priority request in this manual, and it consumes the computing resources that could have been used for high priority requests, i.e., brings a request for positive results to content providers.Positive results can be any response to request, which causes a result that is favorable to content providers, such as the content interaction provided by users and content providers.

[0022] As described in this specification, computing resources can be used efficiently by throttling access to computing resources required to respond to low priority requests while providing access to computing resources required to respond to high priority requests. In determining whether to throttle access to computing resources required to respond to a content request, the content provider first determines whether the content request is a low priority request. To facilitate this determination, the content provider extracts features from the content request and inputs these features into a prioritization model.

[0023] The prioritization model trained using data from a previous set of content requests determines a priority value for a received content request based on features of the content request input to the prioritization model. In turn, the prioritization model outputs a priority value for the content request, which can be used to determine, for example, whether the request is a high priority request or a low priority request, as discussed in more detail below.

[0024] The content provider also determines a specified threshold, which represents the maximum negative result that the content provider can allow, and the specified threshold controls the throttling of access to the computing resource. The specified threshold can be determined based on actual results of previous content request sets, such as scaled by a predetermined target loss percentage, as detailed in this specification.

[0025] The content provider compares the priority value of the content request with the specified threshold. If the priority value of the content request does not meet the specified threshold, the request is considered a low priority request. On the other hand, if the priority value of the content request meets the specified threshold, the request is considered a high priority request.

[0026] The content provider also determines whether to provide access (or allocation) to computing resources required to respond to the content request. If the content provider has determined that the request is a low priority request, the content provider throttles access to computing resources required to respond to the request. On the other hand, if the content provider determines that the request is a high priority request, the content provider provides access to computing resources required to respond to the request.

[0027] To ensure that the prioritization model continues to provide accurate predictions for content requests it receives over time, the content provider can periodically update the prioritization model with a new set of content requests that are not subject to the above throttling, thereby processing all requests in the new set of content requests. Specifically, the content provider can evaluate the actual results and priority values ​​of the new set of content requests, and use the difference between the actual results and the priority values ​​to update the prioritization model.

[0028] Reference below Figures 1 to 5 These and additional features are described in more detail.

[0029] Figure 1 1 is a block diagram of an example environment 100 in which content is distributed for presentation. The example environment 100 includes a network 108, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. The network 108 connects content providers 130 and client devices 102. The example environment 100 can include many different content providers 130 and client devices 102.

[0030] Client devices 102 are electronic devices capable of requesting and receiving content and resources over network 108. Example client devices 102 include personal computers, mobile communication devices, digital assistant devices, and other devices capable of sending and receiving data over network 108. Client devices 102 typically include user applications, such as a web browser, to facilitate sending and receiving data over network 108, although local applications executed by client device 102 may also facilitate sending and receiving content over network 108.

[0031] Examples of content presented at the client device 102 include web pages, word processing documents, portable document format (PDF) documents, images, videos, and search results pages. Content providers 130 can provide content to client devices 102. For example, in some embodiments, content providers 130 include content servers 126 that host content provided in response to content requests. In this example, client device 102 can initiate content request 104. Content request 104 is sent by client device 102 to content provider 130 via network 108 (e.g., a telecommunications network).

[0032] In some implementations, the content provider 130 includes a request analyzer 110, a data store 116, a prioritization model 122, a throttling engine 124, and one or more content servers 126. The content provider 130 may be implemented as a data processing device or reference Figure 5 The computer system described herein, or any other suitable processing system.

[0033] like Figure 1 As shown in the block diagram of , content provider 130 is depicted as a single box with various sub-boxes. However, although a content provider can be a single device or a single set of devices, the present specification contemplates that a content provider can also be a group of devices, or even multiple different systems that communicate to provide content to client devices. For example, a content provider can include one or more of a search system, a video streaming service, an audio streaming service, a navigation service, or any other service. A content provider can include a publisher of content, or can be an entity other than a publisher.

[0034] The above components of content provider 130 determine whether the content request is a low priority request and whether to provide access to computing resources required to respond to the content request. Figure 2 To describe the operation of these components, Figure 2 2 is a flow chart of an example process 200 implemented by a content provider to determine whether to provide access to a required computing resource in response to a content request. The operations of process 200 are described below for illustrative purposes only. The operations of process 200 may be performed by any suitable device or system, such as any suitable data processing device. The operations of process 200 may also be implemented as instructions stored on a computer-readable medium, which, when executed by one or more data processing devices, causes the one or more data processing devices to perform the operations of process 200.

[0035] The content provider 130 receives a content request 104 from a client device (at 202). For example, the content request 104 may be a query containing the words "cheap tax preparation companies" sent by a user through a browser of their mobile device and received by the content provider 130. For example, the content provider 130 may be a search engine or any other type of service (e.g., a web server) configured to respond to the content request 104 by sending content to the client device 102.

[0036] The content provider 130 extracts features from the content request 104 (at 204). Upon receiving the content request 104, the request is sent to the request analyzer 110 component of the content provider 130. In some embodiments, the request analyzer 110 includes two subcomponents: a feature extractor 112 and a priority analyzer 114. The feature extractor 112 extracts one or more features from the content request 104. The term "feature" as used herein may refer to any attribute of the content request 104 that the content provider 130 may use to select appropriate content to provide in response to the request. For example, the feature extractor 112 may extract features from the content request 104 that include, among other features, keywords referenced by the request, entities referenced by the request (e.g., people, places, things), geographic information (e.g., the region in which the request is submitted, network location), the name (or network location) of the requesting device (e.g., client device 102), whether the user is using an ad blocker, the type of device the user is using (browser on desktop, browser on mobile device), the type of browser (e.g., Chrome, Safari, Firefox), the time of the request, and the date of the request. As used herein, extracting features includes retrieving features from a content request and identifying features within a content request.

[0037] In the example above, the content provider extracts features from a query sent by a user through the browser of their mobile device that contains the words "cheap tax preparation company." The features extracted in this example may include the words present in the query ("cheap tax preparation company"), the IP address of the client device 104, which can be used to determine that the query originated from San Mateo, California, and that the user is using a brand X browser on their mobile device.

[0038] The content provider then inputs some or all of the extracted features into a prioritization model (at 206). The prioritization model is configured to generate a numerical value, referred to as a priority value. The priority value may be used to prioritize content requests in order to determine whether to allow a single content request to access a computing resource to respond to the request. The feature extractor 112, as it extracts features from the content request 104, inputs the features into the prioritization model 122. In the example above, the words "cheap tax preparation company", San Mateo, California, mobile device, and brand X browser are input as features of the prioritization model.

[0039] Based on the features input to the prioritization model 122 by the feature extractor 112, the prioritization model 122 outputs a priority value for the content request 104 (at 208). In some implementations, the prioritization model 122 can be generated using data from a set of previous content requests. Specifically, the prioritization model 122 can use the features extracted from the previous content requests and the actual results of the previous content requests to determine the priority value of the content request.

[0040] Prioritization model 122 may include two sub-models: SM1 118 and SM2 120. SM1 118 determines the probability of a request generating a positive outcome, and SM2 120 determines the conditional expected outcome of a content request with a positive outcome. The outputs of SM1 118 and SM2 120 may be combined to generate a priority value.

[0041] The first sub-model SM1118 of the prioritization model 122 predicts the probability that a content request generates a positive result based on the results generated by the set of previous content requests (at 214). The feature extractor 112 extracts features from previous content requests, and the priority analyzer 114 obtains the results of each of these requests. These extracted features and the results of previous content requests are stored in the data store 116. SM1118 accesses data specifying these features and results of previous content requests from the data store 116. In some embodiments, the first sub-model uses a Poisson Regression model, where, for a content request CR with a feature level X, the model predicts the probability that CR generates a positive result based on the following assumptions:

[0042] N X |O X ~Poisson(O X λ X ),

[0043] log(λ X )=Θ T X,

[0044] Among them, O X is the number of times (offset) a content request with feature level X is seen, N X is the number of content requests with expected positive outcomes, and Θ denotes the parameter vector of the model.

[0045] For example, SM 1118 may determine the probability that a query generates a positive result (e.g., a user interacts with content provided in response to the query) based on all result data for previous queries (including queries with both positive and negative results). Thus, the probability that a content request CR results in a positive result (in the example above, whether the query results in providing content with which the user can interact) is estimated as follows:

[0046]

[0047] The second sub-model SM2120 of the prioritization model 122 predicts a conditional expected result of a content request based on requests in the set of previous content requests that generated positive results (at 216). The conditional expected result is a numeric value that represents the magnitude of the positive result of the content request. SM2120 uses data specifying the features and results of previous content requests stored in the data store 116. The model can be implemented using a log-Gaussian regression model. In some embodiments, for each content request CR with a feature level X, the log-Gaussian model predicts the expected result of the CR (which can quantify the user's interaction with the content provided in response to the query) based on the following assumptions:

[0048] log(Rev+ε)~Gaussian(Γ T X, σ 2 ),

[0049] where Γ is the parameter vector of the model, and ε is set to a small positive number (ε ~ ​​10- 6 ) to prevent very small log arguments.

[0050] The prioritization model 122 combines the outputs of SM1 118 and SM2 120 to generate a priority value. Specifically, the prioritization model 122 determines the priority value (at 218) by combining the probability of a positive outcome determined by SM1 118 with the conditional expected outcome determined by SM2 120. This can be represented by the following equation:

[0051] E[interaction] = P(interaction>0) x E[interaction|interaction>0]

[0052] In the above example where a user submits a query containing "cheap tax preparation companies" through a mobile device browser, the prioritization model may determine, based on the input features of the content request, that the content request has a priority value of 65. For ease of explanation, priority values ​​are shown as integers in this specification, but it should be understood that priority values ​​may be measured on any suitable scale (e.g., currency, time, etc.).

[0053] Note that although specific implementations of models are discussed above for purposes of example, other suitable models may also be used, such as linear regression models and neural network models.

[0054] The content provider 130 determines a specified threshold (at 210) based on the target loss percentage and the results of the previous request set. Figure 4 To describe the details of the process by which the content provider determines the specified threshold, Figure 4 is a flow diagram of an example process 400 implemented by content provider 130 to determine a specified threshold.

[0055] As reference Figure 4 As described, the content provider obtains the results of each request in the previous content request set (at 402). As an example, the content provider can access the actual results (e.g., revenue, such as advertising revenue, actions taken by users, or other results) of the previous content request set stored in the data store 116. For example, for the purpose of example, assume that there are ten previous content requests that obtain the following advertising revenue (in US dollars): 100, 150, 200, 50, 30, 10, 25, 400, 15, 20. It should be understood that revenue is only one possible way to quantify the actual results of the previous content requests. Other ways to quantify the results of the previous content requests include user viewing time or interactive time of the content.

[0056] The content provider 130 aggregates the results of the previous content requests to generate an aggregate result (at 404). In the above example, the content provider 130 determines that the total revenue of the previous content requests is $1000 based on the aggregate (e.g., 100+150+200+50+30+10+25+400+15+20). If the actual results are quantified using user viewing time, the content provider 130 determines the total viewing time of the previous content requests by aggregating (summing) the viewing time of each request.

[0057] The content provider scales the aggregate result by the target loss percentage to generate a specified threshold (at 406). The target loss percentage that can be predetermined specifies the proportion of content requests that are denied access to computing resources. Therefore, the content provider 130 can adjust the performance of the system by specifying a target loss percentage that causes the service of a portion of content requests to be denied. In the above example, the content provider 130 can set the target loss percentage to 10%. The content provider therefore scales the total revenue of $1000 by this 10% target loss percentage to generate a specified threshold of 100. If the actual result is quantified using user viewing time, the content provider 130 scales the total viewing time of previous content requests by a target loss percentage of 10%.

[0058] The calculation of the specified threshold value may also be performed in other ways. In some embodiments, the content provider may obtain the predicted and actual results of previous content requests. The content provider aggregates the actual results of the previous content requests. Using the example above, the actual aggregate result of 10 previous content requests is 1000. The content provider then calculates the initial threshold value by scaling the total actual results by 10% of the predetermined target loss threshold. In the above example, the content provider scales the actual aggregate result of 1000 by the predetermined target loss threshold of 5% to obtain an initial threshold value of 50 (i.e., 5 / 100*1000).

[0059] The content provider sorts the predicted results in ascending order and then uses the order to obtain the corresponding actual result set. Then, starting with the first content request in the list of actual results, the content provider adds the actual results of each request until the sum of the actual results of the analyzed requests exceeds the initial threshold. At this point, the content provider selects the predicted result corresponding to the last actual result whose sum of the predicted results does not exceed the initial threshold as the specified threshold.

[0060] In the example above, the content provider obtains the actual revenue [100, 150, 200, 50, 30, 10, 25, 400, 15, 20] and the corresponding predicted revenue [100, 150, 200, 50, 30, 10, 20, 400, 15, 22] for each of the 10 previous content requests. The content provider sorts the predicted revenue in increasing order to obtain the following dataset: 10, 15, 20, 22, 30, 50, 100, 150, 200, 400. The content provider then uses this order to obtain a dataset of actual revenues corresponding to the actual dataset as follows: 10, 15, 25, 20, 30, 50, 100, 150, 200, 400. Then, starting with the first result in the actual revenue dataset, the content provider adds up the actual revenues for each request until the sum of the actual revenues exceeds the initial threshold of 50. The sum of the actual revenues after the first three requests (i.e., 10+15+25) is 50. When the fourth actual revenue, 20, is added to the sum of the actual revenues, the sum of the actual revenues exceeds the initial threshold of 50 (i.e., 10+15+25+20=70, which is greater than 50). Therefore, the content provider selects 20 as the specified threshold because it is the predicted revenue corresponding to the last actual revenue (25) where the sum of the actual revenues did not exceed the initial threshold of 50. If the results of the content requests can be quantified using user viewing time or interactive time of the content (or in any other appropriate manner) instead of revenue, the above analysis will be the same.

[0061] Based on a new set of content requests (referred to in this specification as holdout requests) and their corresponding results, the specified thresholds may be periodically updated using the above process.

[0062] Note that percentages are used in the above examples, but the target loss may be expressed in other ways, such as total aggregate losses, total losses over a specific period, or other measures of loss. Note also that the thresholds may vary based on the content provided, the content provider, and / or the country in which the content is provided.

[0063] refer to Figure 1 and Figure 2 , the content provider 130 determines whether the priority value meets the specified threshold (at 212). The content provider 130 inputs the priority value generated by the prioritization model 122 and the specified threshold to the throttling engine 124. Based on the priority value and the specified threshold, the throttling engine 124 evaluates whether to provide access to computing resources required to respond to the content request 104. In the above example, the throttling engine 124 compares the priority value 65 (generated by the prioritization model 122) to the specified threshold of 100.

[0064] If the throttling engine 124 determines that the priority value does not meet the specified threshold, the throttling engine denies access to the computing resources required to respond to the request at 222. In the above example, because the priority value 65 does not meet the specified threshold 100, the throttling engine denies access to the computing resources required to identify the advertisement list for the content request.

[0065] Upon determining that the priority value does not meet a specified threshold, the throttling engine 124 may notify the content server 126 not to process the content request 104. In some implementations, the throttling engine 124 may send a message to the content server 126 that includes the content request 104, the extracted features, and a "no" indicator as to whether to provide access to the computing resources in response to the content request. Upon receiving the message, the content server 126 ignores the content request 104 and may respond to the client device 102 with "access denied" in a reply 106. As a result, the client device 102 may simply not display any content. Other ways of notifying the content server not to process the content request include setting a data flag that prevents the content server from processing the content request, or simply not instructing the content server to process the content request (e.g., where the default of the content server is to not process the content request unless explicitly instructed).

[0066] On the other hand, if the throttling engine 124 determines that the priority value meets the specified threshold, the throttling engine 124 provides access to the computing resources needed to respond to the request at 220. In the above example, if the priority value has reached or exceeded the specified threshold of 100, the throttling engine will provide access to the computing resources needed to provide the content in response to the content request.

[0067] In some implementations, upon determining that the priority value meets a specified threshold, the throttling engine 124 may notify the content server 126 to process the content request 104. The throttling engine 124 may send a message to the content server 126 that includes the content request 104, the extracted features, and a "yes" indicator as to whether to provide access to computing resources in response to the request. Upon receiving the message, the content server 126 utilizes computing resources to identify appropriate content (e.g., search results) to respond to the content request 104. The content server 126 then provides the content to the client device 102 in a reply 106. As a result, in the case of search results in a search engine, the client device 102 displays the retrieved search results. Other ways of notifying the content server to process the content request include setting a data flag that triggers the content server to process the content request, or simply not sending any instructions to the content server (e.g., where the default of the content server is to process the content request unless explicitly instructed).

[0068] As above reference Figure 1 and Figure 2 As described, the prioritization model is trained using a set of previous content requests, and a priority value is generated for a particular content request based on the set of previous content requests. In some embodiments, the prioritization model is periodically updated using a new set of content requests, such as reference Figure 3 described.

[0069] Figure 3 is a flow diagram of an example process 300 implemented by a content provider 130 to update a prioritization model 122 .

[0070] After receiving a request for a specific content from the client device 102 (refer to Figure 2 ), or at any time, the content provider 130 may identify a new set of content requests received from a plurality of client devices as hold requests (at 302).

[0071] For these retention requests, the content provider 130 provides access to the required computing resources regardless of any priority value of the retention request (at 304). In some implementations, after the feature extractor 112 component extracts features from the retention request, the feature extractor sends the extracted features and the corresponding retention request to the content server 126, the prioritization model 122, and the data store 116 in parallel.

[0072] For a hold request sent to the prioritization model 122, the prioritization model determines a priority value (as described above with reference to Figure 1 and Figure 2 as described above), and store these priority values ​​in the data store 116.

[0073] For a hold request that the feature extractor 112 sends directly to the content server 126, the content server 126 expends computing resources to identify the content of the extracted features of the hold request. In the case of a hold request, the content server 126 does not need any information from the throttling engine 124 about whether to process the content request. For example, when the feature extractor 112 sends the extracted features directly to the content server, it may also include a tag identifier "hold", which may allow the content server 126 to operate without receiving any instructions from the throttling engine 124.

[0074] The content provider 130 monitors the actual results of the content provided in response to the hold request (at 306). In some embodiments, the priority analyzer 114 monitors the actual results of each hold request processed by the content server 126. For example, when the hold request is being processed, the priority analyzer 114 can have an agent running on the content server 126, and the content provided by the content server 126 passes through the agent before being sent to the client device 102. The priority analyzer 114 extracts the content source (e.g., the server that can host the content provided in response to the content request) and the session identifier (i.e., the identifier of the specific content request 102 for which the content is provided) from the metadata included in the content. The agent of the priority analyzer 114 can then query the content source to obtain the actual result data corresponding to the specific session identifier. Upon receiving the actual result data from the content source, the priority analyzer 114 stores the data for each hold request in the data store 116.

[0075] The content provider 130 updates the prioritization model based on the actual results (at 308). This can be done in three steps. First, the content provider 130 obtains the priority value of the content provided in response to the retention request (at 310). In some embodiments, the priority analyzer 114 obtains the priority value of the retention request stored in the data store 116. For example, the priority analyzer 114 can query the data store 116 for the priority value of the retention request. The priority analyzer can send a unique identifier for each retention request to the data store 116, and the data store 116 can store identifiers of different content requests and corresponding data for each of these requests. Therefore, when a query is received from the priority analyzer 114, the data store 116 uses the identifier to locate the corresponding priority value data stored in the data store 116.

[0076] The content provider 130 compares the priority value with the actual result (at 312) and adjusts the prioritization model based on the difference between the priority value and the actual result (at 314). In some embodiments, the priority analyzer 114 obtains the actual result of each retention request from the data store 116. Then, the priority analyzer 114 compares the priority value with the actual result of the retention request. For example, the priority analyzer 114 determines the difference between the actual result and the priority value of each retention request. The actual result and / or the priority value can be scaled (e.g., multiplied by a numerical value) so that they can be compared. In some embodiments, the priority analyzer 114 can generate an average difference based on the difference between the priority value and the actual result, and then use the average difference to update the prioritization model. For example, the priority analyzer 114 can determine that the average difference between the priority value and the actual result is -50. In other words, the priority analyzer 114 can determine that the actual result is 50 less than the priority value on average. One way that the content provider 130 can update the prioritization model 122 is to scale the output by 50. For example, the content provider 130 can subtract 50 from the final determined output of the current prioritization model. Thus, if the current model would generate a priority value of 100, the updated model would generate a priority value of 50 (ie, 100 minus 50).

[0077] Content provider 130 may further adjust the prioritization model based on the relationship between the target loss percentage and the number of discarded requests that are denied access to computing resources (316). For example, based on content requests received by content provider 130 over time, content provider 130 may maintain a distribution of the number of discarded requests that are denied access to computing resources at a particular target loss percentage.

[0078] For example, although the current request drop rate may be 50%, the content provider 130 may determine based on the distribution that for a target loss percentage of 10%, the desired number of dropped requests should be 70%. The content provider 130 may achieve this optimal request drop rate by adjusting the prioritization model. This is illustrated using a specified threshold of 37.5 based on a 10% target loss percentage and sample data of 10 content requests, where the following priority values ​​are determined by the prioritization model: 15, 20, 25, 30, 35, 40, 45, 50, 55, and 60. Using a specified threshold of 37.5, the first five requests with priority values ​​below 37.5 will be dropped.

[0079] In this example, a 10% target loss percentage results in only a 50% request drop rate (i.e., 5 queries out of a total of 10 queries are dropped). Increasing the request drop rate from 50% to 70% at the same target loss threshold can be achieved by scaling the output of the prioritization model by 10. Therefore, the sample data of 10 content requests will now have the following updated priority values: 5, 10, 15, 20, 25, 30, 35, 40, 45, and 50. Now, applying the specified threshold of 37.5, a 70% request drop rate can be achieved. This is because the first seven requests with priority values ​​below 37.5 are dropped.

[0080] Figure 5 5 is a block diagram of an example computer system 500 that can be used to perform the above operations. System 500 includes a processor 510, a memory 520, a storage device 530, and an input / output device 540. Each of the components 510, 520, 530, and 540 can be interconnected, for example, using a system bus 550. Processor 510 is capable of processing instructions executed within system 500. In some embodiments, processor 510 is a single-threaded processor. In another embodiment, processor 510 is a multi-threaded processor. Processor 510 is capable of processing instructions stored on memory 520 or storage device 530.

[0081] Memory 520 stores information within system 500. In one embodiment, memory 520 is a computer readable medium. In some embodiments, memory 520 is a volatile memory unit. In another embodiment, memory 520 is a non-volatile memory unit.

[0082] The storage device 530 can provide mass storage for the system 500. In some embodiments, the storage device 530 is a computer-readable medium. In various embodiments, the storage device 530 may include, for example, a hard disk device, an optical disk device, a storage device shared by multiple computing devices on a network (e.g., a cloud storage device), or some other mass storage device.

[0083] The input / output device 540 provides input / output operations for the system 500. In some embodiments, the input / output device 540 may include one or more of a network interface device (e.g., an Ethernet card), a serial communication device (e.g., an RS-232 port), and / or a wireless interface device (e.g., and an 802.11 card). In another embodiment, the input / output device may include a drive device configured to receive input data and send output data to other input / output devices, such as a keyboard, a printer, and a display device 560. However, other embodiments may also be used, such as a mobile computing device, a mobile communication device, a set-top TV client device, etc.

[0084] Despite Figure 5 An example processing system is described in the specification, but the subject matter and implementation of the functional operations described in this specification may be implemented in other types of digital electronic circuits, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them.

[0085] The subject matter and the embodiments of the operations described in this specification may be implemented in digital electronic circuits, or in computer software, firmware or hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more computer program instruction modules, encoded on a computer storage medium (or multiple media) for execution by a data processing device or for controlling the operation of a data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. The computer storage medium may be or be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. In addition, although the computer storage medium is not a propagation signal, the computer storage medium may be the source or destination of the computer program instructions encoded in the artificially generated propagation signal. The computer storage medium may also be or be included in one or more separate physical components or media (e.g., multiple CDs, disks or other storage devices).

[0086] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0087] The term "data processing apparatus" includes all kinds of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer, a system on a chip, or a plurality or combination of the foregoing. The apparatus may include a dedicated logic circuit, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment may implement a variety of different computing model infrastructures, such as network services, distributed computing, and grid computing infrastructures.

[0088] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0089] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits, and the apparatus can also be implemented as special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits).

[0090] For example, processors suitable for executing computer programs include general and special microprocessors. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more large-capacity storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or be operably coupled to one or more large-capacity storage devices for storing data, to receive data from one or more large-capacity storage devices or to transmit data to one or more large-capacity storage devices or to receive and transmit data. However, a computer does not need to have such a device. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (such as a universal serial bus (USB) flash drive), just to name a few examples. Devices suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and storage devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuits.

[0091] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display), for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and the input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0092] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component, such as a data server, or that includes a middleware component, such as an application server, or that includes a front-end component, such as a client computer with a graphical user interface or a web browser, through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), interconnected networks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0093] The computing system can include a client and a server. The client and the server are usually remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by means of computer programs running on each computer and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML page) to the client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input from the user). Data generated at the client device (e.g., the result of a user interaction) can be received from the client device at the server.

[0094] Although this specification contains many specific implementation details, these should not be interpreted as limitations on any invention or the scope of possible protection, but rather as descriptions of the features of specific embodiments of specific inventions. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. Moreover, although features may be described above as appearing in certain combinations and even requested to be protected as such at the outset, one or more features from the combination requested for protection can be deleted from the combination in some cases, and the combination requested for protection may be directed to a sub-combination or a variation of the sub-combination.

[0095] Similarly, although operations are described in a particular order in the accompanying drawings, this should not be understood as requiring that the operations be performed in the particular order or sequence shown, or that all of the operations shown be performed, to obtain the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0096] Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. Furthermore, the processes described in the accompanying drawings do not necessarily require the particular order or sequence shown to achieve the desired results. In certain embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method, include: Receiving, by a data processing device, a content request from a client device; extracting features from the content request by a data processing device; inputting, by a data processing device, the features into a prioritization model trained using a set of previous content requests; obtaining, by a data processing device, a priority value of the content request from the prioritization model based on the features input to the model; determining a specified threshold based on a target loss percentage and results of the previous set of content requests; and Throttling, by a data processing device, access to a computing resource for responding to the request based on the specified threshold and the priority value of the content request, comprises: providing access to computing resources in response to the request when the prioritization model outputs a priority value that satisfies the specified threshold; and When the prioritization model outputs a priority value that fails to meet the specified threshold, access to the computing resource is denied.

2. The computer-implemented method of claim 1 , further comprising: include: identifying a set of subsequent content requests received after receiving the content request as hold requests; providing access to computing resources required to respond to the hold request regardless of any priority value of the hold request; monitoring actual results of content provided in response to the retention request; and The prioritization model is updated based on the actual results.

3. The computer-implemented method of claim 2, in, Updating the prioritization model includes: obtaining a priority value of content provided in response to the hold request; comparing the priority value to the actual result; and The prioritization model is adjusted based on a difference between the priority value and the actual result.

4. The computer-implemented method of claim 3, further comprising adjusting the prioritization model based on a relationship between a target loss percentage and a percentage of discarded content requests that are denied access to computing resources.

5. The computer-implemented method of claim 4, in, Based on the features input into the model, obtaining, by the data processing device, the priority value of the content request from the model comprises: using the prioritization model, determining a probability that the content request will generate a positive outcome based on outcomes of requests in the set of previous content requests; and using the prioritization model, determining an expected magnitude of positive results for the content request based on positive results generated by the previous content request; and The probability that the content request generates a positive result is combined by the data processing device with the expected magnitude of the positive result to generate the priority value.

6. The computer-implemented method of claim 1, in, Determining a specified threshold based on the target loss percentage and the results of the previous set of content requests includes: obtaining a result for each request in the set of previous content requests; aggregating the results of each request in the set of previous content requests to generate an aggregated result for the set of previous content requests; and The aggregate result is scaled by the target loss percentage to generate the specified threshold value.

7. The computer-implemented method of claim 2, further comprising updating the specified threshold based on the target loss percentage and a result of the hold request.

8. The computer-implemented method of claim 7, in, Updating the specified threshold based on the target loss percentage and the result of the hold request includes: obtaining a result of each of the hold requests; aggregating the results of each of the hold requests to generate an aggregated result for the hold requests; and The aggregate result is scaled by the target loss percentage to generate the specified threshold value.

9. A system for using computing resources, include: one or more memory devices storing instructions; and One or more data processing apparatus configured to interact with the one or more memory devices and, when executing the instructions, perform operations including: receiving a content request from a client device; extracting features from the content request; inputting the features into a prioritization model trained using a set of previous content requests; obtaining a priority value for the content request from the prioritization model based on the features input to the model; determining a specified threshold based on a target loss percentage and results of the previous set of content requests; and Throttling access to computing resources for responding to the request based on the specified threshold and the priority value of the content request includes: providing access to computing resources in response to the request when the prioritization model outputs a priority value that satisfies the specified threshold; and When the prioritization model outputs a priority value that fails to meet the specified threshold, access to the computing resource is denied.

10. The system according to claim 9, in, The one or more data processing devices are configured to perform operations further comprising: identifying a set of subsequent content requests received after receiving the content request as hold requests; providing access to computing resources required to respond to the hold request regardless of any priority value of the hold request; monitoring actual results of content provided in response to the retention request; and The prioritization model is updated based on the actual results.

11. The system according to claim 10, in, Updating the prioritization model includes: obtaining a priority value of content provided in response to the hold request; comparing the priority value to the actual result; and The prioritization model is adjusted based on a difference between the priority value and the actual result.

12. The system according to claim 11, in, The one or more data processing devices are configured to perform operations further comprising: The prioritization model is adjusted based on a relationship between the target loss percentage and a percentage of discarded content requests that are denied access to computing resources.

13. The system according to claim 12, in, Based on the features input to the model, obtaining a priority value of the content request from the model includes: using the prioritization model, determining a probability that the content request will generate a positive outcome based on outcomes of requests in the set of previous content requests; and using the prioritization model, determining an expected magnitude of positive results for the content request based on positive results generated by the previous content request; and The probability that the content request generates a positive result is combined by the data processing device with the expected magnitude of the positive result to generate the priority value.

14. The system according to claim 9, in, Determining the specified threshold based on the target loss percentage and the results of the previous set of content requests includes: obtaining a result for each request in the set of previous content requests; aggregating the results of each request in the set of previous content requests to generate an aggregated result for the set of previous content requests; and The aggregate result is scaled by the target loss percentage to generate the specified threshold value.

15. The system according to claim 10, in, The one or more data processing devices are configured to perform operations further comprising: The specified threshold is updated based on the target loss percentage and a result of the hold request.

16. The system according to claim 15, in, Updating the specified threshold based on the target loss percentage and the result of the hold request includes: obtaining a result of each of the hold requests; aggregating the results of each of the hold requests to generate an aggregated result for the hold requests; and The aggregate result is scaled by the target loss percentage to generate the specified threshold value.

17. A non-transitory computer readable medium storing instructions which, when executed by one or more data processing devices, cause the one or more data processing devices to perform operations comprising: receiving a content request from a client device; extracting features from the content request; inputting the features into a prioritization model trained using a set of previous content requests; obtaining a priority value for the content request from the prioritization model based on the features input to the model; determining a specified threshold based on a target loss percentage and results of the previous set of content requests; and Throttling access to computing resources for responding to the request based on the specified threshold and the priority value of the content request includes: providing access to computing resources in response to the request when the prioritization model outputs a priority value that satisfies the specified threshold; and When the prioritization model outputs a priority value that fails to meet the specified threshold, access to the computing resource is denied.

18. The non-transitory computer readable medium of claim 17, in, When executed by one or more data processing devices, the instructions cause the one or more data processing devices to perform operations including: identifying a set of subsequent content requests received after receiving the content request as hold requests; providing access to computing resources required to respond to the hold requests regardless of any priority value of the hold requests; monitoring actual results of content provided in response to the retention request; and The prioritization model is updated based on the actual results.

19. The non-transitory computer readable medium of claim 18, in, Updating the prioritization model includes: obtaining a priority value of content provided in response to the hold request; comparing the priority value to the actual result; and The prioritization model is adjusted based on a difference between the priority value and the actual result.

20. The non-transitory computer readable medium of claim 19, in, When executed by one or more data processing devices, the instructions cause the one or more data processing devices to perform operations including: The prioritization model is adjusted based on a relationship between the target loss percentage and a percentage of discarded content requests that are denied access to computing resources.

21. The non-transitory computer readable medium of claim 20, in, Based on the features input to the model, obtaining a priority value of the content request from the model includes: using the prioritization model, determining a probability that the content request will generate a positive outcome based on outcomes of requests in the set of previous content requests; and using the prioritization model, determining an expected magnitude of positive results for the content request based on positive results generated by the previous content request; and The probability that the content request generates a positive outcome is combined with the expected magnitude of the positive outcome to generate the priority value.

22. The non-transitory computer readable medium of claim 17, in, Determining the specified threshold based on the target loss percentage and the results of the previous set of content requests includes: obtaining a result for each request in the set of previous content requests; aggregating the results of each request in the set of previous content requests to generate an aggregated result for the set of previous content requests; and The aggregate result is scaled by the target loss percentage to generate the specified threshold value.

23. The non-transitory computer readable medium of claim 18, in, When executed by one or more data processing devices, the instructions cause the one or more data processing devices to perform operations including updating the specified threshold based on the target loss percentage and the result of the retention request.

24. The non-transitory computer readable medium of claim 23, in, Updating the specified threshold based on the target loss percentage and the result of the hold request includes: obtaining a result of each of the hold requests; aggregating the results of each of the hold requests to generate an aggregated result for the hold requests; and The aggregate result is scaled by the target loss percentage to generate the specified threshold value.