Method and apparatus for training a reinforcement learning system for automated bidding

Abstract

The embodiments of the present specification disclose a method and device for training a reinforcement learning system for automatic bidding. The reinforcement learning system comprises an environment model and an agent, the environment model is used to predict the state distribution of a business environment, the business environment state comprises the budget consumption and the obtained business value of a business party, and the agent is used to determine the current bidding parameter of the business party according to the current environment state. The method comprises: firstly training the environment model by using a plurality of historical samples, wherein any first historical sample comprises a first state of a business environment, a first bidding parameter, and a second state of the business environment after the business environment is changed by using the first bidding parameter under the first state; and secondly training the agent by using a plurality of state-bidding parameter pairs constructed based on the sampling of the environment state space and the bidding parameter space, and the predicted state distribution corresponding to each state-bidding parameter pair obtained by inputting the state-bidding parameter pair into the trained environment model.

Description

Technical Field

[0001] This specification relates to the field of business recommendation technology, and more particularly to a method and apparatus for training a reinforcement learning system for automatic bidding, and an automatic bidding method and apparatus. Background Technology

[0002] With the widespread application of computer and internet technologies, more and more users are choosing e-commerce as part of their lives and shopping. At the same time, many new technology business areas, different from traditional industries, have gradually developed along with the rapid growth of the internet. Among them is the online advertising exchange system.

[0003] The typical working principle of an online advertising exchange system is as follows: A portal website provides advertising space on its webpages to display advertising services. When a user is opening a webpage on the portal website, the portal website auctions the advertising space on the webpage visited by the user on the online advertising exchange system. Different merchants can bid according to the specific circumstances of the portal website. The highest bidder wins the opportunity to display the advertising space, and the winning bid price for this auction is the second highest bid.

[0004] However, due to the short bidding time, typically only tens of milliseconds, and the huge number of display opportunities, advertisers cannot manually bid for each different traffic stream individually and need to rely on automatic bidding algorithms. Therefore, the embodiments in this specification provide a new automatic bidding scheme that can meet higher requirements in practical applications, such as improving the accuracy of automatic bidding and reducing algorithm complexity, etc. Summary of the Invention

[0005] This specification describes a method and apparatus for an automatic bidding reinforcement learning system, which can better meet the needs of practical applications.

[0006] According to a first aspect, a method is provided for training a reinforcement learning system for automatic bidding, the reinforcement learning system comprising an environment model and an agent, the environment model being used to predict the state distribution of a business environment, the state of the business environment including the budget consumption and acquired business value of a business party; the agent being used to determine the current bidding parameters of the business party based on the current environment state; the method comprising:

[0007] The environment model is trained using multiple historical samples; wherein any first historical sample includes a first state of the business environment, a first bid parameter, and a second state to which the business environment changes after adopting the first bid parameter in the first state. The agent is trained using multiple state-bid parameter pairs and the corresponding predicted state distribution obtained by inputting each state-bid parameter pair into the trained environment model; the construction of the multiple state-bid parameter pairs is based on multiple states sampled from the state space of the business environment and multiple bid parameters sampled from the bid parameter space.

[0008] In one embodiment, the budget consumption includes the ratio of consumed budget to total budget.

[0009] In one embodiment, training the environment model using multiple historical samples includes: inputting the first state and a first bidding parameter into the environment model to obtain a first predicted state distribution; determining a training loss based on the probability of the second state corresponding to the first predicted state distribution and the rate of change of the state with respect to the bidding parameter, determined based on the first predicted state distribution; the training loss is negatively correlated with the probability, and further negatively correlated with the rate of change when the rate of change is negative; and updating the model parameters of the environment model using the training loss.

[0010] In one embodiment, the plurality of states are all m states in the state space, and the plurality of bid parameters are all n bid parameters in the bid parameter space; wherein, before training the agent, the method further includes: using the m states and n bid parameters, constructing m*n state-bid parameter pairs as the plurality of state-bid parameter pairs.

[0011] In one embodiment, training the agent using multiple state-bid parameter pairs and a predicted state distribution obtained by inputting each state-bid parameter pair into the environment model includes: updating the state value function in the agent using dynamic programming based on each state-bid parameter pair and the corresponding predicted state distribution; and determining the corresponding state-action value function based on the updated state value function.

[0012] In one specific embodiment, both the state value function and the state-action value function include a reward value in their computational terms. The reward value is the product of a scaling factor and the obtained business value, wherein the scaling factor is negatively correlated with the ratio between the actual cost and the predetermined cost threshold, and the actual cost is calculated based on the budget consumption and the obtained business value.

[0013] In one specific embodiment, both the state value function and the state-action value function include a discount coefficient in their computational terms, and the discount coefficient is set to 1.

[0014] According to the second aspect, an automatic bidding method is provided, executed by a device associated with a business party, including:

[0015] A bidding request is received, instructing the presentation of a business object to a first user on a first platform display slot. Based on the bidding request, the current state of the business environment in which the business party operates is input into the agent to obtain current bidding parameters selected from the bidding parameter space; the agent is trained using the method provided in the first aspect above. Based on the target business object provided by the business party, and the characteristics of the first user and / or the first platform display slot, the business value of the display opportunity is determined. Based on the current bidding parameters and the business value, the bid amount for the display opportunity is determined.

[0016] In one embodiment, the agent includes a state-action value function; wherein, inputting the current state of the business environment in which the business party is located into the agent to obtain the current bid parameter selected from the bid parameter space includes: taking state-bid parameter pairs composed of the current state and each bid parameter in the bid parameter space as inputs to the state-action value function to obtain multiple function values; and determining the bid parameter in the state-bid parameter pair corresponding to the largest function value among the multiple function values ​​as the current bid parameter.

[0017] In one embodiment, determining the business value of the display opportunity based on the target business object provided by the business party, and the characteristics of the first user and / or the first platform, includes: inputting the object characteristics of the target business object, the user characteristics of the first user, and / or the display position characteristics of the first platform display position into a pre-trained object recommendation model to obtain the corresponding predicted recommendation degree, which is used as the business value.

[0018] According to a third aspect, an apparatus is provided for training a reinforcement learning system for automatic bidding, the reinforcement learning system including an environment model and an agent, the environment model being used to predict the state distribution of a business environment, the state of the business environment including the budget consumption and acquired business value of a business party; the agent being used to determine the current bidding parameters of the business party based on the current environment state; the apparatus includes:

[0019] The environment model training module is configured to train the environment model using multiple historical samples; wherein any first historical sample includes a first state of the business environment, a first bidding parameter, and a second state to which the business environment changes after adopting the first bidding parameter in the first state. The agent training module is configured to train the agent using multiple state-bidding parameter pairs and a predicted state distribution obtained by inputting each state-bidding parameter pair into the trained environment model; the construction of the multiple state-bidding parameter pairs is based on multiple states sampled from the state space of the business environment and multiple bidding parameters sampled from the bidding parameter space.

[0020] According to the fourth aspect, an automatic bidding device is provided, integrated into a device associated with a business party, comprising:

[0021] A request receiving module is configured to receive a bidding request, which instructs the display opportunity of a business object to be presented to a first user on a first platform display position. A parameter prediction module is configured to, based on the bidding request, input the current state of the business environment in which the business party is located into an intelligent agent to obtain current bidding parameters selected from a bidding parameter space; the intelligent agent is trained using a device provided by a third party. A value determination module is configured to determine the business value of the display opportunity based on the target business object provided by the business party, and the characteristics of the first user and / or the first platform display position. A bid determination module is configured to determine the bid amount for the display opportunity based on the current bid parameters and the business value.

[0022] According to a fifth aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method provided in the first or second aspect.

[0023] According to a sixth aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and the processor, when executing the executable code, implements the method provided in the first or second aspect.

[0024] Using the methods and apparatus disclosed in the embodiments of this specification, 1) the automatic bidding problem is modeled as a coarse-grained MDP decision process based on the optimal bidding model, which greatly reduces the learning complexity of model-based methods; 2) a simple and effective model-based reinforcement learning framework is proposed, which can avoid cumbersome simulation environment construction, and the distribution of future states output by the environment model includes the uncertainty of the bidding environment, overcoming the inconsistency between offline training and online deployment; 3) a probabilistic modeling method is introduced, and a new monotonic constraint is proposed, which can accurately describe the distribution and non-stationarity of the online display environment in the environment model; 4) bidding is carried out in a model-based programming manner, which enhances the interpretability of bidding. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 The Markov decision process modeled for an automated bidding scenario is illustrated.

[0027] Figure 2 This diagram illustrates the implementation architecture of the automatic bidding scheme based on reinforcement learning disclosed in the embodiments of this specification.

[0028] Figure 3 This diagram illustrates a method flow diagram for training a reinforcement learning system for automatic bidding, as disclosed in an embodiment of this specification.

[0029] Figure 4 This diagram illustrates a flowchart of the automatic bidding method disclosed in the embodiments of this specification.

[0030] Figure 5 This diagram illustrates the apparatus structure for training a reinforcement learning system for automatic bidding, as disclosed in an embodiment of this specification.

[0031] Figure 6 This diagram illustrates the structure of the automatic bidding device disclosed in the embodiments of this specification. Detailed Implementation

[0032] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0033] As mentioned earlier, traffic arrives sequentially over a period of time, and businesses compete with each other to win traffic. It's important to understand that traffic refers to the opportunity to display business information in designated display positions, such as advertising spaces within application or website interfaces; business information can be advertisements for goods or services, or content and information; and business partners include enterprises, organizations, merchants, or online platforms that intend to promote their business.

[0034] After traffic is auctioned, the business entity with the highest bid can be recommended to the corresponding user and gain business value such as clicks or conversions. Considering the business value that traffic brings to the business entity, and the long-term constraint of the business entity's budget, the business entity's optimization goal can be written as:

[0035] (1)

[0036] In equation (1), It is the total traffic. Indicates the i-th flow, Indicates whether the business object is pushed or displayed. This represents the business value obtained by the push service object. It is the cost of recommending business objects. It's the budget.

[0037] Under auction mechanisms such as the two-price mechanism (where the highest bidder receives the opportunity to be displayed, and the winning bid is the second-highest bid), the optimal bidding formula is as follows: (2)

[0038] In equation (2), This represents the scaling parameter, the optimal scaling parameter. It can be obtained by scaling based on historical data. Based on this idea, the optimal bid formula shown in equation (2) can be extended to a more general form, which involves multiple constraints, including the budget, and includes parameters related to these constraints:

[0039] (3)

[0040] In equation (3), This indicates the number of constraints. and It can represent any performance metric. It is a core parameter vector related to multiple constraints; It is an indicator function, indicating the first... Is this constraint related to budget expenditure? It is the first An upper bound for the constraint. It should be noted that... , and These are all hyperparameters, which can be provided in advance by the business party or set in advance by the bidding platform.

[0041] Similarly, the optimal parameter vector It can be calculated based on historical data. However, in practical applications, because a complete traffic sequence can only be collected after a certain period of time (such as a day), it is difficult to obtain. .

[0042] Therefore, the embodiments in this specification propose to utilize an agent in reinforcement learning (RL) to adjust the core parameter vector in the current state using an approximate real-time strategy. .

[0043] Specifically, inspired by the optimal bid formulas (2) and (3), it is proposed to incorporate the core parameters (or bid parameters). The online adjustment model is a coarse-grained Markov decision process (MDP).

[0044] Within a certain period (anepisode) or a preset time cycle, the bidding agent, based on the initial bidding parameters, sequentially modifies... For example, assuming the preset total duration is 1 day, and the interval between any two consecutive modifications is 1 hour, then... =24. For example... Figure 1 As shown, at any t-th time step (or t-th time interval) in the T iterations, the agent observes the current state of the business environment of the business party. Then, actions are taken on the business environment. This action is used to generate new bid parameters. This is used to calculate the bid amount within the t-th time interval. When using new bid parameters... After bidding, the business environment is determined by probability. Transition to a new state and provide feedback to the intelligent agent. This return This can be represented as a reward function. .

[0045] Below, we introduce the following five key components of the MDP bidding decision-making process:

[0046] 1) Status From the business perspective, state space The state in the data should reflect the state information required for the agent to make bidding decisions, including budget consumption, the accumulated winning value, or the current timestep.

[0047] In one example, budget consumption includes the ratio of the budget consumed in the current time period to the total budget, which is the percentage of the budget consumed. In another example, budget consumption includes the total budget in the current time period and the budget consumed. It should be understood that the accumulated business value is the cumulative amount of business value obtained in the current time period, and the current time step corresponds to t in the t-th adjustment of the bidding parameters in the current time period.

[0048] 2) Actions Action Space ,in Represents the bid parameter vector The Middle The value pool of dimensional elements. Including A discrete action, wherein Based on this, at the t-th time step, from the action space... Discrete selection action Used to update bid parameters This can be recorded as Understandably, the action space Each action, or bid parameter, can be preset by staff based on practical experience.

[0049] 3) Returns Because the business's optimization goal is to meet... Maximize total business value under certain constraints Therefore, the reward function It's crucial to consider not only the business value gained by the business stakeholders but also the satisfaction of constraints. For example, if a business generates significant value (such as a large number of clicks) but far exceeds its budget or costs, the reward, or feedback value, will be relatively low. This is understandable given the reward function... It can be provided directly by the business party, etc.

[0050] For example, the reward function is as follows:

[0051] (4)

[0052] (5)

[0053] In formulas (4) and (5), Indicates the scaling factor; This represents the cumulative business value achieved. and These represent the true cost and cost threshold per unit of business value, respectively. This can be calculated using budgeted expenditures and accumulated business value; for example, the ratio of budgeted expenditures to accumulated business value can be used as the true cost. Cost threshold. It can be customized by the business side; and It is a hyperparameter used to control the penalty intensity related to cost constraints.

[0054] 4) Transition probability Because market competition fluctuates with traffic, the win rate of businesses with the same bid will also fluctuate. This specification's embodiments explicitly model the non-stationarity of state transitions and assign transition probabilities... Represented as state The probability distribution on.

[0055] 5) Discount coefficient In scenarios where business objects are displayed online, regardless of whether reward decay is considered, the goal is to maximize the total reward. In other words, the business side doesn't need to weigh short-term and long-term interests. Therefore, the discount factor can be set to... It should be understood that the discount factor can also be set to other values, as long as it is a positive number not greater than 1.

[0056] The above are the five key components of the MPD decision-making process.

[0057] The aim of modeling the MPD decision-making process described above is to obtain a deterministic policy. It defines a bidding strategy for automatic bidding in online display business object scenarios.

[0058] Next, we introduce the reinforcement learning framework for automatic bidding proposed in the embodiments of this specification. For example... Figure 2 As shown, the reinforcement learning system includes an environment dynamics model and a bidding agent. In the first stage of this framework, historical data collected from a Real-Time Bidding (RTB) system is utilized. Construct a parameterized probabilistic environmental dynamic model; in the second stage, learn the optimal bidding strategy based on the environmental dynamic model. , Figure 2 The diagram illustrates how dynamic programming is used to plan the bid and obtain the optimal result. Value function It should be noted that other methods can also be used to learn the optimal strategy. .

[0059] Furthermore, in response to bidding requests initiated by the real-time bidding system, the bidding agent, based on the current environmental state, utilizes the optimal Q-value function. From the action space Select action As the current bid parameter And then according to Calculate the bids from participating businesses and enter into an auction. For example, under a two-price mechanism, the business with the highest bid gets the display opportunity and only needs to pay the second-highest price.

[0060] The implementation steps under the above framework are described below with reference to more embodiments.

[0061] Figure 3 This diagram illustrates a method flow diagram for training a reinforcement learning system for automatic bidding, as disclosed in embodiments of this specification. The reinforcement learning system includes an environment model for predicting the state distribution of the business environment in which the business operates. This state distribution refers to the probability distribution of environment states. For an introduction, please refer to the aforementioned related content. The reinforcement learning system also includes a competitive agent, used to determine the current bidding parameters of the business party based on the current environmental state.

[0062] Figure 3 The execution entity of the method shown can be any device, platform, server, or equipment cluster with computing and processing capabilities. For example, if a business entrusts a bidding platform to automatically submit bids, the execution entity of this method can be the bidding platform. Figure 3 As shown, the method includes the following steps:

[0063] Step S310: Train the environment model using multiple historical samples. Any first historical sample includes the first state of the business environment, the first bidding parameter, and the second state the business environment changes to after adopting the first bidding parameter in the first state. It should be noted that the terms "first" in "first historical sample," and similar terms like "second" in the text, are used to distinguish similar items and do not serve any ranking or other purpose. Furthermore, the environment model can be implemented based on machine learning algorithms such as neural networks (NNs).

[0064] Let's denote the sample set consisting of multiple historical samples as... ,in , and Representing any number of... The first state, the first bid parameter, and the second state in each sample. This represents the total number of historical samples.

[0065] In the RTB system, the business environment status This includes budget consumption and business value, which accumulate gradually. Based on this, environmental models can be used to predict the incremental distribution of environmental states within a specific time interval. Furthermore, the next state depends on the previous state and the predicted incremental distribution.

[0066] In one embodiment, for any first A sample, including the first state and the first bid parameter The common input environment model yields the predicted increment distribution. Then, based on the probability of the corresponding label increment in the predicted increment distribution, the training loss is determined. This training loss is negatively correlated with the probability; that is, it is desirable for the probability to be as high as possible. The label increment can be the difference between the second state and the first state, i.e.: .

[0067] In one specific embodiment, assuming the output of the environment model follows a log-normal distribution, the negative logarithmic predicted probability can be considered as part of the loss function. For example, the formula for calculating the training loss includes:

[0068] (6)

[0069] In another specific embodiment, it is assumed that the output of the environment model follows a multivariate normal distribution, and the mean vector of this distribution... and diagonal covariance matrix The parameters involved are as follows and Thus, the parameters of the environmental model are obtained. The predicted increment can be denoted as a multivariate Gaussian distribution: At this point, the loss function represented by formula (6) becomes:

[0070]

[0071] (7)

[0072] The above takes into account the uncertainty of matching the predicted results with the actual actions when determining the training loss.

[0073] In another embodiment, a pattern observed during the bidding process is also considered: an increase in bid price typically leads to an increase in both budget and win rate. Therefore, a method is proposed to optimize the loss function by ensuring the monotonicity of this pattern. Specifically, this is based on the predicted increment distribution. Determine the state Depending on the bid parameters The rate of change is determined, and when the rate of change is determined to be less than 0, a loss term is added to the training loss so that the training loss is negatively correlated with the rate of change, that is, the closer the rate of change is to 0, the better.

[0074] For example, assuming the output of the environmental model follows a multivariate normal distribution, the loss term considering monotonicity can be denoted as:

[0075] (8)

[0076] In one example, the final loss function could be adopted. In the form of, It is a weight hyperparameter with a value range of (0,1).

[0077] After determining the training loss above, the parameters in the environment model can be updated using the training loss. This process yields a trained environment model. In this way, an environment model can be trained to predict the distribution of state increments. The increment distribution predicted based on the previous state can then be added to the previous state to obtain the state distribution for the next state. The relationship between the increment distribution and the state distribution can be expressed as: .

[0078] In fact, the trained environment model can also directly predict the state distribution of the next state, simply by changing the labels during training from incremental labels to state labels. For example, the second state can be directly used. As a status label.

[0079] In one embodiment, a first state and a first bid parameter can be input into an environment model to obtain a first predicted state distribution. Then, based on the probability of the second state corresponding to the first predicted state distribution, a training loss is determined, which is negatively correlated with that probability. Further, in a specific embodiment, the training loss is also determined based on the rate of change of the state with respect to the bid parameter. This rate of change can be calculated based on the first predicted state distribution. When the rate of change is negative, the training loss is negatively correlated with the rate of change.

[0080] It should be understood that for an introduction to the environment model for training the direct prediction of state distribution, please refer to the aforementioned introduction to the environment model for training the prediction of incremental distribution, which will not be repeated here.

[0081] Therefore, a trained environment model can be obtained using supervised learning with historical data. Based on the trained environment model, the interaction between the agent and the environment can be realized to train a bidding strategy. .

[0082] In step S320, the agent is trained using multiple state-bid parameter pairs and the predicted state distribution obtained by inputting each state-bid parameter pair into the trained environment model; wherein the construction of multiple state-bid parameter pairs is based on multiple states sampled from the state space of the business environment and multiple bid parameters sampled from the bid parameter space.

[0083] For constructing the aforementioned multiple state-bid parameter pairs, in one embodiment, all m states in the state space and all n bid parameters in the bid parameter space can be sampled, and m*n state-bid parameter pairs can be obtained through exhaustive combination. In another embodiment, the state-bid pairs are constructed through multiple random samplings.

[0084] In one implementation, dynamic programming can be used to train the agent. It's important to understand that the core idea of ​​dynamic programming is to break down the problem into subproblems, remembering past mistakes and reducing redundant computations. The following section describes how to break down the problem into subproblems to obtain the state-value function that needs to be iterated. The conception process will be introduced.

[0085] Specifically, after obtaining a trained environment model, a series of bidding trajectories can be obtained by sequentially predicting bidding results under different strategies. Based on these bidding trajectories, the agent's current optimal strategy can be determined by maximizing the expected reward of a successful bid.

[0086] (9)

[0087] Based on formula (9), a dynamic programming approach is proposed to estimate the state-action value function. See formula (10), which represents the initial state. Take action And implement strategies And the expected reward:

[0088] (10)

[0089] in, ; The state-value function represents the state in the initial state. The following strategy The expected return is calculated as follows:

[0090] (11)

[0091] Similarly, based on state-value functions The state can be determined. The optimal strategy is as follows:

[0092] (12)

[0093] To solve the problem in formula (12) For the problem of finding integrals, a discretized environment state is proposed, and based on the discretized state space... The optimal value function is calculated using the following formula (13). Approximate value:

[0094] (13)

[0095] In equation (13), the return value It can be based on state It is obtained by calculating the return function R.

[0096] Thus, the state value function is obtained. The iterative calculation formula is as follows. Below, we will introduce how to obtain the optimal strategy of the agent through offline training based on the trained environment model and formula (13), using pseudocode as an example. It is embodied in the optimal state-action value function. The process is as follows. The pseudocode is as follows:

[0097] 1 for each do

[0098] 2. Initialization ;

[0099] 3 end for

[0100] 4 for to do

[0101] 5 for each pair do

[0102] 6. State distribution based on the output of the environment model Get the estimated

[0103] 7. Iteration based on formula (13)

[0104] 8 end for

[0105] 9. Determine the action based on the following formula:

[0106] 10

[0107] 11 end for

[0108] The above describes training an agent using dynamic programming. Further, based on the determined optimal state-action value function... The state can be The optimal action is determined as follows:

[0109] (14)

[0110] Alternatively, this step can be implemented in other ways, such as using the Dyna class method or the Analytical gradient method to train the agent.

[0111] In summary, the method for training a reinforcement learning system for automatic bidding disclosed in the embodiments of this specification: 1) models the automatic bidding problem as a coarse-grained MDP decision process based on the optimal bidding model, which greatly reduces the learning complexity of model-based methods; 2) proposes a simple and effective model-based reinforcement learning framework that avoids cumbersome simulation environment construction, and the distribution of future states output by the environment model includes the uncertainty of the bidding environment, overcoming the inconsistency between offline training and online deployment; 3) introduces a probabilistic modeling method and proposes a new monotonic constraint that can accurately describe the distribution and non-stationarity of the online display environment in the environment model; 4) adopts a model-based planning approach for bidding, enhancing the interpretability of bidding.

[0112] After completing the training of the reinforcement learning system, the agent within it can be used to predict online actions.

[0113] Figure 4 This diagram illustrates a flowchart of an automatic bidding method disclosed in an embodiment of this specification, executed by a device associated with a business entity. In one example, the business entity has usage rights to the device. In another example, the business entity delegates bidding to a bidding platform, and accordingly, the device may belong to the bidding platform. It is understood that the device can be any apparatus with computing and processing capabilities, such as a server. Figure 4 As shown, the method includes the following steps:

[0114] Step S410: Receive a bidding request, which instructs the first user to be presented with a display opportunity for a business object on the first platform display position; Step S420: Based on the bidding request, input the current state of the business environment in which the business party is located into the intelligent agent to obtain the current bidding parameters selected from the bidding parameter space; Step S430: Based on the target business object provided by the business party, and the characteristics of the first user and / or the first platform display position, determine the business value of the display opportunity; Step S440: Based on the current bidding parameters and the business value, determine the bid amount for the display opportunity.

[0115] The steps above are explained in detail below:

[0116] First, in step S410, a bidding request is received, which indicates the opportunity to display a business object to the first user on the first platform display position.

[0117] For example, the first platform display position can be an advertising space or content section in a website or application; the display opportunity limits the form of the business object, such as text, image, video or audio.

[0118] Then, in step S420, based on the bidding request, the current state of the business environment in which the business party is located is input into the intelligent agent to obtain the current bidding parameters selected from the bidding parameter space.

[0119] In one embodiment, the agent includes an optimal state-action value function. Based on this, this step can be implemented as follows: The state-bid parameter pairs, composed of the current state and each bid parameter in the bid parameter space, are used as inputs to the state-action value function to obtain multiple function values; then, the bid parameter in the state-bid parameter pair corresponding to the largest function value among the multiple function values ​​is determined as the current bid parameter. For further information, please refer to formula (14).

[0120] In another embodiment, the agent includes a parameterized policy model. , the current state Input policy model The optimal action can be obtained directly from the output. .

[0121] From the above, we can see the current bid parameters. .

[0122] In step S430, based on the target business object provided by the business party and the characteristics of the first user and / or the first platform display position, the business value of the display opportunity is determined. It should be noted that step S420 can be executed before or after step S430, or simultaneously; this embodiment of the specification does not limit this.

[0123] In one embodiment, the bidding request includes the user ID of the first user. Accordingly, the user's authorized feature data, such as consumption preferences, can be accessed from the user data center based on the user ID. In another embodiment, the bidding request includes features of the first platform's display position, such as restricted display formats and platform category (e.g., entertainment, technology).

[0124] In one embodiment, this step includes: inputting the object features of the target business object, the user features of the first user, and / or the display position features of the first platform display position into a pre-trained object recommendation model to obtain the corresponding predicted recommendation score, which serves as the business value. For example, the target business object is the target advertisement, the first platform display position is the first advertisement position, the object recommendation model is a click-through rate prediction model, and the predicted recommendation score is the predicted click-through rate; the object recommendation model is a machine learning model, which can be implemented based on neural networks, decision trees, etc.

[0125] In another embodiment, this step includes: matching the object characteristics of the target business object with the user characteristics of the first user, and calculating the proportion of the number of successfully matched feature items to the total number of feature items as the business value.

[0126] Therefore, the business value of the display opportunity for a specific business can be determined. .

[0127] S440, based on current bid parameters and business value Determine the bid amount for the display opportunity. .

[0128] For example, the bid amount can be calculated using the formula (2) or (3) above. .

[0129] Therefore, it is possible to use a trained reinforcement learning system to achieve automatic bidding.

[0130] Corresponding to the above training method and bidding method, the embodiments of this specification also disclose a training device and a bidding device.

[0131] Figure 5This diagram illustrates a device structure for training a reinforcement learning system for automatic bidding, as disclosed in embodiments of this specification. The reinforcement learning system includes an environment model and an agent. The environment model predicts the state distribution of a business environment, including the business party's budget consumption and acquired business value. The agent determines the business party's current bidding parameters based on the current environment state. Figure 5 As shown, the training device 500 includes:

[0132] The environment model training module 510 is configured to train the environment model using multiple historical samples; wherein any first historical sample includes a first state of the business environment, a first bidding parameter, and a second state to which the business environment changes after adopting the first bidding parameter in the first state. The agent training module 520 is configured to train the agent using multiple state-bidding parameter pairs and a predicted state distribution obtained by inputting each state-bidding parameter pair into the trained environment model; the construction of the multiple state-bidding parameter pairs is based on multiple states sampled from the state space of the business environment and multiple bidding parameters sampled from the bidding parameter space.

[0133] In one embodiment, the budget consumption includes the ratio of consumed budget to total budget.

[0134] In one embodiment, the environment model training module 510 is specifically configured to: input the first state and the first bid parameter into the environment model to obtain a first predicted state distribution; determine a training loss based on the probability of the second state corresponding to the first predicted state distribution and the rate of change of the state with respect to the bid parameter, as determined based on the first predicted state distribution; the training loss is negatively correlated with the probability, and also negatively correlated with the rate of change when the rate of change is negative; and update the model parameters of the environment model using the training loss.

[0135] In one embodiment, the plurality of states are all m states in the state space, and the plurality of bid parameters are all n bid parameters in the bid parameter space; the training device 500 further includes: a state-action pair construction unit 530, configured to: construct m*n state-bid parameter pairs using the m states and n bid parameters, as the plurality of state-bid parameter pairs.

[0136] In one embodiment, the agent training module 520 is specifically configured to: update the state value function in the agent using dynamic programming based on the state-bid parameter pairs and the corresponding predicted state distribution; and determine the corresponding state-action value function based on the updated state value function.

[0137] In one specific embodiment, both the state value function and the state-action value function include a reward value in their computational terms. The reward value is the product of a scaling factor and the obtained business value, wherein the scaling factor is negatively correlated with the ratio between the actual cost and the predetermined cost threshold, and the actual cost is calculated based on the budget consumption and the obtained business value.

[0138] In one specific embodiment, both the state value function and the state-action value function include a discount coefficient in their computational terms, and the discount coefficient is set to 1.

[0139] Figure 6 This diagram illustrates the structure of an automatic bidding device disclosed in an embodiment of this specification, integrated into a device associated with a business partner. For example... Figure 6 As shown, the bidding device 600 includes:

[0140] The request receiving module 610 is configured to receive bidding requests, which indicate the opportunity to display a business object to a first user on a first platform display position. The parameter prediction module 620 is configured to input the current state of the business environment in which the business party is located into the intelligent agent based on the bidding request, and obtain the current bidding parameters selected from the bidding parameter space. The value determination module 630 is configured to determine the business value of the display opportunity based on the target business object provided by the business party, and the characteristics of the first user and / or the first platform display position. The bid determination module 640 is configured to determine the bid amount for the display opportunity based on the current bid parameters and the business value.

[0141] In one embodiment, the agent includes a state-action value function; the parameter prediction module 620 is specifically configured to: take the state-bid parameter pairs composed of the current state and each bid parameter in the bid parameter space as inputs to the state-action value function to obtain multiple function values; and determine the bid parameter in the state-bid parameter pair corresponding to the largest function value among the multiple function values ​​as the current bid parameter.

[0142] In one embodiment, the value determination module 630 is specifically configured to: input the object features of the target business object, the user features of the first user, and / or the display features of the first platform display position into a pre-trained object recommendation model to obtain the corresponding predicted recommendation degree, which is used as the business value.

[0143] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 3 or Figure 4 The method described.

[0144] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements a combination... Figure 3 or Figure 4 The methods described herein. Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0145] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method of training a reinforcement learning system for automated bidding, the reinforcement learning system comprising an environment model and a bidding agent, the environment model for predicting a state distribution of a business environment, the state of the business environment comprising a budget consumption of a business party and a realized business value. The bidding agent is configured to determine a current bidding parameter of the service provider according to a current state of the environment; the method comprises: training the environment model using a plurality of historical samples; wherein any first historical sample comprises a first state of the service environment, a first bidding parameter, and a second state of the service environment after the service environment is changed by using the first bidding parameter in the first state; training the bidding agent using a plurality of state-bidding parameter pairs and a predicted state distribution corresponding to each state-bidding parameter pair obtained by inputting the state-bidding parameter pair into the trained environment model; the plurality of state-bidding parameter pairs are constructed based on a plurality of states sampled from a state space of the service environment and a plurality of bidding parameters sampled from a bidding parameter space.

2. The method of claim 1, wherein, The budget consumption comprises a ratio of a consumed budget to a total budget.

3. The method of claim 1, wherein, Training the environment model using a plurality of historical samples comprises: inputting the first state and the first bidding parameter into the environment model to obtain a first predicted state distribution; determining a training loss according to a probability of a second state in the first predicted state distribution and a rate of change of the state with respect to the bidding parameter determined based on the first predicted state distribution; the training loss is negatively correlated with the probability, and is negatively correlated with the rate of change when the rate of change is negative; updating model parameters of the environment model using the training loss.

4. The method of claim 1, wherein, The plurality of states are all m states in the state space, and the plurality of bidding parameters are all n bidding parameters in the bidding parameter space; wherein, before training the bidding agent, the method further comprises: constructing m*n state-bidding parameter pairs as the plurality of state-bidding parameter pairs using the m states and the n bidding parameters.

5. The method of claim 1 or 4, wherein, Training the bidding agent using a plurality of state-bidding parameter pairs and a predicted state distribution obtained by inputting each state-bidding parameter pair into the environment model comprises: updating a state value function in the bidding agent in a dynamic programming manner based on the each state-bidding parameter pair and the corresponding predicted state distribution; determining a corresponding state-action value function based on the updated state value function.

6. The method of claim 5, wherein, The state value function and the state-action value function both comprise a reward value in an operation term thereof, the reward value being a product of a scaling coefficient and the obtained service value, wherein the scaling coefficient is negatively correlated with a ratio between an actual cost and a predetermined cost threshold, and the actual cost is calculated based on the budget consumption and the obtained service value.

7. The method of claim 5, wherein, The state value function and the state-action value function both comprise a discount coefficient, and the discount coefficient is set to 1.

8. An automatic bidding method performed by a device associated with a service provider, comprising: receiving a bidding request indicating a display opportunity of a service object to a first user on a first platform display position; inputting a current state of a service environment in which the service provider is located into a bidding agent to obtain a current bidding parameter selected from a bidding parameter space according to the bidding request; the bidding agent is trained by the method of claim 1. determine a business value of the display opportunity based on the target business object provided by the business party and a characteristic of the first user and / or the first platform display position; determine a bid amount for the display opportunity based on the current bid parameter and the business value.

9. The method of claim 8, wherein, The bid agent comprises a state-action value function; wherein the current state of the business environment in which the business party is located is input into the bid agent to obtain the current bid parameter selected from the bid parameter space, comprising: input the state-bid parameter pairs composed of the current state and each bid parameter in the bid parameter space into the state-action value function respectively to obtain a plurality of function values; determine the bid parameter in the state-bid parameter pair corresponding to the maximum function value in the plurality of function values as the current bid parameter.

10. The method of claim 8, wherein, determine a business value of the display opportunity based on the target business object provided by the business party and a characteristic of the first user and / or the first platform display position, comprising: input the object characteristic of the target business object, the user characteristic of the first user, and / or the display position characteristic of the first platform display position into a pre-trained object recommendation model to obtain a corresponding predicted recommendation degree as the business value.

11. An apparatus for training a reinforcement learning system for automated bidding, the reinforcement learning system comprising an environment model and a bidding agent, the environment model for predicting a state distribution of a business environment, the state of the business environment comprising a budget consumption of a business party and a realized business value; The bid agent is configured to determine the current bid parameter of the business party according to the current environment state; and the device comprises: an environment model training module configured to train the environment model using a plurality of historical samples; wherein any first historical sample comprises a first state of a business environment, a first bid parameter, and a second state of the business environment after the business environment is changed by using the first bid parameter in the first state; a bid agent training module configured to train the bid agent using a plurality of state-bid parameter pairs and predicted state distributions obtained by inputting each state-bid parameter pair into the trained environment model; the plurality of state-bid parameter pairs are constructed based on a plurality of states sampled from a state space of the business environment and a plurality of bid parameters sampled from a bid parameter space.

12. An automatic bid device integrated in a device associated with a business party, comprising: a request receiving module configured to receive a bid request indicating a display opportunity of displaying a business object to a first user on a first platform display position; a parameter prediction module configured to input a current state of a business environment in which a business party is located into a bid agent according to the bid request to obtain a current bid parameter selected from a bid parameter space; the bid agent is trained by using the device of claim 11; a value determination module configured to determine a business value of the display opportunity based on a target business object provided by the business party and a characteristic of the first user and / or the first display position; a bid determination module configured to determine a bid amount for the display opportunity based on the current bid parameter and the business value.

13. A computer readable storage medium having stored thereon a computer program, wherein, When the computer program is executed in the computer, the computer is caused to perform the method of any one of claims 1-10.

14. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and the processor executes the executable code to implement the method in any one of claims 1-10. The memory stores executable code, and the processor executes the executable code to implement the method in any one of claims 1-10. The memory stores executable code, and the processor executes the executable code to implement the method in any one of claims 1-10. The memory stores

Images