Negotiation agent method based on deep reinforcement learning

Abstract

The invention provides a negotiation agent method based on deep reinforcement learning, which learns a strategy through a historical negotiation data set, and comprises the following steps: collecting historical negotiation data through an automatic negotiation platform, and constructing a data set for learning; learning an effective negotiation strategy for the intelligent agent by using an offline data set; and the learned strategy is quickly and finely adjusted through interaction in an online environment and adapts to the change of the preference or strategy of the opponent. According to the method, the problem of high online interaction training cost in the automatic negotiation field is effectively solved, a universal offline initial strategy is learned at the same time, and the negotiation agent can adapt to the change of online opponent preferences or strategies.

Description

Technical Field

[0001] The present invention relates to the technical field of negotiation agent methods, and specifically to a negotiation agent method based on deep reinforcement learning. Background Art

[0002] Negotiation is a process in which different interested parties exchange offers to explore the possibility of achieving mutual benefit, resolving conflicts, or finding mutually acceptable solutions. Therefore, negotiation can serve as a fundamental and powerful mechanism for managing conflicts. However, for humans, this mechanism can be time-consuming and costly. Due to the advantages of automated negotiation over non-computerized negotiation, such as reducing the workload of human negotiators and achieving better results by compensating for the limitations of human computing and reasoning capabilities, automated negotiation has become an important research direction in multi-agent systems in the past decade.

[0003] Reinforcement learning (RL) is a powerful learning paradigm for control tasks. Specifically, RL can automatically acquire approximately optimal behavioral skills (represented by a policy) for a given task. The successful application of RL algorithms in different fields such as natural language processing, computer vision, and complex games has also triggered explorations in automated negotiation. Despite the remarkable progress achieved so far, however, current work is affected by various restrictive assumptions: 1) Traditional RL methods for negotiation usually focus on online learning from active interactions with the environment (i.e., everything in the negotiation scenario, including the opponent and the domain) to iteratively collect data for policy improvement. However, such online learning is of limited value and is often impractical for negotiation, mainly because data collection based on online interactions is expensive. For example, training an RL agent from scratch to compete with a negotiation partner in an e-commerce scenario may result in a large number of unacceptable results and low-quality customer experience. Although previous methods partially alleviated this problem by training with an opponent simulator (i.e., an agent that applies a known policy), it is often difficult to fully capture the opponent's negotiation strategy in real-world situations because user states and behaviors are uncertain, the environment is noisy, and negotiators aim to hide information related to their strategies to prevent the opponent from exploiting them. Therefore, designing autonomous agents that can learn effective negotiation strategies without online interaction remains a major open problem. Another important challenge is 2) adapting to the opponent's behavior. Since online RL is of limited value for automated negotiation, a key question is whether data collected from previous negotiation sessions can be effectively used by the agent to learn its negotiation skills. In particular, can the agent learn and adapt its negotiation strategy based on offline datasets when the opponent's preferences or strategies change?

[0004] Current RL-based automated negotiation agents have the characteristic of actively interacting with opponents because they focus on online RL methods to train their agents. This often leads to the implementation of simple but impractical negotiation strategies. This is mainly because training RL agents requires a lot of interaction with opponents, but in real-world applications, the data collection process for agents can be expensive and difficult. The recent emergence of offline RL methods provides a new option for learning policies directly from fixed, pre-collected datasets. An effective strategy is to first pre-train a preliminary policy using offline reinforcement learning or offline imitation learning, and then use online reinforcement learning to further fine-tune the performance of this policy. The advantage of this method is that it not only reduces the dependence of offline policy learning on the quality of the dataset, but also provides a better starting strategy for online reinforcement learning.

[0005] A key challenge in offline RL is the distribution shift problem, which is caused by counterfactual queries of out-of-distribution (OOD) samples by function approximators (e.g., value functions, policy networks). It is desirable to ensure that the learned policy remains close to the behavioral policy during training and to mitigate the impact of out-of-distribution actions during training. However, in the context of offline-to-online RL, it is crucial to consider the future online optimization process. Methods such as those that penalize the Q-values ​​of out-of-distribution actions directly penalize out-of-distribution actions in the offline phase, which may lead to severe underestimation of the Q-values ​​of these unknown actions. This may lead to severe bootstrapping errors during fine-tuning, thus destroying the good initial policy obtained through offline RL. However, the focus of the online optimization phase is to explore these unknown actions. This discrepancy may hinder the accurate evaluation of these actions during the online optimization process, thereby hindering the decision of the optimal action. Therefore, is it possible to develop a method to prevent overoptimism about unfamiliar actions in new states during the initial training phase and accelerate the ability of fine-tuned agents to adapt to changing environments? Summary of the invention

[0006] In order to solve the above problems, the present invention proposes a negotiation agent method based on deep reinforcement learning, which learns strategies through historical negotiation data sets, including: collecting historical negotiation data through an automatic negotiation platform to construct a dataset for learning; using offline datasets to learn effective negotiation strategies for the agent; and quickly fine-tuning the learned strategies and adapting to changes in the opponent's preferences or strategies through online interactions. The present invention effectively solves the problem of high cost of online interactive training in the field of automatic negotiation, and at the same time learns a general offline initial strategy, and the negotiation agent can adapt to changes in the online opponent's preferences or strategies.

[0007] The present invention provides a negotiation agent method based on deep reinforcement learning, the method comprising:

[0008] S100, collecting historical negotiation data through the automatic negotiation platform and constructing a data set for learning;

[0009] S200, using an offline data set to learn an effective negotiation strategy for the agent;

[0010] S300, quickly fine-tune learned strategies and adapt to changes in opponent preferences or strategies through interaction in an online environment.

[0011] According to a negotiation agent method based on deep reinforcement learning provided by the present invention, historical negotiation data is collected through an automatic negotiation platform to construct a data set for learning, which specifically includes:

[0012] S110, build a negotiation environment;

[0013] S120, construct the state space, action space and reward function settings in the reinforcement learning training process, and define the Markov decision process;

[0014] S130, using different strategies to collect intelligent agents and opponent intelligent agents to negotiate, record the offers and acceptance / rejection actions of both parties in each round of negotiation, and collect and process historical negotiation data sets.

[0015] According to a negotiation agent method based on deep reinforcement learning provided by the present invention, the negotiation environment is constructed, and the negotiation domain consists of a domain description and the preference files of both parties. The domain description is a description of the legal topics and the value of each topic. Each automatic negotiation domain contains a mapping, in which each key is a negotiation topic, and the legal values ​​of the corresponding topics form a set. Formally, a domain contains n topics: D = (I1, ..., I n ), each topic I contains k values: Based on this, an agent can make an offer ω, which selects the corresponding value from each issue (the selected value is represented by c), represented as a value vector: All the options constitute the negotiation space Ω. The preference profile of the domain determines the utility function, and the utility function of agent i maps an offer ω from the outcome space Ω to a real number in [0,1].

[0016]

[0017] where ω is a vector representation of the quotes, where each issue corresponds to a value, v jk represents the kth value of topic j, represents the evaluation function of agent i for topic j, which maps the value corresponding to the topic to the real number range [0,1]; Represents the preference weight of agent i for issue j.

[0018] According to a negotiation agent method based on deep reinforcement learning provided by the present invention, the definitions of the state space, action space and reward function are designed based on the agent's own perspective, and the state s at time t t The definition of the state space takes two factors into account. First, time, because if no agreement can be reached before the deadline, the negotiation will fail. Second, the negotiation history of both agents has an important influence on the decision of the agents. Therefore, the state space is defined as follows:

[0019]

[0020] Where T max Indicates the maximum round of current negotiation. and They represent the offerω proposed by the opponent and the agent at time tn respectively. s represents the utility function of the agent.

[0021] The action space is composed of [u r ,1] range. So the action at time t is defined as: in Indicates the utility value of the next offer. In order to get the real offer, we define an inverse utility function Map a utility r to the best choice of an offer ω by estimating the opponent's utility. Formally, the inverse utility function is defined as follows:

[0022]

[0023] Where U' o Represents the opponent's utility function estimated based on the distribution frequency of the opponent's historical bids.

[0024] The reward function is specifically defined as when an agreement is reached, the agent will receive a positive reward, and if an agreement cannot be reached before the deadline, it will receive a penalty of -1. Formally, the reward function is defined as follows:

[0025]

[0026] The Markov decision process (MDP) is defined as a 5-tuple, namely Where S is the state space, A is the action space, P:S×A→S' is the transition function, r(s,a) is the reward function, γ∈[0,1) is the discount issue, and the strategy is a distribution π(a|s), which means that in the current state s t Take action a t The goal of the RL agent is to find a solution that maximizes the expected return strategy, everything in the negotiation scenario, including the opponent, is considered as the environment.

[0027] According to a negotiation agent method based on deep reinforcement learning provided by the present invention, the offline data set is used to learn an effective negotiation strategy for the agent, specifically including:

[0028] S210 uses conservative Q learning (CQL) to train the policy network and action value network, where

[0029] Explicitly penalize Q-values ​​of actions that do not exist in the offline dataset during training of the action-value network;

[0030] S220 builds an integrated strategy to construct an offline pre-trained automatic negotiation agent.

[0031] According to a negotiation agent method based on deep reinforcement learning provided by the present invention, the conservative Q learning (CQL) is used to train the policy network and the action value network, wherein the Q value of the action that does not exist in the offline data set is explicitly penalized during the training of the action value network; specifically, CQL pessimistically evaluates the current strategy and makes its Q function the lower bound of the true value. The method is based on the SAC framework, and the optimization objective of its value function is specifically expressed as follows:

[0032]

[0033] Among them, α is the weight issue. The first two terms of the above formula constitute a regularization term. The first term aims to prevent the overestimation of the Q value of the OOD action by minimizing the Q value under the policy, and the second term is balanced by maximizing the Q value of the action in the dataset after the behavior policy. The third term is the standard TD error, which measures the error between the predicted value and the more realistic value, and is represented by the Bellman operator Composition, among which,

[0034]

[0035] In addition, the optimization objectives of the negotiation strategy are as follows:

[0036]

[0037] where π φ is the strategy function of the negotiation agent, is the value function of the agent taking action a under the price adjustment of state s, and α is the temperature hyperparameter, which is used to control the balance between exploration and exploitation in the policy function.

[0038] According to a negotiation agent method based on deep reinforcement learning provided by the present invention, the construction of the integration strategy, the construction of the offline pre-trained automatic negotiation agent, the value network Q of the actor-critic network corresponding to the agent is as follows:

[0039]

[0040] The corresponding policy network π is as follows:

[0041]

[0042] in

[0043] According to a negotiation agent method based on deep reinforcement learning provided by the present invention, the method of quickly fine-tuning the learned strategy and adapting to the changes in the opponent's preferences or strategies through interaction in an online environment specifically includes:

[0044] S310, using the strategy network obtained by offline pre-training to initialize the online negotiation strategy intelligence

[0045] Entity, construct its optimization goal;

[0046] S320, introducing a priority sampling component for balanced experience replay, which utilizes online data by sampling offline data related to the current strategy and generates corresponding optimization targets;

[0047] S330: Based on the optimization target, update the strategies of the intelligent agent and the balance experience playback component.

[0048] According to a negotiation agent method based on deep reinforcement learning provided by the present invention, the online negotiation strategy agent is initialized using the strategy network obtained by offline pre-training, and its optimization goal is constructed. The agent is based on the SAC framework, and the optimization goal of its value function is specifically as follows:

[0049]

[0050] where π φ is the probability distribution of the agent selecting action a in the current state s, r(s,a) is the reward function, is an online dataset, Q θ (s,a) is the value function corresponding to the agent’s strategy.

[0051] The negotiation strategy of the agent has the following optimization objectives:

[0052]

[0053] in is the entropy, α>0 is the temperature parameter, γ is the discount issue, rt is the reward function at timestamp t.

[0054] According to a negotiation agent method based on deep reinforcement learning provided by the present invention, the priority sampling component of balanced experience replay is introduced, and this component uses the density ratio ω(s, a): = d on (s,a) / d off (s, a) measures the linearity of the sample. In practice, this is done by training a neural network called the density ratio estimator ω ψ (s, a) to estimate the density ratio. The density ratio estimator ω ψ The training of (s,a) uses the variational representation of f divergence, assuming Jensen-Shannon (JS) divergence is defined as D JS (P||Q)=∫ X f(dP(x) / dQ(x))dQ(x); then the density ratio estimator ω is updated by maximizing the lower bound of the JS divergence ψ , the corresponding optimization objective is expressed as follows:

[0055]

[0056] where f * is the convex conjugate of f. For the first term in the equation, the expected value is estimated by sampling from the online data, and the second term is sampled from the offline data.

[0057] The beneficial effects of the present invention are as follows: the method can learn effective strategies using previously collected negotiation datasets without the need to interact with the opponent. This is in contrast to existing RL-based negotiation methods, which all rely on active interaction with the opponent. In addition, a strategy fine-tuning mechanism is included to cope with opponent preferences or strategy changes. The performance of the framework is evaluated based on a diverse set of state-of-the-art baselines under different settings. Experimental results show that the method can learn effective strategies using only offline datasets, and can also effectively adapt to changes in opponent preferences or strategies. The present invention effectively solves the problem of high trial and error costs in the field of intelligent negotiation, while alleviating offline reinforcement learning distribution shift and improving offline to online fast fine-tuning capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0059] Figure 1This is one of the flow charts of a negotiation agent method based on deep reinforcement learning provided by the present invention;

[0060] Figure 2 This is the second flow chart of a negotiation agent method based on deep reinforcement learning provided by the present invention;

[0061] Figure 3 This is the third flow chart of a negotiation agent method based on deep reinforcement learning provided by the present invention;

[0062] Figure 4 This is the fourth flow chart of a negotiation agent method based on deep reinforcement learning provided by the present invention; DETAILED DESCRIPTION

[0063] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. It will be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the drawings. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0065] Combine the following Figure 1-Figure 4 The present invention describes a negotiation agent method based on deep reinforcement learning.

[0066] The present invention provides a negotiation agent method based on deep reinforcement learning, such as Figure 1 As shown, the method includes:

[0067] S100, collecting historical negotiation data through the automatic negotiation platform and constructing a data set for learning;

[0068] S200, using an offline data set to learn an effective negotiation strategy for the agent;

[0069] S300, quickly fine-tune learned strategies and adapt to changes in opponent preferences or strategies through interaction in an online environment.

[0070] The negotiation agent reinforcement learning method of the present invention adopts an offline-to-online reinforcement learning method, that is, a negotiation strategy is first trained offline, and then the strategy is fine-tuned online in the online reinforcement learning stage.

[0071] Specifically, in the offline reinforcement learning stage, the present invention uses the CQL algorithm to train multiple policy networks and value networks on the basis of the SAC framework. It explicitly punishes the Q values ​​of actions outside the offline data set by adding regularization terms to reduce the impact of distribution shift. At the same time, it compensates the Q values ​​of actions that are within the offline data set. CQL pessimistically evaluates the current strategy and makes its Q function the lower bound of the true value. In order to mitigate its randomness, multiple trained policy networks and value networks are used to form an integrated network. In the online reinforcement learning stage, the present invention uses a priority sampling scheme component called balanced experience recovery on the basis of the SAC framework. This component utilizes online data by sampling offline data related to the current strategy. In this way, the agent can implicitly identify changes in the opponent's strategy or utility function without explicitly modeling the opponent.

[0072] Unlike existing methods, this method can train a good negotiation strategy without online interaction and effectively alleviates the problem of distribution shift. At the same time, unknown opponents may be encountered in the online stage. For familiar opponents, we can obtain favorable results through offline strategies to obtain higher constraint strength. On the contrary, for unknown opponents, safe transition conversion is used to promote fast and safe performance improvement.

[0073] Figure 2 The example of collecting historical negotiation data through the automatic negotiation platform and constructing a dataset for learning is given, which specifically includes:

[0074] S110, build a negotiation environment;

[0075] S120, construct the state space, action space and reward function settings in the reinforcement learning training process, and define the Markov decision process;

[0076] S130, using different strategies to collect intelligent agents and opponent intelligent agents to negotiate, record the offers and acceptance / rejection actions of both parties in each round of negotiation, and collect and process historical negotiation data sets.

[0077] First, the negotiation environment is described. Automated negotiation is the process of using different agents to represent humans in autonomous negotiation. A domain consists of a domain description and the preference files of both parties. The domain description is a description of the legal topics and the values ​​of each topic. Each automated negotiation domain contains a mapping, where each key is a negotiation topic and the legal values ​​of the corresponding topic form a set. Formally, a domain contains n topics: D = (I1,…,I n), each topic I contains k values: Based on this, an agent can make an offer ω, which selects the corresponding value from each issue (the selected value is represented by c), represented as a value vector: All the options constitute the negotiation space Ω. The preference profile of the domain determines the utility function, and the utility function of agent i maps an offer ω from the outcome space Ω to a real number in [0,1].

[0078]

[0079] where ω is a vector representation of the quotes, where each issue corresponds to a value, v jk represents the kth value of topic j, represents the evaluation function of agent i for topic j, which maps the value corresponding to the topic to the real number range [0,1]; Represents the preference weight of agent i for issue j.

[0080] Specifically, the definitions of the state space, action space, and reward function are designed based on the agent's own perspective. The state s at time t t The defined state space takes two factors into account. First, time, because if no agreement can be reached before the deadline, the negotiation will fail. Second, the negotiation history of both agents has an important influence on the decision of the agents. Therefore, the state space is defined as follows:

[0081]

[0082] Where T max Indicates the maximum round of current negotiation. and They represent the offerω proposed by the opponent and the agent at time tn respectively. s represents the utility function of the agent.

[0083] The action space is composed of [u r ,1] range. So the action at time t is defined as: in Indicates the utility value of the next offer. In order to get the real offer, we define an inverse utility function Map a utility u into the best choice of an offer ω by estimating the opponent's utility. Formally, the inverse utility function is defined as follows:

[0084]

[0085] Where U' oRepresents the opponent's utility function estimated based on the distribution frequency of the opponent's historical bids.

[0086] The reward function is specifically defined as when an agreement is reached, the agent will receive a positive reward, and if an agreement cannot be reached before the deadline, it will receive a penalty of -1. Formally, the reward function is defined as follows:

[0087]

[0088] The Markov decision process (MDP) is defined as a 5-tuple, namely Where S is the state space, A is the action space, P:S×A→S' is the transition function, r(s,a) is the reward function, γ∈[0,1) is the discount issue, and the strategy is a distribution π(a|s), which means that in the current state s t Take action a t The goal of the RL agent is to find a solution that maximizes the expected return strategy, everything in the negotiation scenario, including the opponent, is considered as the environment.

[0089] The offline dataset is used to learn an effective negotiation strategy for the agent, such as Figure 3 As shown, specifically including:

[0090] S210, using conservative Q learning (CQL) to train the policy network and the action value network, where in the process of training the action value network, actions that do not exist in the offline dataset are explicitly penalized.

[0091] The Q value of the work;

[0092] S220: Build an integrated strategy and construct an offline pre-trained automatic negotiation agent.

[0093] A major problem with online RL methods is that due to the limited and biased data, the model easily makes overly optimistic value estimates for unseen state-action pairs, which leads to poor performance of the strategy in practical applications. In the offline learning stage, the present invention adopts the idea of ​​penalizing the Q-values ​​of out-of-distribution actions by the CQL method to alleviate this problem. CQL avoids making overly optimistic value estimates on state-action pairs outside the training data by introducing conservative constraints. This can prevent the model from making mistakes in unseen environments and improve the robustness of the strategy. Specifically, conservative Q-learning (CQL) is used to train the policy network and the action-value network, wherein the Q-values ​​of actions that do not exist in the offline data set are explicitly penalized during the training of the action-value network;

[0094] Specifically, CQL pessimistically evaluates the current strategy and makes its Q function the lower bound of the true value. The method is based on the SAC framework, and the optimization goal of its value function is specifically expressed as follows:

[0095]

[0096] Among them, α is the weight issue. The first two terms of the above formula constitute a regularization term. The first term aims to prevent the overestimation of the Q value of the OOD action by minimizing the Q value under the policy, and the second term is balanced by maximizing the Q value of the action in the dataset after the behavior policy. The third term is the standard TD error, which measures the error between the predicted value and the more realistic value, and is represented by the Bellman operator Composition, among which,

[0097]

[0098] In addition, the optimization objectives of the negotiation strategy are as follows:

[0099]

[0100] where π φ is the strategy function of the negotiation agent, is the value function of the agent taking action a under the price adjustment of state s, and α is the temperature hyperparameter, which is used to control the balance between exploration and exploitation in the policy function.

[0101] The integration strategy is constructed to construct an offline pre-trained automatic negotiation agent, and the value network Q of the actor-critic network corresponding to the agent θ As shown below:

[0102]

[0103] The corresponding policy network π φ As shown below:

[0104]

[0105] in

[0106] The described method can quickly fine-tune the learned strategy through interaction in the online environment and adapt to changes in the opponent's preferences or strategies, such as Figure 4 As shown, specifically including:

[0107] S310, using the strategy network obtained by offline pre-training to initialize the online negotiation strategy intelligence

[0108] Entity, construct its optimization goal;

[0109] S320, introducing a priority sampling component for balanced experience replay, which utilizes online data by sampling offline data related to the current strategy and generates corresponding optimization targets;

[0110] S330: Based on the optimization target, update the strategies of the intelligent agent and the balance experience playback component.

[0111] Since the adversary may switch to a different strategy or change its preferences in later online negotiations, the environment faced by the offline strategy may be different from the environment seen in the offline dataset. This requires an online fine-tuning procedure, where the strategy can be further improved by collecting additional samples. Therefore, after obtaining the strategy through offline RL, the present invention adopts a policy fine-tuning mechanism to adjust its strategy. In order to effectively adapt to these changes of the adversary, the policy fine-tuning mechanism is designed to safely utilize online samples and more effectively mitigate distribution changes.

[0112] Specific, combined Figure 4 The strategy network obtained by offline pre-training is used to initialize the online negotiation strategy agent and construct its optimization goal. The agent is based on the SAC framework, and the optimization goal of its value function is as follows:

[0113]

[0114] where π φ is the probability distribution of the agent selecting action a in the current state s, r(s,α) is the reward function, is an online dataset, Q θ (s,a) is the value function corresponding to the agent’s strategy.

[0115] The negotiation strategy of the agent has the following optimization objectives:

[0116]

[0117] in is the entropy, α>0 is the temperature parameter, γ is the discount issue, r t is the reward function at timestamp t.

[0118] Due to the distribution shift problem, directly using these online data may quickly destroy a good initial offline policy. This requires an efficient policy fine-tuning mechanism to effectively utilize offline and online data. To this end, a priority sampling component called experience replay is used to utilize online data by sampling offline data related to the current policy. This component utilizes online data by sampling offline data related to the current policy and generates the corresponding optimization target.

[0119] Specifically, a priority buffer is first created to store offline negotiation data D off and online data D on The priority buffer then sorts all available samples according to their onlineness.

[0120] Specifically, to measure the linearity of the sample, the density ratio ω(s,a): = d on (s,a) / d off (s, a) measures the linearity of the sample. In practice, this is done by training a neural network called the density ratio estimator ω ψ (s, a) to estimate the density ratio. The density ratio estimator ω ψ The training of (s,a) uses the variational representation of f divergence, assuming Jensen-Shannon (JS) divergence is defined as D JS (P||Q)=∫ X f(dP(x) / dQ(x))dQ(x); then the density ratio estimator ω is updated by maximizing the lower bound of the JS divergence ψ , the corresponding optimization objective is expressed as follows:

[0121]

[0122] where f * is the convex conjugate of f. For the first term in the equation, the expected value is estimated by sampling from the online data, and the second term is sampled from the offline data.

[0123] In the experimental setup for testing negotiation performance, each agent plays against an opponent several times in each domain. In addition, in each repetition, a pair of agents negotiates twice, and they exchange the order of starting bids. The experiment considers the entire set of domains created for ANAC 2013. During each negotiation session, the discount factor is ignored and the maximum number of negotiation rounds is 1000. The number of repetitions is set to 300. For convenience, the reinforcement learning-based automatic negotiation agent is abbreviated as DRNA (Deep Reinforcement learning Negotiate Agent).

[0124] Since the adversary may change its preference file for a variety of reasons that are difficult to model in subsequent online negotiations, this experiment studies the performance of the DRNA framework when the adversary's preferences change. We assume that the adversary keeps its preference file unchanged for every 250 sessions in the online negotiation and then changes its preferences again. For simplicity, we also assume that the adversary keeps its policy fixed when changing its preferences. 100 different adversary preference files are randomly generated for each domain (i.e., these preferences are different from the preferences used in the offline dataset and from each other). For comparison, three baselines are introduced - a DRNA agent without policy fine-tuning (denoted as DRNA w / o sft), an RL agent learned online from scratch using the SAC algorithm (denoted as SAC agent), and a SAC agent initialized by the parameters of the DRNA agent (denoted as SAC-sft agent).

[0125] Table 1 summarizes the average utility of the agents and the baseline agents after 200 sessions in the small, medium, and large categories, with bounds based on 95% confidence intervals. The DRNA agent significantly outperforms the baseline agent in both learning efficiency and final performance, outperforming DRNA w / o sft by 28% on average. The SAC-sft agent follows closely behind and ranks second in all three categories. In summary, DRNA is able to outperform the baseline agent when the opponent changes his preferences in online negotiations.

[0126] Table 1: Mean utilities in the three category domains, with bounds based on 95% confidence intervals.

[0127] Domain Category DRNA DRNAw / osft SAC SAC-sft Small space domain 0.79±0.02 0.57±0.04 0.61±0.04 0.74±0.03 Middle Space Domain 0.88±0.03 0.71±0.05 0.69±0.06 0.81±0.04 Large spatial domain 0.84±0.03 0.67±0.03 0.65±0.04 0.77±0.04

[0128] This experiment also studies the performance of the DRNA agent when the opponent adopts different strategies. In the subsequent online negotiation, the opponent can use new strategies that have not been seen in the offline learning phase. The opponent strategy pool not only includes the strategies of the aforementioned ANAC winning agents, namely AgreeableAgent2018, PonpokoAgent, Caduceus, and Atlas3, but also considers the strategies of the second-place agents in their respective ANAC competitions - Agent36, CaduceusDC16, YX-Agent, and ParsAgent as new strategies. And the recently proposed effective negotiation strategy MiCRO is also in the strategy pool. The experimental setup is similar to the previous one, except that the opponent can change its strategy while keeping its preference file fixed.

[0129] Table 2 shows the results, where the column “Average Utility vs. Opponent” represents the average utility of all agents for each agent (per row) in the pool of opponent strategies, and the other columns represent the average utility obtained by each agent in each column when playing against the agent in each row. The first part of the table (the first four rows) represents the strategies seen in the offline dataset, and the second part represents the new strategies.

[0130] First, when encountering the four winning strategies used in offline training, DRNA w / osft outperforms the average performance of the opponent strategy (see the second column of the table), outperforming the opponent's average score by 48.96%. However, this performance advantage is reduced by about 13.95% when the opponent switches to an unknown strategy. This shows that although DRNA w / o sft is an effective strategy overall, policy adjustment is required if higher performance is expected when facing unknown strategies. Then, through the policy fine-tuning mechanism, the DRNA agent further improves performance, with an average improvement of 9.58% over its non-fine-tuning variant. The only special case is that the DRNA agent performs slightly worse than DRNA w / o sft (about 1.20%) when playing against AgreeableAgent2018. The reason is that in this case, the initial strategy is good enough that the DRNA agent eventually reaches a similar performance level. The SAC-sft agent lags behind the DRNA agent again. These results verify the effectiveness of the policy fine-tuning mechanism in negotiations where the opponent changes strategy.

[0131] Table 2: Comparison results of the DRNA agent and the baseline agent against opponents with different strategies in online negotiation. All results are based on all 18 domains.

[0132]

[0133] The present invention proposes a negotiation agent method based on deep reinforcement learning, which learns strategies through historical negotiation data sets, including: collecting historical negotiation data through an automatic negotiation platform to construct a dataset for learning; using offline datasets to learn effective negotiation strategies for the agent; and quickly fine-tuning the learned strategies and adapting to changes in the opponent's preferences or strategies through online interactions. The present invention effectively solves the problem of high online interactive training costs in the field of automatic negotiation, and at the same time learns a general offline initial strategy, and the negotiation agent can adapt to changes in the online opponent's preferences or strategies.

[0134] The device embodiments described above are illustrative only, and some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. A person skilled in the art may understand and implement these embodiments without creative effort.

Claims

1. A negotiation agent method based on deep reinforcement learning, characterized in that: include: S100, collecting historical negotiation data through the automatic negotiation platform and constructing a data set for learning; S200, using an offline data set to learn an effective negotiation strategy for the agent; S300, quickly fine-tune learned strategies and adapt to changes in opponent preferences or strategies through interaction in an online environment.

2. According to claim 1, a negotiation agent method based on deep reinforcement learning is characterized in that: The above mentioned method of collecting historical negotiation data through the automatic negotiation platform and constructing a dataset for learning includes: S110, build a negotiation environment; S120, construct the state space, action space and reward function settings in the reinforcement learning training process, and define the Markov decision process; S130, using different strategies to collect intelligent agents and opponent intelligent agents to negotiate, record the offers and acceptance / rejection actions of both parties in each round of negotiation, and collect and process historical negotiation data sets.

3. A negotiation agent method based on deep reinforcement learning according to claim 2, characterized in that: In the negotiation environment, a domain consists of a domain description and the preference files of both parties. The domain description is a description of the legal topics and the value of each topic. Each automatic negotiation domain contains a mapping, in which each key is a negotiation topic, and the legal values ​​of the corresponding topic form a set. Formally, a domain consists of n topics: D = (I1,…,I n ), each topic I contains k values: Based on this, an agent can make an offer ω, which selects the corresponding value from each issue (the selected value is represented by c), represented as a value vector: All the options constitute the negotiation space… The preference profile of the domain determines the utility function, and the utility function of agent i maps an offer ω from the outcome space Ω to a real number in [0,1]. where ω is a vector representation of the quotes, where each issue corresponds to a value, v jk represents the kth value of topic j, represents the evaluation function of agent i for topic j, which maps the value corresponding to the topic to the real number range [0,1]; Represents the preference weight of agent i for issue j.

4. According to the deep reinforcement learning-based negotiation agent method of claim 2, it is characterized in that: The definitions of the state space, action space, and reward function are designed based on the agent's own perspective. The state s at time t is t The state space is defined as follows: Where T max Indicates the maximum round of current negotiation. and They represent the offerω proposed by the opponent and the agent at time tn respectively. s represents the utility function of the agent. The action space is composed of [u r ,1] range. So the action at time t is defined as: in Indicates the utility value of the next offer. In order to get the real offer, we define an inverse utility function Map a utility u into the best choice of an offer ω by estimating the opponent's utility. Formally, the inverse utility function is defined as follows: Where U' o Represents the opponent's utility function estimated based on the distribution frequency of the opponent's historical bids. The reward function is specifically defined as when an agreement is reached, the agent will receive a positive reward, and if an agreement cannot be reached before the deadline, it will receive a penalty of -1. Formally, the reward function is defined as follows: The Markov decision process (MDP) is defined as a 5-tuple, namely Where S is the state space, A is the action space, P:S×A→S' is the transition function, r(s,a) is the reward function, γ∈[0,1) is the discount issue, and the strategy is a distribution π(a|s), which means that in the current state s t Take action a t The goal of the RL agent is to find a solution that maximizes the expected return strategy, everything in the negotiation scenario, including the opponent, is considered as the environment.

5. According to claim 1, a negotiation agent method based on deep reinforcement learning is characterized in that: The utilizing of the offline data set to learn an effective negotiation strategy for the agent specifically includes: S210, training the policy network and the action value network using conservative Q learning (CQL), wherein the Q values ​​of actions that do not exist in the offline dataset are explicitly penalized during the training of the action value network; S220: Build an integrated strategy and construct an offline pre-trained automatic negotiation agent.

6. A negotiation agent method based on deep reinforcement learning according to claim 5, characterized in that: The method uses conservative Q learning (CQL) to train the policy network and the action value network, wherein the Q value of the action that does not exist in the offline data set is explicitly penalized during the training of the action value network; specifically, CQL pessimistically evaluates the current policy and makes its Q function a lower bound of the true value. The method is based on the SAC framework, and the optimization objective of its value function is specifically expressed as follows: Among them, α is a weight issue. The first two terms of the above formula constitute a regularization term. The first term aims to prevent the overestimation of the Q value of the action outside the offline distribution by minimizing the Q value under the policy, and the second term is balanced by maximizing the Q value of the action in the dataset after the behavior policy. The third term is the standard TD error, which measures the error between the predicted value and the more realistic value, which is represented by the Bellman operator Composition, among which, In addition, the optimization objectives of the negotiation strategy are as follows: where π φ is the strategy function of the negotiation agent, is the value function of the agent taking action a under the price adjustment of state s, and α is the temperature hyperparameter, which is used to control the balance between exploration and exploitation in the policy function.

7. A negotiation agent method based on deep reinforcement learning according to claim 5, characterized in that: The above-mentioned integrated strategy is used to construct an offline pre-trained automatic negotiation agent. The value network Q of the actor-critic network corresponding to the agent is as follows: The corresponding policy network π is as follows: in 8. The negotiation agent method based on deep reinforcement learning according to claim 1, characterized in that: The method of quickly fine-tuning the learned strategy and adapting to the opponent's preference or strategy changes through interaction in the online environment specifically includes: S310, using the strategy network obtained by offline pre-training to initialize the online negotiation strategy agent and construct its optimization target; S320, introducing a priority sampling component for balanced experience replay, which utilizes online data by sampling offline data related to the current strategy and generates corresponding optimization targets; S330: Based on the optimization target, update the strategies of the intelligent agent and the balance experience playback component.

9. A negotiation agent method based on deep reinforcement learning according to claim 8, characterized in that: The strategy network obtained by offline pre-training is used to initialize the online negotiation strategy agent and construct its optimization goal. The agent is based on the SAC framework and its value function Q θ The optimization objectives are as follows: where π φ is the probability distribution of the agent selecting action a in the current state s, r(s,a) is the reward function, is an online dataset, Q θ (s,a) is the value function corresponding to the agent’s strategy. The negotiation strategy of the agent has the following optimization objectives: in is the entropy, α>0 is the temperature parameter, γ is the discount issue, r t is the reward function at timestamp t.

10. A negotiation agent method based on deep reinforcement learning according to claim 8, characterized in that: The priority sampling component introduced to balance experience replay uses the density ratio ω(s,a): = d on (s,a) / d off (s, a) measures the linearity of the sample. In practice, this is done by training a neural network called the density ratio estimator ω ψ (s, a) to estimate the density ratio. The density ratio estimator ω ψ The training of (s,a) uses the variational representation of f divergence, assuming Jensen-Shannon (JS) divergence is defined as D JS (P||Q)=∫ X f(dP(x) / dQ(x))dQ(x); then the density ratio estimator ω is updated by maximizing the lower bound of the JS divergence ψ , the corresponding optimization objective is expressed as follows: where f * is the convex conjugate of f. For the first term in the equation, the expected value is estimated by sampling from the online data, and the second term is sampled from the offline data.

Images