Automated Negotiation Agent Adaptation

Abstract

The automated negotiation agent adaptation is implemented by detecting a change in a utility function associated with the automated negotiation between the assistant agent and the other agent while the assistant agent is operating according to a first negotiation strategy model, generating a plurality of training samples from the automated negotiation between the assistant agent and the other agent while the assistant agent is operating according to a baseline negotiation strategy model, and training the initialized negotiation strategy model using the plurality of training samples to generate a second negotiation strategy model.

Description

[Technical field]

[0001] The present disclosure relates to computer-readable media, computer-implemented methods, and apparatus. This application claims priority to U.S. Provisional Application No. 63 / 292,383, filed December 21, 2021, and U.S. Application No. 17 / 575,908, filed January 14, 2022, the entire contents of which are incorporated herein by reference in their entireties. [Background technology]

[0002] Negotiation is a decision-making process between two or more parties aiming to reach a mutually beneficial agreement. Automated negotiation involves negotiation between automated agents acting on behalf of real-world entities in order to achieve mutually beneficial agreements while reducing the time and effort involved in negotiation. Summary of the Invention

[0003] According to a first exemplary aspect of the present disclosure, a computer-readable medium includes instructions executable by a computer to cause a computer to perform operations including detecting a change in a utility function associated with an automated negotiation between an assistant agent and a counter agent while the assistant agent is operating according to a first negotiation strategy model; generating a plurality of training samples from an automated negotiation between the assistant agent and the counter agent while the assistant agent is operating according to a baseline negotiation strategy model; and training an initialized negotiation strategy model using the plurality of training samples to generate a second negotiation strategy model.

[0004] According to a second exemplary aspect of the present disclosure, a computer-implemented method includes detecting a change in a utility function associated with an automated negotiation between an assistant agent and a counter agent while the assistant agent is operating according to a first negotiation strategy model, generating a plurality of training samples from the automated negotiation between the assistant agent and the counter agent while the assistant agent is operating according to a baseline negotiation strategy model, and training an initialized negotiation strategy model using the plurality of training samples to generate a second negotiation strategy model.

[0005] According to a third exemplary aspect of the present disclosure, an apparatus comprises a controller including circuitry configured to detect a change in a utility function associated with an automated negotiation between an assistant agent and a counter agent while the assistant agent is operating according to a first negotiation strategy model, generate a plurality of training samples from the automated negotiation between the assistant agent and the counter agent while the assistant agent is operating according to a baseline negotiation strategy model, and train an initialized negotiation strategy model using the plurality of training samples to generate a second negotiation strategy model. [Brief description of the drawings]

[0006] Aspects of the present disclosure are best understood from the following detailed description when read in conjunction with the accompanying drawings, in which: It should be noted that, in accordance with standard industry practice, various features have not been drawn to scale, and in fact, dimensions of various features may be arbitrarily expanded or reduced for clarity of illustration. [Figure 1] FIG. 1 is a schematic diagram of a system for automated negotiation agent adaptation, according to at least some embodiments of the present disclosure. [Diagram 2] 1 is an operational flow for automated negotiation agent adaptation, according to at least some embodiments of the present disclosure. [Diagram 3] 1 is an operational flow for opposing utility function change detection, according to at least some embodiments of the present disclosure. [Figure 4] 1 is an operational flow for supporting utility function change detection, in accordance with at least some embodiments of the present disclosure. [Diagram 5] 1 is an operational flow for training sample generation, according to at least some embodiments of the present disclosure. [Figure 6] 1 is an operational flow for training a negotiation strategy model according to at least some embodiments of the present disclosure. [Figure 7] FIG. 1 is a block diagram of a hardware configuration for automated negotiation agent adaptation, in accordance with at least some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] The following disclosure provides many different embodiments or examples for implementing different features of the provided subject matter. Hereinafter, specific examples of components, values, operations, materials, arrangements, etc. will be described to simplify the disclosure. Of course, these are merely examples and are not intended to be limiting. Other components, values, operations, materials, arrangements, etc. are contemplated. In addition, the disclosure may repeat reference numbers and / or letters in various examples. This repetition is for simplicity and clarity and does not in itself dictate a relationship between the various embodiments and / or configurations described.

[0008] Bilateral automated negotiation is a negotiation between two automated agents. Repeated negotiation between two fixed entities is a continuous negotiation. An exemplary negotiation scenario consists of a utility function and a negotiation domain for each agent. An exemplary negotiation protocol is a stacked alternating offer protocol. In at least some embodiments, each agent operates according to a negotiation strategy that is a combination of an acceptance strategy and a bidding strategy.

[0009] In at least some embodiments, a negotiation domain consists of one or more issues. The outcome space of a negotiation is the set of all possible negotiation outcomes, which in at least some embodiments is defined as follows:

number

number

number

number

[0010] When historical data, such as records of previous negotiation traces, are available, machine learning techniques can be used to create domain-specific negotiation strategies. However, such strategies become less effective when the utility functions of opposing agents in a negotiation change during frequent negotiations.

[0011] According to at least some embodiments described herein, the negotiation agent is trained to negotiate from historical data and to adapt to changes in the utility function of the opposing agent. In doing so, at least some embodiments herein enable the negotiation agent to more efficiently negotiate agreements and obtain more profitable outcomes when the utility function changes. At least some embodiments herein include detecting changes in the utility function of the opposing agent. In doing so, at least some embodiments herein enable the negotiation agent to continue to learn and improve over time.

[0012] In at least some embodiments herein, a parameter-based transfer learning approach involves transferring knowledge via shared parameters of a currently trained negotiation strategy model and a newly trained negotiation strategy model. In at least some embodiments, the currently trained negotiation strategy model, which is trained in the source domain, has already learned a well-defined structure, and the structure is transferable to the newly trained negotiation strategy model because the task does not change with changes to any utility function.

[0013] 1 is a schematic diagram of a system for automated negotiation agent adaptation in accordance with at least some embodiments of the present disclosure. The system includes an apparatus 100, an assisting agent 112, and an opposing agent 119. In at least some embodiments, the apparatus 100 and the assisting agent 112 are one or more computers, such as a personal computer, a server, a mainframe, an instance of cloud computing, etc., that include instructions executed by a controller to implement the automated negotiation agent adaptation.

[0014] The apparatus 100 is in communication with the assisting agent 110 and includes a detection section 170, a generation section 172, and a training section 174. In at least some embodiments, the detection section 170 is configured to detect changes in utility functions, such as an assisting utility function of the assisting agent 110 and an opponent utility function of the opponent agent 119. In at least some embodiments, the detection section 170 is configured to receive recent traces 124 and past traces 122 from the negotiation trace storage unit 120 for opponent utility function change detection. In at least some embodiments, the detection section 170 is configured to receive current and updated assisting utility functions from the assisting utility function storage unit 116 for assisting utility function change detection.

[0015] In at least some embodiments, the generating section 172 is configured to generate training samples in response to the detecting section detecting the utility function change. In at least some embodiments, the generating section 172 is configured to receive the baseline traces 126 for training sample generation. In at least some embodiments, the generating section 172 is configured to transmit a plurality of training samples to the training section 174 for use in training the negotiation strategy model.

[0016] In at least some embodiments, training section 174 is configured to train the negotiation strategy model. In at least some embodiments, training section 174 is configured to receive training samples 184 and context information 115 for training the negotiation strategy model. In at least some embodiments, training section 174 is configured to transmit the trained negotiation strategy model to assistance agent 110 for use in live automated negotiation.

[0017] In at least some embodiments, assistance agent 110 is configured to conduct automated negotiation with counter agent 119, whereby offers 118 are sent alternately until an offer is accepted. In at least some embodiments, assistance agent 110 is configured to operate according to a negotiation strategy model, such as a baseline negotiation strategy model 112 or a training negotiation strategy model 114. In at least some embodiments, assistance agent 110 is configured to select a counteroffer based on outstanding offers from counter agent 119 and context information 115. In at least some embodiments, context information 115 is dynamic information that affects the amount of profit.

[0018] In at least some embodiments, baseline negotiation strategy model 112 is an algorithm for selecting offers and counteroffers based on an aided utility function. In at least some embodiments, baseline negotiation strategy model 112 is not trained specifically for negotiating with opposing agents 119. In at least some embodiments, trained negotiation strategy model 114 is a machine learning model trained to select offers and counteroffers. In at least some embodiments, trained negotiation strategy model 114 is trained specifically for negotiating with opposing agents 119.

[0019] 2 is an operational flow for automated negotiation agent adaptation, according to at least some embodiments of the present disclosure. The operational flow provides a method of automated negotiation agent adaptation. In at least some embodiments, the method is performed by a controller of a device that includes sections for performing certain operations, such as the controller and device shown in FIG. 7 described below.

[0020] In S230, the detection section detects changes in the utility functions. In at least some embodiments, the detection section detects changes in the utility functions associated with the automated negotiation between the assisting agent and the opposing agent while the assisting agent is operating according to the first negotiation strategy model. In at least some embodiments, the detection section detects changes in the opposing agent's opposing utility function and the assisting agent's first assist utility function. In at least some embodiments, the opposing utility function change detection proceeds as shown in FIG. 3, described below. In at least some embodiments, the assisting utility function change detection proceeds as shown in FIG. 4, described below. In at least some embodiments, the detection section detects changes in only one utility function at a time.

[0021] At S240, the generation section generates training samples from negotiations according to the baseline strategy model. In at least some embodiments, the generation section generates a plurality of training samples from automated negotiations between the assisting agent and the opposing agent while the assisting agent is operating according to the baseline negotiation strategy model. In at least some embodiments, the generation section instructs the assisting agent to negotiate according to the baseline strategy model. In at least some embodiments, the generation section generates training samples from a plurality of complete negotiations. In at least some embodiments, the generation section generates a plurality of training samples from each complete negotiation.

[0022] In S250, the generation section determines whether a sufficient number of training samples have been generated. In at least some embodiments, the generation section determines whether a sufficient number of training samples have been generated based on the number of training samples. In at least some embodiments, the generation section determines whether a sufficient number of training samples have been generated based on one or more qualifications of the training samples. If the generation section determines that a sufficient number of training samples have been generated, the operational flow proceeds to new negotiation strategy model training in S260. If the generation section determines that a sufficient number of training samples have not yet been generated, the operational flow returns to training sample generation in S240.

[0023] In S260, the training section trains a new negotiation strategy model. In at least some embodiments, the training section trains the initialized negotiation strategy model using a plurality of training samples to generate a second negotiation strategy model. In at least some embodiments, the training section transmits the trained negotiation strategy model to an assistance agent and instructs the assistance agent to use the trained negotiation strategy model in the automated negotiation.

[0024] 3 is an operational flow for opponent utility function change detection, according to at least some embodiments of the present disclosure. The operational flow provides a method of opponent utility function change detection. In at least some embodiments, the method is performed by a detection section of an apparatus, such as the apparatus shown in FIG. 7 described below.

[0025] In S332, the detection section or a subsection thereof obtains a recent negotiation trace. In at least some embodiments, the detection section obtains a recent negotiation trace from an automated negotiation between the assisting agent and the counter agent while the assisting agent is operating according to the first negotiation strategy model, the negotiation trace including a plurality of time steps, each time step in the plurality of time steps including a counter agent offer and an assisting agent offer. In at least some embodiments, the recent negotiation trace is a negotiation trace from a most recent complete negotiation. In at least some embodiments, the detection section obtains two or more recent negotiation traces, such as negotiation traces from the five most recent complete negotiations. In at least some embodiments, the detection section obtains the recent negotiation traces directly from the assisting agent or from a negotiation trace storage unit.

[0026] In S333, the detection section or a subsection thereof acquires a previous negotiation trace. In at least some embodiments, the detection section acquires a previous negotiation trace from an automated negotiation between the assisting agent and the opposing agent while the assisting agent is operating according to the first negotiation strategy model. In at least some embodiments, the previous negotiation trace is a negotiation trace used to generate training samples that are then used to train the first negotiation strategy model. In at least some embodiments, the detection section acquires two or more previous negotiation traces, such as negotiation traces from the first five complete negotiations using the first negotiation strategy model. In at least some embodiments, the detection section acquires the previous negotiation traces directly from the assisting agent or from a negotiation trace storage unit.

[0027] In S335, the detection section or a subsection thereof compares the counter offers of the negotiation traces obtained in S332 and S333. In at least some embodiments, the detection section compares the counter agent offers of the recent negotiation trace with the counter agent offers of the previous negotiation traces from the automated negotiation between the assisting agent and the counter agent while the assisting agent was operating according to the first negotiation strategy model. In at least some embodiments, the detection section applies a classifier to tuples of the average frequency distribution of the counter offer in the previous negotiation trace and the frequency distribution of the counter offer in the recent negotiation trace.

[0028] In at least some embodiments, the classifier is a binary classifier that classifies whether the assistance agent should continue with the currently trained negotiation strategy model or adapt to a new negotiation strategy model for the negotiation strategy. In at least some embodiments, the classifier assumes that the opponent utility function and the assistance utility function do not change significantly for successive sets of negotiations. In at least some embodiments,

number

number

number

[0029] In at least some embodiments, the classifier is an XGBOOST-based classifier, and for training the classifier, the training data is

number

number

number

number

number

[0030] In S336, the detection section or a subsection thereof estimates a change value of the opponent utility function. In at least some embodiments, the detection section estimates a change value that represents an amount of change in the opponent utility function. In at least some embodiments, the change value is an output of a opponent offer comparison, such as the operation in S335. In at least some embodiments where a classifier is used for the opponent offer comparison, the output of the classifier is a Boolean value, and the final classification between the Boolean values ​​is based on a numerical value that represents the difference between groups of opponent offers. In at least some embodiments where a distance-based algorithm is used for the opponent offer comparison, the distance-based algorithm estimates a tuple

number

number

number

[0031] In S338, the detection section or a subsection thereof determines whether the change value estimated in S336 exceeds a threshold. In at least some embodiments, the threshold is an adjustable hyperparameter. In at least some embodiments, increasing the threshold results in fewer instances of training a new negotiation strategy model, which is more computationally efficient but produces less effective results, while decreasing the threshold results in more instances of training a new negotiation strategy model, which is less computationally efficient but produces more effective results. In at least some embodiments, the threshold is adjusted to balance the tradeoff between efficiency and effectiveness. In at least some embodiments where a classifier is used for counterpart offer comparison, the output of the classifier is a Boolean value, and the final classification between the Boolean values ​​involves comparing a numerical value representing the difference between groups of counterpart offers to a threshold. In at least some embodiments where a distance-based algorithm is used for counterpart offer comparison, such as Equation 7, the estimated change value m is adjusted to a threshold value α M is compared to.

[0032] If the detection section determines that the variance value exceeds the threshold, operational flow proceeds to simply wait, e.g., a predetermined period of time, at S339 before returning to obtain a recent negotiation trace at S332. In other words, in at least some embodiments, the detecting is performed periodically.

[0033] If the detection section determines that the change value exceeds the threshold, the operational flow ends. In at least some embodiments, the end of the operational flow for opponent utility function change detection leads to training sample generation, and ultimately to new negotiation strategy model training, such as the operations at S240 and S260 of FIG. 2. In at least some embodiments, the generating and training are performed in response to determining that the change value exceeds the threshold.

[0034] 4 is an operational flow for assisted utility function change detection, according to at least some embodiments of the present disclosure. The operational flow provides a method of assisted utility function change detection. In at least some embodiments, the method is performed by a detection section of an apparatus, such as the apparatus shown in FIG. 7 described below.

[0035] In S431, the detection section or a subsection thereof receives an updated assistance utility function. In at least some embodiments, the detection section receives a second assistance utility function. In at least some embodiments, receiving the updated assistance utility function triggers assistance utility function change detection. In at least some embodiments, assistance utility function change detection (S430) includes operations at S434, S437, and S438 and is performed in response to receiving the updated assistance utility function in S431. In other words, in at least some embodiments, the detecting is performed in response to receiving the second assistance utility function.

[0036] In at least some embodiments, one simple method is to use the assistance utility function change, i.e.,

number

[0037] At S434, the detection section or a subsection thereof compares the previous aid utility function with the updated aid utility function. In at least some embodiments, the detection section compares the first aid utility function with the second utility function. In at least some embodiments where the aid utility function includes an ordered result set, such as Equation 2, the detection section compares the ordering of results between the previous aid utility function and the updated aid utility function.

[0038] In at least some embodiments, the detection section comprises a metric based on the Levenshtein distance.

number

number

number

number

number

number

number

number

number

[0039] For example,

number

number

number

number

[0040] In S437, the detection section or a subsection thereof estimates a change value of the aided utility function. In at least some embodiments, the detection section estimates a change value that represents an amount of change between the first aided utility function and the second aided utility function. In at least some embodiments, the change value is an output of an aided utility function comparison, such as the operation in S434. The detection section may estimate a metric based on the Levenshtein distance.

number

number

number

number

number

[0041] In S438, the detection section or a subsection thereof determines whether the change value estimated in S437 exceeds a threshold. In at least some embodiments, the threshold is an adjustable hyperparameter. In at least some embodiments, increasing the threshold results in fewer instances of new negotiation strategy model training, which is more computationally efficient but produces less effective results, while decreasing the threshold results in more instances of new negotiation strategy model training, which is less computationally efficient but produces more effective results. In at least some embodiments, the threshold is adjusted to balance the tradeoff between efficiency and effectiveness. In at least some embodiments in which the detection section measures the change between the assistance utility functions using a metric based on the Levenshtein distance, such as Equations 9-11, the estimated change value l is adjusted to a value greater than or equal to the threshold.

number

[0042] If the detection section determines that the variance value exceeds the threshold, operational flow returns to receiving an updated assistance utility function in S431.

[0043] If the detection section determines that the change value exceeds the threshold, the operational flow ends. In at least some embodiments, the end of the operational flow for opponent utility function change detection leads to training sample generation, and ultimately to new negotiation strategy model training, such as the operations at S240 and S260 of FIG. 2. In at least some embodiments, the generating and training are performed in response to determining that the change value exceeds the threshold.

[0044] 5 is an operational flow for training sample generation according to at least some embodiments of the present disclosure. The operational flow provides a method of training sample generation. In at least some embodiments, the method is performed by a generation section of an apparatus such as the apparatus shown in FIG. 7 described below.

[0045] In S541, the generating section or a subsection thereof obtains a baseline negotiation trace. In at least some embodiments, the generating section obtains a negotiation trace from an automated negotiation between the assisting agent and the counter agent while the assisting agent is operating according to the baseline negotiation strategy model, the negotiation trace including a plurality of time steps, each time step in the plurality of time steps including a counter agent offer and an assisting agent offer. In at least some embodiments, the generating section obtains a plurality of negotiation traces while the assisting agent is operating according to the baseline negotiation strategy model. In at least some embodiments, the legacy system is temporarily used as a baseline negotiation strategy model in place of the currently trained negotiation strategy model to negotiate with the counter agent for a small number of complete negotiations, and H smalland generating a small set of negotiation traces with final agreed-upon offers, denoted as (a) and (b). In at least some embodiments, the legacy system includes a single or multiple instances of a compatible automated negotiation agent of any type. In at least some embodiments, the generating section obtains recent negotiation traces directly from an assisting agent or from a negotiation trace storage unit.

[0046] In S543, the generation section or a subsection thereof begins generating a single training sample by including the first i time steps. In at least some embodiments, the generation section includes a complete set of offers in the training sample from the first i-1 time steps. In at least some embodiments, the complete set of offers from a time step includes a counter offer and a supporting offer. In at least some embodiments, i ranges from 2 to n and increases by 1 for each iteration, where n is the number of time steps in the negotiation trace.

[0047] In S544, the generation section or a subsection thereof continues to generate a single training sample by including a counter offer for time step i. In at least some embodiments, the counter offer for time step i is a counter offer after the complete set of offers in the training sample from the first i-1 time steps included in S543.

[0048] In S545, the generation section or a subsection thereof continues to generate a single training sample by labeling the training sample with the assistance offer for time step i. In at least some embodiments, the assistance offer for time step i is included in the training sample such that the training section can identify the assistance offer as a label rather than as an input to the model. In at least some embodiments, in the operations of S543, S544, and S545, the generation section generates each training sample of the plurality of samples to include, as input, a portion of consecutive time steps of the plurality of time steps from the first time step and a counterpart agent offer for a subsequent time step of the plurality of time steps following the portion, and further includes the assistance agent offer for the subsequent time step as a label. In at least some embodiments, the generation section generates all training samples of the plurality of samples for training.

number

[0049] In S547, the generating section or its subsection determines whether all of the time steps in the baseline negotiation trace have been processed. In at least some embodiments, the generating section determines that all of the time steps in the baseline negotiation trace have been processed after the operations in S543, S544, and S545 have been performed for time step n. If the generating section determines that there are unprocessed time steps remaining in the baseline negotiation trace, the operational flow returns to S543, and after incrementing i by 1 in S548, begins generating another single training sample for the next time step. If the generating section determines that all of the time steps in the baseline negotiation trace have been processed, the operational flow proceeds to training sample weighting in S549.

[0050] In S549, the generation section or a subsection thereof weights the training samples. In at least some embodiments, the generation section weights each training sample based on a utility value of a final agreed offer of the negotiation trace, where the utility value is obtained by applying an auxiliary utility function to the final agreed offer ... T, while using the generated input-output set from the training samples generated from the negotiation trace T, by applying a loss weight (U s (ω * T )) k By using D We overcome the poor model training caused by the negotiation trace in,ω,. * T is the final agreed offer, U S (ω * T ) is the utility value for the final agreed offer, and k is a tunable hyperparameter.

[0051] 6 is an operational flow for training a negotiation strategy model according to at least some embodiments of the present disclosure. The operational flow provides a method of training a negotiation strategy model. In at least some embodiments, the method is performed by a training section of an apparatus such as the apparatus shown in FIG. 7 described below.

[0052] At S662, the training section or a subsection thereof initializes at least a portion of the currently trained negotiation strategy model. In at least some embodiments, the training section initializes a portion of the first negotiation strategy model with random values. In at least some embodiments, the negotiation strategy model uses a bidirectional long short-term memory (LSTM) based architecture with an embedding layer at the beginning and a single dense layer with softmax activation at the end. In at least some embodiments, the embedding layer captures partial information about the utility functions of both the helping agent and the opponent agent.

[0053] A change in the utility function, in at least some embodiments, reduces the performance of the currently trained negotiation strategy model due to the use of an embedding layer that captures partial information of both utility functions. In at least some embodiments, the embedding layer is retrained when either utility function changes. In at least some embodiments, both the embedding layer and the dense layer are retrained to recapture new information regarding the updated opponent or assist utility function. In at least some embodiments where the new strategy model training is responsive to the updated assist utility function, the training section reuses the weights and biases of the bidirectional LSTM layer from the currently trained negotiation strategy model to reduce the total number of trainable parameters and reduce the time and data utilized to train the model. In at least some embodiments, the training section initializes one or more of the embedding layer and the dense layer with random values ​​while maintaining the bidirectional LSTM layer.

[0054] At S664, the training section, or a subsection thereof, applies the negotiation strategy model to the training samples. In at least some embodiments, the training section inputs the training samples to the negotiation strategy model and reads output from the negotiation strategy model. In at least some embodiments, the training section inputs the sequences of offers of the training samples and reads output assisted offers from the negotiation strategy model. In at least some embodiments, the training section also inputs contextual data to the negotiation strategy model.

[0055] In S666, the training section or a subsection thereof adjusts the negotiation strategy model based on the output from the negotiation strategy model. In at least some embodiments, the training section compares the output from the negotiation strategy model to the labels of the training samples. In at least some embodiments, the training section compares the output assistance offers to actual assistance offers from the baseline negotiation traces with which the training samples were labeled. In at least some embodiments, the training section adjusts the negotiation strategy model by applying a loss function to the output and the labels and deriving a loss value that is used to adjust the weights of the negotiation strategy model. In at least some embodiments, the strategy model adjustment in S666 is not performed for every iteration of the operations of S664, S666, S668, and S669, but rather periodically with respect to the number of iterations or in response to a loss value exceeding a threshold.

[0056] In S668, the training section or a subsection thereof determines whether a termination condition is met. In at least some embodiments, the termination condition is a number of training iterations, such as a number of epochs. In at least some embodiments, the termination condition is met when the loss value falls below a threshold. If the training section determines that the termination condition is not yet met, the operational flow proceeds to select the next training sample (S669) before returning to the strategy model application in S664. If the training section determines that the termination condition is met, the operational flow ends.

[0057] In at least some embodiments, the training section fine-tunes the entire negotiation strategy model with a very small learning rate and a small number of epochs. In at least some embodiments, the fine-tuning results in better model accuracy, but the utility value of the final agreed-upon offer is not large compared to the final agreed-upon offer reached with the negotiation strategy model without fine-tuning.

[0058] FIG. 7 is a block diagram of a hardware configuration for automated negotiation agent adaptation, according to at least some embodiments of the present disclosure.

[0059] An exemplary hardware configuration includes an apparatus 700 that interacts with an assist agent 710 and communicates with a network 707. In at least some embodiments, the apparatus 700 is integrated with the assist agent 710. In at least some embodiments, the apparatus 700 is a computer system that executes computer-readable instructions to perform operations for physical network function device access.

[0060] The device 700 includes a controller 702, a storage unit 704, a communication interface 706, and an input / output interface 708. In at least some embodiments, the controller 702 includes a processor or programmable circuit that executes instructions, which cause the processor or programmable circuit to perform operations according to the instructions. In at least some embodiments, the controller 702 includes analog or digital programmable circuits, or any combination thereof. In at least some embodiments, the controller 702 includes physically separate storage devices or circuits that interact through communication. In at least some embodiments, the storage unit 704 includes a non-volatile computer readable medium capable of storing executable and non-executable data for the controller 702 to access during execution of instructions. The communication interface 706 transmits and receives data to and from a network 707. The input / output interface 708 connects to and exchanges information with an input device 708 via a parallel port, a serial port, a keyboard port, a mouse port, a monitor port, or the like.

[0061] The controller 702 includes a detection section 770, a generation section 772, a training section 774, and a transmission section 786. The storage unit 704 includes a utility function 780, a negotiation trace 782, training samples 784, and training parameters 786.

[0062] The detection section 770 is circuitry or instructions of the controller 702 configured to detect changes to the utility function. In at least some embodiments, the detection section 770 is configured to detect changes in the utility function associated with an automated negotiation between the assisting agent and the opposing agent while the assisting agent is operating according to the first negotiation strategy model. In at least some embodiments, the detection section 770 utilizes information in the storage unit 704, such as the negotiation trace 782 and the utility function 780. In at least some embodiments, the detection section 770 includes subsections for implementing additional functions, as described in the preceding flow charts. In at least some embodiments, such subsections are referenced by names associated with the corresponding functions.

[0063] The generating section 772 is circuitry or instructions of the controller 702 configured to generate training samples. In at least some embodiments, the generating section 772 is configured to generate a plurality of training samples from an automated negotiation between the assisting agent and the opposing agent while the assisting agent is operating according to the baseline negotiation strategy model. In at least some embodiments, the generating section 772 utilizes information in the storage unit 704, such as the negotiation trace 782, and records information, such as the training samples 784, in the storage unit 704. In at least some embodiments, the generating section 772 includes subsections for implementing additional functions, as described in the preceding flow charts. In at least some embodiments, such subsections are referenced by names associated with the corresponding functions.

[0064] The training section 774 is circuitry or instructions of the controller 702 configured to train the negotiation strategy model. In at least some embodiments, the training section 774 is configured to train the initialized negotiation strategy model using a plurality of training samples to generate a second negotiation strategy model. In at least some embodiments, the training section 774 utilizes information from the storage unit 704, such as the training samples 784 and the training parameters 786. In at least some embodiments, the training section 774 includes subsections for performing additional functions, as described in the preceding flow charts. In at least some embodiments, such subsections are referenced by names associated with the corresponding functions.

[0065] In at least some embodiments, the apparatus is a separate device capable of processing logical functions to perform the operations herein. In at least some embodiments, the controller and the storage unit need not be completely separate devices, and in some embodiments share circuitry or one or more computer-readable media. In at least some embodiments, the storage unit includes a hard drive that stores both computer-executable instructions and data accessed by the controller, and the controller includes a combination of a central processing unit (CPU) and RAM, and the computer-executable instructions can be copied in whole or in part for execution by the CPU during performance of the operations herein.

[0066] In at least some embodiments where the apparatus is a computer, a program installed on the computer can cause the computer to function as an apparatus of the embodiments described herein or to perform operations associated with the apparatus, and in at least some embodiments, such a program can be executed by a processor to cause the computer to perform specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0067] At least some embodiments are described with reference to flow charts and block diagrams, where the blocks represent (1) steps of a process in which an operation is performed, or (2) sections of a controller responsible for performing an operation. In at least some embodiments, certain steps and sections are performed by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable medium, and / or a processor provided with computer-readable instructions stored on a computer-readable medium. In at least some embodiments, the dedicated circuitry includes digital and / or analog hardware circuitry, including integrated circuits (ICs) and / or discrete circuits. In at least some embodiments, the programmable circuitry includes reconfigurable hardware circuitry comprising logical AND, OR, XOR, NAND, NOR, and other logic operations, flip-flops, registers, memory elements, etc., e.g., field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), etc.

[0068] In at least some embodiments, a computer-readable storage medium includes a tangible device that can hold and store instructions for use by an instruction execution device. In some embodiments, a computer-readable storage medium includes, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), static random access memories (SRAMs), portable compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, punch cards or mechanically encoded devices such as raised structures in grooves with instructions recorded thereon, and any suitable combination of the above. As used herein, a computer-readable storage medium should not be construed as being a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through wires.

[0069] In at least some embodiments, the computer-readable program instructions described herein are downloadable from a computer-readable storage medium to the respective computing / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. In at least some embodiments, the network includes copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. In at least some embodiments, a network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing / processing device.

[0070] In at least some embodiments, the computer readable program instructions for performing the operations described above are either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk, C++, and traditional procedural programming languages ​​such as the "C" programming language or similar programming languages. In at least some embodiments, the computer readable program instructions execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In at least some embodiments, in the latter scenario, the remote computer is connected to the user's computer through any type of network, including a local area network (LAN) or wide area network (WAN), or is connected to an external computer (e.g., through the Internet using an Internet Service Provider). In at least some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), executes computer readable program instructions to individualize the electronic circuitry by utilizing state information of the computer readable program instructions to implement aspects of the present disclosure.

[0071] Although the embodiments of the present disclosure have been described above, the technical scope described in the claims is not limited to the above-mentioned embodiments. Those skilled in the art will understand that various modifications and improvements to the above-mentioned embodiments are possible. Those skilled in the art will also understand from the claims that the embodiments with such modifications or improvements are included in the technical scope of the present disclosure.

[0072] The operations, procedures, steps, and process stages performed by the apparatus, systems, programs, and methods shown in the claims, embodiments, or figures may be performed in any order unless the order is indicated by "before," "prior to," or the like, and unless output from a previous process is used in a later process. Even if a process flow is described in the claims, embodiments, or figures using phrases such as "first" or "next," such description does not necessarily imply that the processes must be performed in the order described.

[0073] According to at least some embodiments of the present disclosure, the automated negotiation agent adaptation is implemented by detecting changes in a utility function associated with an automated negotiation between the assistant agent and the counter agent while the assistant agent is operating according to a first negotiation strategy model, generating a plurality of training samples from the automated negotiation between the assistant agent and the counter agent while the assistant agent is operating according to a baseline negotiation strategy model, and training the initialized negotiation strategy model using the plurality of training samples to generate a second negotiation strategy model.

[0074] Some embodiments include instructions in a computer program, a method implemented by a processor executing the instructions of the computer program, and an apparatus for implementing the method. In some embodiments, the apparatus includes a controller including circuitry configured to perform operations in the instructions.

[0075] The above outlines the features of some embodiments so that those skilled in the art can better understand the aspects of the present disclosure. Those skilled in the art should understand that the present disclosure can be easily used as a basis for designing or modifying other processes and structures for carrying out the same purpose and / or achieving the same advantages as the embodiments introduced herein. Those skilled in the art should also recognize that such equivalent constructions do not depart from the spirit and scope of the present disclosure, and that various changes, substitutions, and alterations can be made herein without departing from the spirit and scope of the present disclosure.

[0076] Some or all of the above exemplary embodiments can be described as follows, but are not limited to the following:

[0077] (Appendix 1) On the computer, Detecting a change in a utility function accompanying an automatic negotiation between the assistant agent and a counter agent while the assistant agent is operating according to a first negotiation strategy model; generating a plurality of training samples from automated negotiations between the assistant agent and the opposing agent while the assistant agent is operating according to a baseline negotiation strategy model; training the initialized negotiation strategy model using the plurality of training samples to generate a second negotiation strategy model; A computer-readable medium comprising instructions executable by said computer for performing operations including:

[0078] (Appendix 2) 2. The computer-readable medium of claim 1, wherein the utility function is an opponent utility function of the opponent agent.

[0079] (Appendix 3) Detecting the change includes: obtaining a recent negotiation trace from an automated negotiation between the assisting agent and the opposing agent while the assisting agent is operating according to the first negotiation strategy model, the negotiation trace including a plurality of time steps, each time step of the plurality of time steps including an opposing agent offer and an assisting agent offer; said assisting agent comparing said opposing agent offer of said recent negotiation trace with an opposing agent offer of a previous negotiation trace from an automated negotiation between said assisting agent and said opposing agent while operating according to said first negotiation strategy model; 3. The computer-readable medium of claim 2, comprising:

[0080] (Appendix 4) The detecting step further includes estimating a change value representing an amount of change in the other party utility function; The generating and training are performed in response to determining that the variance value exceeds a threshold. 4. The computer-readable medium of claim 3.

[0081] (Appendix 5) 3. The computer-readable medium of claim 2, wherein the detecting is performed periodically.

[0082] (Appendix 6) 2. The computer-readable medium of claim 1, wherein the utility function is a first assistance utility function of the assistance agent.

[0083] (Appendix 7) The operation includes: Receiving a second assistance utility function Further comprising: detecting the change includes comparing the first assistance utility function to the second utility function. 7. The computer-readable medium of claim 6.

[0084] (Appendix 8) the comparing includes estimating a change value representing an amount of change between the first assistance utility function and the second assistance utility function; The generating and training are performed in response to determining that the variance value exceeds a threshold. 8. The computer-readable medium of claim 7.

[0085] (Appendix 9) 8. The computer-readable medium of claim 7, wherein the detecting is performed in response to receiving a second assistance utility function.

[0086] (Appendix 10) 2. The computer-readable medium of claim 1, wherein generating the plurality of training samples includes obtaining a negotiation trace from an automated negotiation between the assisting agent and the opposing agent while the assisting agent is operating according to a baseline negotiation strategy model, the negotiation trace including a plurality of time steps, each time step of the plurality of time steps including a opposing agent offer and an assisting agent offer.

[0087] (Appendix 11) 11. The computer-readable medium of claim 10, wherein each training sample of the plurality of samples includes as input a portion of consecutive time steps of the plurality of time steps from a first time step and an opposing agent offer for a subsequent time step of the plurality of time steps following the portion, and further includes a assisting agent offer for the subsequent time step as a label.

[0088] (Appendix 12) Generating the plurality of training samples comprises: weighting each training sample based on a utility value of the final agreed offer of the negotiation trace, the utility value being obtained by applying an auxiliary utility function to the final agreed offer. 12. The computer-readable medium of claim 11, comprising:

[0089] (Appendix 13) The training comprises: generating an initialized negotiation strategy model by initializing a portion of the first negotiation strategy model with a random value; 2. The computer-readable medium of claim 1,

[0090] (Appendix 14) Detecting a change in a utility function accompanying an automatic negotiation between the assistant agent and a counter agent while the assistant agent is operating according to a first negotiation strategy model; generating a plurality of training samples from automated negotiations between the assistant agent and the opposing agent while the assistant agent is operating according to a baseline negotiation strategy model; training the initialized negotiation strategy model using the plurality of training samples to generate a second negotiation strategy model; 4. A computer-implemented method comprising:

[0091] (Appendix 15) 15. The computer-implemented method of claim 14, wherein the utility function is an opponent utility function of the opponent agent.

[0092] (Appendix 16) Detecting the change includes: obtaining a recent negotiation trace from an automated negotiation between the assisting agent and the counter agent while the assisting agent is operating according to the first negotiation strategy model, the negotiation trace including a plurality of time steps, each time step among the plurality of time steps including a counter agent offer and an assisting agent offer; and comparing the counter agent offer of the recent negotiation trace with a counter agent offer of a previous negotiation trace from an automated negotiation between the assisting agent and the counter agent while the assisting agent is operating according to the first negotiation strategy model; 16. The computer-implemented method of claim 15, comprising:

[0093] (Appendix 17) The detecting step further includes estimating a change value representing an amount of change in the other party utility function; The generating and training are performed in response to determining that the variance value exceeds a threshold. 17. The computer-implemented method of claim 16.

[0094] (Appendix 18) 15. The computer-implemented method of claim 14, wherein the utility function is a first assistance utility function of the assistance agent.

[0095] (Appendix 19) Receiving a second assistance utility function Further comprising: detecting the change includes comparing the first assistance utility function to the second utility function. 19. The computer-implemented method of claim 18.

[0096] (Appendix 20) Detecting a change in a utility function accompanying an automatic negotiation between the support agent and a counterpart agent while the support agent is operating according to a first negotiation strategy model; generating a plurality of training samples from automated negotiations between the assistant agent and the opposing agent while the assistant agent is operating according to a baseline negotiation strategy model; training the initialized negotiation strategy model using the plurality of training samples to generate a second negotiation strategy model; A controller including a circuit configured to An apparatus comprising:

Claims

1. On the computer, Detecting a change in a utility function associated with an automatic negotiation between the assistant agent and a counter agent while the assistant agent is operating according to a first negotiation strategy model; generating a plurality of training samples from automated negotiations between the assistant agent and the opposing agent while the assistant agent is operating according to a baseline negotiation strategy model; training the initialized negotiation strategy model using the plurality of training samples to generate a second negotiation strategy model; A computer-readable medium comprising instructions executable by said computer for performing operations including:

2. The computer-readable medium of claim 1 , wherein the utility function is an opponent utility function of the opponent agent.

3. Detecting the change includes: obtaining a recent negotiation trace from an automated negotiation between the assisting agent and the opposing agent while the assisting agent is operating according to the first negotiation strategy model, the negotiation trace including a plurality of time steps, each time step of the plurality of time steps including an opposing agent offer and an assisting agent offer; said assisting agent comparing said counter agent offer of said recent negotiation trace with a counter agent offer of a previous negotiation trace from an automated negotiation between said assisting agent and said counter agent while operating according to said first negotiation strategy model; The computer readable medium of claim 2 , comprising:

4. The detecting step further includes estimating a change value representing an amount of change in the other party utility function; The generating and training are performed in response to determining that the variance value exceeds a threshold. The computer-readable medium of claim 3.

5. The computer-readable medium of claim 2 , wherein the detecting is performed periodically.

6. The computer-readable medium of claim 1 , wherein the utility function is a first assistance utility function of the assistance agent.

7. The operation includes: Receiving a second assistance utility function. Further comprising: detecting the change includes comparing the first assistance utility function to the second utility function. The computer-readable medium of claim 6.

8. the comparing includes estimating a change value representing an amount of change between the first assistance utility function and the second assistance utility function; The generating and training are performed in response to determining that the variance value exceeds a threshold. The computer-readable medium of claim 7.

9. The computer-readable medium of claim 7 , wherein the detecting is performed in response to receiving the second assistance utility function.

10. 2. The computer-readable medium of claim 1, wherein generating the plurality of training samples comprises obtaining a negotiation trace from an automated negotiation between the assisting agent and the opposing agent while the assisting agent is operating according to a baseline negotiation strategy model, the negotiation trace comprising a plurality of time steps, each time step of the plurality of time steps comprising an opposing agent offer and an assisting agent offer.

11. 11. The computer-readable medium of claim 10, wherein each training sample of the plurality of samples includes as input a portion of consecutive time steps of the plurality of time steps from a first time step and an opposing agent offer for a subsequent time step of the plurality of time steps following the portion, and further includes a assisting agent offer for the subsequent time step as a label.

12. Generating the plurality of training samples comprises: weighting each training sample based on a utility value of the final agreed offer of the negotiation trace, the utility value being obtained by applying an auxiliary utility function to the final agreed offer. The computer readable medium of claim 11 , comprising:

13. The training comprises: generating an initialized negotiation strategy model by initializing a portion of the first negotiation strategy model with a random value; The computer readable medium of claim 1 , comprising:

14. Detecting a change in a utility function associated with an automatic negotiation between the assistant agent and a counter agent while the assistant agent is operating according to a first negotiation strategy model; generating a plurality of training samples from automated negotiations between the assistant agent and the opposing agent while the assistant agent is operating according to a baseline negotiation strategy model; training the initialized negotiation strategy model using the plurality of training samples to generate a second negotiation strategy model; 4. A computer-implemented method comprising:

15. The computer-implemented method of claim 14 , wherein the utility function is an opponent utility function of the opponent agent.

16. Detecting the change includes: obtaining a recent negotiation trace from an automated negotiation between the assisting agent and the counter agent while the assisting agent is operating according to the first negotiation strategy model, the negotiation trace including a plurality of time steps, each time step among the plurality of time steps including a counter agent offer and an assisting agent offer; and comparing the counter agent offer of the recent negotiation trace with a counter agent offer of a previous negotiation trace from an automated negotiation between the assisting agent and the counter agent while the assisting agent is operating according to the first negotiation strategy model; 20. The computer-implemented method of claim 15, comprising:

17. The detecting step further includes estimating a change value representing an amount of change in the other party utility function; The generating and training are performed in response to determining that the variance value exceeds a threshold.

20. The computer-implemented method of claim 16.

18. The computer-implemented method of claim 14 , wherein the utility function is a first assistance utility function of the assistance agent.

19. Receiving a second assistance utility function. Further comprising: detecting the change includes comparing the first assistance utility function to the second utility function.

20. The computer-implemented method of claim 18.

20. Detecting a change in a utility function accompanying an automatic negotiation between the assisting agent and a counter agent while the assisting agent is operating according to a first negotiation strategy model; generating a plurality of training samples from automated negotiations between the assistant agent and the opposing agent while the assistant agent is operating according to a baseline negotiation strategy model; training the initialized negotiation strategy model using the plurality of training samples to generate a second negotiation strategy model; A controller including a circuit configured to An apparatus comprising:

Images