Multi-entity mediation system, and multi-entity mediation method
The multi-subject mediation system addresses the imbalance in existing decision-making systems by calculating decisions that balance overall and individual indicators, ensuring fairness through reinforcement learning and optimization, leading to improved outcomes for all entities.
Patent Information
- Application Number
- JP2023077415
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-09
- Publication Date
- 2024-11-21
AI Technical Summary
Existing decision-making systems fail to effectively balance overall and individual indicators, leading to suboptimal outcomes where entities improve individual indicators at the expense of overall indicators, resulting in a Nash equilibrium with fairness discrepancies.
A multi-subject mediation system that calculates decisions based on input subject data and evaluation functions for both overall and individual indices, using reinforcement learning and mathematical optimization to propose decisions that consider both types of indicators, incorporating fairness constraints.
The system enables balanced decision-making that considers both overall and individual indicators, allowing entities to assess proposed decisions and their future impacts, thereby improving overall and individual outcomes while ensuring fairness.
Smart Images

Figure 2024162146000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a multi-master shutdown system and a multi-master shutdown method. [Background technology]
[0002] Multiple agents make decisions in the same environment, and the state of the environment changes over time depending on each decision, and the indicators of each agent are determined by each decision and state transition. Here, we consider the case where there are multiple overall indicators that are common to each agent, and multiple individual indicators that are specific to each agent. In this case, the overall indicators and the individual indicators are not necessarily in a win-win relationship, but may be in a trade-off relationship. We assume that all agents do not set weight coefficients for each overall indicator that are sufficiently larger than the individual indicators, and that the game is set as a non-cooperative game. In this case, each agent improves each individual indicator of the other agents, and the overall indicators cannot be improved, or some agents improve their overall indicators while sacrificing their own individual indicators, resulting in unfairness among agents, converging to a Nash equilibrium.
[0003] Patent Document 1 discloses a computer system in which terminal devices of multiple users are connected to a server via a network, the server optimizes plans transmitted from each of the multiple terminal devices, and provides the optimized plans to the terminal devices of each of the multiple users, wherein the server has a controller that executes a program stored in a memory, and the controller evaluates the plans transmitted from each of the multiple terminals, estimates a utility function for each of the multiple users based on the results of the evaluation, sets a plan to be recommended to the multiple users based on the utility function, transmits the recommended plan to the terminal devices of each of the multiple users, repeats the evaluation and setting of the recommended plan until the multiple users agree on the recommended plan, and determines the recommended plan agreed upon by the multiple users as the optimized plan. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2023-006347 A Summary of the Invention [Problem to be solved by the invention]
[0005] In the invention described in Patent Document 1, there is room for improvement in the calculation of the decision-making of each subject. [Means for solving the problem]
[0006] A multi-subject mediation system according to a first aspect of the present invention is a multi-subject mediation system that mediates the decisions of multiple subjects that affect at least one overall indicator, a target overall indicator, and at least one individual indicator, a target individual indicator, and includes a decision calculation unit that calculates the decision of each of the multiple subjects based on subject data, which is observations of the multiple subjects input from each of the multiple subjects, and evaluation functions corresponding to each of the target overall indicator and the target individual indicator, and proposes the decision to each of the multiple subjects. A second aspect of the present invention is a multi-subject mediation method executed by a computer for reconciling decisions of multiple subjects affecting a target overall indicator, which is at least one overall indicator, and a target individual indicator, which is at least one individual indicator, the method including obtaining subject data, which is observations of the multiple subjects, from each of the multiple subjects, calculating a decision of each of the multiple subjects based on an evaluation function corresponding to each of the target overall indicator and the target individual indicator, and proposing the decision to each of the multiple subjects. Effect of the Invention
[0007] According to the present invention, a decision-making process for multiple entities that takes into account both overall and individual indicators can be calculated and presented to each entity as a proposal. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]
[0008] [Figure 1] Diagram of the multi-maintenance outage system [Diagram 2] Flowchart of multi-maintenance health outage system [Diagram 3] Diagram of the decision-making proposal system [Figure 4] Diagram of the proposal basis generation system [Diagram 5] Configuration diagram of the adjustment request application system [Figure 6] Diagram of a multi-master shutdown system including reinforcement learning functionality [Figure 7] Reinforcement learning block diagram [Figure 8] Hardware configuration diagram of the multi-maintenance shutdown system [Figure 9] User interface for multi-maintenance shutdown system [Figure 10] User interface for multi-maintenance shutdown system [Figure 11] Configuration diagram of multi-master shutdown system in variant 1 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] --First embodiment-- Hereinafter, a first embodiment of a multi-main shutdown system will be described with reference to Figs.
[0010] FIG. 1 is a functional configuration diagram of a multi-master health shutdown system 100. The multi-master health shutdown system 100 exchanges various information with multiple subjects. In FIG. 1, only subject i is shown as a representative of the multiple subjects, but in reality there are multiple subjects. For example, there are subject 1, subject 2, ..., subject |I|. A subject is something that acts on and has an influence on the environment and other subjects, such as a system or a human being. The specific content exchanged between each subject and the multi-master health shutdown system 100 is different, but the meaning of the data is the same.
[0011] Each subject exchanges data with the multi-master health shutdown system 100 via the data input / output unit 101. The data input / output unit 101 is, for example, a general-purpose computer. When a subject, a human being, inputs input data 102 into the data input / output unit 101 using a keyboard or mouse, the data input / output unit 101 transmits the input data 102 to the multi-master health shutdown system 100 as subject data 104, either as is or after processing. The input data 102 and subject data 104 also include observations of each subject i. The processing of the input data 102 by the data input / output unit 101 is, for example, the addition of a subject identifier. The data input / output unit 101 further outputs the data received from the multi-master health shutdown system 100 as output information 103, which is video or audio. However, when the subject is a system, the input data 102 and the output information 103 may be electronic data. Furthermore, when the subject is a system, the data input / output unit 101 is not a required component, and the subject may directly exchange various data with the multi-master health shutdown system 100. In this case, the subject transmits subject data 104 directly to the multi-subject health shutdown system 100 .
[0012] Each subject receives a proposal 113 corresponding to the subject data 104 transmitted from the multi-subject health shutdown system 100. The subject that receives the proposal 113 may optionally output at least one of an additional presentation request 105 and a decision-making adjustment request 106 via the data input / output unit 101. That is, each data input / output unit 101 always outputs the subject data 104, but may not transmit both the additional presentation request 105 and the decision-making adjustment request 106, may transmit only one of them, or may transmit both.
[0013] The multi-subject health shutdown system 100 comprises a decision making proposal system 110, a proposal basis generation system 1200, and an adjustment request application system 1300. Upon receiving subject data 104 from all subjects, the decision making proposal system 110 performs the process described below to generate a proposal 113 and transmits it to each data input / output unit 101. Upon receiving an additional presentation request 105 from each subject, the proposal basis generation system 1200 performs the process described below for the subject that sent the additional presentation request 105 and generates a future intention 1203. The proposal basis generation system 1200 transmits the generated future intention 1203 to the subject that sent the additional presentation request 105.
[0014] When the adjustment request application system 1300 receives a decision-making adjustment request 106 from each entity, it performs the processing described below for the entity that sent the decision-making adjustment request 106 and generates a future intention 1303. The adjustment request application system 1300 transmits the generated future intention 1303 to the entity that sent the decision-making adjustment request 106. In other words, the proposal basis generation system 1200 and the adjustment request application system 1300 may perform processing for some entities and transmit data to only some of those entities. The entities that the proposal basis generation system 1200 processes and the entities that the adjustment request application system 1300 processes may be the same or different.
[0015] 2 is a flowchart showing the processing of the multi-subject health shutdown system 100. In step S400, the subject data 104 related to all subjects is acquired from the data input / output unit 101. In the following step S401, the decision-making proposal system 110 calculates the decision-making of each subject as a proposal 113 using the subject data 104 acquired in step S400, and outputs the calculated proposal 113 to the data input / output unit 101 of each subject and the proposal basis generation system 1200. In step S402, the proposal basis generation system 1200 calculates the proposal basis 1237 based on the proposal 113 calculated in step S401, and transmits it to the data input / output unit 101 of each subject. Furthermore, the decision-making proposal system 110 outputs the proposal basis 1237 of all subjects to the adjustment request application system 1300 and proceeds to step S403.
[0016] In step S403, the proposal basis generation system 1200 judges whether there is a further future request for basis presentation from each of the subjects. If the proposal basis generation system 1200 receives a further future request for basis presentation from any of the subjects, it proceeds to step S404, and if it has not received a further future request for basis presentation from any of the subjects, it proceeds to step S405. In step S404, the proposal basis generation system 1200 calculates a further future proposal basis 1237 for the subject that made the request, transmits it to the data input / output unit 101 of the subject that made the request, and returns to step S403.
[0017] In step S405, the adjustment request application system 1300 judges whether or not there is a decision-making adjustment request 106 from any of the entities. If there is a decision-making adjustment request 106 from any of the entities, the adjustment request application system 1300 proceeds to step S406, and if there is no decision-making adjustment request 106 from any of the entities, it ends the processing shown in Fig. 2. In step S406, the adjustment request application system 1300 calculates the impact of applying the requested adjustment based on the decision-making adjustment requests 106 from each entity, outputs it to the data input / output unit 101 of each entity, and returns to step S405.
[0018] 3 is a block diagram showing the functional configuration of the decision-making proposal system 110. The decision-making proposal system 110 calculates decisions to be proposed to each subject. The decision-making proposal system 110 has a decision-making calculation unit 111 and an environment model 112 that models the behavior of the environment.
[0019] The decision-making calculation unit 111 has an evaluation function capable of calculating at least one overall index and one individual index. Instead of integrating multiple indexes to calculate one overall evaluation value, the decision-making calculation unit 111 calculates each index separately. In this embodiment, these evaluation functions that calculate evaluation indexes separately are also called "evaluation functions divided for multiple overall indexes and multiple individual indexes." The evaluation function j in the decision-making calculation unit 111 is the j-th evaluation function of the divided evaluation functions. The evaluation function j may also include a penalty function of a constraint condition related to fairness as an index.
[0020] The decision-making proposal system 110 has an algorithm for calculating a proposal 113 based on subject data 104, which is information on each subject i obtained via each data input / output unit 101, and each divided evaluation function j in the decision-making calculation unit 111. This algorithm may use mathematical optimization as in prior art documents, or may use reinforcement learning as described below. The proposal 113 calculated by this algorithm is displayed as output information 103 to the subject i via the data input / output unit 101.
[0021] When the algorithm in the decision-making calculation unit 111 is based on mathematical optimization, a proposal 113 at each time is input to the environmental model 112. The environmental model 112 is a model created in advance. When the proposal 113 at the current time (e.g., time t0) and the subject data 104 of each subject i are input, the environmental model 112 outputs an environmental state 115 that is the state of the environment at the next time (e.g., time t1) and a subject observation 116 that is the observation of each subject i at the next time. When the algorithm in the decision-making calculation unit 111 is based on mathematical optimization, the algorithm in the decision-making calculation unit 111 uses the environmental state 115 at the next time output by the environmental model 112 and the subject observation 116 of each subject i at the next time to calculate the proposal 113.
[0022] FIG. 4 is a block diagram showing the functional configuration of the proposal basis generation system 1200. In the following description, the current time is time t0, and future times are times t1, t2, .... The proposal basis generation system 1200 calculates the basis for the proposal 113 at time t0 calculated by the decision making proposal system 110. The proposal basis generation system 1200 has a proposal basis calculation unit 1201 and an environmental model 1202 that models the behavior of the environment. However, the environmental model 1202 has a configuration similar to that of the environmental model 112 shown in FIG. 3. The proposal basis calculation unit 1201 has an evaluation function j similar to the decision making calculation unit 111. In addition, the evaluation function j may also include a penalty function of a constraint condition related to fairness as an index. The proposal basis calculation unit 1201 has an algorithm similar to that of the decision making calculation unit 111.
[0023] The proposal basis calculation unit 1201 receives as input the subject data 104 at time t0 obtained via each data input / output unit 101, the proposal 113 at time t0 calculated by the decision calculation unit 111, the environmental state 1209 at each time output from the environmental model 1202, and subject observation 1210 which is the observation of each subject i at each time. The proposal basis calculation unit 1201 uses the evaluation function j to calculate future intention 1203 which is a decision planned for the future, and inputs the calculated future intention 1203 to the environmental model 1202.
[0024] Future intention 1203 is the same type of information as proposal 113 calculated by decision making proposal system 110, but the time axis they target is different. That is, proposal 113 is an optimal decision at time t0, which is the current time, and future intention 1203 is an optimal decision at future times t1, t2, .... Proposal 113 and future intention 1203 have in common that they are calculated by multi-master health shutdown system 100 at time t0, but differ in which time they target.
[0025] The environmental model 1202 receives as input proposals 1208, which are the decisions of each subject at each time calculated by the proposal basis calculation unit 1201, and subject data 104 for each subject i. The environmental model 1202 calculates an environmental state 1209, which is the state of the environment at the next time, and subject observations 1210, which are the observations of each subject i at the next time, and outputs these to the proposal basis calculation unit 1201. Since the environmental model 1202 has already calculated the subject observations 1210 at the next time, it uses this as a substitute for the subject data 104. In other words, the subject data 104 is used as an initial value for the subject observations 1210. In this way, the proposal basis calculation unit 1201 and the environmental model 1202 work together to successively calculate the environment and optimal decision-making at future times.
[0026] The proposal basis calculation unit 1201 calculates a future index 1205 which is an index for the subject i at each time in the future, based on the evaluation function j, the calculated future intention 1203, and the environmental state 1209 at each time calculated by the environmental model 1202. Based on the calculated future index 1205, the proposal basis calculation unit 1201 calculates a fair state 1206 which is a state related to fairness defined by a constraint condition related to fairness in the future, a time series total value 1207 which is a time series total value for each index at each time in the future, and a time series total value 1207 for each index at each time in the future.
[0027] Hereinafter, the future intention 1203, future index 1205, fair state 1206, and time series total value 1207 calculated by the proposal basis calculation unit 1201 and the subject observation 1210 calculated by the environmental model 1202 are collectively referred to as proposal basis 1237. The proposal basis calculation unit 1201 transmits the proposal basis 1237 to the data input / output unit 101 of each subject i. The data input / output unit 101 presents the received data to the subject i as output information 103. The subject i can refer to the future intention 1203, future index 1205, fair state 1206, time series total value 1207, and subject observation 1210 to determine whether to accept the currently proposed decision-making.
[0028] The proposal basis generating system 1200 calculates the future intention 1203 and subject observation 1210 for all subjects, but may calculate the future index 1205, the fair state 1206, and the time series total value 1207 only for the subject i for which calculation is required. Also, the proposal basis 1237 sent to each data input / output unit 101 does not need to include data related to other subjects i.
[0029] The subject i can transmit an additional presentation request 105 to the proposal basis generation system 1200 via the data input / output unit 101. When the proposal basis generation system 1200 receives the additional presentation request 105 from the subject i, it calculates the proposal basis 1237 at a future time and transmits it to the data input / output unit 101 that transmitted the additional presentation request 105. However, it is assumed that each additional presentation request 105 includes an identifier of the subject i, or that the corresponding subject i can be identified using the identifier of the data input / output unit 101 that transmitted the additional presentation request 105. The additional presentation request 105 may be data that at least clearly indicates that a specific subject i is requesting additional data.
[0030] The proposal basis generation system 1200 may calculate and send data up to a specified future time without receiving any particular instruction. For example, when the decision making proposal system 110 calculates the proposal 113 at time t0, the proposal basis generation system 1200 may calculate only the proposal basis 1237 at time t0, or may calculate the proposal basis 1237 from time t0 to t2. The proposal basis 1237 calculated by the proposal basis generation system 1200 without receiving any particular instruction is transmitted to all data input / output units 101, and the proposal basis 1237 calculated upon receiving a request for additional presentation 105 is transmitted to the data input / output unit 101 that transmitted the request for additional presentation 105.
[0031] FIG. 5 is a block diagram showing the functional configuration of the adjustment request application system 1300. The adjustment request application system 1300 operates when it receives a decision-making adjustment request 106. The subject i transmits the decision-making adjustment request 106 to the adjustment request application system 1300 using the data input / output unit 101. The decision-making adjustment request 106 includes a value desired by a specific subject i. For example, when the proposal 113 is "$5" for the unit price of a price that a specific subject i has the decision-making power to decide, the decision-making adjustment request 106 includes a different price such as "$8" as an adjustment item. When the adjustment items included in the decision-making adjustment request 106 are reflected, the adjustment request application system 1300 calculates the degree of influence 1337 on each proposal basis. However, even when the decision-making adjustment request 106 is received from multiple subjects i, the calculation in the adjustment request application system 1300 is performed independently for each subject i. In other words, the adjustment request application system 1300 performs calculations using the value of the proposal 113 for subjects other than the subject i that is the calculation target.
[0032] The adjustment request application system 1300 has an application calculation unit 1301, a difference calculation unit 1311, and an environment model 1302. The application calculation unit 1301 has an evaluation function j, similar to the decision-making calculation unit 111. Each evaluation function may also include a penalty function of a constraint condition related to fairness as an index. The application calculation unit 1301 has an algorithm similar to that of the decision-making calculation unit 111 and the proposal basis calculation unit 1201. The environment model 1302 has a configuration similar to that of the environment model 112 shown in FIG. 3 and the environment model 1202 shown in FIG. 4, and performs the same operation.
[0033] The application calculation unit 1301, like the proposal basis calculation unit 1201, calculates a future intention 1303, a future index 1305, a fair state 1306, and a time series total value 1307 at each time. The difference between the application calculation unit 1301 and the proposal basis calculation unit 1201 is that the calculation is performed for the current time, that is, time t0. Specifically, like the proposal basis calculation unit 1201, the application calculation unit 1301 calculates the ideal decision of each entity at time t0, but uses the value of the received decision adjustment request 106 only for the decision of the entity i to be processed. The calculations performed by the application calculation unit 1301 for times after t1 are the same as those performed by the proposal basis calculation unit 1201.
[0034] The correspondence between Figure 4 and Figure 5 is as follows: future intention 1203 corresponds to future intention 1303, future indicator 1205 corresponds to future indicator 1305, fair state 1206 corresponds to fair state 1306, time series total value 1207 corresponds to time series total value 1307, environmental state 1209 corresponds to environmental state 1309, and subject observation 1210 corresponds to subjective observation 1310. The difference in these correspondences is that future intention 1303 differs from future intention 1203 in that the decision for subject i that is the subject of calculation at time t0 is not calculated, but is a value included in decision adjustment request 106 desired by subject i. Due to this difference, the specific values of the future index 1305, the fair state 1306, the time series total value 1307, the environmental state 1309, and the subjective observation 1310 are different from the future index 1205, the fair state 1206, the time series total value 1207, the environmental state 1209, and the subjective observation 1210. The difference calculation unit 1311 calculates the difference between the time series total value 1307 calculated by the application calculation unit 1301 and the time series total value 1207 calculated by the proposal basis generation system 1200 as a request difference 1312.
[0035] In the following, the future intention 1303, future index 1305, fair state 1306, and time series total value 1307 calculated by the subjective observation 1310, the subjective observation 1310 calculated by the environmental model 1302, and the requested difference 1312 calculated by the difference calculation unit 1311 are collectively referred to as the influence 1337. The adjustment request application system 1300 transmits the influence 1337 to the data input / output unit 101 that transmitted the decision-making adjustment request 106. The data input / output unit 101 outputs the received influence 1337 as the output information 103. Each subject i can refer to the influence 1337 to determine whether or not to accept the proposal 113 output from the decision-making proposal system 110.
[0036] Fig. 6 is a block diagram showing the functional configuration of the multi-master health shutdown system when the algorithms of the decision making proposal system 110, the proposal basis generation system 1200, and the adjustment request application system 1300 are reinforcement learning. In Fig. 6, a learning unit 600, which will be described later, exists outside the multi-master health shutdown system 100, but the learning unit 600 may be included in the multi-master health shutdown system 100. The outline of the operation of the decision making proposal system 110, the proposal basis generation system 1200, and the adjustment request application system 1300 has been described above. Below, differences when using reinforcement learning instead of mathematical optimization will be mainly described.
[0037] The decision-making calculation unit 111 shown in Fig. 6 has an agent model 211 having a Q function divided for each divided evaluation function j, with each divided evaluation function j being a reward. The Q function j in the agent model 211 is the j-th Q function of the divided Q functions. The agent model 211 is Q learning based on each divided Q function j, or reinforcement learning by Actor-Critic.
[0038] 6, when the agent model 211 is Q-learning, calculates proposal 113, which is the decision-making of each agent to be proposed, based on parameters 601 synchronized from the learning unit 600 and the environmental state s(t) included in the agent data 104 obtained from the data input / output unit 101, according to Equation 1 of the agent model. Here, observations of each agent i, which will be described later, may be input instead of the environmental state.
[0039]
number
[0040] Here, a1, a2, . . ., a in Equation 1 |I| is the decision-making of each entity, c j is the weighting coefficient for evaluation function j, Q j is the Q function j, and J is a set of the divided evaluation functions j. The proposals 113, which are the calculated decisions of each subject, are displayed as output information 103 to the subject i via the data input / output unit 101.
[0041] 6, when the agent model 211 is Actor-Critic, the decision making proposal system 110 calculates a proposal 113 by Equation 2 of the agent model based on the parameters 601 synchronized from the learning unit 600 and the environmental state s(t) included in the subject data 104 obtained from the data input / output unit 101. Here, instead of the environmental state, the observation of each subject i, which will be described later, may be input.
[0042]
number
[0043] Here, a1, a2, . . ., a in Equation 2 |I| is the decision-making process of each entity, π actor is a conditional probability distribution and is the policy of the actor model. Here, in Equation 2, a1, a2, . . ., a |I| is π actorThe calculated proposal 113 is displayed as output information 103 to the subject i via the data input / output unit 101.
[0044] 6 has an agent model 221 having each divided Q function j, similar to the agent model 211. It also has an environment model 1202 that models the behavior of the environment. The agent model 221 is Q-learning or Actor-Critic reinforcement learning based on each divided Q function j.
[0045] 6 executes a rollout of the agent model 221 at each time t according to Equation 1 when the agent model 221 is Q-learning. For this rollout, parameters 601 synchronized from the learning unit 600, the state s(t) of the environment included in the subject data 104 obtained from the data input / output unit 101, the calculated proposal 113, each divided evaluation function j, and an environment model 1202 are used. Note that instead of the state of the environment, observations of each subject i, which will be described later, may be input.
[0046] Here, rollout refers to the following series of calculations. Based on the state of the environment at a certain time, the decision-making of each agent is calculated and input into the environmental model, which outputs the state of the environment at the next time and the observations of each agent. Based on the calculated state of the environment at the next time, each indicator is calculated using each divided evaluation function j, and in some cases the state related to fairness is also calculated. Thereafter, similar calculations are repeated based on the state of the environment at the next time, thereby calculating the decision-making of each agent, the state of the environment, the observations of each agent, each indicator, and the state related to fairness at each time.
[0047] 6 executes a rollout of the agent model 221 at each time t according to Equation 2 when the agent model 221 is Actor-Critic. This rollout uses parameters 601 synchronized from the learning unit 600, the state of the environment s(t) included in the subject data 104 obtained from the data input / output unit 101, the calculated proposal 113, each divided evaluation function j, and an environment model 1202. Note that instead of the state of the environment, observations of each subject i, which will be described later, may be input.
[0048] The proposal basis generation system 1200 in Fig. 6 calculates a future intention 1203, a future indicator 1205, a fair state 1206, and a subject observation 1210 through rollout. The proposal basis generation system 1200 in Fig. 6 calculates each divided Q function j as a time series total value 1207, which is a time series total value for each indicator for each time. The proposal basis generation system 1200 outputs the calculated proposal basis 1237 to the data input / output unit 101. The data input / output unit 101 displays the received proposal basis 1237 to the subject i as output information 103. Based on the proposal basis 1237, the subject i can determine whether or not to accept the current proposed decision-making.
[0049] When the proposal basis generation system 1200 receives an additional presentation request 105 from the subject i, it calculates a further future proposal basis 1237 by rollout and transmits it to the data input / output unit 101 corresponding to the subject i. Based on the further future information, the subject i can decide whether or not to accept the current proposed decision-making.
[0050] 6 includes an agent model 231 having each divided Q function j, and an environment model 1302 that models the behavior of the environment, similar to the agent model 211. The adjustment request application system 1300 further includes a difference calculation unit 1311 for calculating an influence degree 1337 when the decision-making adjustment request 106 is applied. The agent model 231 is Q-learning or Actor-Critic reinforcement learning based on each divided Q function j.
[0051] 6 uses the following data when executing the rollout of the agent model 231 according to Formula 1 at each time t when the agent model 231 is Q-learning. That is, the adjustment request application system 1300 uses the parameters 601 synchronized from the learning unit 600, the state s(t) of the environment included in the subject data 104 obtained from the data input / output unit 101, each divided evaluation function j, and an environment model 1302. Note that instead of the state of the environment, the observation of each subject i described later may be input.
[0052] 6, when the agent model 231 is Actor-Critic, executes rollout of the agent model 231 at each time t according to Equation 2. For this execution, parameters 601 synchronized from the learning unit 600, the state s(t) of the environment included in the subject data 104 obtained from the data input / output unit 101, each divided evaluation function j, and an environment model 1302 are used. Note that instead of the state of the environment, observations of each subject i, which will be described later, may be input.
[0053] The adjustment request application system 1300 in Fig. 6 calculates each divided Q function j as a time series total value 1307 based on the decision making adjustment request 106 from the subject i, the parameters 601 synchronized from the learning unit 600, and the environmental state s(t) included in the subject data 104 obtained via each data input / output unit 101. The difference calculation unit 1311 calculates the difference between the time series total value 1207 calculated by the proposal basis generation system 1200 and the time series total value 1307 calculated by the application calculation unit 1301 as a request difference 1312, as described with reference to Fig. 5.
[0054] 7 is a block diagram showing the functional configuration of the learning unit 600. The learning unit 600 includes a parameter management unit 240, a data collection unit 250, and a data storage unit 260. The parameter management unit 240 updates each parameter of the agent model, holds the parameters, and synchronizes the parameters 601 with each agent model in FIG.
[0055] The parameter management unit 240 has an agent model 241 having each divided Q function j, similar to the agent model 211. The parameter management unit 240 updates each parameter based on a data set 603 obtained from the data storage unit 260. When each agent model is Q-learning, the parameter management unit 240 updates each parameter by updating the loss function L in Equation 3. j The parameters of each Q function in the agent model 241 are updated so as to minimize
[0056]
number
[0057] Here, a1, a2, . . ., a in Equation 3 |I| is the decision-making of each entity, Q j are the Q functions j, r in the agent model 241 j is the reward according to the divided evaluation function j obtained from the data set 603, γ is the discount rate, Q target j is the target network corresponding to Q function j in the agent model 241, J is the set of divided evaluation functions j, c j is a weighting coefficient for the evaluation function j. When each agent model is an actor-critic model, the parameter management unit 240 updates each parameter by using the loss function L j The parameters of the Q function of each critic model in the agent model 241 are updated so as to minimize
[0058]
number
[0059] Here, a1, a2, . . ., a in Equation 4 |I| is the decision-making of each entity, Q j are the Q functions j, r in the agent model 241 j is the reward according to the divided evaluation function j obtained from the data set 603, γ is the discount rate, Q target j is the target network corresponding to the Q function j in the agent model 241, and π actor is a conditional probability distribution and is the actor model's policy. The actor model is trained appropriately depending on the type of actor-critic.
[0060] The data collection unit 250 has a decision-making calculation unit 251 and an environment model 252 that models the behavior of the environment. The decision-making calculation unit 251 has each divided evaluation function j, and has an agent model 253 having each divided Q function j, similar to the agent model 211 in FIG. 6.
[0061] The data collection unit 250 collects the state of the environment, the decision-making of each subject, and each reward calculated by each divided evaluation function j as transition data through a simulation using the agent model 253 and the environment model 252, and stores the data in the data storage unit 260. Here, instead of the state of the environment, observations of each subject i, which will be described later, may be used.
[0062] When the agent model 253 is Q-learning, the data collector 250 calculates a(t), which is the decision-making 255 of each subject, from s(t), which is the state of the environment 254 output from the environment model 252, using Equation 5 based on parameters 601 synchronized from the parameter manager 240, and inputs the result to the environment model 252. Here, instead of the state of the environment 254, observations 256 of each subject i, which will be described later, may be input.
[0063]
number
[0064] Here, a1, a2, . . ., a in Equation 5 |I| is the decision-making of each entity, C j is the weighting coefficient for evaluation function j, Q j is a Q function j in the agent model 253, J is a set of divided evaluation functions j, and ε is a real number from 0 to 1. s(t+1), which is the state 254 of the environment at the next time, and each observation 256 at the next time are calculated from the environmental model 252, and each reward is calculated by each divided evaluation function j in the decision-making calculation unit 251. By repeating this process thereafter, transition data 602 is collected and stored in the data storage unit 260.
[0065] When the agent model 253 is Actor-Critic, the data collection unit 250 calculates the parameters 601 synchronized from the parameter management unit 240, s(t) which is the state of the environment 254 output from the environment model 252, and a(t) which is the decision-making 255 of each subject using Equation 2, and inputs the calculated values to the environment model 252. Note that instead of the state of the environment 254, the observation 256 of each subject i described later may be input. The environment model 252 calculates s(t+1) which is the state of the environment 254 at the next time, and each observation 256 at the next time, and each reward is calculated by each divided evaluation function j in the decision-making calculation unit 251. By repeating this process thereafter, the data is collected as transition data 602 and stored in the data storage unit 260.
[0066] The data storage unit 260 stores the transition data 602 stored from the data collection unit 250. When each parameter of the parameter management unit 240 is updated, the data storage unit 260 creates a data set 603 from the stored transition data 602 and outputs the data set 603 to the parameter management unit 240.
[0067] Although each agent model has treated the state of the environment as an input, instead of the state of the environment, the observation of each subject i may be input. Furthermore, the observation may be partially masked due to confidentiality of the information. Furthermore, each agent model may appropriately use a time series model such as an RNN (Recurrent Neural Network) to extract features from each past observation information, and model each Q function or model an actor model. With this time series modeling, even if the multi-subject health shutdown system 100 cannot access all information, it is possible to calculate the proposal 113 and the proposal basis 1237 by extracting features from each past observation information that can be accessed.
[0068] Each agent model may be solved by single-agent reinforcement learning that combines policies into one, or by multi-agent reinforcement learning that divides policies for each decision as necessary. When solving by multi-agent reinforcement learning, agents may communicate with each other as necessary.
[0069] The above formulas 1 to 5 are for the case where a single agent model is adopted. When a multi-agent model is adopted for the agent model, formulas 1 to 5 can be replaced with the following formulas 6 to 10. However, formulas 6 to 8 and 10 are written for a certain subject, so when calculating for other subjects, a1 should be replaced with a2, a3, . . ., a |I| is changed to and calculated.
[0070]
number
[0071]
number
[0072]
number
[0073]
number
[0074]
number
[0075] 8 is a block diagram showing the hardware configuration of the multi-main system for shutting down the system 100. The multi-main system for shutting down the system 100 includes a server 310 and a plurality of terminal devices 320. The server 310 includes a controller 311 and a server communication device 312. The controller 311 includes a decision-making proposal system 110, a proposal basis generating system 1200, and an adjustment request applying system 1300. The controller 311 includes a CPU, a ROM, and a RAM, which are central processing units (not shown), and realizes the decision-making proposal system 110, the proposal basis generating system 1200, and the adjustment request applying system 1300 by expanding a program stored in the ROM into the RAM and executing it with the CPU.
[0076] However, instead of a combination of a CPU, ROM, and RAM, the decision-making proposal system 110, the proposal basis generating system 1200, and the adjustment request applying system 1300 may be realized by an FPGA (Field Programmable Gate Array), which is a rewritable logic circuit, or an ASIC (Application Specific Integrated Circuit), which is an application specific integrated circuit. Also, instead of a combination of a CPU, ROM, and RAM, the decision-making proposal system 110, the proposal basis generating system 1200, and the adjustment request applying system 1300 may be realized by a combination of different configurations, for example, a combination of a CPU, ROM, RAM, and an FPGA.
[0077] The terminal device 320 includes an input device 321, an output device 322, and a terminal communication device 323. The terminal device 320 corresponds to the data input / output unit 101 in FIG. 1. The terminal device 320 is, for example, a general-purpose computer. A terminal device 320 exists for each subject i. The terminal device 320 transmits information input by the subject i using the input device 321 to the server 310 via the terminal communication device 323 and the network 330.
[0078] Based on the input information of each subject, the server 310 calculates the proposal 113, proposal basis 1237, and impact 1337 using the decision making proposal system 110, proposal basis generation system 1200, and adjustment request application system 1300 in the controller 311. The server 310 transmits the calculated proposal 113, proposal basis 1237, and impact 1337 to the terminal device 320 via the server communication device 312 and the network 330.
[0079] The input device 321 is a device for inputting information that each subject desires to input, such as a keyboard, mouse, microphone, touch panel, etc. The output device 322 is a device for outputting data and information within the system, such as a display, speaker, printer, etc.
[0080] 9 and 10 are diagrams showing an example of the output information 103 presented to the subject i by the data input / output unit 101. Here, the output device 322 is a display or a printer, and is shown as an example of a user interface that visually shows the output information 103.
[0081] U500 in Fig. 9 is a screen display that shows each subject the proposal 113, which is the current decision proposed by the multi-subject health shutdown system 100, and information about the expected time series total value for each index or the Q function when all subjects i accept the proposal 113. Each subject can confirm the proposed current decision and the expected time series total value for each index or the information about the Q function by referring to the display shown in U500.
[0082] 9 is a screen display showing, for each subject, the future intention 1203, subject observation 1210 which is the observation information of each subject in the future, the value of each index at each time, and the fairness state 1206. Each subject i can confirm the proposal basis 1237 by using U501.
[0083] U502 in Fig. 10 is a screen display that shows, for each subject, the decision that subject i, included in the decision-making adjustment request 106, is requesting adjustment, and the time series total value or Q function and time series total value for each index that is expected when this adjustment is applied. Each subject can check the impact 1337 of the adjustment request by comparing U500 and U502. Note that the difference before and after applying the adjustment request may also be displayed in the time series total value for each index in U502.
[0084] 10 is a screen display showing for each subject the future intention 1303 when the adjustment request is applied, the subjective observation 1310 which is the observation information of each subject in the future, the value of each index at each time, and the fairness state 1306. Each subject can check the impact 1337 of the adjustment request by comparing U501 with I503.
[0085] According to the above-described first embodiment, the following advantageous effects can be obtained. (1) The multi-subject matter control system 100 mediates the decision-making of multiple subjects that affects at least one overall indicator, a target overall indicator, and at least one individual indicator, a target individual indicator. The multi-subject matter control system 100 includes a decision-making calculation unit 111 that calculates the decision-making of each of the multiple subjects as a proposal 113 based on subject data 104, which is the observation of the multiple subjects input from each of the multiple subjects, and an evaluation function corresponding to each of the target overall indicator and the target individual indicator, and proposes the proposal 113 to each of the multiple subjects. Therefore, the decision-making of the multiple subjects that takes into account both the overall indicator and the individual indicator can be calculated and presented to each subject as the proposal 113.
[0086] (2) The multi-subject health shutdown system 100 includes an environmental model 1202 that calculates subject data 104 at the next time based on proposal 113, which is the decision-making of multiple subjects at the current time, and subject data 104 at the current time, and a proposal basis calculation unit 1201 that calculates an evaluation function, subject data 104 at the current time, and environmental model 1202, and proposal basis 1237, which is the basis for the proposal, and presents proposal basis 1237 to each of the multiple subjects. Therefore, each subject can view proposal basis 1237 and accept the proposal 113 after becoming convinced.
[0087] (3) The multi-subject health suspension system 100 includes an application calculation unit 1301 that calculates the impact degree 1337 when an adjustment request from a target subject, which is one of the subjects, is applied to each decision of the multiple subjects calculated by the decision-making calculation unit 111, and presents the impact degree 1337 to the target subject. Therefore, each subject can know the extent to which its own decision-making adjustment request 106 will affect other indicators, and can judge whether or not to accept the proposal 113.
[0088] (4) The decision-making calculation unit 111 calculates a decision-making process using the definition of fairness determined among multiple entities as a constraint. Therefore, a fairness restriction is imposed on the proposal 113, and extreme unfairness between entities can be prevented.
[0089] (5) The decision-making calculation unit 111 unifies the measures for calculating the loss function for each target overall index and target individual index, while providing parameters obtained by reinforcement learning that divides the Q function, and calculates the decision-making using the parameters.
[0090] (6) The decision-making calculation unit 111 includes an agent model trained by reinforcement learning, and calculates the decision-making of each subject based on the subject data 104 to simultaneously improve the overall target index and the individual target index.
[0091] (7) The multi-subject health shutdown system 100 includes an environmental model 1202 that calculates subject data 104 at the next time based on the decision-making of multiple subjects at the current time and the subject data 104 at the current time, and a proposal basis calculation unit 1201 that calculates decision-making at a future time and the subject data 104 at a future time by performing a rollout based on the agent model, the environmental model 1202, the subject data 104 at the current time, and the decision-making calculated by the decision-making calculation unit 111, and calculates a time-series total value of at least one of the target overall indicators and the target individual indicators using a Q function.
[0092] (8) An application calculation unit 1301 is provided which calculates the impact 1337 when an adjustment request from a target subject, which is one of the multiple subjects, is applied to each decision of the multiple subjects calculated by the decision calculation unit 111, and presents the impact 1337 to the target subject.
[0093] (9) The apparatus includes an environment model 1302 that calculates the subject data 104 at the next time based on the decision-making of the multiple subjects at the current time and the subject data 104 at the current time. The application calculation unit 1301 executes a rollout based on the decision-making, agent model, environment model 1302, and subject data 104 at the current time to which an adjustment request from a target subject, which is one subject included in the multiple subjects, is applied to each decision-making of the multiple subjects at the current time calculated by the decision-making calculation unit 111, and calculates the impact on the time-series total value of at least one index among the target overall index and the target individual index at a future time and presents it to the subject.
[0094] (10) The agent model includes a time series model and extracts features from the subject data 104. Therefore, even if a part of the subject data 104 is concealed, the proposal 113 and the like can be calculated in the same way as when the subject data 104 is not concealed.
[0095] (11) The agent model is a single-agent model, in which each decision of multiple agents is calculated using a single policy model.
[0096] (12) The agent model is a multi-agent model, and a different policy model is used for each decision made by multiple agents.
[0097] (Variation 1) In the above-described embodiment, the multi-master health shutdown system 100 includes the decision making proposal system 110, the proposal basis generating system 1200, and the adjustment request applying system 1300. However, the multi-master health shutdown system 100 only needs to include at least the decision making proposal system 110, and may not include at least one of the proposal basis generating system 1200 and the adjustment request applying system 1300.
[0098] 11 is a configuration diagram of a multi-master health shutdown system 100A in Modification 1. The multi-master health shutdown system 100A includes a decision making proposal system 110, but does not include a proposal basis generating system 1200 or an adjustment request application system 1300. The multi-master health shutdown system 100A can provide a proposal 113 with a simple configuration.
[0099] In each of the above-mentioned embodiments and modifications, the configuration of the functional blocks is merely an example. Some functional configurations shown as separate functional blocks may be integrated, or a configuration shown in one functional block diagram may be divided into two or more functions. In addition, some of the functions of each functional block may be provided by other functional blocks.
[0100] In the above-mentioned embodiments and modifications, the program is stored in a ROM (not shown) in the controller 311, but the program may be stored in a non-volatile storage device. The controller 311 may also have an input / output interface (not shown), and the program may be read from another device via the input / output interface and a medium available to the controller 311 when necessary. The medium here refers to, for example, a storage medium that is detachable from the input / output interface, or a communication medium, that is, a network such as a wired, wireless, or optical network, or a carrier wave or digital signal that propagates through the network. Some or all of the functions realized by the program may be realized by a hardware circuit or an FPGA.
[0101] The above-mentioned embodiments and modifications may be combined with each other. Although various embodiments and modifications have been described above, the present invention is not limited to these. Other aspects that are conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention. [Explanation of symbols]
[0102] 100: Multi-master outage system 104: Subject data 105: Additional presentation request 106: Decision-making coordination request 110: Decision-making proposal system 112: Environmental Model 113: Proposal 115: Environmental conditions 116: Subject Observation 1200: Proposal basis generation system 1237: Basis for proposal 1300: Adjustment request application system 1337: Impact
Claims
1. A multi-subject arbitration system that arbitrates decision-making among multiple subjects that affects at least one overall indicator, a target overall indicator, and at least one individual indicator, a target individual indicator, A multi-subject health shutdown system comprising a decision-making calculation unit that calculates a decision for each of the multiple subjects based on subject data, which is observations of the multiple subjects input from each of the multiple subjects, and evaluation functions corresponding to each of the target overall indicators and the target individual indicators, and proposes the decision to each of the multiple subjects.
2. 2. The multi-maintenance shutdown system according to claim 1, an environment model for calculating the subject data at a next time based on the decision-making of the multiple subjects at a current time and the subject data at the current time; A multi-subject health shutdown system further comprising a proposal basis calculation unit that calculates the evaluation function, the subject data at the current time, and the environmental model, and a proposal basis that is the basis for the proposal, and presents the proposal basis to each of the multiple subjects.
3. 2. The multi-maintenance shutdown system according to claim 1, A multi-subject health shutdown system further comprising an application calculation unit that calculates the impact of applying an adjustment request from a target subject, which is one of the multiple subjects, to each decision of the multiple subjects calculated by the decision calculation unit, and presents the impact to the target subject.
4. 2. The multi-maintenance shutdown system according to claim 1, The decision-making calculation unit calculates the decision-making using a definition of fairness determined among the multiple entities as a constraint condition.
5. 2. The multi-maintenance shutdown system according to claim 1, The decision-making calculation unit is provided with parameters obtained by reinforcement learning that divides a Q function while unifying the measures at the time of calculating the loss function for each of the target overall index and the target individual index, and calculates the decision-making using the parameters. A multi-subject health shutdown system.
6. 6. A multi-maintenance shutdown system according to claim 5, The decision-making calculation unit is equipped with an agent model learned by the reinforcement learning, and calculates the decision-making of each subject that simultaneously improves the target overall indicator and the target individual indicator based on the subject data.
7. 7. A multi-maintenance shutdown system according to claim 6, an environment model for calculating the subject data at a next time based on the decision-making of the multiple subjects at a current time and the subject data at the current time; a proposal basis calculation unit that calculates the decision-making at a future time and the subject data at the future time by executing a rollout based on the agent model, the environment model, the subject data at the current time, and the decision-making calculated by the decision-making calculation unit, and calculates a time-series total value of at least one of the target overall indicators and the target individual indicators using the Q function.
8. 7. A multi-maintenance shutdown system according to claim 6, A multi-subject health shutdown system further comprising an application calculation unit that calculates the impact of applying an adjustment request from a target subject, which is one of the multiple subjects, to each decision of the multiple subjects calculated by the decision calculation unit, and presents the impact to the target subject.
9. 9. A multi-maintenance shutdown system according to claim 8, an environment model for calculating the subject data at a next time based on the decision-making of the multiple subjects at a current time and the subject data at the current time; A multi-subject health shutdown system in which an application calculation unit executes a rollout based on the decision-making, the agent model, the environmental model, and the subject data at the current time to which an adjustment request from a target subject, which is one of the multiple subjects, is applied to the decision-making of each of the multiple subjects at the current time calculated by the decision-making calculation unit, thereby calculating the impact on a time-series total value of at least one of the target overall indicators and the target individual indicators at a future time and presenting the impact to the subjects.
10. 7. A multi-maintenance shutdown system according to claim 6, The agent model includes a time series model, and extracts features from the subject data.
11. 7. A multi-maintenance shutdown system according to claim 6, A multi-subject health shutdown system, in which the agent model is a single-agent model, and the decision-making of each of the multiple subjects is calculated using one policy model.
12. 7. A multi-maintenance shutdown system according to claim 6, The agent model is a multi-agent model, and a multi-subject health shutdown system is implemented in which a different policy model is used for each decision of the multiple subjects.
13. A multi-subject mediation method implemented by a computer for mediating decisions of multiple subjects affecting at least one overall indicator, a target overall indicator, and at least one individual indicator, a target individual indicator, comprising: acquiring subject data from each of the plurality of subjects, the subject data being observations of the plurality of subjects; calculating a decision for each of the multiple entities based on an evaluation function corresponding to each of the target overall indicators and the target individual indicators, and proposing the decision to each of the multiple entities.
Citation Information
Patent Citations
Game machine system
JP2023006347A
Cited By
Information processing systems, information processing methods, and programs
JP7950081B1
