Measure management method and measure management system

Abstract

To make it possible to prepare measures which can cope with various tasks represented by various pieces of environment information and KPI in reinforcement learning.SOLUTION: A measure management system for assigning a measure for determining an action in causing a task to be executed by an agent in reinforcement learning is configured to: calculate weight information for a KPI on the basis of environment information on environment and KPI weight calculation parameters for calculating a weight of the KPI; define the task on the basis of the environment information, KPI, and weight information and collect an environment status with an evaluation measure applied to the task, an action of the agent, and a reward for the action to generate history data; project the task on a task inclination space using the environment information, weight information, and history data; calculate an adaptive value for evaluating adaptability of a measure with the measure assigned to the task; and divide the task inclination space into a measure application region to which the measure is applied on the basis of the adaptive value.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to a method and a system for managing a policy. [Background technology]

[0002] Reinforcement learning is known as an effective method for control, decision support, planning, etc., and is based on the interaction between an agent and the environment. The agent determines the action based on a policy for the state obtained from the environment. The environment receives the action from the agent and returns the next state and reward for the action to the agent.

[0003] Here, the optimal action for a certain state can be determined by a task expressed by information about the environment (hereinafter referred to as environmental information) and an evaluation index (Key Performance Indicator, hereinafter referred to as KPI) that determines a reward. When applying reinforcement learning, a policy that outputs an action for a state is learned using a reinforcement learning model (hereinafter referred to as a policy) for the task during advance preparation, and the learned policy is applied to the task when actually applied. When applying reinforcement learning in practice, it is necessary to handle a variety of tasks because environmental information and KPIs become diverse.

[0004] For example, in Patent Document 1, policies are learned for a task during advance preparation, and the learned policies are assumed to be used when actually applied. Various tasks can be handled by changing the policy to be applied from N learned policies (N is an integer equal to or greater than 2) depending on the task.

[0005] In addition, in Non-Patent Document 1, meta-reinforcement learning is used to learn a single robust policy for multiple tasks during advance preparation, and the meta-reinforcement learned policy is used during actual operation, allowing a single policy to handle a variety of tasks. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent Publication No. 2021-89446 [Non-patent literature]

[0007] [Non-Patent Document 1] B. Mehta etal., “Curriculum in Gradient-Based Meta-ReinforcementLearning”, ICLR 2020 workshop: Beyond tabula rasa in RL (BeTR-RL). Summary of the Invention [Problem to be solved by the invention]

[0008] In the above-mentioned Patent Document 1, at least one measure is prepared for each of N different tasks during advance preparation. If the environmental information or KPI differs, it will result in a different task, so during actual operation, it is necessary to consider countless tasks that are not present during advance preparation. In addition, it is necessary to appropriately determine the aforementioned N so that an appropriate measure can be assigned to each task, but there is a possibility that N will become infinite.

[0009] Furthermore, the above-mentioned non-patent document 1 merely prepares one measure for multiple tasks, and as described in non-patent document 1, it is not possible to deal with the countless tasks that may arise in actual applications.

[0010] In order to apply reinforcement learning to multiple tasks with different environmental information and KPIs, it is necessary to prepare as many policies as there are tasks, because a single policy cannot handle all tasks. However, preparing as many policies as there are tasks is difficult in reality.

[0011] The present invention has been made in consideration of the above, and aims to make it possible to prepare measures in reinforcement learning that can handle a variety of tasks expressed by diverse environmental information and KPIs. [Means for solving the problem]

[0012] As one aspect of solving the above problem, a policy management method is implemented by a policy management system that assigns policies to tasks in order to determine actions taken by an agent when the agent executes a task in an environment in reinforcement learning, the method comprising: a task setting step of acquiring a KPI weight calculation parameter for calculating a weight of a KPI (Key Performance Indicator) and calculating weight information for the KPI based on the environmental information and the KPI weight calculation parameter; a history data generation step of defining the task based on the environmental information, the KPI, and the weight information, and generating history data by acquiring a state of the environment, the action of the agent, and a reward for the action when a policy for evaluation is applied to the task; a dimensionality reduction step of projecting the task onto the task tendency space, which is a subspace of a lower dimension than a task space constituted by the environmental information and the KPI, using the environmental information, the weight information, and the history data, and predetermined parameters for projecting the history data onto a task tendency space, which is a subspace of a lower dimension than a task space constituted by the environmental information and the KPI, based on features of the task; an adaptation value calculation step of calculating an adaptation value for evaluating the suitability of the policy when the policy is assigned to the task; and a policy domain division step of dividing the task tendency space into policy application domains to which the policies are applied, based on the adaptation value. Effect of the Invention

[0013] According to the present invention, it is possible to prepare a policy capable of dealing with various tasks expressed by diverse environmental information and KPIs in reinforcement learning. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram showing the configuration of a policy management system according to an embodiment. [Diagram 2] FIG. 4 is a diagram showing a parameter group according to the embodiment. [Figure 3A] FIG. 4 is a diagram showing an example of the configuration of environmental information according to the embodiment. [Figure 3B]FIG. 4 is a diagram showing an example of the configuration of KPI weight information according to the embodiment. [Figure 3C] FIG. 4 is a diagram showing an example of the configuration of history data according to the embodiment. [Figure 3D] FIG. 2 is a diagram showing an example of the configuration of a policy set according to the embodiment. [Figure 3E] FIG. 4 is a diagram showing an example of the configuration of a policy application domain parameter according to the embodiment. [Figure 4A] FIG. 4 is a diagram showing a detailed example of an adaptive threshold value according to the embodiment. [Figure 4B] 5A and 5B are diagrams showing detailed examples of simulation control information and actual application control information according to the embodiment. [Figure 4C] FIG. 4 is a diagram showing a detailed example of an adaptive value set according to the embodiment. [Figure 4D] FIG. 4 is a diagram showing a detailed example of projection information according to the embodiment. [Figure 4E] FIG. 4 is a diagram showing a detailed example of history data generation parameters according to the embodiment. [Figure 4F] FIG. 4 is a diagram showing a detailed example of display axis information according to the embodiment. [Diagram 5] FIG. 13 is a diagram showing an area graphic according to the embodiment. [Figure 6] 1 is a flowchart showing a measure management process according to an embodiment. [Figure 7] 4 is a flowchart showing a task setting process according to the embodiment. [Figure 8] 5 is a flowchart showing a history data generation process according to the embodiment. [Figure 9] 11 is a flowchart showing a dimensional compression process according to the embodiment. [Figure 10] 11 is a flowchart showing a measure allocation process according to the embodiment. [Figure 11] 4 is a flowchart showing a simulation process according to the embodiment. [Figure 12] 11 is a flowchart showing an actual application process according to the embodiment. [Figure 13] 5 is a flowchart showing an adaptive value calculation process according to the embodiment. [Figure 14] 11 is a flowchart showing a task definition update process according to the embodiment. [Figure 15] 11 is a flowchart showing a strategy area division process according to the embodiment. [Figure 16] 11 is a flowchart showing a task tendency space update process according to the embodiment. [Figure 17] 11 is a flowchart showing a visualization process according to the embodiment. [Figure 18] 6A to 6C are views for explaining a task tendency space generation process / a strategy area division process according to the embodiment. [Figure 19] FIG. 13 is a diagram for explaining task tendency space regeneration according to the embodiment. [Figure 20] FIG. 13 is a diagram for explaining visualization of a task tendency space according to the embodiment. [Figure 21] FIG. 2 is a diagram for explaining a train rescheduling operation according to the first embodiment. [Figure 22] FIG. 2 is a diagram for explaining a train rescheduling operation according to the first embodiment. [Diagram 23] FIG. 11 is a diagram for explaining an automatic cleaning operation performed by a robot according to a second embodiment. [Figure 24] FIG. 11 is a diagram for explaining an automatic cleaning operation performed by a robot according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In this embodiment, the same components are generally designated by the same reference numerals, and repeated explanations will be omitted. Note that this embodiment is merely an example for realizing the present invention, and does not limit the technical scope of the present invention.

[0016] In the following explanation, the processing performed by a program may be described. A computer performs processing defined by a program using a processor (e.g., a CPU (Central Processing Unit), a GPU (Graphics Processing Unit)) while using memory in a main storage device, etc. Therefore, the processor may be the entity that performs processing by executing a program. The processor executes a program to realize a functional unit that performs processing.

[0017] Similarly, the subject of the processing performed by executing the program may be a controller, device, system, computer, or node having a processor. The subject of the processing performed by executing the program may be a computing unit, and may include a dedicated circuit that performs specific processing. The dedicated circuit is, for example, an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0018] In the following description, a program may be installed in a computer from a program source. The program source may be, for example, a program distribution server or a non-transitory storage medium readable by a computer. When the program source is a program distribution server, the program distribution server may include a processor and a storage resource (storage) for storing a program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. In addition, in an embodiment, two or more programs may be realized as one program, and one program may be realized as two or more programs.

[0019] [Embodiment] FIG. 1 is a diagram showing a configuration of a policy management system 100 according to an embodiment. The policy management system 100 assigns a policy to a task for determining an action to be taken when an agent executes a task in an environment in reinforcement learning. The policy management system 100 divides a task tendency space generated by dimensionality reduction while considering a range to which the policy can be applied. A task is determined by environmental information and KPIs and is the target of optimization. The task tendency space is a space onto which tasks with similar policies for the task are projected, among the tasks determined by environmental information and KPIs.

[0020] The strategy management system 100 comprises a processor 601 , a data storage device 602 , a communication device 603 , an input device 604 , an output device 605 , and a program storage device 606 .

[0021] The policy management system 100 can access the database 102. It is assumed that the database 102 is stored in the data storage device 602. The database 102 is not limited to this, and may be stored in another device within the policy management system 100 or an external computer capable of communicating with the policy management system 100.

[0022] The database 102 stores a parameter group 2000 shown in Fig. 2. It also stores sensor information obtained by a simulator that executes reinforcement learning and an external computer, and information calculated by each part of the policy management system 100.

[0023] The processor 601 is a processor such as a CPU (Central Processing Unit), and controls the policy management process.

[0024] The data storage device 602 serves as a working area for the processor 601. The data storage device 602 is a non-temporary or temporary recording medium for storing various programs and data. Examples of the data storage device 602 include a read only memory (ROM), a random access memory (RAM), a hard disk drive (HDD), a solid state drive (SSD), and a flash memory.

[0025] The communication device 603 is connected to a network and transmits and receives data. The input device 604 is a device for inputting data and information to the system. Examples of the input device 604 include a keyboard, a mouse, a touch panel, a numeric keypad, a scanner, and a microphone.

[0026] The output device 605 is a device that outputs data and information held by the system. Examples of the output device 605 include a display, a printer, and a speaker.

[0027] The program storage device 606 stores the task tendency space projection unit 103, the strategy allocation unit 113, the simulation unit 114, the actual application unit 115, the adaptation value calculation unit 116, the task tendency space adjustment unit 104, the visualization unit 120, and the data input / output unit 101.

[0028] The task tendency space projection unit 103, the policy assignment unit 113, ..., and the data input / output unit 101 are realized by making the processor 601 execute a program stored in the data storage device 602. The program is stored in the storage device of the policy management system 100 or an external computer.

[0029] The data input / output unit 101 accepts data input from an input device such as a keyboard, or an external computer capable of communicating with the policy management system 100. The data input / output unit 101 displays data calculated within the policy management system 100 by an output device such as a display.

[0030] The task tendency space projection unit 103 includes a task setting unit 110 , a history data generation unit 111 , and a dimensionality reduction unit 112 .

[0031] The task setting unit 110 has a KPI weight calculation parameter 205. The task setting unit 110 acquires environment information 200 and a KPI set 201, and stores in KPI weight information 202 the importance of each KPI in the KPI set 201 calculated by a model using the KPI weight calculation parameter 205.

[0032] Specifically, the environment information 200 represents, in the form of parameters, the elements of the environment that define the task. The KPI set 201 is a collection of indices for evaluating actions for the task. The KPI weight 202 represents the importance of each KPI in the KPI set. The KPI set 201 and the KPI weight 202 are used when determining a reward function in reinforcement learning. The reward function r is a weighted sum expressed, for example, as shown in Equation (1), using the weights stored in the KPI weight 202 for each element in the KPI set 201. r=w_1*KPI_1+w_2*KPI_2+… ···(1)

[0033] Here, r on the left side of formula (1) is the reward obtained, and w_1 and w_2 are the weights of the reward for the first KPI_1 and the second KPI_2, respectively. Note that when there are three or more KPIs, the KPIs are similarly multiplied by the weight w and added together. The method of determining the reward function is merely an example and is not limited to this.

[0034] The history data generator 111 acquires the environment information 200, the KPI set 201, the KPI weight 202, and the evaluation policy 203, and determines the parameters using a simulator. The history data generator 111 stores the action history (a_t, a_(t+1), ...) selected by the evaluation policy 203, the state history (s_t, s_(t+1), ...) transitioned according to the action, and the reward history (r_t, r_(t+1), ...) according to the KPI obtained by the action as history data 214 in the database 102. Here, a means the action, s means the state, r means the reward, and t means the step during the simulation. The step is expressed by a discrete value, and as the state transition progresses, the next step is reached. For example, a_t means the action at step t.

[0035] The dimension compression unit 112 acquires the environment information 200, the KPI weight information 202 stored by the task setting unit 110, and the history data 214 stored by the history data generation unit 111. Then, the dimension compression unit 112 projects the multidimensional task space configured by the environment information 200 and the KPI weight information 202 onto a low-dimensional task tendency space. When projecting, the dimension compression unit 112 uses a dimension compressor having parameters (hereinafter, task tendency space generation parameters 212) that can be projected onto a low-dimensional space using the history data 214. The dimension compression unit 112 stores the coordinates of the projection destination onto the task tendency space in the projection information 215 of the database 102.

[0036] The dimension reduction method used by the dimension reduction unit 112 includes, for example, principal component analysis, autoencoder, t-SNE, etc. The dimension reduction in this embodiment does not necessarily require reversible conversion before and after the dimension reduction.

[0037] The policy allocation unit 113 acquires the projection information 215 stored in the database 102 by the dimensionality reduction unit 112, and stores the corresponding policy as an applied policy 213 in the database 102 according to the policy application region in the task tendency space. Hereinafter, the policy application region in the task tendency space is referred to as a policy application region parameter 206.

[0038] The simulation unit 114 obtains the applied policy 213 stored in the policy assignment unit 113, and executes a simulation for the task using the applied policy 213. The simulation unit 114 stores the state, action, and reward obtained during the simulation in the database 102 as simulation control information 209.

[0039] The actual application unit 115 acquires the application policy 213 stored in the policy allocation unit 113, and stores in the database 102 the state, action, and reward obtained when the application policy 213 is actually applied to a person or a system as actual application control information 210.

[0040] The adaptation value calculation unit 116 acquires simulation control information 209 stored in the simulation unit 114 and actual application control information 210 stored in the actual application unit 115. Then, the adaptation value calculation unit 116 stores an adaptation value calculated based on the acquired reward of reinforcement learning, the success probability of a task, etc., in an adaptation value set 211 of the database 102.

[0041] Examples of the adaptation value include the acquired reward and the task success rate, but the indicator is not limited as long as it can evaluate the policy. For example, the adaptation value may be a weighted sum of the acquired reward and the task success rate, or may be calculated from the change in the acquired reward when reinforcement learning is performed for only a few episodes.

[0042] The task definition update unit 117 acquires the KPI weight calculation parameter 205, simulation control information 209 stored in the simulation unit, actual application control information 210 stored in the actual application unit 115, and adaptation value set 211 stored by the adaptation value calculation unit 116. Then, the task definition update unit 117 acquires the adaptation value stored in the simulation unit 114 and the adaptation value stored in the actual application unit 115 from the adaptation value set 211, and compares the two adaptation values. The task definition update unit 117 determines whether to adjust the KPI weight calculation parameter 205 depending on the magnitude relationship.

[0043] The policy domain division unit 118 acquires a policy set 204 and an adaptive threshold 207. The policy domain division unit 118 acquires an adaptive value of each policy in the policy set 204 from an adaptive value set 211, and updates a policy application domain parameter 206 according to the magnitude relationship between the adaptive value and the adaptive threshold 207.

[0044] The task tendency space update unit 119 acquires the policy application area parameter 206 and the projection information 215. The task tendency space update unit 119 compares the policy application area parameter 206, the projection information 215, and the area regeneration threshold 208 to determine whether or not to regenerate the task tendency space.

[0045] The visualization unit 120 acquires the measure application domain parameters 206, the environment information 200, the KPI set 201, the KPI weights 202, and the display axis information 217, and visualizes the task space by limiting the axes in two or three dimensions.

[0046] (Parameter set 2000 according to the embodiment) 2 is a diagram showing a parameter group 2000 according to the embodiment. The parameter group 2000 is stored in the database 102, for example. The parameter group 2000 includes environment information 200, a KPI set 201, a KPI weight 202, an evaluation measure 203, a measure set 204, a KPI weight calculation parameter 205, a measure application area parameter 206, an adaptation threshold 207, and an area regeneration threshold 208. The parameter group 2000 also includes simulation control information 209, actual application control information 210, an adaptation value set 211, a task tendency space generation parameter 212, an application measure 213, history data 214, projection information 215, and a history data generation parameter 216. The parameter group 2000 also includes display axis information 217.

[0047] The environment information 200 includes parameters related to the environment that define the task. For example, when using reinforcement learning for a task of train traffic rescheduling, which eliminates delay time by rescheduling trains when a delay occurs, the environment information 200 defines a disruption pattern. Specifically, the environment information includes a disruption start time indicating when a disruption will occur, a disruption duration indicating how long the disruption will continue, a disruption section indicating the size of the disruption range, a disruption type indicating a service stop or slowdown, and a disruption direction indicating a disruption in only upbound / downbound directions or in both directions.

[0048] The KPI set 201 includes indicators (KPIs) for evaluating tasks. Tasks change depending on how each KPI is weighted. For example, in the case of a train rescheduling task, KPIs include a delay recovery time, a train rescheduling plan that does not violate constraints, and customer satisfaction. A collection of KPIs corresponds to the KPI set 201.

[0049] The KPI weights 202 include parameters that define weights of real values ​​for all KPIs included in the KPI set 201. The KPI weights 202 may be estimated from the environmental information 200, or may be set by a person via the data input / output unit 101.

[0050] The evaluation policy 203 includes a policy parameter that determines an action for a state. The evaluation policy 203 may change an action depending on a state, or may use a set of actions prepared in advance in sequence regardless of the state. The evaluation policy 203 may be determined by a random number, or may be input via the data input / output unit 101 such as teacher data. A plurality of evaluation policies 203 may be prepared.

[0051] The policy set 204 includes learned policy models. When a policy application region is divided in the task tendency space, the policy to be assigned to that region is added to the policy set 204. The policy set 204 also includes a general policy to be applied when a task that does not fall into any region is received. The general policy may be a policy learned using a task used in advance preparation, or may be a heuristic model including human experience and know-how.

[0052] The KPI weight calculation parameters 205 include parameters for calculating the KPI weights 202 for the environmental information 200. The KPI weight calculation parameters 205 may prepare a model for calculating KPI weights for the number of KPIs and store each independently, or may be a single model whose output corresponds to the number of KPIs.

[0053] The strategy application area parameters 206 include an ID for identifying the area, an ID of the strategy corresponding to the area, the center coordinates within the area, an area division ID used for the area division, area graphic parameters corresponding to the area division ID, and an adaptive threshold 207 used to determine the area division.

[0054] The adaptive threshold 207 includes an adaptive threshold value that determines whether or not to perform region division. The adaptive threshold 207 may be a fixed real value, or may be a variable real value that changes depending on the measure application region parameter 206, etc. A plurality of fixed real values ​​may be prepared and switched depending on the situation.

[0055] The region regeneration threshold 208 includes a threshold for determining whether or not to regenerate the task tendency space. The region regeneration threshold 208 may be a fixed real number, or may be a variable real number that changes depending on the situation. For example, when the number of outlying areas in the policy application region exceeds the region regeneration threshold, the task tendency space is regenerated.

[0056] The simulation control information 209 includes each state, action, and reward calculated in the simulation using the applied policy 213. The simulation control information 209 includes information corresponding to the task, and the information may be a real value or a flag indicated by True or False.

[0057] The actual application control information 210 includes each state, action, and reward obtained when actually applying using the application policy 213. The actual application control information 210 includes information according to the task, and the information may be a real value or a flag indicated by True or False.

[0058] The adaptation value set 211 includes an evaluation value calculated based on the simulation control information 209 and the actual application control information 210. The adaptation value may be an index used to evaluate reinforcement learning, such as an acquired reward or a success probability of a task. The adaptation value may also be calculated from two or more real values, such as an acquired reward or a success probability. The adaptation value may be a negative value.

[0059] The task tendency space generation parameters 212 include parameters of a dimension reduction model used when generating the task tendency space. The dimension reduction may be a lossy transformation. For example, when an autoencoder is used as the dimension reduction model, the neural network inside the autoencoder is stored as the task tendency space generation parameters.

[0060] The applied policy 213 includes one policy from the policy set 204 stored in the policy assignment unit 113 .

[0061] The historical data 214 includes states, actions, and rewards when the evaluation strategy 203 is used for a given task.

[0062] The historical data 214 may include environmental information 200 and KPI weights 202, as well as other information obtained through reinforcement learning.

[0063] It is not necessary to include all of one episode (n steps from the beginning to the end of a task) in reinforcement learning. For example, it is possible to use state-action-reward histories for any number of steps, or state-action-reward histories for every few steps.

[0064] The number of steps is obtained from the history data generation parameters 216 .

[0065] The projection information 215 includes the coordinates of the tasks projected in the dimensionality reduction unit 112 in the task tendency space.

[0066] When one task has multiple projection targets due to multiple evaluation policies 203 or divided history data 214, the center of gravity (centroid) of the coordinates may be taken, or a representative point (medoid) may be taken.

[0067] The history data generation parameters 216 include setting information related to the generation of history data, such as the number of steps, when the history data 214 is generated. The setting information is stored via a setting file or the data input / output unit 101.

[0068] The display axis information 217 includes information for limiting the axes to two or three dimensions in order to visualize a high-dimensional task space, which is used in the visualization unit 120. Specifically, an n-dimensional task space is a space having n axes, and includes criteria for selecting two or three of the axes.

[0069] The display axis information 217 may be set not only to the axes of the task space, but also to new axes that are easily understandable to humans.

[0070] 3A to 3E and 4A to 4F are diagrams showing detailed examples of parameters according to the embodiment.

[0071] As shown in Fig. 3A, the environment information 200 includes a task ID and each variable of the environment information (environment information 1, environment information 2, environment information 3, ...). The task ID is identification information that specifies a task. The environment information 1, environment information 2, environment information 3, ... represent environment information. In the environment information 200, the number of pieces of environment information is not limited to three, that is, the environment information 1 to 3.

[0072] Each variable of the environment information in the environment information 200 may be a discrete categorical variable or a continuous variable. It may also be in the form of a flag expressed as True or False. When there are tasks having the exact same environment information 200, the task IDs may be combined into one, or may be assigned as separate task IDs.

[0073] As shown in FIG. 3B, the KPI weight information 202 includes a task ID and a weight for each KPI included in the KPI set 201. The task ID is identification information that specifies a task. KPI1, KPI2, KPI3, and KPI4 represent each KPI included in the KPI set 201, and the weight for each KPI is stored for each task ID. The number of KPIs in the KPI weight information 202 is not limited to four, KPI1 to 4. The KPI weight information 202 differs depending on the task or the environment information 200, and the KPI weight information 202 is calculated based on a KPI weight calculation parameter 205. The KPI weight information 202 is used when setting a reward function for reinforcement learning. Each value of the KPI weight information 202 may be a positive real value or a negative real value. A weight of 0.0 means that the KPI is not used.

[0074] For example, the reward function is expressed as a weighted linear sum using the KPI weight information 202. In this case, for a task with a task ID of 0, the reward can be defined as shown in equation (2). r=0.14*KPI1+0.0*KPI2+1.0+1.0*KPI3+0.23*KPI4+… ···(2)

[0075] In formula (2), the value of each KPI is calculated based on the state of the task, the action for the task, etc. In this example, the reward function is expressed as a weighted linear sum, but this is not limited to this.

[0076] 3C, the history data 214 includes a task ID, an evaluation policy ID, a state, an action, a reward, etc. The evaluation policy ID is a number that uniquely identifies a policy included in the evaluation policy 203.

[0077] In the history data 214, state 1_1 represents the first element of the state of the first step included in the history data 214. Similarly, state 1_2 represents the second element of the state of the first step included in the history data 214. Action 1 represents the action when state 1 in the first step is given. Reward 1 represents the reward when action 1 in the first step is executed for state 1. State 2_1 represents the first element of the state of the second step included in the history data 214. Thereafter, similar storage is performed for the n states included in the history data 214.

[0078] Note that the column names and storage method of the history data 214 are merely examples, and each column may contain information for all steps included in the history data 214. For example, the state column may store information in the form of a vector, such as (state 1 in the first step, state 2 in the second step, ...).

[0079] The states in the history data 214 may be continuous real values ​​or discrete values, or may be in the form of a flag expressed as True or False. Similarly, the actions and rewards may be continuous real values ​​or discrete values.

[0080] As shown in Fig. 3D, the policy set 204 includes a policy ID, a task ID, and flag information indicating whether the corresponding policy is a general policy. The columns of Task ID1, Task ID2, etc. indicate which tasks were used to learn the policy during advance preparation. If the number of tasks used for learning is small compared to the number of columns, "-", "None", "Null", etc. are stored to indicate that there is no corresponding task and the column is left blank.

[0081] Note that during preparation, more than one task may be used to learn the policy. In that case, additional task ID columns are added, such as a column for task ID 2, a column for task ID 3, and so on.

[0082] The general policy column contains a flag indicating whether the policy has been assigned as a general policy.

[0083] If the policy does not fall into any policy application domain or if there is no policy application domain, a general policy learned with the task used in advance preparation is stored as the applied policy 213. As a method for creating a general policy, for example, if N tasks are used to generate the history data 214, reinforcement learning may be performed for all the N tasks to adjust one policy, or a part of the tasks may be used for learning.

[0084] The strategy application region parameters 206 include a region ID, a strategy ID, region center coordinates, a region division ID, region graphic parameters, and an adaptive threshold 207, as shown in FIG. 3E.

[0085] The domain ID is identification information for specifying the policy application domain. The policy ID is used to determine which policy is to be used within the policy application domain.

[0086] The region center coordinates determine the coordinates that are the center when determining the policy application region. For example, when projected onto a two-dimensional space by the dimension reduction unit 112, the region center coordinate 1 includes the x coordinate, and the region center coordinate 2 includes the y coordinate. The x coordinate and the y coordinate may be written together in vector notation as the column of center coordinates.

[0087] The region division ID is identification information for identifying a figure or a method used when dividing a region. FIG. 5 is a diagram showing a region figure according to the embodiment. As shown in FIG. 5, for example, region division ID=0 indicates a circular region figure 501. Region division ID=1 indicates a rectangular region figure 501. Region division ID=2 indicates a hexagonal region figure 503. Region division ID=4 indicates a pentagonal region figure 504. The region figure does not have to be a convex figure like region figure 504. In addition, the region division ID may determine a region division method, such as a region that satisfies the adaptive threshold value 207 or a region obtained by Voronoi division.

[0088] The area graphic parameters include the parameters of the area graphic indicated by the area division ID. For example, when a circular area graphic 501 with area division ID=0 is used, the radius is stored in area graphic parameter 1. Since the radius is the only parameter that characterizes a circle, "-" is stored in area graphic parameter 2 and onward. When a rectangular area graphic 502 with area division ID=1 is used, the coordinates of each vertex are stored in area graphic parameter 1, and the length of each side is stored in area graphic parameter 2.

[0089] The column of adaptive threshold stores the adaptive threshold 207 used when performing region division.

[0090] As shown in FIG. 4A, the adaptive threshold 207 stores an adaptive threshold and a condition. The adaptive threshold varies depending on the condition. Whether or not a condition is satisfied is determined from simulation control information and actual application control information including states, actions, and the like. For example, a condition is set in the columns of condition 1 and condition 2 shown in FIG. 4A, and True is stored if the condition is satisfied, and False is stored if the condition is not satisfied. For example, a condition is "Are there five or more policy application regions in the task tendency space?". Although two conditions are illustrated in the adaptive threshold 207, the number of conditions is not limited to two.

[0091] The conditions may be predetermined based on domain knowledge, or may be variable. Also, the adaptive threshold may be variable, instead of being managed in a table format like adaptive threshold 207.

[0092] 4B, the simulation control information 209 includes a task ID, a state, an action, and a reward, similar to the history data 214. Note that the simulation control information 209 and the actual application control information 210 have the same configuration.

[0093] Image information may be stored as the state in the simulation control information 209. In addition, for problem settings in which the order of actions does not have a meaning as control information and only the final state and reward are important, only the final state and reward may be stored as control information without including the transition of states and rewards due to actions.

[0094] Simulation control information 209 and actual application control information 210 refer to vectors that store only information for one task under consideration during operation. History and control information for other tasks are not stored here.

[0095] 4C, the adaptation value set 211 stores a measure ID, a simulation control information flag, and an adaptation value. The simulation control information flag stores True when the adaptation value is calculated from the simulation control information, and stores False when the adaptation value is calculated from the actual application control information. Note that the actual application control information flag may be added to the simulation control information flag for expression.

[0096] 4D, the projection information 215 includes a task ID and a projection destination coordinate. For example, when a task is compressed into two dimensions by the dimension compression unit 112, the projection destination coordinate 1 is synonymous with the x-axis, and the projection destination coordinate 2 is synonymous with the y-axis.

[0097] The history data generation parameters 216 include a start step, an end step, and an interval, as shown in Fig. 4E. The history data 214 is stored for each interval from the start step to the end step.

[0098] The start step and end step are non-negative values ​​that indicate the number counting from the first element. The start step and end step are negative values ​​that indicate the number counting from the last element. For example, if the start step is 0, the end step is -1, and the interval is 5, this indicates that every 5 steps from the first step to the last step are stored in the history data 214. If the start step is -5, the end step is -1, and the interval is 1, this indicates that every step from the 5th step counting from the last step to the last step are stored in the history data 214.

[0099] The display axis information 217 includes parameters that specify the coordinate axes of a projection space in which the visualization unit 120 projects a high-dimensional task space having the environment information 200 and the KPI weight information 202 as its axes onto a two-dimensional or three-dimensional space.

[0100] 4F, when a two-dimensional space is used, "-", which means that there is no corresponding input, is input to the coordinate axis of the dimension to be deleted among the three axes, X-axis, Y-axis, and Z-axis, for the display axis information 217. In the display axis information 217, a part of the environment information 200 or the KPI set 201 is selected as an axis, but an axis that is easy for humans to understand, such as an index calculated from domain knowledge, may be set.

[0101] (Policy management process according to the embodiment) 6 is a flowchart showing a measure management process according to the embodiment. The measure management process is executed in response to an instruction from an operator when a measure is prepared in advance or when a measure is selected.

[0102] First, in step S103, the task tendency space projection unit 103 executes a task tendency space projection process.

[0103] In step S110 in step S103, the task setting unit 110 executes a task setting process. In the task setting process, the task setting unit 110 calculates an evaluation weight for each KPI stored in the KPI set 201 from the environment information 200 and the KPI weight calculation parameter 205. The task setting process will be described in detail later with reference to FIG.

[0104] Next, in step S111 in step S103, the history data generation unit 111 executes a history data generation process. In the history data generation process, the history data generation unit 111 defines a task using the environment information 200, the KPI set 201, and the KPI weight information 202. Then, for the defined task, the history data generation unit 111 obtains state transitions, actions, and rewards using an evaluation policy, and stores them as history data 214. The history data 214 may include the environment information 200 and the KPI weight information 202, or may include other information obtained by reinforcement learning. Details of the history data generation process will be described later with reference to FIG. 8.

[0105] Next, in step S112 in step S103, the dimensionality reduction unit 112 executes a history data generation process. In the dimensionality reduction process, the dimensionality reduction unit 112 executes dimensionality reduction of the task space using the environment information 200, the KPI weight information 202, and the history data 214 obtained from the task, and projects the dimensionally reduced task space onto the task tendency space. Details of the dimensionality reduction process will be described later with reference to FIG. 9.

[0106] Next, in step S113, the measure allocation unit 113 executes a measure allocation process. The details of the measure allocation process will be described later with reference to FIG.

[0107] Next, in step S114, the simulation unit 114 executes a simulation process. The details of the simulation process will be described later with reference to FIG.

[0108] In step S115, the actual application unit 115 executes an actual application process. Details of the actual application process will be described later with reference to Fig. 12. Note that steps S114 and S115 may be executed in reverse order, or may be executed in parallel.

[0109] After steps S114 and S115 are completed, the adaptive value calculation unit 116 executes an adaptive value calculation process in step S116. The details of the adaptive value calculation process will be described later with reference to FIG.

[0110] Next, after step S116 is completed, in step S104, the task tendency space adjustment section 104 executes a task tendency space adjustment process.

[0111] In step S117 in step S104, the task definition update unit 117 executes the task definition update process when the system is in operation, and skips the execution of the task definition update process when the system is in preparation. In the task definition update process, the task definition update unit 117 acquires the simulation control information 209, the actual application control information 210, the adaptation value set 211, and the KPI weight calculation parameter 205, and adjusts the KPI weight information 202 according to the magnitude relationship of the respective adaptation values. Details of the task definition update process will be described later with reference to FIG. 14.

[0112] Next, in step S118 in step S104, the policy area dividing unit 118 executes a policy area dividing process. In the policy area dividing process, the policy area dividing unit 118 acquires the adaptive value set 211, the adaptive threshold 207, the policy application area parameter 206, and the projection information 215, and divides the policy application area in the task tendency space according to the magnitude relationship of the respective adaptive values ​​and the threshold. Details of the policy area dividing process will be described later with reference to FIG. 15.

[0113] Next, in step S119 in step S104, the task tendency space update unit 119 executes the task tendency space update process when the system is in operation, and skips the execution of the task tendency space update process when the system is in preparation. In the task tendency space update process, the task tendency space update unit 119 acquires the adaptation value set 211, the area regeneration threshold 208, the measure application area parameter 206, the task tendency space generation parameter 212, and the projection information 215. Then, the task tendency space is regenerated according to the magnitude relationship of the adaptation values ​​and the thresholds. Details of the task tendency space update process will be described later with reference to FIG. 16.

[0114] (Task setting process according to the embodiment) FIG. 7 is a flowchart showing the task setting process (step S110 (FIG. 6)) according to the embodiment.

[0115] First, in step S110a, the task setting unit 110 acquires the environment information 200 and the KPI weight calculation parameters 205. Next, in step S110b, the task setting unit 110 calculates the weight for each KPI stored in the KPI set 201 using a model having the KPI weight calculation parameters 205.

[0116] The model in step S110b may be configured by rules, or may be expressed by a machine learning model. For example, in the case of a rule-based model, the model determines the weight of a KPI for each value of the environmental information 200. The model may be expressed in the form of a function, or in the form of an If-then. In the case of a machine learning model, a model that uses the environmental information 200 as an input and the weight of each KPI as an output is trained and used.

[0117] Next, in step S110c, the task setting section 110 stores the result calculated in step S110b in the KPI weight information 202.

[0118] (History Data Generation Process According to the Embodiment) FIG. 8 is a flowchart showing the history data generation process (step S111 (FIG. 6)) according to the embodiment.

[0119] First, in step S111a, the history data generation unit 111 acquires the environment information 200, the KPI weight information 202, and the evaluation policy 203. Next, in step S111b, the history data generation unit 111 defines a task from the environment information 200 and the KPI weight information 202, and acquires a state, an action, and a reward using the evaluation policy 203. Note that in step S111b, either one task or multiple evaluation policies 203 may be used for one task.

[0120] Next, in step S111c, the history data generation unit 111 stores the data set of the state, acquisition, reward, and action acquired in step S111b in the history data 214. Note that in step S111c, the history data 214 for one episode from the start to the end of the task may be divided and handled as multiple history data 214. For example, a state / action / reward history for any number of steps may be used, or a state / action / reward history for every few steps may be used. The number of steps is acquired from the history data generation parameters 216, and is stored via a predetermined setting file (appended figure) or the data input / output unit 101.

[0121] (Dimensional compression process according to the embodiment) FIG. 9 is a flowchart showing the dimensionality reduction process (step S112 (FIG. 6)) according to the embodiment.

[0122] First, in step S 112 a , the dimensionality reduction unit 112 acquires the history data 214 from the data storage device 602 .

[0123] Next, in step S112b, the dimensionality reduction unit 112 determines whether the current system usage is during preparation or during operation. The method of determining whether the current system usage is during preparation or during operation includes designation by a command line argument when the program is executed, designation via the data input / output unit 101, etc. Also, a method of writing in a configuration file may be used, or automatic switching may be performed periodically using the system time.

[0124] The dimensionality reduction unit 112 advances the process to step S112c if the current system usage is during advance preparation, and advances the process to step S112e if the current system usage is during operation.

[0125] In step S112c, the dimension reduction unit 112 learns a dimension reduction model using a plurality of history data 214 obtained from a plurality of tasks. Next, in step S112d, the dimension reduction unit 112 stores the parameters of the dimension reduction model learned in step S112c in the task tendency space generation parameters 212. After step S112d is completed, the dimension reduction unit 112 moves the process to step S112f.

[0126] In step S112e, the dimensionality reduction unit 112 acquires the task tendency space generation parameters 212 stored in step S112d during advance preparation. Then, the task space is projected onto the task tendency space using the dimensionality reduction model, the environment information 200, the KPI weight information 202, and the history data 214. After step S112e is completed, the dimensionality reduction unit 112 advances the process to step S112f.

[0127] In step S112f, the dimensionality reduction unit 112 stores the coordinates of the projection destination on the task tendency space in step S11e in the projection information 215 of the data storage device 602.

[0128] (Measure allocation process according to the embodiment) FIG. 10 is a flowchart showing the measure allocation process (step S113 (FIG. 6)) according to the embodiment.

[0129] First, in step S113 a, the policy allocation unit 113 acquires the coordinates of the projection destination on the task tendency space from the projection information 215 stored in the data storage device 602, and acquires the policy application region parameter 206 and the policy set 204.

[0130] Next, in step S113b, the policy allocation unit 113 determines whether the coordinates of the projection destination acquired in step S113a are included in any of the policy application areas represented by the policy application area parameter 206. If the coordinates of the projection destination are included in any of the policy application areas, the policy allocation unit 113 shifts the process to step S113c, and if not, shifts the process to step S113d.

[0131] In step S113c, the policy assignment unit 113 acquires, from the policy set 204, a policy ID assigned to the policy application region of the policy application region parameter 206 including the coordinates of the projection destination, and stores the ID in the applied policy 213.

[0132] On the other hand, in step S113 d, the policy assignment unit 113 selects a policy whose general policy flag is True from among the policies included in the policy set 204, and stores the selected policy in the application policy 213.

[0133] FIG. 11 is a flowchart showing the simulation process (step S114 (FIG. 6)) according to the embodiment.

[0134] First, in step S 114 a , the simulation unit 114 acquires the environment information 200 , the KPI set 201 , the KPI weight information 202 , and the applied policy 213 stored by the policy assignment unit 113 .

[0135] Next, in step S114b, the simulation unit 114 executes a simulation using the task ID and the applied policy 213, and stores the obtained state, action, reward, and the like in the simulation control information 209.

[0136] The information stored in the simulation control information 209 in step S114b may be any information used subsequently in the adaptation value calculation unit 116 or the task definition update unit 117. For example, when only the reward is used in the adaptation value calculation unit 116 or the task definition update unit 117, only the reward may be stored in the actual application control information 210.

[0137] FIG. 12 is a flowchart showing the actual application process (step S115 (FIG. 6)) according to the embodiment.

[0138] First, in step S 115 a , the actual application unit 115 acquires the environment information 200 , the KPI set 201 , the KPI weight information 202 , and the application policy 213 stored in the policy assignment unit 113 .

[0139] Next, in step S115b, the actual application unit 115 stores in the actual application control information 210 the task ID, the agent state, the action, and the reward obtained when the application policy 213 is actually applied to the actual system.

[0140] The information stored in the actual application control information 210 in step S115b may be any information used subsequently in the adaptation value calculation unit 116 or the task definition update unit 117. For example, when only the reward is used in the adaptation value calculation unit 116 or the task definition update unit 117, only the reward may be stored in the actual application control information 210.

[0141] Note that the actual application unit 115 may acquire, as control information, the result of a user adjusting the control information acquired by the simulation unit 114 in step S114, instead of acquiring the state, behavior, and reward of the agent by actually applying the task ID and the application policy 213 to the actual system. This case corresponds to Example 1 described later.

[0142] (Adaptation Value Calculation Process According to the Embodiment) FIG. 13 is a flowchart showing the adaptive value calculation process (step S116 (FIG. 6)) according to the embodiment.

[0143] First, in step S 116 a , the adaptation value calculation unit 116 acquires the simulation control information 209 and the actual application control information 210 from the data storage device 602 .

[0144] Next, in step S116b, the adaptation value calculation unit 116 calculates an adaptation value based on the state, action, and reward stored in the simulation control information 209. The adaptation value calculation unit 116 also calculates an adaptation value based on the state, action, and reward stored in the actual application control information 210.

[0145] Next, in step S116c, the adaptive value calculation section 116 stores each adaptive value calculated in step S116b in the adaptive value set 211 in association with a simulation control information flag.

[0146] Examples of the adaptive value include the acquired reward and the task success rate, but any value that can evaluate the policy may be used. For example, the adaptive value may be a weighted sum of the acquired reward and the task success rate, or a calculated value based on the change in the acquired reward when reinforcement learning is performed for only a few episodes.

[0147] (Task definition update process according to the embodiment) 14 is a flowchart showing the task definition update process (step S117 (FIG. 6)) according to the embodiment. The task definition update process is executed when the system is in operation, and is skipped during advance preparation.

[0148] First, in step S117 a, the task definition update unit 117 acquires the KPI weight calculation parameters 205 , the simulation control information 209 , the actual application control information 210 , and the adaptation value set 211 .

[0149] Next, in step S117b, the task definition update unit 117 refers to the adaptation value set 211 and compares the magnitude relationship between the adaptation value of the simulation control information 209 and the adaptation value of the actual application control information 210. If the adaptation value of the simulation control information 209 is greater than the adaptation value of the actual application control information 210, the task definition update unit 117 moves the process to step S117c. On the other hand, if the adaptation value of the simulation control information 209 is equal to or less than the adaptation value of the actual application control information 210, the task definition update unit 117 ends the task definition update process and moves the process to step S118 (FIG. 6).

[0150] In step S117c, the task definition update unit 117 adjusts the KPI weight calculation parameter 205 so that the adaptation value of the actual application control information 210 exceeds the adaptation value of the simulation control information 209, and stores the adjusted parameter in the data storage device 602.

[0151] Next, in step S117d, the task definition update unit 117 causes the task tendency space projection unit 103 to reproject the task using the new KPI weight information 202 calculated based on the KPI weight calculation parameters 205 updated in step S117c.

[0152] (Policy area division process according to the embodiment) FIG. 15 is a flowchart showing the strategy area division process (step S118 (FIG. 6)) according to the embodiment.

[0153] First, in step S118 a, the policy domain dividing unit 118 acquires the policy set 204, the adaptive threshold 207, and the adaptive value set 211 from the data storage device 602.

[0154] Next, in step S118b, the policy domain division unit 118 obtains from the adaptation value set 211 the adaptation values ​​of all policies in the policy set 204 for the given latest task.

[0155] Next, in step S118c, the policy domain division unit 118 determines whether or not the adaptation values ​​of all the policies acquired in step S118b are below the adaptation threshold 207. If the adaptation values ​​of all the policies are below the adaptation threshold 207, the policy domain division unit 118 shifts the process to step S118d, and if there is a policy whose adaptation value is equal to or greater than the adaptation threshold 207, the policy domain division unit 118 shifts the process to step S118e.

[0156] In step S118d, the policy domain division unit 118 trains a new policy so that the adaptation threshold 207 is equal to or greater than the adaptation threshold 207, updates the policy application domain parameter 206 so that the new policy generates a new domain to which the new policy can be applied by division, and stores the parameter in the data storage device 602.

[0157] On the other hand, in step S118e, the policy area dividing unit 118 updates the policy application area of ​​the policy having the highest fitness value based on the position coordinates in the task tendency space of the latest given task.

[0158] 16 is a flowchart showing the task tendency space update process (step S119 (FIG. 6)) according to the embodiment. The task tendency space update process is executed when the system is in operation, and is skipped during advance preparation.

[0159] First, in step S119 a, the task tendency space update unit 119 acquires the measure application domain parameters 206 and the projection information 215 from the data storage device 602.

[0160] Next, in step S119b, the task tendency space update unit 119 determines whether or not the number of policy application regions on the current task tendency space exceeds the region regeneration threshold 208. If the number of policy application regions on the current task tendency space exceeds the region regeneration threshold 208, the task tendency space update unit 119 moves the process to step S119c, and if not, ends the task tendency space update process.

[0161] In step S119c, the task tendency space update unit 119 regenerates the task tendency space so that the policy application region in the task tendency space is equal to or smaller than the region regeneration threshold 208, and updates and stores the task tendency space generation parameters 212. Regenerating the task tendency space means executing the task tendency space projection process (step S103 (FIG. 6)) again.

[0162] As will be described later with reference to Figure 19, even if the distance between two tasks exceeds a threshold before and after updating the KPI weight information 202, the task tendency space may be regenerated so that the distance between the two tasks is equal to or less than the threshold.

[0163] (Visualization Processing According to the Embodiment) 17 is a flowchart showing the visualization process according to the embodiment. The visualization process is executed by the visualization unit 120 in response to an instruction from an operator.

[0164] First, in step S120 a, the visualization unit 120 acquires the measure application domain parameters 206 , the projection information 215 , the environment information 200 , the KPI set 201 , the KPI weights 202 , the adaptive value set 211 , and the display axis information 217 from the data storage device 602 .

[0165] Next, in step S120b, the visualization unit 120 displays the policy allocation area and the coordinates of the task on a display unit (not shown) of the output device 605 based on the policy application area parameter 206 and the projection information 215.

[0166] Next, in step S120c, the visualization unit 120 selects two or three variables from the environment information 200 and the KPI set 201 based on the display axis information 217, and performs visualization on a display unit (not shown) of the output device 605.

[0167] Here, the display axis information 217 of the task space may be input in advance, or may be specified via the data input / output unit 101. At this time, any combination of variables from the environment information 200 and the KPI set 201 may be selected, or multiple two-dimensional or three-dimensional spaces based on different combinations may all be visualized. Furthermore, in addition to the environment information 200 and the KPI set 201, a new axis that is easy for humans to understand may be selected by utilizing domain knowledge.

[0168] FIG. 18 is a diagram for explaining the task tendency space generation process / policy area division process according to the embodiment.

[0169] 18, a task tendency space projection diagram 2100 represents the processing functions of the task tendency space projection unit 103 and the policy area division unit 118. Tasks 2105 and 2106 exist on a task space 2012 configured by coordinate axes 2103 and 2104. A task tendency space generation process 2113 of the task tendency space projection unit 103 applies the evaluation policy 203 using a dimension reducer 2107 to project the task space 2012 onto a task tendency space 2108.

[0170] The task tendency space 2108 is composed of coordinate axes 2111 and 2112. Tasks 2105 and 2106 on the task space 2012 are projected onto tasks 2109 and 2110 on the task tendency space 2108 by a task tendency space generation process 2113.

[0171] Note that task space 2102 is actually a high-dimensional space and cannot be visualized accurately, so this means that other axes exist in the task space, such as coordinate axis 2103, which is drawn with a dashed line.

[0172] 18 is the task tendency space before the policy application area is divided. Task 2115 is the current task. Policy application area 2116 is the policy application area to which the policy learned in task 2118 is assigned. Policy application area 2117 is the policy application area to which the policy learned in task 2119 is assigned. Areas surrounded by different line types, such as the dashed and dotted lines, dashed lines, and dotted lines, are different policy application areas, and areas surrounded by the same line type are the same policy application area.

[0173] In the task tendency space 2114 shown in FIG. 18, a task 2115 does not apply to any policy application region in the task tendency space 2114, and therefore a general policy is assigned to the task 2115 through the policy region division process performed by the policy region division unit 118.

[0174] 18, the adaptability value with the policy learned in the task 2119 is high. Therefore, the policy application area 2117 of the policy learned in the task 2119 is assigned as the policy application area 2123 of the task 2115 by the policy area division process performed by the policy area division unit 118.

[0175] 18, the adaptation value with the policy learned in the task 2118 is high. Therefore, the policy application area 2116 of the policy learned in the task 2118 is assigned as the policy application area 2124 of the task 2115 by the policy area division process performed by the policy area division unit 118.

[0176] In addition, in the task tendency space 2122 shown in FIG. 18, since the adaptation values ​​of all the existing policies are insufficient, a new policy is assigned as the policy application domain 2125 of the task 2115 by the policy domain division process performed by the policy domain division unit 118.

[0177] FIG. 19 is a diagram for explaining task tendency space regeneration according to the embodiment.

[0178] The task tendency space is regenerated when the number of outlying areas (areas indicated by dashed lines in the task tendency space 2201) where areas to which the same measures are assigned are not adjacent exceeds a certain number, as in the task tendency space 2201 shown in Fig. 19. Task tendency space 2202 represents the task tendency space after regeneration.

[0179] In addition, if multiple tasks that were close in coordinates in the task tendency space before updating the KPI weight information 202 become separated by a certain distance or more in the task tendency space after updating the KPI weight information 202, the task tendency space may be regenerated.

[0180] For example, as shown in diagram 2203 of FIG. 19, assume that task 2204 before the update of KPI weight information 202 becomes task 2206 after the KPI weight update, and task 2205 before the update of KPI weight information 202 becomes task 2207 after the update of KPI weight information. Task 2204 and task 2205 are multiple tasks with similar environmental information. If the distance in the task tendency space between two tasks 2206 and 2207 after the update of KPI weight information 202 is greater than the area regeneration threshold 208, the task tendency space is regenerated. In other words, the distance in the task tendency space of each task is set close to the area regeneration threshold 208 or less. Diagram 2208 represents the task tendency space after regeneration.

[0181] FIG. 20 is a diagram for explaining visualization of a task tendency space according to the embodiment.

[0182] 20 shows an example of visualizing an area to which a policy is assigned by a task tendency space. In the display example 2301, a policy is assigned to a task 2302 according to a policy application area 2304 learned in a task 2303.

[0183] A task tendency space 2305 in FIG. 20 represents a task tendency space in which a policy application area 23041 learned in task 2302 is newly generated by dividing the policy application area 2304 assigned to task 2302 and task 2303.

[0184] A task tendency space 2306 in FIG. 20 represents the task tendency space after spatial adjustment such that task 2302 is assigned another policy according to policy application domain 2315 learned in task 2314, rather than policy application domain 2304.

[0185] 20 shows a visualized task space. A task space 2308 in the visualized display 2307 is a high-dimensional task space. A task tendency space 2309 shows a diagram in which the task space 2308 is divided along two axes.

[0186] 20 is a diagram showing a mode of visualizing information in a task tendency space via a GUI (Graphical User Interface). When a mouse pointer 2311 is focused on a task and clicked, the environmental information and KPIs of the task are displayed.

[0187] The visualization unit 120 can display changes made when the task tendency space is regenerated, using the task space visualization display 2307 and the task tendency space visualization display 2310, to support a human being in understanding the policy application area. By displaying the changes, a human being may change the threshold value via the data input / output unit.

[0188] Furthermore, the visualization unit 120 can display changes made when updating the policy application domain using the task space visualization display 2307 and the task tendency space visualization display 2310, allowing the user to confirm changes made when the task tendency space is regenerated. Based on changes in the task tendency space, a human being may cancel the regeneration of the task tendency space or request the system to regenerate it again via the data input / output unit 101.

[0189] (Effects of the embodiment) In reinforcement learning, if the environmental information and KPI are different, the task becomes different, so the number of policies corresponding to the tasks becomes huge. In this embodiment, the task space in which tasks with the environmental information and KPI as coordinate axes are distributed is projected into a task tendency space, which is a subspace with reduced dimensions. Then, the task tendency space is divided into policy application areas based on the policies with large fitness values ​​when applied to each task, thereby narrowing down the policies corresponding to the tasks and making it possible to handle any task. EXAMPLES

[0190] Example 1 is an example in which the policy management system 100 according to the above-described embodiment is applied to railway traffic rescheduling.

[0191] In this embodiment, a method is required to plan train operation rescheduling for various disruption patterns. However, since KPIs change depending on the disruption pattern and the dispatcher, it is necessary to skillfully switch between multiple methods.

[0192] 21 and 22 are diagrams for explaining a train rescheduling operation according to Example 1. Fig. 21 is a diagram relating to problem setting and tasks in train rescheduling.

[0193] A diagram 2400 in Fig. 21 shows a train operation timetable 2401. In the diagram 2400, the vertical axis 2402 represents stations discretely, and the horizontal axis 2403 represents time. The timetable 2401 shows the operation schedules of all trains.

[0194] The timetable 2401 shows terminal stations 2404, 2405 at both ends of the line, and an intermediate station 2406. For simplicity, it is assumed here that there is one intermediate station 2406. The number of stations is not limited to three. Depending on the train, operation may be discontinued midway, and a station other than the terminal stations 2404, 2405 may become the end point.

[0195] The timetable 2401 shows an up train 2407 heading from a terminal station 2405 to a terminal station 2404. The timetable 2401 also shows a down train 2408 heading from a terminal station 2404 to a terminal station 2405. The timetable 2401 shows a train turnaround 2409, and after the train has traveled to the terminus, the operation is changed from the up train to a down train, or from the down train to an up train.

[0196] Diagram 2410 in FIG. 21 shows a disruption of the timetable 2401 when a disruption occurs in railway operation. Diagram 2410, unlike diagram 2400, omits the vertical and horizontal axes of the timetable. Hereinafter, the vertical and horizontal axes of the timetable will be omitted. Diagram 2410 includes disruption 2411, disruption start time 2412, disruption duration 2413, and disruption station distance 2414 as environmental information 200. Solid lines in diagram 2410 indicate trains that are running on time, and dashed lines indicate trains that are delayed because they cannot run on time due to disruption 2411. By appropriately rescheduling operations such as suspending service for trains that are experiencing delays, the delays in the timetable can be eliminated.

[0197] Chart 2415 in FIG. 21 represents environmental information 200 for train operation rescheduling. In environmental information 200, the disruption start time does not need to be input in time format, and for example, the total number of seconds until 11:00 a.m. may be input as the time, with 00:00 a.m. as 0 seconds. The disruption time is not limited to units such as minutes or seconds. A disruption station interval of 0 represents a disruption from station A to station B, and a disruption station interval of 2 represents a disruption from station A to station C. As in the example of chart 2415, the information input for the disruption station interval may be specified by assigning one ID to all patterns, or the disruption station ID may be given as an input for each.

[0198] In the train traffic rescheduling work of this embodiment, a task is defined using the environment information 200 representing a disruption pattern as shown in chart 2415 and the KPI weights 202 as shown in chart 2416.

[0199] In the advance preparation stage, policies are trained for various tasks in advance, and the trained policies are used as evaluation policies. Policy training may be performed for each task, or one policy may be trained for multiple tasks. Also, instead of a trained policy, a rule-based policy or a random policy may be used.

[0200] In order to switch the policy to be applied based on the tendency and characteristics of the problem, a task tendency space is generated using a task tendency space projection unit 103. A policy application area on the task tendency space is divided using a policy allocation unit 113, a simulation unit 114, an adaptation value calculation unit 116, and a policy area division unit 118. Tasks used for dividing the policy application area may be tasks learned during advance preparation, or may be new tasks.

[0201] During operation, one of the actual disruption patterns is input to the policy management system 100 via the data input / output unit 101, and based on that information, it is projected onto the task tendency space by the task tendency space projection unit 103. The policy allocation unit 113 determines the policy to be applied, and a traffic rescheduling plan for the disruption is proposed.

[0202] FIG. 22 is a diagram for explaining the operation of the policy management system 100 of this embodiment. In FIG. 22, a timetable re-arrangement proposal 2500 is a policy output by the policy management system 100. A simulation of the policy output by the policy management system 100 is not immediately performed, and the timetable re-arrangement proposal 2500 is presented to a human. A timetable re-arrangement proposal 2501 in FIG. 22 is a timetable re-arrangement proposal in which the timetable re-arrangement proposal 2500 is modified by a human. A train 2502 whose timetable is displayed in the timetable re-arrangement proposal 2501 is a train whose operating time has been changed due to the timetable re-arrangement proposal 2501 modified by a human. The timetable re-arrangement proposal 2500 output by the policy management system 100 is modified by a human if necessary, and the final timetable re-arrangement proposal 2501 is determined.

[0203] The timetable rescheduling proposal 2500 output by the strategy management system 100 is stored in the simulation control information 209 , and the timetable rescheduling proposal 2501 revised by a human being is stored in the actual application control information 210 .

[0204] In the train rescheduling problem, the final timetable based on the train rescheduling proposals 2500 and 2501 is important, and the order of train rescheduling such as train cancellation is not important. Therefore, the information stored in the simulation control information 209 and the actual application control information 210 does not need to include transitions of states and rewards due to actions.

[0205] In this embodiment, the delay time of each train, route information, and train information such as express trains at the end of traffic reorganization are stored as states in the simulation control information 209 and the actual application control information 210. Traffic reorganization such as service suspension is stored as an action, and the delay recovery time and the penalty for traffic reorganization are stored as rewards.

[0206] After planning a traffic rescheduling plan for the disruption, the policy application area is updated for the disruption pattern. In train traffic rescheduling, a human being modifies the traffic rescheduling plan proposed using the policy as necessary, and the final traffic rescheduling is decided. In reinforcement learning, a reward function is designed based on KPIs prioritized by humans. Normally, the adaptation value based on the actual application control information 210, in which the traffic rescheduling plan 2501 modified by a human being is stored, does not fall below the adaptation value based on the simulation control information 209, in which the traffic rescheduling plan 2500 output by the policy management system 100 is stored.

[0207] When the adaptation value of the actual application control information 210 falls below the simulation control information 209, it is considered that the KPI weight information 202 has changed due to factors outside the environment. Factors outside the environment include, for example, a change in the number of passengers due to a change in the traffic controller, who is the user, or the establishment of a large commercial facility. The task definition update unit 117 responds to the change in the KPI weight information 202 that may occur due to factors outside the environment by updating the KPI weight calculation parameter 205.

[0208] The task definition update unit 117 updates the KPI weight calculation parameter 205 so that the adaptive value of the timetable rescheduling proposal 2501 corrected by a human being is greater than the adaptive value of the timetable rescheduling proposal 2500 output by the policy management system 100. Diagram 2416 in Fig. 21 shows an example of the KPI weight information 202 before the KPI weight calculation parameter 205 is updated. Diagram 2503 in Fig. 22 shows an example of the KPI weight information 202 after the KPI weight calculation parameter 205 is updated. Diagram 2503 shows an example in which the KPI weights have been changed to reduce the number of service cancellations rather than to eliminate delays in the schedule.

[0209] Based on the updated KPI weights 202 and the actual application control information 210 in which the timetable rescheduling proposal 2501 corrected by a human is stored, the disruption patterns of the timetable rescheduling proposal 2500 having a higher adaptation value than the adaptation value calculated by the adaptation value calculation unit 116 are reprojected onto the task tendency space.

[0210] If the adaptation value of the simulation control information 209 is smaller than the adaptation threshold 207, a new policy application area is created, and a new policy exceeding the adaptation threshold 207 is learned for the current failure pattern. Here, the adaptation threshold 207 may be a predetermined value, or may be calculated based on the adaptation value of the actual application control information 210.

[0211] On the other hand, if the adaptation value of the simulation control information 209 is greater than the adaptation threshold value 207, it is assumed that the policy application domain is functioning, and nothing is done.

[0212] By the continuous operation as described above, the number of outlying areas of the policy application area increases in the task tendency space as shown in the task tendency space 2201 in Fig. 19. When the value of the area regeneration threshold 208 is exceeded, a request is made to the system to regenerate the task tendency space, and the task tendency space is remade. Here, the area regeneration threshold 208 sets the number of outlying areas.

[0213] Alternatively, consider a case where tasks that were originally similar and close in the task tendency space, such as task 2204 and task 2205 in Fig. 19, have become distant from each other, such as task 2206 and task 2207 in Fig. 19, due to a change in KPI weight information 202. In this case, the task tendency space will no longer be able to capture the task tendency, and the value of the area regeneration threshold 208 will be exceeded. For this reason, a request is made to the system to regenerate the task tendency space, and the task tendency space is recreated. Here, the area regeneration threshold 208 sets the norm on the task tendency space.

[0214] A human may determine which reconstructed task tendency space to adopt while checking the task tendency spaces before and after the reconstruction via the visualization unit 120. The adoption of the task tendency space may be determined by the system according to the region reconstruction threshold 208.

[0215] A chart 2504 in FIG. 22 shows a task space. In the chart 2504, the axes selected from the display axis information 217 are taken as coordinate axes, and it is possible to confirm which policy is assigned to which task in the task space formed by the coordinate axes. A task 2505 in FIG. 22 is plotted based on the coordinate information of the two selected axes. For example, the task 2505 is drawn with a dotted line, but this is a policy application area included in a policy application area 2506 drawn with the same dotted line, which means that the policy learned in the task 2507 is applied. The policy assigned to each task may be displayed by coloring the plotted task 2505, labeling it with a policy ID, or the like, without being limited to a frame line.

[0216] The visualization diagram 2508 in Fig. 22 shows a diagram in which the policy application region in the task tendency space is visualized. The visualization diagram 2508 makes it possible to confirm the range in which the policy is applicable. The dotted line, dashed line, and dashed-dotted line used in the visualization diagram 2508 each represent a different policy application region. For example, the policy application region 2506 indicates the range of the region in which the policy learned for the task 2507 is applicable. In the visualization diagram 2508, all rectangles are used as region figures, but any shape can be used without being limited to a rectangle.

[0217] When the task tendency space is adjusted by the policy area dividing unit 118 or the task tendency space updating unit 119, it is possible to confirm the adjustment by visualization of the chart 2504 or the visualization diagram 2508, so that a person can review the task tendency space and support understanding of the policy application area. At this time, the adaptive threshold 207 that determines the policy application area may be determined by utilizing domain knowledge. EXAMPLES

[0218] Example 2 is an example in which the policy management system 100 according to the above-described embodiment is applied to an automatic cleaning operation.

[0219] In this embodiment, an automatic cleaning robot that performs an automatic cleaning task is required to have a policy applicable to various tasks determined by the layout of the floor on which the automatic cleaning task is performed, the location of garbage, etc. On the other hand, since the evaluation of the floor layout and collision with a wall varies depending on the user, it is necessary to switch and apply a policy suitable for the user from among multiple policies.

[0220] Fig. 23 and Fig. 24 are diagrams for explaining an automatic cleaning task by a robot according to Example 2. Fig. 24 is a diagram relating to problem setting and tasks in the automatic cleaning robot.

[0221] Diagram 2600 in FIG. 23 shows an embodiment of an automatic cleaning robot 2601. The automatic cleaning robot 2601 moves by itself while avoiding obstacles 2602, including walls, placed on the floor to be cleaned, and sucks up dirt 2603. The obstacles 2602 are beds, shelves, walls, etc., and are not moved frequently. The position of the dirt 2603 does not change significantly from day to day, but it is never exactly the same. The automatic cleaning robot 2601 sucks up the dirt 2603 scattered on the floor while circulating around the floor.

[0222] The motion simulation environment 2604 of the automatic cleaning robot 2601 simulates the motion of the automatic cleaning robot 2601. The actual motion environment 2605 is an environment in which the automatic cleaning robot 2601 actually travels. The behavior of the automatic cleaning robot 2601 is determined based on the selection result of a patrol strategy 2607 that indicates the route to patrol within the floor. The reward and adaptation value are calculated based on how far the automatic cleaning robot 2601 traveled according to each patrol strategy 2607, how much garbage 2603 it collected, how many collisions it did not have with obstacles 2602, and the like.

[0223] 23 represents the environment information 200 for the automatic cleaning robot 2601, and graph 2609 represents the KPI weight information 202. A negative value for the KPI weight information 202 shown in graph 2609 indicates a penalty, but it is not necessary to impose a penalty, and 0 may be used instead of a penalty.

[0224] The automatic cleaning robot 2601 is equipped with a contact sensor and has the function of retreating and changing direction when it collides with an obstacle 2602. The automatic cleaning robot 2601 also has an infrared sensor and can observe the relative position information of the obstacle 2602 with respect to the automatic cleaning robot 2601 and the amount of collected garbage 2603. However, the infrared sensor contains noise and therefore contains errors.

[0225] The automatic cleaning robot 2601 is rechargeable and automatically returns to a power source after a certain period of time has passed or when the remaining charge falls below a threshold. Increasing the suction power of the dirt allows the robot to collect more dirt, but the battery will be consumed more quickly. On the other hand, decreasing the suction power of the dirt reduces battery consumption and increases the time the robot can travel, but the amount of dirt that can be sucked up at one time decreases.

[0226] During advance preparation, before product shipment, policies are trained in advance for various room environments, and the trained policies are used as evaluation policies. Policy training may be performed for each task, or one policy may be trained for multiple tasks. Moreover, instead of a trained policy, a rule-based policy or a random policy may be used.

[0227] A policy learned using all tasks used in training the evaluation policies is used as a general policy.

[0228] A task tendency space is generated according to the environment information 200 shown in chart 2608 of Fig. 26 and the KPI weight information 202 shown in chart 2609, using the task tendency space projection unit 103. In addition, a policy application area on the task tendency space is divided using the policy allocation unit 113, the simulation unit 114, the adaptation value calculation unit 116, and the policy area division unit 118. Tasks used for dividing the policy application area may be tasks learned during advance preparation, or may be new tasks.

[0229] A diagram 2700 in Fig. 24 shows a policy application region on a task tendency space in the second embodiment. The diagram 2700 is an example in which the region division ID of the policy application region parameter 305 is specified so as to divide the policy application region along a boundary that satisfies the adaptive threshold 207. That is, the policy region dividing unit 118 divides the task tendency space based on the adaptive value or the coordinate space so that the policy application region on the task tendency space becomes as large as possible. This makes it possible to divide the policy application region while maintaining a large region.

[0230] Like chart 2700, chart 2701 in Fig. 24 also shows a policy application region on a task tendency space in the second embodiment. Chart 2701 is an example in which the region division ID of the policy application region parameter 305 is specified so that Voronoi division is performed according to the norm of the task in the task tendency space, not according to the adaptive threshold 207. Although division may be performed by the policy application region as shown in chart 2701, the division method shown in chart 2700 is adopted in the second embodiment.

[0231] When the automatic cleaning robot 2601 is in use, the actual layout of the room is input by a human via the data input / output unit 101 and stored in the environment information 200. Based on that information, the task of cleaning the room is projected onto the task tendency space by the task tendency space projection unit 103. The policy allocation unit 113 determines the policy to be applied, and a patrol route is determined in a simulation.

[0232] The layout of the room may be automatically determined based on sensor information obtained by test driving of the automatic cleaning robot 2601 without a human input via the data input / output unit 101 .

[0233] The travel path including the action and state and the acquired reward output by the policy management system 100 on the operation simulation environment 2604 are stored in the simulation control information 209. In addition, the travel path and the acquired reward when the policy is applied in the actual operation environment 2605 are stored in the actual application control information 210.

[0234] The KPI weight information 202 shown in chart 2609 is adjusted based on the difference between the simulation control information 209 and the actual application control information 210, which is caused by the influence of the position of the garbage 2603, the size and positional deviation of the obstacle 2602, friction with the ground, etc. At this time, the environment information 200 shown in chart 2608 may be updated based on the acquired sensor information.

[0235] If the adaptation value of the simulation control information 209 is smaller than the adaptation threshold 207, a new policy application area is created, and a new policy exceeding the adaptation threshold 207 is learned for the current failure pattern. Here, the adaptation threshold 207 may be a predetermined value, or may be calculated based on the adaptation value of the actual application control information 210.

[0236] If the adaptation value of the simulation control information 209 is greater than the adaptation threshold 207, it is assumed that the policy application domain is functioning, and nothing is done.

[0237] When the unused degree of the policy application area divided during advance preparation or the policy application area before the layout change due to rearrangement or the like exceeds the value of the area regeneration threshold 208 due to continuous operation, the system is requested to regenerate the task tendency space and the task tendency space is reconstructed. Here, the area regeneration threshold 208 is set, for example, when the policy application area has not been selected ten times in a row.

[0238] The task tendency space may be regenerated based on differences in environmental information 200 such as layout and KPI weight information 202 .

[0239] The task tendency space before and after regeneration may be checked by a human via the visualization unit 120, and regeneration of the task tendency space may be adopted, or may be determined by the system in accordance with the region regeneration threshold 208.

[0240] Diagram 2702 in Fig. 24 shows the adjustment of the task tendency space through human operation. Depending on the tasks displayed via the GUI, the system can be requested to review the task tendency space. For example, if the layout has changed significantly due to a redecoration and the policy for the old layout 2703 is no longer necessary, there is no need to retain the policy for that task, so the human requests the system to modify the policy application area.

[0241] 24 is a diagram after the system that received the request has modified the policy application area on the task tendency space 2705 by the policy area dividing unit 118. As in the task tendency space 2705, the policy application area disappears, and the policy assigned to this area becomes a general policy.

[0242] When unnecessary areas increase due to rearrangement, as in the task tendency space 2705 in Fig. 23, a human requests the system to regenerate the task tendency space. Regeneration may be automatically determined by defining the area regeneration threshold 208.

[0243] During operation, there may be cases where tasks change and measures that were previously required become unnecessary, but the policy management system 100 of this embodiment updates the task definition, policy application area, and task tendency space as appropriate.

[0244] Although the embodiments of the present disclosure have been described above in detail, the present disclosure is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present disclosure. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present disclosure is not necessarily limited to those having all of the configurations described. In addition, it is possible to add, delete, or replace part of the configuration of the above-described embodiments with other configurations.

[0245] Furthermore, the above-mentioned configurations, functional units, processing units, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. The above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), an IC card, an SD card, a DVD, or other recording medium.

[0246] In addition, in each of the above figures, the control lines and information lines are shown as those considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the actual implementation. For example, it may be considered that almost all the components are actually connected to each other.

[0247] The above-described arrangement of the functions and data of the policy management system 100 is merely an example. The arrangement of the functions and data can be changed to an optimal arrangement in terms of the performance of the hardware and software, processing efficiency, communication efficiency, and the like. [Explanation of symbols]

[0248] 100: Policy management system, 103: Task tendency space projection unit, 104: Task tendency space adjustment unit, 110: Task setting unit, 111: History data generation unit, 112: Dimension reduction unit, 113: Policy allocation unit, 114: Simulation unit, 115: Actual application unit, 116: Adaptation value calculation unit, 117: Task definition update unit, 118: Policy area division unit, 119: Task tendency space update unit, 120: Visualization unit, 121: Data input / output unit, 200: Environmental information, 201: KPI set, 202: K PI weight information, 203: evaluation policy, 204: policy set, 205: KPI weight calculation parameter, 206: policy application area parameter, 207: adaptation threshold, 208: area regeneration threshold, 209: simulation control information, 210: actual application control information, 211: adaptation value set, 212: task tendency space generation parameter, 213: applied policy, 214: history data, 215: projection information, 216: history data generation parameter, 217: display axis information, 305: policy application area parameter.

Claims

1. A policy management method executed by a policy management system that assigns a reinforcement learning model, which is used to determine an action when an agent executes a task in an environment in reinforcement learning, to the task as a policy, comprising: a task setting step of acquiring environmental information including settings and parameters related to the environment and KPI weight calculation parameters for calculating weights of KPIs (Key Performance Indicators), and calculating weight information for the KPIs based on the environmental information and the KPI weight calculation parameters; a history data generation step of defining the task based on the environmental information, the KPI, and the weight information, and acquiring the state of the environment, the action of the agent, and a reward for the action when an evaluation policy is applied to the task, to generate history data; a dimension reduction step of projecting the task onto the task tendency space, which is a subspace of lower dimension than the task space configured from the environmental information and the KPIs, using predetermined parameters for projecting the history data based on the characteristics of the task, the environmental information, the weight information, and the history data; a fitness value calculation step of calculating a fitness value for evaluating the suitability of the policy when the policy is assigned to the task; a policy domain division step of dividing the task tendency space into policy application domains to which the policies are applied based on the adaptation value; A policy management method comprising:

2. 2. The method of claim 1, further comprising: a simulation step of acquiring first control information including the state, the action, and the reward when the policy is applied to the task in a simulation; an actual application step of acquiring, as second control information, a result of adjustment by a user made to the first control information acquired by the simulation step; In the adaptive value calculation step, calculating a first adaptive value based on the first control information, and a second adaptive value based on the second control information; In the step of dividing the policy domain, Dividing the task tendency space into the policy application regions based on a comparison result of comparing the first adaptation value with the second adaptation value. A policy management method characterized by:

3. 2. The method of claim 1, further comprising: a simulation step of acquiring first control information including the state, the action, and the reward when the policy is applied to the task in a simulation; an actual application step of applying the policy to a real system to obtain second control information including the state, the action, and the reward; In the adaptive value calculation step, calculating a first adaptive value based on the first control information, and a second adaptive value based on the second control information; In the step of dividing the policy domain, Dividing the task tendency space into the policy application regions based on a comparison result of comparing the first adaptation value with the second adaptation value. A policy management method characterized by:

4. 4. The method for managing measures according to claim 2 or 3, a task definition update step of updating the weight information of the KPI based on the comparison result; a task tendency space updating step of comparing a distance in the task tendency space after updating the weight information of the plurality of tasks whose environmental information is similar to each other with a threshold, and regenerating the task tendency space if the distance exceeds the threshold so that the distance is equal to or less than the threshold; A policy management method comprising:

5. 2. The method of claim 1, further comprising: a task tendency space updating step of determining whether the number of policy application areas divided by the policy area dividing step exceeds a threshold, and if the number of policy application areas exceeds the threshold, regenerating the task tendency space so that the number of policy application areas is equal to or less than the threshold. A policy management method comprising:

6. 2. The method of claim 1, further comprising: a policy allocation step of allocating the policy to the task based on whether the task is included in the policy application region in the task tendency space; A policy management system comprising:

7. 2. The method of claim 1, further comprising: In the step of dividing the policy domain, Dividing the task tendency space based on the adaptation value so that the policy application region in the task tendency space is as large as possible. A policy management system characterized by:

8. 2. The method of claim 1, further comprising: a visualization step of visualizing the task tendency space of the policy application area divided by the policy area division step in two or three dimensions with the environmental information or the KPI as a coordinate axis; A policy management method comprising:

9. A policy management system that assigns a reinforcement learning model, which is used to determine an action when an agent executes a task in an environment in reinforcement learning, to the task as a policy, a task setting unit that acquires environmental information including settings and parameters related to the environment and KPI (Key Performance Indicator) weight calculation parameters for calculating weights of KPIs, and calculates weight information for the KPIs based on the environmental information and the KPI weight calculation parameters; a history data generation unit that defines the task based on the environmental information, the KPI, and the weight information, and generates history data by acquiring the state of the environment, the action of the agent, and a reward for the action when an evaluation measure is applied to the task; a dimension reduction unit that projects the task onto the task tendency space, which is a subspace of lower dimension than the task space configured from the environmental information and the KPIs, using predetermined parameters for projecting the history data based on the characteristics of the task, and the environmental information, the weight information, and the history data; an adaptability value calculation unit that calculates an adaptability value for evaluating the suitability of the policy when the policy is assigned to the task; a policy domain division unit that divides the task tendency space into policy application domains to which the policies are applied based on the adaptation value; A policy management system comprising:

Images