Policy selection program, policy selection system, and policy selection method

Abstract

To achieve policy selection closer to the preference of a user.SOLUTION: Based on a plurality of utility values for learning calculated according to a plurality of combinations of an action value relating to an evaluation value for each of a plurality of objectives of a predetermined policy for learning and a preference value for learning relating to a priority of each of the plurality of objectives when selecting an action, A policy set calculation unit configured to calculate a policy set including a plurality of policies corresponding to a plurality of objectives, a preference value acquisition unit configured to acquire a predetermined preference value, and a policy selection unit configured to select at least one policy included in the plurality of policies based on a selection-time mismatch degree regarding a degree of mismatch between an action value of each of the plurality of policies and the predetermined preference value, the selection-time mismatch degree being calculated according to the action value of each of the plurality of policies and the predetermined preference value.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a measure selection program, a measure selection system, and a measure selection method. [Background technology]

[0002] Conventionally, techniques for selecting a strategy according to the state of the environment under multiple objectives have been known.

[0003] For example, a computer implementation method described in Non-Patent Document 1 has proposed a method for solving sequential decision-making problems with multiple objectives, in which a Pareto front is learned using multi-objective reinforcement learning, and a policy is selected and used from the Pareto front according to the preference of the entity (e.g., user) who will enjoy the benefits from the execution of the policy, as to which objective is emphasized. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Jie Xu, Yunsheng Tian, ​​Pingchuan Ma, Daniela Rus, Shinjiro Sueda, and Wojciech Matusik. “Prediction-guided multi-objective reinforcement learning for continuous robot control” In International Conference on Machine Learning, pp. 10607-10616. PMLR, 2020. Summary of the Invention [Problem to be solved by the invention]

[0005] However, the computer-implemented method described in Non-Patent Document 1 focuses on comprehensively learning Pareto-optimal policies, and does not describe or suggest a method for selecting a policy desired by a user from the Pareto front.

[0006] Therefore, the present invention aims to realize a policy selection that is closer to the user's preferences (highly accurate in the sense that the user's preferences and the selected policy are close in distance). [Means for solving the problem]

[0007] A policy selection program according to one embodiment of the present invention causes a computer to implement the following: a policy set calculation unit that calculates a policy set including a plurality of policies corresponding to a plurality of objectives based on a plurality of learning utility values ​​calculated according to a plurality of combinations of an action value related to the evaluation value of a predetermined learning policy for each of a plurality of objectives and a learning preference value related to the priority of each of the plurality of objectives when selecting an action; a preference value acquisition unit that acquires the predetermined preference value; and a policy selection unit that selects at least one policy included in the plurality of policies based on a selection-time mismatch degree related to the degree of mismatch between the action value of each of the plurality of policies and the predetermined preference value, which is calculated according to the action value of each of the plurality of policies and the predetermined preference value. [Effects of the Invention]

[0008] According to the present invention, it is possible to realize a policy selection that is closer to the user's preferences. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an overview of processing in a measure selection system 100 according to an embodiment of the present invention. [Figure 2] 1 is a diagram showing the configuration of a measure selection system 100 according to an embodiment of the present invention. [Figure 3] FIG. 10 is a diagram showing an outline of a policy set calculation process performed by a policy set calculation unit 120. [Figure 4]FIG. 10 is a diagram showing an outline of a measure selection process performed by a measure selection unit 150. [Figure 5] FIG. 10 is a diagram illustrating an execution result of the present embodiment. [Figure 6A] FIG. 10 is a diagram illustrating an execution result of the present embodiment. [Figure 6B] FIG. 10 is a diagram illustrating an execution result of the present embodiment. [Figure 7] 1 is a flowchart showing an example of processing in the measure selection system 100. [Figure 8] FIG. 8 is a diagram illustrating an example of the hardware configuration of a computer 800. DETAILED DESCRIPTION OF THE INVENTION

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will now be described with reference to the accompanying drawings, in which: Figure 1 is a diagram showing an overview of processing in a measure selection system 100 according to one embodiment of the present invention.

[0011] The policy selection system 100 is an information processing system implemented by a policy selection program. The policy selection system 100 selects a policy depending on the state of the environment.

[0012] First, the policy selection system 100 calculates a policy set including multiple policies corresponding to multiple objectives (S101). Next, the policy selection system 100 acquires the environmental state from the environment (S102). The policy selection system 100 selects a policy from the policy set according to the environmental state and predetermined preferences (S103).

[0013] The policy selection system 100 outputs the selected policy (S104). At this time, the policy selection system 100 may, for example, execute an action according to the selected policy. Then, the policy selection system 100 obtains a reward according to the policy from the environment (S105).

[0014] In this way, the policy selection system 100 can function as, for example, an agent of a sequential decision-making model that sequentially selects policies according to the state of the environment. In this case, the policy selection system 100 is, for example, an agent of a multi-objective sequential decision-making model that can aim at optimizing multiple objectives.

[0015] The policy selection system 100 may use, for example, machine learning techniques, particularly, for example, multi-objective reinforcement learning techniques, to select policies sequentially.

[0016] The measures selected by the measure selection system 100 may be, for example, measures that are optimal for achieving multiple objectives. Note that the "optimal measures" in this embodiment include, for example, measures that are considered to be effective in achieving multiple objectives under predetermined constraints.

[0017] In addition, in this embodiment, the case where the plurality of objectives is two objectives will be described as an example, but the number of the plurality of objectives is not limited to this.

[0018] 2 is a diagram showing the configuration of a measure selection system 100 according to one embodiment of the present invention. The measure selection system 100 is communicably connected to an environmental system 200 and a user device 300 via a network such as the Internet. Details of the measure selection system 100 will be described later.

[0019] The environmental system 200 outputs state information indicating the state of the environment to the measure selection system 100 .

[0020] When the strategy selection system 100 selects at least one of a strategy involving acceleration and a strategy involving deceleration for a vehicle, the environmental information may include, for example, the coordinates and speed of the vehicle and the coordinates and speeds of other vehicles.When the strategy selection system 100 selects a control strategy for a valve in a plant, the environmental information may be the state of the plant (for example, the temperature, color, etc. inside a blast furnace).

[0021] Furthermore, the environment system 200 receives an action according to a measure from the measure selection system 100 and outputs a reward according to the action to the measure selection system 100.

[0022] When the strategy selection system 100 selects at least one of a strategy involving acceleration and a strategy involving deceleration in the vehicle, the action accepted by the environmental system 200 may be at least one of a strategy involving acceleration and a strategy involving deceleration in the vehicle.

[0023] The user device 300 is an information processing device used by a user who uses the policy selection system 100, and is, for example, a computer, a smartphone, a tablet terminal, a personal computer, or the like.

[0024] A user accesses the policy selection system 100 using the user device 300, and outputs to the policy selection system 100, for example, preference values ​​relating to the priorities of the user's respective objectives when selecting an action.

[0025] Here, the user may be, for example, a subject who receives the execution result of a measure selected by the measure selection system 100. That is, for example, when the measure selection system 100 selects at least one of a measure involving acceleration and a measure involving deceleration for a vehicle, the user may be a person who uses the vehicle (a driver or a passenger).

[0026] The user may also be, for example, an administrator who manages the measure selection system 100.

[0027] Although one environmental system 200 and one user device 300 are shown in FIG. 2, there may be a plurality of environmental systems 200 and a plurality of user devices 300.

[0028] Next, details of the policy selection system 100 will be described. The policy selection system 100 includes a storage unit 110, a policy set calculation unit 120, a preference value acquisition unit 130, a state acquisition unit 140, a policy selection unit 150, and a policy output unit 160. Each unit shown in Fig. 2 can be realized, for example, by using a storage area or by having a processor execute a program stored in the storage area.

[0029] The storage unit 110 stores information to be processed in the policy selection system 100. The storage unit 110 can store, for example, policy set information, preference value information, and state information, which will be described later.

[0030] The policy set calculation unit 120 calculates a policy set including multiple policies corresponding to multiple objectives based on multiple learning utility values ​​calculated according to multiple combinations of behavioral values ​​related to the evaluation values ​​of a specified learning policy for each of multiple objectives and learning preference values ​​related to the priorities of each of the multiple objectives when selecting an action, and stores policy set information related to the policy set in the memory unit 110.

[0031] Here, the action value indicates the evaluation value of each of a plurality of objectives (for example, two objectives) when a measure is selected (for example, executed). The action value may be, for example, a vector indicating the evaluation value of each of the two objectives. A predetermined action value is set for a predetermined measure.

[0032] The training strategies include, for example, multiple strategies that the strategy selection system 100 can select.

[0033] The training preference values ​​indicate preference values ​​for each of a plurality of objectives (for example, two objectives), and may be, for example, a vector indicating preference values ​​for each of the two objectives.

[0034] The preference values ​​for each of the multiple objectives included in the training preference values ​​may be relative preference values. That is, the sum of the preference values ​​for each of the multiple objectives included in the training preference values ​​may be, for example, 1. The training preference values ​​may be at least one of multiple possible combinations of the preference values ​​for each of the multiple objectives. That is, for example, if the multiple objectives are two objectives, the training preference values ​​may be in increments of 0.1, such as (1,0), (0.9,0.1), (0.8,0.2), etc. Note that the increments may also be other increments (e.g., 0.01).

[0035] The learning preference values ​​indicate the priority of each of multiple objectives when selecting a policy. For example, if the preference value for the first objective is 0.9 and the preference value for the second objective is 0.1, an agent with these preference values ​​will learn a policy that prioritizes the first objective. Also, if the preference value for the first objective is 0.5 and the preference value for the second objective is 0.5, an agent with these preference values ​​will evaluate both the first and second objectives equally and select a policy.

[0036] The utility value is a value indicating the utility obtained according to preferences when a policy is selected, and is calculated, for example, based on a utility function based on action values ​​and preference values. The training utility value is calculated, for example, based on action values ​​and training preference values.

[0037] Here, the utility function is not particularly limited, but may be, for example, a linear scalarized utility function, specifically, w Q (Equation (1)) Here, w may be, for example, a vector indicating preference values, and Q may be, for example, a vector indicating action values. The utility function may also be a utility function based on a Chebyshev function.

[0038] The training utility value may also be calculated based on a training mismatch degree relating to the degree of mismatch between the behavioral value and the training preference value, i.e., in this case, the training utility value may be calculated based on a utility function that is further based on the training mismatch degree.

[0039] Here, the learning mismatch degree may be, for example, a mismatch degree according to the relationship between the multiple objectives indicated by the action value of the learning policy and the relationship between the multiple objectives indicated by the learning preference value. In particular, for example, the learning mismatch degree may be a mismatch degree according to the direction of the vector indicated by the action value of the learning policy and the direction of the vector indicated by the learning preference value. More specifically, the learning mismatch degree may be a mismatch degree according to a vector obtained by normalizing the vector indicated by the action value of the learning policy and a vector obtained by normalizing the vector indicated by the learning preference value. In this case, the learning mismatch degree may be, for example, Equation (2).

[0040] The degree of mismatch during training is expressed, for example, by equation (2): Here, ∥Q∥ indicates, for example, the L2 norm of vector Q.

[0041]

number

[0042] When the mismatch degree at the time of learning is expressed by Equation (2), a utility function further based on the mismatch degree at the time of learning is, for example, w·Q―kM(w,Q) (Eq.(3)) Here, k is a predetermined coefficient that balances w·Q and M(w,Q). Note that the utility function further based on the degree of mismatch during training is not limited to this.

[0043] The training policy and the policies included in the policy set may be the same policy. That is, the training policy is, for example, a policy (e.g., all selectable policies) that the policy set calculator 120 uses in the policy set calculation process among the policies selectable by the policy selection system 100.

[0044] Furthermore, the degree of mismatch during learning may be a degree of mismatch during learning that takes into consideration predetermined weighting for each of a plurality of purposes.

[0045] The policy set calculation unit 120 can calculate a policy set based on a training utility value based on the mismatch degree at training time.

[0046] Furthermore, the policy set calculation unit 120 can calculate a policy set including multiple policies that are Pareto optimal for achieving multiple objectives. In this case, the policy set may be, for example, a Pareto front.

[0047] In this way, the policy set calculator 120 can calculate (learn) a policy set based on the training utility values.

[0048] The policy set calculation algorithm used by the policy set calculation unit 120 may be a calculation algorithm based on existing information science technology. Specifically, for example, the policy set calculation unit 120 may calculate the policy set using an evolutionary algorithm or a multi-objective reinforcement learning algorithm. The policy set calculation unit 120 can calculate (learn) a policy set (e.g., a Pareto front) by updating randomly initialized policies using, for example, a multi-objective policy gradient method.

[0049] Conventional techniques for calculating the Pareto front do not take into account the degree of mismatch during training. Therefore, for example, when the scales of the action values ​​for multiple objectives are different, the scale of the action values ​​significantly contributes to the training utility value, resulting in a mismatch between the training utility value and the training preference value. Specifically, consider a case where the action value for the first objective in policy A is 1 and the action value for the second objective is 20, and the action value for the first objective in policy B is 5 and the action value for the second objective is 1, and the training preference values ​​indicate a priority of the first objective of 0.8 and a priority of the second objective of 0.2. In this case, when the utility function shown in Equation (1) is applied, the training utility value for policy A is 4.8 and the training utility value for policy B is 4.2, so policy A is selected and the Pareto front is calculated using a predetermined technique. Thus, even though the training preference values ​​prioritize the first objective, the selected policy A has a lower action value for the first objective than policy B.

[0050] Therefore, the policy set calculation unit 120 calculates a policy set taking into account the degree of mismatch during training. As a result, the policy selection system 100 can calculate, for example, a Pareto front suitable for the training preference values. Specifically, when the utility function shown in equation (3) is applied, the training utility value of policy A is 3.6 and the training utility value of policy B is 4.1, so policy B is selected and the Pareto front is calculated using a predetermined technique.

[0051] FIG. 3 is a diagram showing an outline of the process of calculating a set of measures by the policy set calculator 120. As shown in FIG.

[0052] Area 301 indicates the area when the entire set of measures selectable by the policy selection system 100 under predetermined constraints is mapped onto a graph showing the action values ​​of objectives 1 and 2. Point 302 indicates a measure included in the Pareto front from the entire set of measures selectable by the policy selection system 100. The same applies to points other than point 302. In other words, point 302 and the other points constitute the Pareto front.

[0053] The preference value acquisition unit 130 acquires a predetermined preference value and stores the acquired preference value in the storage unit 110 .

[0054] The preference value acquiring unit 130 may acquire preference values ​​from, for example, the user device 300. In this case, the preference values ​​may indicate the priorities of each of a plurality of objectives when the user selects an action.

[0055] The state acquisition unit 140 acquires environmental information from the environmental system 200 and stores the acquired environmental information in the storage unit 110 .

[0056] The policy selection unit 150 selects at least one policy from the plurality of policies based on a mismatch degree at selection, which is calculated based on the action value of each of the plurality of policies and a predetermined preference value, and relates to the mismatch degree between the action value of each of the plurality of policies and the predetermined preference value. The policy selection unit 150 can select at least one policy that is optimal for the environment indicated by the environmental information, further based on the environmental information.

[0057] Here, the mismatch degree at selection may be, for example, the mismatch degree according to the relationship between the multiple objectives indicated by the action values ​​of each of the multiple measures and the relationship between the multiple objectives indicated by a predetermined preference value. In particular, the mismatch degree at selection may be the mismatch degree according to the direction of a vector indicated by the action values ​​of each of the multiple measures and the direction of a vector indicated by a predetermined preference value. More specifically, the mismatch degree at selection may be the mismatch degree according to a vector obtained by normalizing the vector indicated by the action values ​​of each of the multiple measures and a vector obtained by normalizing the vector indicated by the predetermined preference value. In this case, the mismatch degree at selection may be, for example, Equation (2).

[0058] The mismatch degree at the time of selection may be a mismatch degree at the time of learning that takes into consideration predetermined weighting for each of a plurality of purposes.

[0059] Furthermore, the learning mismatch degree and the selection mismatch degree may be values ​​expressed by the same formula, or may be values ​​expressed by different formulas.

[0060] The policy selection unit 150 can select at least one policy included in the policy set based on the magnitude of the mismatch degree at the time of selection for each of the policies included in the policy set. In this case, the policy selection unit 150 may select a policy having a mismatch degree of a predetermined value (e.g., a value equal to or less than a predetermined threshold), or may select a policy with a predetermined rank, for example, the smallest mismatch degree.

[0061] FIG. 4 is a diagram showing an outline of the measure selection process performed by the measure selection unit 150.

[0062] Point 401 is one policy included in the policy set. Vector 402 indicates the action value vector of the policy indicated by point 401. Point 402 is a policy different from point 401 that is included in the policy set. Vector 404 indicates the action value vector of the policy indicated by point 403. Vector 405 indicates the preference value vector acquired by the preference value acquisition unit 130.

[0063] Comparing the degree of mismatch at selection with vector 405, vector 404 has a smaller degree of mismatch at selection with vector 405 than vector 402. Therefore, the policy selection unit 150 can select the policy indicated by point 403 rather than the policy indicated by point 401. Note that the policy selection unit 150 may select a policy with an even smaller degree of mismatch at selection.

[0064] The measure output unit 160 outputs the measure selected by the measure selection unit 150 .

[0065] The measure output unit 160 may output the selected measure (in other words, execute the selected measure) to, for example, the environmental system 200. The measure output unit 160 may also output the selected measure to the user device 300.

[0066] Next, a first embodiment will be described in which the measure selection system 100 is applied to a vehicle driving assistance system. Here, the vehicle driving assistance system may be, for example, a system that performs automatic driving. That is, in this case, the measure selection system 100 functions as, for example, an agent that controls the automatic driving of the vehicle.

[0067] In the first embodiment, the multiple objectives may be, for example, an objective to safely operate the vehicle while maintaining a speed and an objective to operate the vehicle at a faster speed. The preference values ​​may be, for example, a preference value indicating a priority given to safety by a user of the vehicle and a preference value indicating a priority given to a driving speed. The multiple strategies may be, for example, a strategy involving acceleration and a strategy involving deceleration by the vehicle. The multiple strategies may also be a strategy for keeping the vehicle in its lane and a strategy for overtaking by the vehicle.

[0068] In the first embodiment, the environmental system 200 may be a vehicle system provided in a vehicle. The vehicle system acquires environmental information indicating the coordinates and speed of the vehicle and the coordinates and speeds of other vehicles from predetermined sensors provided in the vehicle, and controls the speed and direction of the vehicle.

[0069] The policy set calculation unit 120 calculates a policy set that forms the Pareto front based on the utility function expressed by, for example, equation (3).

[0070] The preference value acquisition unit 130 acquires a preference value of a vehicle user, for example, based on the operation of the user. The user, for example, uses the user device 300 to output a preference value indicating a priority for safety and a preference value indicating a priority for driving speed to the strategy selection system 100. When the priority for safety is high, the user's preference value indicates that the user prefers, for example, to refrain from overtaking and drive more safely. When the priority for driving speed is high, the user's preference value indicates, for example, that the user prefers to actively overtake and drive at a faster speed.

[0071] The measure selection system 100 may include, as constraints for implementing measure selection, speed conditions based on traffic regulations and conditions that take safety into consideration to a certain extent.

[0072] The strategy selection unit 150 selects at least one of a strategy involving acceleration and a strategy involving deceleration based on the mismatch degree at the time of selection. The strategy output unit 160 outputs the selected strategy to the environmental system 200 to assist the driving of the vehicle.

[0073] Fig. 5 is a diagram showing the execution results of this embodiment. The diagram shown in Fig. 5 is a diagram in which the Pareto front calculated by the policy set calculation unit 120 is mapped based on the action value for each objective 1 and objective 2 when (A) the policy set calculation unit 120 uses the utility function shown in formula (3) and when (B) the policy set calculation unit 120 uses the utility function shown in formula (1).

[0074] Compared to (B) using the utility function shown in equation (1), (A) using the utility function shown in equation (3), which takes into account the degree of mismatch during learning, a wider range of Pareto fronts was calculated.

[0075] FIG. 6 is a diagram showing the execution results of this embodiment. The diagram shown in FIG. 6 shows a case where, when selecting one policy from a plurality of policies included in a policy set, the policy with the smallest mismatch degree at selection is selected (FIG. 6A), and a case where the policy with the highest utility indicated by Equation (1) is selected without considering the mismatch degree at selection (FIG. 6B). The selected policies are mapped based on the rewards for each of Objectives 1 and 2 obtained by the selected policies. In FIGS. 6A and 6B, among the objectives (Objective 1) of operating the vehicle at a faster speed and the objective (Objective 2) of operating the vehicle safely while maintaining the speed, the coloring is based on the strength of Objective 1 relative to Objective 2. In other words, if the user places more importance on Objective 1, the coloring is lighter.

[0076] In FIG. 6A, as in the case of the policy mapped to area 601, the more importance is attached to objective 1, the more likely a policy with a higher reward for objective 1 is selected. That is, in FIG. 6A, a policy suited to the user's preferences is selected. On the other hand, in FIG. 6B, as in the case of the policy mapped to area 602, a policy with a higher reward for objective 2 is often selected despite the fact that more importance is attached to objective 1. That is, in FIG. 6B, a policy suited to the user's preferences is not selected. In this way, when the policy selection unit 150 selects a policy taking into account the degree of mismatch at the time of selection, the policy selection system 100 can select a policy suited to the preference value.

[0077] Next, a second embodiment will be described in which the measure selection system 100 is applied to a plant control system. Here, the plant control system may be, for example, a system that controls a plant (e.g., a blast furnace) that refines a predetermined substance. That is, in this case, the measure selection system 100 functions as an agent that controls the plant.

[0078] In the second embodiment, the multiple objectives may be, for example, an objective to minimize the number of operations for controlling the plant and an objective to increase the purified purity of a predetermined substance. Furthermore, the preference values ​​may be, for example, a preference value indicating a priority for minimizing the number of operations for controlling the plant and a preference value indicating a priority for increasing the purified purity of a predetermined substance. Furthermore, the multiple measures may be, for example, a measure involving acceleration and a measure involving deceleration in a vehicle. Furthermore, the multiple measures may be, for example, a valve control measure (e.g., a measure to change the opening / closing state of a predetermined valve to a predetermined state).

[0079] In this regard, in the past, skilled workers controlled the plant by checking the plant's status (for example, the temperature and color inside the blast furnace). When functioning as an agent that controls the plant, the measure selection system 100 can select an appropriate measure depending on the plant's status. This reduces the workload and risks of the plant's workers.

[0080] FIG. 7 is a flowchart showing an example of processing in the measure selection system 100.

[0081] First, the policy set calculation unit 120 calculates a policy set including multiple policies corresponding to multiple objectives based on multiple training utility values ​​(S701). The policy set calculation unit 120 can calculate a policy set that becomes the Pareto front, for example. Next, the preference value acquisition unit 130 acquires preference values ​​corresponding to the user's preferences, for example (S702). The policy selection unit 150 selects at least one policy included in the multiple policies based on the mismatch degree at selection related to the mismatch degree between the action value and the acquired preference values, and outputs the selected at least one policy (S703).

[0082] Next, an example of a hardware configuration in which the measure selection system 100 is realized by a computer 800 will be described with reference to Fig. 8. Fig. 8 is a diagram showing an example of a hardware configuration of the computer 800.

[0083] As shown in FIG. 8, a computer 800 includes, for example, a processor 801, a memory 802, a storage device 803, an input I / F unit 804, a data I / F unit 805, a communication I / F unit 806, and a display device 807.

[0084] Computer 800 may be, for example, a server computer, a personal computer (e.g., desktop, laptop, tablet, etc.), a media computing platform (e.g., cable, satellite set-top box, digital video recorder, etc.), a handheld computing device (e.g., PDA, email client, etc.), or any other type of computing or communications platform.

[0085] The processor 801 is a control unit that controls various processes in the computer 800 by executing programs stored in the memory 802 .

[0086] The memory 802 is a storage medium such as a RAM (Random Access Memory), etc. The memory 802 temporarily stores the program code of the program executed by the processor 801 and data required when the program is executed.

[0087] The storage device 803 is a non-volatile storage medium such as a hard disk drive (HDD), flash memory, etc. The storage device 803 stores an operating system and various programs for realizing the above-mentioned components.

[0088] The input I / F unit 804 is a device for receiving input from a user. The input I / F unit 804 is, for example, a keyboard, a mouse, a touch panel, various sensors, a wearable device, etc. The input I / F unit 804 may be connected to the computer 800 via an interface such as a USB (Universal Serial Bus).

[0089] The data I / F unit 805 is a device for inputting data from outside the computer 800. The data I / F unit 805 is, for example, a drive device for reading data stored in various storage media. The data I / F unit 805 may be provided outside the computer 800. When the data I / F unit 805 is provided outside the computer 800, the data I / F unit 805 is connected to the computer 800 via an interface such as a USB.

[0090] The communication I / F unit 806 is a device for performing data communication via a network such as the Internet, either wired or wirelessly, with a device external to the computer 800. The communication I / F unit 806 may be provided outside the computer 800. When the communication I / F unit 806 is provided outside the computer 800, the communication I / F unit 806 is connected to the computer 800 via an interface such as a USB.

[0091] The display device 807 is a device for displaying various types of information. The display device 807 is, for example, a liquid crystal display, an organic EL (Electro-Luminescence) display, a display of a wearable device, or the like. The display device 807 may be provided outside the computer 800. When the display device 807 is provided outside the computer 800, the display device 807 is connected to the computer 800 via, for example, a display cable. Furthermore, when a touch panel is adopted as the input I / F unit 804, the display device 807 may be configured as an integral part of the input I / F unit 804.

[0092] The above describes one embodiment of the present invention. The policy selection system 100 calculates a set of policies based on multiple training utility values, obtains a predetermined preference value, and selects at least one policy from the multiple policies based on the mismatch degree at the time of selection. This allows the policy selection system 100 to select a policy that is closer to the user's preference.

[0093] Furthermore, the policy selection system 100 can select at least one policy based on a mismatch degree at selection time, which includes a mismatch degree according to the relationship between the multiple objectives indicated by the action values ​​of each of the multiple policies and the relationship between the multiple objectives indicated by a predetermined preference value. Furthermore, the policy selection system 100 can select at least one policy based on a mismatch degree at selection time, which includes a mismatch degree according to the direction of a vector indicated by the action values ​​of each of the multiple policies and the direction of a vector indicated by a predetermined preference value. This allows the policy selection system 100 to select, for example, a policy with a small mismatch degree with a preference value.

[0094] Furthermore, the policy selection system 100 can calculate a set of policies based on the training utility values ​​that are based on the mismatch degree during training, thereby enabling the policy selection system 100 to calculate the Pareto front over a wider area.

[0095] Furthermore, the policy selection system 100 can calculate a set of policies including multiple policies that are Pareto optimal for achieving multiple objectives. This allows the policy selection system 100 to select at least one policy from among the policies that make up the Pareto front.

[0096] Furthermore, the policy selection system 100 can calculate a set of policies based on a mismatch degree during learning, which includes a mismatch degree according to the relationship between multiple objectives indicated by the action values ​​of the learning policies and the relationship between multiple objectives indicated by the learning preference values. Furthermore, the policy selection system 100 can calculate a set of policies based on a mismatch degree during learning, which includes a mismatch degree according to the direction of a vector indicated by the action values ​​of the learning policies and the direction of a vector indicated by the learning preference values. This allows the policy selection system 100 to calculate a Pareto front over a wider area.

[0097] Furthermore, the measure selection system 100 can select at least one of a measure involving acceleration and a measure involving deceleration based on the mismatch degree at the time of selection. This allows the measure selection system 100 to function as, for example, an agent that assists vehicle driving.

[0098] It should be noted that the present embodiment is provided to facilitate understanding of the present invention and is not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit thereof, and equivalents thereof are also included in the present invention.

[0099] In addition, in the present invention, a "unit" does not simply mean a physical means, but also includes cases where the functions of the "unit" are realized by software. Furthermore, the functions of one "unit" or device may be realized by two or more physical means, devices, or software, and the functions of two or more "units" or devices may be realized by one physical means, device, or software. [Explanation of symbols]

[0100] 100 policy selection system, 110 memory unit, 120 policy set calculation unit, 130 preference value acquisition unit, 140 state acquisition unit, 150 policy selection unit, 160 policy output unit, 200 environmental system, 300 user device

Claims

1. On the computer, a policy set calculation unit that calculates a policy set including a plurality of policies corresponding to a plurality of objectives based on a plurality of training utility values ​​calculated according to a plurality of combinations of an action value related to an evaluation value of a predetermined training policy for each of a plurality of objectives and a training preference value related to the priority of each of the plurality of objectives when selecting an action; a preference value acquisition unit that acquires a predetermined preference value; a policy selection unit that selects at least one policy included in the plurality of policies based on a mismatch degree at selection time regarding a mismatch degree between the action value of each of the plurality of policies and the predetermined preference value, the mismatch degree being calculated according to the action value of each of the plurality of policies and the predetermined preference value; A policy selection program to achieve this.

2. 2. The program according to claim 1, wherein the mismatch degree at the time of selection includes a mismatch degree according to a relationship between the plurality of objectives indicated by the behavioral values ​​of each of the plurality of measures and a relationship between the plurality of objectives indicated by the predetermined preference value.

3. 3. The program according to claim 2, wherein the mismatch degree at the time of selection includes a mismatch degree according to the direction of a vector indicated by the action value of each of the plurality of measures and the direction of a vector indicated by the predetermined preference value.

4. the learning utility value is calculated based on a learning mismatch degree regarding a mismatch degree between the action value of the learning policy and the learning preference value; 4. The program according to claim 1, wherein the policy set calculation unit calculates the policy set based on a training utility value based on the degree of mismatch during training.

5. 4. The program according to claim 1, wherein the policy set calculation unit calculates the policy set including a plurality of policies that are Pareto optimal for achieving the plurality of objectives.

6. 5. The program according to claim 4, wherein the learning mismatch degree includes a mismatch degree according to a relationship between the plurality of objectives indicated by the behavioral value of the learning strategy and a relationship between the plurality of objectives indicated by the learning preference value.

7. 7. The program according to claim 6, wherein the mismatch degree during learning includes a mismatch degree according to a direction of a vector indicated by an action value of the learning policy and a direction of a vector indicated by the learning preference value.

8. The preference values ​​include a preference value indicating a priority given to safety and a preference value indicating a priority given to a driving speed of a user of a vehicle, the plurality of strategies include a strategy involving acceleration and a strategy involving deceleration in a vehicle; the measure selection unit selects at least one of the measure involving acceleration and the measure involving deceleration based on the mismatch degree at the time of selection. The program according to claim 1 or 2.

9. a policy set calculation unit that calculates a policy set including a plurality of policies corresponding to a plurality of objectives based on a plurality of training utility values ​​calculated according to a plurality of combinations of an action value related to an evaluation value of a predetermined training policy for each of a plurality of objectives and a training preference value related to the priority of each of the plurality of objectives when selecting an action; a preference value acquisition unit that acquires a predetermined preference value; a policy selection unit that selects at least one policy included in the plurality of policies based on a mismatch degree at selection time regarding a mismatch degree between the action value of each of the plurality of policies and the predetermined preference value, the mismatch degree being calculated according to the action value of each of the plurality of policies and the predetermined preference value; A policy selection system comprising:

10. The computer calculating a policy set including a plurality of policies corresponding to a plurality of objectives based on a plurality of training utility values ​​calculated according to a plurality of combinations of an action value relating to an evaluation value for each of a plurality of objectives of a predetermined training policy and a training preference value relating to the priority of each of the plurality of objectives when selecting an action; Obtaining a predetermined preference value; selecting at least one measure included in the plurality of measures based on a mismatch degree at selection time regarding a mismatch degree between the action value of each of the plurality of measures and the predetermined preference value, the mismatch degree being calculated according to the action value of each of the plurality of measures and the predetermined preference value; Strategy selection method.

Images