Reinforcement learning device, reinforcement learning method, and reinforcement learning program

Abstract

To enable distributed reinforcement learning in consideration of a risk to be taken by an actor and a learner at the time of action selection.SOLUTION: A reinforcement learning device includes: a setting unit configured to set a selection range of a first parameter related to a first risk to be taken when an action to be applied to an analysis target is selected from an action group to a partial range, and set a second parameter related to a second risk to be taken in learning of a value function; an actor configured to select the action on the basis of the value function and the first parameter within the partial range, update a state of the analysis target, and calculate a reward increased as the updated state becomes a new state; a learner configured to update the value function on the basis of the reward and the second parameter; and a determination unit configured to determine, on the basis of a history of the reward calculated when each of a plurality of the first parameters is used, a target output to the actor as a specific first parameter used when the actor selects a specific action of updating the state to the new state.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a reinforcement learning device, a reinforcement learning method, and a reinforcement learning program that executes reinforcement learning. [Background technology]

[0002] In reinforcement learning, which has been developing rapidly in recent years, one model that has shown particularly remarkable progress is distributed reinforcement learning with curiosity (Non-Patent Document 1). In distributed reinforcement learning, learning is performed by a large number of actors that perform trials independently of each other in a certain learning environment, and a learner that aggregates the experiences of these actors and predicts rewards obtained from their actions. Distributed reinforcement learning with curiosity is a learning model that allows for more efficient exploration by giving a large reward to the actor when he or she selects an action that is curious, that is, one that leads to an unknown state.

[0003] In typical reinforcement learning models, including the distributed reinforcement learning of Non-Patent Document 1, learning is designed to increase the expected value of the reward obtained. However, depending on the field in which reinforcement learning is applied, rather than increasing the reward on average, there is a demand for high-risk, high-return or low-risk, low-return learning, in which it is desired to obtain a higher reward even if only once, or to reduce the possibility of obtaining a low reward as much as possible.

[0004] Patent Literature 1 discloses a method for quantifying risk, searching for policies, and deploying automated secure policies. The system of Patent Literature 1 includes one or more computing devices configured to perform operations including controlling deployment of a received policy based at least in part on a quantification of a risk of a tendency to associate a received policy with a contrary deployed policy, the control including predicting values ​​of performance measures of the received policy using reinforcement learning and concentration heterogeneity on deployment data describing the deployment of the deployed policy, quantifying the risk by calculating one or more statistical guarantees regarding the predicted values, and converting the deployed policy into a received policy corresponding to a determination that the one or more statistical guarantees indicate at least a confidence level that the measured values ​​of the performance measures correspond at least to a threshold based at least in part on the performance measures of the deployed policy.

[0005] Patent Document 2 discloses a method for determining an optimal action considering risk for each state of each period of a target period using a computer device. This method includes the steps of: a) selecting one of the states that can be taken in the current period; b) selecting one of the executable action candidates; c) calculating a probability distribution of an evaluation value that depends on a reward obtained by executing one of the selected action candidates and an optimal value in the state of the next period; d) calculating a risk index using the probability distribution of the evaluation value; e) calculating a value index when the action candidate is executed by weighting according to a predetermined preference that considers the risk index; f) repeating steps b) to e) for unselected action candidates; and g) comparing the value indexes to determine one of the action candidates as the optimal action. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] U.S. Patent Publication No. 2016-0148251 [Patent Document 2] JP 2012-068780 A [Non-patent literature]

[0007] [Non-Patent Document 1] Badia, Adria Puigdomenech, et al. "Agent57: Outperforming the atari human benchmark." International Conference on Machine Learning. PMLR, 2020 [Non-Patent Document 2] Mihatsch, O., Neuneier, R. "Risk-Sensitive Reinforcement Learning." Machine Learning 49, 267-290 (2002) Summary of the Invention [Problem to be solved by the invention]

[0008] Patent Documents 1 and 2 each estimate the risk of selecting a certain action and weight the reward according to the risk value, thereby learning in consideration of the risk. The methods of Patent Documents 1 and 2 suppress learning when the risk is high, but cannot learn to aim for a higher reward by imposing a high risk. In addition, they do not describe a method of learning in consideration of risk in an algorithm that determines curiosity in distributed reinforcement learning with curiosity.

[0009] Non-Patent Document 2 proposes that when a learner learns a reward prediction model using each action of an actor, if the action has a higher reward than the reward predicted by the current prediction model, the κ% learning rate is lowered, and if the action has a lower reward, the κ% learning rate is raised. When κ is positive, the learning avoids risk by placing emphasis on cases where the reward is low, and when κ is negative, the learning corresponds to risk-seeking learning that places emphasis on cases where a higher reward than expected is obtained. This document also does not describe a method of learning that takes risk into consideration in the algorithm that determines curiosity in distributed reinforcement learning with curiosity.

[0010] The present invention aims to enable distributed reinforcement learning that takes into account the risks that actors and learners should take when selecting an action. [Means for solving the problem]

[0011] A reinforcement learning device according to one aspect of the invention disclosed in the present application includes a setting unit that sets a selection range of a first parameter related to a first risk to be taken when selecting an action to be applied to an analysis target from a group of actions to a part of the selection range, and sets a second parameter related to a second risk to be taken in learning a value function that calculates a value serving as a guideline for selecting the action; an actor that selects the action based on the value function and the first parameter within the part of the range, updates the state of the analysis target, and calculates a higher reward the more new the updated state is; a learner that updates the value function based on the reward and the second parameter; and a determination unit that determines the first parameter to be output to the actor to a specific first parameter used when the actor selected a specific action that updated the analysis target to a new state, based on a history of the reward calculated by the actor when each of the multiple first parameters within the part of the range was used, and outputs the first parameter to the actor. Effect of the Invention

[0012] According to a representative embodiment of the present invention, distributed reinforcement learning that takes into account the risks that actors and learners should take when selecting an action becomes possible. Problems, configurations, and effects other than those described above will become clear from the following description of the examples. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is an explanatory diagram illustrating an example of execution of a cyber-attack AI according to the first embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a hardware configuration of the reinforcement learning device. [Diagram 3] FIG. 3 is an explanatory diagram showing an example of the analysis target DB. [Figure 4] FIG. 4 is an explanatory diagram illustrating an example of the behavior table. [Diagram 5] FIG. 5 is a block diagram of an example of a circuit configuration of the reinforcement learning circuit according to the first embodiment. [Figure 6] FIG. 6 is an explanatory diagram illustrating an example of a remuneration history table. [Figure 7] FIG. 7 is a block diagram showing an example of the configuration of the learner shown in FIG. [Figure 8] FIG. 8 is an explanatory diagram showing an example of a two-dimensional array. [Figure 9] FIG. 9 is an explanatory diagram showing an example of a state table relating to states. [Figure 10] FIG. 10 is an explanatory diagram showing an example of the operation of the reinforcement learning circuit. [Figure 11] FIG. 11 is an explanatory diagram of an example of an input / output screen displayed on an output device of the reinforcement learning device according to the first embodiment. [Figure 12] FIG. 12 is a flowchart of an example of a data processing procedure according to the first embodiment. [Figure 13] FIG. 13 is a diagram of an example of the analysis target DB according to the second embodiment. [Figure 14] FIG. 14 is a block diagram of an example of a circuit configuration of a reinforcement learning circuit according to the second embodiment. [Figure 15]FIG. 15 is a diagram illustrating an example of an input / output screen displayed on an output device of the reinforcement learning device according to the second embodiment. [Figure 16] FIG. 16 is a flowchart of an example of a data processing procedure according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS EXAMPLES

[0014] Hereinafter, an example of a reinforcement learning device, a data processing method, and a data processing program according to a first embodiment will be described with reference to the accompanying drawings. In the first embodiment, a data group to be analyzed is, for example, a set of connection relationships, vulnerabilities, and statuses of each node in a network consisting of websites, their users, and directory services and repository services used by them, which may be targets of cyber attacks. Note that the network configuration is an example.

[0015] <An example of analysis> 1 is an explanatory diagram showing an example of execution of a cyber-attack AI 100 according to Example 1. The cyber-attack AI (Artificial Intelligence) 100 is an attacker that attacks and occupies nodes indicated by ellipses in a network 106, and includes an actor 101 and a learner 102.

[0016] The actor 101 is an AI that selects an action to be taken by the cyber-attack AI 100 from the action table 105 based on a certain value function and the history of the value obtained so far. In FIG. 1, there is one actor 101, but there may be two or more actors.

[0017] The learner 102 is an AI that updates a value function that determines the value of actions (methods of attacking nodes) that the actor 101 can take, based on the actions that the actor 101 has taken so far, the resulting state of the network 106, and the reward r obtained from that state.

[0018] User U is a person who uses the cyber-attack AI 100, and may be, for example, a doctor, a scholar, a researcher, or an operator who provides an analysis service by the cyber-attack AI 100.

[0019] (1) The cyber-attack AI 100 reads the information in the analysis target DB 104. The analysis target DB 104 stores the properties (status, vulnerability, connection state) of each node in the network 106 to be analyzed.

[0020] (2) User U selects risk parameters 130 that determine the risks of the actor 101 and the learner 102 respectively. The cyber-attack AI 100 reads and sets the risk parameters 130 in the risk parameter table 103 according to the operation of user U. Risk is the uncertainty of the action selected by the actor 101.

[0021] The risk parameter 130 is a combination of an actor risk parameter 131 related to the actor 101 and a learner risk parameter 132 related to the learner 102. The actor risk parameter 131 is a parameter for adjusting the magnitude of the risk to be taken by the actor 101, and is set by a value range [a, b] (0 ≤ a < b ≤ 1) which is a part of the selection range [0.0, 1.0].

[0022] The actor risk parameter 131 may be set by a plurality of value ranges, for example, [a, b], [c, d] (0 ≤ a < b < c < d ≤ 1). In the example of FIG. 1, the actor risk parameter 131 is [0.0, 0.9], and is a discrete value between 0.0 and 0.9. The learner risk parameter 132 is set by a certain value κ (-1 ≤ κ ≤ 1).

[0023] (3) Actor 101 selects a method of attacking a node as an action from action table 105. Actor 101 selects an action that maximizes a value function that introduces, for example, Agent57-type curiosity. Specifically, for example, actor 101 (5) calculates the expected value of the value when each discretized value of curiosity is selected in order to determine the magnitude of curiosity, and (3) adopts the value of curiosity with the largest expected value. Note that actor risk parameter 131 is a parameter that weights this expected value.

[0024] (4) The actor 101 updates the state of the network 106 based on the selected behavior and the vulnerabilities held in the analysis target DB 104, and calculates the reward r. The reward r is, for example, the number of nodes occupied by the cyber-attack AI 100.

[0025] Network 106 is composed of, for example, the status of each node held by DB 104 to be analyzed and the connection relationships between the nodes. The ovals are nodes, and the arrows are edges that indicate the connection relationships between the nodes. "GitHubProject" is an example of a repository service. "Website.Directory" is an example of a directory service. "Website1" is a website that uses "GitHubProject" and "Website.Directory". "client" is a computer that uses "Website1". "user" is a computer within network 106. "User" cannot use "Website1" because it is not connected to "Website1" by an edge.

[0026] The black nodes ("Website1", "client") are nodes that are occupied (owned) by the cyber-attack AI 100. The shaded nodes ("GitHubProject", "Website.Directory") are nodes that have been discovered but not occupied (discovered) by the cyber-attack AI 100. The white nodes ("user") are nodes that have not been discovered (undiscovered) by the cyber-attack AI 100.

[0027] (5) The learner 102 updates the value function based on the updated state of the network 106 and the reward r, for example, by calculation using a neural network. The learner risk parameter 132 is a parameter that weights the updated value of this value function. After this, the cyber-attack AI 100 repeatedly executes (3) to (5).

[0028] (6) The cyber-attack AI 100 may update the risk parameter table based on the state of the network 106 obtained by repeatedly executing (3) to (5) and the obtained reward r, by the operation of the user U. After executing (2), the cyber-attack AI 100 repeatedly executes (3) to (5).

[0029] In this way, the cyber-attack AI 100 selects an attack method from the behavior table 105 based on the risk parameters and updates the state of the network 106.

[0030] <Hardware Configuration Example of Reinforcement Learning Device 200> FIG. 2 is a block diagram showing an example of a hardware configuration of a reinforcement learning device 200. The reinforcement learning device 200 includes a processor 201, a storage device 202, an input device 203, an output device 204, and a communication interface (communication IF) 205. The processor 201, the storage device 202, the input device 203, the output device 204, and the communication IF 205 are connected by a bus 206. The processor 201 controls the reinforcement learning device 200. The storage device 202 serves as a working area for the processor 201. The storage device 202 is a non-transient or temporary recording medium that stores various programs, data, and an analysis target DB. Examples of the storage device 202 include a read only memory (ROM), a random access memory (RAM), a hard disk drive (HDD), and a flash memory. The input device 203 inputs data. Examples of the input device 203 include a keyboard, a mouse, a touch panel, a numeric keypad, and a scanner. The output device 204 outputs data. Examples of the output device 204 include a display and a printer. The communication IF 205 connects to a network and transmits and receives data.

[0031] The reinforcement learning device 200 implements the cyber-attack AI 100 shown in Fig. 1. The cyber-attack AI 100 is realized by causing a processor to execute a program stored in a storage device 202. In addition, in the reinforcement learning device 200, the analysis target DB 104 and the behavior table 105 shown in Fig. 1 are stored in the storage device 202.

[0032] Furthermore, the reinforcement learning device 200 may include a reinforcement learning circuit 207. The reinforcement learning circuit 207 is a circuit configuration that executes learning of a value function. The reinforcement learning circuit 207 executes the processes (1) to (6) shown in FIG. 1 by referring to the action table 105. The action table 105 may be stored in, for example, a storage area (not shown) in the reinforcement learning circuit 207. Note that the reinforcement learning circuit 207 is realized by a circuit configuration, but may also be realized by causing the processor 201 to execute a program stored in the storage device 202.

[0033] <Analysis target DB104> 3 is an explanatory diagram showing an example of the analysis target DB 104. The analysis target DB 104 has, as fields, a node ID 301, a status 302, a vulnerability 303, and a connection relationship 304. A combination of values ​​of each field in the same row becomes the analysis target data of one node.

[0034] The nodes 301 are components of the network 106 to be analyzed, and are represented by, for example, "client", "Website", "Website.Directory", "user", and "GitHubProject".

[0035] The status 302 indicates the state of the node 301 that may be changed by an attack by the cyber-attack AI 100. For example, the status 302 classifies the node 301 into "undiscovered" if the node 301 has not been discovered by the cyber-attack AI 100, "discovered" if the node 301 has been discovered, or "owned" if the node 301 is owned by the cyber-attack AI 100.

[0036] Vulnerability 303 indicates a flaw that may threaten the security of the node through a specific attack. For example, if vulnerability 303 of node ID 301 "client" has "Search web history for list of accessed websites", it means that a connection to "Website1" can be obtained by the action "ScanWebHistory".

[0037] A connection relationship 304 indicates the node to which the node is connected. If the connection relationship 304 of the node ID "client" includes "Website", it means that a connection is possible from "client" (the start point of the arrow) to the "Website node" (the end point of the arrow).

[0038] <Action Table 105> 4 is an explanatory diagram showing an example of the action table 105. The action table 105 is a table that defines, as actions, methods of attacking nodes of the analysis target network 106. The contents of the action table 105 are set in advance.

[0039] The behavior table 105 has fields of a behavior ID 401 and a behavior 402. The behavior ID 401 is identification information that uniquely identifies the behavior 402.

[0040] Actions 402 are attack methods that can be selected by the actor 101. Actions 402 are classified into local attack actions, remote attack actions, and connection actions. A local attack action has a variable of one node (node1) that is the target of a local attack. Specifically, for example, a local attack action is an action that uses node1 as a starting point to discover or possess other nodes. For example, actions 402 with action IDs 401 of "a0" and "a1" correspond to local attack actions.

[0041] A remote attack behavior has two nodes (node1, node2) that are the starting points of a remote attack as variables. Specifically, for example, a remote attack behavior is a behavior that starts from node1 and node2 and discovers or possesses other nodes. For example, behavior 402 with behavior IDs 401 of "a20" and "a21" corresponds to a remote attack behavior.

[0042] A connection action has the connection start point and two target nodes (node1, node2) as variables. Specifically, for example, a connection action connects two nodes (node1, node2) with an edge, or disconnects the connection. For example, the action 402 with the action ID 401 of "a50" corresponds to a connection action.

[0043] The value z of xxxIDyy (xxx is a name, yy is a code, and z is an alphanumeric string) is represented as xxxIDz. For example, the behavior 402 with the behavior ID 401 of "a50" is represented as behavior a50.

[0044] <Example of reinforcement learning circuit configuration> 5 is a block diagram showing an example of a circuit configuration of the reinforcement learning circuit 207 according to Example 1. The reinforcement learning circuit 207 includes a data memory 500, an actor unit 510, a selection unit 520, a learner 102, a risk parameter table 103, and a behavior table 105.

[0045] The data memory 500 stores all data from the storage device 202 to the analysis target DB 104 , that is, the node ID 301 , the status 302 , the vulnerability 303 , and the connection relationship 304 .

[0046] The data memory 500 also stores a history of the discrete values ​​of the actor risk parameter 131 selected by the actor unit 510 so far, and a history of the reward r obtained as a result of the action 402 selected by the actor based on the selected discrete value.

[0047] An actor unit 510 is one or more actors 101. The number of actors 101 can be set by a user U's operation.

[0048] The actor risk parameter 131 corresponds to a curiosity parameter called "arm" in Non-Patent Document 1. Curiosity is generally an interest in unknown matters held by humans, but in this example, it is an element related to the action 402 and reward r for an unknown state of the network 106 specified by the actor 101. The unknown state is a new state that does not exist in the state group in which the network 106 has been updated from the initial state since the actor 101 started the action.

[0049] Specifically, for example, when the state of the network 106 is unknown to the actor 101, when the actor 101 selects an action 402 that leads to the unknown state, the reward r obtained is larger than when the state of the network 106 is known to the actor 101. In this example, the reward r becomes higher as the status 302 changes from "undiscovered" to "discovered" and from "discovered" to "owned." The selection of the action 402 to obtain such reward r is curiosity, and the information that quantifies the curiosity is the curiosity parameter. In this example, the discrete value of the actor risk parameter 131 becomes the curiosity parameter.

[0050] In the following, for convenience, the discrete value (curiosity parameter) of the actor risk parameter 131 may be referred to as an "arm," and the identification information that uniquely identifies the discrete value (curiosity parameter) of the actor risk parameter 131 may be referred to as an "arm ID."

[0051] The selection unit 520 selects an arm ID and determines the discrete value of the actor risk parameter 131 used by the learner 102. The actor risk parameter 131 is an external value function z(t) based on the reward r(t) in the value function Q(t) presented by the learner 102. e An internal value function z(t) that rewards more exploratory actions 402 for (t) i That is, the value function Q(t) is expressed by the following formula (1), where t indicates the time step.

[0052] Q(t)=z e (t) + βz i (t) (1)

[0053] In the above formula (1), β is a discrete value of the actor risk parameter 131. A plurality of arm IDs selectable in the selection unit 520 are set by the operation of the user U. In this example, the number of arm IDs is 32. The selection unit 520 allocates, from the actor risk parameter 131, the discrete values ​​of the actor risk parameter 131 corresponding to the number of set arm IDs as curiosity parameters.

[0054] Specifically, for example, the selection unit 520 includes an extraction unit 521 and a determination unit 513. The extraction unit 521 samples a certain number (for example, 100) of rewards r based on the reward history for each arm ID stored in the data memory 500. This sampling is performed for each actor 101.

[0055] [Reward History Table] 6 is an explanatory diagram showing an example of a reward history table. The reward history table 600 is stored in the data memory 500. A reward history table 600 exists for each actor 101. The reward history table 600 has episode count 601 and reward history 602 as fields.

[0056] The episode count 601 is the number of times an episode has been executed. In reinforcement learning, one episode is executed for time step t=m times (m is an integer equal to or greater than 1) of learning (see FIG. 10). The example in FIG. 6 shows that the upper limit of episodes is 10,000 times. The smaller the value of the episode count 601, the newer the episode is.

[0057] A row in which the value of the episode count 601 is "1" is the latest episode for each arm ID, but even if the episode is in the same row, it does not necessarily mean that it is the same episode between different arm IDs.

[0058] Note that an episode may be executed more than 10,000 times. In this case, the reward history table 600 stores the latest reward history 602 for a maximum of 10,000 times for each arm ID.

[0059] The reward history 602 is a history of reward r for each arm ID. In Fig. 6, as an example, since there are 32 arm IDs, 32 different discrete values ​​of the actor risk parameter 131 are selected from the actor risk parameter 131.

[0060] The arm ID values ​​"1" to "32" are written as arm ID 1 to arm ID 32. When arm ID 1 to arm ID 32 are not to be distinguished, they are simply written as arm IDs.

[0061] The values ​​in each column of Arm ID1 to Arm ID32 are the history of rewards r obtained when the cyber-attack AI 100 takes the action 402 selected using the discrete value of the actor risk parameter 131 specified by that arm ID. Specifically, it is the reward r(t) output at the last time step t in the episode.

[0062] For example, in the column of arm ID1, 10,000 rewards r are stored, which indicates that the discrete value of the actor risk parameter 131 identified by arm ID1 was used to select 10,000 actions 402, and 10,000 rewards r were calculated by the selection of the actions 402. For example, when the episode count 601 is "1", the reward r = 0.21 is the reward r in the latest episode.

[0063] In addition, in the column of arm ID2, three rewards r are stored, which indicates that the discrete value of the actor risk parameter 131 identified by arm ID2 was used to select three actions 402, and three rewards r were calculated by the selection of the actions 402. For example, when the episode count 601 is "1", the reward r = 0.01 indicates that it is the reward r in the latest episode.

[0064] However, the reward r=0.21 when the episode count 601 in arm ID1 is "1" and the reward r=0.01 when the episode count 601 in arm ID2 is "1" are not necessarily the rewards r for the same episode.

[0065] The extraction unit 521 generates probability distributions d1 to d132 of reward r for each arm ID in the reward history 602, and samples a certain number (for example, up to 100) of rewards r from the reward history table 600 for each arm ID based on the generated probability distributions d1 to d132.

[0066] The horizontal axis of probability distributions d1 to d132 (when not distinguishing between them, simply referred to as probability distribution d) indicates reward r in reward history 602 indicated by the column of arm ID. The vertical axis indicates the occurrence probability P of reward r specified in reward history 602.

[0067] For example, the extraction unit 521 assumes that the reward history 602 of each arm follows a normal distribution and estimates the mean and variance of the reward r in the reward history 602. The extraction unit 521 generates a probability distribution d based on the mean and variance of the estimated reward r. The extraction unit 521 randomly samples a certain number of rewards rf1 to rf132 (simply referred to as reward rf when there is no need to distinguish between them) according to the probability distribution d.

[0068] 5, the determination unit 522 selects an arm ID that maximizes the expected value of the reward r based on a certain number of rewards rf1 to rf132 sampled by the extraction unit 521, and outputs the arm ID to the actor unit 510. The selection of the arm ID is performed for each actor 101.

[0069] Specifically, for example, for each of a certain number of rewards rf1 to rf132 (i.e., for each arm ID), the determination unit 522 extracts a group of rewards ranging from the top a percent to the top b percent according to percentiles defined by the value range [a, b] of the actor risk parameter 131 in the risk parameter table 103. The determination unit 522 calculates the expected value of reward r for each arm ID using the group of rewards extracted for each arm ID. The expected value of reward r may be the average value or the median value of the extracted group of rewards.

[0070] a is the lower limit of the actor risk parameter 131, and 0≦a<1. b is the upper limit of the actor risk parameter 131, and 0 <b≦1となる。

[0071] The determination unit 522 identifies an arm ID that has a specific expected value (for example, the maximum value) among the expected values ​​of the reward r calculated for each arm ID from among the set of arm IDs. For example, it is assumed that the value range of the actor risk parameter 131 is set to [0.0, 0.9].

[0072] In this case, the determination unit 522 calculates the expected value of the reward r for each arm ID based on the average value of the reward group ranging from the top 0% reward to the top 90% reward for a certain number of sampled rewards rf. The determination unit 522 determines the arm specified by the arm ID that maximizes the expected value of the reward r as the discrete value of the actor risk parameter 131 used by the actor unit 510.

[0073] In this way, by determining the arm (curiosity), the determination unit 522 can determine the action 402 by ignoring the top 10 percentile of the reward r that can be expected from the past reward history 602 when the value range of the actor risk parameter 131 is [0.0, 0.9]. In other words, by ignoring the top 10 percentile of the reward r that can be obtained by chance, it is possible to make the actor unit 510 more likely to select an action with lower risk.

[0074] Conversely, when the actor risk parameter 131 has a value range of [0.1, 1.0], the actor unit 510 can be made more likely to take actions 402 that are riskier but may yield a higher reward r by ignoring the bottom 10 percentile of rewards r that can be expected based on the reward history 602 to date.

[0075] Instead of using the percentile that determines the reward sample to be considered when calculating the expected value from the sampled reward as the actor risk parameter 131, the percentile that determines the reward sample to be considered when calculating the expected value from the reward history 602 itself may be used as the actor risk parameter 131. Also, the actor risk parameter 131 may be used as the percentile that determines the reward sample to be considered when calculating the expected value from some reward sequence other than the above.

[0076] The actor unit 510 and the learner 102 constitute a part of the cyber-attack AI 100 shown in Fig. 1. The actor unit 510 calculates the value of the value function Q (the value of the action 402) using the curiosity parameter (discrete value of the actor risk parameter 131) of the arm ID selected by the selection unit 520 and the current state of the network 106, and selects the action 402 that maximizes the value of the value function Q.

[0077] Note that there may be a plurality of actors 101 in the actor unit 510. In this case, the same number of selection units 520 may be arranged for each of the plurality of actors 101, or a single selection unit 520 may be arranged for the plurality of actors 101. When there are a plurality of actors 101, the state s(t) of the network 106 is updated for each actor 101. This makes it possible to collect more actions 402 in the same processing time. Also, by making the threshold value eth of the random value e generated by the random unit 703 described later different for each actor 101, it is possible to give each actor 101 individuality.

[0078] Learner 102 is the external value function z e (t) and the internal value function z i This is a value function model in curiosity-driven reinforcement learning that learns two value functions, a(t) and b(t). The learner 102 obtains the action a(t) selected by the actor unit 510 in the state s(t) of the network 106 at the current time step t, the state s(t+1) of the network 106 updated by the action a(t), and the reward r(t+1) obtained from the state s(t+1).

[0079] The learner 102 controls the actor unit 510 and the selection unit 520. Specifically, for example, when the learner 102 receives the state s(t) of the network 106 from the actor unit 510, the learner 102 updates the value function Q that determines the action 402 selected by the actor unit 510.

[0080] [Learner 102] Fig. 7 is a block diagram showing an example of the configuration of the learner 102 shown in Fig. 5. The learner 102 includes a network unit 700, a replay memory 720, and a learning parameter update unit 730. The network unit 700 includes a Q* network 701, a Q network 702, and a random unit 703.

[0081] The Q* network 701 and the Q network 702 are identically configured action-value functions that learn an action a(t) that maximizes value. In this case, value is an index value that represents the magnitude of reward r(t+1) obtained from state s(t+1) of the network 106 updated by taking the action 402 defined by the action a(t).

[0082] Specifically, for example, the Q* network 701 receives the state s(t) of the network 106 as an input, and calculates the extrinsic value function z e The value of (t) and the internal value function z iIt is a deep reinforcement learning DQN (Deep Q-Network) that outputs a two-dimensional array indicating the value of a value function z(t) including the values ​​of z(t).

[0083] [2D array] 8 is an explanatory diagram showing an example of a two-dimensional array. The two-dimensional array 800 is an external value function z e A one-dimensional array 802 showing the value of (t) and an internal value function z i The node configuration 801 is defined by the number of types of nodes in the network 106, which are variables of the behavior 402, and their combinations.

[0084] For example, since the action a0 has one variable, the node configuration 801 has five types of nodes, that is, five combinations. On the other hand, since the action a0 has two variables, the node configuration 801 has ten combinations (5C2).

[0085] 7, the Q network 702 is a deep reinforcement learning DQN with the same configuration as the Q* network 701. The Q network 702 uses a learning parameter θ and calculates the value of each action 402 in the current state s(t) of the network 106 for each node configuration 801 using the above formula (1).

[0086] The random unit 703 outputs a random value e between 0 and 1. The random value e is compared with a threshold eth. For example, if the random value e is greater than or equal to the threshold eth, the actor 101 randomly selects the behavior 402, and if the random value e is less than the threshold eth, the actor 101 selects based on the Q* network 701. The threshold eth can be set to any value by the user U. To eliminate random behavior selection, the threshold eth can be set to 1.

[0087] The learning parameter update unit 730 includes a gradient calculation unit 631. The learning parameter update unit 730 uses the gradient calculation unit 631 to calculate a gradient g that takes into account both the reward r(t) calculated from a state s(t+1) of the network 106 obtained by a certain action a(t) and the learner risk parameter 132 in the risk parameter table 103. The learning parameter update unit 730 updates the learning parameter θ by adding the gradient g to the learning parameter θ.

[0088] The replay memory 720 stores data packs D(t), which include the state s(t), state s(t+1), actions a(t), and rewards r(t) of the network 106 at time step t.

[0089] 7, a specific example of the configuration of the Q* network 701 will be described. A state s(t) is input to the Q* network 701. Here, a specific example of the state s(t) will be described.

[0090] [State Table] 9 is an explanatory diagram showing an example of a state table related to state s(t). A state table 900 has a connection source node 901, a status 902, and connection information 903.

[0091] The originating node 901 is the node 301 connected to the destination node, that is, the node 301 connected to the starting point of the edge indicated by the arrow in the network 106. The status 902 is the status 302 of the originating node 901.

[0092] The status 902 changes when the action 402 taken by the actor 101 is executed. For example, when the action 402 discovers a node 301 that has not been discovered by the cyber-attack AI 100, the status 902 is updated from "undiscovered" to "discovered." When the action 402 causes the node 301 discovered by the cyber-attack AI 100 to be owned, the status 902 is updated from "discovered" to "owned."

[0093] The connection information 903 indicates whether or not there is a connection with a destination node. Specifically, for example, the connection information 903 indicates whether or not there is a connection with the source node 901 by using a flag of "1" or "0" with the node 301 connected to the end of the edge indicated by the arrow in the network 106 as the destination node. "1" indicates a connection and "0" indicates no connection.

[0094] Returning to FIG. 7, the Q* network 701 is e Network 701(1) outputs a one-dimensional array 802 for (t) and an internal value function z i The network 701(2) outputs a one-dimensional array 803 related to (t).

[0095] For example, the first and second layers of the network 701(1) are fully connected networks (number of neurons: 256, activation function: ReLU). The output layer of the network 701(1) is a fully connected network, and outputs a one-dimensional array 802, which uses actions a0 to a50 of the action table 105 as indexes, as an output signal.

[0096] The first layer of network 701(2) is a layer that calculates 51 types of states obtained when each of actions a0 to a50 in action table 105 is performed in state s(t) of network 106. The second and third layers of network 701(2) are fully connected networks (number of neurons: 256, activation function: ReLU) that receive each of the 51 types of states s(t) as input and output 51 types of vectors.

[0097] The output layer of network 701(2) calculates the minimum Euclidean distance between each of the 51 types of vectors and the vectors obtained by inputting the past states s stored in replay memory 620 into the fully connected networks of the second and third layers, and outputs a one-dimensional array 803 in which actions a0 to a50 serve as indices of the 51 types of minimum values.

[0098] [Example of operation of reinforcement learning circuit 207] Fig. 10 is an explanatory diagram showing an example of the operation of the reinforcement learning circuit 207. For ease of understanding, Fig. 10 shows an example of the operation from time step t=0 to t=2 in a certain episode.

[0099] (Step S1000: t=0) The determination unit 522 in the selection unit 520 selects one of the discrete values ​​dv from the 32 discrete values ​​dv1 to dv32 (simply the discrete value dv when no distinction is made between them) extracted from the value range [0.0, 0.9] of the actor risk parameter 131. In FIG. 10, it is assumed that the discrete value dv1 is selected. The discrete values ​​dv1 to dv32 correspond to arm ID1 to arm ID32, respectively.

[0100] (Step S1001: t=0) In step S1001, it is assumed that the random value e is less than the threshold value eth. The actor 101 calculates the two-dimensional array 800 in the arm ID1 corresponding to the discrete value dv1. The actor 101 calculates the z e (t=0) is calculated by z e (t) and assign it to z of the one-dimensional array 802. i (t=0) is calculated by z i For example, in the case of the node configuration 801 “client” of the action a0, the actor 101 assigns “1.53” to z e Substitute "0.60" for (t) and z i Substitute into (t).

[0101] If the discrete value dv1 = 0.1, the value function Q(t = 0) in this case is 1.53+0.1×0.60=1.59 The actor 101 executes such a calculation for all node configurations 801 of actions a0 to a50 in arm ID1.

[0102] Q(x, yz) in FIG. 10 indicates the value of the value function. x indicates the node ID. y is a numeric number in the action ID 401 (y=0 for a0). z is a number that uniquely indicates the node configuration 801 in the action ID 401. For example, Q(1, 0-2) indicates the value in "Website1", which is the second node configuration 801 in action a0 in node ID1.

[0103] (S1002:t=0) Actor 101 selects action 402 that provides the maximum value from the calculated value function values ​​Q(1,0-1) to Q(1,50-10). In this example, action a0 corresponding to Q(1,0-2) is selected.

[0104] (Step S1003: t=0) The actor 101 updates the state s(t=0) of the network 106 to state s(t=1) by executing the selected action a0 (attack method).

[0105] (Step S1004: t=0) Actor 101 calculates reward r(t=0) based on state s(t=1).

[0106] (Step S1005: t=0) The learner 102 updates the value function Q(t=0) to a value function Q(t=1).

[0107] (Step S1006: t=0) As shown in FIG. 6, the extraction section 521 of the selection unit 520 randomly samples a certain number of rewards rf1 to rf132 from the reward history table 600 according to the probability distribution d.

[0108] (Step S1007: t=0) The determination unit 522 of the selection unit 520 calculates an expected value of reward r for each of the fixed number of rewards rf1 to rf132 sampled by the extraction unit 521. The determination unit 522 identifies the arm ID that is the sampling source of the fixed number of rewards rf from which the maximum expected value is calculated, among the expected values ​​of reward r calculated for each of the fixed number of rewards rf1 to rf132.

[0109] (Step S1008: t=0) The determination unit 522 of the selection unit 520 identifies the discrete value dv in the actor risk parameter 131 corresponding to the arm ID identified in step S1007. Here, the identified discrete value dv does not change until the episode ends. In this example, it is assumed that the discrete value dv32 is identified.

[0110] (Step S1009: t=0) The determination section 522 of the selection unit 520 outputs the discrete value dv32 identified in step S1008 to the actor 101.

[0111] (Step S1011: t=1) In step S1011, it is assumed that the random value e is less than the threshold value eth. The actor 101 calculates the two-dimensional array 800 in the arm ID32 corresponding to the discrete value dv32, in the same manner as in step S1001.

[0112] (Step S1012: t=1) Actor 101 selects action 402 that provides the maximum value from the calculated value function values ​​Q(32,0-1) to Q(32,50-10). In this example, action a50 corresponding to Q(32,50-10) is selected.

[0113] (Step S1013: t=1) The cyber-attack AI 100 updates the state s(t=1) of the network 106 to state s(t=2) by executing the selected action a50 (attack method).

[0114] (Step S1014: t=1) The cyber-attack AI 100 calculates the reward r(t=1) based on the state s(t=2).

[0115] (Step S1015: t=1) The learner 102 updates the value function Q(t=1) to a value function Q(t=2).

[0116] (Step S1019: t=1) The determination section 522 of the selection unit 520 outputs the discrete value dv32 identified in step S1008 to the actor 101.

[0117] (Step S1021: t=2) In step S1021, it is assumed that the random value e is less than the threshold value eth. The actor 101 calculates the two-dimensional array 800 in the arm ID32 corresponding to the discrete value dv32, in the same manner as in step S1011.

[0118] (Step S1022: t=2) Actor 101 selects action 402 that has the maximum value from the calculated value function values ​​Q(32,0-1) to Q(32,50-10). In this example, action a0 corresponding to Q(32,0-3) is selected. Thereafter, similar actions are executed until the episode ends.

[0119] <Example of input / output screen> 11 is a diagram illustrating an example of an input / output screen displayed on the output device 204 of the reinforcement learning device 200 according to the first embodiment. The input / output screen 1100 includes a load button 1110, a number of actors input area 1120, a number of arms input area 1130, a risk parameter input area 1140, a reward input area 1150, a start button 1160, a network display area 1170, and a stop / resume button 1180.

[0120] The load button 1110 is a user interface for loading entries of the analysis target DB 104 into the data memory 500 when pressed by the user U.

[0121] The actor number input area 1120 is an area for receiving input of the number of actors 101 to be placed. The arm number input area 1130 is an area for receiving input of the number of arms in the selection unit 520, that is, the number of discrete values ​​of the actor risk parameter.

[0122] The risk parameter input area 1140 includes an input area 1141 for the actor risk parameter 131 and an input area 1142 for the learner risk parameter 132. The reward selection area 1150 is an area for accepting the selection of the type of reward r(t) to be given for the state s(t) of the obtained network 106. Specifically, for example, the number of occupied nodes (status information is owned) can be selected. As a result, as described above, the reward r is calculated as the number of occupied nodes. Note that the selectable types of reward r may be other types such as the percentage of the number of occupied nodes.

[0123] The start button 1160 is a user interface that, when pressed by the user U, starts the operation of the cyber-attack AI 100 using the number of actors, number of arms, risk parameters, and rewards set in the number of actors input area 1120, number of arms input area 1130, risk parameter input area 1140, and reward input area 1150 in the network 106 loaded by the load button 1110.

[0124] The network display area 1170 displays the network 106 updated by the actions 402 of the actor 101. The network 106 displayed in the network display area 1170 may be the state table 900 or a graph network diagram.

[0125] The stop / resume button 1180 includes a stop button 1181 and a resume button 1182. The stop button 1181 is a user interface for temporarily suspending the operation of the cyber-attack AI 100 when pressed by the user U. The resume button 1182 is a user interface for temporarily suspending the operation of the cyber-attack AI 100 that was paused by the stop button 1181 when pressed by the user U.

[0126] The input / output screen 1100 is displayed, for example, on a display, which is an example of the output device 204 of the reinforcement learning device 200. The input / output screen 1100 may be displayed on a display of another computer that is communicatively connected to the communication IF 205 by transmitting information about the input / output screen 1100 from the communication IF 205 of the reinforcement learning device 200 to the other computer.

[0127] <Example of data processing procedure> 12 is a flowchart illustrating an example of a data processing procedure according to the embodiment 1. It is assumed that before the start of the processing, the entries of the analysis target DB 104 are loaded into the data memory 500 by pressing the load button 1110 on the input / output screen 1100 in FIG.

[0128] (Step S1201) The reinforcement learning device 200 executes initialization. Specifically, for example, the reinforcement learning device 200 sets the calculation step m to m=1. The reinforcement learning device 200 also initializes the learning parameter θ* of the Q* network 701 with a random weight. The reinforcement learning device 200 also initializes the learning parameter θ of the Q network 702 with a random weight. The reinforcement learning device 200 also sets all cells of the reward history 602 to blank, the time step t to t=0, and the state s(t) to the state s(t=0) of the initial network 106 held in the analysis target DB 104.

[0129] (Step S1202) The reinforcement learning device 200 sets a risk parameter table 103 in which the value of the risk parameter 130 input by the user U is stored.

[0130] (Step S1203) The learner 102 determines the two-dimensional array 800 by inputting the state s (t=0) to the Q* network 701. The learner 102 also generates a random value e by the random unit 603.

[0131] (Step S1204) 10, the selection unit 520 specifies an arm ID that maximizes the expected value of the reward r based on the actor risk parameter 131 and the reward history 602 stored in the data memory 500, and outputs the discrete value dv of the actor risk parameter 131 corresponding to the specified arm ID. Note that at time step t=0, the selection unit 520 selects a random discrete value dv from the value range of the actor risk parameter 131 by the determination unit 522, and outputs it to the actor 101.

[0132] (Step S1205) The actor 101 obtains a random value e generated by the random unit 703 of the learner 102. Then, the actor 101 determines a selection policy (random or selection based on the two-dimensional array 800) for the behavior 402 based on the random value e, and selects the behavior 402 according to the determined selection policy.

[0133] Specifically, for example, if the random value e is greater than or equal to a threshold value eth (e.g., eth=0.5), the actor 101 randomly selects one behavior from the behavior table 105, and randomly selects the number of target nodes required for the node that is the target of the selected behavior 402.

[0134] For example, if the action 402 randomly selected from the action table 105 is "CredScanBashHistory(node1)" of action a1, one node is randomly selected as node1 from the network 106. For example, if "Website.Directory" is selected as node1, the actor 101 will adopt "CredScanBashHistory(Website.Directory)" as the action 402 with node1 specified.

[0135] On the other hand, if the random number value output by the random unit 703 is less than eth, the actor 101 inputs the state s(t) to the Q* network 701 in the network unit 700 and calculates the two-dimensional array 800.

[0136] The actor 101 calculates the external value function z e The value of the one-dimensional array 802 indicating the value of (t) and the internal value function z i The value of the one-dimensional array 803 indicating the value of (t) is substituted into β, and the discrete value dv (dv1 in the example of FIG. 10) of the actor risk parameter 131 corresponding to the arm selected by the selection unit 520 is substituted into β.

[0137] For example, when the discrete value dv1 = 0.5, the maximum value of the value function Q(t = 0) is the external value function z e The value of (t=0) "3.88" and the internal value function z i The sum of the value "0.02" at (t=0) and the value "0.01" obtained by multiplying the discrete value dv1=0.5 is "3.89".

[0138] In the behavior table 105, the behavior 402 with the behavior ID a0 is "SearchEdgeHistory(node1)". The actor 101 selects "SearchEdgeHistory(Website1)" which has the maximum value. In this way, the actor 101 can select a more valuable behavior.

[0139] (Step S1206) 10, the actor 101 executes an update to the state s(t+1) of the network 106 after performing the action 402 selected in step S1205 at time step t, and a calculation of the reward r(t). Specifically, for example, the actor 101 updates the status 902 and the connection information 903 in the state table 900 based on the action 402 selected in step S1205.

[0140] Specifically, for example, the actor 101 calculates the number of nodes whose status 902 is “owened” in the state table 900 in the updated state s(t+1) as the reward r(t). The actor 101 stores the reward r(t) in the data memory 500 and outputs it to the learner 102.

[0141] The reward r(t) is not limited to the number of nodes whose status 902 is "owened". For example, points may be given according to the status 902, and the total may be the reward r(t). For example, by setting the points higher as the state transitions to a new state that is more effective in attack, such as 0 points for "undiscovered" and 1 point for "discovered", and 3 points for "owned", the reward r(t) can be increased when an unknown state s(t+1) is obtained. The types of reward r can also be selected in the reward selection area 1150 described above.

[0142] (Step S1207) Next, the network unit 700 stores in the replay memory 720 a data pack D(t) that is a set of data including the reward r(t), the reward history 602, and the states s(t) and s(t+1) of the network 106.

[0143] Furthermore, the reinforcement learning device 200 displays the updated state s(t+1) of the network 106. Specifically, for example, the reinforcement learning device 200 loads the data pack D(t) stored in the storage device 202, draws a graph network diagram using the status 902 and the connection information 903 in the state s(t+1) of the network 106 in the data pack D(t), and displays it in the network display area 1170.

[0144] Furthermore, the reinforcement learning device 200 displays the reward r(t) in the data pack D(t) in the network display area 1170. If the data pack D(t) is not stored in the storage device 202, the reinforcement learning device 200 may display an analysis result indicating a failure of the analysis.

[0145] (Step S1208) If the time step t is non-zero and the remainder of t / T is zero for a given period T (T is an integer equal to or greater than 1), the reinforcement learning device 200 proceeds to step S1209. If not (step S1208: No), the time step t is set to t=t+1, and the process returns to step S1203. This makes it possible to execute learning in steps S1209 and S1210 for the time step t of the given period T.

[0146] (Step S1209) The learning parameter update unit 730 randomly loads J data packs D(j) (j=1, ..., J) (hereinafter, referred to as data pack group Ds) from the replay memory 720, and updates the teacher signal y(j) according to the following formula (2). The teacher signal y(j) is a predicted value of the value of the action 402. In the first embodiment, the upper limit of J is set to 100.

[0147]

number

[0148] In the above formula (2), r(j) on the right side is the reward in the j-th data pack D(j). Also, γ on the right side is the discount rate, which is set to a fixed value of γ=0.998 in the first embodiment. Also, maxQ(s(j+1);θ) on the right side is a calculation process that calculates the maximum value of the value function Q(s(j+1);θ).

[0149] The value function Q(s(j+1);θ) indicates the value of an action in state s(j+1) on the condition that a learning parameter θ is applied. Specifically, for example, the network unit 700 inputs state s(j+1) to the Q network 702, which applies the learning parameter θ to calculate a two-dimensional array 800 for t=j. The network unit 700 calculates the external value function z e The value of (s(j+1);θ) and the internal value function z iThe value of the action 402, which is the sum of the value of (s(j+1);θ) weighted by the curiosity value β (discrete value dv), is calculated for each node configuration 801.

[0150]

number

[0151] The external value function z on the right side of the above equation (3) e (s(j+1);θ) is the extrinsic value function z in state s(j+1) subject to the application of the learning parameter θ. e The value of the inner value function z on the right side of the above formula (3) is shown. i (s(j+1);θ) is the internal value function z in state s(j+1) subject to the application of the learning parameter θ. i Indicates the value of (j+1).

[0152] For example, the external value function z e The value of (j) “3.88” and the internal value function z i If the value of Q(s(j+1);θ) "3.89" obtained from the value of (j) "0.02" is greater than the value of Q(s(j+1);θ) of any other node configuration 801, the calculation process maxQ(s(j+1);θ) outputs the value "3.89" indicated by the action value of action a0 in the node configuration 801 "Website1". In this way, the learning parameter update unit 730 updates the teacher signal y(j).

[0153] (Step S1210) Next, the learning parameter update unit 730 executes the learning calculation. Specifically, for example, the gradient calculation unit 631 outputs a gradient g for the learning parameter θ using the following formula (4), and updates the learning parameter θ by adding the gradient g to the learning parameter θ.

[0154]

number

[0155] In the above formula (4), κ is the learner risk parameter 132. When κ is 0<κ≦1, learning is performed with a smaller learning rate (1-κ) or (1+κ) when the teacher signal y(j) is larger than the value Q(s(j);θ) predicted by the Q* network 701, and a larger learning rate is used in the opposite case. In other words, this is low-risk, low-return learning that emphasizes risk avoidance, in that no learning is performed when a value higher than expected is obtained, and strong learning is performed when the value is lower than expected.

[0156] Conversely, when κ is -1≦κ<0, learning is not performed when the value is lower than expected, and learning is performed strongly when the value is higher than expected, resulting in high-risk, high-return learning that emphasizes taking risks and aiming for higher value.

[0157] The second term on the right-hand side of the above formula (4) is the gradient g of the value of the action 402. This allows the Q network 702 to show actions 402 that increase the reward, for example, the number of nodes whose "owned" status 902 is "owened", by using the updated learning parameter θ that takes into account the reward r(j).

[0158] In addition, in the learning calculation (step S1210), the learning parameter update unit 730 overwrites the updated learning parameter θ of the Q network 702 to the learning parameter θ* of the Q* network 701. That is, the learning parameter θ* becomes the same value as the updated learning parameter θ. This enables the Q* network 701 to identify the action 402 that is expected to increase the prediction accuracy of the action value, i.e., the objective variable.

[0159] (Step S1211) The reinforcement learning device 200 determines whether or not at least one of the risk parameter table 103 and the analysis target DB 104 has been changed (step S1211). If there is no change in either the risk parameter table 103 or the analysis target DB 104 (step S1211:1), the time step t is updated to t=t+1, and the process proceeds to step S1203.

[0160] If only the risk parameter table 103 out of the risk parameter table 103 and the analysis target DB 104 has been changed (step S1211:2), the time step t is updated to t=t+1, and the process proceeds to step S1202. For example, this corresponds to a case where the stop button 1101 on the input / output screen 1100 is pressed, a value is re-entered in the risk parameter input area 1140, and the resume button 1182 is pressed.

[0161] When only the analysis target DB 104 out of the risk parameter table 103 and the analysis target DB 104 has been changed (step S1211:3), the process proceeds to step S1201. For example, this corresponds to a case where the stop button 1181 is pressed on the input / output screen 1100, the load button 1110 is pressed again, and the resume button 1182 is pressed.

[0162] When the time step t reaches the upper limit, the episode ends, and the process for the next episode is restarted from step S1201. When the number of episodes reaches the upper limit, the data processing ends.

[0163] In this way, according to the first embodiment, it is possible to automatically obtain a cyber attack that maximizes a certain reward r against a certain specified network 106 after specifying the risk to be taken. EXAMPLES

[0164] Next, a second embodiment will be described. In the first embodiment, the cyber-attack AI 100 that is responsible for the cyber-attacks has been described. In contrast to this, in the second embodiment, an example including a defense AI independent of the cyber-attack AI 100 in addition to the cyber-attack AI 100 will be described. The defense AI has actors and learners, just like the cyber-attack AI 100. The defense AI is an AI that defends nodes in the network 106 so that the status 902 of the node does not become "owened", and takes back nodes whose status 902 has become "owened" (changing the status from "owened" to "discovered"). In the second embodiment, the differences from the first embodiment will be mainly described, so that the same components as those in the first embodiment are given the same reference numerals and their description will be omitted.

[0165] <Database to be analyzed> 13 is an explanatory diagram illustrating an example of an analysis target DB according to the embodiment 2. The analysis target DB 1300 has a node ID 301, an attacking side status 1301, a defending side status 1302, a vulnerability 303, an attacking side connection relationship 1303, and a defending side connection relationship 1304. A combination of values ​​of each field in the same row becomes analysis target data of one node.

[0166] The attacker status 1301 is the status 302 of the node as viewed from the cyber-attack AI 100. Specifically, for example, the attacker status 1301 indicates the state of the node 301 that may be changed by an attack by the cyber-attack AI 100.

[0167] The defending side status 1302 is the status 302 of the node as seen from the defending AI. Specifically, for example, the defending side status 1302 indicates the state of the node 301 that may be changed due to the defense of the defending AI.

[0168] The attacking side connection relationship 1303 indicates a connected node of the node from the viewpoint of the cyber attack AI 100. The defending side connection relationship 1304 indicates a connected node of the node from the viewpoint of the defending AI.

[0169] <Reinforcement learning circuit> 14 is a block diagram showing an example of a circuit configuration of the reinforcement learning circuit 207 according to the second embodiment. In the second embodiment, the reinforcement learning circuit 207 has an attacking circuit configuration 1400A and a defending circuit configuration 1400D. The attacking circuit configuration 1400A is a circuit configuration showing one cyber-attack AI 100. When multiple cyber-attack AIs 100 are arranged, multiple attacking circuit configurations 1400A are provided.

[0170] The defense circuit configuration 1400D is a circuit configuration representing one defense AI. If multiple defense AIs are deployed, multiple defense circuit configurations 1400D are provided. The attacking circuit configuration 1400A and the defending circuit configuration 1400D each have the configuration shown in FIG.

[0171] 5 with "A" at the end is the configuration of the attacking circuit configuration 1400A, and "attacking" is added to the beginning of the name of the configuration. For example, the selection unit 520 with "A" at the end, 520A, is referred to as the attacking selection unit 520A.

[0172] 5 with "D" at the end are configurations on the defending circuit configuration 1400D side, and "Defensive" is added to the beginning of the configuration name. For example, the selection unit 520 with "D" at the end, 520D, is referred to as the defending selection unit 520D.

[0173] When there is no distinction between the attacking side and the defending side, the "A" and "D" at the end of the reference numerals and the "attacking side" and "defensive side" at the beginning are omitted. The data memory 500 is shared by the attacking side circuit configuration 1400A and the defending side circuit configuration 1400D.

[0174] The difference between the attacking circuit configuration 1400A and the defending circuit configuration 1400D is in the action table 105. Specifically, for example, the attacking action table 105A specifies actions 402 required for the attacking circuit configuration 1400A, which is the cyber-attack AI 100, to attack a node in the network 106, as shown in FIG.

[0175] On the other hand, the defending side action table 105D specifies actions 402 required for the defending side circuit configuration 1400D, which is a defense AI, to defend nodes in the network 106. Specifically, for example, the actions 402 in the defending side action table 105D are defense methods selectable by the defending side actor unit 510D. The actions 402 are classified into local defense actions, remote defense actions, and connection actions.

[0176] A local defense action takes a node (node1) that is the target of local defense as a variable. Specifically, for example, a local defense action is an action that starts from node1 and hides other nodes ("discovered" → "undiscovered") or seizes owned nodes ("owned" → "discovered").

[0177] Remote defense actions have two nodes (node1, node2) as variables, which are the starting points of remote defense. Specifically, for example, remote attack actions are actions that start from node1 and node2 and hide other nodes ("discovered" → "undiscovered") or seize owned nodes ("owned" → "discovered").

[0178] That is, the defending side action table 105D includes an action 402 that opposes the action 402 in the attacking side action table 105A.

[0179] The attacking circuit configuration 1400A and the defending circuit configuration 1400D share only the actions 402 they have performed and the resulting state s(t) of the network 106 in the data memory 500. Specifically, for example, the attacking circuit configuration 1400A and the defending circuit configuration 1400D share the state table 900 shown in Fig. 9. Here, the state s(t) of the network 106 updated by the attacking circuit configuration 1400A is defined as state s(t)A, and the state s(t) of the network 106 updated by the defending circuit configuration 1400D is defined as state s(t)D.

[0180] For example, at a certain time step t, if the attacking circuit configuration 1400A updates state s(t)A to state s(t+1)A, the defending circuit configuration 1400D updates the state s(t+1)A updated by the attacking circuit configuration 1400A to state s(t+1)D, taking the state s(t+1)A updated by the attacking circuit configuration 1400A as state s(t)D. Then, the attacking circuit configuration 1400A updates state s(t+1)A to state s(t+1)D.

[0181] Here, we have explained an example of updating the state s(t) such that at the same time step t, the attacking circuit configuration 1400A attacks first and the defending circuit configuration 1400D then defends, but it is also possible to update the state s(t) such that the defending circuit configuration 1400D defends first and the attacking circuit configuration 1400A then attacks.

[0182] In addition, if it is not specified which will attack or defend first, and if the attacking circuit configuration 1400A and the defending circuit configuration 1400D select an action 402 that causes a conflict, neither action 402 will be adopted, and the state s(t) may not change.

[0183] On the other hand, the respective behavior tables 105, the earned rewards r, the reward history tables 600, and the data packs D(t) stored in the replay memory 720 by the network units 700 in the learner 102 are not shared.

[0184] The reward r(t) in the attacking circuit configuration 1400A is as shown in Example 1. On the other hand, the reward r(t) in the defending circuit configuration 1400D is, for example, a value obtained by subtracting the number of nodes whose status 902 is “owened” from the total number of nodes in the network 106.

[0185] In addition, the number of nodes whose status 902 is "owened" is not limited. For example, points may be given according to the status 902, and the total may be set as the reward r(t). For example, the points are set lower as the state transitions to a new state that is more effective for attack, such as 3 points for "undiscovered," 1 point for "discovered," and 0 points for "owned."

[0186] In this way, the reward r(t) in the defender circuit configuration 1400D becomes lower as the state s(t+1) becomes unknown.

[0187] In addition, the attacking side circuit configuration 1400A and the defending side circuit configuration 1400D may share the time step t and simultaneously and in parallel execute processing similar to that of Figure 12, or while either the attacking side circuit configuration 1400A or the defending side circuit configuration 1400D advances one time step (steps S1203 to S1207), the other unit may repeat steps S1203 to S1207 multiple times.

[0188] <Example of input / output screen> 15 is an explanatory diagram showing an example of an input / output screen displayed on the output device 204 of the reinforcement learning device 200 according to the second embodiment. The input / output screen 1500 is divided into configurations that are applied to the attacking side circuit configuration 1400A and the defending side circuit configuration 1400D in the configuration of the input / output screen 1100.

[0189] 11 with "A" at the end are configurations used for execution in the attacking circuit configuration 1400A, and "attacking" is added to the beginning of the configuration name. For example, the input area 1141 with "A" at the end, 1141A, is referred to as the attacking input area 1141A.

[0190] 11 are configurations used for execution in the defending circuit configuration 1400D, and the name of the configuration is preceded by "Defensive side." For example, the input area 1141 with the suffix "D" 1141D is referred to as the defending side input area 1141D.

[0191] When no distinction is made between attacking and defending teams, the "A" and "D" suffixes and the "attacking" and "defending" suffixes are omitted.

[0192] The input / output screen 1500 has an input area 1501 for the number of attackers and the number of defenders. The input area 1501 for the number of attackers and the number of defenders has an attacker number input area 1511 and a defender number input area 1512. The attacker number input area 1511 is an area that accepts input of the number of attackers through operation by the user U. The number of attackers is the number of cyber attack AIs 100. The defender number input area 1512 is an area that accepts input of the number of defenders through operation by the user U. The defender number is the number of defense AIs.

[0193] In addition, the network display area 1170 displays the reward r obtained for each of the attacker (cyber attack AI 100) and the defender (defense AI).

[0194] 15, in the attacker risk parameter input area 1140A, "0.1:1.0" is input into the input area 1141A of the attacker actor risk parameter 131. Thus, by ignoring the bottom 10 percentile of the sampled rewards r, the cyber-attack AI 100 causes the attacker actor unit 510A to perform actions 402 that are riskier but may yield a higher reward r.

[0195] Also, "-0.5" is entered in the input field 1142A of the attacking learner risk parameter 132. Therefore, the cyber-attack AI 100 causes the attacking learner 102A to perform high-risk, high-return learning that emphasizes taking risks and aiming for higher value, by not learning when the value is lower than expected and learning strongly when the value is higher than expected.

[0196] 15, in the defender risk parameter input area 1140D, "0.0:0.9" is input in the input area 1141D of the defender actor risk parameter 131. Therefore, the defense AI can cause the defender actor unit 510D to select a lower-risk action by ignoring the top 10 percentile reward r that can be obtained by chance.

[0197] Also, "0.5" is input in the input field 1142D of the defending learner risk parameter 132. Therefore, the defense AI causes the defending learner 102D to execute a low-risk, low-return type that emphasizes risk avoidance, in which learning is not performed when a value higher than expected is obtained, and learning is performed intensively when a value lower than expected is obtained.

[0198] Next, an example will be described in which the attacking side circuit configuration 1400A and the defending side circuit configuration 1400D share the time step t and simultaneously and in parallel execute the same processing as in FIG.

[0199] <Example of data processing procedure> Fig. 16 is a flowchart illustrating an example of a data processing procedure according to the embodiment 2. It is assumed that before the start of the processing, the entries of the analysis target DB 104 are loaded into the data memory 500 by pressing the load button 1110 on the input / output screen 1100 in Fig. 16.

[0200] 16, the steps with "A" at the end are processes executed by the attacking circuit configuration 1400A, and "attack" is added to the beginning of the step name. For example, step S1203A with "A" added to the end of learner execution (step S1203) is called attacking learner execution.

[0201] 16 are processes executed by the defending circuit configuration 1400D, and the name of the step is preceded by "Defensive side." For example, step S1203D, which has "D" added to the end of learner execution (step S1203), is referred to as defensive learner execution.

[0202] In the second embodiment, in order to share the state s(t) between the attacking circuit configuration 1400A and the defending circuit configuration 1400D, in step S1606 of FIG. 16, as described above, at the same time step t, either one of the attacking circuit configuration 1400A or the defending circuit configuration 1400D (for example, the attacking circuit configuration 1400A) first updates the state s(t)A to state s(t+1)A, and the defending circuit configuration 1400 updates the updated state s(t+1)A to state s(t)D, and the attacking circuit configuration 1400A sets the updated state s(t+1)D to the updated state s(t+1)A. This allows the state to be shared between the attacking circuit configuration 1400A and the defending circuit configuration 1400D.

[0203] In addition, the calculation of reward r(t) is counted by the number of "owned" in state table 900 at the updated state s(t+1).

[0204] In this way, according to the second embodiment, it is possible to automatically determine a cyber attack that maximizes the attacker's reward r and a defense that maximizes the defender's reward r for a specified network 106, after specifying the risks to be taken.

[0205] In the second embodiment, a configuration in which one attacking circuit configuration 1400A and one defending circuit configuration 1400D are arranged has been described, but one or more attacking circuit configurations 1400A and one or more defending circuit configurations 1400D may be arranged. Also, the configuration may be made up of only multiple attacking circuit configurations 1400A.

[0206] In addition, in the first and second embodiments, the network 106 is taken as an example of an attack target or a defense target, but the attack target or defense target is not limited to such a cyber attack or defense against it. For example, it can be applied to stock trading in a certain market, and the analysis target DB 104, 1300 may be a data set with company IDs and company information such as the company's stock price, net profit, number of employees, and sales as a variable set, the action table 105 may be stock buying and selling, the target of the action in the action table 105 may be each company, and the state s(t) may be the company information in the analysis target DB 104, 1300. In addition, in the first embodiment, the reward r(t) may be the total profit obtained by the user. In addition, in the second embodiment, the defender and the attacker may be multiple competitive stock traders.

[0207] As described above, the reinforcement learning device 200 according to this embodiment adjusts the learner risk parameter 132, which is the risk that the learner 102 should take in learning the value function that serves as a guideline for selecting the action 402, and the actor risk parameter 131, which is the risk that the actor 101 should take when selecting the arm to be adopted when selecting the action 402, i.e., the value of the discretized curiosity parameter, in distributed reinforcement learning with curiosity. Therefore, high-risk high-return type or low-risk low-return type learning that takes both risks into consideration is possible.

[0208] The present invention is not limited to the above-described embodiments, and includes various modified examples and equivalent configurations within the spirit of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those having all of the configurations described. Also, a part of the configuration of one embodiment may be replaced with a configuration of another embodiment. Also, a configuration of another embodiment may be added to a configuration of one embodiment. Also, a part of the configuration of each embodiment may be added, deleted, or replaced with another configuration.

[0209] Furthermore, each of the aforementioned configurations, functions, processing units, processing means, etc. may be realized in hardware, for example by designing some or all of them as an integrated circuit, or may be realized in software by a processor interpreting and executing a program that realizes each function.

[0210] Information such as programs, tables, files, etc. that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC (Integrated Circuit) card, an SD card, or a DVD (Digital Versatile Disc).

[0211] In addition, the control lines and information lines shown are those considered necessary for the explanation, and do not necessarily show all the control lines and information lines necessary for implementation. In reality, it can be considered that almost all components are connected to each other. [Explanation of symbols]

[0212] 103 Risk Parameter Table 105 Behavior Table 106 Network 131 Actor Risk Parameter 132 Learner Risk Parameter 200 Reinforcement Learning Device 402 Action 510 Actor Unit 520 Selection Units 521 Extraction part 522 Decision Section 600 Reward History Table 700 Network Unit 800 2D Array 900 Status Table 100 Cyber ​​Attack AI 104 Analysis target database 1300 Analysis target DB 1400A Attacker circuit configuration 1400D Defence circuit configuration

Claims

1. a setting unit that sets a selection range of a first parameter related to a first risk to be taken when selecting an action to be applied to an analysis target from a group of actions to a part of the selection range, and sets a second parameter related to a second risk to be taken in learning a value function that calculates a value serving as a guideline for selecting the action; an actor that selects the action based on the value function and the first parameter within the range, updates the state of the analysis target, and calculates a higher reward as the updated state becomes newer; a learner that updates the value function based on the reward and the second parameter; a determination unit that determines, based on a history of the rewards calculated by the actor when each of the plurality of first parameters within the partial range is used, the first parameter to be output to the actor as a specific first parameter used when the actor selected a specific action that updated the analysis target to a new state, and outputs the first parameter to the actor; A reinforcement learning device comprising:

2. The reinforcement learning device according to claim 1 , the determination unit calculates an expected value of the reward for each history of the reward for each of the plurality of first parameters within the partial range, and determines the specific first parameter to be used for selecting a next action based on the expected value of the reward for the first parameter; A reinforcement learning device characterized by:

3. The reinforcement learning device according to claim 2, The determination unit determines the first parameter within the partial range in which the expected value of the reward is maximized as the specific parameter. A reinforcement learning device characterized by:

4. The reinforcement learning device according to claim 1 , the lower limit of the partial range is the lower limit of the selected range, and the upper limit of the partial range is smaller than the upper limit of the selected range; A reinforcement learning device characterized by:

5. The reinforcement learning device according to claim 1 , The lower limit of the partial range is greater than the lower limit of the selected range, and the upper limit of the partial range is the upper limit of the selected range. A reinforcement learning device characterized by:

6. The reinforcement learning device according to claim 1 , The learner updates a learning parameter of the value function based on the second parameter and a gradient of the value function. A reinforcement learning device characterized by:

7. The reinforcement learning device according to claim 1 , a plurality of execution entities each having the setting unit, the actor, the learner, and the determination unit; The actor of each of the plurality of execution entities shares the updated state. Reinforcement learning device.

8. The reinforcement learning device according to claim 1 , a first execution entity having the setting unit, the actor, the learner, and the determination unit; a second execution entity having the setting unit, the actor, the learner, and the decision unit, and the set of actions including an action opposing the set of actions in the first execution entity; The actors of the first execution subject and the second execution subject share the updated state; The actor of the second execution entity selects the action based on the value function, updates the state of the analysis target, and calculates the reward so that the reward is lower as the updated state is newer. A reinforcement learning device characterized by:

9. A reinforcement learning method executed by a reinforcement learning device having an actor that performs action selection in reinforcement learning, a learner that determines a value of a selected action in the reinforcement learning, a setting unit, and a determination unit, a setting process in which the setting unit sets a selection range of a first parameter related to a first risk to be taken when selecting an action to be applied to an analysis target from a group of actions to a part of the selection range, and sets a second parameter related to a second risk to be taken in learning a value function that calculates a value serving as a guideline for selecting the action; a calculation process in which the actor selects the action based on the value function and the first parameter within the partial range, updates the state of the analysis target, and calculates a higher reward as the updated state becomes newer; an update process in which the learner updates the value function based on the reward and the second parameter; a determination process in which the determination unit determines, based on a history of the rewards calculated by the actor when each of the plurality of first parameters within the partial range is used, that the first parameter to be output to the actor is a specific first parameter used when the actor selected a specific action that updated the analysis target to a new state, and outputs the first parameter to the actor; A reinforcement learning method comprising:

10. A processor that controls actors and learners in reinforcement learning. a setting process for setting a selection range of a first parameter related to a first risk to be taken when selecting an action to be applied to an analysis target from a group of actions to a part of the selection range, and setting a second parameter related to a second risk to be taken in learning a value function for calculating a value serving as a guideline for selecting the action; a calculation process in which the actor selects the action based on the value function and the first parameter within the partial range, updates the state of the analysis target, and calculates a higher reward as the updated state becomes newer; an update process in which the learner updates the value function based on the reward and the second parameter; a determination process for determining, based on a history of the rewards calculated by the actor when each of the plurality of first parameters within the partial range is used, the first parameter to be output to the actor as a specific first parameter used when the actor selected a specific action that updated the analysis target to a new state, and outputting the first parameter to the actor; A reinforcement learning program characterized by executing the following:

Images