Reinforcement learning device, reinforcement learning method, and reinforcement learning program

Abstract

To prevent learning from becoming difficult.SOLUTION: A machine learning device includes a generation unit configured to generate a behavior of an environment, a calculation unit configured to calculate, based on an action for the environment and the behavior generated by the generation unit, a simulation reward indicating how much the action simulates the behavior, and a selection unit configured to select an action for the environment based on a policy. A collection unit that collects experience data including the action, a state of the environment when the action is performed on the environment, and a reward obtained from the environment as a result of the action being performed, and a learning unit that learns the policy based on the reward collected by the collection unit and the simulated reward calculated by the calculation unit.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a reinforcement learning device, a reinforcement learning method, and a reinforcement learning program for performing reinforcement learning. [Background technology]

[0002] Developing autonomous agents in reinforcement learning requires a suitable environment in which different alternatives can be rapidly evaluated, particularly how to construct training scenarios that pit attackers and defenders against each other.

[0003] The following Patent Document 1 discloses CyberBattleSim. CyberBattleSim has a function for training an attacking AI (Artificial Intelligence) and a function for training a defensive AI. Training the defensive AI strengthens defense against attacks. In particular, training the defensive AI together with the attacking AI improves the defensive capabilities of the defensive AI to block advanced attacks from the attacking AI. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Thomas Kunz, Christian Fisher, James La Novara-Gsell, Christopher Nguyen, Li Li, “A Multiagent CyberBattleSim for RL Cyber ​​Operation Agents” https: / / arxiv.org / pdf / 2304.11052.pdf, 3 Apr 2023 Summary of the Invention [Problem to be solved by the invention]

[0005] During training, if a series of learning sessions in which no reward or only low rewards are obtained occurs before a high reward is obtained, i.e., if a series of learning sessions in which sparse rewards are obtained occurs, learning becomes difficult. This causes the defensive AI to overwhelm the attacking AI in the early stages of learning, making it difficult to collect good experience data necessary for learning.

[0006] The present invention aims to prevent learning from becoming more difficult. [Means for solving the problem]

[0007] A reinforcement learning device according to one aspect of the invention disclosed in the present application comprises a generation unit that generates environmental behavior; a calculation unit that calculates a mimicry reward indicating the degree to which an action mimics an action based on the action toward the environment and the behavior generated by the generation unit; a collection unit that selects an action toward the environment based on a policy and collects empirical data including the action, the state of the environment when the action is performed toward the environment, and a reward obtained from the environment as a result of the action; and a learning unit that learns the policy based on the reward collected by the collection unit and the mimicry reward calculated by the calculation unit. [Effects of the Invention]

[0008] According to the exemplary embodiment of the present invention, it is possible to prevent learning from becoming more difficult. Problems, configurations, and effects other than those described above will become clear from the following description of the examples. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram illustrating an example of the hardware configuration of a reinforcement learning device. [Figure 2] FIG. 2 is an explanatory diagram showing an example of a reinforcement learning process. [Figure 3] FIG. 3 is an explanatory diagram showing a first example of problem setting in reinforcement learning. [Figure 4] FIG. 4 is an explanatory diagram showing a second example of problem setting in reinforcement learning. [Figure 5] FIG. 5 is a block diagram illustrating a first example of a functional configuration of a reinforcement learning device. [Figure 6] FIG. 6 is an explanatory diagram showing an example of attack control based on priority. [Figure 7] FIG. 7 is a graph showing an example of setting priorities. [Figure 8] FIG. 8 is a block diagram showing a second example of a functional configuration of a reinforcement learning device. [Figure 9] FIG. 9 is a block diagram illustrating a detailed functional configuration example of the attack learning unit. [Figure 10] FIG. 10 is a block diagram illustrating a second example of a functional configuration of a reinforcement learning device. DETAILED DESCRIPTION OF THE INVENTION

[0010] <Figure 1. Example of hardware configuration for a reinforcement learning device> FIG. 1 is a block diagram showing an example of the hardware configuration of a reinforcement learning device. The reinforcement learning device 100 includes a processor 101, a storage device 102, an input device 103, an output device 104, and a communication interface (communication IF) 105. The processor 101, the storage device 102, the input device 103, the output device 104, and the communication IF 105 are connected via a bus 106. The processor 101 controls the reinforcement learning device 100. The storage device 102 serves as a working area for the processor 101. The storage device 102 is a non-transitory or temporary recording medium that stores various programs and data. Examples of the storage device 102 include a read-only memory (ROM), a random access memory (RAM), a hard disk drive (HDD), and a flash memory. The input device 103 inputs data. Examples of the input device 103 include a keyboard, a mouse, a touch panel, a numeric keypad, a scanner, a microphone, and a sensor. The output device 104 outputs data. The output device 104 includes, for example, a display, a printer, and a speaker. The communication IF 105 connects to a network and transmits and receives data.

[0011] <Figure 2 Reinforcement learning process> 2 is an explanatory diagram showing an example of a reinforcement learning process. In reinforcement learning, the reinforcement learning device 100 asynchronously and repeatedly executes the experience collection process 201, the experience storage process 202, and the learning process 203, such as the following: experience collection process 201 ⇒ experience storage process 202 ⇒ learning process 203 ⇒ experience collection process 201 ⇒ experience storage process 202 ⇒ learning process 203 ⇒ ...

[0012] The experience collection process 201, the experience storage process 202, and the learning process 203 are specifically realized, for example, by causing the processor 101 to execute a program stored in the storage device 102. The experience collection process 201, the experience storage process 202, and the learning process 203 will be specifically described below.

[0013] [Experience Gathering Process 201] The experience collection process 201 repeats the following steps P11 and P12. Note that t below is the number of time steps, and is an ascending integer starting from 0. That is, the number of time steps t is the number of learning iterations, and when the learning process 203 is executed, t is incremented, and when t reaches a predetermined value, the reinforcement learning ends.

[0014] P11: The agent 210 observes the current state s(t) of the environment 211, selects an action a(t) according to the policy π(t) of the agent 210, and executes it on the environment 211, obtaining a reward r(t) and a next state s(t+1) as a result of the action a(t). The agent 210 sends {state s(t), action a(t), reward r(t), next state s(t+1)} to the experience memory process 202 as experience data e(t).

[0015] Reward r(t) is the reward obtained when transitioning to state s(t+1) by taking action a(t) in state s(t).

[0016] P12: The agent 210 receives the latest policy π(t) from the learning process 203 at regular intervals and updates the policy π(t) of the agent 210.

[0017] [Experiential Memory Process 202] The experience memory process 202 repeats the following steps P21 and P22.

[0018] P21: The reinforcement learning device 100 receives the empirical data e(t) collected by the experience collection process 201, calculates the priority of the empirical data e(t), and stores it in the storage device 102. The priority of this empirical data e(t) is calculated using an index called the TD (Temporal Difference) error, which increases as the empirical data e(t) becomes more valuable to learn. There are several methods for determining the priority of the empirical data e(t) from the TD error, but a representative method is Proportional, which defines the absolute value of the TD error as the priority of the empirical data e(t). However, because the priority of the empirical data e(t) needs to be a probability distribution, the priorities of all the empirical data e(t) are normalized so that the sum of their priorities becomes 1.

[0019] P22: Upon receiving a request from the learning process 203, the reinforcement learning device 100 selects empirical data e(t) from the accumulated empirical data group 220 based on the priority of the empirical data e(t). The selected empirical data e(t) is denoted as selected empirical data e(s). The selected empirical data e(s) is not necessarily the empirical data e(t) for the time step number t. The reinforcement learning device 100 sends the selected empirical data e(s) to the learning process 203.

[0020] [Learning Process 203] The learning process 203 repeats the following steps P31 and P32.

[0021] P31: Learning the latest policy π(t) The agent 231 before learning the policy π(t-1) receives the selection experience data e(s) from the experience memory process 202 and learns the latest policy π(t) based on the selection experience data e(s). The policy π(t) is the probability of taking the action a(t) in the state s(t). The agent 231 has the same configuration as the agent 210.

[0022] P32: The agent 231 (hereinafter, agent 232) that has learned the latest policy π(t) sends the latest policy π(t) to the experience collection process 201 at regular intervals.

[0023] <Figure 3 Example of problem setting in reinforcement learning 1> 3 is an explanatory diagram showing a problem setting example 1 in reinforcement learning. In the example 1, an attacking AI 301 launches a cyber-attack on an environment 211, and a security countermeasure agent 302 defends against the cyber-attack from the attacking AI 301.

[0024] The environment 211 is a network simulator consisting of multiple nodes and links connecting the nodes. For example, the environment 211 is a simulator that simulates the behavior of an intra-organizational network that shows human relationships within the organization, where the nodes represent personal computers used by people and servers within the organization, and the links represent the relationships between people, between personal computers used by people, and between personal computers and servers.

[0025] The security countermeasure agent 302 is, for example, internal security countermeasure software implemented in the environment 211, and is executed by the processor 101. The network state of the environment 211 before the defensive action b(t) by the security countermeasure agent 302 is the state s(t) in Figure 2, and the network state of the environment 211 after the defensive action b(t) by the security countermeasure agent 302 in response to the attack action a(t) is the next state s(t+1).

[0026] Attack AI 301 is agents 210, 231, and 232 that execute and learn cyber-attacks on environment 211 by repeating experience collection process 201, experience memory process 202, and learning process 203 shown in Figure 2. Specifically, for example, attack AI 301, as agent 210, executes attack action a(t) on environment 211 and obtains reward r(t), thereby collecting experience data e(t).

[0027] The goal of the attack AI 301 is to gain ownership of all nodes in the network made up of the environment 211, for example, by obtaining the login IDs and passwords of personal computers or servers, or by infecting them with malware. Therefore, the reward r(t) depends on the next state s(t+1) of the environment 211, for example, r=50 if new authority is obtained, or r=5000 if authority for all nodes is obtained.

[0028] Furthermore, the attack AI 301, as the agent 231, learns an attack, that is, a policy π(t) that defines a method for selecting an action a(t) based on the selection experience data e(s).

[0029] The attack AI 301 receives a feature f(t) as input. The feature f(t) is a vector indicating the attack result, such as the number of nodes acquired from the environment 211, the number of nodes discovered, and cache information of the internal network of the organization.

[0030] In the experience collection process 201, when the attack AI 301 receives the feature f(t), it calculates the expected value of future cumulative rewards when each of multiple attack actions a(t) is performed according to the strategy π(t), and outputs a vector storing the expected value of the cumulative rewards. The expected value of the cumulative reward is the sum of the expected values ​​of future rewards r(t), r(t+1), r(t+2), ... (added until the end condition of the environment 211 (e.g., obtaining ownership of all nodes) is met). This expected value of cumulative rewards is calculated using a state value function.

[0031] The attack AI 301 refers to the expected value of the cumulative reward of each attack action a(t), and selects from among the multiple attack actions a(t) an attack action a(t) with a high expected value of the cumulative reward, that is, an attack action a(t) that is likely to yield a higher reward r(t) (= good), and attacks the environment 211. This attack action a(t) causes the state s(t) of the environment 211 to transition to the next state s(t+1).

[0032] Furthermore, in the learning process 203, the attack AI 301 optimizes its parameters based on the selected experience data e(s) so that the cumulative reward can be accurately predicted. The parameters here refer to the policy π when the state s(t) is input, and the parameters are optimized, for example, by minimizing the TD error of the experience data e(t). Specifically, for example, the attack AI 301 learns the policy π(t) so as to maximize the expected value of the cumulative reward.

[0033] The environment 211 receives the offensive action a(t) and defensive action b(t), rewrites the state s(t) within the organization's network to the next state s(t+1), and passes the reward r(t) and feature f(t) calculated based on the reward rules to the attack AI 301. The reward rules are basically designed to output a high reward when a desirable state is achieved (such as when the game is won). For example, when CyberBattleSim is used, when ownership of one node (a device within the network) within the environment 211 is acquired, a reward of approximately 10 to 100 is earned depending on the importance of the node. Furthermore, when ownership of all nodes is acquired, a reward of 5,000 is earned and the game ends.

[0034] <Figure 4 Reinforcement learning problem setting example 2> FIG. 4 is an explanatory diagram showing a problem setting example 2 in reinforcement learning. In FIG. 4, the problem setting example 1 in FIG. 3 is added with a mimicry reward r m This is a configuration with (t) added. m(t) is an index value for evaluating whether normal behavior in the environment 211 can be imitated. In other words, if normal behavior in the environment 211 is imitated, the attack behavior a(t) will be more likely to infiltrate the organization's network without being detected by the security measure agent 302.

[0035] The attack AI 301 calculates the reward r(t) and the mimicry reward r(t) weighted by the priority α(t) calculated by the attack AI 301. m (t) and the composite reward r s Expected cumulative value of (t) Q s (t) is calculated. That is, the mimicry reward r m The larger (t) is (the more it is able to imitate normal behavior), the expected value of the cumulative composite reward Q s Therefore, in the experience collection process 201, the attack AI 301 increases the expected value Q s By calculating (t), it is possible to learn the probability of selecting such an action a(t), i.e., the policy π(t).

[0036] <Figure 5 Functional configuration example 1 of the reinforcement learning device 100> 5 is a block diagram showing a first example functional configuration of the reinforcement learning device 100. The reinforcement learning device 100 includes a reinforcement learning unit 501 and a control unit 502. Specifically, the reinforcement learning unit 501 and the control unit 502 are realized by, for example, causing the processor 101 to execute a program stored in the storage device 102 shown in FIG.

[0037] The state s(t) shown in Fig. 5 corresponds to the feature f(t) shown in Fig. 3 and Fig. 4. That is, the feature f(t) is a vector obtained by quantifying the state s(t), and the conversion from the state s(t) to the feature f(t) is performed by the policy π(t).

[0038] [Reinforcement Learning Section 501] The reinforcement learning unit 501 has an attack experience collection unit 511, an experience memory unit 512, and an attack learning unit 513. The attack experience collection unit 511 executes the above-described experience collection process 201 as the attack AI 301 on the network simulator 500, which is an example of the environment 211, selects an attack action a(t) based on the strategy π(t), and attacks the network simulator 500. The security countermeasure software 542 is an example of the security countermeasure agent 302, and takes a defensive action b(t) on the network simulator 500 in response to the attack action a(t).

[0039] The network simulator 500 outputs the state s(t) before the attack action a(t) and the next state s(t+1) after the defensive action b(t) against the attack action a(t). The network simulator 500 outputs the reward r(t) obtained when the attack experience collecting unit 511 takes the attack action a(t) in the state s(t) and transitions to the state s(t+1), based on the reward rule described above.

[0040] Furthermore, when the priority α(t) is input from the setting unit 523, the attack experience collecting unit 511 outputs an attack behavior a(t) based on the priority α(t) and the strategy π(t). Details of the attack experience collecting unit 511 will be described later with reference to FIG.

[0041] The experience memory unit 512 executes the above-mentioned experience collection process 201 as the attack AI 301. In addition, the experience data e(t) in the above-mentioned experience collection process 201 stores {state s(t), action a(t), reward r(t), next state s(t+1)}, but here, the mimicry reward r m (t), and the priority α(t) are also stored.

[0042] The attack learning unit 513, as the attack AI 301, executes the above-mentioned learning process 203 based on the selected experience data e(s), generates a policy π(t), and outputs it to the attack experience collection unit 511. Details of the attack learning unit 513 will be described later with reference to FIG.

[0043] [Control unit 502] The control unit 502 controls the reinforcement learning unit 501 to generate a mimicry reward r m Specifically, the control unit 502 includes a generation unit 521, a calculation unit 522, and a setting unit 523.

[0044] The generating unit 521 uses the network setting information 514 from the network simulator 500 to generate, as the behavior of the environment 211, communication feature amount data c(t) indicating communications flowing in the intra-organization network configured by the network simulator 500.

[0045] The network setting information 514 includes the nodes that make up the intra-organization network, the node type (personal computer / server), the number of nodes n (n is an integer of 1 or more), and the links that connect the nodes.

[0046] The communication feature data c(t) is data in which the number of communications from the i-th node (i is an integer satisfying 1≦i≦n) to the j-th node (j is an integer satisfying 1≦j≦n) in the time step number t is stored in the element of the i-th row and j-th column (hereinafter referred to as element ij) of an n×n matrix. In other words, the communication feature data c(t) is communications indicating normal behavior (normal communications) within the network simulator 500.

[0047] The calculation unit 522 calculates the mimicry reward r based on the communication feature data c(t) and the aggressive behavior a(t). m The attack behavior a(t) is data obtained by further storing the number of communications from the i-th node to the j-th node that occurred due to an attack on the network simulator 500 in the element ij of the communication feature data c(t).

[0048] Mimic reward r m (t) is, for example, the inverse of the Euclidean distance between the communication feature data c(t) and the attacking behavior a(t), and indicates the similarity between normal communication and the attacking behavior a(t). To prevent the denominator from becoming 0, a constant (for example, 1) may be added to the denominator. The shorter the Euclidean distance between the communication feature data c(t) and the attacking behavior a(t), the higher the mimicry reward rm The larger the value of (t), the more similar the normal communication indicated by the communication feature data c(t) is to the communication indicated by the attack behavior a(t), and the more the attack behavior a(t) can imitate the normal communication indicated by the communication feature data c(t). Therefore, the more likely it is that such attack behavior a(t) will infiltrate the organization's internal network without being detected by the security countermeasure agent 302. The mimicry reward r m (t) is stored in the empirical data e(t).

[0049] The setting unit 523 sets the mimicry reward r m The setting unit 523 sets a priority α(t) indicating how much priority to give to (t), and outputs the priority α(t) to the attack experience collecting unit 511. Specifically, for example, the setting unit 523 sets the priority α(t) using a priority selection history 531 and / or a reward history 532. The priority selection history 531 is a history of selecting priorities α(1) to α(t-1) up to the time step number t-1. The reward history 532 is a calculation history of rewards r(1) to r(t-1) up to the time step number t-1.

[0050] [Figure 6 Attack experience collection unit 511] 6 is a block diagram showing a detailed example of the functional configuration of the attack experience collection unit 511. The attack experience collection unit 511 has a behavior decision unit 600. The behavior decision unit 600 has a reward prediction unit 601, a mimicry reward prediction unit 602, a calculation unit 603, and a behavior selection unit 604.

[0051] The reward prediction unit 601 is a neural network that receives the priority α(t) and the state s(t) and calculates the cumulative reward prediction value Q(t). The mimicry reward prediction unit 602 receives the priority α(t) and the state s(t) and calculates the cumulative mimicry reward prediction value Q m In other words, the policy π(t) is a weight set in the reward prediction unit 601 and the mimicry reward prediction unit 602.

[0052] The cumulative reward prediction value Q(t) is the expected value of the future cumulative reward when each of the multiple aggressive actions a(t) is performed. m(t) is the expected value of the future cumulative mimicry reward when each of the multiple aggressive behaviors a(t) is performed. That is, the cumulative reward predicted value Q(t) and the cumulative mimicry reward predicted value Q m (t) is a real vector with dimensions equal to the number of attack actions a(t), and the expected value when the attack action is executed is stored for each dimension.

[0053] The calculation unit 603 calculates the cumulative mimicry reward predicted value Q m (t) is weighted by the priority α(t), and the cumulative reward prediction value Q(t) and the weighted cumulative mimicry reward prediction value Q m (t) × α(t) and the cumulative composite reward prediction value Q s (t). When learning is completed and inference is executed, the calculation unit 603 sets the priority α(t)=0 and outputs the cumulative combined reward predicted value Q s (t) is calculated. That is, the cumulative mimicry reward prediction value Q m (t) is used only to collect good experiences when collecting experiences during learning.

[0054] The action selection unit 604 calculates the cumulative combined reward prediction value Q s (t). Specifically, for example, the action selection unit 604 selects an offensive action a(t) based on the cumulative combined reward predicted value Q s (t) is selected to maximize the cumulative composite reward prediction value Q s (t) is also the cumulative reward prediction value Q(t) and the cumulative mimicry reward prediction value Q m Similar to (t), it is a real vector with dimensions equal to the number of offensive actions a(t). The action selection unit 604 calculates the cumulative combined reward prediction value Q s The cumulative composite reward prediction value Q from the real vector of (t) s Select the attack action a(t) that maximizes (t).

[0055] The priority α(t) input to the reward prediction unit 601 and the mimicry reward prediction unit 602 is selected, for example, according to the UCB (Upper Confidence Bound) method. The UCB method is a method for selecting which hyperparameter (in this example, priority αi) to use to collect experience in reinforcement learning. i represents one of the numbers 1 to n (n is an integer equal to or greater than 1) in the priority selection history 531.

[0056] The setting unit 523 selects the priority αi as the priority α(t) based on the reward history 532 obtained in the most recent few episodes, taking into consideration the following points (A) and (B). (A) The setting unit 523 selects a priority αi that is likely to provide as much reward as possible. (B) The setting unit 523 preferentially selects a priority αi that has not been selected before (because there is a possibility that a large reward will be obtained by trying a priority αi that has not yet been tried).

[0057] [Figure 7 UCB method] In order to take into account the above points (A) and (B), a score called the UCB score μi is defined by the following formula (1), and the setting unit 523 selects the priority αi with the largest μi at the beginning of each episode as the priority α(t). One episode is the period from the time step t=1 to the time step t=n (n is an integer equal to or greater than 1).

[0058]

number

[0059] In the above formula (1), μi(T) on the left side is the composite reward r of the options obtained by selecting the priority αi up to the Tth episode. s (t) is the average value of the cumulative values.

[0060] In the above formula (1), the first term on the right side is the most recent average combined reward term, that is, the combined reward r obtained in the episode where priority αi was selected among the most recent few episodes up to the (T-1)th episode. s(t) is the average value of the cumulative value of (t). In other words, the most recent average composite reward is the composite reward r that can be expected when selecting priority αi. s It can be considered as the cumulative value of (t).

[0061] The second term on the right-hand side is a correction term. Ni in the second term on the right-hand side is the number of episodes in which αi was selected in the last few episodes up to the (T-1)th episode. The correction term becomes larger the fewer the number of times Ni that priority αi has been selected up to now.

[0062] In other words, selecting a priority αi where the first and second terms on the right-hand side are large, i.e., μi(T) is large, is equivalent to selecting a priority αi that can be expected to provide a large composite reward and that has not been selected much in the past.

[0063] Figure 7 is a chart showing the selection results of priority αi using the UCB method. Specifically, Figure 7 shows the priority selection history 531 of priority αi for the most recent five episodes, the reward history 532 at that time, the composite reward average term in the first term on the right-hand side of the above formula (1) and the correction term in the second term on the right-hand side, and the UCB score μi(T=10) for the current episode T=10. Among the UCB scores μi(T=10), the UCB score μ3(T=10) = "9" for i=3 is the highest score, so in the T=10th episode, priority α3 = 9 is selected.

[0064] [Figure 8 Example of setting priority α(t)] Fig. 8 is a graph showing an example of setting the priority α(t). In Fig. 8, for example, the setting unit 523 sets the priority α(t) according to the number of time steps t using an exponential function or a linear function. Specifically, for example, the setting unit 523 changes the priority α(t) selected by the method shown in Fig. 7 according to the number of time steps t using an exponential function or a linear function.

[0065] [Figure 9 Attack learning unit 513] 9 is a block diagram showing a detailed example of the functional configuration of the attack learning unit 513. The attack learning unit 513 has an action decision unit 900, a TD error calculation unit 901, and a mimicry TD error calculation unit 902. The action decision unit 900 has a configuration in which the calculation unit 603 and the action selection unit 604 are removed from the action decision unit 600.

[0066] The attack learning unit 513 calculates the cumulative reward prediction value Q(t) and the cumulative mimicry reward prediction value Q m The reward prediction unit 601 and the mimicry reward prediction unit 602, which are composed of neural networks, are optimized so that (t) and (t) can be accurately predicted.

[0067] Specifically, if weights for the reward prediction unit 601 and the mimicry reward prediction unit 602 are obtained that minimize the TD error L defined by the following equation (2) for all empirical data e(t), then these weights become the optimal policy π(t).

[0068]

number

[0069] E[] in the above equation (2) is the expected value of the TD error L, and is, for example, the average value based on multiple pieces of empirical data e(t) sampled from the empirical data group 220. γ on the right-hand side of the above equation (2) is a hyperparameter called a time discount rate. r(t) is the reward for the empirical data e(t) at time step t. maxQ(t+1) is the maximum value of the state value function Q at time step t+1. Q(t) is the state value function Q at time step t.

[0070] The above formula (2) is the simplest TD error. In practice, techniques such as importance sampling may be used to appropriately weight each empirical data e(t) to remove bias caused by differences between the policy π(t) at the time of experience collection and the policy π(t) at the time of learning. This embodiment does not rely on the formula for the TD error itself, and is therefore applicable to such TD error derivation techniques.

[0071] Specifically, for example, during learning in the attack learning unit 513, the TD error calculation unit 901 calculates the TD error L by the above formula (2).

[0072] The reward prediction unit 601 learns and updates the weights so that the TD error L is minimized with respect to the cumulative reward prediction value Q(t), thereby optimizing the reward prediction unit 601.

[0073] Similarly, during learning in the attack learning unit 513, the mimicry TD error calculation unit 902 calculates the cumulative mimicry reward prediction value Q m (t) and the mimetic reward r included in the selection experience data e(s). m (t) and the predicted value of the mimic reward maxQ m (t+1), the simulated TD error L is calculated using the following equation (3). m Calculate.

[0074]

number

[0075] The mimicry reward prediction unit 602 calculates the mimicry TD error L m is the cumulative predicted value of the mimicry reward Q m The weights are updated by learning so that (t) is minimized. This optimizes the mimicry reward prediction unit 602.

[0076] <Fig. 10 Functional configuration example 2 of the reinforcement learning device 100> Fig. 10 is a block diagram showing a second functional configuration example of the reinforcement learning device 100. Fig. 10 shows an example in which the reinforcement learning device 100 is applied to measures against SNS (Social Networking Service) stealth marketing. The difference from Fig. 5 is that a stealth marketing detector 1002 is used as an example of the security measure agent 302, and an SNS simulator 1000 is used as an example of the environment 211.

[0077] The stealth marketing detector 1002 is software that detects stealth marketing, which is a social media post a(t) that promotes a product or service without disclosing that it is a promotion or advertisement.

[0078] Like the network simulator 500, the SNS simulator 1000 is a simulator that simulates the behavior of an intra-organizational network that shows human relationships within the organization, where nodes represent personal computers (e.g., smartphones) used by people and servers that provide SNS, and links represent relationships between people, between personal computers used by people, and between personal computers and servers.

[0079] The attack experience collecting unit 511 outputs the SNS post a(t) as the attack behavior a(t). The generating unit 521 generates communication feature amount data c(t) related to the SNS post a(t).

[0080] Thus, according to this embodiment, the mimicking reward r m By using (t), it is possible to reduce learning with sparse rewards and prevent learning from becoming more difficult. This allows attacks that mimic normal communications to be learned preferentially, making it possible to deceive the security countermeasure agent 302 and facilitating the collection of good experience data e(t) required for learning.

[0081] The present invention is not limited to the above-described embodiments, and includes various modifications and equivalent configurations within the spirit and scope of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to configurations including all of the described configurations. Furthermore, part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, part of the configuration of each embodiment may be added to, deleted from, or replaced with other configurations.

[0082] Furthermore, the aforementioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or may be realized in software by having a processor interpret and execute a program that realizes each function.

[0083] Information such as programs, tables, files, etc. that realize each function can be stored in storage devices such as memory, hard disks, SSDs (Solid State Drives), or recording media such as IC (Integrated Circuit) cards, SD cards, and DVDs (Digital Versatile Discs).

[0084] In addition, the control lines and information lines shown are those that are considered necessary for explanation, and do not necessarily represent all the control lines and information lines that are necessary for implementation. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0085] 100 Reinforcement Learning Device 101 processors 102 Storage Devices 201 Experience Gathering Process 202 Experiential Memory Process 203 Learning Process 211 Environment 302 Security Agent 501 Reinforcement Learning Department 502 control section 511 Attack Experience Collection Department 512 Experience Memory Department 513 Attack Learning Department 514 Network setting information 521 Generation part 522 Calculation Unit 523 Settings Department

Claims

1. a generator for generating environmental behavior; a calculation unit that calculates a mimicry reward indicating how well the action mimics the behavior based on the action relative to the environment and the behavior generated by the generation unit; a collection unit that selects an action for the environment based on a strategy, and collects experience data including the action, a state of the environment when the action is performed on the environment, and a reward obtained from the environment as a result of the action; a learning unit that learns the policy based on the reward collected by the collection unit and the mimicry reward calculated by the calculation unit; A reinforcement learning device comprising:

2. The reinforcement learning device according to claim 1 , The calculation unit calculates a similarity between the action and the behavior as the mimicry reward. A reinforcement learning device characterized by:

3. The reinforcement learning device according to claim 1 , A setting unit for setting a priority of the mimicry reward, the learning unit learns the policy based on the priority set by the setting unit. A reinforcement learning device characterized by:

4. The reinforcement learning device according to claim 3, The setting unit setting the priority based on the number of times learning is repeated by the learning unit; A reinforcement learning device characterized by:

5. The reinforcement learning device according to claim 1 , The learning unit has a first reward prediction unit that calculates a first cumulative reward prediction value, which is a prediction value of a cumulative value of the reward based on the reward, and a first mimicry reward prediction unit that calculates a first cumulative mimicry reward prediction value, which is a prediction value of a cumulative value of the mimicry reward based on the mimicry reward, and learns the first reward prediction unit so that a difference between the reward and the first cumulative reward prediction value becomes small, and learns the first mimicry reward prediction unit so that a difference between the mimicry reward and the first cumulative mimicry reward prediction value becomes small. A reinforcement learning device characterized by:

6. The reinforcement learning device according to claim 1 , an action decision unit that decides the action based on the measures when the state of the environment is input; A reinforcement learning device comprising:

7. 7. The reinforcement learning device according to claim 6, A setting unit for setting a priority of the mimicry reward, When the priority is input, the action determination unit determines the action based on the priority and the measure. A reinforcement learning device characterized by:

8. 7. The reinforcement learning device according to claim 6, The behavior determination unit has a second reward prediction unit that calculates a second cumulative reward prediction value, which is a prediction value of a cumulative value of the reward, based on the reward, and a second mimicry reward prediction unit that calculates a second cumulative mimicry reward prediction value, which is a prediction value of a cumulative value of the mimicry reward, based on the mimicry reward, and determines the behavior based on the second cumulative reward prediction value and the second cumulative mimicry reward prediction value. A reinforcement learning device characterized by:

9. The reinforcement learning device according to claim 8, A setting unit for setting a priority of the mimicry reward, When the priority is input, the behavior determination unit determines the behavior based on the second cumulative reward predicted value and the second cumulative mimicry reward predicted value weighted by the priority. A reinforcement learning device characterized by:

10. A reinforcement learning method executed by a reinforcement learning device having a processor that executes a program and a storage device that stores the program, the processor: a generation process for generating environmental behavior; a calculation process for calculating a mimicry reward indicating how well the action mimics the behavior based on the action relative to the environment and the behavior generated by the generation process; a collection process for selecting an action for the environment based on the strategy and collecting experience data including the action, a state of the environment when the action is taken on the environment, and a reward obtained from the environment as a result of the action; a learning process that learns the policy based on the reward collected by the collection process and the mimic reward calculated by the calculation process; A reinforcement learning method characterized by executing the following.

11. The processor a generation process for generating environmental behavior; a calculation process for calculating a mimicry reward indicating how well the action mimics the behavior based on the action relative to the environment and the behavior generated by the generation process; a collection process for selecting an action for the environment based on the strategy and collecting experience data including the action, a state of the environment when the action is taken on the environment, and a reward obtained from the environment as a result of the action; a learning process that learns the policy based on the reward collected by the collection process and the mimic reward calculated by the calculation process; A reinforcement learning program characterized by executing the following.

Images