Information processing apparatus, information processing method, program product, and storage medium

Abstract

The invention provides a technology capable of evaluating or learning a model by using appropriate noise for ensuring robustness of the model. An information processing device according to the present invention is provided with: a determination means for determining the distribution of reactive noise with respect to a model to be processed, using a predetermined pre-distribution; and a learning control means that learns at least one of an action value function and a policy function of the model to be processed on the basis of an action value of an action in a perturbated state in which the resistance noise is added to a state in an environment used in the model to be processed. The determination means determines the distribution of the antagonistic noise, which reduces the action value of the model to be processed, on the basis of using a constraint after a divergence indicating the proximity between the distribution of the antagonistic noise and a predetermined pre-distribution.

Description

TECHNICAL FIELD

[0001] The present application relates to an information processing apparatus, an information processing method, a program product, and a storage medium. BACKGROUND

[0002] It is known that if an actual acquired state, environment deviates from a state, environment assumed (for example, at the time of learning, etc.), the performance of a reinforcement learning algorithm does not function. Therefore, a reinforcement learning system is proposed that generates a new state to which noise is added to an acquired state, and calculates an action value function using the state to which the noise is added, whereby a deviation of the state can be considered (Patent Literature 1).

[0003] PRIOR ART DOCUMENT

[0004] PATENT LITERATURE

[0005] Patent Literature 1: International Publication No. 2023 / 037504 SUMMARY

[0006] PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] Note that, in order to evaluate the robustness of a model, or to cause a model to learn so as to ensure robustness, it is necessary to give an appropriate noise. This is because the ensuring of robustness often has a trade-off relationship with the performance of a controller at the time of normal action, and giving a noise that cannot be generated in reality, ensuring excessive robustness only leads to deterioration of performance. However, in the above-described prior art, only a noise based on a random variable is considered to be added to a state.

[0008] The present application was made in view of the above-described problems, and aims to provide a technology capable of evaluating or learning a model using an appropriate noise for ensuring the robustness of the model.

[0009] MEANS FOR SOLVING THE PROBLEMS

[0010] In order to solve this problem, for example, the information processing apparatus of the present application has:

[0011] a decision mechanism that decides a distribution of an adversarial noise with respect to a model of a processing target using a predetermined pre-distribution; and

[0012] a learning control mechanism that causes at least either of an action value function or a policy function of the model of the processing target to learn based on an action value of an action in a state within an environment used in the model of the processing target, the state to which the adversarial noise is added,

[0013] The decision-making mechanism determines the distribution of adversarial noise that reduces the action value of the model of the object being processed, based on constraints using divergence, which represents the proximity of the distribution of the adversarial noise to the predetermined pre-distribution.

[0014] To solve this problem, for example, the information processing method of the present invention performs each step by an information processing device, which has the following characteristics:

[0015] A decision step, in which the distribution of adversarial noise relative to the model of the object being processed is determined using a predetermined pre-distribution; and

[0016] A learning control step is performed in which at least one of the action value function or policy function of the model of the processing object is learned, based on the action value of actions in a perturbed state with added adversarial noise within the environment used in the model of the processing object.

[0017] In the decision step, based on constraints using divergence representing the proximity of the distribution of the adversarial noise to the predetermined pre-distribution, the distribution of the adversarial noise that reduces the action value of the model of the object being processed is determined.

[0018] Invention Effects

[0019] According to the present invention, techniques are provided that enable the evaluation or learning of models using appropriate noise to ensure the robustness of the models. Attached Figure Description

[0020] Figure 1 This is a block diagram illustrating an example of the functional configuration of the vehicle according to Embodiment 1.

[0021] Figure 2 This is a diagram illustrating the relationship between the functional components involved in the robustness evaluation of Implementation Method 1.

[0022] Figure 3 This is a diagram illustrating the evaluation of the reinforcement learning model involved in Implementation Method 1.

[0023] Figure 4A This is a diagram illustrating the details of the noise addition involved in Implementation Method 1.

[0024] Figure 4B This is a diagram illustrating the generated noise distribution involved in Implementation Method 1.

[0025] Figure 5 This is a flowchart illustrating a series of actions involved in the noise additional processing described in Implementation 1.

[0026] Figure 6 This is a flowchart representing a series of actions involved in the robustness evaluation process described in Implementation 1.

[0027] Figure 7 This is a diagram illustrating the main components of the vehicle involved in Embodiment 2.

[0028] Figure 8 This diagram illustrates the learning method of the reinforcement learning model involved in Implementation Method 2.

[0029] Figure 9 This is a flowchart representing a series of actions involved in the learning process of the model involved in Implementation Method 2.

[0030] Figure 10 This is a diagram illustrating the evaluation of the reinforcement learning model involved in Implementation Method 3.

[0031] Figure 11 This diagram illustrates the process of changing the environment involved in Implementation Method 3.

[0032] Figure 12 This is a flowchart representing a series of actions involved in the robustness evaluation process described in Implementation Method 3.

[0033] Explanation of reference numerals in the attached figures

[0034] 100: Vehicles;

[0035] 101: Sensors Department;

[0036] 108: Control Department;

[0037] 113: Noise Additions;

[0038] 114: Model Processing Department;

[0039] 116: Learning Control Department. Detailed Implementation

[0040] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Furthermore, the embodiments described below do not limit the invention involved in the technical solution, and not all of the feature combinations described in the embodiments are essential to the invention. Two or more features from the plurality of features described in the embodiments can be arbitrarily combined. Additionally, the same or identical components are labeled with the same reference numerals, and repeated descriptions are omitted.

[0041] (Implementation Method 1)

[0042] In Embodiment 1, the robustness evaluation involved in this embodiment will be described. In this robustness evaluation, the robustness of the machine learning model of the evaluated object relative to noisy input is assessed. Therefore, in this embodiment, appropriate noise is provided for such a robustness evaluation. Furthermore, the following embodiment will be described using the implementation of the invention in a vehicle as an example. However, the following embodiment can also be executed in one or more information processing devices, such as a server device. Additionally, the vehicles described below include not only four-wheeled or two-wheeled passenger vehicles, but also vehicles that guide or follow people when no one is riding. Furthermore, the following embodiment can be applied not only to the aforementioned vehicles, but also to robots capable of autonomous or maneuverable movement. The following embodiment is not limited to devices capable of movement themselves, but can also be applied to robots that move objects (e.g., robotic arms), and information processing devices (control devices) that directly or remotely control devices capable of movement.

[0043] <Vehicle Composition>

[0044] First, refer to Figure 1 A functional configuration example of the vehicle 100 according to this embodiment will be described. Furthermore, the functional blocks described with reference to the following figures can be integrated or separated, and the described functions can also be implemented by other functional blocks. Additionally, functional blocks described as hardware can also be implemented as software, and vice versa.

[0045] In the following example, the case where a control unit 108 is assembled in a vehicle 100 will be used for explanation. However, the control unit 108 of the vehicle 100 can be configured as a control module or information processing device having the configuration of the control unit 108. That is, the present invention can be implemented as a control module or information processing device having the processor 110, model processing unit 114, etc. included in the control unit 108.

[0046] The sensor unit 101 includes various sensors present in the vehicle 100, and outputs sensor data related to the behavior of the vehicle 100. Among these sensors are, for example, a vehicle speed sensor for measuring the vehicle speed, an acceleration sensor for measuring the vehicle's body acceleration, and a suspension displacement sensor for measuring the travel behavior (speed, displacement) of the shock absorbers. Other sensors include a steering angle sensor for measuring steering input, a sensor for measuring the torque generated in the power unit 105, and a GPS device for acquiring its own position. Furthermore, the sensor unit 101 may include a camera (image capture unit) that outputs images captured of the front of the vehicle 100 (or further, images captured of the sides and rear of the vehicle). The sensor unit 101 may also include a LiDAR (Light Detection and Ranging) system that outputs distance images obtained by measuring the distance to the front of the vehicle (or further, images captured of the sides and rear of the vehicle).

[0047] The acceleration, position information, steering angle, torque, captured images, and distance images of the vehicle 100, and other sensor data, are used as one of the states for controlling the vehicle's actions, for example, by a reinforcement learning model included in the model processing unit 114.

[0048] The communication unit 102 is, for example, a communication device including communication circuits, which communicates with external information processing servers (not shown), surrounding traffic systems, etc., via mobile communication such as LTE, LTE-Advanced, or so-called 5G. For example, the communication unit 102 receives some or all of map data, traffic information, etc., from other information processing servers or surrounding traffic systems. The communication unit 102 can obtain at least one of the hyperparameters of the learning model used in the model processing unit 114, the learned parameters, or the pre-distribution of noise, the pre-distribution of environmental parameters, etc., described later, from the external information processing server.

[0049] The operation unit 103 includes, in addition to operating components such as buttons and touch panels installed within the vehicle 100, components that receive inputs for driving the vehicle 100, such as a steering wheel and brake pedal. The power supply unit 104 includes, for example, a battery composed of a lithium-ion battery, which supplies power to various parts within the vehicle 100. The power unit 105 includes, for example, an engine or motor that generates power to move the vehicle. The reporting unit 106 reports predetermined sounds, such as warning sounds, to the occupants (in some cases, the driver).

[0050] The storage unit 107 includes a non-volatile, high-capacity storage device such as a semiconductor memory. It temporarily stores various sensor data output from the sensor unit 101. In addition, it stores trajectory information including the completed learning parameters of the machine learning model executed by the model processing unit 114, and the set of actions and states of reinforcement learning described later.

[0051] The control unit 108 includes, for example, a processor 110, RAM 111, and ROM 112, and controls the actions of various parts of the vehicle 100. Furthermore, the control unit 108 can acquire sensor data from the sensor unit 101, thereby performing processing for vehicle 100 action control based on the reinforcement learning model (described later) and processing for robustness evaluation of the reinforcement learning model. The processor 110 loads and executes a computer program stored in the ROM 112 in the RAM 111, thereby enabling the control unit 108 to perform the functions of various units, such as the model processing unit 114.

[0052] Processor 110 may include one or more processors, such as a CPU. In addition to a CPU, processor 110 may include other processors or circuits such as a GPU or ASIC for high-speed execution of the model processing unit 114. RAM 111 is composed of volatile storage media such as DRAM and functions as the working memory of processor 110. ROM 112 is composed of non-volatile storage media and stores computer programs executed by processor 110, setting values ​​for operating control unit 108, etc.

[0053] The noise addition unit 113 generates adversarial noise, thereby adding the generated adversarial noise to the sensor data (e.g., torque, captured images) received from the sensor unit 101. Adversarial noise is sometimes referred to as adversarial samples, etc. Adversarial noise is obtained by identifying inputs for which the learned model cannot output optimal results or predicts low evaluation values ​​(low performance). By applying adversarial noise to the model, the robustness of the model can be evaluated, or the model can be trained to improve its robustness. Furthermore, when evaluating the robustness of the machine learning model in the model processing unit 114 or when training the machine learning model, the noise addition using the noise addition unit 113 is performed. That is, the noise addition unit 113 is not used during the operation of the vehicle 100 without evaluation of the machine learning model. In this case, the sensor data output from the sensor unit 101 can be output to the model processing unit 114.

[0054] The model processing unit 114 executes a machine learning model that implements a reinforcement learning algorithm, using sensor data to determine the actions of the vehicle 100. For example, it uses sensor data such as torque and captured images to output action instructions to control the power unit 105 (in a manner that controls acceleration, deceleration, or steering). Furthermore, this control example is one in which arbitrary action instructions can be output using arbitrary sensor data.

[0055] The action control unit 115 controls the movement of the vehicle 100 based on the action instructions output from the model processing unit 114. For example, the action control unit 115 controls the power unit 105 according to the action instructions for controlling the power unit 105 based on the model processing unit 114. In this embodiment, the model processing unit 114 and the action control unit 115 are described separately, but the action control unit 115 may also be included in the model processing unit 114.

[0056] Figure 2 This illustrates the relationship between the main functional components involved in the robustness evaluation according to Embodiment 1. For example, sensor data (e.g., torque) is acquired via sensor unit 101. Noise addition unit 113 adds adversarial noise (described later) to the sensor data. Model processing unit 114 uses the sensor data with adversarial noise to determine the action output (e.g., acceleration / deceleration, steering control amount) corresponding to the action that the vehicle 100 should take. Action control unit 115 controls power unit 105 based on the action output from model processing unit 114.

[0057] <Evaluation of Reinforcement Learning Models>

[0058] Next, refer to Figure 3 The evaluation of the reinforcement learning model involved in this embodiment will be explained. Sensor data is acquired at a certain time t. If sensor data is acquired, as described above, the noise addition unit 113 adds adversarial noise (described later) to the sensor data (adversarial noise addition 301). The model processing unit 114 receives the sensor data with added adversarial noise and outputs the control quantity (policy-based action output 302) obtained (through the execution of a machine learning algorithm). At this time, in reinforcement learning, the sensor data corresponds to the state (s) of the environment. t The control quantity is equivalent to the action relative to the environment (a) t Additionally, if for state (s) t Adding adversarial noise results in a state (s) with added adversarial noise. ～ t ).

[0059] Subsequently, if the action control unit 115 controls the power unit 105 based on the control input, new sensor data (action and state observations 303 in the environment) is acquired at time t+1. In reinforcement learning, this sensor data corresponds to the state (s) in the environment. t+1 The model processing unit 114 determines the reward (r) in reinforcement learning based on sensor data from the sensor unit 101. t (Or penalty) (Reward determination 304). The reward is, for example, a reward value related to the vehicle's behavior obtained from a predetermined combination of sensor data. As time progresses, the processing steps 301 to 304 are repeated to accumulate the reward for actions over multiple steps (cumulative reward 305). The model processing unit 114 compares, for example, the cumulative reward obtained without the addition of adversarial noise with the cumulative reward 305, and evaluates the robustness relative to the model (robustness evaluation 306). For example, if the cumulative reward 305 changes by more than a predetermined value compared to the cumulative reward without the addition of adversarial noise, it means that the action deviates from the expected action of the model, indicating low robustness relative to adversarial noise.

[0060] The model processing unit 114, for example, makes the reinforcement learning model constituting the actor-critic method act. The actor chooses action (a) based on policy π(a|s). The critic is the body that evaluates the policy π(a|s) currently used by the actor, and under the action of policy π, for example, has an action value function Q representing the expected discounted reward and the action value function Q given that action a has been taken in state s. π (s, a). Furthermore, as described later, in the actor-critic approach, the actor who decides the action is improved while the critic learns the evaluation strategy. However, it is argued that since the action output is discrete, even if methods such as selecting the optimal action from multiple action candidates are attempted, the actor will choose the action candidate whose evaluation value (the value of the action value function) is maximized. Therefore, it can also be used in methods other than actor-critic approaches such as Q-learning and DQN.

[0061] <Details of noise addition>

[0062] Next, refer to Figure 4A Details regarding the noise additive involved in this embodiment will be explained. This processing is relative to the state s acquired at time t. t This is performed in the noise addition unit 113. The noise addition unit 113 obtains the pre-distribution of noise as described below from the storage unit 107 or the communication unit 102.

[0063] The noise addition unit 113, for example, in the case of approximation in sampling, samples noise according to a predetermined pre-distribution into n data points. The pre-distribution can be various distributions that assume the distribution of noise that can be generated in the environment used in the model of the evaluation object. In addition to the normal distribution, various distributions can be used.

[0064] Noise addition part 113 for state s t The additional noise samples are used to generate the perturbed state (s). ～ t1 、…、s ～ ti 、…、s ～ tn Furthermore, in the following description, the superscript ~ indicates the value after perturbation due to the influence of noise. The noise addition unit 113's action a in calculating the perturbated state (using an actor) ～ ti Based on this, (using commentators) calculate in state s t The following action has been taken. ～ ti The value of actions Q(s) under certain circumstances t a ～ ti ).

[0065] The noise addition unit 113 calculates the distribution of adversarial noise relative to the model (reinforcement learning model) being evaluated. The distribution of adversarial noise is obtained by adding constraints using divergence, which represents the proximity of the calculated adversarial noise distribution to a predetermined pre-distribution, while simultaneously determining the noise distribution that minimizes the model's action value (i.e., noise that is difficult for the model to handle). In state s ～ t The adversarial noise distribution ν that minimizes action value * (s ～ t |s t When the divergence is f, it can be obtained by the following equation (1).

[0066] [Number 1]

[0067]

[0068] Here, D f (ν||p) represents the f-divergence between the noise distribution ν and the pre-distribution p of the noise, α attk This is an adjustment coefficient used to adjust the strength of the constraint based on this divergence. That is, in equation (1), when calculating the state s... t The following action has been taken. ～ tThe value of an action Q in a given situation π～ (s t a ～ t When minimizing the expected value of the noise distribution, it is possible to achieve this through α. attk The degree to which the distribution ν of the noise is constrained by the pre-distributed p is adjusted.

[0069] α attk The closer the value of ν is to 0, the better. ＊ (s ～ t |s t The distribution of α, as an example, tends to have a peak-like distribution that is closer to the noise value at a point that minimizes the action value. On the other hand, α attk The larger the value of ν, the closer it is to the pre-defined distribution p. ＊ (s ～ t |s t The distribution of α. That is, in this embodiment, if α is appropriately set... attk The value of can then include the characteristics of the noise distribution (which minimizes the action value) and obtain the adjusted adversarial noise distribution in a manner with the characteristics of the pre-distributed p.

[0070] Next, given that the divergence is defined as KL divergence, the adversarial noise distribution ν minimized in equation (1) can be expressed by equation (2). ＊ (s ～ t |s t The analytical solution to can be obtained by using the Fincher-Legend transform. However, it is difficult to compute when dealing with continuous state and action spaces. Furthermore, even when approximated by known methods such as the Markov chain Monte Carlo method, it requires multiple accesses to the policy π and action value function Q to compute the value at each time t. π However, this presents the problem of high computational costs.

[0071] [Number 2]

[0072]

[0073] Therefore, as one embodiment of this invention, equation (2) is assumed to approximate the pre-distribution p(s) of the noise. ～ t |s t The number of restrictive elements (e.g., in) Figure 4A The approximate adversarial noise distribution ν is obtained from the sample values ​​of 1 to n (as described in the text) using equation (3). * (s ～t |s t ).

[0074] [Number 3]

[0075]

[0076] This is to compare the distribution p(s) with the pre-distributed distribution used for sampling. ～ t |s t The weights for each sample are modified from the pre-distributed adversarial noise distribution to the shape of the numerator on the right-hand side of equation (2) after subtracting the term p. Additionally, this is in the calculation... Figure 4A The action value Q(s) corresponding to the sample is shown. t a ～ ti Based on this, the adversarial noise distribution is calculated relative to the sample. Specifically, the adversarial noise distribution is calculated using the action value Q(s) t a ～ ti ) and adjustment coefficient α attk The value is obtained by using an exponential function of the expected value of the ratio of the values.

[0077] Alternatively, as a method different from the sample approximation, consider the following approach: Prepare a model ν to generate adversarial noise. model π (s ～ t |s t The model learns in parallel with the actor-critic method in the same way as the distribution in Equation (2). For example, the adversarial noise model can be easily obtained by updating Equation (4) in a way that minimizes the KL divergence between the noise distribution generated by the model and Equation (2).

[0078] [Number 4]

[0079]

[0080]

[0081] Here, s t ~D(·) means that the expected value is calculated by extracting multiple trajectories from the storage unit 107 only according to the batch size. Z represents the denominator (partition function) on the right side of equation (2), which does not depend on s ～ t (It is eliminated by integration), so it is treated as a constant term and does not participate in the learning of the adversarial noise distribution, thus allowing the term to be ignored.

[0082] Figure 4BThe noise distribution generated by the noise addition unit 113 is schematically represented. The noise distribution obtained from equation (3) becomes the noise distribution that minimizes the action value ( ). Figure 4B The characteristics of the left side of the distribution are similar to those of the pre-distribution. Figure 4B The distribution after the characteristics of the center) Figure 4B (on the right side). Furthermore, in Figure 4B In the example shown, with Figure 4B The sample corresponding to the peak of the distribution shown on the right corresponds to the sample representing the peak in the noise distribution that minimizes the action value. In other words, the noise at the peak of the distribution obtained by Equation (3) corresponds to the noise that minimizes the action value. However, sometimes by sampling methods such as subtracting the number of samples n, the distribution obtained by Equation (3) may not include the noise that minimizes the action value. Alternatively, the peak of the distribution obtained by Equation (3) can be the noise that is peripheral to or far from the noise that minimizes the action value, according to a pre-distributed noise.

[0083] In this embodiment, the action value Q(s) is based on the pre-distributed sampled noise. t a ～ ti In order to calculate the approximate adversarial noise distribution ν based on equation (3), ＊ (s ～ t |s t Alternatively, the action value Q(s) can be calculated using a pre-learned adversarial noise model via equation (4), without the need for angle methods or similar methods. t a ～ t The optimal calculation of the minimum value of ) is achieved. Since the optimal calculation of the angle method, which requires a large amount of computational resources, is not needed, the computational cost can be greatly reduced and the processing speed can be increased significantly.

[0084] Noise addition 113 selects (e.g., the most antagonistic) noise from the obtained adversarial noise distribution for state s. t This noise is added as a state s with added adversarial noise. ～ t And then output. In this way, it is possible to generate reasonable noise along a pre-assumed noise distribution, such as reducing the action value of the model. In other words, it is possible to provide appropriate noise that moderately selects difficult-to-handle samples from a pre-assumed noise distribution for evaluating the robustness of the model.

[0085] Furthermore, when approximating this embodiment through sampling, if the sample size *n* of the pre-distributed noise is large, the probability of obtaining the most adversarial noise that minimizes the model's action value (making it the most difficult for the model to handle) increases. On the other hand, if the sample size *n* is small, the probability of including this most adversarial noise decreases, and the probability of obtaining noise with characteristics according to the pre-distributed noise increases. In other words, when the frequency of the most adversarial noise is extremely low, the user can adjust the sample size according to the evaluation purpose to evaluate a reasonable model. Even when the frequency of the most adversarial noise is extremely low, if a robustness evaluation of the model using this noise is required, the sample size can be sufficiently increased to evaluate the model. On the other hand, it is sufficient to evaluate the model using noise with characteristics according to the pre-distributed noise; if an evaluation of noise with a relatively low frequency of occurrence is not necessarily required, the sample size can be reduced to evaluate the model within a reasonable noise range. Of course, in this case, the model evaluation can be performed at high speed.

[0086] <Noise Additional Processing Actions>

[0087] Reference Figure 5 The following describes a series of operations for approximation-based noise augmentation processing based on sampling in noise augmentation unit 113. Furthermore, the noise augmentation processing is implemented, for example, by processor 110 loading and executing a computer program stored in ROM 112 or storage unit 107 in RAM 111. Unless otherwise specified, the following processing operates in noise augmentation unit 113 as the processing entity, and at time t, the state s of the environment used in the reinforcement learning model of the object. t The time is acquired from the sensor unit 101.

[0088] In S501, the noise addition unit 113 samples n noise values ​​from the pre-distributed noise. In S502, the noise addition unit 113 generates a state s with each noise value added. ～ ti That is, the noise addition part 113 affects state s. t The additional noise samples are used to generate the perturbed state (s). ～ t1 、…、s ～ ti 、…、s ～ tn ).

[0089] In S503, the noise addition unit 113 calculates each state s ～ ti The following action a ～ ti Based on this, calculate the action a taken in state st.～ ti The value of actions Q(s) under certain circumstances t a ～ ti ).

[0090] In S504, the noise addition part 113 is based on the action value Q(s) t a ～ ti and adjustment coefficient α attk The value of ν is used to calculate the adversarial noise distribution. At this time, the noise addition unit 113 calculates the approximate adversarial noise distribution ν based on equation (3) when the divergence is KL divergence. * (s ～ t |s t ).

[0091] In S505, the noise addition unit 113 selects a noise value according to the probability weight represented by the weighting formula (3) of the adversarial noise distribution, and outputs the noise value to be added to state s. t The subsequent perturbation state s ～ t Subsequently, the noise addition unit 113 completes a series of actions. Here, in α attk When →0, it is equivalent to the action value Q(s) t a ～ ti The lowest (most aggressive and difficult to process) noise value and make α attk The case of sufficiently large (α) attk →∞), in equation (3), the contribution of the action value becomes smaller, becoming noise according to the pre-distributed p. Thus, it is possible to determine the value based on α. attk The adjustment should focus on the degree to which difficult-to-process samples are selected and evaluated with high probability.

[0092] <Actions for Robust Evaluation Processing>

[0093] Next, refer to Figure 6 A series of actions for robustness evaluation processing will be described. Furthermore, this processing is implemented, for example, by the processor 110 loading and executing a computer program stored in the ROM 112 or the storage unit 107 in the RAM 111. Unless otherwise specified, the model processing unit 114 in the following processing operates as the processing body.

[0094] In S601, the control unit 108 acquires sensor data from the sensor unit 101 at time t to obtain the state s of the environment used in the reinforcement learning model of the object. t .

[0095] In S602, the noise addition unit 113 generates an adversarial noise distribution and acquires the perturbation state s by performing the noise addition process described above. ～ t .

[0096] In S603, the model processing unit 114 determines the perturbation state s, for example, based on the actor's strategy π. ～ t The following action a t In S604, the model processing unit 114 takes action in the environment. t (e.g., output and action a) t (corresponding control quantity), obtain the new state s t+1 (For example, sensor data from sensor unit 101).

[0097] Model processing unit 114 determines in S605 based on action a t Reward r t In S606, the accumulated reward is updated. In S607, the model processing unit 114 determines whether the termination condition is met. If the termination condition is met, the processing proceeds to S608; if the termination condition is not met, the processing returns to S601 to allow the processing to resume. In the case of repeated processing, the model processing unit 114 moves the time step from t to t+1. The termination condition can be any condition, and time t can be a case where the time exceeds a predetermined time T, etc.

[0098] In S608, the model processing unit 114 compares, for example, the cumulative reward obtained without added adversarial noise with the cumulative reward in S606, and evaluates the robustness of the model relative to the adversarial noise. As described above, for example, if the cumulative reward in S606 is greater than a predetermined change in the cumulative reward without added adversarial noise, it is determined that the robustness relative to the adversarial noise is low. Thereafter, the model processing unit 114 ends this process.

[0099] Furthermore, the above explanation uses the case where a sample size n is given in advance as an example, but a setting mechanism for this sample size can also be set, which the user can set according to the evaluation.

[0100] As explained above, in this embodiment, the control unit 108 acquires noise according to a predetermined pre-distributed pattern and adds this noise to the state within the environment used in the model of the evaluation object, thereby calculating the action value of actions in the perturbated state. Then, while adding constraints using a divergence representation of the proximity of the adversarial noise distribution relative to the model of the evaluation object to the predetermined pre-distribution, the control unit 108 samples or learns an adversarial perturbation model based on the distribution of adversarial noise determined by the action value, thereby enabling approximate generation. In this way, reasonable noise, such as reducing the action value of the model, can be generated along a pre-assumed noise distribution. In other words, the model can be evaluated using appropriate noise to ensure the model's performance and robustness.

[0101] (Implementation Method 2)

[0102] Next, Embodiment 2 will be described. In Embodiment 2, an example of model learning will be described using the adversarial noise distribution described in Embodiment 1. By using the adversarial noise described in Embodiment 1 to learn the model, the robustness of the learned model can be improved. Furthermore, in Embodiment 2, except for the aspect that the control unit 108 has the configuration of the learning control unit 116 described later, and the aspect that the model learning processing is performed by the learning control unit 116, the vehicle configuration and other processing are substantially the same as in Embodiment 1. Therefore, common configurations or processing are labeled with the same reference numerals and the description is omitted, and the differences are explained in detail.

[0103] <Vehicle Composition>

[0104] Reference Figure 7 The configuration of the vehicle according to Embodiment 2 will be described. In this embodiment, the control unit 108 includes a learning control unit 116. The learning control unit 116, for example, uses... Figure 8 The model learns through reinforcement learning, as shown. Furthermore, in this embodiment, the example described uses a reinforcement learning model that constitutes the actor-critic method, but other reinforcement learning models can also be used. The learning of the reinforcement learning model executed by the learning control unit 116 will be described later.

[0105] Figure 8As an example of reinforcement learning, an offline policy-based reinforcement learning method is presented. In offline policy-based reinforcement learning, the action output 801 according to the policy, the observation of actions and states in the environment 802, and the reward determination 803 are repeated a predetermined number of times. The learning control unit 116 collects multiple sets of states, actions, rewards, and next states obtained through this repetition, forming a time-series data (trajectory 804), and stores it in the storage unit 107, for example. The learning control unit 116 retrieves the saved trajectory, updates the action value function, and then updates the policy function. The learning control unit 116 repeats the updating of the action value function and the updating of the policy function. If learning is completed, the policy function of the model used in the model processing unit 114 is updated using the learned policy function. The noise-addition processing involved in this embodiment is used in the repeatedly executed update of the action value function.

[0106] <Actions processed by the model during learning>

[0107] Reference Figure 9 The following describes a series of actions related to the learning process of the model in the learning control unit 116. Furthermore, the model learning process is implemented, for example, by the processor 110 loading and executing a computer program stored in the ROM 112 or the storage unit 107 in the RAM 111. Unless otherwise specified, the learning control unit 116 operates as the processing unit in the following processes.

[0108] In S901, the learning control unit 116 collects temporal data (trajectory) including state and action by observing actions and states in the environment. The learning control unit 116 repeats the observations of actions and states in the environment a predetermined number of times. In addition, the learning control unit 116 saves the collected trajectory in the storage unit 107, for example.

[0109] In S902, the learning control unit 116 reads the saved trajectory. In S903, the learning control unit 116 calculates the action value Q(s) based on a distribution that approximates the trajectory and adversarial noise, similar to Embodiment 1. t+1 a ～ t+1 ).

[0110] In S904, the learning control unit 116 is based on the action value Q(s) t+1 a ～ t+1 ) and adjustment coefficient α attk And calculate the objective yt(s) t a t s t+1 Specifically, using sample approximation, noise for n next states is obtained from a pre-distributed p, i.e., s is performed. ～t+1 ,i~p(·|s t+1 (i = 1, 2, ..., n), the action to be taken under this noise is obtained through the strategy (controller), i.e., a. ～ t+1 ,i~π(·|s ～ t+1 In this case, the learning control unit 116 calculates the target y according to the following equation (6). t (s t a t s t+1 In equation (6), r(s) t a t ) represents the reward, and γ represents the discounted reward. The term estimating the action value in the next state on the right side of equation (6) is the action value obtained by substituting the term inside argmin on the right side of equation (1) into equation (2) as the analytical solution. This term becomes the estimation of the action value in the next state considering adversarial noise, and becomes the form in which the smaller the action value under noise, the larger the weight is placed. Here, the method of placing the weight is also, α attk The smaller it is, the more extreme it becomes, α attk The larger the sum, the more it becomes an average sum.

[0111] Furthermore, the adversarial perturbation model ν, which is approximated in the previous implementation method 1, was used. model π (s ～ t+1 |s t+1 In the case of ), based on equation (7), the input s of the adversarial perturbation model is... t+1 Therefore, the adversarial perturbation s of the next state can be directly calculated. ～ t+1 Calculate the target y t (s t a t s t+1 ).

[0112] In S905, the learning control unit 116, according to equation (5), makes the action value function Q... θ (s t a t ) and target y t (s t a t s t+1 The parameter θ of the action value function Q is determined by minimizing the difference between s and s. Here, s t a t s t+1 ~D(·) means that the expected value is calculated by taking only multiple batch-sized trajectories from the storage unit 107.

[0113] [Number 5]

[0114]

[0115] [Number 6]

[0116]

[0117] [Number 7]

[0118]

[0119] In S906, the learning control unit 116 updates the policy function. For example, when calculating using sample approximation, the learning control unit 116 updates the policy function in a way that maximizes the action value by applying the result of actions in a state with added adversarial noise, according to equations (8) and (9). Specifically, after taking i = 1, 2, ..., n samples from the pre-distribution p, the weight coefficient w, which represents the adversarial perturbation to the weights modified (divided) from the pre-distributed weights, is used. Furthermore, even here, the adversarial perturbation model ν approximated in the previous embodiment 1 is used... model π (s ～ t |s t In the case of ), Equation (10) is used for the input state s of the adversarial perturbation model. t Therefore, the adversarial perturbation s can be directly calculated. ～ t Based on this, the policy function is updated in a way that maximizes the value of the action. This also means that s t ~D(·) retrieves only the current state s of multiple batch sizes from storage unit 107. t And calculate the expected value.

[0120] [Number 8]

[0121]

[0122] [Number 9]

[0123]

[0124] [Number 10]

[0125]

[0126] If the policy function is optimized, the learning control unit 116 updates the policy of the model processing unit 114, and then the process ends.

[0127] As explained above, in this embodiment, adversarial noise, as shown in Embodiment 1, is used during the learning of the reinforcement learning model. Specifically, the learning control unit 116 optimizes the action value function of the model being processed based on a predetermined pre-distribution and the action value of actions in a state perturbed by this adversarial noise within the environment used in the model of the processing object. At this time, the noise addition unit 113 determines the distribution of the adversarial noise based on a constraint using divergence, which represents the proximity of the distribution of the adversarial noise to the predetermined pre-distribution. This allows for the provision of appropriate noise to ensure the robustness of the model, and enables robust and high-performance model learning based on the assumed noise (pre-)distribution and degree of adversarial nature.

[0128] (Implementation Method 3)

[0129] Next, Embodiment 3 will be described. In Embodiments 1 and 2, the case where adversarial noise is added to the state in reinforcement learning (the case where the observed data contains noise) was described. In Embodiment 3, the case where it is difficult to output the correct model result due to environmental perturbations in reinforcement learning will be described. Furthermore, in Embodiment 3, the internal processing of the noise addition unit 113, the model processing unit 114, and the learning control unit 116 are different, but other configurations and processing are substantially the same as in the embodiments described above. Therefore, common configurations or processing are labeled with the same reference numerals and the description is omitted, and the differences are explained in detail.

[0130] <Evaluation of Reinforcement Learning Models>

[0131] Reference Figure 10 The evaluation of the reinforcement learning model involved in this embodiment will be explained. Sensor data is acquired at a certain time t. If sensor data is acquired, the model processing unit 114 accepts the sensor data and outputs the control quantity (policy-based action output 1001) obtained (through the execution of a machine learning algorithm). At this time, in reinforcement learning, the sensor data is equivalent to the state of the environment (s). t The control quantity is equivalent to the action taken on the environment (a) t ).

[0132] The environment is divided into {environment 1, ..., environment i, ..., environment n} based on different environmental parameters ξ (e.g., friction). Here, for illustrative purposes, it is described as environment i, but even if the environmental parameters are set as continuous variables like the friction coefficient, or discrete variables like the type of vehicle being controlled, the same classification can be performed. Furthermore, even in these multiple combinations, ξ can be treated similarly by considering it as a vector. In this embodiment, the environmental parameters are parameters according to a predetermined pre-distribution. The model processing unit 114 obtains this pre-distribution from the storage unit 107 or the communication unit 102. The pre-distribution can be various distributions assuming the distribution of environmental parameters that can be generated in the environment used in the model of the evaluation object; various distributions besides the normal distribution can be used. If the machine learning model performs actions a in each environment... t Then, through different dynamic characteristics F (states s) t+1 |Status t Action a t The environmental parameter ξ) is used to transition to a new state s. t+1 (Observations of actions and states in the environment 1002). At this time, if the model processing unit 114 selects the action value Q (state s) of the machine learning model... t Action a t If the environment parameter ξ) is the lowest, then the environment is the most difficult for the reinforcement learning model to handle. In other words, the model processing unit 114 selects the action value Q (state s) of the machine learning model. t Action a t If the environmental parameter ξ is the lowest environmental parameter, and the reward rt of the model in that environment is determined (reward determined by 1003), then robustness is evaluated in an adversarial environment.

[0133] If time progresses, the processing steps 1001 to 1003 are repeated, accumulating the reward for actions taken over multiple steps (cumulative reward 1004). The model processing unit 114 compares, for example, the cumulative reward obtained in a non-adversarial environment with the cumulative reward 1004, and evaluates the robustness of the model (robustness evaluation 1005). This means that, for example, if the cumulative reward 1004 is greater than a predetermined change in the cumulative reward in a non-adversarial environment, the action deviates from the action originally expected by the model, indicating low robustness relative to an adversarial environment.

[0134] <Environmental Changes>

[0135] Next, the changes in the environment involved in this embodiment will be explained. This process determines the state s obtained at time t. t Action a t Then the model processing unit 114 uses the following distribution: the environmental parameter ξ is distributed according to the action value function Q(s) added.t a t The objective function of the constraint after the divergence of ξ and f becomes the adversarial distribution ν(ξ|s) that is minimized. t a t That is, the distribution of equation (11).

[0136] [Number 11]

[0137]

[0138] Here, D f (ν||p) represents the f-divergence between the antagonistic distribution ν and the pre-distribution p of the environmental parameters, α attk This is an adjustment coefficient used to adjust the strength of the constraint based on this divergence. That is, it can be adjusted by α. attk The degree to which the environmental antagonistic distribution ν is constrained by the pre-distributed p.

[0139] α attk The closer the value of ν is to 0, the better. ＊ (ξ|s t a t The distribution of α, as an example, can more closely approximate a peaked distribution, such as that of environmental parameters that have a peak at a point where the action value is minimized. On the other hand, α attk The larger the value of ν, the closer it is to the pre-defined distribution p. ＊ (ξ|s t a t The distribution of α. That is, in this embodiment, if α is appropriately set... attk The value of includes the characteristics of the adversarial distribution of the environment (which minimizes the value of the action), and can be obtained in a manner that has the characteristics of a pre-distributed p.

[0140] Next, given that the divergence is defined as the KL divergence, the minimized adversarial distribution ν is... ＊ (ξ|s t a t The analytical solution of equation (12) is obtained by performing the Fincher-Legend transform on equation (11).

[0141] [Number 12]

[0142]

[0143] Here, similar to implementation 1, the sampling from the pre-distribution p(ξ) is limited to a certain number (i = 1, 2, ..., n), and can be modified by subtracting the probability weight from equation (12) to approximate it as equation (13).

[0144] [Number 13]

[0145]

[0146] This is related to the calculation Figure 11 The action value Q(s) corresponding to the sample is shown. t a i ξ i Based on this, the adversarial distribution corresponding to the sample is calculated. Specifically, this is achieved by calculating the action value Q(s). t a i ξ i ) and adjustment coefficient α attk The ratio of the values ​​is used as an exponential function of the variable to find the adversarial distribution.

[0147] Additionally, as another approximation method, an adversarial environmental parameter distribution model ν is prepared separately for the parameterized environment. model π (ξ|s t a t This allows for learning in a manner consistent with equation (12), which is an analytical solution. Similar to implementation 1, this can be easily achieved by updating the distribution of adversarial environment parameters in a way that minimizes the KL divergence equation (14) on the right-hand side of equation (12).

[0148] [Number 14]

[0149]

[0150] Similar to the implementation described above, the action value Q(s) corresponding to the perturbation sampled in the sampling-based approximation is used. t a i ξ i ), calculate the approximate adversarial distribution ν based on equation (13). * (ξ|s t a t This method directly calculates the adversarial distribution from the adversarial environment parameter distribution model. Therefore, it eliminates the need for optimal calculations related to action value using the angle method. The computationally expensive angle method optimization is unnecessary, thus significantly reducing computational costs and accelerating processing speed.

[0151] The model processing unit 114 selects an environment parameter ξ from the obtained adversarial distribution according to the probability weights in accordance with equation (12), either from the obtained sample in the case of sampling approximation or from the approximate output distribution in the case of approximation of the adversarial environment parameter distribution model, and uses the state s transitioned through this environment. t+1 Here, we choose the adjustment factor α. attk Smaller is in αattk →The environment with the lowest action value at time 0 (the most difficult to handle, the most adversarial), in α attk When the value is sufficiently large, it is selected based on the pre-assumed distribution p(ξ) of the environmental parameters. In this way, it is possible to select an appropriate environment for evaluating the robustness of the required model according to the pre-assumed distribution.

[0152] Furthermore, in this embodiment where sampling-based approximation is used, the robustness evaluation characteristics based on sample size are the same as in the embodiments described above. That is, the user can adjust the sample size according to the evaluation objective to perform a reasonable model evaluation. Even when the frequency of occurrence of the most adversarial environment is extremely low, if a robustness evaluation of the model using that environment is required, the sample size can be sufficiently increased for model evaluation. On the other hand, it is sufficient to evaluate the model in an environment according to pre-distributed characteristics; if evaluation in an environment that does not necessarily occur with extremely low frequency is required, the sample size can be reduced for model evaluation. Of course, in this case, model evaluation can be performed at high speed.

[0153] <Actions for Robust Evaluation Processing>

[0154] Next, refer to Figure 12 The following describes a series of actions for robustness evaluation processing. Furthermore, this processing is implemented, for example, by processor 110 loading and executing a computer program stored in ROM 112 or storage unit 107 in RAM 111. Unless otherwise specified, the model processing unit 114 operates as the main processing unit in the following processes.

[0155] In S1201, the control unit 108 acquires sensor data from the sensor unit 101 at time t, thereby acquiring the state s of the environment used in the reinforcement learning model of the object. t .

[0156] In S1202, the model processing unit 114 determines the state s, for example, based on the actor's strategy π. t The following action a t .

[0157] In S1203, the model processing unit 114, for example, in Figure 11 By performing the above processing, an adversarial distribution of the environment is generated, and the (selected) (e.g., most adversarial) environment parameter ξ is obtained.

[0158] In S1204, the model processing unit 114 takes action in an environment including the environmental parameter ξ. t (e.g., output and action a) t (corresponding control quantity), obtain the new state s t+1 .

[0159] In S1205, the model processing unit 114 decides based on action a t Reward r t In S1206, the accumulated reward is updated. In S1207, the model processing unit 114 determines whether the termination condition is met. If the termination condition is met, the processing proceeds to S1208; if the termination condition is not met, the processing returns to S1201 to repeat the process. In the case of repeated processing, the model processing unit 114 moves the time step from t to t+1. The termination condition can be any condition, but it can also be a case where time t exceeds a predetermined time T, etc.

[0160] In S1208, the model processing unit 114 compares, for example, the cumulative reward obtained when an environment as the evaluation benchmark is selected (e.g., a non-adversarial environment is selected) with the cumulative reward in S1206, and evaluates the robustness of the model relative to the benchmark. As described above, for example, if the cumulative reward in S1206 is greater than a predetermined change in cumulative reward when no adversarial environment is selected, it is determined that the robustness relative to the adversarial environment is low. Thereafter, the model processing unit 114 terminates this process.

[0161] <Learning Processing of Models Using Adversarial Distributions in the Context>

[0162] Next, an example of model learning using the adversarial distribution of the environment described above in Embodiment 3 will be explained. By using an adversarial environment for model learning, the robustness of the learned model can be improved. The learning of the model based on the learning control unit 116 can substantially be compared with... Figure 8 The same processing described in the text is performed.

[0163] That is, in the learning of the action value function, as shown in equation (15), the model is updated in a way that the objective y is consistent with the action value function. Here, s t a t ~D(·) means that the expected value is calculated by reading multiple tracks from the storage unit 107 only according to the batch size.

[0164] In the case of sampling-based approximation, a finite number of samples ξ are obtained from a predetermined pre-distribution p of environmental parameters. i (Here, let's set it to n elements of i = 1, 2, ..., n), using the environment parameter ξ i State s t Action a t and using the environment model T(s) t+1 |s t a t , ξ) and calculate the next state s t+1 i. Next state s t+1The calculation of i can be performed directly in a learning environment using a known environment model T, such as a simulator. Furthermore, even in an unknown environment, it can be calculated from multiple trajectory data s stored in storage unit 107. t a t s t+1 The learning prediction model is easy to compute. Based on the information obtained, the target y is calculated by Equation (16), and the action value function is updated by Equation (15).

[0165] In addition, as shown in Equation (14), when the adversarial distribution of environmental parameters in the model is approximated, as shown in Equation (17), the target y can be calculated by directly obtaining the environmental parameter ξ from the adversarial environmental parameter distribution model.

[0166] [Number 15]

[0167]

[0168] [Number 16]

[0169]

[0170] [Number 17]

[0171]

[0172] Similarly, the update of the policy function will be explained. In the case of sampling-based approximation, a finite number of samples (here, i = 1, 2, ..., n) are obtained from the predetermined pre-distribution p(ξ), and the policy function is updated in a way that maximizes the action value function, even based on environmental parameter perturbations, using equations (18) and (19). Here, as in the above implementation, w is a weight correction term that modifies the probability weights of the pre-distribution p, which uses the distribution on the right-hand side of equation (12) for sampling.

[0173] Furthermore, as shown in Equation (14), in the case of an adversarial distribution of environmental parameters in the model, as shown in Equation (20), the policy function can be updated by directly obtaining the environmental parameter ξ from the adversarial environmental parameter distribution model.

[0174] [Number 18]

[0175]

[0176] [Number 19]

[0177]

[0178] [Number 20]

[0179]

[0180] At this point, if the environmental parameter ξ remains unchanged over time, the same environmental parameter ξ can be used until the end of the learning event. In this case, Figure 11 After selecting the environmental parameters through the above processing, the same environmental parameters can be used to learn a conventional reinforcement learning model. That is, the learning control unit 116 learns the action value function and the policy function in a way that maximizes the action value of actions in the environment (determined by the selected environmental parameters). In this way, a robust model can be learned relative to the environment with the selected environmental parameters.

[0181] Furthermore, when environmental parameters ξ change over time (e.g., due to road surface friction), at each moment, Figure 11 The environmental parameters are selected through the above processing. This allows the model to learn in an environment where the adversarial environmental parameters are constantly changing.

[0182] As explained above, in this embodiment, the control unit 108 determines the adversarial distribution of the environment based on action value while adding constraints that represent the divergence between the adversarial distribution of the environment relative to the model of the evaluation object and a predetermined pre-distribution. Furthermore, the determined adversarial distribution of the environment is applied to the machine learning model of the evaluation object, and the robustness of the machine learning model of the evaluation object is evaluated based on the changes between the application of the adversarial environment and the application of the non-adversarial environment. In this way, an appropriate environment can be selected for evaluating the robustness of the model. Additionally, an appropriate environment can be used to evaluate the robustness of the model.

[0183] Furthermore, in this embodiment, selected adversarial environment parameters are used during the learning of the reinforcement learning model. Specifically, the learning control unit 116 uses a predetermined pre-distribution to determine the adversarial distribution of the environment relative to the model being processed, and optimizes the action value function of the model being processed based on the action value of actions in the adversarial environment. This allows the model to learn in a more robust manner relative to the adversarial environment. In other words, it enables the evaluation of the model's robustness or the provision of an appropriate environment for ensuring its robustness.

[0184] The invention is not limited to the above-described embodiments, and various modifications and alterations can be made within the scope of the inventive intent.

Claims

1. An information processing apparatus characterized by comprising: a decision mechanism that decides a distribution of an adversarial noise with respect to a model of a processing target using a predetermined pre-distribution; and a learning control mechanism that causes at least either of a policy function or an action value function of the model of the processing target to learn based on an action value of an action in a perturbed state in which the adversarial noise is added to a state within an environment used in the model of the processing target, the decision mechanism deciding the distribution of the adversarial noise that reduces the action value of the model of the processing target on the basis of a constraint using a divergence that indicates a closeness of the distribution of the adversarial noise to the predetermined pre-distribution. 2.The information processing apparatus according to claim 1, characterized in that the decision mechanism controls a size of the constraint by multiplying the divergence by an adjustment coefficient. 3.The information processing apparatus according to claim 1, characterized in that the divergence includes a KL divergence. 4.The information processing apparatus according to claim 1, characterized in that the learning control mechanism repeats a process of updating the policy function and the action value function of the model in order to cause the action value function of the model to learn. 5.The information processing apparatus according to claim 4, characterized in that the learning control mechanism updates the action value function of the model on the basis of the distribution of the adversarial noise decided by the decision mechanism. 6.The information processing apparatus according to claim 1, characterized in that the learning control mechanism saves, in a storage medium, time-series data at a plurality of time points obtained by repeating a predetermined number of times of a process of determining a reward based on an action in the environment, a state observation, and a reward before causing the action value function of the model to learn. 7.The information processing apparatus according to claim 1, characterized in that the information processing apparatus further comprises a setting mechanism that sets a number of samples that sample noise in accordance with the predetermined pre-distribution, the distribution of the adversarial noise is such that the larger the number of samples, the more likely it is to include noise that minimizes the action value of the model of the processing target, and the smaller the number of samples, the more likely it is to include noise in accordance with the predetermined pre-distribution. 8.The information processing apparatus according to claim 1, characterized in that the decision mechanism approximates the distribution of the adversarial noise by a modeled adversarial noise model. 9.The information processing apparatus according to claim 8, characterized in that the adversarial noise model is obtained by updating parameters of the adversarial noise model in such a manner that a divergence of the distribution of the adversarial noise and an output distribution of the adversarial noise model becomes minimum using recorded trajectory data. 10.The information processing apparatus according to claim 1, characterized in that the information processing apparatus is included in a vehicle or a robot. 11.The information processing apparatus according to claim 1, characterized in that the information processing apparatus is included in a server apparatus. ​ 12. An information processing method, each step of which is executed by an information processing apparatus, characterized by comprising: a decision step of deciding a distribution of an adversarial noise with respect to a model of a processing target using a predetermined pre-distribution; and a learning control step of causing at least either of a policy function or an action value function of the model of the processing target to learn based on an action value of an action in a perturbed state in which a state within an environment used in the model of the processing target is added with the adversarial noise, in the decision step, the distribution of the adversarial noise that reduces the action value of the model of the processing target is decided on the basis of a constraint using a divergence that indicates a closeness of the distribution of the adversarial noise to the predetermined pre-distribution.

13. A computer program product including a computer program or commands, characterized by, if the computer program or commands are executed by a processor, realizing steps of the information processing method according to claim 12.

14. A computer-readable storage medium storing a computer program or commands, characterized by, if the computer program or commands are executed by a processor, realizing steps of the method according to claim 1. ​

Images