Information processing device, information processing method, method for evaluating a machine learning model, learning method of a machine learning model and storage medium

Abstract

An information processing device takes a predetermined prior distribution of an environmental parameter, calculates an action value for an action in each of the environments perturbed by the environmental parameter, and determines an antagonistic distribution of an environment for a model to be processed based on the action value. The device determines the antagonistic distribution of the environment, which reduces the action value of the model to be processed, while adding a constraint using a divergence that indicates a proximity between the antagonistic distribution of the environment and the predetermined prior distribution.

Description

Background of the invention Field of the invention

[0001] The present invention relates to an information processing device, an information processing method, a method for evaluating a machine learning model, a learning method for a machine learning model and a storage medium. Description of the related prior art

[0002] It is known that a self-reinforcing learning algorithm may fail to work well if a state or environment that is actually to be recorded changes with respect to an assumed state or environment (for example, at the time of learning).

[0003] Therefore, a self-reinforcing learning system is proposed which generates a new state by adding noise to a recorded state and computes an action value function using the state in which the noise is added, thus enabling it to take into account variations of the state (International Publication No. 2023 / 037504).

[0004] Regarding a case where the assumed environment changes in the amplified learning model, it is desirable to evaluate robustness and train the model in such a way as to ensure robustness. In this respect, the related prior art described above has only considered adding noise to a random number to change a state. Additionally, there is often a trade-off between ensuring robustness and the performance of a control unit during normal operation, and preparing for noise that cannot occur in reality and ensuring excessive robustness can lead to an unnecessary degradation in performance. Summary of the invention

[0005] The present invention was made in view of the above problems and it is an object of it to provide a technique which is able to provide a suitable environment for evaluating or ensuring the robustness of a model.

[0006] To solve the aforementioned problems, one aspect of the present disclosure provides an information processing device comprising: a recording means configured to record a predetermined prior distribution of an environmental parameter; a computational means configured to calculate an action value of an action in each of the environments disturbed by the environmental parameter;and a determining means which is configured to determine a hostile distribution of an environment for a model to be processed based on the action value, wherein the determining means is configured to determine the hostile distribution of the environment which reduces the action value of the model to be processed, while adding a constraint using a divergence which indicates a proximity between the hostile distribution of the environment and the predetermined prior distribution.

[0007] Another aspect of the present disclosure provides an information processing device comprising: a determining means configured to determine a hostile distribution of an environment for a model to be processed using a predetermined prior distribution; and a learning control means configured to train at least one of an action value function or a precept function of the model to be processed based on an action value of an action in an environment selected from the hostile distribution of the environment, wherein the determining means is configured to determine the hostile distribution of the environment which reduces the action value of the model to be processed under a constraint using a divergence indicating a proximity between the hostile distribution of the environment and the predetermined prior distribution.

[0008] A further aspect of the present disclosure provides an information processing procedure in which each step is performed by an information processing device, the information processing procedure comprising: taking a predetermined prior distribution of an environmental parameter; calculating an action value of an action in each of environments perturbed by the environmental parameter; and determining an antagonistic distribution of an environment for a model to be processed based on the action value, wherein determining the antagonistic distribution includes determining the antagonistic distribution of the environment that reduces the action value of the model to be processed, while adding a constraint using a divergence that indicates a proximity between the antagonistic distribution of the environment and the predetermined prior distribution.

[0009] A further aspect of the present disclosure provides a method for evaluating a machine learning model, in which each step is performed by an information processing device, the method comprising: recording a predetermined prior distribution of an environmental parameter; calculating an action value of an action in each of the environments perturbed by the environmental parameter; determining a hostile distribution of an environment for a machine learning model to be processed based on the action value;and applying an environment selected from the hostile distribution of the environment to the machine learning model to be processed, in order to evaluate the robustness of the machine learning model to be processed based on a change between a case in which the environment is applied and a case in which the environment is not applied, wherein determining the hostile distribution includes determining the hostile distribution of the environment, which reduces the action value of the machine learning model to be processed, while adding a constraint using a divergence, which indicates a proximity between the hostile distribution of the environment and the predetermined prior distribution.

[0010] Yet another aspect of the present disclosure provides a learning procedure of a machine learning model in which each step is performed by an information processing device, wherein the learning procedure comprises: determining a hostile distribution of an environment for a model to be processed using a predetermined prior distribution;and training at least one from an action value function or a precept function of the model to be processed based on an action value of an action in an environment selected from the hostile distribution of the environment, wherein determining the hostile distribution includes determining the hostile distribution of the environment which reduces the action value of the model to be processed under a constraint using a divergence which indicates a proximity between the hostile distribution of the environment and the predetermined prior distribution.

[0011] A further aspect of the present disclosure provides a computer-readable storage medium which stores a program to induce a computer to execute an information processing procedure comprising: recording a predetermined prior distribution of an environment parameter; calculating an action value of an action in each of environments perturbed by the environment parameter; and determining an antagonistic distribution of an environment for a model to be processed based on the action value, wherein determining the antagonistic distribution includes determining the antagonistic distribution of the environment which reduces the action value of the model to be processed, while adding a constraint using a divergence which indicates a proximity between the antagonistic distribution of the environment and the predetermined prior distribution.

[0012] According to the present invention, it is possible to provide the technology which is capable of providing a suitable environment for evaluating or ensuring the robustness of a model. Brief description of the drawings Fig. Figure 1 is a block diagram illustrating a functional configuration example of a vehicle according to a first embodiment; Fig. 2 is a view for describing a relationship between functional configurations in a robustness evaluation according to the first embodiment; Fig. 3 is a view describing an evaluation of a self-reinforcing learning model according to the first embodiment; Fig. 4A is a view describing details of adding noise according to the first embodiment; Fig. 4B is a view for describing a noise distribution generated according to the first embodiment; Fig. Figure 5 is a flowchart illustrating a series of operations of an addition processing for noise according to the first embodiment; Fig. Figure 6 is a flowchart illustrating a series of operations of a robustness evaluation process according to the first embodiment; Fig. Figure 7 is a diagram describing a main configuration of a vehicle according to a second embodiment; Fig. Figure 8 is a view describing a learning procedure of a self-reinforcing learning model according to the second embodiment; Fig. Figure 9 is a flowchart illustrating a series of processes of a model learning process according to the second embodiment; Fig. 10 is a view describing an evaluation of a self-reinforcing learning model according to a third embodiment; Fig. 11 is a view describing a process for modifying an environment according to the third embodiment; and Fig. Figure 12 is a flowchart illustrating a series of operations of a robustness evaluation processing according to the third embodiment. Description of the embodiments

[0013] The following embodiments will be described in detail with reference to the accompanying drawings. It should be noted that the following embodiments are not intended to limit the scope of the claimed invention, and no limitation is imposed to an invention requiring a combination of all features described in the embodiments. Two or more of the several features described in the embodiments can be suitably combined. Furthermore, the same reference numerals are assigned to the same or similar configurations, and redundant descriptions thereof are omitted. (First embodiment)

[0014] In a first embodiment, a robustness evaluation according to the present embodiment will be described. In the robustness evaluation described here, it is assessed whether a machine learning model under evaluation exhibits robustness with respect to an input that includes noise. Therefore, in the present embodiment, appropriate noise is added to perform such a robustness evaluation. It should be noted that a case in which the present invention is implemented in a vehicle will be described as an example in the embodiments described below. However, the embodiments described below can be implemented in one or more information processing devices, such as a server device.In addition, the vehicle described below includes a four-wheeled or two-wheeled passenger vehicle, and also includes a vehicle that carries a person or moves towards a person without a person entering the vehicle. Furthermore, the embodiments described below can also be applied to a robot that can move autonomously or according to a process, in addition to the vehicle described above. The embodiments described below are not limited to a device that can move on its own and are also applicable to a robot that moves an object (for example, a robot arm), as well as an information processing device (control device that directly or remotely controls a movable device). <fahrzeugkonfiguration>

[0015] First, a functional configuration example of a vehicle 100 according to the present embodiment is given with reference to Fig. 1. It should be noted that each of the functional blocks to be described with reference to the following drawings can be integrated or separate. Furthermore, a function to be described can be implemented in a different block. Additionally, a functional block described as hardware can be implemented by software, and vice versa.

[0016] In the following example, a case in which a control unit 108 is integrated into the vehicle 100 will be described as an example, and the control unit 108 of the vehicle 100 can be configured as a control module or an information processing device comprising a configuration of the control unit 108. That is to say, the present invention can be implemented as a control module or an information processing device comprising configurations such as a processor 110 and a model processing unit 114, which are included in the control unit 108.

[0017] A sensor unit 101 comprises various sensors provided in the vehicle 100 and outputs sensor data regarding the vehicle 100's behavior. These sensors include, for example, a vehicle speed sensor for measuring the vehicle 100's speed, an accelerometer for measuring the vehicle's acceleration, and a suspension displacement sensor for measuring the shock absorber's movement (speed or displacement). Additionally, it includes a steering angle sensor, which measures steering input, a sensor that measures torque generated by a power unit 105, a GPS unit that records its own position, and similar sensors. Furthermore, the sensor unit 101 may include a camera (an image capture unit) that outputs a captured image of the view in front of the vehicle 100 (or views in front of, beside, and behind the vehicle).The sensor unit 101 can also include a light detection and distance measurement (Li-DAR) system which outputs a range image obtained by measuring a distance to an object in front of the vehicle (or distances to objects in front of, beside and behind the vehicle).

[0018] One or more pieces of sensor data, such as acceleration, position information, steering angle, torque, a captured image and the range image of the vehicle 100, are used as one of states to control an action of the vehicle by a self-reinforcing learning model, which is included, for example, in the model processing unit 114.

[0019] A communication unit 102 is a communication device that, for example, includes a communication circuit and communicates with an external information processing server, a transport system arranged around the vehicle, and similar devices, for example, via Long Term Evolution (LTE), LTE-Advanced, or mobile communication, which is standardized as the so-called fifth-generation mobile communication system (5G). For example, the communication unit 102 receives some or all of the map data, traffic information, and similar data from another information processing server or the transport system arranged around the vehicle.The communication unit 102 can, for example, receive from the external information processing server at least one arbitrary hyperparameter of a learning model used by the model processing unit 114, a learned parameter, a previous distribution of noise, a previous distribution of an environment parameter to be described later, or similar.

[0020] An actuation unit 103 comprises an actuation element, such as a button or a touch panel, installed in the vehicle 100, as well as elements that receive input for driving the vehicle 100, such as a steering wheel and a brake pedal. A power supply unit 104 comprises a battery, such as a lithium-ion battery, and supplies electrical power to each unit in the vehicle 100. The power unit 105 comprises, for example, a motor or an internal combustion engine, which generates power to propel the vehicle. A notification unit 106 notifies an occupant (or a driver) of a predetermined sound, such as a warning sound.

[0021] A memory unit 107 comprises a high-capacity, non-volatile storage device, such as a semiconductor memory. Various types of sensor data output by the sensor unit 101 are temporarily stored. Additionally, a learned parameter of a machine learning model, executed by the model processing unit 114, and information about a trajectory are stored. This trajectory comprises a set of actions and states of a reinforcing learning process, which will be described later.

[0022] The control unit 108, for example, comprises the processor 110, random access memory (RAM) 111, and read-only memory (ROM) 112, and controls the operation of each unit of the vehicle 100. Additionally, the control unit 108 can receive sensor data from the sensor unit 101 and execute a process of controlling an action of the vehicle 100 by means of the augmenting learning model, which will be described later, as well as a process of evaluating the robustness of the augmenting learning model. The control unit 108 causes each unit, such as the model processing unit 114, which is included in the control unit 108, to perform its function by instructing the processor 110 to transfer a computer program stored in the ROM 112 to the RAM 111 and to execute the computer program.

[0023] The processor 110 comprises one or more processors, such as a CPU. In addition to the CPU, the processor 110 may include other processors or circuits, such as a graphics processing unit (GPU) and an application-specific integrated circuit (ASIC) for performing processing by the model processing unit 114 at high speed. The RAM 111 comprises a volatile memory medium, such as dynamic RAM (DRAM), and acts as the working memory of the processor 110. The ROM 112 comprises a non-volatile memory medium and stores a computer program to be executed by the processor 110, a set value to be used when the control unit 108 is operated, and similar data.

[0024] A noise addition unit 113 generates hostile noise and adds it to the sensor data (for example, a torque or a captured image) received from the sensor unit 101. The hostile noise can be referred to as a hostile sample or similar. The hostile noise is obtained by identifying an input for which a trained model cannot produce an optimal result or for which a predictive evaluation value is low (performance is low). When the hostile noise is added to the model, the robustness of the model can be evaluated, or the model can be trained to become more robust.It should be noted that noise addition using the noise addition unit 113 is performed when the robustness of a machine learning model is being evaluated by the model processing unit 114 or when the machine learning model is being trained. This means that the noise addition unit 113 is not used during vehicle 100 movement that does not involve evaluating the machine learning model. In this case, the sensor data output by the sensor unit 101 can be input to the model processing unit 114.

[0025] The model processing unit 114 executes a machine learning model, which implements a self-reinforcing learning algorithm, and determines an action for the vehicle 100, which uses the sensor data. For example, an instruction to control the power unit 105 (to control acceleration / deceleration or steering) is issued using the sensor data, such as torque or the captured image. It should be noted that this control example is just that—an example—and any instruction can be issued using any sensor data.

[0026] An action control unit 115 controls the movement of the vehicle 100 based on the action instruction issued by the model processing unit 114. For example, the action control unit 115 controls the power unit 105 according to the action instruction for controlling the power unit 105 issued by the model processing unit 114. Although the model processing unit 114 and the action control unit 115 are described separately in the present embodiment, the action control unit 115 can be included within the model processing unit 114.

[0027] Fig. Figure 2 illustrates a relationship between key functional configurations in the robustness evaluation according to the first embodiment. For example, the sensor unit 101 acquires sensor data (e.g., a torque). The noise addition unit 113 adds hostile noise, which will be described later, to the sensor data. The model processing unit 114 determines an action output (e.g., a control quantity of acceleration / deceleration or steering) according to an action to be performed by the vehicle 100, using the sensor data to which the hostile noise has been added. The action control unit 115 controls the power unit 105 according to the action output by the model processing unit 114. <Evaluation des sich verstärkenden Lernmodells>

[0028] Next, an evaluation of a self-reinforcing learning model according to the present embodiment will be presented with reference to Fig. 3. At a specific time t, sensor data is acquired. When the sensor data is acquired, the noise addition unit 113 adds hostile noise, which is described later, to the sensor data (hostile noise addition 301). The model processing unit 114 receives the sensor data to which the hostile noise is added and outputs a control quantity that has been obtained (by executing a machine learning algorithm) (action output according to rule 302). At this time, in the reinforcing learning process, the sensor data corresponds to a state (st) of an environment, and the control quantity corresponds to an action (at) with respect to the environment. Additionally, the hostile noise is added to the state (st) to obtain a state (s~t) in which hostile noise is added.

[0029] According to this, when the action control unit 115 controls the power unit 105 based on the control quantity, new sensor data are recorded at time t+1 (action and state observation in the environment 303). In the reinforcing learning process, this sensor data corresponds to a state (s t+1 ) in the environment. The model processing unit 114 determines a reward (rt) (or penalty) in the reinforcing learning process based on sensor data from sensor unit 101 (reward determination 304). The reward is, for example, a reward value for a behavior of the vehicle, which is obtained from a combination of pieces of predetermined sensor data. As time passes, the processing from 301 to 304 is repeated, and a reward for an action over multiple steps is accumulated (cumulative reward 305). For example, the model processing unit 114 compares a cumulative reward obtained in a case where no hostile noise is added with the cumulative reward 305 and evaluates robustness with respect to the model (robustness evaluation 306).For example, in a case where the cumulative reward 305 has changed by a predetermined value or more from the cumulative reward that would be received in the case where no hostile noise is added, this means that an action of the model deviates from an originally expected action, indicating that the robustness against hostile noise is low.

[0030] Model processing unit 114, for example, operates a self-reinforcing learning model that forms actor-critic units. An actor selects an action (a) based on a rule π(a|s). A critic is a mechanism for evaluating the rule π(a|s) currently used by the actor and exhibits an action-value function Q. π (s,a) represents a reduced reward sum expected when an action a is taken in a state s, for example, under a rule π. It should be noted that in actor-critic, the critic is trained to evaluate a rule while the actor is improved to determine an action, as will be described later. However, a procedure other than actor-critic, such as Q-learning or DQN, can be used concurrently if it is taken into account that the actor selects one with the largest evaluation value (the largest value of the action value function) from a plurality of action candidates for a procedure in which an action output is discrete and an optimal action is selected from the plurality of action candidates. <details des hinzufügens von rauschen>

[0031] Next, details of adding noise according to the present embodiment will be given with reference to Fig. 4A will be described. This processing is implemented in the noise addition unit 113 for a state s. t The noise addition unit 113 takes a previous distribution of noise, described below, from the storage unit 107 or the communication unit 102.

[0032] For example, in a case where approximation is performed by sampling, the noise addition unit 113 performs sampling of noise into n pieces of data according to a predetermined prior distribution. The prior distribution can be various distributions that take on a noise distribution that might occur in an environment used in a model to be evaluated, and various distributions can be used in addition to a normal distribution.

[0033] The noise addition unit 113 adds each sampled noise to state s t in addition, to disturbed states (s~ t1 , ...,s~ ti , ... and s~ tn ) to generate. It should be noted that the superscript “~” indicates a value that is affected by noise. After calculating actions a~ ti In the disturbed states (using the actor), the noise addition unit calculates 113 action values ​​Q(s). t ,a~ ti ) the actions in a case in which the actions a~ ti in that state s t undertaken (using the critic).

[0034] The noise-adding unit 113 computes a hostile noise distribution for the model (a self-reinforcing learning model) to be evaluated. The hostile noise distribution is obtained by finding a noise distribution that minimizes the model's action value (i.e., noise that weakens the model), while adding a constraint using a divergence that represents the proximity between the computed hostile noise distribution and the predetermined prior distribution. A hostile noise distribution v*(s~ t |s t ), which has an action value in the state s~ t minimized, is obtained by the following formula (1) if the divergence is an f-divergence. [Formula 1] vπ*(st∼|st)=argminv∈N Est∼∼v[Wat∼∼π(⋅|st∼)[Qπ∼(st,at∼)]] +αattkDf(v(⋅|st)‖p(⋅|st))

[0035] Here, D f (v||p) the f-divergence between a distribution v and a previous distribution of noise p, and α attk is an adjustment factor for adjusting the strength of the restriction imposed by the divergence. That is, in formula (1), the degree to which the distribution of the noise v is restricted by the preceding distribution p is given by α. attk can be adjusted if the distribution of the noise, which has an expected value of an action value Q π∼ (st, a~ t ) of an action minimized in a case in which the action a~ t in that state s t undertaken, will be obtained.

[0036] If a value of α attk closer to 0, the distribution of v*(s~t|s can change t ) approximate a jagged distribution which has a peak at a value of noise that minimizes the action value, as an example. On the other hand, if the value of α attk larger, the distribution v*(s~ t |s t ) closer to the distribution of the previous distribution p. That is, in the present embodiment, when the value of α attk If appropriately set, it is possible to obtain the distribution of enemy noise which is adapted to exhibit characteristics of the preceding distribution p, while including characteristics of the noise distribution (which minimizes the action value).

[0037] Next, assuming the divergence is a Kullback-Leibler (KL) divergence, an analytical solution of the minimized hostile noise distribution v*(s~ t |s t ) from formula (1) can be expressed by formula (2) using the Legendre-Fenchel transformation. However, in a case where a continuous state space and a continuous action space are involved, a calculation is difficult, and even if an approximation is performed using a known method, such as a Markov chain Monte Carlo method, it is necessary to refer to the rule π and the action value function Q. π Accessing the computer multiple times to calculate a value at each time t presents the problem that the computation costs are high. [Formula 2] vπ*(st∼|st)=p(st∼|st)exp(Eat∼∼π(⋅|st)[−Qπ(st,at∼) / αattk ])∫st∼p(st∼|st)exp(Eat∼∼π(⋅|st∼)[−Qπ(st,at∼) / αattk])dst∼

[0038] Therefore, one of the present embodiments is an approximate distribution of hostile noise v*(s~ t |s t ) obtained by formula (3) by applying formula (2) using a limited number (for example, 1 to n, as described in Fig. 4A) of sample values ​​from a previous distribution p(s~ t |s t ) is used for noise. [Formula 3] vπ*(st∼|st)≅vπ*(sti∼|st)∝exp(Eati∼∼π(⋅|sti∼)[−Qπ(st,ati∼)αattk])

[0039] This is obtained by subtracting the term p from the numerator on the right-hand side of formula (2) to give a weight for each sample according to the preceding distribution p(s~ t |s t ) to correct which is used for sampling from the previous distribution to the distribution of enemy noise. Additionally, this corresponds to calculating the action values ​​Q(st, a~t). i ), which correspond to the samples which are in Fig. 4A are illustrated, and then a calculation of the enemy noise distribution for the samples. In particular, the enemy noise distribution is obtained by using an exponential function with respect to an expected value of a ratio between the action value Q(s) t , a~t i ) and a value of the adjustment factor α attk is calculated.

[0040] Additionally, a procedure for separately preparing a model v is provided. mo-del π(s∼t|s t The process of generating hostile noise and training the model in parallel with actor-critic learning, in order to exhibit the same function from formula (2), is considered a method different from the sample approximation. This can be easily obtained, for example, by updating the hostile noise model to minimize formula (4) based on the noise distribution generated by the model and the KL divergence from formula (2). [Formula 4] Loss=Est∼D(⋅)[DKL(vπmodel(⋅|st)|p(⋅|st)exp(Eat∼∼π(⋅|st∼)[−Qπ(st,at∼)αattk])Z)] ∝Est∼D(⋅)[Est∼∼vπmodel(⋅|st)[αattklog vπmodel(sπ∼|st)−αattklog p(sπ∼|st) +Eat∼∼π(⋅|st∼)[Qπ(st,at∼)]+const.]]

[0041] Here, s t ~D(·), that a plurality of trajectories are extracted from storage unit 107 by the size of a stack to calculate an expected value. Z denotes the denominator (distribution function) on the right-hand side of formula (2), and this expression does not depend on s~ t ab (is fully integrated), and therefore does not contribute to learning the distribution of hostile noise as a constant expression const. and can be ignored.

[0042] Fig. Figure 4B schematically illustrates a noise distribution generated by the noise addition unit 113. The noise distribution obtained by formula (3) is a distribution (right in Fig. 4B), in which the characteristics of the noise distribution (left in Fig. 4B), which minimizes the action value, and the characteristics of the previous distribution (middle in Fig. 4B) are added. It should be noted that in the Fig. 4B illustrated example shows a sample that corresponds to a peak in the distribution, which is on the right in Fig. Figure 4B illustrates a sample that exhibits a peak in the noise distribution, minimizing the action value. This means that the noise at the peak of the distribution obtained by formula (3) corresponds to noise that minimizes the action value. However, there is a case where the distribution obtained by formula (3) does not include noise that minimizes the action value, depending on how samples are taken, such as by reducing the number n of samples. Instead, the peak of the distribution obtained by formula (3) may be noise around the action-value-minimizing noise, or away from the noise, according to the previous distribution.

[0043] In the present embodiment, since the approximate distribution of the hostile noise v*(s~ t |s t ) according to formula (3) based on the action value Q(s t , a~t i ) is calculated, which corresponds to the noise obtained by sampling the previous distribution or the enemy noise model, which is previously trained according to formula (4), and is used for the calculation; it is not necessary to perform an optimization calculation to find the minimum value of the action value Q(s). t ,a~ t ) using a gradient descent method or similar. Since the optimization calculation using the gradient descent method, which requires high computational costs, becomes unnecessary, the computational costs can be largely reduced, and the processing speed can be increased.

[0044] The noise addition unit 113 selects (for example, the most hostile) noise from the received distribution of hostile noise and adds the noise to the state s. t add and gives the state s~ t from which hostile noise is added. In this way, it is possible to generate meaningful noise that reduces the action value of the model while adhering to the previously assumed noise distribution. In other words, it is possible to add suitable noise by selecting a sample that is appropriately weak in the previously assumed noise distribution in order to evaluate the robustness of the model.

[0045] Additionally, in a case where the approximation is performed by sampling in the present embodiment, increasing the number n of samples used to sample noise from a preceding distribution greatly increases the probability of obtaining the most hostile noise, which minimizes the action value of a model (makes the model the weakest). Conversely, if the number n of samples is small, the probability of capturing the most hostile noise decreases, and the probability of obtaining noise corresponding to the characteristics of the preceding distribution increases. This means that if the frequency of occurrence of the most hostile noise is extremely low, a user can perform an appropriate model evaluation by adjusting the number of samples according to the purpose of the evaluation.Even if the frequency of occurrence of the most hostile noise is extremely low, a model is evaluated by sufficiently increasing the number of samples in cases where evaluating the model's robustness using noise is required. Conversely, in cases where it is sufficient to perform an evaluation using noise according to the characteristics of the preceding distribution, and evaluation for noise with an extremely low frequency is not necessarily needed, a model can be evaluated within a reasonable range of noise by reducing the number of samples. Naturally, in this case, the model evaluation can be performed at high speed. <Vorgang einer Rauschen-Hinzufügungsverarbeitung>

[0046] A series of operations of a processing for adding noise using the approximation by sampling in the noise addition unit 113 is described with reference to Fig. 5. It should be noted that the noise addition processing is implemented, for example, by the processor 100, which transfers a computer program stored in the ROM 112 or the memory unit 107 to the RAM 111 and executes the computer program. Unless otherwise specified, the following processing is started when the noise addition unit 113 is operating as a processing unit, and the state s t An environment used in a reinforcing target learning model is recorded by sensor unit 101 at time t.

[0047] In S501, the noise addition unit 113 samples n noise values ​​from a previous noise distribution. In S502, the noise addition unit 113 generates states s~ ti , to which the corresponding noise values ​​are added. This means that the noise addition unit 113 adds each sampled noise to the state s t adds to disturbed states (s~ t1 , ...,s~ ti , ... and s~ tn to generate.

[0048] In S503, the noise addition unit 113 calculates the actions a~ ti in the respective states s~ ti and then calculates the action values ​​Q(s t ,a~ ti ) the actions in a case in which the actions a~ ti in that state s t undertaken.

[0049] In S504, the noise addition unit 113 calculates a distribution of enemy noise based on the action values ​​Q(s). t , a~t i ) and a value of the adjustment factor α attk At this point, the noise addition unit 113 calculates the approximate distribution of hostile noise v*(s~ t |s t ) according to formula (3) in a case where a divergence is the KL divergence.

[0050] In S505, the noise addition unit 113 selects a noise value according to a probability weight represented by formula (3) of the weight of the hostile noise distribution and outputs the disturbed state s~ t from which, by adding the noise value to the state s t is obtained. The noise addition unit 113 then terminates the sequence of processes. In one case, it corresponds to α. attk → 0 a selection of the (most hostile and weakest) noise value with the lowest action value Q(s t , a~ ti ), and in a case where α attk is sufficiently large (α attk → ∞), the contribution of the action value in formula (3) is small, and noise according to the preceding distribution p is obtained. In this way, it is possible to continuously detect any degree of weakness of a selected and evaluated sample with high probability according to α attk to adapt. <Vorgang der Robustheit-Evaluationsverarbeitung>

[0051] Next, a series of processes for robustness evaluation processing will be described with reference to Fig. 6. It should be noted that this processing is implemented, for example, by the processor 110, which transfers a computer program stored in the ROM 112 or the memory unit 107 to the RAM 111 and executes the computer program. The model processing unit 114 performs the following processing as a processing unit, unless otherwise specified.

[0052] In S601, the control unit 108 receives sensor data from the sensor unit 101 at time t and records the state s t an environment which is used in a reinforcing target learning model.

[0053] In S602, the noise addition unit 113 performs the noise addition processing described above to generate a distribution of hostile noise and the disturbed state s~ t to obtain.

[0054] In S603, the model processing unit 114 determines the action at in the disturbed state s~ t according to, for example, the actor's rule π. In S604, the model processing unit 114 performs the action at in the environment (for example, outputs a control quantity corresponding to the action at) and assumes a new state s. t+1 on (for example, sensor data from sensor unit 101).

[0055] Model Processing Unit 114 determines a reward rt for the action at in S605 and updates a cumulative reward in S606. In S607, Model Processing Unit 114 determines whether a termination condition is met, transfers the processing to S608 if the termination condition is met, and returns the processing to S601 to repeat the processing if it is not. When the processing is repeated, Model Processing Unit 114 transfers the time from t to t+1. The termination condition can be any condition, but could, for example, be a case in which time t exceeds a predetermined time T, or something similar.

[0056] In S608, for example, the model processing unit 114 compares a cumulative reward received in a case where no hostile noise is added with the cumulative reward in S606 and evaluates the robustness of the model. As described above, for example, if the cumulative reward in S606 has changed by a predetermined value or more compared to the cumulative reward in the case where no hostile noise is added, it is determined that the robustness against hostile noise is low. The model processing unit 114 then terminates the current processing.

[0057] It should be noted that a case in which the number n of samples is given beforehand has been described as an example in the above description, but that a setting unit which sets the number of samples may be provided so that the user can set the number of samples according to the evaluation.

[0058] As described above, in the present embodiment, the control unit 108 acquires noise according to the predetermined prior distribution, adds the noise to the environment used in the model to be evaluated, and calculates the action value of the action in the perturbed state. The control unit 108 can then approximate and generate the hostile noise distribution, which is determined based on the action value by sampling or training a hostile perturbation model, while adding the constraint using the divergence that indicates the proximity between the hostile noise distribution and the predetermined prior distribution for the model to be evaluated. In this way, it is possible to generate meaningful noise that reduces the action value of the model while adhering to the previously assumed noise distribution.In other words, the model can be evaluated using appropriate noise to ensure the model's performance and robustness. (Second embodiment)

[0059] Next, a second embodiment will be described. In the second embodiment, an example will be described in which a model is trained using a hostile noise distribution as described in the first embodiment. By training the model using the hostile noise described in the first embodiment, the robustness of the trained model can be improved. It should be noted that in the second embodiment, the vehicle configuration and other processing are essentially the same as those of the first embodiment, except that the control unit 108 has a configuration of a learning control unit 116, which will be described later, and model learning processing is performed by the learning control unit 116.Therefore, the same configurations and processing operations are designated with the same reference symbols, and since no description of such a configuration and processing operation is given, the following description focuses mainly on differences. <fahrzeugkonfiguration>

[0060] The configuration of the vehicle according to the second embodiment is described with reference to Fig. 7 will be described. In the present embodiment, the control unit 108 comprises the learning control unit 116. The learning control unit 116 trains a model, for example, by means of a Fig. Figure 8 illustrates reinforcing learning. It should be noted that the description of the present embodiment uses, for example, a case in which an actor-critic reinforcing learning model is employed, but that a different reinforcing learning model can be used. Training of the reinforcing learning model, which is executed by the learning control unit 116, will be described later.

[0061] Fig. Figure 8 illustrates a learning process of reinforcing off-rule learning as an example of reinforcing learning. In reinforcing off-rule learning, an action output 801 according to a rule, an action and state observation in an environment 802, and a reward determination 803 are repeated a predetermined number of times. The learning control unit 116 stores, for example, time-series data (trajectory 804) in the memory unit 107. This data is obtained by collecting multiple sets of states, actions, rewards, next states, and the like, which are obtained through repetition. The learning control unit 116 extracts the stored trajectory, updates an action value function, and then updates a rule function.The learning control unit 116 repeatedly updates the action value function and the rule function and updates a rule function of the model, which is used in the model processing unit 114, with the trained rule function when the learning is complete. A noise addition processing according to the present embodiment is used in the update of the action value function, which is executed repeatedly. <Vorgang der Modell-Lernverarbeitung>

[0062] A number of processes of model learning processing in the learning control unit 116 are described with reference to Fig. 9. It should be noted that the model learning processing is implemented, for example, by the processor 110, which transfers a computer program stored in the ROM 112 or the memory unit 107 to the RAM 111 and executes the computer program. The learning control unit 116 performs the following processing as a processing unit, unless otherwise specified.

[0063] In S901, the learning control unit 116 collects time-series data (trajectories) encompassing states and actions by observing actions and states within an environment. The learning control unit 116 repeats this action and state observation in the environment a predetermined number of times. Additionally, the learning control unit 116 stores, for example, the collected trajectories in the storage unit 107.

[0064] In S902, the learning control unit 116 reads the stored trajectory. In S903, the learning control unit 116 calculates an action value Q(s). t+1 , a~ t+1 ) similar to the first embodiment based on the trajectory and a distribution that approximates hostile noise.

[0065] In S904, the learning control unit 116 calculates a target y. t (s t , a t , s t+1 ) based on the action value Q(s t+1 ,a~ t+1 ) and the adjustment factor α attk In particular, when using a sample approximation, noise from n nearest states is picked up from the preceding distribution p, i.e., s~ t+1,i ~p(·|s t+1 ) (where i = 1, 2, ..., and n) is obtained and an action is taken under the noise, i.e. a~ t+1,i ~π(·|s~ t+1,i ) is obtained according to a rule (control unit). In this case, the learning control unit 116 calculates the target y. t (s t , at, s t+1 ) according to the following formula (6). In formula (6), r(st, at) represents a reward and γ represents a reduced reward. An expression estimating an action value in a next state on the right-hand side of formula (6) is a value of the action value obtained as a result of substituting formula (2), which is an analytical solution, into an expression argmin on the right-hand side of formula (1). This expression is the estimate of the action value in the next state taking into account hostile noise and has a form in which a greater weight is added as the action value decreases under the noise. Here, the way in which the weight is added becomes more extreme as α attk is smaller, and an average addition is performed when α attk is larger.

[0066] Additionally, in a case where a hostile perturbation model v model π(s∼ t+1 |s t+1 ), which is approximated in the first embodiment, is used, s t+1 to the hostile disturbance model based on formula (7) in order to directly generate a hostile disturbance s~ t+1 to calculate the next state and the target y t (s t , at, s t+1 to calculate.

[0067] In S905, the learning control unit 116 determines a parameter θ of the action value function Q in order to calculate a difference between an action value function Qe(st, at) and the goal y. t (s t , at, s t+1 ) according to formula (5) to minimize . Here, “s” denotes t , a t , s t+1 ~D(·)", that a plurality of trajectories are extracted from the storage unit 107 by a size of a stack to compute an expected value. [Formula 5] L(Q)=Est,at,st+1∼D(⋅)[|y(st,at,st+1)−Qπ(st,at)|2] [Formula 6] y(st,at,st+1)=r(st,at)+γ[−αattk log(1N∑i=1nexp(−Qπ(st+1,at+1,i∼)αattk))] [Formula 7] y(st,at,st+1)=r(st,at)+γEst+1∼∼vπmodel(⋅|st+1)[Eat+1∼∼π(⋅|st+1∼)[Qπ(st+1,at+1∼)]]

[0068] In S906, the learning control unit 116 updates a precept function. For example, in a case where the computation is performed using the sample approximation, the learning control unit 116 updates the precept function such that the outcome of an action in a state to which hostile noise is added maximizes the action value according to formulas (8) and (9). In particular, a weighting factor w, obtained by taking n samples from the preceding distribution p (i = 1, 2, ... and n) and then correcting (dividing) the weight of the preceding distribution by the hostile disturbance, expressed by formula (2), is used. In this case, where the hostile disturbance model v model π(s∼ t |s t ), which is approximated in the first embodiment, is used, the state s t to the hostile disturbance model using formula (10) to directly calculate the hostile disturbance s~ t to calculate, and the regulatory function is updated to determine the action value based on the hostile disturbance s~ t to maximize. Here, s denotes t ~D(·), that a plurality of instantaneous states s t The expected value is extracted from memory unit 107 by the size of the stack. [Formula 8] J(π)=Est∼D(⋅)[∑i=1nw(st,i∼|st)Eat,i∼∼π(⋅|st,i∼)[Qπ(st,at,i∼)]] [Formula 9] w(st,i∼|st)∝ exp(Eat,i∼∼π(⋅|st,i∼)[−Qπ(st,at,i∼)αattk]) [Formula 10] J(π)=Est∼D(⋅)[Est∼∼vπmodel(⋅|st)[Eat∼∼π(⋅|st∼)[Qπ(st,at∼)]]]

[0069] When the rule function is optimized, the learning control unit 116 updates the rule of the model processing unit 114 and then terminates the current processing.

[0070] As described above, in the present embodiment, the hostile noise described in the first embodiment is used during the training of the amplified learning model. That is, the learning control unit 116 optimizes the action value function of the model to be processed based on a predetermined prior distribution and the action value of the action in the perturbed state, which is obtained by adding the hostile noise to the state in the environment used in the model to be processed. At this time, the noise addition unit 113 determines the distribution of the hostile noise under a constraint using a divergence that represents the proximity between the hostile noise distribution and the predetermined prior distribution.In this way it is possible to add appropriate noise to ensure the robustness of the model, and it is possible to train a model whose robustness and performance have been ensured according to the assumed (previous) distribution of the noise and the degree to which the noise is hostile.

[0071] Next, a third embodiment will be described. In the first and second embodiments, a case has been described in which hostile noise is added to a state in self-enhancing learning (a case in which noise is added to observed data). In the third embodiment, a case will be described in which it is difficult for a model to output a correct result due to a disturbance in a self-enhancing learning environment. It should be noted that the processing within the noise-adding unit 113, the model processing unit 114, and the learning control unit 116 differs in the third embodiment, but that other configurations and processing are essentially the same as those of the embodiments described above.Therefore, the common configurations and processing will be referred to with the same reference symbols, and since no description of such configurations and processing will be given, the following description focuses mainly on differences. <Evaluation eines sich verstärkenden Lernmodells>

[0072] An evaluation of a self-reinforcing learning model according to the present embodiment is carried out with reference to Fig. 10 can be described. At a specific time t, sensor data is acquired. When the sensor data is acquired, the model processing unit 114 receives the sensor data and outputs a control quantity that has been obtained (by executing a machine learning algorithm) (action output according to rule 1001). At this time, in the reinforcing learning process, the sensor data corresponds to a state (s). t ) an environment, and the tax quantity corresponds to an action (at) with respect to the environment.

[0073] The environment is distinguished as {environment 1, ..., environment i, ..., and environment n} based on a difference in an environment parameter ξ (for example, friction). Although described in the form of an environment i for descriptive purposes, environment parameters can be treated as continuous variables, such as coefficients of friction, or as discrete parameters, such as vehicle types of vehicles to be controlled. Additionally, even in multiple combinations thereof, ξ can be considered a vector, which is to be treated in the same way. In the present embodiment, it is assumed that the environment parameters follow a predetermined prior distribution. The model processing unit 114 receives the prior distribution from the storage unit 107 or the communication unit 102.The preceding distribution can be various distributions if a distribution of environmental parameters is assumed that might occur in an environment used in a model being evaluated, and various distributions can be used in addition to a normal distribution. When a machine learning model performs the action `at` in environments, the state transitions to a new state `s`. t+1 through different dynamics F over (state s t+1 |Condition s t , action at; environment parameter ξ) (action and state observation in environment 1002). At this time, when the model processing unit 114 selects an environment in which an action value Q(state s) exists t , action at; environment parameter ξ) of the machine learning model is lowest, the environment is given which is the weakest for the reinforcing learning model. That is, if the model processing unit 114 selects an environment in which the action value Q(state s) is lowest, the environment is given which is the weakest for the reinforcing learning model. That is, if the model processing unit 114 selects an environment in which the action value Q(state s) is lowest, the environment is the weakest for the reinforcing learning model. t , action at; environment parameter ξ) of the machine learning model is the lowest and then determines the reward rt of the model in the environment (reward determination 1003), which evaluates robustness in a hostile environment.

[0074] As time passes, the processing from 1001 to 1003 is repeated, and rewards for actions over multiple steps are accumulated (cumulative reward 1004). For example, the model processing unit 114 compares a cumulative reward received in a non-hostile environment with cumulative reward 1004 to evaluate the robustness of the model (robustness evaluation 1005). For example, if cumulative reward 1004 has changed by a predetermined value or more from the cumulative reward in the non-hostile environment, this means that an action of the model deviates from an originally expected action, indicating that the robustness with respect to the hostile environment is low. <Change in the environment>

[0075] Next, a change in an environment according to the present embodiment will be described. In this processing, when the action is performed in state s t , which is obtained at time t, is determined by the model processing unit, the environment parameter ξ in the action value function Q(s t , a t ; ξ) and a hostile distribution v(ξ|s t , a t ), in which an objective function obtained by adding a constraint to an f-divergence is minimized, i.e. a distribution according to formula (11). [Formula 11] v*(ξ|st,at)=argminv∈N Eξ∼v[Qπ(st,at;ξ)]+αattkDf(v(⋅|st,at)‖p(⋅))

[0076] Here, D f (v||p) the f-divergence between the hostile distribution v and the preceding distribution p of the environmental parameters, and α attk is an adjustment factor for adjusting the strength of the constraint imposed by the divergence. That is, by α attk It can be adjusted how much the hostile distribution v is restricted to the previous distribution p.

[0077] If a value of α attk closer to 0, a distribution of v*(ξ|s can occur t , at) approximate a jagged distribution which has a peak at an environmental parameter that, for example, minimizes the action value. On the other hand, if the value of α attk larger, the distribution of v*(ξ|s t , a t ) closer to the distribution of the previous distribution p. That is, in the present embodiment, when the value of α attk If the distribution is set appropriately, it is possible to obtain the hostile distribution, which is adapted to exhibit characteristics of the preceding distribution p, while including the characteristics of the hostile distribution of the environment (which minimizes the action value).

[0078] Next, assuming that the divergence is a KL divergence, an analytical solution of formula (12) is obtained by the Legendre-Fenchel transformation of formula (11) of the minimized hostile distribution v*(ξ|s). t , at) received. [Formula 12] vπ∗(ξ|st,at)=p(ξ)exp(−Qπ(st,at;ξ) / αattk)∫ξp(ξ)exp(−Qπ(st,at;ξ) / αattk)dξ

[0079] In this case, similar to the first embodiment, an approximation can be carried out in formula (13) by sampling a limited number (i = 1, 2, ..., and n) from the previous distribution p(ξ) and subtracting a probability weight from formula (12) for correction. [Formula 13] v*(ξ|st,at)≅v*(ξi|st,at)∝ exp(−Qπ(st,at;ξi)αattk)

[0080] This corresponds to calculating the action values ​​Q(s). t , ai; ξ i ), which samples correspond to which in Fig. Figure 11 illustrates this, and then calculates the hostile noise distribution for the samples. In particular, the hostile distribution is calculated by calculating an exponential function using a ratio between the action value Q(s). t ;ai; ξ i ) and a value of the adjustment factor α attk preserved as a variable.

[0081] Additionally, a hostile environmental parameter distribution model can be used as a further approximation unit. modell π(ξ|s t , a t ) for a parameterized environment must be prepared and trained separately to conform to formula (12), which is the analytical solution. Similar to the first embodiment, this can be easily achieved by updating the hostile environment parameter distribution such that the model and formula (14) of the KL divergence on the right-hand side of formula (12) are minimized. [Formula 14] Loss=Est,at~D(⋅)[DKL(vπmodel(⋅|st,at)|p(⋅)exp(−Qπ(st,at;⋅) / αattk)Z)]∝Est,at~D(⋅)[Eξ~vπmodel(⋅|st,at)[αattklog vπmodel(ξ|st,at)−αattklog p(ξ) +Qπ(st,at;ξ)+const.]]

[0082] Similar to the embodiments described above, an approximately hostile distribution v*(s) is approximated by sampling. t , ai; ξ i ) according to formula (13) based on the action value Q(ξ|s t , a t The value of the action is calculated according to a perturbation obtained by sampling the preceding distribution. In a case where the hostile environment parameter distribution model is used, the hostile distribution is calculated directly from the model. Therefore, it is not necessary to perform an optimization calculation with respect to the action value using a gradient descent method. Since the optimization calculation using the gradient descent method, which requires high computational costs, is eliminated, the computational costs can be significantly reduced, and the processing speed can be increased.

[0083] The model processing unit 114 selects the environment parameter ξ according to the probability weight according to formula (12) from the obtained hostile distribution in the case of approximation by sampling or from an approximate output distribution in the case of approximation by the hostile environment parameter distribution model and uses the state s t+1 , which transitions according to the environment. Here, the (weakest and most hostile) environment with the lowest action value is selected when the adaptation factor α attk is small, i.e., when α attk → 0, and the environment is selected according to the distribution p(ξ) of the environment parameters, which are previously assumed when α attk is sufficiently large. In this way, it is possible to select an appropriate environment for evaluating the robustness necessary for the model, while following the distribution that is assumed in advance.

[0084] It should be noted that in a case where the approximation is performed by sampling in the present embodiment, the characteristics of a robustness evaluation according to the size of the number of samples are similar to those in the embodiments described above. That is, a user can evaluate a suitable model by adjusting the number of samples according to the purpose of the evaluation. Even if the frequency of occurrence of the most hostile environment is extremely low, a model can still be evaluated by sufficiently increasing the number of samples in a case where the evaluation of the model's robustness using the environment is required.On the other hand, in a case where it is sufficient to perform an evaluation in an environment corresponding to the characteristics of the previous distribution, and an evaluation in an environment with an extremely low frequency of occurrence is not necessarily required, a model can be evaluated by reducing the number of samples. Naturally, in this case, the model evaluation can be performed at high speed. <Vorgang der Robustheit-Evaluationsverarbeitung>

[0085] Next, a series of processes for robustness evaluation processing will be described with reference to Fig. 12. It should be noted that this processing is implemented, for example, by the processor 110, which transfers a computer program stored in the ROM 112 or the memory unit 107 to the RAM 111 and executes the computer program. The model processing unit 114 performs the following processing as a processing unit, unless otherwise specified.

[0086] In S1201, the control unit 108 receives sensor data from the sensor unit 101 at time t and receives the state s. t an environment that is used in a reinforcing target learning model.

[0087] In S1202, the model processing unit 114 determines the action at in the state s t according to, for example, the actor's rule π.

[0088] In S1203, the model processing unit 114 generates a hostile distribution of the environment, for example by executing the processing described above. Fig. 11 is described, and receives the (for example, most hostile) environment parameter ξ (selects it).

[0089] In S1204, the model processing unit 114 performs the action at in the environment, which includes the environment parameter ξ (for example, outputs a control quantity corresponding to the action at), and assumes a new state s t+1 on.

[0090] Model Processing Unit 114 determines a reward rt for the action at in S1205 and updates a cumulative reward in S1206. In S1207, Model Processing Unit 114 determines whether a termination condition is met, transfers the processing to S1208 if the termination condition is met, and returns the processing to S1201 to repeat the processing if it is not. When the processing is repeated, Model Processing Unit 114 transfers the time from t to t+1. The termination condition can be any condition, but could be, for example, a case in which time t exceeds a predetermined time T, or something similar.

[0091] In S1208, for example, the model processing unit 114 compares a cumulative reward received in a case where an environment serving as an evaluation criterion is selected (for example, in a case where a non-hostile environment is selected) with the cumulative reward in S1206 and evaluates the robustness of the model. As described above, for example, if the cumulative reward in S1206 has changed by a predetermined value or more from the cumulative reward in a case where the hostile environment is not selected, the robustness with respect to the hostile environment is determined to be low. The model processing unit 114 then terminates the current processing. <Modell-Lernverarbeitung unter Verwendung einer feindlichen Verteilung einer Umgebung>

[0092] Next, an example will be described in which a model is trained using the hostile distribution of the environment in the third embodiment. Training the model using the hostile environment improves the robustness of the trained model. The training of the model by the learning control unit 116 can be carried out in essentially the same way as described in Fig. 8 described processing.

[0093] This means that when learning the action-value function, the model is updated such that the goal y coincides with the action-value function, as shown in formula (15). Here, “s” means t , a t ~D(·)", that a plurality of trajectories are extracted from the storage unit 107 by the size of a stack to compute an expected value.

[0094] In a case where the approximation is performed by sampling, a finite number of samples ξ are used. i (where n(i = 1, 2, ..., and n)) are taken from the predetermined preceding distribution p of the environment parameters, and a next state s t+1,i is done using an environment model T(s) t+1 |s t , a t , ξ) using the environment parameters ξ i , the condition s t and the action is calculated. The next state s t+1,i can be calculated directly if a known environment model T, such as a simulator, is used for a learning environment, and can be easily trained by training a predictive model from a plurality of pieces of trajectory data s t , at and s t+1 The values ​​stored in memory unit 107 are calculated, even in an unknown environment. The target y is calculated in formula (16) based on pieces of the recorded information, and the action value function is updated by formula (15).

[0095] Additionally, in a case where the hostile distribution of the environment parameter is approximated by the model as shown in formula (14), the target y can be calculated by directly obtaining the environment parameter ξ from the hostile environment parameter distribution model, as expressed by formula (17). [Formula 15] L(Q)=Est,at~D(⋅)[|y(st,at)−Qπ(st,at;ξ)|2] [Formula 16] y(st,at)=r(st,at)+γ[−αattklog(1N∑i=1nexp(−Qπ(st+1,i,at+1,i;ξi)αattk))] [Formula 17] y(st,at)=r(st,at)+γEξ~vπmodel(⋅|st,at)[Est+1~T(⋅|st,at;ξ)[Eat+1~π(⋅|st+1)[Qπ(st+1,at+1;ξ)]]]

[0096] Similarly, the updating of the rule function will be described. In a case where the approximation is performed by sampling, finite samples (i = 1, 2, ..., and n, where n) are similarly taken from the predetermined prior distribution p(ξ), and the rule function is updated to maximize the action-value function itself under a perturbation of the environment parameter by formulas (18) and (19). Here, w is a weight correction expression obtained by correcting the probability weight of the prior distribution p using the distribution on the right-hand side of formula (12) for sampling, similar to the embodiments described above.

[0097] Additionally, in a case where the hostile distribution of the environment parameter is approximated by the model as shown in formula (14) above, the regulation function can be obtained by directly obtaining the environment parameter ξ from the hostile environment parameter distribution model, as expressed by formula (20). [Formula 18] J(π)=Est~D(⋅)[Eat~π(⋅|st)[∑i=1nw(ξi|st,at)Qπ(st,at;ξi)]] [Formula 19] w(ξi|st,at)∝exp(−Qπ(st,at;ξi)αattk) [Formula 20] J(π)=Est~D(⋅)[Eat~π(⋅|st)[Eξ~vπmodel(⋅|st,at)[Qπ(st,at;ξ)]]

[0098] At this time, in a case where the environment parameter ξ does not change over time, the same environment parameter ξ can be used until the end of the learning episode. In this case, after the environment parameter is selected by the processing described above, the following can be used: Fig. As described in section 11, normal training of the reinforcing learning model is performed using the same environment parameter. This means that the learning control unit 116 trains the action value function and the rule function such that the action value of the action in the environment (specified by the selected environment parameter) is maximized. In this way, it is possible to train the model robustly against the environment of the selected environment parameter.

[0099] Additionally, in a case where the environmental parameter ξ changes over time (for example, the friction of a road surface changes), the environmental parameter is selected by the processing described above. Fig. 11 is described for each time. In this way, the model can be trained in the environment, which changes from moment to moment due to the hostile environment parameter.

[0100] As described above, in the present embodiment, the control unit 108 determines the hostile distribution of the environment based on the action value, while the constraint is added using the divergence, which indicates the proximity between the hostile distribution of the environment for the model to be evaluated and the predetermined prior distribution. Additionally, the determined hostile distribution of the environment is applied to a machine learning model to be evaluated, and the robustness of the machine learning model to be evaluated is assessed based on a change between a case in which the hostile environment is applied and a case in which the hostile environment is not applied. In this way, it is possible to select a suitable environment for evaluating the robustness of the model. Furthermore, the robustness of the model can be evaluated using the appropriate environment.

[0101] Additionally, the selected hostile environment parameter is used during the training of the reinforcing learning model in the present embodiment. This means that the learning control unit 116 determines the hostile distribution of the environment with respect to the model being processed, using the predetermined prior distribution, and optimizes the action value function of the model being processed based on the action value of the action in the hostile environment. In this way, the model can be trained to be more robust against the hostile environment. In other words, a suitable environment can be provided to evaluate or ensure the robustness of the model.

[0102] The invention is not limited to the embodiments described above, and various variations / modifications are possible within the spirit of the invention. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] WO 2023 / 037504

[0003] < / fahrzeugkonfiguration> < / details> < / fahrzeugkonfiguration>

Claims

[1] Information processing device, comprising: a recording device (113) which is designed to record a predetermined prior distribution of an environmental parameter; a computational tool (114) which is designed to calculate an action value of an action in each of environments which are disturbed by the environment parameter; and a determining instrument (116) which is designed to determine a hostile distribution of an environment for a model to be processed on the basis of action value, wherein the determining means is set up to determine the hostile distribution of the environment, which reduces the action value of the model to be processed, while a constraint is added using a divergence which indicates a proximity between the hostile distribution of the environment and the predetermined prior distribution. [2] Information processing device according to claim 1, wherein the determining means is configured to control a size of the restriction by multiplying the divergence by an adjustment factor. [3] Information processing device according to claim 2, wherein the determining means is configured to determine the hostile distribution of the environment closer to the predetermined prior distribution when the adjustment factor is larger. [4] Information processing device according to claim 2, wherein the determining means is configured to determine the hostile distribution of the environment closer to the hostile distribution of the environment which minimizes the action value when the adjustment factor is smaller. [5] Information processing device according to claim 2, wherein the divergence comprises a KL divergence, and the determining means is configured to determine the hostile distribution of the environment using a ratio between the action value and the adaptation factor. [6] Information processing device according to claim 1, further comprising an evaluation means which is configured to apply an environment selected from the hostile distribution of the environment to the model to be processed in order to evaluate a robustness of the model to be processed based on a change between a case in which the environment is applied and a case in which the environment is not applied. [7] Information processing device according to claim 1, further comprising a sampling means which is configured to sample a finite number of samples from the predetermined prior distribution of the environmental parameter, wherein the determining means is configured to determine the hostile distribution of the environment which is approximated using the samples. [8] Information processing device according to claim 7, further comprising a setting agent which is configured to set the number of samples to be used for sampling from the predetermined prior distribution. [9] Information processing device according to claim 8, wherein the hostile distribution of the environment is more likely to include an environment which minimizes the action value of the model to be processed when the number of samples is larger, and the hostile distribution of the environment is more likely to include an environment according to the predetermined preceding distribution when the number of samples is smaller. [10] Information processing device according to claim 1, wherein the determining means is configured to approximate the hostile distribution of the environment with a modeled hostile environment parameter distribution model. [11] Information processing device according to claim 10, wherein the hostile environment parameter distribution model is obtained by updating a parameter of the hostile environment parameter distribution model using recorded trajectory data in such a way that a divergence between the hostile distribution of the environment and an output distribution of the hostile environment parameter distribution model is minimized. [12] Information processing device according to claim 1, wherein the information processing device is comprised in a vehicle or a robot. [13] Information processing device according to claim 1, wherein the information processing device is comprised in a server device. [14] Information processing device, comprising: a determining tool designed to determine a hostile distribution of an environment for a model to be processed using a predetermined prior distribution; and a learning control tool designed to train at least one of an action value function or a precept function of the model to be processed, based on an action value of an action in an environment selected from the hostile distribution of the environment, wherein the determining means is set up to determine the hostile distribution of the environment, which reduces the action value of the model to be processed under a constraint using a divergence that indicates a proximity between the hostile distribution of the environment and the predetermined prior distribution. [15] Information processing method in which each step is carried out by an information processing device, the information processing method comprising: Recording a predetermined prior distribution of an environmental parameter; Calculating an action value for an action in each of the environments disturbed by the environmental parameter; and Determining a hostile distribution of an environment for a model to be processed based on the action value, where determining the hostile distribution involves determining the hostile distribution of the environment, which reduces the action value of the model to be processed, while adding a constraint using a divergence that indicates a proximity between the hostile distribution of the environment and the predetermined prior distribution. [16] Method for evaluating a machine learning model in which each step is performed by an information processing device, the method comprising: Recording a predetermined prior distribution of an environmental parameter; Calculating an action value of an action in each of environments that are disturbed by the environmental parameter; Determining a hostile distribution of an environment for a machine learning model to be processed, based on action value; and Applying an environment selected from the hostile distribution of the environment to the machine learning model to be processed, to evaluate the robustness of the machine learning model to be processed based on a change between a case in which the environment is applied and a case in which the environment is not applied, where determining the hostile distribution involves determining the hostile distribution of the environment, which reduces the action value of the machine learning model to be processed, while adding a constraint using a divergence that indicates a proximity between the hostile distribution of the environment and the predetermined prior distribution. [17] Learning method of a machine learning model in which each step is performed by an information processing device, wherein the learning method comprises: Determining a hostile distribution of an environment for a model to be processed using a predetermined prior distribution; and Training at least one from an action value function or a precept function of the model to be processed, based on an action value of an action in an environment selected from the hostile distribution of the environment, where determining the hostile distribution involves determining the hostile distribution of the environment, which reduces the action value of the model to be processed under a constraint using a divergence that indicates a proximity between the hostile distribution of the environment and the predetermined prior distribution. [18] Computer-readable storage medium which stores a program to cause a computer to execute an information processing procedure comprising: Recording a predetermined prior distribution of an environmental parameter; Calculating an action value for an action in each of the environments disturbed by the environmental parameter; and Determining a hostile distribution of an environment for a model to be processed based on the action value, where determining the hostile distribution involves determining the hostile distribution of the environment, which reduces the action value of the model to be processed, while adding a constraint using a divergence that indicates a proximity between the hostile distribution of the environment and the predetermined prior distribution.

Images