Adjustable agent behavior through a continuous reward weight-based target space

Abstract

For each component of an artificial intelligence agent within a domain, a single policy can be trained to handle user selection of parameters over a predetermined range. The agent can be trained over a number of weights within a desired range for each component. These weights determine how much the agent should consider the reward portion of each component during training. Thus, an improved formulation for UVFA-like goals can be realized based on a compositional reward function parameterized by the component weights. In the field of autonomous racing games, a set of reward components has been determined that, when combined with the improved UVFA formulation, allows for the training of a single racing agent that generalizes over multidimensional continuous motion. This can be used by game designers to fine-tune the skills and personality of the trained agent.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to reinforcement learning systems and methods. Specifically, embodiments of the present invention relate to methods and systems for goals such as universal value function approximators (UVFAs) based on compositional reward functions parameterized by component weights.

Background Art

[0002] The following background information can present examples of specific aspects of the prior art (e.g., without limitation, techniques, facts, or general notions), and these examples are expected to help convey further aspects of the prior art to the reader, but should not be construed as limiting the present invention or any of its embodiments to any matter mentioned or implied within these examples or speculated about these examples.

[0003] In the field of reinforcement learning, the value function V π (s) is used to model the expected future reward of an agent starting from state s and following policy π. At this time, the agent directly determines the action to take using this value function, or informs another policy function in an actor-critic setting of the learning process to stabilize it.

[0004] Universal value function approximator (UVFA) V π(s,g) is an extension of the value function that is further conditional on goal g, i.e., these estimate future rewards starting from state s using a reward function that depends on the active goal g. As a result, UVFA-based agents can learn how to operate optimally under multiple goals and potentially generalize to unknown goals. Exemplary goals for UVFA include a discrete set of goal states (e.g., 2D goal locations in a grid world where the agent is rewarded for reaching the active goal location) or a vector representation of arbitrary pseudo-reward functions. [Overview of the project] [Problems that the invention aims to solve]

[0005] Therefore, an improved formulation for UVFA-like goals based on a compositional reward function parameterized by the weights of the components is needed. [Means for solving the problem]

[0006] Aspects of the present invention provide an improved formulation for UVFA-like goals based on a compositional reward function parameterized by component weights. Furthermore, aspects of the present invention provide a set of reward components for the field of autonomous racing games, which, when combined with the improved UVFA formulation, allows for the training of a single racing agent that generalizes over multidimensional continuous motion. This can be used by game designers to fine-tune the skills and personality of the trained agent.

[0007] Embodiments of the present invention provide a method for training an artificial intelligence agent that generalizes over multidimensional continuous operations, comprising: defining a state- and action-based reward function as a linear combination of a plurality of component reward functions and weights for each of the plurality of component reward functions; sampling multiple dimensions of the weights for each of the plurality of component reward functions from a continuous distribution between a maximum weight and a minimum weight; and training a single policy for the artificial intelligence agent over a continuous target space including a plurality of parameterized reward functions represented by the continuous distribution of weights for each of the plurality of component reward functions; and providing a non-temporary computer-readable storage medium that tangibly embodies computer-readable program code having computer-readable instructions that cause a computer device to execute such a method at runtime.

[0008] Embodiments of the present invention provide a method for providing an artificial intelligence agent capable of tuning to one or more skill components and / or one or more personality components in a racing game, comprising: defining a state- and action-based reward function as a linear combination of a plurality of component reward functions and weights for each of the plurality of component reward functions; sampling multiple dimensions of weights for each of the plurality of component reward functions from a continuous distribution between a maximum weight and a minimum weight; and training a single policy for the artificial intelligence agent over a continuous target space including a plurality of parameterized reward functions represented by a continuous distribution of weights for each of the plurality of component reward functions, wherein the plurality of component reward functions include a base reward that motivates the artificial intelligence agent to complete the race in the shortest possible time, and one or more further component reward functions that provide one or more skill components and / or one or more personality components.

[0009] These and other features, aspects and advantages of the present invention will be better understood by referring to the following drawings, description and claims.

[0010] Several embodiments of the present invention are shown, not as limitations, but as examples, with reference to the figures in the accompanying drawings, where the same reference numerals may indicate similar elements. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows the input modification of an actor network for an exemplary quantile regression soft actor-critic (QR-SAC) algorithm that informs an agent of reward weights, according to an exemplary embodiment. [Figure 2] This figure shows the input modification of the critic network of an exemplary quantile regression soft actor-critic (QR-SAC) algorithm that informs the agent of the reward weights, according to an exemplary embodiment. [Figure 3] This is a functional block diagram of a computer hardware platform that can be used to implement a specially configured computer device capable of hosting an AI agent training engine. [Modes for carrying out the invention]

[0012] Unless otherwise specified, the diagrams are not necessarily to scale.

[0013] The present invention and its various embodiments can be better understood by referring to the following detailed description of the illustrated embodiments. It should be clearly understood that the illustrated embodiments are for illustrative purposes only and do not limit the invention to the final claims.

[0014] The terms used herein are for the purpose of describing specific embodiments and are not intended to limit the invention. The terms “and / or” as used herein include any combination of one or more of the items described herein. The singular forms “a, an” and “the” as used herein are intended to include the plural form as well as the singular form unless otherwise explicitly indicated in the context. Furthermore, the terms “comprises and / or comprising” as used herein indicate the presence of the features, steps, actions, elements and / or components referred to, but should be understood not to exclude the presence or addition of one or more other features, steps, actions, elements, components and / or groups thereof.

[0015] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as those generally understood by those skilled in the art to which the present invention pertains. Furthermore, terms defined in commonly used dictionaries should be interpreted as having the meanings corresponding to those meanings in the context of the relevant art and this disclosure, and should not be interpreted in an ideal or overly formal sense unless explicitly defined herein.

[0016] It will be understood that the description of this invention discloses multiple techniques and steps. Each of these has its own individual benefit and may be used in conjunction with one or more, or possibly all, of the other techniques disclosed. Therefore, for clarity, this specification refrains from unnecessarily repeating as many possible combinations of the individual steps as possible. However, this specification and the claims should be read with the understanding that such combinations are fully included in the scope of this invention and the claims.

[0017] The following description provides numerous specific details to allow for a complete understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention can be carried out without these specific details.

[0018] This disclosure should be considered illustrative of the present invention and is not intended to limit the invention to specific embodiments shown by the drawings and description.

[0019] "Computer" or "computer device" can mean one or more devices and / or one or more systems that accept structured input, process the structured input according to predefined rules, and produce the results of the processing as output. Examples of computers or computer devices include computers, fixed and / or portable computers, computers having a single processor, multiple processors, or multiple core processors capable of operating in parallel and / or non-parallel, supercomputers, mainframes, superminicomputers, minicomputers, workstations, microcomputers, servers, clients, interactive televisions, web appliances, communication devices with internet access, hybrid combinations of computers and interactive televisions, portable computers, tablet personal computers (PCs), personal digital assistants (PDAs), mobile phones, application-specific hardware that emulates computers and / or software, such as digital signal processors (DSPs), field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific instruction set processors (ASIPs), chips, multiple chips, systems on a chip or chipsets, data collectors, optical computers, quantum computers, biocomputers, and devices that generally accept data, process the data according to one or more stored software programs, produce results, and typically include input devices, output devices, memory devices, arithmetic units, logic units and control units.

[0020] "Software" or "application" can mean the defined rules for operating a computer. Examples of software or applications can include code segments in one or more computer-readable languages, graphic and / or text instructions, applets, pre-compiled code, interpreted code, compiled code, and computer programs.

[0021] Also, by storing these computer program instructions, which can be used to instruct a computer, other programmable data processing device, or other device to function in a specific manner, in a computer-readable medium, it is possible to create an article of manufacture in which the instructions stored in the computer-readable medium include instructions for performing the functions / acts specified within one or more blocks of a flowchart and / or block diagram.

[0022] Furthermore, although process steps, method steps, or algorithms may be described in a certain order, such processes, methods, and algorithms can also be configured to function in a different order. In other words, any order or sequence of steps that can be described does not necessarily indicate that these steps must be executed in this order. The process steps described in this specification can be executed in any practical order. Additionally, several steps can also be executed simultaneously.

[0023] It will be readily apparent that the various methods and algorithms described in this specification can be executed, for example, by appropriately programmed general-purpose computers and computing devices. Typically, a processor (e.g., a microprocessor) receives instructions from memory or a similar device and executes these instructions to perform the process defined by these instructions. Furthermore, programs for executing such methods and algorithms can be stored and transmitted using various known media.

[0024] As used herein, the term "computer-readable medium" means any medium that participates in providing data (e.g., instructions) that can be read by a computer, processor, or similar device. Such a medium can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical or magnetic disks and other permanent memories. Volatile media typically includes dynamic random access memory (DRAM) that constitutes main memory. Transmission media includes coaxial cables, copper wire, and fiber optics, including wires that include a system bus coupled to a processor. Transmission media can include, or convey, acoustic waves, light waves, and electromagnetic radiation such as those generated during radio frequency (RF) and infrared (IR) data communications. Common forms of computer-readable media include floppy disks, flexible disks, hard disks, magnetic tape, any other magnetic medium, CD-ROM, DVD, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASHEEPROM, any other memory chip or cartridge, a carrier wave as described hereinafter, or any other medium that can be read by a computer.

[0025] Various forms of computer-readable media can be involved in conveying a series of instructions to a processor. For example, a series of instructions can be supplied to the processor from (i) RAM, can be conveyed via a wireless transmission medium, and / or can be formatted according to numerous formats, standards, or protocols such as Bluetooth, TDMA, CDMA, 3G, 4G, and 5G.

[0026] Embodiments of the present invention can include an apparatus that performs the operations disclosed herein. The apparatus can include a device specially configured for a desired purpose or can include a general-purpose device selectively actuated or configured by a program stored therein.

[0027] Unless otherwise stated, and as may become apparent from the following description and claims, throughout this specification, any use of terms such as “process,” “calculate,” “calculate,” or “determine” should be understood to mean the actions and / or processes of a computer, computer system, or similar electronic device that manipulates data represented as physical quantities, such as quantities of electrons in the registers and / or memory of a computer system, and / or transforms it into other data similarly represented as physical quantities in the memory, registers, or other such information storage, transmission, or display device of a computer system.

[0028] Similarly, the term “processor” can mean any device or part of a device that processes electronic data from registers and / or memory and converts this electronic data into other electronic data that can be stored in registers and / or memory or transmitted to an external device, so as to cause a physical change or operation of an external device.

[0029] As is well known to those skilled in the art, designing the optimal configuration for a commercial implementation of any method or system, in particular embodiments of the present invention, usually requires considerable deliberation and compromise. A commercial implementation of the spirit and teachings of the present invention can be configured according to the needs of a particular application, thereby enabling those skilled in the art to suitably omit, include, adapt, mix, match, or improve and / or optimize any (one or multiple) aspects, (one or multiple) features, (one or multiple) functions, (one or multiple) results, (one or multiple) components, (one or multiple) methods, or (one or multiple) steps of any (one or multiple) teaching relating to any described embodiment of the present invention, to achieve a desired implementation that meets the needs of a particular application.

[0030] In general, embodiments of the present invention provide systems and methods for developing artificial intelligence (AI) policies for artificial agents for various fields, including gaming fields such as racing games. The behavior of such AI agents can be selected by the user at runtime by selecting parameters of several different factors. A single policy can be trained to handle user selection of parameters over a predetermined range for each component. The agent can be trained over a number of weights within a desired range for each component. These weights determine how much the agent should consider the reward portion of each component during training. Thus, an improved formulation for UVFA-like goals can be realized based on a compositional reward function parameterized by the component weights. In the field of autonomous racing games, a set of reward components has been determined that, when combined with the improved UVFA formulation, allows for the training of a single racing agent that generalizes over multidimensional continuous operation. This can be used by game designers to tune the skills and personality of the trained agent.

[0031] When developing AI policies for different games, different players play at different skill levels. Therefore, when playing against an AI agent, players often want this agent to play at at least the same skill level as the player. An aspect of the present invention provides a reinforcement learning training process that defines what different skill levels mean and adjusts behavior by using AI techniques.

[0032] For example, one component could represent the aggressiveness of an agent in a racing game. The reward function could be based on the number of collisions the agent causes during a race, and a reward could be provided in this case. Therefore, if an agent that drives aggressively is desirable, a higher reward can be given for more collisions. Policies can be trained across a reward range so that the user can select the aggressiveness component of the opposing agent at runtime. Thus, instead of training multiple policies for low, medium, and high aggression, for example, a single policy can be implemented for this component and any further components for the agent.

[0033] As will be described in more detail below, aspects of the present invention provide the ability to train an agent across multiple weights for various components of a field, such as racing games, where the weights become inputs to a neural network, allowing the user to select the weights at runtime. In some embodiments, weights can be provided as inputs to both a policy and a Q function. While inputting weights to the Q function is not required in all aspects of the present invention, such inputs can help achieve more stable learning during training.

[0034] In some embodiments, the agent is trained in a training step so that the user can select parameters during gameplay. In other embodiments, the agent can be trained during gameplay, in which case the user can select parameters for gameplay and, if necessary, feed these results back into the neural network to update the policy. For example, if the user selects an aggression level of 5 (e.g., on a scale of 1 to 10), and the agent performs with more collisions than the desired number of collisions at this selected level, this information can be fed back into the network to appropriately update the policy to further limit the number of collisions at this selected level for this selected component.

[0035] Continuous reward weights as the target space Referring to Figures 1 and 2, as is done in many, if not most, reinforcement learning (RL) applications, the reward function R of the environment can be defined as a linear combination of m components based on the state s and the action a. JPEG2026520331000002.jpg10170 Here, w i is the weight of the scalar component, R i (s,a) is the reward function of the i-th component. Typically, RL applications keep w fixed as a single scalar vector during experiments, and often explore the optimal w for the application across multiple experiments. However, aspects of the present invention allow an agent to be trained over a continuous target space, including a parameterized reward function represented by weights.

[0036] In aspects of the present invention, instead of keeping w fixed, one or more dimensions i of w can be sampled from a continuous (e.g., uniform) distribution. This subset of non-fixed dimensions of w is It can be represented as JPEG2026520331000003.jpg6170. To give more weight to a particular weight or weight range segment and improve the agent's performance on that segment, skewed distributions such as a log-uniform distribution can be used. In the basic version of this method, The image JPEG2026520331000004.jpg6170 can be sampled once at the start of each training rollout and then kept fixed thereafter.

[0037] A possible extension is to make the agent robust to changes in the reward function during the current trajectory, during rollout. It is suggested that JPEG2026520331000005.jpg6170 be resampled repeatedly.

[0038] To inform the trained agent of the reward function under which it is operating, additional inputs are provided to both the policy (actor) and value function (critic) of the training algorithm. JPEG2026520331000006.jpg6170 can be provided. By concatenating the weights of the non-fixed reward component with the remaining inputs, the policy of the neural network can be changed from π(s) In JPEG2026520331000007.jpg6170, the action-value function Q(s,a) is also... It can be updated to JPEG2026520331000008.jpg6170.

[0039] When evaluating the agent's policy during inference, The JPEG2026520331000009.jpg6170 can be set to any of the weights covered during training, and the agent can adapt its behavior accordingly, striving to perform optimally under the expressed reward function without requiring retraining.

[0040] Reward parts that affect the driving behavior of autonomous racing agents Aspects of the present invention provide a set of reward parts that can be used in combination with the reward weight-based goal formulation described above for the field of designing autonomous opponents in racing games. These reward parts enable the encoding of different desired behavior types into a reward function. In the environment step, progress along the center line of the racing course achieved by the agent can be used as a base reward to motivate the agent to complete the racing course as quickly as possible. At this time, new reward parts The sampled weights in JPEG2026520331000010.jpg6170 represent the importance of these parts in relation to fixed-weight progress rewards. In addition to progress rewards, we propose using the following reward parts individually or in combination with each other: (1) A penalty for tire wear of the agent's car during environmental steps. This penalty motivates the agent to conserve tires and drive a more conservative trajectory, similar to a cautious human driver, resulting in a slower trajectory. (2) A penalty for fuel consumption of the agent during environmental steps. This penalty motivates the agent to drive a more economical and smoother trajectory by minimizing extreme speed changes. (3) A linear penalty for tire slip ratio and angle. This penalty motivates the agent to minimize the possibility of slipping and therefore drive a safer trajectory, similar to an inexperienced or cautious human driver who brakes earlier before approaching a curve and accelerates later when exiting the curve, resulting in a slower trajectory overall. (4) A linear positive reward for tire slip ratio and angle. This reward motivates the agent to drift, especially on curves where drifting allows for the achievement of relatively high progress rewards. (5) Edge distance penalty that increases linearly as the agent approaches the edge of the racing track. By setting the weight of this reward, it is possible to influence how wide the width of the track the agent uses. (6) A set of reward parts that penalize the agent for driving within corresponding slices of the track defined by distances to the center line, such as center line to center line, 2m to 4m, and 4m to 6m. The game designer can influence the driving line of the trained agent based on the weights configured for these reward parts during inference. (7) Overtaking reward with positive and negative parts that are independently weighted for overtaking other cars and being overtaken by other cars, respectively.(8) Penalties for changes in steering angle during environmental steps. This penalty motivates the agent to reduce steering changes, making steering smoother and braking earlier to achieve a tighter steering range through curves. (9) Penalties for collisions with other vehicles. Based on the weight of this penalty, the agent's aggression and assertiveness towards other drivers nearby can be adjusted. (10) Penalties for going off-track, measured by the center of the vehicle or tires. Based on the weight of this penalty, the frequency with which the agent crosses the track boundary, or, when used with a relatively high weight, the degree to which the agent travels near the edge of the track can be adjusted.

[0041] The advantages of continuous reward weights for tuning agent behavior (1) Scaling agent behavior and designing post-training driver personality. Game designers can configure the behavior of agents trained according to embodiments of the present invention along multiple axes, including caution and aggression, without having to retrain the policies each time. In this way, for example, designers can adjust the agent's skills to match those of human players by appropriately setting the weights of previously introduced reward components, such as tire slip penalties, during gameplay. In addition, game designers can also design the agent's personality based on a driving style represented by the selection of specific reward weights. For example, a cautious and courteous driver can be configured by increasing the weights of tire slip and collision penalties, or the weights can be adjusted to match the driving style of a famous real-world racing driver. Allowing game designers to adjust these behaviors to their liking after training increases the degree of freedom in the design process and reduces reliance on reinforcement learning experts who trained the agents.

[0042] (2) Support for context-based reward weights for in-game behavior tweaking. An aspect of the present invention allows for training an agent that is robust to online reward weight changes by resampling the reward weights midway through the rollout during training. As a result, game designers can update the agent's behavior in the middle of the game. For example, this can be used in combination with a tire slip penalty to change the agent's overall speed based on its distance from the human driver by increasing the weight of the tire slip penalty when the agent is too far from the human driver and decreasing this weight when the agent is too far behind.

[0043] (3) Sharing of neural network inference for multiple differently behaving agents. A neural network policy trained using this method encodes a range of consecutive agent behaviors within a single, shared network weight set. Running multiple different reward weight configurations of such agents side by side allows for easy parallelization of the network forward pass, effectively reducing the inference time for multiple differently behaving agents to the time required for inference for a single agent. Furthermore, running multiple differently behaving agents only requires loading a single neural network into memory. As a result, memory consumption is significantly reduced compared to using a neural network policy trained separately for all desired agent behaviors.

[0044] (4) Replacement of reward weight search by single experiment. The method according to an aspect of the present invention enables the training of an agent over a continuous range of reward weights in a single training run. This method can be used even if the goal is to ultimately use only a single fixed vector of reward weights during inference. The method according to an aspect of the present invention can significantly reduce the time required to search for a reward weight vector that yields the desired agent behavior. Using the aspect of the present invention, instead of training separate models from scratch for all reward weight combinations of interest, the range of reward weights of interest can be covered in a single training run, and then promising reward weight combinations can be evaluated, which is significantly faster and less costly to implement.

[0045] (5) Improving agent search by varying reward weights. It is hypothesized that training an agent with varying reward weights can improve the algorithm's search process by providing the searching agent with more diverse data with different reward weight settings. For example, training an agent with a collision penalty allows the agent to learn to avoid collisions early in the training process. However, this increases the sparsity of the collision penalty signal, significantly reducing the amount of collision data that can be learned from the remaining training time. By simultaneously training the agent across a range of low-penalty weights, the agent can collect data containing collisions from these low-weight rollouts. This additional data allows the agent to learn a more accurate value function.

[0046] While the above disclosure focuses on the field of racing games, it should be understood that aspects of the present invention can be applied to AI agents used in a variety of different fields. For example, the AI ​​agent could be in the field of animation or locomotion, in which case one of the components could be provided to make the AI ​​agent's movements slower, faster, or more expressive by changing the weight of energy costs, for example. As another non-limiting example, in fighting games, an AI agent could be trained using various proficiencies with various weapons, in which case a single policy could provide a range of proficiencies with weapons such as bows, swords, and axes.

[0047] Figure 3 is a functional block diagram of a computer hardware platform 300 that can be used to implement a specially configured computer device capable of hosting the AI ​​agent training engine 350. As described above, the AI ​​agent training engine 350 may include an actor network 352, an arbitrary critic network 354, and a number of components 356, each of which can be individually weighted to train the AI ​​agent.

[0048] The computer platform 300 may include a central processing unit (CPU) 302, a hard disk drive (HDD) 304, random access memory (RAM) and / or read-only memory (ROM) 306, a keyboard 308, a mouse 310, a display 312, and a communication interface 314, all of which are connected to a system bus 316.

[0049] In one embodiment, the HDD 304 has the ability to store a program that can execute various processes, such as an AI agent training engine 350, in a manner that performs the methods described herein.

[0050] Unless otherwise specified, all features disclosed herein, including any attached abstracts and drawings, may be replaced by other features that are identical, equivalent, or serve a similar purpose. Accordingly, unless otherwise specified, each disclosed feature is merely an example of a general set of equivalent or similar features.

[0051] Elements and steps of the claims herein may be numbered and / or lettered solely for readability and comprehensibility. Any such numbering and lettering is not intended, nor should be construed, to indicate an ordering of elements and / or steps within the claims.

[0052] Those skilled in the art can make numerous changes and modifications without departing from the spirit and scope of the present invention. Therefore, the illustrated embodiments are merely examples and should not be interpreted as limiting the present invention as defined by the following claims. For example, even if the elements of the claims are shown below in specific combinations, it should be clearly understood that the present invention includes other combinations of elements that are fewer, more, or different from those disclosed.

[0053] The words used herein to describe the present invention and its various embodiments should be understood to include not only their generally defined meanings but also, by their specific definitions herein, comprehensive structures, materials, or actions that represent a single kind.

[0054] Accordingly, the definitions of words or elements in the following claims are not limited to combinations of elements explicitly stated in the text. In this sense, it is conceivable that one of the elements in the following claims may be replaced by two or more equivalent substitutes of elements, or a single element may be replaced by two or more elements in the claims. While elements are described above as functioning in specific combinations, and claims may initially be made in this manner, it should be clearly understood that, in some cases, one or more elements resulting from the claimed combinations may be removed from these combinations, and the claimed combinations may be directed towards lower combinations or variations of lower combinations.

[0055] It is explicitly assumed that any minor changes from the subject matter, as seen by those skilled in the art, whether currently known or later devised, are equally included in the claims. Accordingly, obvious substitutions, both currently known and hereafter known to those skilled in the art, are also defined as being included in the scope of the specified elements.

[0056] Therefore, the claims should be understood to include those specifically illustrated and described above, those that are conceptually equivalent, those that are clearly substituted, and those that incorporate the fundamental ideas of the present invention. [Explanation of symbols]

[0057] 300 Computer Hardware Platforms 302 CPU 304 HDD 306 RAM / ROM 308 keyboards 310 Mouse 312 displays 314 Communication Interface 316 System Bus 350 Artificial Intelligence Agent Training Engines 352 Actor Network 354 Critic Network 356 Multiple ingredients

Claims

1. A method for training an artificial intelligence agent that generalizes over a series of multidimensional actions, The reward function based on state and behavior is defined as a linear combination of multiple component reward functions and the weights for each of the multiple component reward functions, The multiple dimensions of the weights for each of the multiple component reward functions are sampled from a continuous distribution between the maximum weight and the minimum weight, Training a single policy of the artificial intelligence agent over a continuous target space including a plurality of parameterized reward functions represented by a continuous distribution of the weights for each of the plurality of component reward functions, A method characterized by including the following.

2. The further includes improving the performance of the artificial intelligence agent across segments of the continuous distribution of the weights by providing a gradient distribution of the weights, wherein the training is performed across the gradient distribution of the weights for one or more of the plurality of component reward functions. The method according to claim 1.

3. The gradient distribution of the aforementioned weights is a log-uniform distribution. The method according to claim 2.

4. The continuous distribution of the weights is further sampled once for each training rollout at the start of an episode. The method according to claim 1.

5. The artificial intelligence agent further includes repeatedly resampling the continuous distribution of the weights during the training rollout, so that the agent is robust to changes in the reward function during the current trajectory. The method according to claim 1.

6. This further includes applying the continuous distribution of the weights to both the training algorithm's policy and its value function. The method according to claim 1.

7. Continuous distribution of the aforementioned weights By linking this to an input related to state s, the neural network policy can be expressed as π(s) Updated to, the action-value function Q(s, a) Further updates include, The method according to claim 6.

8. The method further includes evaluating the single policy of the artificial intelligence agent at inference time by selecting selected weights for each of the plurality of component reward functions, wherein the artificial intelligence agent operates appropriately under the selected reward function without retraining. The method according to claim 1.

9. The aforementioned artificial intelligence agent operates in a racing game environment. The method according to claim 1.

10. As one of the aforementioned multiple component reward functions, a basic reward is provided to motivate the artificial intelligence agent to complete the race in the shortest possible time. To provide one or more further component reward functions that provide one or more skill component reward functions and / or one or more personality component reward functions from among the plurality of component reward functions, Further including, The method according to claim 9.

11. The continuous distribution of weights for the one or more further component reward functions represents the importance of each of the one or more further component reward functions in relation to the fixed weight for the basic reward. The method according to claim 10.

12. Of the aforementioned plurality of component reward functions, the one or two or more further component reward functions are: (a) Penalties for the deterioration of the tires of the artificial intelligence agent's car during the environmental step, (b) Penalties imposed by the artificial intelligence agent on fuel consumption during the environmental step, (c) Linear penalties for tire slip ratio and angle, (d) A linear positive reward for the slip ratio and angle of the tire, (e) an edge distance penalty that increases linearly as the artificial intelligence agent approaches the edge of the racing track, (f) A set of reward parts that penalizes the artificial intelligence agent for driving within a corresponding slice of the racing track, determined by the distance to the center line of the racing track. (g) Overtaking reward having positive and negative parts that are independently weighted for overtaking other vehicles and being overtaken by other vehicles, (h) Penalties for changes in steering angle during environmental steps, (i) Penalties for collisions with other vehicles, (j) Penalties for the artificial intelligence agent's car driving off course, Including at least one of the following: The method according to claim 10.

13. Each of the one or more further component reward functions among the plurality of component reward functions is defined within a single policy of the artificial intelligence agent. The method according to claim 12.

14. A method for providing an artificial intelligence agent capable of synchronizing with one or more skill components and / or one or more personality components in a racing game, The reward function based on state and behavior is defined as a linear combination of multiple component reward functions and the weights for each of the multiple component reward functions, The multiple dimensions of the weights for each of the multiple component reward functions are sampled from a continuous distribution between the maximum weight and the minimum weight, Training a single policy of the artificial intelligence agent over a continuous target space including a plurality of parameterized reward functions represented by a continuous distribution of the weights for each of the plurality of component reward functions, Includes, The plurality of component reward functions include a base reward that motivates the artificial intelligence agent to complete the race in the shortest possible time, and one or more further component reward functions that provide one or more skill components and / or one or more personality components. A method characterized by the following:

15. The further includes improving the performance of the artificial intelligence agent across segments of the continuous distribution of the weights by providing a gradient distribution of the weights, wherein the training is performed across the gradient distribution of the weights for one or more of the plurality of component reward functions. The method according to claim 14.

16. The continuous distribution of the weights is further applied to both the policy and the value function of the training algorithm. By linking this to inputs related to state s and action a, the neural network policy is derived from π(s) The function has been updated, and the action-value function has changed from Q(s, a) It will be updated to The method according to claim 14.

17. The method further includes evaluating the single policy of the artificial intelligence agent at inference time by selecting selected weights for each of the plurality of component reward functions, wherein the artificial intelligence agent operates optimally under the selected reward function without retraining. The method according to claim 14.

18. A non-temporary computer-readable storage medium that tangibly embodies computer-readable program code having computer-readable instructions, wherein the computer-readable instructions cause a computer device to execute a method for training an artificial intelligence agent that is generalized over a series of multidimensional operations, and the method is The reward function based on state and behavior is defined as a linear combination of multiple component reward functions and the weights for each of the multiple component reward functions, The multiple dimensions of the weights for each of the multiple component reward functions are sampled from a continuous distribution between the maximum weight and the minimum weight, Training a single policy of the artificial intelligence agent over a continuous target space including a plurality of parameterized reward functions represented by a continuous distribution of the weights for each of the plurality of component reward functions, A non-temporary computer-readable storage medium characterized by including [a certain element].

19. The aforementioned artificial intelligence agent is part of a racing game environment. The method according to claim 18.

20. The method further includes evaluating the single policy of the artificial intelligence agent at inference time by selecting selected weights for each of the plurality of component reward functions, wherein the artificial intelligence agent operates optimally under the selected reward function without retraining. The method according to claim 18.

Images