Action output method, network training method and device for reinforcement learning process

Abstract

Disclosed are an action output method, a network training method, and a device for a reinforcement learning process. The action output method for a reinforcement learning process includes: determining a first state of an environment in which an intelligent agent is located at a current time point; determining a first candidate action for the intelligent agent at a current time point based on the first state and a first historical action output by the intelligent agent to the environment at a previous time point; selecting a target action from the first candidate action and the first historical action; and controlling the intelligent agent to output the target action at the current time point. In an embodiment of the present disclosure, the same action can be repeated multiple times by a binary switching decision to achieve the effect that the action spans multiple time points, thereby shortening the task timeline, and further simplifying the reward distribution problem, which helps to ensure the implementation of intelligent agents enabled by deep reinforcement learning in actual application scenarios.

Description

Technical Field

[0001] The present disclosure relates to the technical field of reinforcement learning, and in particular to an action output method, a network training method and a device for a reinforcement learning process. Background Art

[0002] At present, the use of reinforcement learning is becoming more and more common. When using reinforcement learning methods, when faced with a new control task, robots and other intelligent agents may need a lot of trial and error, spend a lot of time, and risk hardware damage in the process before they can learn the task through reward signals. It should be noted that reinforcement learning methods need to solve the credit assignment problem. Generally speaking, the longer the timeline of the task, the more difficult it is to solve the credit assignment problem, which will affect the implementation of intelligent agents enabled by deep reinforcement learning in actual application scenarios. Summary of the invention

[0003] In order to solve the above technical problems, the present disclosure is proposed. The embodiments of the present disclosure provide an action output method, a network training method and a device for a reinforcement learning process.

[0004] According to one aspect of an embodiment of the present disclosure, there is provided an action output method for a reinforcement learning process, comprising:

[0005] Determine a first state of the agent's environment at the current time point;

[0006] Determine a first candidate action for the agent at a current time point based on the first state and a first historical action output by the agent to the environment at a previous time point;

[0007] Selecting a target action from the first candidate action and the first historical action;

[0008] Control the agent to output the target action at the current time point.

[0009] According to another aspect of an embodiment of the present disclosure, a network training method is provided, including:

[0010] Acquire historical data, wherein the acquired historical data records a second state of an environment in which an agent is located at a first time point and a second historical action output by the agent to the environment at a second time point, wherein the first time point is a time point next to the second time point;

[0011] Based on the second state and the second historical action, determining, via a second network, a second candidate action for the agent at the first time point;

[0012] Determining a probability of selecting the second candidate action based on the second state, the second historical action, and the second candidate action;

[0013] Determine a parameter gradient of the second network based on the second state, the second historical action, the second candidate action, and the selected probability;

[0014] The second network is trained based on the parameter gradient.

[0015] According to another aspect of the embodiments of the present disclosure, there is provided a motion output device for a reinforcement learning process, comprising:

[0016] A first determination module, used to determine a first state of the environment where the agent is located at a current time point;

[0017] A second determination module, configured to determine a first candidate action for the agent at a current time point based on the first state determined by the first determination module and a first historical action output by the agent to the environment at a previous time point;

[0018] A selection module, configured to select a target action from the first candidate action and the first historical action determined by the second determination module;

[0019] An output module is used to control the agent to output the target action selected by the selection module at a current time point.

[0020] According to another aspect of the embodiments of the present disclosure, a network training device is provided, including:

[0021] A second acquisition module is used to acquire historical data, wherein the acquired historical data records a second state of the environment where the agent is located at a first time point and a second historical action output by the agent to the environment at a second time point, wherein the first time point is a time point next to the second time point;

[0022] a third determining module, configured to determine, via a second network, a second candidate action for the agent at the first time point based on the second state and the second historical action recorded in the historical data acquired by the acquiring module;

[0023] a fourth determination module, configured to determine a probability of the second candidate action being selected based on the second state and the second historical action recorded in the historical data acquired by the acquisition module, and the second candidate action determined by the third determination module;

[0024] a fifth determination module, configured to determine a parameter gradient of the second network based on the second state and the second historical action recorded in the historical data acquired by the acquisition module, the second candidate action determined by the third determination module, and the probability of being selected determined by the fourth determination module;

[0025] A second training module is used to train the second network based on the parameter gradient determined by the fifth determination module.

[0026] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, wherein the computer program is used to execute the above-mentioned action output method for a reinforcement learning process, or to execute the above-mentioned network training method.

[0027] According to another aspect of the embodiments of the present disclosure, there is provided an electronic device, including:

[0028] processor;

[0029] a memory for storing instructions executable by the processor;

[0030] The processor is used to read the executable instructions from the memory and execute the instructions to implement the above-mentioned action output method for the reinforcement learning process, or to execute the above-mentioned network training method.

[0031] Based on the above-mentioned embodiments of the present disclosure, an action output method, network training method, device, computer-readable storage medium and electronic device for a reinforcement learning process can be provided. The first candidate action for the agent at the current time point can be determined based on the first state of the environment in which the agent is located at the current time point and the action output to the environment by the agent at the previous time point, and a target action can be selected from the first candidate action and the first historical action. The agent can then be controlled to output the target action at the current time point, which is equivalent to selecting a suitable action from the first candidate action and the first historical action to output to the environment through a binary switching decision. Theoretically, the same action can be repeated multiple times by the binary switching decision to achieve the effect that the action spans multiple time points, thereby shortening the task timeline and simplifying the reward distribution problem. This helps to ensure the practical use of agents enabled by deep reinforcement learning in actual application scenarios, and is conducive to improving the problem of low data utilization efficiency in deep reinforcement learning.

[0032] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0034] Figure 1 It is a flowchart of an action output method for a reinforcement learning process provided by an exemplary embodiment of the present disclosure.

[0035] Figure 2 It is a schematic diagram of the action decision process in an embodiment of the present disclosure.

[0036] Figure 3 It is a flowchart of an action output method for a reinforcement learning process provided by another exemplary embodiment of the present disclosure.

[0037] Figure 4 It is a flowchart of an action output method for a reinforcement learning process provided by yet another exemplary embodiment of the present disclosure.

[0038] Figure 5 It is a flowchart of a network training method provided by an exemplary embodiment of the present disclosure.

[0039] Figure 6 It is a flowchart of a network training method provided by another exemplary embodiment of the present disclosure.

[0040] Figure 7 It is a structural diagram of an action output device for a reinforcement learning process provided by an exemplary embodiment of the present disclosure.

[0041] Figure 8 It is a structural diagram of an action output device for a reinforcement learning process provided by another exemplary embodiment of the present disclosure.

[0042] Fig. 9 It is a structural diagram of a network training device provided by an exemplary embodiment of the present disclosure.

[0043] Fig.10 It is a structural diagram of a network training device provided by another exemplary embodiment of the present disclosure.

[0044] Fig.11 is a structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0045] Below, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0046] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0047] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, etc., and neither represent specific technical meanings nor indicate a necessary logical order.

[0048] It should also be understood that in the embodiments of the present disclosure, “plurality” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0049] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0050] The term "and / or" in this disclosure is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this disclosure generally indicates that the associated objects before and after are in an "or" relationship.

[0051] It should also be understood that the description of the various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0052] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0053] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0054] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0055] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0056] The disclosed embodiments can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0057] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0058] Application Overview

[0059] In some cases, there is a demand to enable intelligent agents to efficiently and autonomously learn to complete different control tasks from different experience data through the same set of algorithms, while avoiding human intervention and reprogramming as much as possible in the process; intelligent agents include but are not limited to unmanned vehicles, robots, etc.; control tasks include but are not limited to autonomous driving control, robot motion, robotic arm control, etc.

[0060] In response to the above needs, a promising research branch is to use deep reinforcement learning (Deep RL) to allow the agent to continuously try and adjust its behavior from the task reward signal, so as to ultimately achieve the goal of completing the task. Common algorithms with wide applications in this research branch include: Deep Deterministic Policy Gradient (DDPG), Soft Actor-Critic (SAC), and Proximal Policy Optimization (PPO). The characteristics of the method of this research branch are: emphasizing data-driven learning, basically not requiring professional knowledge in the control field, and good transferability to different control tasks. However, due to the data hunger caused by data-driven learning, when facing a new control task, the agent may need a lot of trial and error, spend a lot of time, and risk hardware damage in the process to learn the task through reward signals.

[0061] It should be noted that reinforcement learning methods often have the problem of low data utilization efficiency (sample efficiency). Taking the currently popular method SAC as an example, the amount of data required to learn a simple object grabbing operation is probably several orders of magnitude more than what humans need. An important reason for the low data utilization efficiency is that reinforcement learning methods need to solve the reward allocation problem. A current action may not play a key role in getting rewards / punishments until a certain point in the future on the timeline, and this reward / punishment signal needs to be passed from back to front step by step for review. In this process, the intelligent agent needs to correctly attribute this signal to a specific early behavior. Generally speaking, the longer the timeline of the task, the more difficult it is to solve the reward allocation problem, which makes it difficult for intelligent agents enabled by deep reinforcement learning to land in actual application scenarios.

[0062] In order to solve the problem of low data utilization efficiency, one subfield of deep reinforcement learning is the hierarchical control method, and the other subfield is the action duration prediction method. The hierarchical control method abstracts the action of the agent by establishing a multi-level control timeline. The timeline of each level is greatly shortened, thereby simplifying the reward distribution problem. However, in the hierarchical control method, the multi-level structure is generally proposed for specific scene tasks, and it is not easy to automatically migrate to new scenes. The hierarchical structure design between different scenes has low sharing, and the multi-level model is generally not very stable and more difficult to train. The action span prediction method outputs the number of times the action is executed before outputting each action. This method belongs to the open-loop control method and has low flexibility. If a longer action execution span has been predicted, it cannot immediately stop the repeated execution of the action in an emergency in the environment.

[0063] Exemplary Systems

[0064] In the embodiments of the present disclosure, reinforcement learning can be considered as a decision search problem in a Markov decision process (MDP). It is understandable that an MDP contains a set of interactive objects, namely, an agent and the environment in which the agent is located; wherein the agent is an agent that performs machine learning in the MDP, can perceive the state of the environment and make decisions, take actions on the environment and adjust the decisions through feedback from the environment; the environment is a collection of all things outside the agent in the MDP, whose state will change due to the influence of the agent's actions, and the change can be fully or partially perceived by the agent, and the environment may give feedback to the agent after each decision.

[0065] In specific implementation, intelligent agents, terminal devices, servers and other electronic devices can execute the action output method for the reinforcement learning process provided in the following embodiments of the present disclosure to shorten the task timeline, thereby simplifying the reward distribution problem and ensuring the implementation of the intelligent agent enabled by deep reinforcement learning in actual application scenarios. For ease of understanding, the following description is based on the case where the electronic device that executes the above action output method is an intelligent agent. In addition, intelligent agents, terminal devices, servers and other electronic devices can also execute the network training method provided in the following embodiments of the present disclosure to train an accurate and reliable second network so that the second network can be applied to the above action output method.

[0066] Exemplary Methods

[0067] Figure 1It is a flowchart of an action output method for a reinforcement learning process provided by an exemplary embodiment of the present disclosure. Figure 1 The method shown includes step 101, step 102, step 103 and step 104, and each step is described below.

[0068] Step 101, determining a first state of the environment in which the agent is located at the current time point.

[0069] It should be noted that the time points involved in the embodiments of the present disclosure may also be referred to as time steps, so the current time point in step 101 may also be referred to as the current time step, and the previous time point in the following step 102 may also be referred to as the previous time step. In addition, the intelligent agent involved in the embodiments of the present disclosure may be any device capable of deep reinforcement learning, including but not limited to unmanned vehicles, robots, etc., and the robots may specifically include intelligent robots such as sweeping robots.

[0070] In step 101, the agent may sense its own environment to determine the state of its own environment at the current time point, and this state may be used as the first state. Optionally, when the agent is an unmanned vehicle, the first state may be represented by camera image data of the unmanned vehicle; when the agent is a sweeping robot, the first state may be represented by laser radar data of the sweeping robot.

[0071] Step 102 , based on the first state and the first historical action output by the agent to the environment at the previous time point, determine a first candidate action for the agent at the current time point.

[0072] It should be noted that, in the process of deep reinforcement learning, the actions that the agent has output to the environment at various time points can be recorded, for example, in a data revisit cache; wherein the data revisit cache can be a data cache area set inside the agent. Optionally, in the case where the agent is an unmanned vehicle, the actions that the agent has output to the environment can include the motor torque change action of the unmanned vehicle; in the case where the agent is a sweeping robot, the actions that the agent has output to the environment can include the speed change action of the sweeping robot, the steering action of the sweeping robot, etc.

[0073] In step 102, based on the information recorded in the data revisit cache, the action outputted to the environment by the agent at the previous time point can be determined very conveniently and reliably, and the action can be used as the first historical action; then, based on the first state and the first historical action, the first candidate action for the agent at the current time point can be determined. Optionally, the specific method of determining the first candidate action can refer to the method of determining the action to be outputted to the environment at the current time point based on the state at the current time point and the action outputted to the environment at the previous time point in the current SAC, which will not be repeated here.

[0074] Step 103: Select a target action from the first candidate action and the first historical action.

[0075] In step 103, the relative quality of the first candidate action and the first historical action can be compared, so as to select the action that is more suitable for the current time point from the first candidate action and the first historical action according to the comparison result, and use the selected action as the target action.

[0076] Step 104, controlling the agent to output the target action at the current time point.

[0077] In step 104, the agent can control itself to output the target action to the environment at the current time point. At this time, the state of the environment will change due to the influence of the target action, and the change can be perceived by the agent, and the agent can execute the above step 101 again.

[0078] It should be noted that steps 101 to 103 in the embodiments of the present disclosure can be regarded as an action decision process, which includes two stages of decision making, namely, the first stage of decision making and the second stage of decision making. Figure 2 As shown in the figure, the first state of the environment where the agent is located at the current time point can be represented by S, and the first historical action output by the agent to the environment at the previous time point can be represented by a - Indicates that when making the first stage decision, we can base our decision on S and a - , determine the first candidate action for the agent at the current time point, the first candidate action can be used Indicates; when making the second stage decision, it can be compared with a as the first historical action - and as the first candidate action The relative goodness of - and Select an action as the target action, which can be represented as a.

[0079] Since the first candidate action is determined The method can refer to the corresponding method in SAC. It can be considered that on the basis of SAC, the embodiments of the present disclosure propose and use a temporally abstract soft actor-critc (TASAC). TASAC adds a second-stage decision after the SAC decision (which corresponds to the decision in the first stage above). The decision can be called a binary switch policy. The binary switch decision can decide which action to use as the action output to the environment in the current time step (e.g., the target action) by comparing the relative quality of the action generated by SAC (e.g., the first candidate action) and the action output to the environment in the previous time step (e.g., the first historical action), that is, decide whether to stop repeating the previous action (e.g., the first historical action).

[0080] Based on the above embodiment of the present disclosure, a method for outputting an action for a reinforcement learning process can be provided. Based on the first state of the environment in which the agent is located at the current time point and the action output to the environment by the agent at the previous time point, the first candidate action for the agent at the current time point can be determined, and a target action can be selected from the first candidate action and the first historical action. After that, the agent can be controlled to output the target action at the current time point, which is equivalent to selecting a suitable action from the first candidate action and the first historical action to output to the environment through a binary switching decision. In theory, the same action can be repeated multiple times by the binary switching decision to achieve the effect that the action spans multiple time points, thereby shortening the task timeline, and thus simplifying the reward allocation problem, which helps to ensure the landing use of the agent enabled by deep reinforcement learning in actual application scenarios, and is conducive to improving the problem of low data utilization efficiency in deep reinforcement learning. In addition, the binary switching decision in the embodiment of the present disclosure can be applied to a variety of scenarios, not limited to specific scenarios, and the binary switching decision belongs to closed-loop control, which can decide whether to stop repeating the previous action (such as the first historical action) according to the specific state of the environment (such as the first state), so it has high flexibility and can handle emergencies in the environment.

[0081] exist Figure 1 Based on the embodiment shown, Figure 3 As shown, step 103 includes:

[0082] Step 1031: Based on the first state and the first candidate action, obtain a future reward impact prediction value corresponding to the first candidate action via the first network.

[0083] Here, the first network may be a network obtained through training, which can calculate and output the predicted value of future reward impact according to a given state and action. Optionally, when the first network has been trained, the first network may be updated and trained regularly or irregularly.

[0084] In step 1031, the first state and the first candidate action only need to be provided to the first network, and the first network can perform calculation processing by itself to obtain and output a future reward impact prediction value, which can be used as the future reward impact prediction value corresponding to the first candidate action. It should be noted that the future reward impact prediction value corresponding to the first candidate action can refer to: in the first state, when the first candidate action is selected to be output to the environment at the current time point, it is expected that the sum of all rewards that the intelligent agent can get from the environment in the future.

[0085] Step 1032: Based on the first state and the first historical action, obtain a future reward impact prediction value corresponding to the first historical action via the first network.

[0086] Similar to step 1031, in step 1032, it is only necessary to provide the first state and the first historical action to the first network, and the first network can perform calculation processing by itself to obtain and output a future reward impact prediction value, which can be used as the future reward impact prediction value corresponding to the first historical action. It should be noted that the future reward impact prediction value corresponding to the first historical action may refer to: in the first state, when the first historical action is selected to be output to the environment at the current time point, it is expected that the sum of all rewards that the intelligent agent can get from the environment in the future.

[0087] Step 1033: Select, from the first candidate action and the first historical action, an action corresponding to a larger predicted value of future reward impact as the target action.

[0088] In step 1033, the predicted future reward impact values ​​corresponding to the first candidate action and the first historical action may be compared. If the comparison result shows that the predicted future reward impact value corresponding to the first candidate action is greater than the predicted future reward impact value corresponding to the first historical action, it can be considered that the first candidate action is better than the first historical action and the first candidate action is more suitable for the current time point, then the first candidate action can be selected as the target action; otherwise, the first historical action can be selected as the target action.

[0089] It can be seen that in the embodiments of the present disclosure, through the first network, the future reward impact prediction values ​​corresponding to the first candidate action and the first historical action can be determined very conveniently and reliably, so as to compare the relative quality of the first candidate action and the first historical action, thereby conveniently and reliably selecting a suitable action from the first candidate action and the first historical action as the target action and output it to the environment.

[0090] exist Figure 3 Based on the embodiment shown, Figure 4 As shown, the method also includes:

[0091] Step 111, obtain multiple sets of historical data, each set of historical data obtained records the state of the environment, the action output by the intelligent agent to the environment, the next state of the environment, and the reward generated by the state transition of the environment, and the multiple sets of historical data are adjacent to each other on the timeline, and the actions recorded in the multiple sets of historical data are the same.

[0092] It should be noted that in the process of deep reinforcement learning, the state of the environment (which can be expressed as s t ), the action output by the agent to the environment (which can be represented as a t ), the next state of the environment (which can be represented as s t+1 ), the reward generated by the state of the environment transferring to the next state (which can be expressed as r(s t , a t ,s t+1 )), and record the determined state, action, next state, and reward through a set of historical data. In this way, each set of historical data can correspond to a historical time point respectively, and each set of historical data can be expressed as (state, action, next state, reward generated by this state transition) or (s t , a t ,s t+1 ,r(s t , a t ,s t+1 )). Optionally, all historical data may be stored in a data revisit cache.

[0093] In step 111, the data revisit cache may be sampled to obtain multiple sets of historical data from the data revisit cache. The multiple sets of historical data may be adjacent to each other in the timeline and record actions (i.e., s t It should be noted that two sets of historical data are adjacent on the timeline means that the historical time point corresponding to one set of historical data is the later time point of the historical time point corresponding to the other set of historical data.

[0094] Optionally, when obtaining multiple groups of historical data from the data revisit cache, in addition to referring to the condition that the historical data are adjacent on the timeline and the recorded actions are the same (for the sake of convenience, it will be referred to as the first condition later), other conditions can also be referred to. For example, if there are 10 groups of historical data in the data revisit cache that are adjacent to each other on the timeline and record the same action (assuming that the action is action C), and in the order from early to late of the corresponding historical time points, these 10 groups of historical data are the first group of historical data, the second group of historical data, ..., the tenth group of historical data, then the action in the first group of historical data can be used as the historical action, and the state in the second group of historical data can be used as the state of the environment in which the intelligent agent is located, and the two-stage decision in the above text is used to determine the action to be output to the environment (equivalent to the process of determining the target action in the above text); the action in the second group of historical data can be used as the historical action, and the state in the third group of historical data can be used as the state of the environment in which the intelligent agent is located, and the two-stage decision in the above text is used to determine the action to be output to the environment; in a similar manner, a total of 9 actions to be output to the environment can be determined. Assuming that the 9 actions to be output to the environment are all action C, the 10 groups of historical data can be obtained from the data revisit cache for the subsequent network training in step 112; assuming that only the first Y (Y is an integer not greater than 8) of the 9 actions to be output to the environment are action C, the first Y+1 groups of historical data among the 10 groups of historical data can be obtained from the data revisit cache for the subsequent network training in step 112. It should be noted that in this example, the other conditions are used to ensure that the action to be output determined under the same state is the same as the action recorded in the historical data based on the two-stage decision, which is conducive to ensuring the accuracy and reliability of the first network trained in step 112. For the sake of convenience, the other conditions will be referred to as the second conditions later.

[0095] Step 112, using multiple groups of historical data as training data and utilizing a multi-step bootstrap method to train the first network.

[0096] Optionally, when the first network is trained using the multi-step bootstrap method, the first network may be trained using the following objective function:

[0097]

[0098]

[0099]

[0100] θ represents the algorithm parameters of the algorithm used by the first network to calculate the predicted value of future reward impact, D represents a set of multiple groups of historical data, s t 、a t 、s t+1It represents the state, action, and next state in the same historical data. Q represents the algorithm used by the first network to calculate the predicted value of future reward impact. represents the historical version of θ, γ represents the discount constant (which means that future rewards will depreciate with time), t represents the earliest time point corresponding to multiple sets of historical data, T represents the latest time point corresponding to multiple sets of historical data, r represents the reward, V is obtained by calculating the expectation of Q for the action under the same state, π ta represents a two-stage decision distribution (corresponding to the two-stage decision in the above text), where the first stage is used to determine the candidate action (corresponding to the first stage decision in the above text), and the second stage is used to select the target action (corresponding to the second stage decision in the above text), S T+1 Indicates the state of the agent at the next time point corresponding to the latest time point of multiple sets of historical data, a T Represents the action output to the environment at the latest time point corresponding to multiple sets of historical data.

[0101] Assuming that the multiple sets of historical data are specifically 10 sets of historical data, and the 10 historical time points corresponding to these 10 sets of historical data are historical time point 1, historical time point 2,..., historical time point 10 in order from early to late, then the earliest time point t corresponding to the multiple sets of historical data is specifically historical time point 1, and the latest time point T corresponding to the multiple sets of historical data is specifically historical time point 10.

[0102] In specific implementation, when multiple groups of historical data are obtained from the data revisit cache, the state, action, next state, and reward recorded in each group of historical data in the multiple groups of historical data can be substituted into the above objective function. By minimizing the value of the objective function, the algorithm parameter θ of the algorithm used by the first network to calculate the predicted value of the impact of future rewards can be calculated. When θ is calculated, it can be considered that the first network is successfully trained. At this time, as long as the state and action are given, the first network can calculate and output the predicted value of the impact of future rewards.

[0103] In the embodiment of the present disclosure, since TASAC adds a second-stage decision (i.e., a binary switching decision) after the SAC decision, there will be a situation where the same action is repeatedly output in multiple consecutive time steps. Therefore, based on the first condition and the second condition above, multiple groups of historical data can be acquired, and the multiple groups of historical data can be used as training data to train the first network using the multi-step bootstrap method. This is equivalent to using the bootstrap method between the first time step and the last time step in multiple time steps corresponding to the same action, without being limited to using the bootstrap method only between two adjacent time steps in SAC, which results in the need to use the bootstrap method multiple times, thereby effectively improving the training speed of the first network (i.e., improving the learning efficiency of the Q value).

[0104] It should be noted that due to the existence of two-stage decision-making, TASAC in the embodiment of the present disclosure can be regarded as a hierarchical control method. Due to the existence of temporal abstraction, TASAC in the embodiment of the present disclosure can have the following two important features: 1. Persistent exploration in a certain direction of action; 2. Multi-step temporal difference learning (multi-step TD learning). The above two features can greatly improve the efficiency of data utilization, so that TASAC can achieve ideal learning effects with very little data compared to SAC in multiple control tasks.

[0105] Figure 5 It is a flowchart of a network training method provided by an exemplary embodiment of the present disclosure. Figure 5 The method shown includes step 501, step 502, step 503, step 504 and step 505, and each step is described below.

[0106] Step 501, obtain historical data, the obtained historical data records the second state of the environment where the agent is located at a first time point and the second historical action output by the agent to the environment at a second time point, and the first time point is the next time point of the second time point.

[0107] It should be noted that in the deep reinforcement learning process, the state of the environment in which the agent is located and the action output by the agent to the environment at the next time point after the time point can be determined for any past time point, and the determined state and action can be recorded through a set of historical data. Optionally, all historical data can be stored in a data revisit cache; wherein the data revisit cache can be a data cache area set inside the agent.

[0108] In step 501, the data revisit cache can be sampled, for example, the data revisit cache can be sampled with equal probability to obtain historical data from the data revisit cache. The state recorded in the obtained historical data can be used as the second state, and the action recorded in the obtained historical data can be used as the second historical action.

[0109] Optionally, the number of historical data obtained from the data revisit cache may be multiple groups, and the following text only takes the related processing for one group of historical data as an example for description.

[0110] Step 502: Based on the second state and the second historical action, determine a second candidate action for the agent at the first time point via the second network.

[0111] Here, the second network may be a network obtained through training and capable of determining candidate actions according to given states and actions. Optionally, steps 501 to 505 may be performed periodically, and the second network used in step 502 in the current cycle may be the second network trained by step 505 in the previous cycle.

[0112] In step 502, the second state and the second historical action only need to be provided to the second network, and the second network can perform calculations and processing by itself to determine a candidate action, which can be used as the second candidate action for the agent at the first time point.

[0113] Step 503: Determine the probability of the second candidate action being selected based on the second state, the second historical action and the second candidate action.

[0114] Here, the probability of the second candidate action being selected may refer to: in the second state, when TASAC is used, the probability of selecting the second candidate action as the action output to the environment at the first time point from the second historical action and the second candidate action. Optionally, the probability of the second candidate action being selected may be expressed as

[0115] Step 504: Determine a parameter gradient of the second network based on the second state, the second historical action, the second candidate action, and the probability of being selected.

[0116] Here, a formula for calculating the parameter gradient can be predefined. In step 504, by substituting the second state, the second historical action, the second candidate action and the probability of being selected into the formula, a parameter gradient can be conveniently and reliably calculated, and the parameter gradient can be used as the parameter gradient of the second network. Optionally, the parameter gradient of the second network can be expressed as Δφ.

[0117] Step 505: Train the second network based on the parameter gradient.

[0118] Here, each parameter in the second network can be adjusted based on the parameter gradient to achieve the update training of the second network. It should be noted that there is also a process of network training based on parameter gradient in SAC. The specific implementation process of step 505 can refer to the relevant training process in SAC, which will not be repeated here.

[0119] Based on a network training method provided by the above-mentioned embodiment of the present disclosure, after obtaining historical data recording the second state of the environment in which the agent is located at a first time point (which is the next time point of the second time point) and the second historical action output by the agent to the environment at the second time point, the second candidate action for the agent at the first time point can be determined through the second network based on the second state and the second historical action. Next, the probability of the second candidate action being selected can be determined based on the second state, the second historical action and the second candidate action. After that, the parameter gradient of the second network can be determined based on the second state, the second historical action, the second candidate action and the probability of being selected, and the second network can be trained based on the parameter gradient to achieve updated training of the second network. The second network after updated training can be used for determining the first candidate action in an action output method for a reinforcement learning process provided by the above-mentioned embodiment of the present disclosure to ensure the accuracy and reliability of the determined first candidate action, thereby ensuring the reasonable reliability of the target action finally selected and output, which is beneficial to ensuring the effect of deep reinforcement learning.

[0120] exist Figure 5 Based on the embodiment shown, Figure 6 As shown, step 503 includes:

[0121] Step 5031: Based on the second state and the second candidate action, obtain the future reward impact prediction value corresponding to the second candidate action via the first network.

[0122] Here, the first network may be a network obtained through training that can calculate and output the predicted value of future reward impact according to a given state and action. The training method of the first network may refer to the description of the corresponding part of the action output method for a reinforcement learning process provided in the above embodiment of the present disclosure, and will not be repeated here.

[0123] In step 5031, the second state and the second candidate action only need to be provided to the second network, and the second network can perform calculation processing by itself to obtain and output a future reward impact prediction value, which can be used as the future reward impact prediction value corresponding to the second candidate action. It should be noted that the future reward impact prediction value corresponding to the second candidate action can refer to: in the second state, when the second candidate action is selected to be output to the environment at the first time point, it is expected that the sum of all rewards that the intelligent agent can get from the environment in the future.

[0124] Step 5032: Based on the second state and the second historical action, obtain a future reward impact prediction value corresponding to the second historical action via the first network.

[0125] Similar to step 5031, in step 5032, it is only necessary to provide the second state and the second historical action to the second network, and the second network can perform calculation processing by itself to obtain and output a future reward impact prediction value, which can be used as the future reward impact prediction value corresponding to the second historical action. It should be noted that the future reward impact prediction value corresponding to the second historical action may refer to: in the second state, when the second historical action is selected to be output to the environment at the first time point, it is expected that the sum of all rewards that the intelligent agent can get from the environment in the future.

[0126] Step 5033, based on the future reward impact prediction values ​​corresponding to the second candidate action and the second historical action, determine the probability of the second candidate action being selected.

[0127] In a specific implementation, step 5033 includes:

[0128] Calculate a first ratio of the predicted value of the future reward impact corresponding to the second candidate action to a specified constant;

[0129] Calculate a second ratio of the predicted value of the future reward impact corresponding to the second historical action to a specified constant;

[0130] Determining a first processing result based on a preset constant and a first ratio;

[0131] Determining a second processing result based on the preset constant and the second ratio;

[0132] Determining a normalized score of the first processing result based on the first processing result and the second processing result;

[0133] Based on the normalized score, the probability of the second candidate action being selected is determined.

[0134] Here, the predicted value of future reward impact corresponding to the second candidate action can be expressed as Q(1), the predicted value of future reward impact corresponding to the second historical action can be expressed as Q(0), the specified constant can be expressed as α, and the preset constant can be expressed as e.

[0135] In this implementation, Q(1) may be divided by α to obtain z1 as a first ratio, and Q(0) may be divided by α to obtain z2 as a second ratio.

[0136] Next, the first processing result can be determined based on e and z1, and the second processing result can be determined based on e and z2. Optionally, an exponential function operation can be performed with e as the base and z1 as the exponent to obtain the operation result as the first processing result, so that the first processing result can be expressed as exp(z1); similarly, an exponential function operation can be performed with e as the base and z2 as the exponent to obtain the operation result as the second processing result, so that the second processing result can be expressed as exp(z2).

[0137] Afterwards, the normalized score of exp(z1) as the first processing result may be determined based on exp(z1) as the first processing result and exp(z2) as the second processing result. Optionally, the normalized score of exp(z1) may be expressed as the following formula:

[0138]

[0139] Similarly, the normalized fraction of exp(z2) can be expressed as follows:

[0140]

[0141] Afterwards, the probability of the second candidate action being selected may be determined based on the normalized score of exp(z1). For example, the normalized score of exp(z1) may be directly used as the probability of the second candidate action being selected.

[0142] In this implementation, the probability of the second candidate action being selected can be calculated very conveniently and reliably by combining the predicted value of the future reward impact corresponding to the second candidate action, the predicted value of the future reward impact corresponding to the second historical action, the specified constant, the preset constant, and simple division operations, exponential function operations, and normalization operations.

[0143] It can be seen that in the embodiments of the present disclosure, through the first network, the future reward impact prediction values ​​corresponding to the second candidate action and the second historical action can be determined very conveniently and reliably, and based on the determined future reward impact prediction values, the probability of the second candidate action being selected can be reasonably determined, so that the probability of the second candidate action being selected can be used for training the second network.

[0144] In an optional example, the formula used to determine the parameter gradient of the second network may be:

[0145]

[0146] Δφ represents the parameter gradient, represents the probability of being selected, Q represents the algorithm used by the first network to calculate the predicted value of future reward impact, θ represents the algorithm parameters of the algorithm used by the first network to calculate the predicted value of future reward impact, s represents the second state, represents the second candidate action, α represents a specified constant, π represents the algorithm used by the second network to determine the candidate action, φ represents the algorithm parameter of the algorithm used by the second network to determine the candidate action, a - represents the second historical action, and f represents the algorithm used by the specified subnetwork of the second network.

[0147] In the embodiment of the present disclosure, if Q, θ, π, and φ are known, the second state, the second candidate action, and the second historical action can be substituted into the above formula to conveniently and reliably calculate Δφ as the parameter gradient of the second network. By comparing with the parameter gradient in the current SAC, the parameter gradient calculated in the embodiment of the present disclosure is actually There is an extra coefficient in front

[0148] The following is an introduction to the origin of the formula used in determining the parameter gradient of the second network.

[0149] Assume that the decision in the second stage above is expressed as Where b=0 means selecting a - As the target action, b = 1 means selection As the target action, in order to simplify the expression, we record Then, the decision distribution of the two stages above can be expressed as:

[0150]

[0151] Assumptions Then we have:

[0152]

[0153] The following derivation process can be followed:

[0154]

[0155] in,

[0156] To learn how to get π φ and β, the following objective function can be set:

[0157]

[0158] Then for any The combination of can derive an optimal β distribution, which can be expressed as:

[0159]

[0160] The derivation process can be specifically:

[0161] Assume that we have N values, any value can be represented as X(i), i = 0, ..., N-1, and we want to find a discrete distribution P through the following objective function:

[0162]

[0163]

[0164] Among them, α can be greater than 0.

[0165] By using the Lagrange multiplier λ, the objective function can be transformed into an unconstrained optimization problem, namely:

[0166]

[0167] By taking the derivative of each P(i) in the above equation and setting it to 0, we can get the following equation:

[0168]

[0169] Among them, λ needs to ensure that the following formula is satisfied:

[0170]

[0171] In order to give any (s, a - )~D, b~β all derive the optimal β distribution, we can let And order In this way, the optimal β distribution can be like P above * The same was found.

[0172] It should be noted that, intuitively speaking, the form of the above optimal β distribution tells us that the selection of the action at the previous moment or the current candidate action is mainly to compare their corresponding Q values. The probability of the action being selected is proportional to the exp() of the Q value. In view of this, in the specific implementation of the above step 5033, the probability of the second candidate action being selected can be calculated by combining division operation, exponential function operation, and normalization operation. In addition, by substituting this optimal β distribution form into the learning method to obtain π φ The objective function set by and β can be simplified to obtain a function that is only related to π as shown below: φ The objective function is:

[0173]

[0174] By taking the derivative of φ with this objective function, we can get the formula used to determine the parameter gradient of the second network above.

[0175] It should be noted that in the above text, when the multi-step bootstrap method is used to train the first network to improve the training speed of the first network, it is also beneficial to improve the training speed of the second network (i.e., promote π φ The learning is accelerated), thus, the TASAC in the embodiments of the present disclosure will be more data efficient than the current SAC.

[0176] Any action output method for a reinforcement learning process provided in the embodiments of the present disclosure may be executed by any appropriate device with data processing capabilities, including but not limited to: a terminal device, a server, an intelligent agent, etc. Alternatively, any action output method for a reinforcement learning process provided in the embodiments of the present disclosure may be executed by a processor, such as the processor executing any action output method for a reinforcement learning process mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.

[0177] Any network training method provided in the embodiments of the present disclosure may be executed by any appropriate device with data processing capabilities, including but not limited to: terminal devices, servers, agents, etc. Alternatively, any network training method provided in the embodiments of the present disclosure may be executed by a processor, such as the processor executing any network training method mentioned in the embodiments of the present disclosure by calling corresponding instructions stored in a memory. This will not be described in detail below.

[0178] Exemplary Devices

[0179] Figure 7 It is a structural diagram of an action output device for a reinforcement learning process provided by an exemplary embodiment of the present disclosure. Figure 7 The device shown includes a first determination module 701 , a second determination module 702 , a selection module 703 and an output module 704 .

[0180] A first determination module 701 is used to determine a first state of the environment where the agent is located at a current time point;

[0181] A second determination module 702 is used to determine a first candidate action for the agent at a current time point based on the first state determined by the first determination module 701 and a first historical action output by the agent to the environment at a previous time point;

[0182] A selection module 703, configured to select a target action from the first candidate action and the first historical action determined by the second determination module 702;

[0183] The output module 704 is used to control the agent to output the target action selected by the selection module 703 at the current time point.

[0184] In an alternative example, Figure 8 As shown, the selection module 703 includes:

[0185] A first acquisition submodule 7031 is used to acquire, via the first network, a future reward impact prediction value corresponding to the first candidate action determined by the second determination module 702 based on the first state determined by the first determination module 701 and the first candidate action determined by the second determination module 702;

[0186] A second acquisition submodule 7032 is used to acquire, via the first network, a future reward impact prediction value corresponding to the first historical action based on the first state and the first historical action determined by the first determination module 701;

[0187] The selection submodule 7032 is used to select, from the first candidate action and the first historical action determined by the second determination module 702, the action with the larger corresponding future reward impact prediction value among the future reward impact prediction values ​​obtained by the first acquisition submodule 7031 and the second acquisition submodule 7032 as the target action.

[0188] In an alternative example, Figure 8 As shown, the device also includes:

[0189] The first acquisition module 711 is used to acquire multiple sets of historical data, each set of historical data acquired records the state of the environment, the action output by the agent to the environment, the next state of the environment, and the reward generated by the state transition of the environment, and the multiple sets of historical data are adjacent to each other on the timeline, and the actions recorded in the multiple sets of historical data are the same;

[0190] The first training module 712 is used to train the first network using the multiple groups of historical data obtained by the first obtaining module 711 as training data and using a multi-step bootstrap method.

[0191] Optionally, the first acquisition module 711 may acquire multiple sets of historical data from the data revisit cache based only on the first condition above; or, the first acquisition module 711 may acquire multiple sets of historical data from the data revisit cache based on the first condition and the second condition above.

[0192] In an optional example, the first network is trained using the following objective function:

[0193]

[0194] θ represents the algorithm parameters of the algorithm used by the first network to calculate the predicted value of future reward impact, D represents a set of multiple groups of historical data, st 、a t 、s t+1 It represents the state, action, and next state in the same historical data. Q represents the algorithm used by the first network to calculate the predicted value of future reward impact. represents the historical version of θ, γ represents the discount constant, t represents the earliest time point corresponding to multiple sets of historical data, T represents the latest time point corresponding to multiple sets of historical data, r represents the reward, V is obtained by calculating the expectation of the action under the same state Q, π ta represents a two-stage decision distribution, where the first stage is used to determine the candidate actions and the second stage is used to select the target action. T+1 Indicates the state of the agent at the next time point corresponding to the latest time point of multiple sets of historical data, a T Represents the action output to the environment at the latest time point corresponding to multiple sets of historical data.

[0195] Fig. 9 It is a structural diagram of a network training device provided by an exemplary embodiment of the present disclosure. Fig. 9 The device shown includes a second acquisition module 901 , a third determination module 902 , a fourth determination module 903 , a fifth determination module 904 , and a second training module 905 .

[0196] The second acquisition module 901 is used to acquire historical data, wherein the acquired historical data records a second state of the environment where the agent is located at a first time point and a second historical action output by the agent to the environment at a second time point, wherein the first time point is a time point next to the second time point;

[0197] A third determination module 902 is used to determine a second candidate action for the agent at the first time point via a second network based on the second state and the second historical action recorded in the historical data acquired by the second acquisition module 901;

[0198] A fourth determination module 903 is used to determine the probability of the second candidate action being selected based on the second state and the second historical action recorded in the historical data acquired by the second acquisition module 901 and the second candidate action determined by the third determination module 902;

[0199] A fifth determination module 904 is used to determine the parameter gradient of the second network based on the second state and the second historical action recorded in the historical data acquired by the second acquisition module 901, the second candidate action determined by the third determination module 902, and the probability of being selected determined by the fourth determination module 903;

[0200] The second training module 905 is used to train the second network based on the parameter gradient determined by the fifth determination module 904.

[0201] In an alternative example, Fig.10 As shown, the fourth determination module 903 includes:

[0202] A third acquisition submodule 9031 is used to acquire, via the first network, a future reward impact prediction value corresponding to the second candidate action determined by the third determination module 902 based on the second state recorded in the historical data acquired by the second acquisition module 901 and the second candidate action determined by the third determination module 902;

[0203] A fourth acquisition submodule 9032 is used to acquire, via the first network, a future reward impact prediction value corresponding to the second historical action recorded in the historical data acquired by the second acquisition module 901 based on the second state and the second historical action recorded in the historical data acquired by the second acquisition module 901;

[0204] The determination submodule 9033 is used to determine the probability of selection of the second candidate action determined by the third determination module 902 based on the future reward impact prediction value corresponding to the second candidate action obtained by the third acquisition submodule 9031 and the future reward impact prediction value corresponding to the second historical action obtained by the fourth acquisition submodule 9032.

[0205] In an alternative example, Fig.10 As shown, the determination submodule 9033 includes:

[0206] A first calculation unit 90331 is used to calculate a first ratio of a future reward impact prediction value corresponding to the second candidate action obtained by the third acquisition submodule 9031 to a specified constant;

[0207] The second calculation unit 90332 is used to calculate a second ratio of the future reward impact prediction value corresponding to the second historical action obtained by the fourth obtaining submodule 9032 to a specified constant;

[0208] A first determining unit 90333 is used to determine a first processing result based on a preset constant and a first ratio calculated by the first calculating unit 90331;

[0209] A second determining unit 90334 is used to determine a second processing result based on a preset constant and a second ratio calculated by the second calculating unit 90332;

[0210] A third determining unit 90335 is used to determine a normalized score of the first processing result based on the first processing result determined by the first determining unit 90333 and the second processing result determined by the second determining unit 90334;

[0211] The fourth determining unit 90336 is used to determine the probability of the second candidate action being selected based on the normalized score determined by the third determining unit 90335.

[0212] In an alternative example, the formula used in determining the parameter gradient is:

[0213]

[0214] Δφ represents the parameter gradient, represents the probability of being selected, Q represents the algorithm used by the first network to calculate the predicted value of future reward impact, θ represents the algorithm parameters of the algorithm used by the first network to calculate the predicted value of future reward impact, s represents the second state, represents the second candidate action, α represents a specified constant, π represents the algorithm used by the second network to determine the candidate action, φ represents the algorithm parameter of the algorithm used by the second network to determine the candidate action, a - represents the second historical action, and f represents the algorithm used by the specified subnetwork of the second network.

[0215] Exemplary Electronic Devices

[0216] Below, reference Fig.11 The electronic device according to the embodiment of the present disclosure is described. The electronic device may be any one or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive the collected input signals from them.

[0217] Fig.11 A block diagram of an electronic device according to an embodiment of the present disclosure is illustrated.

[0218] like Fig.11 As shown, the electronic device 1100 includes one or more processors 1101 and a memory 1102 .

[0219] The processor 1101 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1100 to perform desired functions.

[0220] The memory 1102 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1101 may run the program instructions to implement the action output method or network training method for the reinforcement learning process of the various embodiments of the present disclosure described above. Various contents such as input signals, signal components, noise components, etc. may also be stored in the computer-readable storage medium.

[0221] In one example, the electronic device 1100 may further include: an input device 1103 and an output device 1104 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0222] For example, when the electronic device is the first device or the second device, the input device 1103 may be a microphone or a microphone array. When the electronic device is a stand-alone device, the input device 13 may be a communication network connector for receiving collected input signals from the first device and the second device.

[0223] In addition, the input device 1103 may also include, for example, a keyboard, a mouse, and the like.

[0224] The output device 1104 can output various information to the outside, including the determined distance information, direction information, etc. The output device 1104 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0225] Of course, to simplify, Fig.11 Only some of the components related to the present disclosure in the electronic device 1100 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application scenarios, the electronic device 1100 may also include any other appropriate components.

[0226] Exemplary computer program products and computer-readable storage media

[0227] In addition to the above-mentioned methods and devices, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the action output method or network training method for the reinforcement learning process according to various embodiments of the present disclosure described in the above "Exemplary Method" section of this specification.

[0228] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0229] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, causes the processor to execute the steps of the action output method or network training method for a reinforcement learning process according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0230] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0231] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0232] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0233] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0234] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0235] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0236] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0237] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. An action output method for a reinforcement learning process, comprising: Determine a first state of an environment in which an intelligent agent is located at a current time point; wherein the intelligent agent is an unmanned vehicle or a robot; Determine a first candidate action for the agent at a current time point based on the first state and a first historical action output by the agent to the environment at a previous time point; wherein the first candidate action is determined via a second network; Selecting a target action from the first candidate action and the first historical action; wherein the target action is an action selected from the first candidate action and the first historical action according to a comparison result of the relative quality of the first candidate action and the first historical action; Control the agent to output the target action at the current time point.

2. The method according to claim 1, wherein: The selecting a target action from the first candidate action and the first historical action includes: Based on the first state and the first candidate action, obtaining, via a first network, a predicted value of future reward impact corresponding to the first candidate action; Based on the first state and the first historical action, obtaining, via the first network, a predicted value of future reward impact corresponding to the first historical action; From the first candidate action and the first historical action, select an action corresponding to a larger predicted value of future reward impact as the target action.

3. The method according to claim 2, wherein: The method further comprises: Acquire multiple sets of historical data, each set of the acquired historical data records the state of the environment, the action output by the agent to the environment, the next state of the environment, and the reward generated by the state transition of the environment, and the multiple sets of historical data are adjacent to each other on the timeline, and the actions recorded in the multiple sets of historical data are the same; The first network is trained using the multiple groups of historical data as training data and a multi-step bootstrap method.

4. A network training method, comprising: Acquire historical data, wherein the acquired historical data records a second state of an environment in which an intelligent agent is located at a first time point and a second historical action output by the intelligent agent to the environment at a second time point, wherein the first time point is a time point next to the second time point; wherein the intelligent agent is an unmanned vehicle or a robot; Based on the second state and the second historical action, determining, via a second network, a second candidate action for the agent at the first time point; Determining a probability of selecting the second candidate action based on the second state, the second historical action, and the second candidate action; Determine a parameter gradient of the second network based on the second state, the second historical action, the second candidate action, and the selected probability; Based on the parameter gradient, the second network is trained; wherein the trained second network is applied to the action output method for the reinforcement learning process as described in any one of claims 1-3.

5. The method according to claim 4, wherein: The determining, based on the second state, the second historical action, and the second candidate action, a probability of the second candidate action being selected includes: Based on the second state and the second candidate action, obtaining, via the first network, a future reward impact prediction value corresponding to the second candidate action; Based on the second state and the second historical action, obtaining, via the first network, a future reward impact prediction value corresponding to the second historical action; Based on the future reward impact prediction values ​​corresponding to the second candidate action and the second historical action respectively, the probability of the second candidate action being selected is determined.

6. The method according to claim 5, wherein: The determining the probability of the second candidate action being selected based on the future reward impact prediction values ​​respectively corresponding to the second candidate action and the second historical action includes: Calculate a first ratio of a future reward impact prediction value corresponding to the second candidate action to a specified constant; Calculate a second ratio of the predicted value of future reward impact corresponding to the second historical action to a specified constant; Determining a first processing result based on a preset constant and the first ratio; Determining a second processing result based on the preset constant and the second ratio; Determining a normalized score of the first processing result based on the first processing result and the second processing result; Based on the normalized score, a probability of the second candidate action being selected is determined.

7. An action output device for strengthening a learning process, comprising: A first determination module is used to determine a first state of an environment in which an intelligent agent is located at a current time point; wherein the intelligent agent is an unmanned vehicle or a robot; A second determination module is configured to determine a first candidate action for the agent at a current time point based on the first state determined by the first determination module and a first historical action output by the agent to the environment at a previous time point; wherein the first candidate action is determined via a second network; A selection module, configured to select a target action from the first candidate action and the first historical action determined by the second determination module; wherein the target action is an action selected from the first candidate action and the first historical action according to a comparison result of the relative quality of the first candidate action and the first historical action; An output module is used to control the agent to output the target action selected by the selection module at a current time point.

8. A network training device, comprising: A second acquisition module is used to acquire historical data, wherein the acquired historical data records a second state of the environment in which the agent is located at a first time point and a second historical action output by the agent to the environment at a second time point, wherein the first time point is a time point next to the second time point; wherein the agent is an unmanned vehicle or a robot; a third determination module, configured to determine, via a second network, a second candidate action for the agent at the first time point based on the second state and the second historical action recorded in the historical data acquired by the second acquisition module; a fourth determination module, configured to determine a probability of the second candidate action being selected based on the second state and the second historical action recorded in the historical data acquired by the acquisition module, and the second candidate action determined by the third determination module; a fifth determination module, configured to determine a parameter gradient of the second network based on the second state and the second historical action recorded in the historical data acquired by the acquisition module, the second candidate action determined by the third determination module, and the probability of being selected determined by the fourth determination module; A second training module is used to train the second network based on the parameter gradient determined by the fifth determination module; wherein the trained second network is applied to the action output method for the reinforcement learning process as described in any one of claims 1 to 3.

9. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the action output method for a reinforcement learning process described in any one of claims 1 to 3, or to execute the network training method described in any one of claims 4 to 6.

10. An electronic device comprising: processor; a memory for storing executable instructions; The processor is used to read the executable instructions from the memory and execute the instructions to implement the action output method for the reinforcement learning process described in any one of claims 1-3, or to execute the network training method described in any one of claims 4-6.

Images