Intelligent agent reinforcement learning decision method and device, electronic equipment and storage medium thereof

Abstract

The application belongs to the technical field of intelligent decision-making, and particularly relates to an agent reinforcement learning decision-making method and device, an electronic device and a storage medium thereof. The method first establishes a constraint equation according to a plurality of constraint conditions of a problem to be decided, designs a plurality of simple examples that can be explicitly solved and solved, adds a suitable reward function, and obtains a series of sparse reward expert knowledge data. The expert knowledge data is placed in a DQN replay buffer module to obtain an improved replay buffer module Ex-Replay buffer having both expert knowledge data and environment learning data. After the agent and the environment interact, the probability distribution of the action is input into an action filtering module to obtain a legal action, and a confidence function is used to determine whether to select the filtered action. An adaptive term is added to the loss function to adjust the frequency of using the action filtering module. The data set collection process of the method is more efficient and convenient, and the confidence function can select the best strategy for the agent corresponding to the task.

Description

Technical Field

[0001] This application belongs to the field of intelligent decision-making technology, and specifically relates to an intelligent agent reinforcement learning decision-making method, device, electronic device and its storage medium. Background Technology

[0002] With the gradual development of intelligent technology, the demand for autonomous decision-making by intelligent agents is constantly increasing. Many fields are now applying reinforcement learning-based intelligent decision-making technologies, such as robotics, autonomous driving, and intelligent transportation. The problems requiring decision-making are becoming increasingly large and complex. While these factors provide a solid foundation for intelligent agents to make complex decisions, they also significantly increase the difficulty for agents to learn correct strategies. Currently, decisions made based on reinforcement learning typically need to be flexible enough to adapt to various working conditions while ensuring a high success rate in task execution. Decision errors can result in huge losses and even endanger the lives of those involved. Therefore, ensuring the diversity and robustness of decisions made by intelligent agents has significant social importance and can bring substantial economic and social benefits.

[0003] To ensure the security of intelligent decision-making while promoting the development of intelligent technologies, an increasing number of intelligent systems utilize reinforcement learning to assist agents in learning various strategies. This allows agents to learn initially in simple scenarios under the guidance of reinforcement learning, laying the groundwork for intelligent decision-making in more complex scenarios. However, reinforcement learning is a trial-and-error method, while real-world scenarios are highly complex and informationally incomplete, posing significant challenges to fully reinforcement learning-based intelligent decision-making in practical applications. For model-based reinforcement learning, the main problem stems from the model itself. Due to the complexity of intelligent decision-making scenarios, it is difficult to build a complete model that is isomorphic to the scenario. For model-free reinforcement learning, when the decision problem is highly complex, a large amount of data is required to ensure that the agent learns the corresponding strategy. The reasons for needing a large amount of data are mainly twofold: 1) massive amounts of data can better fit the model for solving the problem; 2) the data contains many invalid and erroneous data points, which affect the agent's learning efficiency and results. However, in practical intelligent decision-making tasks, acquiring large amounts of data incurs high time and sampling costs. Summary of the Invention

[0004] This application aims to at least address the problems in existing technologies. Based on the inventor's understanding of the following facts and problems, the design of reward functions in reinforcement learning for decision-making problems requires significant effort. Complex decision-making problems involve numerous constraints. While the reward upon completion of the task is relatively easy to determine, the rewards for intermediate processes are difficult to design in detail. Improper design can lead to the agent learning incorrect knowledge and failing to execute the task, or even irreversible consequences. For such sparse reward problems, a large amount of ineffective exploration occurs in the early stages of learning, and many incorrect strategies fail to yield feedback from the reward function, resulting in extremely low learning efficiency. Therefore, using an action filter based on expert knowledge to filter the generated actions and convert invalid actions into valid ones is an effective means of solving this problem. Currently, reinforcement learning with sparse rewards remains a challenging issue.

[0005] In view of this, this disclosure proposes a reinforcement learning decision-making method, device, electronic device and storage medium for intelligent agents, to realize a reinforcement learning decision-making method based on confidence action filtering. The method fills the Replay Buffer in the reinforcement learning algorithm with expert knowledge data, uses expert knowledge to filter the learned actions to obtain an effective action, and determines whether to adopt the expert action based on confidence, thus solving the important technical problem of inefficient learning in reinforcement learning based on sparse rewards.

[0006] According to a first aspect of this disclosure, a reinforcement learning decision-making method for intelligent agents is proposed, comprising:

[0007] Step 1: Construct an expert knowledge data set for agent reinforcement learning decision-making;

[0008] Step 2: Construct a replay buffer module for a reinforcement learning network;

[0009] Step 3: Train the reinforcement learning network to obtain the agent's initial policies and actions;

[0010] Step 4: Set up an action filtering module to filter the agent's initial strategy and use a confidence function to determine the agent's final action.

[0011] This disclosure presents a reinforcement learning decision-making method for sparse reward scenarios. Compared with existing methods, it exhibits excellent versatility and ease of operation for solving complex problems. This is mainly reflected in two aspects: First, when constructing the expert training dataset, this disclosure simplifies the scenario to a solution that can be obtained using linear programming or dynamic programming, and then adds a reward function to this solution to form the expert training dataset. Second, when selecting the agent's final action, adding a simple confidence method ensures that the agent selects the optimal action that meets the conditions.

[0012] Optionally, the construction of the expert knowledge data set for agent reinforcement learning decision-making includes:

[0013] (1) For scenarios where both the objective equation and the constraint equation are linearly represented, linear programming is used to obtain the action and state information of the agent. For scenarios that require iterative solutions, dynamic programming is used to obtain the action and state information of the agent.

[0014] (2) Using linear programming or dynamic programming methods, obtain the current state information s, action information a, next time step state information s′, and the flag d indicating whether the task has terminated for N different simple scenarios of the agent. Combine the above information into a knowledge data set.

[0015] Where i represents the relevant information for the agent's decision-making in the i-th scenario;

[0016] (3) is the knowledge data set Each data point in the dataset is assigned a reward function r. i ;

[0017]

[0018] (4) The reward function r i With the set By merging these elements, a set of expert knowledge data for agent reinforcement learning decision-making is obtained.

[0019] Optionally, the replay buffer module of the reinforcement learning network It consists of two parts, the first part being a fixed list. Used to store expert knowledge data, the first part, which occupies 30% of the length of the playback buffer module, is called the expert data playback buffer; the second part is a first-in-first-out queue. The queue is used to store samples collected by the agent from the environment and is called the environmental data playback buffer. The queue is empty when it is initialized and is updated according to the first-in-first-out algorithm when the queue is full.

[0020] Optionally, training the reinforcement learning network to obtain the agent's initial strategy and actions includes the following steps:

[0021] (1) In complex environmental problems, the agent collects samples from the environment multiple times to obtain at least one batch of environmental data with a quantity of B, and stores the environmental data into the first-in-first-out queue. middle;

[0022] (2) From the fixed list respectively First-In-First-Out (FIFO) queue Training samples are randomly selected from the dataset to form training data with a batch size of B, wherein 30% of the training data comes from... 70% from The training data is input into the reinforcement learning algorithm, which outputs the agent's initial policy. and the initial actions of the intelligent agent

[0023] in It concerns the probability distribution of the agent's initial actions.

[0024] Optionally, the action filtering module filters the agent's initial strategy and determines the agent's final action using a confidence function, including:

[0025] (1) Set up an action filtering module F af The action filtering module represents the constraints of the complex environmental problem as a system of inequality equations, and then applies the initial strategy of the agent. As input to the action filtering module, the output is the agent filtering action.

[0026] (2) Define a confidence function f c :

[0027] From the initial actions of the intelligent agent With agent filtering actions Select the agent's final action 'a' and use the final action to train the subsequent network;

[0028] (3) Set a loss function L, aim to minimize the loss function, and train the reinforcement learning network using the Adam algorithm until the loss function converges to obtain the trained reinforcement learning network.

[0029] Where β is a hyperparameter, F af For the aforementioned action filtering module, ||·|| 2 Let L be the norm 2, i be the index of the training data, B be the B data points in a batch, and L′ be the loss function of the initial reinforcement learning algorithm, defined as L′=(r+γ·max(Q(s′,a′))-Q(s,a)). 2 Where s, a, and r represent the agent's current state information, action information, and reward function, respectively; s′ and a′ represent the agent's state information and action information at the next moment, respectively; γ = 0.9; and Q(s,a) are variables related to the state and action information, generated by the neural network. 2 For square operators;

[0030] (4) Use the state information of each step in the task as the input of the trained reinforcement learning network to obtain the final action of the agent.

[0031] According to a second aspect of this disclosure, an agent reinforcement learning decision-making device is also proposed, comprising:

[0032] A module for constructing an expert knowledge data set is used to build an expert knowledge data set for agent reinforcement learning decision-making.

[0033] A replay buffer module is constructed to build a replay buffer module for a reinforcement learning network.

[0034] The agent's initial action generation module is used to train the reinforcement learning network to obtain the agent's initial strategy and actions;

[0035] The final action generation module for the agent is used to set up the action filtering module, filter the agent's initial strategy, and determine the agent's final action using a confidence function.

[0036] According to a third aspect of this disclosure, an electronic device is proposed, comprising:

[0037] Memory is used to store instructions that can be executed by the processor;

[0038] A processor configured to execute any agent reinforcement learning decision-making method of claims 1-5.

[0039] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being configured to cause the computer to execute:

[0040] Construct an expert knowledge data set for agent reinforcement learning decision-making;

[0041] Construct a replay buffer module for a reinforcement learning network.

[0042] The reinforcement learning network is trained to obtain the agent's initial policies and actions;

[0043] An action filtering module is set up to filter the agent's initial strategy and use a confidence function to determine the agent's final action.

[0044] According to a fourth invention of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being used to cause the computer to execute:

[0045] Construct an expert knowledge data set for agent reinforcement learning decision-making;

[0046] Construct a replay buffer module for a reinforcement learning network.

[0047] The reinforcement learning network is trained to obtain the agent's initial policies and actions;

[0048] An action filtering module is set up to filter the agent's initial strategy and use a confidence function to determine the agent's final action.

[0049] Existing methods for solving reinforcement learning problems with sparse rewards in complex scenarios commonly employ the construction of demonstration datasets, which are also a form of expert knowledge datasets. The main idea behind constructing such datasets is to collect trajectory information consisting of a series of complete state and action information required for the agent to complete the task. The expert knowledge dataset disclosed in this paper does not require several complete trajectory data sets. Instead, it simplifies the complex scenario and uses the results of linear programming and dynamic programming as expert knowledge data, making the dataset collection process more efficient and convenient. Furthermore, the optimality of the planning results is theoretically guaranteed, and the design of the reward function becomes very simple. This disclosure also introduces a confidence function to compare the initial action of the agent generated by the reinforcement learning neural network with the filtered action generated by the action filtering module to determine the agent's final action. Other algorithms consider the filtered action to be optimal, but reinforcement learning may learn an initial action superior to the filtered action after multiple explorations. Therefore, the confidence function can select the best strategy for the agent in the corresponding task.

[0050] Additional aspects and advantages of this disclosure will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this disclosure. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0052] Figure 1 This is a flowchart illustrating an agent reinforcement learning decision-making method according to an embodiment of the present disclosure.

[0053] Figure 2 This is a schematic diagram of the overall scheme of a reinforcement learning decision-making method according to an embodiment of the present disclosure.

[0054] Figure 3 This is a structural block diagram of an intelligent agent reinforcement learning decision-making device according to an embodiment of the present disclosure.

[0055] Figure 4 This is a defensive scenario diagram illustrated according to an embodiment of the present disclosure.

[0056] Figure 5 yes Figure 4 A top-down view of the defensive scene.

[0057] Figure 6 This is a training result diagram shown according to an embodiment of the present disclosure.

[0058] Figure 7 This is a test result diagram shown according to an embodiment of the present disclosure. Detailed Implementation

[0059] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0060] Figure 1 This is a schematic flowchart illustrating an agent reinforcement learning decision-making method according to an embodiment of the present disclosure. The agent reinforcement learning decision-making method of this embodiment can be applied to user devices, such as mobile phones, tablets, etc.

[0061] like Figure 1 and Figure 2 As shown, the agent reinforcement learning decision-making method may include the following steps:

[0062] In step 1, an expert knowledge data set is constructed for the agent's reinforcement learning decision-making.

[0063] In one specific embodiment, the construction of an expert knowledge data set for agent reinforcement learning decision-making utilizes action and state information in simple scenarios and a manually set reward function to construct the expert knowledge data set, including:

[0064] (1) For scenarios where both the objective equation and the constraint equation are linearly represented, linear programming is used to obtain the action and state information of the agent. For scenarios that require iterative solutions, dynamic programming is used to obtain the action and state information of the agent.

[0065] (2) Using linear programming or dynamic programming methods, obtain the current state information s, action information a, next time step state information s′, and the flag d indicating whether the task has terminated for N different simple scenarios of the agent. Combine the above information into a knowledge data set.

[0066] Where i represents the relevant information for the agent's decision-making in the i-th scenario;

[0067] (3) is the knowledge data set Each data point in the dataset is assigned a reward function r. i ;

[0068]

[0069] (4) The reward function r i With the set By merging these elements, a set of expert knowledge data for agent reinforcement learning decision-making is obtained.

[0070] In step 2, a replay buffer module for a reinforcement learning network is constructed. Part of this module stores the aforementioned expert knowledge data set, and the remaining part stores the data of samples collected by the agent from the environment during network training.

[0071] In one embodiment, the replay buffer module of the reinforcement learning network The replay buffer consists of two parts; the first part is a fixed list. Used to store expert knowledge data, the first part, which occupies 30% of the length of the playback buffer module, is called the expert data playback buffer; the second part is a first-in-first-out queue. The queue is used to store samples collected by the agent from the environment and is called the environmental data playback buffer. The queue is empty when it is initialized and is updated according to the first-in-first-out algorithm when the queue is full.

[0072] In step 3, the reinforcement learning network is trained to obtain the agent's initial policies and actions.

[0073] In one embodiment, training the reinforcement learning network to obtain the agent's initial policy and actions includes the following steps:

[0074] (1) In complex environmental problems, the agent collects samples from the environment multiple times to obtain at least one batch of environmental data with a quantity of B, and stores the environmental data into the first-in-first-out queue. middle;

[0075] (2) From the fixed list respectively First-In-First-Out (FIFO) queue Training samples are randomly selected from the dataset to form training data with a batch size of B, wherein 30% of the training data comes from... 70% from The training data is input into the reinforcement learning algorithm, which outputs the agent's initial policy. and the initial actions of the intelligent agent

[0076] in It concerns the probability distribution of the agent's initial actions. x0 = arg max(f(x)) means that the function f(x) reaches its maximum value when x = x0.

[0077] In step 4, an action filtering module is set up to filter the agent's initial strategy based on the constraints in the decision-making task required by the agent, thereby obtaining the agent's filtered actions. The final action of the agent is determined using a confidence function.

[0078] In one embodiment, the action filtering module filters the agent's initial strategy and determines the agent's final action using a confidence function, including:

[0079] (1) Set up an action filtering module F af The action filtering module represents the constraints of the complex environmental problem as a system of inequality equations, and then applies the initial strategy of the agent. As input to the action filtering module, the output is the agent filtering action.

[0080] (2) Define a confidence function f c :

[0081]

[0082] Where p(·) represents the probability, and α is a manually set significance level. To avoid violating system constraints Compare A better-performing set of agent actions;

[0083] From the initial actions of the intelligent agent With agent filtering actions Select the agent's final action 'a' and use the final action to train the subsequent network;

[0084] (3) Define a loss function L, With the goal of minimizing the loss function, the reinforcement learning network is trained using the Adam algorithm until the loss function converges, thus obtaining the trained reinforcement learning network.

[0085] Where β is a hyperparameter that adaptively changes during the training process of the learning algorithm, F af For the aforementioned action filtering module, ||·|| 2 Let L be the norm 2, i be the index of the training data, B be the B data points in a batch, and L′ be the loss function of the initial reinforcement learning algorithm, defined as L′=(r+γ·max(Q(s′,a′))-Q(s,a)). 2Where s, a, and r represent the agent's current state information, action information, and reward function, respectively; s′ and a′ represent the agent's state information and action information at the next moment, respectively; γ = 0.9; and Q(s, a) are variables related to the state and action information, generated by the neural network. 2 For square operators;

[0086] (4) Use the state information of each step in the task as the input of the trained reinforcement learning network to obtain the final action of the agent.

[0087] This disclosure presents a reinforcement learning decision-making method for sparse reward scenarios. Compared with existing methods, it exhibits excellent versatility and ease of operation for solving complex problems. This is mainly reflected in two aspects: First, when constructing the expert training dataset, this disclosure simplifies the scenario to a solution that can be obtained using linear programming or dynamic programming, and then adds a reward function to this solution to form the expert training dataset. Second, when selecting the agent's final action, adding a simple confidence method ensures that the agent selects the optimal action that meets the conditions.

[0088] Corresponding to the agent reinforcement learning decision-making method of this disclosure, this disclosure also proposes an agent reinforcement learning decision-making device.

[0089] Figure 3 The diagram shown is a schematic representation of an agent reinforcement learning decision-making device according to an embodiment of this disclosure. Figure 3 As shown, the agent reinforcement learning decision-making device includes:

[0090] A module for constructing an expert knowledge data set is used to build an expert knowledge data set for agent reinforcement learning decision-making.

[0091] A replay buffer module is constructed to build a replay buffer module for a reinforcement learning network.

[0092] The agent's initial action generation module is used to train the reinforcement learning network to obtain the agent's initial strategy and actions;

[0093] The final action generation module of the agent is used to set the action filtering module. Based on the constraints in the decision task required by the agent, the initial strategy of the agent is filtered to obtain the agent's filtered actions. The final action of the agent is determined by the confidence function.

[0094] Embodiments of this disclosure also propose an electronic device, including:

[0095] Memory is used to store instructions that can be executed by the processor;

[0096] Processor, the processor being configured to execute:

[0097] Construct an expert knowledge data set for agent reinforcement learning decision-making;

[0098] Construct a replay buffer module for a reinforcement learning network.

[0099] The reinforcement learning network is trained to obtain the agent's initial policies and actions;

[0100] An action filtering module is set up to filter the agent's initial strategy and use a confidence function to determine the agent's final action.

[0101] Embodiments of this disclosure also provide a computer-readable storage medium having a computer program stored thereon for causing the computer to execute:

[0102] Construct an expert knowledge data set for agent reinforcement learning decision-making;

[0103] Construct a replay buffer module for a reinforcement learning network.

[0104] The reinforcement learning network is trained to obtain the agent's initial policies and actions;

[0105] An action filtering module is set up to filter the agent's initial strategy and use a confidence function to determine the agent's final action.

[0106] Existing methods for solving reinforcement learning problems with sparse rewards in complex scenarios commonly employ the construction of demonstration datasets, which are also a form of expert knowledge datasets. The main idea behind constructing such datasets is to collect trajectory information consisting of a series of complete state and action information required for the agent to complete the task. The expert knowledge dataset disclosed in this paper does not require several complete trajectory data sets. Instead, it simplifies the complex scenario and uses the results of linear programming and dynamic programming as expert knowledge data, making the dataset collection process more efficient and convenient. Furthermore, the optimality of the planning results is theoretically guaranteed, and the design of the reward function becomes very simple. This disclosure also introduces a confidence function to compare the initial action of the agent generated by the reinforcement learning neural network with the filtered action generated by the action filtering module to determine the agent's final action. Other algorithms consider the filtered action to be optimal, but reinforcement learning may learn an initial action superior to the filtered action after multiple explorations. Therefore, the confidence function can select the best strategy for the agent in the corresponding task.

[0107] It should be noted that, in the embodiments of this disclosure, the processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. The memory can be used to store the computer program and / or modules. The processor implements various functions of the automotive parts image dataset creation device by running or executing the computer program and / or modules stored in the memory, and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.), etc. Furthermore, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD), flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices. Modules / units of the wind power system's stable operating domain construction device, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided in this disclosure, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0108] The present disclosure will now be described in detail with reference to the accompanying drawings and examples.

[0109] This disclosure validates an expert knowledge-driven decision-making method for sparse reward problems in deep reinforcement learning by using a game of defending against aircraft attacks as an example.

[0110] A flowchart of an expert knowledge-driven decision-making process for the sparse reward problem in deep reinforcement learning is shown below. Figure 1 As shown, it mainly includes:

[0111] (1) Construct an expert knowledge dataset to simplify the agent defense problem in complex scenarios where aircraft can fly randomly, transforming it into a linear programming problem for straight-line flight or a dynamic programming problem for segmented flight. The knowledge dataset can be obtained using a planning algorithm. And based on human experience, a reward function r is assigned to each piece of data in the knowledge dataset. i Then the knowledge dataset becomes an expert knowledge dataset.

[0112] (2) Obtain the improved replay buffer module, put all the data of the expert knowledge dataset into the expert data replay buffer, and put the sample data collected by the agent from the environment into the environment data replay buffer.

[0113] (3) Obtain the agent's initial actions. Randomly select 128 samples from the replay buffer module as training data, of which 30% is expert knowledge data and 70% is environmental data. Use the above samples as input to the DDQN network to obtain the agent's initial strategy and initial actions;

[0114] (4) Obtain the agent's final action and construct an action filtering module based on constraints such as the weapon ammunition quantity and firing range of the defense system. Use the agent's initial strategy as input to the action filtering module to obtain the agent's filtered actions. During the learning process, continuously construct a set of agent actions that perform better than the agent's filtered actions without violating system constraints, and use a confidence function to determine the agent's final action.

[0115] Obtaining the status of the defending target includes: the coordinates of the defended target;

[0116] Obtaining the status of defensive weapons includes: the coordinates of the three weapons, the remaining ammunition in each weapon, and the weapon's firing range. The relationship between the defensive target and the weapons is as follows: Figure 4 , Figure 5 As shown.

[0117] The aircraft status is obtained by including the current position and heading of all aircraft, and whether each aircraft is flying in a straight line or along a segmented fixed curve.

[0118] The weapon launch timing is calculated based on linear programming or dynamic programming when multiple aircraft are in different initial positions and flying straight. This result, along with a pre-designed reward function, is stored as expert knowledge data in the Ex-replay-buffer. Then, the actions generated by the agent's interaction with the environment are filtered by a filtering function generated by the expert system. The weapon's actions after interacting with the environment are also stored in the Ex-replay-buffer. During training, random sampling is performed proportionally from the Ex-replay-buffer to obtain batch training data. The training data is then input into the reinforcement learning neural network to obtain the weapon's initial actions.

[0119] An action filtering module is constructed based on constraints such as weapon range. Preliminary actions are input into the action filtering module to obtain filtered actions, and the final action is determined using a confidence function.

[0120] When 12 aircraft attack in any manner, and each defensive weapon has 4 rounds, the training results are as follows: Figure 6 As shown, the system converged after only 30+ cycles. Furthermore, the number of valid actions remained consistently 1. The test data is shown in Figure 7, demonstrating a test accuracy of 97.3%.

Claims

1. A reinforcement learning decision-making method for intelligent agents, characterized in that, include: Construct an expert knowledge data set for agent reinforcement learning decision-making; Construct a replay buffer module for a reinforcement learning network; The reinforcement learning network is trained to obtain the agent's initial policies and actions; An action filtering module is set up to filter the agent's initial strategy and use a confidence function to determine the agent's final action. The construction of the expert knowledge data set for agent reinforcement learning decision-making utilizes action and state information in simple scenarios and a manually set reward function to construct the expert knowledge data set, including: (1) For scenarios where both the objective equation and the constraint equation are linearly represented, linear programming is used to obtain the action and state information of the agent. For scenarios that require iterative solutions, dynamic programming is used to obtain the action and state information of the agent. (2) Using linear programming or dynamic programming methods, obtain the current state information s, action information a, next time step state information s′, and the flag d indicating whether the task has terminated for N different simple scenarios of the agent. Combine the above information into a knowledge data set. ; Where i represents the relevant information for the agent's decision-making in the i-th scenario; (3) is the knowledge data set A reward function is set for each piece of data. ; (4) The reward function mentioned above With the set By merging these elements, a set of expert knowledge data for agent reinforcement learning decision-making is obtained. ; The replay buffer module of the reinforcement learning network It consists of two parts, the first part being a fixed list. It is used to store expert knowledge data. The first part, which occupies 30% of the length of the playback buffer module, is called the expert data playback buffer; the second part is a first-in-first-out queue. This is used to store samples collected by the agent from the environment, and is called the environmental data playback buffer.

2. The agent reinforcement learning decision-making method according to claim 1, characterized in that, The process of training the reinforcement learning network to obtain the agent's initial policies and actions includes the following steps: (1) In complex environmental problems, the agent collects samples from the environment multiple times to obtain at least one batch of environmental data with a quantity of B, and stores the environmental data into the first-in-first-out queue. middle; (2) From the fixed list respectively First-In-First-Out (FIFO) queue Training samples are randomly selected from the dataset to form training data with a batch size of B, wherein 30% of the training data comes from... 70% from The training data is input into the reinforcement learning algorithm, and the reinforcement learning algorithm outputs the agent's initial policy. and the initial actions of the intelligent agent ; in It concerns the probability distribution of the agent's initial actions. .

3. The agent reinforcement learning decision-making method according to claim 2, characterized in that, The action filtering module filters the agent's initial strategy and determines the agent's final action using a confidence function, including: (1) Set up an action filtering module F af The action filtering module represents the constraints of the complex environmental problem as a system of inequality equations, and then applies the initial strategy of the agent. As input to the action filtering module, the output is the agent filtering action. ; (2) Define a confidence function f c : From the initial actions of the intelligent agent With agent filtering actions Select the agent's final action 'a' and use the final action to train the subsequent network; (3) Set a loss function L, aim to minimize the loss function, and train the reinforcement learning network using the Adam algorithm until the loss function converges to obtain the trained reinforcement learning network. (4) Use the state information of each step in the task as the input of the trained reinforcement learning network to obtain the final action of the agent.

4. An agent-based reinforcement learning decision-making device, suitable for control using the agent-based reinforcement learning decision-making method according to any one of claims 1-3, characterized in that, include: A module for constructing an expert knowledge data set is used to build an expert knowledge data set for agent reinforcement learning decision-making. Construct a replay buffer module to build a replay buffer module for a reinforcement learning network; The agent's initial action generation module is used to train the reinforcement learning network to obtain the agent's initial strategy and actions; The final action generation module for the intelligent agent is used to set up the action filtering module, filter the initial strategy of the intelligent agent, and determine the final action of the intelligent agent using a confidence function; The construction of the expert knowledge data set for agent reinforcement learning decision-making utilizes action and state information in simple scenarios and a manually set reward function to construct the expert knowledge data set, including: (1) For scenarios where both the objective equation and the constraint equation are linearly represented, linear programming is used to obtain the action and state information of the agent. For scenarios that require iterative solutions, dynamic programming is used to obtain the action and state information of the agent. (2) Using linear programming or dynamic programming methods, obtain the current state information s, action information a, next time step state information s′, and the flag d indicating whether the task has terminated for N different simple scenarios of the agent. Combine the above information into a knowledge data set. ; Where i represents the relevant information for the agent's decision-making in the i-th scenario; (3) is the knowledge data set A reward function is set for each piece of data. ; (4) The reward function mentioned above With the set By merging these elements, a set of expert knowledge data for agent reinforcement learning decision-making is obtained. ; The replay buffer module of the reinforcement learning network It consists of two parts, the first part being a fixed list. It is used to store expert knowledge data. The first part, which occupies 30% of the length of the playback buffer module, is called the expert data playback buffer; the second part is a first-in-first-out queue. This is used to store samples collected by the agent from the environment, and is called the environmental data playback buffer.

5. An electronic device, characterized in that, include: Memory is used to store instructions that can be executed by the processor; A processor configured to execute the agent reinforcement learning decision-making method according to any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program for causing the computer to execute the agent reinforcement learning decision-making method according to any one of claims 1-3.

Images