Reinforcement learning based agent decision neural network model design method

Abstract

The present application relates to the technical field of multi-agent and swarm intelligence, and particularly relates to a kind of agent decision neural network model design method based on reinforcement learning. Including: from input to output successively for Extractor module, Encoder module and Header layer module;Extractor module is used to extract the original observation information obtained by agent from environment, adopts the structure of multilayer perceptron, convolutional neural network, the structure of multilayer perceptron and long short memory neural network joint or the structure of convolutional neural network and long short memory neural network joint;Encoder module is used to encode the observation information extracted, adopts the encoder structure in Transformer;Header layer module is used as output layer, for making action decision according to coded information. Advantage lies in: the work of defining neural network structure is independent of reinforcement learning algorithm;It can be reused, and algorithm design process is simplified.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-agent and swarm intelligence, and particularly relates to a method for designing an agent decision-making neural network model based on reinforcement learning. BACKGROUND

[0002] The main tasks of intelligent decision-making include target optimization problems in static environments and action control problems in dynamic environments, and the output of intelligent decision-making can directly affect the environment itself. The present application mainly aims at the action control problem in the dynamic environment, sequentially makes one decision after another, and continuously sees new observations until the task ends. Deep reinforcement learning (DRL) is a product of the combination of deep learning and reinforcement learning, which integrates the powerful understanding ability of deep learning in visual perception problems and the decision-making ability of reinforcement learning, and realizes end-to-end learning. The emergence of deep reinforcement learning makes the reinforcement learning technology truly practical and solves complex problems in real-world scenarios. Since the appearance of DQN (deep Q network) in 2013, a large number of algorithms and papers on solving practical application problems have appeared in the field of deep reinforcement learning.

[0003] Multi-agent system is a kind of distributed computing technology, which can be used to solve problems in various fields, including robot systems, distributed decision-making, traffic control and business management, etc. Multi-agent reinforcement learning is an important branch in the field of multi-agent system research, which applies reinforcement learning technology, game theory, etc. to multi-agent system, so that multiple agents can complete more complex tasks through interaction and decision-making in higher-dimensional and dynamic real scenes.

[0004] The prior art generally designs an agent model based on a task scenario, defines different network structures (Actor and Critic networks) for different state observations and action spaces, and the neural network model has strong customization and is difficult to reuse, and has poor universality. In terms of multi-source information processing capability, the agent model designed based on the task scenario cannot well adapt to the changes of observation information types (observation images, observation vectors, time series information, etc.) and information dimensions. In terms of adaptability to different types of action spaces, the action space types include continuous action space, discrete action space, and action with causal relationship, etc. The agent model designed based on the task scenario has insufficient universality in action space types, and is difficult to reuse and integrate. SUMMARY

[0005] The present application provides a method for designing an agent decision-making neural network model based on reinforcement learning to solve the above problems.

[0006] The application aims to provide a reinforcement learning-based agent decision-making neural network model design method, comprising:

[0007] An Extractor module, an Encoder module and a Header layer module are designed, and the modules are sequentially arranged from input to output as the Extractor module, the Encoder module and the Header layer module.

[0008] The Extractor module is used to extract original observation information obtained by the agent from the environment; the Extractor module adopts a structure of a multilayer perceptron, a convolutional neural network, a structure of a combination of a multilayer perceptron and a long short-term memory neural network or a structure of a combination of a convolutional neural network and a long short-term memory neural network.

[0009] The Encoder module is used to encode the extracted observation information; the Encoder module adopts an encoder structure in a Transformer.

[0010] The Header layer module is used to make action decisions according to the encoded information.

[0011] Preferably, the original observation information is an observation vector, an observation image or time series information; if the original observation information of the agent is an observation vector, the Extractor module adopts a structure of a multilayer perceptron to extract information of the observation vector; if the original observation information of the agent is an observation image, the Extractor module extracts information in the observation image by using a convolutional neural network; if the original observation information of the agent is time series information, the Extractor module increases a long short-term memory neural network on the basis of a multilayer perceptron or a convolutional neural network to extract the time series information; if the agent is multiple, the Extractor module also needs to increase an attention mechanism to establish relative relationship information between the agents.

[0012] Preferably, the structure of the multilayer perceptron comprises the following from input to output:

[0013] (1) an input layer: a fully connected layer is used as the input layer, and the number of nodes is the length of the observation vector;

[0014] (2) a hidden layer: including two hidden layers and an LN normalization layer; the hidden layer is a fully connected layer with 128 nodes, and a Relu function is used as an activation function;

[0015] (3) an output layer: a fully connected layer plus a softmax function is used as the output layer, and the number of nodes is 128.

[0016] Preferably, the structure of the long short-term memory neural network is sequentially connected from input to output as a single time information extraction unit, an LSTM unit, and a gate circuit.

[0017] The network structure of the single time information extraction unit depends on the type of the original observation information. If the original observation information is an observation vector, the structure of a multi-layer perceptron is adopted. If the original observation information is an observation image, a convolutional neural network is adopted.

[0018] The output of the single time information extraction unit is used as the input of the LSTM unit, and the LSTM unit is linked to the gate circuit in residual connection. The gate circuit is used to balance the coupling relationship between the time sequence information and the current information, and the output result length is 128.

[0019] Preferably, the network structure of the attention mechanism from input to output is sequentially connected as a single agent information extraction unit, a multi-head attention unit, and a feedforward neural network unit, specifically including the following steps:

[0020] (1) After all agents complete the single agent information extraction, all observation information is combined into a sequence as the input of the multi-head attention unit, and the relative relationship information extraction is started.

[0021] (2) The multi-head attention unit includes a multi-head attention module, a superposition module, and a normalization module. The input of the multi-head attention module needs to be connected in residual connection to the superposition module, and then superimposed with the output of the multi-head attention module, and then input to the normalization module.

[0022] (3) The feedforward neural network unit includes a feedforward neural network module, a superposition module, and a normalization module. The input of the feedforward neural network module needs to be connected in residual connection to the superposition module, and then superimposed with the output of the feedforward neural network module, and then input to the normalization module.

[0023] (4) The final output node number is 128 (observation information length).

[0024] The attention mechanism follows the following formula:

[0025]

[0026] In the formula, Q represents a query vector, K represents a searched vector, V represents a content vector, d k represents the dimension size of Key, and T is the transpose symbol of the matrix, that is, the dot product operation of Q and K is implemented.

[0027] Preferably, the encoder structure in the Transformer is sequentially connected from input to output as an input embedding layer module, a multi-head attention unit, and a feedforward neural network unit, specifically as follows:

[0028] (1) the input embedding layer module is used for encoding observation information, and is a two-layer fully connected layer with 64 nodes;

[0029] (2) the multi-head attention unit includes a multi-head attention module, a superposition module and a normalization module; the input of the multi-head attention module needs to be connected in residual to the superposition module, and then is superposed with the output of the multi-head attention module, and then is input into the normalization module;

[0030] (3) the feedforward neural network unit includes a feedforward neural network module, a superposition module and a normalization module; the input of the feedforward neural network module needs to be connected in residual to the superposition module, and then is superposed with the output of the feedforward neural network module, and then is input into the normalization module;

[0031] (4) finally, encoded information is output.

[0032] Preferably, the Header layer module includes two fully connected layers and a softmax layer; the fully connected layers are used to connect the encoded information given by the Encoder module, and the softmax layer is used to output action decisions.

[0033] Preferably, the spatial type of the action includes a continuous action space, a discrete action space and an action with a causal relationship;

[0034] If the decision of a discrete action needs to be output, the number of output nodes is the number of discrete actions;

[0035] If the decision of a continuous action needs to be output, it is assumed that the decision distribution of the continuous action satisfies a Gaussian distribution, and the mean value and variance parameters are output;

[0036] If there is an action with a causal relationship with the action after the action, the decision network of the subsequent action needs to be connected.

[0037] Preferably, the decision network of the subsequent action is also designed by using the general neural network model design method of the decision intelligent agent based on reinforcement learning.

[0038] Preferably, the Gaussian distribution is a continuous probability distribution, and the distribution of the Gaussian distribution is determined by the mean value and variance, and the probability density function of the Gaussian distribution is:

[0039] ;

[0040] In the formula, μ represents the mean value, and σ2 represents the variance.

[0041] Compared with the prior art, the application can achieve the following beneficial effects:

[0042] ​The work of defining the neural network structure is independent of the reinforcement learning algorithm design, facilitating algorithm integration in a large simulation system; the defined neural network model can be reused by multiple algorithms, simplifying the algorithm design process. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 A general neural network model architecture diagram of a decision-making agent based on reinforcement learning is provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0044] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the following description, the same modules are denoted by the same reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated.

[0045] In order to make the objectives, technical solutions and advantages of the present application clearer, further detailed descriptions will be made to the present application in combination with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not constitute a limitation on the present application.

[0046] Referring to Figure 1 , the present application provides a general neural network model design method of a decision-making agent based on reinforcement learning, comprising:

[0047] An Extractor module is designed to extract original observation information obtained by the agent from the environment; the original observation information is an observation image, an observation vector or time series information;

[0048] If the original observation information of the agent is an observation vector, the Extractor module uses a structure of a multi-layer perceptron (MLP) to extract information from the observation vector, and the network structure from the input to the output is as follows:

[0049] (1) Input layer: a fully connected layer is used as the input layer, and the number of nodes is the length of the observation vector;

[0050] (2) Hidden layer: including two hidden layers and an LN normalization layer; the hidden layer is a fully connected layer with 128 nodes, and uses a Relu function as an activation function;

[0051] (3) Output layer: a fully connected layer plus a softmax function is used as the output layer, and the number of nodes is 128 (the length of the observation information);

[0052] If the original observation information of the agent is an observation image, the Extractor module extracts information in the observation image using a convolutional neural network (CNN). The network structure from input to output uses the public ResNet-101 model, and the last fully connected layer has 128 output nodes (the length of the observation information).

[0053] If the original observation information of the agent is a time sequence, the Extractor module needs to consider the time sequence information. Therefore, the Extractor module needs to add a long short-term memory network (LSTM) to extract the time sequence information based on the MLP or CNN. The network structure from input to output is as follows:

[0054] (1) The single time information extraction unit, the LSTM unit, and the gate circuit are connected in sequence from input to output.

[0055] (2) The network structure of the single time information extraction unit depends on the type of the original observation information. If it is an observation vector, use MPL, if it is an observation image, use CNN. The specific network structure is described above.

[0056] (3) The input of the LSTM unit (i.e., the output of the single time information extraction unit) is linked to the gate circuit after it.

[0057] (4) The gate circuit is used to balance the coupling relationship between the time sequence information and the current information, so that the entire network can handle both memory and reaction tasks.

[0058] (5) The length of the output result of the network structure is 128 (the length of the observation information).

[0059] If there are multiple agents, the Extractor module also needs to add an attention mechanism to establish the relative relationship information between the agents. Specifically, the network structure from input to output is a single agent information extraction unit, a multi-head attention unit, and a feedforward neural network unit connected in sequence. The process is as follows:

[0060] (1) After all agents complete the single agent information extraction, all observation information is combined into a sequence as the input of the multi-head attention unit to start the relative relationship information extraction.

[0061] (2) The multi-head attention unit includes a multi-head attention module, a stacking module, and a normalization module. The input of the multi-head attention module needs to be connected to the stacking module in residual connection, and then stacked with the output of the multi-head attention module, and then input to the normalization module.

[0062] (3) The feedforward neural network unit comprises a feedforward neural network module, a superposition module and a normalization module; the input of the feedforward neural network module needs to be connected in residual connection to the superposition module, and then superposed with the output of the feedforward neural network module, and then input to the normalization module;

[0063] (4) The final output node number is 128 (observation information length);

[0064] Among them, the attention mechanism follows the following formula:

[0065]

[0066] In the formula, Q represents the query vector Query, K represents the searched vector Key, V represents the content vector Value, d k represents the dimension size of Key, T is the transpose symbol of the matrix, that is, the dot product operation of Q and K is realized; the attention mechanism network will explore the internal relationship according to the Q, K, V parameters, and find out the relationship that should be paid attention to.

[0067] An Encoder module is designed to encode the extracted observation information, explore the internal relationship of the observation information, and make better decisions; the Encoder module adopts the encoder structure in the Transformer, and from input to output, the Input Embedding module, the Multi-Head Attention unit and the Feed Forward unit are connected in turn, and the process is as follows:

[0068] (1) First, the Input Embedding module is used to encode the observation information, and the structure is two fully connected layers, and the node number is 64;

[0069] (2) The Multi-Head Attention unit comprises a Multi-Head Attention module, a superposition module and a normalization module; the input of the Multi-Head Attention module needs to be connected in residual connection to the superposition module, and then superposed with the output of the Multi-Head Attention module, and then input to the normalization module;

[0070] (3) The feedforward neural network unit comprises a feedforward neural network module, a superposition module and a normalization module; the input of the feedforward neural network module needs to be connected in residual connection to the superposition module, and then superposed with the output of the feedforward neural network module, and then input to the normalization module;

[0071] (4) The final output encoding information.

[0072] The Header layer module is designed as an output layer for making action decisions according to the encoding information; specifically, the Header layer module overall structure includes two full connection layers and a softmax layer; the full connection layer is used to link the encoding information given by the Encoder module, and the softmax layer is used to output the action decision; the action space types include continuous action space, discrete action space, and action with causal relationship;

[0073] (1) If the decision of discrete action needs to be output, the number of output nodes is the number of discrete actions;

[0074] (2) If the decision of continuous action needs to be output, it is assumed that the decision distribution of continuous action satisfies Gaussian distribution, and the mean and variance parameters are output;

[0075] The Gaussian distribution is a continuous probability distribution, and its distribution is determined by the mean and variance, and its probability density function is:

[0076] ;

[0077] In the formula, represents the mean, represents the variance;

[0078] (3) If there is an action after the action that has a causal relationship with the action, the decision network of the subsequent action needs to be linked; the decision network of the subsequent action is designed by using the above general neural network model design method of the decision agent based on reinforcement learning.

[0079] The design of the Header layer module mainly improves the adaptability of the neural network model to different types of action spaces (action space types include continuous action space, discrete action space, and action with causal relationship), solves the problem of insufficient generality of action space types, and is difficult to reuse and integrate.

[0080] Embodiment 1

[0081] Taking the multi-agent reinforcement learning decision network controlling multi-UAV cooperative electronic countermeasure combat as an example, an agent decision neural network model is designed; specifically including:

[0082] The Extractor module is designed to extract the original observation information obtained by the agent from the environment; the original observation information is observation image, observation vector or time series information.

[0083] 1. If the original observation information of the agent is an observation vector, for example, the observation information is an observation vector formed by the information such as the position of the UAV, the position of the combat target, and the distance between the UAV and the friendly forces, then the structure of the multi-layer perception (MLP) is used to extract the information of the vector, and the network structure is:

[0084] (1) Input layer: the input layer is a fully connected layer, and the number of nodes is the length of the observation vector;

[0085] (2) Hidden layer: including two hidden layers and LN normalization layer; the hidden layer is a fully connected layer with 128 nodes, and the Relu function is used as the activation function;

[0086] (3) Output layer: the output layer is a fully connected layer plus a softmax function, and the number of nodes is 128 (the length of the observation information).

[0087] 2. If the original observation information of the agent is an observation image, such as a radar information map of the battlefield, the position of the unmanned aerial vehicle, the position of the combat target, and the relative position of the unmanned aerial vehicle and the friendly forces need to be extracted, and the public ResNet-101 convolutional neural network model is used for extraction, and the output node number of the last fully connected layer is 128 (the length of the observation information).

[0088] 3. If the original observation information of the agent is a time sequence, some time sequence information needs to be considered in decision-making; for example, the extraction of the successful experience of the unmanned aerial vehicle reconnaissance in the past time period, then a long short-term memory neural network (LSTM) needs to be added on the basis of MLP or CNN for time sequence information extraction, and the network structure from input to output is as follows:

[0089] (1) From input to output, the single time information extraction unit, the LSTM unit and the gate circuit are connected in turn;

[0090] (2) The network structure of the single time information extraction unit depends on the type of the original observation information. If it is an observation vector, MPL is used, and if it is an observation image, CNN is used. The specific network structure is referred to the above two cases;

[0091] (3) The input of the LSTM unit (i.e. the output of the single time information extraction unit) is linked to the gate circuit after it;

[0092] (4) The gate circuit is used to balance the coupling relationship between time sequence information and current information, so that the whole network can consider both memory type and reaction type tasks;

[0093] (5) The output result length of the network structure at the end is 128 (the length of the observation information);

[0094] 4、If the intelligent agent is multiple, such as the interference task needs to consider the cooperation between multiple unmanned aerial vehicles to complete, the relative position between the unmanned aerial vehicles needs to be considered when making decisions, etc., the Extractor module also needs to increase the attention mechanism to establish the relative relationship information between the intelligent agents; Specifically, the network structure from input to output is a single intelligent agent information extraction unit, a multi-head attention unit, and a feedforward neural network unit connected in turn, and the process is as follows:

[0095] (1) After all the intelligent agents complete the information extraction of the single intelligent agent, all the observation information is combined into a sequence as the input of the multi-head attention unit to start the information extraction of the relative relationship;

[0096] (2) The multi-head attention unit includes a multi-head attention module, a superposition module, and a normalization module; the input of the multi-head attention module needs to be connected in residual to the superposition module, and then the output of the multi-head attention module is superimposed and then input to the normalization module;

[0097] (3) The feedforward neural network unit includes a feedforward neural network module, a superposition module, and a normalization module; the input of the feedforward neural network module needs to be connected in residual to the superposition module, and then the output of the feedforward neural network module is superimposed and then input to the normalization module;

[0098] (4) The final output node number is 128 (observation information length).

[0099] The Encoder module is designed to encode the extracted observation information, explore the internal relationship of the observation information, and make better decisions; the Encoder module adopts the encoder structure in the Transformer, and is connected in turn from input to output as an input embedding layer (Input Embedding) module, a multi-head attention unit (Multi-Head Attention), and a feedforward neural network unit (Feed Forward), and the process is as follows:

[0100] (1) First, the input embedding layer module is used to encode the observation information, and the structure is two fully connected layers with node numbers of 64;

[0101] (2) The multi-head attention unit includes a multi-head attention module, a superposition module, and a normalization module; the input of the multi-head attention module needs to be connected in residual to the superposition module, and then the output of the multi-head attention module is superimposed and then input to the normalization module;

[0102] (3) The feedforward neural network unit includes a feedforward neural network module, a superposition module, and a normalization module; the input of the feedforward neural network module needs to be connected in residual to the superposition module, and then the output of the feedforward neural network module is superimposed and then input to the normalization module;

[0103] (4) Finally output the encoding information.

[0104] Design the Header layer module as an output layer, the overall structure includes two fully connected layers and a softmax layer; the fully connected layer is used to connect the encoding information given by the Encoder module, and the softmax layer is used to output the action decision; the action space type includes continuous action space, discrete action space, and action with causal relationship; the Header layer module focuses on making action decisions based on the fused encoding information, and if it is a mixed action space, there can be multiple Header layers;

[0105] (1) If the decision of discrete action needs to be output, for example, when an agent controls the flight speed of a UAV, the action space can be defined as three speed levels, i.e., a discrete action space, and the output node number is 3;

[0106] (2) If the decision of continuous action needs to be output, for example, when an agent controls the pointing direction of a reconnaissance antenna of a UAV, the action space is continuous variation within the antenna pointing range; assuming that the decision distribution of continuous action satisfies Gaussian distribution, the mean and variance parameters are output;

[0107] wherein, Gaussian distribution is a continuous probability distribution, whose distribution is determined by mean and variance, and its probability density function is:

[0108] ;

[0109] In the formula, represents the mean, represents the variance;

[0110] (3) If there is an action that has a causal relationship with the action after the action, for example, when an agent controls the on-off state of a UAV jamming load, only in the on state will the rotation angle of the jamming antenna be decided to point to the jamming target, so the decision network of the rotation angle of the jamming antenna can be connected after the decision network of the on-off state of the jamming load, and the decision network of the rotation angle of the jamming antenna is designed using the above general neural network model design method of the decision agent based on reinforcement learning.

[0111] Thus, the design of the agent decision network for the cooperative task of multiple UAVs is completed.

[0112] It should be understood that the various forms of flow shown above can be used to reorder, add or delete steps. For example, each step described in the present disclosure can be executed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions of the present disclosure can be achieved, which is not limited herein.

[0113] The above detailed description does not limit the scope of the application. Various modifications, combinations, sub-combinations and alternatives can be made to the detailed description. Any modification, equivalent replacement and improvement etc. made within the spirit and principle of the application shall be included in the scope of the application.

Claims

1. A method for designing an agent decision-making neural network model based on reinforcement learning, characterized in that, include: Design an Extractor module, an Encoder module, and a Header layer module, with the Extractor module, Encoder module, and Header layer module arranged sequentially from input to output; The Extractor module is used to extract raw observation information obtained by the agent from the environment; the Extractor module adopts the structure of multilayer perceptron, convolutional neural network, a structure combining multilayer perceptron and long short-term memory neural network, or a structure combining convolutional neural network and long short-term memory neural network. The original observation information can be an observation vector, an observation image, or temporal information. If the original observation information of the agent is an observation vector, the Extractor module uses a multilayer perceptron structure to extract information from the observation vector. If the original observation information of the agent is an observation image, the Extractor module uses a convolutional neural network to extract information from the observation image. If the original observation information of the agent is temporal information, the Extractor module adds a long short-term memory neural network to the multilayer perceptron or convolutional neural network to extract temporal information. If there are multiple agents, the Extractor module also needs to add an attention mechanism to establish the relative relationship information between the agents. The structure of the Long Short-Term Memory Neural Network consists of a single-time information extraction unit, an LSTM unit, and gate circuits connected sequentially from input to output. The network structure of the single-moment information extraction unit depends on the type of the original observation information. If the original observation information is an observation vector, a multilayer perceptron structure is used; if the original observation information is an observation image, a convolutional neural network is used. The output of the single-moment information extraction unit serves as the input of the LSTM unit, and the residual of the LSTM unit is linked to the gate circuit. The gate circuit is used to balance the coupling relationship between the time-series information and the current information, and the output result has a length of 128. The Encoder module is used to encode the extracted observation information. The Encoder module adopts the encoder structure in Transformer. The Header layer module, which serves as the output layer, is used to make action decisions based on the encoded information.

2. The method for designing an agent decision neural network model based on reinforcement learning according to claim 1, characterized in that: The structure of the multilayer perceptron, from input to output, includes: (1) Input layer: The fully connected layer is used as the input layer, and the number of nodes is the length of the observation vector; (2) Hidden layer: includes two hidden layers and LN-to-one layer; the hidden layer is a fully connected layer with 128 nodes and uses the ReLU function as the activation function; (3) Output layer: A fully connected layer with a softmax function is used as the output layer, with 128 nodes.

3. The method for designing an agent decision neural network model based on reinforcement learning according to claim 1, characterized in that: The attention mechanism's network structure from input to output consists of a single agent information extraction unit, a multi-head attention unit, and a feedforward neural network unit connected sequentially, specifically including the following steps: (1) After all agents have completed the information extraction of a single agent, all observation information is grouped into a sequence and used as the input of the multi-head attention unit to start the information extraction of relative relationships; (2) The multi-head attention unit includes a multi-head attention module, an overlay module, and a normalization module; the input of the multi-head attention module needs to be connected to the overlay module by residuals, then overlaid with the output of the multi-head attention module, and then input to the normalization module; (3) The feedforward neural network unit includes a feedforward neural network module, a superposition module and a normalization module; the input of the feedforward neural network module needs to be connected to the superposition module by residual, and then superimposed with the output of the feedforward neural network module, and then input to the normalization module; (4) The final number of output nodes is 128 (observation information length); The attention mechanism follows the formula below: ； In the formula, Q represents the query vector, K represents the searched vector, V represents the content vector, and d k The dimension of Key is represented by T, which is the transpose of the matrix, i.e., the dot product of Q and K is performed.

4. The method for designing an agent decision neural network model based on reinforcement learning according to claim 3, characterized in that: The encoder structure in the Transformer consists of an input embedding layer module, a multi-head attention unit, and a feedforward neural network unit connected sequentially from input to output, as detailed below: (1) The input embedding layer module is used to encode the observation information. It has a structure of two fully connected layers, each with 64 nodes. (2) The multi-head attention unit includes a multi-head attention module, an overlay module and a normalization module; the input of the multi-head attention module needs to be residually connected to the overlay module, then superimposed with the output of the multi-head attention module, and then input to the normalization module; (3) The feedforward neural network unit includes a feedforward neural network module, a superposition module and a normalization module; the input of the feedforward neural network module needs to be residually connected to the superposition module, then superimposed with the output of the feedforward neural network module, and then input to the normalization module; (4) Finally output the encoded information.

5. The method for designing an agent decision neural network model based on reinforcement learning according to claim 1, characterized in that: The Header layer module includes two fully connected layers and a softmax layer; the fully connected layers are used to connect the encoded information given by the Encoder module, and the softmax layer is used to output action decisions.

6. The method for designing an agent decision neural network model based on reinforcement learning according to claim 5, characterized in that: The spatial types of the actions include continuous action space, discrete action space, and actions with causal relationships; If the decision of discrete actions needs to be output, then the number of output nodes is the number of discrete actions; If the decision of continuous actions needs to be output, it is assumed that the decision distribution of continuous actions follows a Gaussian distribution, and the mean and variance parameters are output. If there is a causal relationship between a certain action and another action, then a decision network is needed to connect the subsequent actions.

7. The method for designing an agent decision neural network model based on reinforcement learning according to claim 6, characterized in that: The decision network for the subsequent actions is also designed using the general neural network model design method for decision agents based on reinforcement learning as described in claim 1.

8. The method for designing an agent decision neural network model based on reinforcement learning according to claim 7, characterized in that: The Gaussian distribution is a continuous probability distribution, and its distribution is determined by its mean and variance. Its probability density function is: ； In the formula, Represents the mean. Represents variance.

Images