An expert-assisted agent training method based on generative adversarial networks
By improving the training process of GAN and introducing GRU, the problem of poor results in traditional imitation learning methods in high-dimensional action space is solved, more efficient data expansion and strategy generalization are achieved, and the decision-making ability and robustness of the agent are improved. sex.
Patent Information
- Application Number
- CN202411084772.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-08
AI Technical Summary
Traditional imitation learning methods are not effective in high-dimensional action space and are difficult to deal with complexity, which leads to the generator that may fall into local optimal solutions, lack sample differences, and overfitting, which cannot fully reflect the diversity of samples.
By improving the training process of GAN, regularization technology and gated recurrent units (GRUs) are introduced, loss function design is improved, and a generative adversarial network model is built to learn and expand the expert experience distribution, enhancing the diversity and stability of the model improves the robustness and generalization ability of the strategy.
It significantly increases the amount and diversity of data for imitation learning, reduces the risk of overfitting, improves the initial level of agents and the robustness and generalization capabilities of strategies, and solves the problems of data sparseness and complex task decision-making.
Smart Images

Figure CN118607615B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of calculation, inference or counting, and in particular to an expert-assisted intelligent agent training method based on a generative adversarial network in the field of machine learning and deep reinforcement learning. Background Art
[0002] With the continuous development of artificial intelligence technology, machine learning has become an important means to solve complex problems. In the field of machine learning, imitation learning, as an important learning paradigm, aims to learn and imitate the behavior strategies of experts from expert demonstrations. Imitation learning has been widely used in many fields, such as autonomous driving, robot control, game AI, etc. However, traditional imitation learning methods face some challenges. First, these methods usually require a large amount of expert demonstration data for machine learning algorithms to extract effective behavior strategies from them. However, in practical applications, obtaining sufficient quantity and quality of demonstration data is often a difficult task. The scarcity of expert demonstration data limits the widespread application of imitation learning. Secondly, for high-dimensional and continuous action spaces, traditional imitation learning methods often do not work well. In high-dimensional action spaces, each action may contain multiple degrees of freedom, which makes the learning of strategies more complicated. Traditional imitation learning methods find it difficult to handle this complexity, resulting in poor learning results, or even failure to learn effective behavior strategies.
[0003] In order to overcome these challenges, researchers have begun to explore new imitation learning methods in recent years. Among them, the imitation learning training method based on generative adversarial network (GAN) has attracted much attention. GAN consists of two neural networks, the generator and the discriminator. The generator is responsible for generating data, while the discriminator is responsible for judging whether the generated data is real. In imitation learning, the generator can be trained to imitate the behavior strategy of the expert, while the discriminator is used to distinguish the generated actions from the actions demonstrated by the expert. By introducing GAN, imitation learning can learn and generate similar actions from the distribution of expert demonstration data. The generator can generate actions similar to the expert behavior by learning the potential representation of the demonstration data, while the discriminator provides feedback to the generator by judging the similarity between the generated actions and the expert demonstration, so that it can continuously optimize the generated actions. In addition, the adversarial training method of GAN also makes imitation learning more robust. Traditional imitation learning methods often simply copy the behavior of experts, while the generator of GAN will try to deceive the discriminator during the training process and generate actions that are similar but not exactly the same as the expert demonstration. This adversarial training method can increase the diversity of generated actions and improve the robustness of the generation strategy.
[0004] However, there are still some problems with the current GAN-based imitation learning method. For example, although the introduction of generative adversarial networks can increase the diversity of data and alleviate the covariate drift phenomenon to a certain extent, sometimes the generator tends to generate similar samples and lacks diversity. This lack of diversity comes from the characteristics of the training data and the limitations of the generative adversarial network structure itself. When dealing with complex data distributions, the generator may fall into a local optimal solution, resulting in a lack of differences between the generated samples and even overfitting. In this case, even if GAN can alleviate covariate drift to a certain extent, the true diversity of the samples still cannot be fully reflected. Summary of the invention
[0005] In view of the problems existing in the prior art, the present invention provides a GAN-based imitation learning training method, medium, device and application thereof. By improving the training process of GAN, introducing appropriate regularization technology and improving the loss function design, the model's learning ability for diversity is enhanced and the stability of the model is improved. By introducing GRU, the imitation learning model can better understand and replicate the behavior patterns of experts, thereby achieving more advanced decision-making in complex tasks, further improving the performance and application scope of imitation learning, and effectively improving the initial level of the intelligent agent.
[0006] The technical solution adopted by the present invention is an expert-assisted intelligent agent training method based on a generative adversarial network, the method comprising the following steps:
[0007] S1 collects expert experience and performs preprocessing;
[0008] S2 builds a generative adversarial network model to learn the distribution of expert experience and expand expert experience;
[0009] S3 introduces regularization technology to train the generative adversarial network, and uses the trained generative adversarial network to expand the expert experience preprocessed by S1;
[0010] S4 improves the imitation learning network to process the expanded expert experience;
[0011] S5 evaluates and optimizes the strategy.
[0012] Preferably, in S1, the expert experience is a state-action pair; and the preprocessing is to standardize the state data to a standard normal distribution.
[0013] Preferably, in S2, the generative adversarial network model includes a generator network and a discriminator network;
[0014] In the generator network, a Dropout layer is set after the first fully connected layer;
[0015] In the discriminator network, the first two fully connected layers are followed by a Dropout layer.
[0016] Preferably, in S3, the generative adversarial network model is iteratively trained using a back propagation algorithm and a gradient descent optimization algorithm, and the generator network and the discriminator network are respectively regularized during the training;
[0017] The collected expert experience is expanded with the generator network of the trained generative adversarial network model.
[0018] Preferably, the objective function of the generative adversarial network model is satisfy,
[0019]
[0020] in, is the prior probability distribution Noise samples, independent variables To obey the distribution The real data For the general The probability distribution of classification as true data, E x∼ p data ( x ) [ log D ( x )] To classify the real data as true, E z ∼ p z ( z ) [log (1 - D ( G ( z )))] For the discriminator In the generator The complement of the expected log-likelihood on the generated fake data.
[0021] Preferably, the regularization process includes Dropout regularization and L2 regularization;
[0022] The loss function of the generator satisfies ,
[0023] The loss function of the discriminator satisfies ,
[0024] in, L G = - E z ∼ p z ( z ) [log D ( G ( z ))] , L D = - E x ∼ p data ( x ) [log D ( x )] - E z ∼ p z ( z ) [log (1 - D ( G ( z )))] , and are the parameters of the generator and the discriminator respectively, and is the corresponding regularization coefficient.
[0025] Preferably, in S4, the imitation learning network is improved by a gated recurrent unit, the expanded expert state-action sequence is processed, and the data are learned by a behavioral cloning method, and a state-action mapping relationship is established by fitting the expert data; the gated recurrent unit is arranged before the policy network, the state is input, and the policy network outputs the action decision.
[0026] Preferably, the objective function of the behavior cloning method is:
[0027]
[0028] in, is a parameter The optimal value of Next Strategy The parameters minimize the expected loss; It means that at time step t, given state When the state sequence is The probability distribution of the selected action; is the action sequence in the entire sequence, T is the length of the time sequence, For expert data, is the loss function under supervised learning, To sample all possible state-action pairs under the expert data and calculate the average of these samples.
[0029] Preferably, the loss function is the cross entropy loss function, satisfying
[0030]
[0031] in, is the true distribution for action a at time step t, is the probability of the action predicted by the model;
[0032] Train the GRU network with the expanded expert state action sequence and adjust the model parameters To minimize the loss function .
[0033] Preferably, in S5, a deviation operation is performed between the action decision output by S4 and the action decision obtained by expert experience under the same input state, and the weight of the strategy network is updated based on the deviation value.
[0034] The present invention relates to an expert-assisted intelligent agent training method based on a generative adversarial network, which collects and preprocesses expert experience, constructs a generative adversarial network model for learning the distribution of expert experience and expanding the expert experience, introduces regularization technology, trains the generative adversarial network, and uses the trained generative adversarial network to expand the preprocessed expert experience; improves an imitation learning network to process the expanded expert experience; and evaluates and optimizes strategies.
[0035] The beneficial effects of the present invention are:
[0036] (1) Data enhancement and efficiency improvement: By augmenting expert experience with generative adversarial networks, the amount of data available for imitation learning can be significantly increased, which helps solve the data sparsity problem often encountered in practical applications, thereby improving the efficiency and accuracy of imitation learning.
[0037] (2) Reduce the risk of overfitting: By expanding expert experience, the risk of overfitting during model training can be reduced. Using data generated by generative adversarial networks can increase the number and diversity of training samples and reduce the occurrence of overfitting.
[0038] (3) Enhanced strategy generalization capability: The samples generated by the generative adversarial network may contain new situations or strategies that are not covered in the original expert experience data, so that the intelligent agent trained by model learning can better adapt and take effective actions when facing unknown or changing environments, effectively improving the robustness and generalization of the strategy. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 is a flow chart of the present invention;
[0040] Figure 2 It is the behavior cloning framework diagram of the present invention;
[0041] Figure 3 1 is a comparison chart of application effects in the embodiments of the present invention, wherein (a) is the application effect when only imitation learning is used, and (b) is the application effect when generative adversarial imitation learning with the introduction of regularization technology. DETAILED DESCRIPTION
[0042] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific implementation described herein is only used to explain the present invention and is not intended to limit the present invention.
[0043] The present invention relates to an expert-assisted intelligent agent training method based on a generative adversarial network. Aiming at the problem of scarcity of expert experience data in imitation learning, the distribution of expert experience is learned by GAN, and new data similar to the expert experience data is generated, thereby effectively expanding the expert experience and improving the training effect and model level of imitation learning.
[0044] like Figure 1 As shown, the method comprises the following steps:
[0045] S1 collects expert experience and performs preprocessing;
[0046] S2 builds a generative adversarial network model to learn the distribution of expert experience and expand expert experience;
[0047] S3 introduces regularization technology to train the generative adversarial network, and uses the trained generative adversarial network to expand the expert experience preprocessed by S1;
[0048] S4 improves the imitation learning network to process the expanded expert experience;
[0049] S5 evaluates and optimizes the strategy.
[0050] The method is described below with specific steps.
[0051] (1) Collect expert experience and perform preprocessing.
[0052] The expert experience is a state-action pair; the preprocessing is to standardize the state data to a standard normal distribution.
[0053] In the present invention, the collected expert experience includes but is not limited to the behavior trajectory, state transition, decision action, etc. of the expert agent or expert model when interacting with the environment. Here, the state-action pair when the expert agent or expert model interacts with the environment ;
[0054] After collecting expert experience, in order to eliminate the dimensionality impact of the data, the expert experience data is standardized and preprocessed. This patent adopts a data standardization method to standardize the data with different distribution ranges to conform to the standard normal distribution. The specific method is as follows:
[0055] For expert state-action pair sequences To transform,
[0056]
[0057] in, , ; The new sequence generated is dimensionless and The mean is 0 and the variance is 1; this allows the length of the data in each dimension to appear as a unit vector, that is, scaling the data with different size ranges into a unified dimension;
[0058] By standardizing the expert experience data, the consistency of the data in different dimensions is ensured, so that it has a unified scale standard, thereby improving the training efficiency and accuracy of the model and facilitating the subsequent training of the adversarial generative network.
[0059] (2) Construct a generative adversarial network model to learn the distribution of expert experience and expand expert experience.
[0060] The generative adversarial network model includes a generator network and a discriminator network;
[0061] In the generator network, a Dropout layer is set after the first fully connected layer;
[0062] In the discriminator network, the first two fully connected layers are followed by a Dropout layer.
[0063] The goal of the generator is to generate data that is similar to real expert experience. The discriminator receives data from the generator network or real training data as input and outputs a probability value indicating whether the data is real. Its goal is to accurately distinguish between the data generated by the generator and the real expert experience.
[0064] (3) Introducing regularization technology to train the generative adversarial network, and using the trained generative adversarial network to expand the preprocessed expert experience.
[0065] In the present invention, the generative adversarial network model is iteratively trained using a back-propagation algorithm and a gradient descent optimization algorithm, and the generator network and the discriminator network are respectively regularized during the training;
[0066] The collected expert experience is expanded with the generator network of the trained generative adversarial network model.
[0067] In the process of iterative training of the generative adversarial network with the back-propagation algorithm and the gradient descent optimization algorithm, the generator network G and the discriminator network D are regularized to improve the generalization ability of the model and prevent overfitting. The generator network of the trained generative adversarial network model uses the collected expert experience to generate new and realistic data by learning its inherent distribution characteristics. This process expands the original expert experience data set and uses the expanded expert experience as the training set. Through adversarial training, the parameters of the generator and the discriminator are continuously optimized, so that the generator can generate data that is closer and closer to the real expert experience, so that the performance of the generator network and the discriminator network is gradually improved.
[0068] The objective function of the generative adversarial network model satisfy,
[0069]
[0070] in, is the prior probability distribution A noise sample of Represents real data, following the distribution , in terms of convergence, the generated data distribution is consistent;
[0071] is a probability distribution, which means The probability of being classified as real data rather than generated data, that is, when the independent variable Represents real data, It means the probability of classifying the real data as true, when the independent variable When representing generated data (fake data), It represents the probability of classifying the generated (fake) data as true;
[0072] E x∼ p data ( x ) [ log D ( x )] represents the probability of classifying real data as true, It can reduce the impact of the dimension while ensuring positive correlation;
[0073] E z ∼ p z ( z ) [log (1 - D ( G ( z )))] For the discriminator In the generator The complement of the expected log-likelihood on the generated fake data, i.e., the ability of the discriminator to correctly classify fake samples, where The generator Zhongyou The generated data is put into the function , obey Distribution, that is, input distribution, reduces We need to increase the discriminator False Positive Generator The misclassification rate of the generated data, therefore, we want to increase the discriminator The ability to distinguish between Generators that are mistaken for real data Generated data;
[0074] During the training process, by continuously adjusting the discriminator parameters, so that it can more accurately distinguish these misjudged data, thereby reducing the number of misjudgments and improving the discriminator in future iterations accurate discrimination ability.
[0075] Regularization processing includes Dropout regularization and L2 regularization.
[0076] Dropout regularization refers to the introduction of Dropout layers in the appropriate layers of the generator and discriminator. Specifically, Dropout is applied after the fully connected layer to randomly discard the activation values of some neurons. This operation reduces the network's dependence on individual samples in the training data and enhances the robustness of the model. In the implementation process, the Dropout ratio is set to 0.5, which means that during the training process, each layer has a 50% probability of discarding its neurons.
[0077] L2 regularization is divided into two parts:
[0078] The goal of the discriminator is to maximize its ability to distinguish true and false data, and its loss function is,
[0079] L D = - E x ∼ p data ( x ) [log D ( x )] - E z ∼ p z ( z ) [log (1 - D ( G ( z )))]
[0080] Loss Function It consists of two parts. The first part is the loss of real data, which encourages the discriminator to judge the real data as true, that is, D(x) is close to 1. The second part is the loss of generated data, which encourages the discriminator to judge the generated data as false, that is, D(G(z)) is close to 0.
[0081] The goal of the generator is to minimize the probability that its generated data is correctly classified as fake by the discriminator, and its loss function is,
[0082] L G = - E z ∼ p z ( z ) [log D ( G ( z ))]
[0083] Loss Function Only the loss of generated data encourages the data generated by the generator to deceive the discriminator and make it judge the false data as true, that is, D(G(z)) is close to 1;
[0084] On the basis of the above basic loss function, add the regularization term L2 to control the size of the weight, and add corresponding terms to the loss functions of D and G.
[0085]
[0086] in, and are the parameters of the discriminator and the generator, respectively. and is the regularization coefficient, which controls the strength of the regularization term.
[0087] The goal of function optimization is the generator network Trying to capture the underlying data density and confuse the discriminator network ,and The optimization process is to achieve the real number moment Generating discriminability and distinguishability of data, and All of them can be fitted with neural networks; the collected expert experience is used to train the generative adversarial network. By combining Dropout and L2 regularization, the model's dependence on training data is not only reduced during the training process, but also the scale of the model parameters is limited, thereby improving the model's performance on unseen data and increasing the diversity of generated samples. After such iterative training, the generator network of the resulting generative adversarial network model can effectively expand the initial expert experience and generate a richer and more realistic sample set.
[0088] (4) Improve the imitation learning network and process the expanded expert experience.
[0089] The imitation learning network is improved by using a gated recurrent unit, the expanded expert state-action sequence is processed, and the behavior cloning method is used to learn these data. By fitting the expert data, a state-action mapping relationship is established; the gated recurrent unit is set before the policy network, the state is input, and the policy network outputs the action decision.
[0090] After obtaining the expanded expert experience data, the behavioral cloning method is used to learn these data. Behavior cloning is a supervised learning method that directly establishes the state-action mapping relationship by fitting expert data.
[0091] In the specific implementation process, the state and action in the expert data are constructed into a binary group (state, action), denoted as ,in, is a state, and the corresponding For experts in the state The actions taken when and These are the input and labels in supervised learning. is an integer between 1 and n.
[0092] The constructed gated recurrent unit (GRU) processes the state-action sequence information of the expanded expert data. The GRU model contains the update gate and reset gate and candidate hidden states .
[0093] The update gate formula is:
[0094]
[0095] Reset gate formula:
[0096]
[0097] Candidate hidden state formula:
[0098]
[0099] Final hidden state formula:
[0100]
[0101] in, represents the input of the current time step, W, U, B represent the weights and biases of the GRU unit, is the hyperbolic tangent activation function, represents the hidden state at the previous time step, Represents the sigmoid activation function, which is used for the gating mechanism. Represents element-wise multiplication, used for interaction of update and reset gates.
[0102] The objective function of the behavior cloning method is:
[0103]
[0104] in, It means that at time step t, given state When the state sequence is The probability distribution of the selected action; represents the action sequence in the entire sequence, T is the length of the time series; according to the parameters Strategy Select the corresponding state action The probability distribution of For expert data, is the loss function under supervised learning, It means that under the expert data, all possible state-action pairs (𝑠,𝑎) are sampled and the average value of these samples is calculated. is a parameter The optimal value of Next, strategy The parameters minimize the expected loss.
[0105] In the present invention, the cross entropy loss function is used as the loss function, and the cross entropy loss function is defined as:
[0106]
[0107] in, is the true distribution for action a at time step t, is the probability of the action predicted by the model.
[0108] The GRU network is trained with the expanded expert state action sequence. The goal of the training process is to adjust the model parameters To minimize the loss function , so that the model can learn accurate predictions.
[0109] (5) Evaluate and optimize strategies.
[0110] A deviation operation is performed between the action decision output by (4) and the action decision obtained from expert experience under the same input state, and the weight of the policy network is updated based on the deviation value.
[0111] The behavior cloning framework is shown in the figure Figure 2 As shown in the figure, since expert decision is the goal of decision network learning, the parameters of the decision network are adjusted by calculating the deviation between the strategy decision and the expert decision, aiming to make the generated decision closer to the expert decision. Through multiple rounds of training, the decision-making ability of the neural network will be close to the expert level or reach the expert's decision-making ability. The expert experience used by behavioral cloning is the combination of the initial expert experience mentioned above and the expert experience generated by the generative adversarial network.
[0112] The following uses a 10-on-10 red-blue symmetrical game air combat scenario as an example to illustrate this method.
[0113] (1) Create a battle scene and set the state space and action space of the fighter;
[0114] This confrontation is set as a battle for high ground between drones, and the red and blue sides launched a confrontation for the center position. Therefore, the victory or defeat of this experiment is also judged by calculating the scores obtained by calculating the number of remaining drones and the number of remaining missiles at the end of each round, plus the sum of the scores obtained by the distance of the drones from the center position at each time step. The side with the higher score is the winner of this round of confrontation. Specifically: the red and blue sides each have ten aircraft, each aircraft carries a number of weapons, and can attack enemy aircraft within the detection range. Points are obtained based on the number of enemy aircraft destroyed and the distance from the central strategic area. When training one side, there are three agents, and the ten drones are organized into three small formations, and each agent controls an aircraft formation.
[0115] Collect expert experience and perform standardized preprocessing:
[0116] Collect a large amount of information to summarize multiple expert strategies and write them into code. According to the situation processing and output action division, when the battlefield situation meets the trigger conditions, the expert rules will issue the corresponding upper-level decision, and then the script will issue specific instructions to the simulation system. Collect the expert data generated when the intelligent agent interacts with the environment. Then collect and standardize the generated expert experience data to ensure the consistency of the data in different dimensions, so that it has a unified scale standard, thereby improving the training efficiency and accuracy of the model and facilitating the subsequent training of the adversarial generation network.
[0117] (2) Build a generative adversarial network model and initialize network parameters
[0118] Because two of the three agents control a formation with three drones, and the other agent controls a formation with four drones, the corresponding GAN parameters are different, as shown in Table 1, which are the parameters of the generative adversarial network;
[0119] Table 1 is the parameters of the generative adversarial network of the present invention.
[0120]
[0121] Generator Network:
[0122] Input layer: receives a 154 (or 153) dimensional input vector, which is a noise vector sampled from the prior probability distribution;
[0123] The first fully connected layer: transforms the dimension of the input vector from 154 (or 153) dimensions to 64 dimensions and uses the ReLU activation function, which helps alleviate the gradient vanishing problem during training;
[0124] Dropout layer: Dropout ratio is 0.5;
[0125] Hidden layer: This layer is a 64-dimensional fully connected layer, which also uses the ReLU activation function;
[0126] The second fully connected layer: transforms the output of the hidden layer from 64 dimensions back to 154 (or 153) dimensions, also using the ReLU activation function;
[0127] Output layer: The final generator outputs a 154 (or 153) dimensional vector, which is a representation of the generated data and should theoretically be similar to the distribution of the real data.
[0128] Discriminator network:
[0129] Input layer: receives a 308 (or 306)-dimensional input vector, which usually consists of real data and data generated by the generator;
[0130] The first fully connected layer: transforms the dimension of the input vector from 308 (or 306) dimensions to 64 dimensions, and uses the LeReLU activation function, which allows negative inputs to pass a smaller gradient to prevent neurons from being completely inactive;
[0131] Dropout layer: Dropout ratio is 0.5;
[0132] The second fully connected layer: transforms the output of the first layer from 64 dimensions to 128 dimensions, using the LeReLU activation function;
[0133] Dropout layer: Dropout ratio is 0.5;
[0134] The third fully connected layer: transforms the 128-dimensional output into 1-dimensional, uses the ReLU activation function, and outputs the final judgment result of the discriminator;
[0135] Output layer: The output of the discriminator is a 1-dimensional scalar, which represents the probability that the input data is real data; during the training process, the goal of the discriminator is to maximize the probability that it can correctly classify real and generated data.
[0136] (3) Introducing regularization technology, training generative adversarial network models, and expanding the expert experience pool
[0137] This embodiment collects 2000 rounds of data, totaling 1,000,000 pieces of data as initial expert experience, standardizes the collected expert experience, and uses the processed expert experience to train the generative adversarial network to prepare for generating new expert experience;
[0138] The objective function of GAN is:
[0139] .
[0140] The loss function of the discriminator is:
[0141] L D = - E x ∼ p data ( x ) [log D ( x )] - E z ∼ p z ( z ) [log (1 - D ( G ( z )))]
[0142] The loss function of the generator is:
[0143] L G = - E z ∼ p z ( z ) [log D ( G ( z ))]
[0144] On the basis of the above basic loss function, add the regularization term L2. By adding L2 regularization to control the size of the weight, add corresponding terms to the loss functions of D and G:
[0145]
[0146] By penalizing large weights through L2 regularization, the model is encouraged to learn smaller weight values, which helps to reduce the complexity of the model's fitting of the training data, thereby reducing the risk of overfitting and improving the generalization ability of the model.
[0147] (4) Using behavioral cloning for imitation learning
[0148] The trained generative adversarial network is used to generate 1,000,000 expert data and added to the initially collected expert data as expanded expert experience data.
[0149] After obtaining the expanded expert experience data, the behavior cloning method in imitation learning is used for learning;
[0150] After obtaining the expanded expert state action sequence, a GRU model is constructed to process the sequence information. The GRU model contains an update gate and reset gate and candidate hidden states .
[0151] The update gate formula is:
[0152]
[0153] Reset gate formula:
[0154]
[0155] Candidate hidden state formula:
[0156]
[0157] Final hidden state formula:
[0158]
[0159] The cross entropy loss function is defined as:
[0160]
[0161] The objective function of behavior cloning is:
[0162] .
[0163] The core idea of the cross entropy loss function is that if the predicted probability distribution of the model is closer to the true distribution, the loss is smaller and the performance of the model is better. The goal of the training process is to adjust the model parameters To minimize the loss function , so that the model can learn accurate predictions.
[0164] (5) Evaluate and optimize strategies
[0165] The red team's initial model was trained using expert data. After training, the red team model was loaded into the blue team so that both sides used exactly the same model for adversarial testing. This setting evaluates the performance of the model more fairly. After 500 rounds of adversarial testing, the wins and losses of both sides were recorded in detail, and the win rate comparison of both sides was obtained. The experimental results show the performance and adaptability of the model in different combat parties. The specific data of the win rate is shown in Table 2;
[0166] Table 2 Statistics of the winning rate of the red and blue training models in the present invention
[0167]
[0168] like Figure 3 As shown in the figure, the reward function of the red and blue teams in the confrontation test is displayed. From the experimental result diagram, it can be clearly observed that the same model is loaded into the red team and the blue team for confrontation. From Table 2, it can be seen that the winning rates of the red and blue models are 49.4% and 50.6% respectively, which are almost the same. Figure 3 It can be seen that the reward values of both sides are also very similar, which fully demonstrates that the model has comparable levels on different combat sides, which shows the stability of the model.
[0169] The present invention also relates to a computer-readable storage medium on which is stored an expert-assisted intelligent agent training program based on a generative adversarial network, which, when executed by a processor, implements the expert-assisted intelligent agent training method based on a generative adversarial network.
[0170] The present invention also relates to a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the expert-assisted intelligent agent training method based on a generative adversarial network is implemented.
[0171] The present invention also relates to an application of the expert-assisted intelligent agent training method based on a generative adversarial network, which is applied to imitation learning in which the amount of initial expert data is less than a preset value.
[0172] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0173] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0174] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0175] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0176] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0177] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. An expert-assisted agent training method based on generative adversarial networks, applied to the red-blue symmetric game air combat scenario, characterized by: The method comprises the following steps: S1 Battle scene creation, setting the state space and action space of the fighter. When training one side, there are three agents, and ten drones are organized into three small formations, with each agent controlling one aircraft formation; collecting expert experience and preprocessing; the expert experience is a state-action pair; according to situation processing and output action division, when the battlefield situation meets the trigger conditions, the expert rules will issue the corresponding upper-level decision, and then the script will issue specific instructions to the simulation system; S2 constructs a generative adversarial network model for learning the distribution of expert experience and expanding the expert experience; the generative adversarial network model includes a generator network and a discriminator network; In the generator network, a Dropout layer is set after the first fully connected layer; In the discriminator network, the first two fully connected layers are followed by a Dropout layer; S3 introduces regularization technology to train the generative adversarial network, and uses the trained generative adversarial network to expand the expert experience preprocessed by S1; S4 improves the imitation learning network with a gated recurrent unit, processes the expanded expert state-action sequence, and uses a behavior cloning method to learn these data, and establishes a state-action mapping relationship by fitting the expert data; the gated recurrent unit is set before the policy network, inputs the state, and the policy network outputs the action decision; S5 evaluates and optimizes the strategy; performs deviation calculation on the action decision output by S4 and the action decision obtained from expert experience under the same input state, and updates the weight of the strategy network based on the deviation value.
2. The expert-assisted agent training method based on a generative adversarial network according to claim 1, characterized in that: In S1, the preprocessing is to standardize the state data to a standard normal distribution.
3. The expert-assisted intelligent agent training method based on generative adversarial network according to claim 1, characterized in that: In S3, the generative adversarial network model is iteratively trained using a back propagation algorithm and a gradient descent optimization algorithm, and the generator network and the discriminator network are respectively regularized during the training; The collected expert experience is expanded with the generator network of the trained generative adversarial network model.
4. The expert-assisted agent training method based on generative adversarial network according to claim 3, characterized in that: The objective function of the generative adversarial network model satisfy, , in, is the prior probability distribution Noise samples, independent variables To obey the distribution The real data For the general The probability distribution of classification as true data, To classify the real data as true, For the discriminator In the generator The complement of the expected log-likelihood on the generated fake data.
5. The expert-assisted intelligent agent training method based on generative adversarial network according to claim 4, characterized in that: Regularization processing includes Dropout regularization and L2 regularization; The loss function of the generator satisfies , The loss function of the discriminator satisfies , in, , , and are the parameters of the generator and the discriminator respectively, and is the corresponding regularization coefficient.
6. The expert-assisted intelligent agent training method based on generative adversarial network according to claim 5, characterized in that: The objective function of the behavior cloning method is: , in, is a parameter The optimal value of Next Strategy The parameters minimize the expected loss; It means that at time step t, given state When the state sequence is The probability distribution of the selected action; is the action sequence in the entire sequence, T is the length of the time sequence, For expert data, is the loss function under supervised learning, To sample all possible state-action pairs under the expert data and calculate the average of these samples.
7. The expert-assisted intelligent agent training method based on generative adversarial network according to claim 6, characterized in that: The loss function is the cross entropy loss function, satisfying , in, is the true distribution for action a at time step t, is the probability of the action predicted by the model; Train the GRU network with the expanded expert state action sequence and adjust the model parameters To minimize the loss function .
