Training method of generative model for state prediction based on adversarial neural network

Abstract

The present invention provides a training method for a generative model for state prediction based on an adversarial neural network, wherein the adversarial neural network comprises a generative model and a discriminator, wherein the generative model and the discriminator both comprise a multi-layer fully connected network, and the method comprises: S1, sampling a plurality of control action sequences of predicted time domain lengths from all feasible possible control actions under the constraints of a target system, and sending each control action sequence into a real environment of the target system for execution to obtain a target state sequence of the target system corresponding to each control action sequence, and sending each control action sequence into a generative model to obtain a target future state prediction sequence of the target system corresponding to each control action sequence; S2, generating a training set by composing samples based on the current state of the target system, each control action sequence, and the target state sequence of the target system corresponding to each control action sequence after being executed; and S3, training the generative adversarial network multiple times until convergence using the training set.

Description

Technical Field

[0001] The present invention relates to the field of automatic control, in particular to the field of MPC-based continuous trajectory control in automatic control, and more particularly to a training method for a generative model for state prediction based on an adversarial neural network, and an MPC control method and device based thereon. Background Art

[0002] With the development of technology, automatic control technology has been applied to all aspects of production and life, greatly facilitating people's lives. For example, robotic arms, drones, and autonomous driving all use automatic control technology. Among them, Model Predictive Control (MPC), a multivariable control method in automatic control methods, is widely used because it can handle various constraints of each state variable and can be used in more complex nonlinear environments. In order to achieve continuous trajectory control, MPC needs to implement a multi-step prediction process. Among them, the so-called MPC is an advanced process control method, which uses the system model to predict the state sequence of the target system in the future according to the current state of the target system to which automatic control is applied and the control action sequence planned to be executed in the future, that is, multi-step prediction, and selects the control action to control the target system to execute according to the degree of difference between the predicted state sequence and the target state sequence, so that the target system reaches the target state by executing the control action. Among them, a system model based on an efficient multi-step prediction framework and with accurate prediction is very important for the implementation of MPC.

[0003] Under the existing technology, due to the limitation of computational cost and model capability, most MPC methods use an iterative framework mode to complete the multi-step prediction process. The system model under the iterative framework only predicts the system state after one moment at a time, and then predicts the future state after multiple moments in turn through iteration until the required prediction time domain length is reached, such as Figure 1 As shown, t represents the state of the target system at time t, and the prediction time domain length is N (N is used throughout the text to represent the prediction time domain length). Then, the system model under the iterative framework is based on the state s at time t. t and the future control action (represented by u) planned to be executed at time t predicts the state s of the target system at time t+1 t+1 ′, the system model in the iterative framework is based on the predicted state s at time t+1 t+1 ′ and the future control action planned to be executed at time t+1 predicts the state s of the target system at time t+2 t+2 ′, and so on, until the state s of the target system at time t+N is obtained t+N′, all the predicted states are spliced ​​to obtain a complete target system future state prediction sequence. In this iterative process, the system model uses a single model for single-step prediction, which reduces the requirements for the system model. However, the iterative process in a complex environment will bring high computational complexity. In addition, the iterative framework has the problem of cumulative error. For example, in the iterative framework, assuming that the deviation of the system model is δ(s t ,u t ), the system model predicts the output s′ at time t+1 t+1 =f(s t ,u t )+δ(s t ,u t ), it can be seen that the small errors introduced by the system model in the iterative framework will be continuously amplified in the prediction process, and the deviations will continue to accumulate in the iterative process. This error becomes more serious as the prediction time domain length N increases, and eventually leads to huge cumulative errors. Under this iterative architecture, the system model input has no contextual information, which limits the amount of system information that can be obtained by the system model established under this framework, further limiting the capabilities of the system model. At the same time, the time complexity of this process is O(N), where N is the length of the prediction horizon. Since MPC is usually used in environments such as robots, it requires extremely fast control action solution speed. This time complexity determines that the system model cannot be too complex, further limiting the performance of the system model.

[0004] In order to alleviate the problem of cumulative error of system models under the iterative framework, researchers proposed a multi-model synchronous prediction framework. This framework uses multiple system models. Each model predicts the future control actions based on the current state and plan, predicts the future state at a fixed future time, and splices the prediction results of all models to obtain a complete series of future states, such as Figure 2 As shown, t Represents the state of the target system at time t, the prediction time domain length is N, and the system model 1 under the multi-model synchronous prediction framework is based on the state s at time t t and the state s of the target system at time t+1 predicted by the future control action planned to be executed at time t t+1 ′, at the same time, the system model 2 under the multi-model synchronous prediction framework is based on the state s at time t t and the state s of the target system at time t+2 predicted by the future control actions planned to be executed at time t and time t+1 t+2 ′, ..., the system model N under the multi-model synchronous prediction framework is based on the state s at time t t And all future control actions planned to be executed from time t to time t+N-1 predict the state s of the target system at time t+N t+N′, all predicted states are spliced ​​together to obtain a complete target system future state prediction sequence. In the multi-model synchronous prediction framework, each system model is different, which greatly alleviates the problem of single model cumulative error in the iterative framework and improves the speed of the multi-step prediction process. However, the design of multiple models is difficult and the amount of calculation required for prediction increases exponentially. This is because the model corresponding to each moment needs to be designed separately, and as the length of the prediction time domain increases, the later the model, the greater the corresponding calculation amount, because the control actions planned to be executed at all previous prediction moments are required.

[0005] In summary, the existing system models for multi-step prediction have the problems of high computational cost and inaccurate prediction. Therefore, how to design a system model based on an efficient multi-step prediction framework and with accurate prediction is still an urgent problem to be solved in MPC. Summary of the invention

[0006] Therefore, the purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a new system modeling method and an MPC control method and device based thereon.

[0007] According to a first aspect of the present invention, there is provided a training method for a generative model for state prediction based on an adversarial neural network, wherein the adversarial neural network comprises a generative model and a discriminator, wherein the generative model and the discriminator both comprise a multi-layer fully connected network; the generative model is used to select a control action sequence to be executed by the target system at multiple moments in the future according to the current state of the target system and the target state sequence of the target system, wherein the method comprises: S1, sampling a plurality of control action sequences of predicted time domain length from all feasible possible control actions under the constraints of the target system, and sending each control action sequence into the real environment of the target system for execution to obtain a target state sequence of the target system corresponding to each control action sequence, and sending each control action sequence into the generative model to obtain a target system future state prediction sequence corresponding to each control action sequence; S2, generating a training set by composing a sample of the current state of the target system, each control action sequence, and the target state sequence of the target system corresponding to each control action sequence after the execution of each control action sequence; S3, training the generative adversarial network multiple times until convergence using the training set.

[0008] Preferably, the generative model and the discriminator both include a 4-layer fully connected network.

[0009] Preferably, in step S3, the discriminator loss and L2 loss are used to train the generative model in the following manner:

[0010]

[0011] Where α is the discriminator loss weight, β is the L2 loss weight, and α+β=1, m is the batch size of the sampled control action sequence in each training, and s t is the current state of the target system during the current training, U i The i-th control action sequence, S i ′ represents the target system's future state prediction sequence corresponding to the target system executing the ith control action sequence in the current state, D ω (s t ,U i ,S i ′) represents the current state s t and action sequence U i The state S predicted by the generative model i ′ is the probability from the real environment.

[0012] According to a second aspect of the present invention, an MPC control method is provided, which is applied to a target system of automatic control and is used to select control action sequences to be executed by the target system at multiple moments in the future according to the current state of the target system and the target state sequence of the target system. The method comprises: T1, sampling a control action sequence of a preset number of predicted time domain lengths corresponding to the target system, wherein the control action at each moment in the control action sequence is sampled from all feasible possible control actions under the constraints of the target system; T2, using a generative model trained by the method as described in the first aspect of the present invention to predict the future state sequence of the target system after executing the control action sequence based on each sampled control action sequence and the current state of the target system to obtain a target system future state prediction sequence of a preset number of predicted time domain lengths; T3, selecting a control action sequence corresponding to the target system future state prediction sequence that is closest to the target state sequence of the target system from among all the control action sequences corresponding to the sampled target systems.

[0013] Preferably, in step T1, a CEM strategy is used for sampling.

[0014] Preferably, in step T3, a control action sequence corresponding to a target system future state prediction sequence closest to a target state sequence of the target system is selected from control action sequences corresponding to all sampled target systems, and the first two control actions in the control action sequence are selected.

[0015] Preferably, the step T3 includes: T31, calculating the loss between the target system future state prediction sequence corresponding to each sampled control action sequence and the target system target state sequence; T32, selecting the control action sequence with the smallest loss between the corresponding target system future state prediction sequence and the target system target state sequence.

[0016] In some embodiments of the present invention, in step T31, the loss between the target system future state prediction sequence and the target state sequence corresponding to each control action sequence is calculated in the following manner:

[0017]

[0018] Among them, L i represents the loss between the target system future state prediction sequence corresponding to the i-th control action sequence and the target state sequence of the target system, S star represents the target state of the target system, S i ′ represents the target system future state prediction sequence corresponding to the target system executing the i-th control action sequence, U i represents the i-th control action sequence, Q represents the weight of the state L2 loss, R represents the weight of the control action penalty term, (·) T Represents the transpose of a vector.

[0019] Preferably, the target system future state prediction sequence with the corresponding prediction time domain length for each control action sequence is the vector sum of a vector consisting of a preset number of target system current states and a difference sequence of the output of the pre-trained system model.

[0020] According to a third aspect of the present invention, an MPC control device is provided, which is configured in an automatically controlled target system and is used to select control action sequences to be executed by the target system at multiple moments in the future according to the current state of the target system and the target state sequence of the target system. The device comprises: a sampler, which is used to sample multiple control action sequences of predicted time domain lengths corresponding to the target system, wherein the control actions at each moment in the control action sequence are sampled from all feasible possible control actions under the constraints of the target system; a generation model constructed by the method described in the first aspect of the present invention, which is used to predict the future state sequence of the target system after executing the control action sequence based on each control action sequence sampled by the sampler and the current state of the target system to obtain a target system future state prediction sequence of a preset number of predicted time domain lengths; and an optimizer, which is used to select, from all sampled control action sequences, a control action sequence corresponding to the target system future state prediction sequence that is closest to the target system target state sequence.

[0021] Compared with the prior art, the advantages of the present invention are: the single model synchronous prediction framework proposed by the present invention uses one model to complete the multi-step prediction process at one time, which reduces the time required for the multi-step prediction process and improves the prediction accuracy, and finally enhances the ability of MPC in complex environments and improves the MPC decision-making effect; the system model based on the generative adversarial network proposed by the present invention uses the generator as the system model network, so that the system model can learn training data with different distributions with the help of the discriminator, fit the complex environmental characteristics and constraints that are difficult to define artificially, alleviate the training error caused by the training data with non-uniform distribution, improve the model accuracy, and design the system model network training loss that integrates the supervised learning loss and the discriminator loss in the training process, accelerate the system model network training process with the help of the gradient direction that is more obvious in the early stage of training, and in the later stage of training, the system model network can learn various environmental constraints that are difficult to define artificially with the help of the discriminator to improve the model accuracy, accelerate the system model network training process, and improve the final training effect. Based on the scheme of the present invention, the time complexity of the multi-step prediction process is reduced from O(N) to O(1), and since there is no iterative process, the cumulative error problem existing in the iterative framework is eliminated. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0023] Figure 1 A schematic diagram of multi-step prediction based on an iterative framework under the prior art;

[0024] Figure 2 A schematic diagram of multi-step prediction based on a multi-model synchronous prediction framework under the existing technology;

[0025] Figure 3 A schematic diagram of multi-step prediction of a single-model synchronous prediction framework according to an embodiment of the present invention;

[0026] Figure 4 A schematic diagram of the principle of a model based on an adversarial neural network modeling system according to an embodiment of the present invention;

[0027] Figure 5 A schematic diagram of an MPC device framework implemented by a system model based on adversarial neural network modeling according to an embodiment of the present invention;

[0028] Figure 6 Schematic diagram of the principle of multi-step synchronous prediction of a system model based on adversarial neural modeling according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0030] As described in the background technology, under the existing technology, the multi-step prediction process in MPC still has some problems, mainly because the system model for implementing multi-step prediction is still not efficient and accurate enough. In the multi-step prediction process, the system models used are mainly divided into traditional mathematical models and learning-based models.

[0031] The effectiveness of the traditional system models currently used is highly dependent on manual design and is usually used in linear or simple nonlinear environments. It is difficult to manually design a system model that fits the complex nonlinear environment well, and such models cannot learn and tune on their own. The typical traditional mathematical model is a traditional linear model in the form of s(t+1)=As(t)+Bu(t)+w, which is manually defined. In this model, s(t) represents the target system state at time t, u(t) represents the control action taken at time t, and A, B, and w are fixed parameters. Linear models can achieve good results in linear or relatively simple nonlinear environments. However, in more complex environments, many attributes are very difficult to represent by manual definition, such as contextual information between states, state sequence patterns in different environments, etc. Because the environments at different times are different and the correlation between the previous and next states is different, it is difficult to manually define a linear model that can accurately predict different times in a complex environment to achieve multi-step prediction in an iterative framework; similarly, in a multi-model synchronous prediction framework, it is even more difficult to manually define prediction models at different times to achieve multi-model synchronous prediction in a complex environment.

[0032] The learning-based method can learn from the collected data by itself without manual design. It can guide the model to train in the desired direction through the specified loss function, and can fit linear and nonlinear environments. With the development of the field of robotics, some more complex tasks have emerged, such as mobile robot path tracking and autonomous navigation of quadcopters. There are many features or noises in such complex nonlinear system environments, and it is difficult for traditional methods to manually define the mathematical expressions of these features or noises. Learning-based models such as neural networks can fit these noises, so they have advantages. The typical learning-based system model is a fully connected model. The model needs to send the target system to execute by randomly sampling control actions, record the state after each control action is executed to form a data set, and then use this data set to train a fully connected neural network to complete the fitting of the system model. However, the learning-based model needs to design a loss function to guide the optimization direction of the model. The currently commonly used supervised learning loss (L2 loss) helps the model determine the optimization direction by comparing the output of the network model in the same environment with the true value. This loss cannot capture the feature differences between different inputs, such as the difference between different terrains such as grass and mud in the autonomous driving control task, which causes the model fitting result to always have a certain error with the true value. In addition, for learning-based models, training data needs to be collected. In the data collection stage, the collected data are the environmental state and the decisions of human experts in this environment, such as path, control amount, etc. The collected data set provides a basis for the model learning system to learn about the changes in the control amount during the training stage. The training data collected in such complex environments usually contain distribution shifts that are inevitable due to different expert strategies. If the current mainstream supervised learning similar losses are used alone, these data with different distributions will cause the model to optimize in different directions, affecting the model training effect, and ultimately affecting the multi-step prediction effect. The impact is mainly reflected in the inability to directly reflect the contextual information of the environment, especially the drastic state changes that rarely occur in normal environments. Therefore, such a model is difficult to fully fit the real environment, resulting in errors. Due to the different environments at different times and the different correlations between the previous and next states, it is impossible to train a model that can meet the correlations between different states in a complex environment to achieve multi-step prediction in the iterative framework; similarly, in the multi-model synchronous prediction framework, it is also difficult to train the corresponding models at different times to achieve multi-model synchronous prediction. Even if it can be achieved, as the prediction time domain increases, the number of models will increase accordingly, bringing greater computational overhead and reducing the prediction speed.

[0033] The inventor found in the MPC research that the defects in the prior art are caused by the long time-consuming multi-step prediction process of the system model and the inaccurate prediction results. There are two reasons for this phenomenon: first, the existing iterative framework requires multiple iterations to repeatedly call the system model for prediction, which is slow and has cumulative error problems, while the multi-model synchronous prediction framework needs to design a model for each future moment, which is difficult to design, and the large amount of calculation of multiple models leads to slow prediction speed. Second, in the system model currently used, the linear model based on the traditional method is artificially defined, which can be well fitted for linear environments or simple nonlinear environments, but it is difficult to accurately define it to fit the system model in a complex nonlinear environment, and the error is large; and the loss function used by the learning-based model only considers the difference between the predicted results and the actual results, and does not consider the features that are difficult to reflect through fixed mathematical expressions (such as expert strategy differences when collecting data) and the additional constraints inherent in the environment (such as contextual information between states, state sequence patterns of different surfaces such as grass and mud, etc.), which leads to a certain error between the learned model and the real environment, affecting the accuracy of the multi-step prediction process, and ultimately affecting the effect of the model predictive control method.

[0034] In order to solve the above problems, the inventors proposed a scheme to improve the multi-step prediction framework and construct a single-model synchronous prediction framework, wherein improvements in two directions are mainly considered: 1) Abandoning the iterative method and realizing synchronous prediction with the help of the multi-head output characteristics of the neural network, wherein the so-called multi-head output characteristics of the neural network refer to the ability to configure the structure of the neural network so that the neural network has corresponding multiple outputs according to multiple inputs. Since the multi-head output characteristics of the neural network are known to those skilled in the art, the present invention does not elaborate on them in detail; 2) Since synchronous prediction saves computing time and can support more complex system models, multiple artificially defined models are no longer used to synchronously predict future states at different times. Instead, a single learning-based neural network model is used as the system model. When realizing multi-step prediction based on the system model scheme of the single-model synchronous prediction framework proposed in the present invention, one model can be used to complete the multi-step prediction process at one time, such as Figure 3 As shown in the single model synchronous prediction framework, a system model is used based on the state s at time t t The length of the planned execution is N (prediction time domain length). The future control action sequence is predicted at one time to obtain the future state sequence of the target system with a length of N: s t+1 ′、s t+2 ′, …, s t+N′, so the time complexity is O(1). The time complexity brought by the synchronous prediction framework is reduced, so that even if a more complex neural network is used to fit the system model, the real-time performance of MPC can still be guaranteed. Using one model to complete the prediction process avoids the use of multiple models, reduces the complexity of model design, and reduces the computational requirements.

[0035] Based on the single model synchronous prediction framework of the present invention, at each sampling moment, MPC samples multiple control action sequences of length N through a sampler according to the current state of the target system, and uses the system model of the single model synchronous prediction framework to predict the future predicted state sequence of the target system according to each control action sequence, and selects the one closest to the target state sequence from these future predicted state sequences of the target system through an optimizer, and sends the first action or the first two actions or the first multiple actions in the control action sequence corresponding to the future predicted state sequence of the target system as the actions to be executed by the target system in the future to the target system for execution. Since the system model realizes multi-step prediction at one time, the prediction efficiency can be greatly improved.

[0036] In order to make full use of the characteristics of the single-model synchronous prediction framework, a sufficiently powerful system model needs to be designed. Since the real system of the interaction between complex environments such as quadcopters, mobile robots and the environment is difficult to describe, traditional system models are difficult to design. The system model based on pure supervised learning deteriorates rapidly when the training data has multiple distributions. According to one embodiment of the present invention, the present invention uses a generative adversarial network to model the system model. The generative adversarial network is based on game theory and can learn features or constraints that cannot be represented by the pure supervised learning loss function. According to one embodiment of the present invention, the generative model in the generative adversarial network is used as the system model, so that the system model learns training data with different distributions with the help of the discriminator in the generative adversarial network, and fits the complex environment features and constraints that are difficult to define. It can also be understood that by adding a discriminator on the basis of the original system model network, the model can fit the features that are difficult to define artificially in a complex environment, thereby improving the accuracy of the model. Finally, in the process of training the system model, an L2 loss based on supervised learning is added, which helps the network converge quickly in the early stage of training, speeds up the training process, thereby improving the speed and accuracy of the multi-step prediction process of the key step in the model predictive control, and ultimately improving the efficiency and control effect of the model predictive control.

[0037] In order to better understand the present invention, the technical idea of ​​modeling the system model under the single-model synchronous prediction framework of the present invention is first introduced: T1. Design a fully connected network as a generative model (system model network) to predict future states, use the same network structure to design a discriminator, and form a generative adversarial network composed of a generative model and a discriminator, wherein the output of the generative model is the predicted state, and the output of the discriminator is the probability that the state sequence input to the discriminator comes from the real environment; T2. Based on the current target system state, sample a series of planned future control action sequences; T3. Send the sampled control action sequence into the real environment of the target system to obtain the actual future state sequence as the true value; T4. Send the sampled control action sequence into the generative model and output the predicted future state; T5. Calculate the L2 loss of the actual future state and the predicted future state. At the same time, use the discriminator to calculate the probability that the predicted future state belongs to the real environment, and calculate the discrimination loss based on this; T6. The discrimination loss and L2 loss calculated in step T5 are weighted and trained as the loss of the training system model according to a certain weight. Preferably, as Figure 4 As shown, the present invention uses the generative model as the system model G θ (s t ,U), which means from the current state s t and the mapping of the control action sequence U to the future state space and the network parameter is θ, that is, the system model can be based on the current state s t And the control action sequence is used to obtain the future predicted state sequence S′. In the system modeling (training) process, an additional discriminator D is added ω (s t ,U,S′), used to calculate the current state s t and control action sequence U estimates a state S′ from the real future state rather than the system model network G θ (s t,U), and based on this probability, the training effect of the system model is judged. If the probability is 0.5, it means that the output of the system model network is almost the same as the real future state, indicating that the training effect is good. If the probability is biased towards 0 or 1, it means that the system model network does not fit the real environment well, and it is easy to be distinguished as not a real environment. Therefore, the discriminator can provide losses for the training of the system model network, help fit various constraints that are difficult to define artificially, context information that cannot be directly represented by artificially designed losses, etc., especially in the case of drastic state changes that rarely occur in normal environments. In addition, the discriminator is only used in the process of training the system network and will not increase the complexity of the test. In the training process of the adversarial network, in most cases, the system model directly outputs the future state sequence of the target state, but such a training method may make the prediction accuracy of the system model too dependent on the effect of network training. In order to avoid the excessive dependence of the prediction accuracy of the training system model on the effect of network training, the present invention uses the difference prediction sequence between the future state and the current state as the output of the system model for training.

[0038] According to one embodiment of the present invention, the pseudo code of the specific adversarial training process is shown in Table 1.

[0039] in:

[0040] Line 1: Initialize the generative model G, the discriminator D, the prediction time domain length N, the ratio of the generative model and the discriminator training times k, and the hyperparameters α, β, and c (where α is the discriminator loss weight, β is the L2 loss weight, and α+β=1, and c is the training gradient threshold)

[0041] Line 2: Randomly sample an action sequence U of length N in the real environment and execute it. Obtain the state S in the real environment corresponding to each action sequence. Record the actions and states to form a data set P = {(S i ,U i ),....}(The state in the real environment corresponding to the action sequence is the target state, which is the goal to be achieved by generating the model prediction output)

[0042] Line 3: while not converged do (multiple rounds of adversarial training of the generative model and the discriminator until convergence)

[0043] Line 4: fork do (it should be noted that in each round of training, the discriminator is first trained multiple times (expressed as k times) through lines 5 to 10, and then the generative model is trained once through lines 11 to 15)

[0044] Line 5: Sample m state-action pairs from dataset P (Samples are sampled from the data set for training the discriminator. m is the batch size sampled during training. The batch size is related to the computing power of the hardware during training. The state sequence contains the current state and a set of future target states )

[0045] Line 6: After copying N times, it is superimposed into vector S base (The embodiment of the present invention uses the difference prediction sequence between the future state and the current state as the output of the generation model, so it is necessary to copy the current state multiple times for superposition with the output of the generation model to obtain each future prediction state)

[0046] Line 7: Use the generative model to predict future state sequences Indicates that the generated model is based on the current state And the action sequence U i The output is a prediction sequence of the difference between the future state and the current state. Each difference in the prediction sequence plus the current state constitutes the predicted future state sequence.

[0047] Line 8: Calculate the gradient is the gradient of the discriminator, where D ω (s t ,U i ,S i ′) represents the current state s t and action sequence U i The state S predicted by the generative model i ′ is the probability of coming from the real environment, D ω (s t ,U i ,S i ) represents the target state S i Probability from the real environment)

[0048] Line 9: Backward pass after gradient truncation (ω D represents the parameters of the discriminator, and clip represents gradient truncation. Since gradient truncation is a technique known to those skilled in the art, it will not be described in detail here.)

[0049] Line 10: end for

[0050] Line 11: Sample m state-action pairs from dataset P (samples are sampled from the dataset for training the generated model, m is the batch size sampled during training, the batch size is related to the computing power of the hardware during training, and the state sequence contains the current state and a set of future target states )

[0051] Line 12: After copying N times, it is superimposed into vector S base (The embodiment of the present invention uses the difference prediction sequence between the future state and the current state as the output of the generation model, so it is necessary to copy the current state multiple times for superposition with the output of the generation model to obtain each future prediction state)

[0052] Line 13: Predict future state sequence Indicates that the generated model is based on the current state And the action sequence U i The output is a prediction sequence of the difference between the future state and the current state. Each difference in the prediction sequence plus the current state constitutes the predicted future state sequence.

[0053] Line 14: Calculate the gradient ( is the gradient of the generative model, where D ω (s t ,U i ,S i ′) represents the current state s t and action sequence U i The state S predicted by the generative model i ′ is the probability from the real environment, represents the discrimination loss, represents the L2 loss of the generative model, α is the discriminator loss weight, β is the L2 loss weight, and α+β=1, m is the batch size sampled during training)

[0054] Line 15: Gradient backpropagation (θ G are the parameters of the generative model and are updated based on gradient backpropagation)

[0055] Line 16: end while

[0056] It should be noted that the structure of the system model may be different in different target systems. For example, when MPC is applied to the path tracking task in autonomous driving, the present invention adopts a system model and a discriminator of a 4-layer fully connected network, and the number of neurons in each layer is 128, 256, 256, and 128, respectively.

[0057] It can be seen from the above pseudo code that compared with the existing traditional adversarial generative training, the present invention adds a supervised learning loss based on L2 to the training loss, and accelerates the training process with the help of the more obvious gradient direction of supervised learning in the early stage of training. The system model loss function used in the present invention can be obtained by adding the supervised learning loss and the discriminator loss according to a certain weight:

[0058]

[0059] Among them, α is the discriminator loss weight, β is the L2 loss weight, and α+β=1. The specific α and β are determined according to the specific task. Usually, it is guaranteed that when the training tends to be stable, the L2 loss is less than the discriminator loss. m is the batch size of the sampled control action sequence in each training, and s t is the current state of the target system during the current training, U i The i-th control action sequence, S i ′ represents the target system's future state prediction sequence corresponding to the target system executing the ith control action sequence in the current state, D ω (s t ,U i ,S i ′) represents the current state s t and action sequence U i The state S predicted by the generative model i ′ is the probability from the real environment.

[0060] The system model obtained through the above training process can perform multi-step prediction under the single model synchronous prediction framework. The following describes in detail how the system model under the single model synchronous prediction framework of the present invention implements MPC in combination with the accompanying drawings and embodiments.

[0061] During the implementation of MPC, the main purpose is to find the current state s at time t. t The best control action sequence And sent to the real environment of the target system for execution, where the current state s is predicted through a multi-step prediction process. t And the given control action sequence U t Predict the future state sequence S of length N, and select the action sequence corresponding to the state closest to the target state as the optimal control action sequence According to one embodiment of the present invention, the MPC architecture implemented based on the system model under the single model synchronous prediction framework of the present invention is as follows Figure 5 As shown, sampler 1 first generates a batch of control action sequences according to the sampling strategy The generated control action sequence is simultaneously input into the system model 3 generated based on the above embodiment under the single model synchronous prediction framework 2 to predict the state at each moment from t to t+N. The optimizer 4 selects the best state (the best state is the state closest to the target state) from the predicted series of future states according to the task goal, and obtains the control action sequence corresponding to this state, which is the optimal control action sequence. This sequence will be sent to the real environment 5 of the target system as a decision result for execution. In actual application, in order to ensure the real-time performance of MPC, the optimal control action sequence is generally The actions that are earlier in the sequence (such as the first or first two actions) are sent to the real environment for execution, and the above process is repeated at each moment to continuously obtain the actions executed by the control target system in the real environment and realize real-time control.

[0062] According to one embodiment of the present invention, the sampler adopts the CEM strategy for sampling, and uses the normal distribution as the action sampling distribution, wherein the action refers to all possible feasible actions under the constraints of the target system. For example, in autonomous driving, assuming that the acceleration (in km / s) constraint is [5,10], and the angular velocity (in degrees / second) constraint is [3,6], then as long as the actions satisfying these two constraints are feasible actions, the action sampling is to sample the acceleration and angular velocity pairs corresponding to each moment, and obtain an action sequence composed of acceleration and angular velocity pairs at multiple moments. Simply put, in the sampling process, the expectation and variance of the action sampling are first initialized, and multiple action sequences of length N are sampled on this distribution. After the sampled action sequences are sent to the multi-step prediction process and optimizer, several better action sequences are obtained. The expectation of the normal distribution is recalculated according to the feedback of these action sequences in the real environment (expressed as environmental rewards) (according to one embodiment of the present invention, the expectation in the present invention refers to finding the action sequence that satisfies the expectation). The best action sequence of U * , where L i is the preset prediction loss, and L i =(S tar -S′ I ) T Q(S tQr -S′ i )+U i T RU i , S tar is the target state, S′ i is the predicted state, (S tar -S′ i ) is the L2 loss of the state, U iis the sampled action sequence, Q and R are the weights of the state L2 loss and action penalty term, and Q and R are determined by the specific target system task) and variance, as the action sampling distribution of the next round, and so on. If the gap between the two rounds of expectations is less than a certain value, it means that a better action expectation has been found. Since the CEM strategy is a technology known to those skilled in the art, the present invention will not be described in detail.

[0063] Among them, Figure 6 As shown, each time a prediction is made, the sampler samples multiple action sequences of length N (represented by U) based on the sampling strategy. The system model under the single model synchronous prediction framework of the present invention is based on the current state s t And each action sequence predicts the future state corresponding to the action sequence, that is, predicts all future states corresponding to all action sequences (represented by S′). The optimizer selects the target state S from all action sequences based on the future state corresponding to the action sequence. tar The action sequence corresponding to the closest future state is taken as the optimal action sequence U * , and send the best action in the best action sequence to the environment for execution.

[0064] According to one embodiment of the present invention, the pseudo code of the system model MPC under the single model synchronous prediction framework of the present invention is shown in Table 2.

[0065]

[0066] in:

[0067] Line 1: Use the trained generative model parameters to initialize the system model G, set the prediction time domain length N, the maximum number of iterations K, and the hyperparameters Q and R (the system model is initialized using the generative model previously trained based on the generative adversarial network, so that it can achieve single-model synchronous prediction for the multi-step prediction process. The maximum number of iterations represents the number of action sequence entries sampled during synchronous prediction, or the number of samplings. The more action sequences sampled, the better the best action sequence obtained from them. Q and R represent the L2 loss weight of the state and the weight of the action penalty term, respectively)

[0068] Line 2: Set the target state sequence S sll-tar ={s1, s2, ....} (the target state sequence is the sequence of all target states that the target system needs to predict in the future)

[0069] Line 3: while the final target state is not reached do (before reaching the final target state, predict the future state in each prediction time domain through lines 4 to 9)

[0070] Line 4: Update the current state s t(In each prediction time domain, the initial current state is different, so it is necessary to update the starting current state corresponding to each prediction time domain at each prediction)

[0071] Line 5: Based on the current state s t With the target state sequence S sll-tar , determine the target state S for the next prediction time domain N tar (The target state sequence contains all the target states, which include target states corresponding to multiple time domains. The target state corresponding to each time domain is different. In each prediction, it is necessary to determine the target state of the next time domain based on the current state, and then predict the future state of the next time domain through lines 6 to 9)

[0072] Line 6: for i=1,…,Kdo (predict the future state of the next time domain by sampling K action sequences to find the best action sequence)

[0073] Line 7: Sample action sequence according to the CEM strategy (The CEM strategy is a technique known to those skilled in the art and will not be described in detail in the present invention)

[0074] Line 8: Replace s t After copying N times, it is superimposed into vector S base (The embodiment of the present invention uses the difference prediction sequence between the future state and the current state as the output of the generation model, so it is necessary to copy the current state multiple times for superposition with the output of the generation model to obtain each future prediction state in the next prediction time domain)

[0075] Line 9: Predict the future state sequence S i ′=G θ (s t ,U i )+S base Indicates that the generated model is based on the current state And the action sequence U i The output is a prediction sequence of the difference between the future state and the current state. Each difference in the prediction sequence plus the current state constitutes the predicted future state sequence.

[0076] Line 10: Calculate the loss L i =(S tar -S′ i ) T Q(S tar -S′ i )+U i T RU i ((S tar -S′i ) represents the L2 loss of the state, Q and R represent the L2 loss weight of the state and the action penalty weight respectively, L i Indicates the difference between the predicted future state and the target state, L i The larger the value, the greater the difference between the predicted future state and the target state. Conversely, the closer the difference between the predicted future state and the target state is.)

[0077] Line 11: end for

[0078] Line 12: Calculate the best action sequence (Find the action sequence whose corresponding predicted future state sequence is closest to the target state sequence from all action sequences)

[0079] Line 13: From U * Select the two most forward actions as u t , and u t Send it to the target system for execution in the real environment (send the first two actions in the optimal action sequence to the real environment for execution, and continuously predict new optimal actions to achieve real-time control of the target system)

[0080] Line 14: end while

[0081] It can be seen from the above embodiments that the present invention uses a single-model synchronous prediction framework and a generative adversarial network system model to solve the problems of high time consumption and reduced prediction accuracy in the multi-step prediction process in model predictive control in complex environments, and can provide better decision-making efficiency and decision-making results in complex environments and under limited computing power.

[0082] In order to better verify the effect of the present invention, the inventors used the single model synchronous prediction framework of the present invention and other frameworks under the prior art to perform multi-step prediction in different task environments (game environment 1, game environment 2, path tracking environment, and real car autonomous driving environment) (wherein, Iter-Fixed refers to fixed linear iterative prediction under the iterative framework, Iter-CNN refers to convolutional neural network iterative prediction under the iterative framework, and SMS-DNN refers to deep neural network prediction under the multi-model synchronous prediction framework), and compared the accuracy and prediction time of each prediction method respectively. The experimental results are shown in Table 3.

[0083] Table 3

[0084]

[0085] It can be seen from Table 3 that the prediction accuracy of the method of the present invention is significantly higher than that of other methods, and the time required is significantly shorter than that of other methods, which can better meet the real-time requirements of multi-step prediction.

[0086] In summary, compared with the prior art, the single model synchronous prediction framework proposed in the present invention uses one model to complete the multi-step prediction process at one time, which reduces the time required for the multi-step prediction process and improves the prediction accuracy, and finally enhances the ability of MPC in complex environments and improves the MPC decision-making effect; the system model based on the generative adversarial network proposed in the present invention uses the generator as the system model network, so that the system model can learn training data with different distributions with the help of the discriminator, fit the complex environmental characteristics and constraints that are difficult to define artificially, alleviate the training error caused by the training data with non-uniform distribution, improve the model accuracy, and design the system model network training loss that integrates the supervised learning loss and the discriminator loss in the training process, accelerate the system model network training process with the help of the gradient direction that is more obvious in the early stage of training, and in the later stage of training, the discriminator can be used to help the system model network learn various environmental constraints that are difficult to define artificially to improve the model accuracy, accelerate the system model network training process, and improve the final training effect. Based on the scheme of the present invention, the time complexity of the multi-step prediction process is reduced from O(N) to O(1), and since there is no iterative process, the cumulative error problem existing in the iterative framework is eliminated.

[0087] It should be noted that although the above describes the various steps in a specific order, it does not mean that the various steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0088] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0089] Computer readable storage medium can be a tangible device that holds and stores instructions used by an instruction execution device. Computer readable storage medium can include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.

[0090] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A training method for a generative model for state prediction based on an adversarial neural network, wherein the adversarial neural network comprises a generative model and a discriminator, characterized in that: The generation model and the discriminator both include a multi-layer fully connected network; the generation model is used to select a control action sequence to be executed by the target system at multiple moments in the future according to the current state of the target system and the target state sequence of the target system, and the method includes: S1. From all possible feasible control actions under the constraints of the target system, multiple control action sequences with predicted time domain lengths are sampled, and each control action sequence is sent into the real environment of the target system for execution to obtain the target state sequence of the target system corresponding to each control action sequence, and each control action sequence is sent into the generation model to obtain the target system future state prediction sequence corresponding to each control action sequence; S2, generating a training set by composing samples based on the current state of the target system, each control action sequence, and the target state sequence of the target system corresponding to each control action sequence after being executed; S3. Use the training set to train the generative adversarial network multiple times until convergence, wherein the generative adversarial network is trained by the following steps: Initialize the generative model G, discriminator D, prediction time domain length N, the ratio of generative model and discriminator training times k, and hyperparameters α, β, and c, where α is the discriminator loss weight, β is the L2 loss weight, and α+β=1, and c is the training gradient threshold; In the real environment, randomly sample action sequences U of length N for execution, obtain the state S in the real environment corresponding to each action sequence, record the actions and states, and form a data set P = {(S i ,U i ),…}, where the state in the real environment corresponding to the action sequence is the target state, which is the goal that the generative model needs to achieve in predicting the output; Perform multiple rounds of adversarial training on the generative model and the discriminator until convergence. In each round of training, the discriminator is trained multiple times before the generative model is trained once. When training the discriminator, it includes: Sample m state-action pairs from the dataset P S i is the state sequence, U i is the action sequence, and the state sequence includes the current state and a set of future target states Will After copying N times, it is superimposed into vector S base ; Using generative models to predict future state sequences in, Indicates that the generated model is based on the current state And the action sequence U i The output is a prediction sequence of differences between the future state and the current state. Each difference in the prediction sequence plus the current state constitutes a predicted future state sequence. Computing Gradients in, is the gradient of the discriminator, D ω (s t ,U i ,S i ′) represents the current state s t and action sequence U i The state S predicted by the generative model i ′ is the probability of coming from the real environment, D ω (s t ,U i ,S i ) represents the target state S i Probability from the real environment; Backward pass after gradient truncation Among them, ω D Represents the parameters of the discriminator, and clip represents gradient truncation; When training the generator, it includes: Sample m state-action pairs from the dataset P Will After copying N times, it is superimposed into vector S base ; Predicting future state sequences in, Indicates that the generated model is based on the current state And the action sequence U i The output is a prediction sequence of differences between the future state and the current state. Each difference in the prediction sequence plus the current state constitutes a predicted future state sequence. Computing Gradients in, is the gradient of the generative model, D ω (s t ,U i ,S i ′) represents the current state s t and action sequence U i The state S predicted by the generative model i ′ is the probability from the real environment, represents the discrimination loss, represents the L2 loss of the generative model, α is the discriminator loss weight, β is the L2 loss weight, and α+β=1; Gradient Backpropagation Among them, θ G are the parameters of the generative model and are updated based on gradient backpropagation.

2. The method according to claim 1, characterized in that: The generation model and the discriminator both include a 4-layer fully connected network.

3. The method according to claim 2, characterized in that In step S3, the generation model is trained using the discriminator loss and the L2 loss in the following manner: Where α is the discriminator loss weight, β is the L2 loss weight, and α+β=1, m is the batch size of the sampled control action sequence in each training, and s t is the current state of the target system during the current training, U i The i-th control action sequence, S i ′ represents the target system's future state prediction sequence corresponding to the target system executing the i-th control action sequence in the current state, D ω (s t ,U i ,S i ′) represents the current state s t and action sequence U i The state S predicted by the generative model i ′ is the probability from the real environment.

4. An MPC control method, applied to a target system of automatic control, for selecting a control action sequence to be executed by the target system at multiple moments in the future according to the current state of the target system and the target state sequence of the target system, characterized in that: The method comprises: T1, sampling a control action sequence of a preset number of predicted time domain lengths corresponding to the target system, wherein the control action at each moment in the control action sequence is sampled from all possible feasible control actions under the constraints of the target system; T2, using a generative model trained by the method described in any one of claims 1 to 3 to predict the future state sequence of the target system after executing the control action sequence based on each sampled control action sequence and the current state of the target system to obtain a target system future state prediction sequence of a preset number of prediction time domain lengths; T3. Select the control action sequence corresponding to the target system future state prediction sequence that is closest to the target state sequence of the target system from the control action sequences corresponding to all sampled target systems.

5. The method according to claim 4, characterized in that In step T1, the CEM strategy is adopted for sampling.

6. The method according to claim 4, characterized in that In step T3, a control action sequence corresponding to the target system future state prediction sequence closest to the target state sequence of the target system is selected from the control action sequences corresponding to all sampled target systems, and the first two control actions in the control action sequence are selected.

7. The method according to claim 4, characterized in that The step T3 comprises: T31, by calculating the loss between the target system future state prediction sequence corresponding to each sampled control action sequence and the target state sequence of the target system; T32. Select the control action sequence with the smallest loss between the corresponding future state prediction sequence of the target system and the target state sequence of the target system.

8. The method according to claim 7, characterized in that In step T31, the loss between the target system future state prediction sequence and the target state sequence corresponding to each control action sequence is calculated in the following manner: Among them, L i represents the loss between the target system future state prediction sequence corresponding to the i-th control action sequence and the target state sequence of the target system, S star represents the target state of the target system, S i ′ It represents the target system future state prediction sequence corresponding to the target system executing the i-th control action sequence, U i represents the i-th control action sequence, Q represents the weight of the state L2 loss, R represents the weight of the control action penalty term, (·) T Represents the transpose of a vector.

9. The method according to claim 8, characterized in that The target system future state prediction sequence with the corresponding prediction time domain length for each control action sequence is the vector sum of the difference sequence of the vector composed of a preset number of target system current states and the output of the pre-trained system model.

10. An MPC control device, which is configured in an automatically controlled target system and is used to select a control action sequence to be executed by the target system at multiple moments in the future according to the current state of the target system and the target state sequence of the target system, characterized in that: The device comprises: A sampler, used for sampling a plurality of control action sequences of predicted time domain lengths corresponding to the target system, wherein the control action at each moment in the control action sequence is sampled from all possible feasible control actions under the constraint conditions of the target system; A generative model constructed by the method as described in any one of claims 1 to 3 is used to predict the future state sequence of the target system after executing the control action sequence based on each control action sequence sampled by the sampler and the current state of the target system to obtain a future state prediction sequence of the target system with a preset number of prediction time domain lengths; The optimizer is used to select, from all sampled control action sequences, a control action sequence corresponding to a target system future state prediction sequence that is closest to a target system target state sequence.

11. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of any method described in claims 1-3, 4-9.

12. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the electronic device to implement the steps of the method as described in any one of claims 1-3, 4-9.

Images