Robot reinforcement learning safety controller online updating method based on deep Koopman

Abstract

The invention relates to the technical field of safety reinforcement learning, and discloses a deep Koopman-based robot reinforcement learning safety controller online updating method, which comprises the steps of collecting a trajectory state of random input control in simulation, training a deep Koopman neural network, and obtaining a corresponding lifting function and an evolution matrix; carrying out dimension reduction processing on the model by adopting an intrinsic orthogonal decomposition method to obtain a projection matrix and a new nominal model; carrying out reinforcement learning strategy migration in the real machine, obtaining an observation error according to the nominal model and the current observation state in interaction, and training an online update network to obtain a residual matrix; the nominal model and the residual model are combined to serve as model constraints of model prediction control, and safety control input is obtained. According to the method, the safety controller can be updated online, the reinforcement learning safety guarantee performance is improved, and the method is suitable for complex situations that model differences exist in the process of migrating a reinforcement learning strategy from simulation to a real machine, and disturbed dynamic environments exist in the physical world and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of safety reinforcement learning, and particularly relates to an online update method for a safety controller of robot reinforcement learning based on deep Koopman. Background Art

[0002] In recent years, reinforcement learning has achieved remarkable results in the control of robot systems with its unique interaction-learning-exploration mechanism. However, its inherent trial-and-error mechanism requires more attention to safety in the physical world. Existing methods reduce safety problems during the training process through the transfer from simulation to real machine. However, due to the non-negligible model differences between simulation and real machine, safety problems are still inevitable during the policy transfer process. Therefore, the research on the safety guarantee mechanism of robots in the context of reinforcement learning is a current research focus.

[0003] The reinforcement learning safety guarantee method based on safety region restriction is a typical one in safety reinforcement learning. Obtaining a more effective safety region is the core of this method, and the range of the safety region is closely related to the performance of the safety controller, which has an important impact on the exploration of robot reinforcement learning policies and safety guarantee performance. Currently, various designs of safety controllers have also been proposed in many literatures in the context of safety reinforcement learning, including methods such as PD controllers, Lyapunov function controllers, and pre-trained reinforcement learning policies. The process of transferring the reinforcement learning policy from simulation to real machine needs to face problems caused by model differences. These fixed safety controllers are also difficult to achieve the expected effect due to model differences and dynamic environments with disturbances, and cannot guarantee the safety performance of robots.

[0004] Updating the safety controller online according to the exploration process of reinforcement learning can, on the one hand, effectively solve the above problems, and on the other hand, can dynamically expand the safety region to make the exploration of robot reinforcement learning task policies more sufficient. Currently, the online update methods for controllers can be divided into two categories. One is to directly update the controller according to online data represented by reinforcement learning. However, as mentioned above, the trial-and-error mechanism is contradictory to safety guarantee. The other is to update the model according to online data and then design the controller based on this. However, the complex nonlinear characteristics of robots lead to a high computational burden in the solution method of nonlinear controllers. The Koopman operator shows great potential in alleviating the difficulties of nonlinear control under unknown dynamics. It maps the nonlinear dynamics into a high-dimensional linear space, and a linear controller can be designed based on this. Existing research online incrementally updates the dynamics model through the iterative least squares method to achieve online controller update. However, there are still problems in determining the appropriate dimension of the high-dimensional linear lifting space and the computational efficiency of controller update in the high-dimensional space, and it has not been well studied in the field of safety reinforcement learning for complex robot systems. Summary of the Invention

[0005] To solve the above technical problems, the present invention provides an online update method for a safety controller of robot reinforcement learning based on deep Koopman, which is applied to the safety guarantee in the process of robot reinforcement learning policy transfer. The model dimensionality reduction method proposed by the present invention avoids subjectively selecting the dimensionality of the high-dimensional lifting space, and at the same time can improve the computational efficiency of the online update of the safety controller, and is applicable to various general robots. The present invention designs the deep Koopman loss function adopted, considering more model matching factors, which can effectively improve the accuracy of learning and the effect of model prediction. The online update safety controller method proposed by the present invention can be applied to complex scenarios such as the existence of model differences in the process of transferring the reinforcement learning policy from simulation to real machine and the dynamic environment with disturbances in the physical world, which is of great significance for improving the safety performance of robot reinforcement learning, and significantly improves the applicable scenarios and practicality of safety reinforcement learning.

[0006] To solve the above technical problems, the present invention adopts the following technical solutions:

[0007] An online update method for a safety controller of robot reinforcement learning based on deep Koopman, comprising:

[0008] Randomly generate a control input sequence of the robot in the simulation environment, and collect the corresponding state set of the robot;

[0009] Input the control input sequence and the state set of the robot into the deep Koopman network for training, optimize the parameters of the deep Koopman network through a loss function including reconstruction error, prediction error and lifted state error, and obtain a lifting function from the state space where the robot state is located to the linear lifting space and the Koopman operator space evolution matrix;

[0010] Map the state of the robot to the linear lifting space based on the lifting function to form a data matrix, and obtain a projection matrix and a reduced-dimensional linear evolution equation matrix through dimensionality reduction processing;

[0011] Construct a linear model with the linear evolution equation matrix as a constraint, predict the control optimization problem, adjust the parameters through simulation, and design a safety controller with the safety region equilibrium point as the target;

[0012] Transfer the reinforcement learning policy in the offline simulation environment to the actual robot as the initial policy, and deploy the safety controller; obtain the control input at the current time step through the reinforcement learning policy, and the robot executes the control input and interacts with the environment;

[0013] When the safety controller determines that the current robot state is safe, the reinforcement learning policy is updated: the state of the robot is collected in real time, the observed state of the robot is calculated through the lifting function and the projection matrix, the state error is predicted based on the nominal model composed of the linear evolution equation matrix, and the residual matrix is obtained and the online nominal model is updated.

[0014] When the safety controller determines that the current robot state is unsafe, the safety controller is activated to solve the control input until a safe state is returned, and then the reinforcement learning policy is updated.

[0015] In one embodiment, randomly generating a control input sequence of the robot in the simulation environment and collecting the corresponding state set of the robot specifically includes:

[0016] Randomly generate a control input sequence to form a control input sequence set U = {U 1 , U 2 , …, U i , …, U n}, where U i is the i-th control input sequence of the robot, and U i = {u 1 , u 2 , …, u k , …, u T}, and u k is the k-th control input in U i .

[0017] Collect a set X = {X 1 , X 2 , …, X i , …, X n} of n trajectories of the robot, where X i is the set of robot states in the i-th trajectory, and X i = {x 1 , x 2 , …, x k , …, x T}, and x k is the k-th robot state in X i .

[0018] In one embodiment, obtaining the lifting function from the state space where the robot state is located to the linear lifting space and the evolution matrix of the Koopman operator space specifically includes:

[0019] The state of the robot in the linear lifting space is defined as where x k is the k-th robot state, and the lifting function f NIt consists of a fully connected deep neural network. I represents the identity matrix, and the dimension of I is the same as that of x k The state estimate of the original state space reconstructed based on the linear lifting space is where the evolution matrix C = [I 0]; the evolution equation of the Koopman operator space can be expressed as:

[0020]

[0021] represents the prediction of the state at time k + 1 through the linear evolution equation in the high-dimensional linear space; the evolution matrices A and B are the weights of the deep neural network, and the fully connected neurons with linear activation functions are used to simulate the linear Koopman dynamics;

[0022] After the training is completed, the deep Koopman network will obtain the lifting function as well as the evolution matrices A, B, and C of the Koopman operator space.

[0023] In one embodiment, the parameters of the deep Koopman network are optimized by a loss function that includes the reconstruction error, prediction error, and lifted state error, specifically including:

[0024] The loss function loss of the deep Koopman network is as follows:

[0025] loss = α 1 L rec + α 2 L pred + α 3 L lift + β 1 ‖W‖ 1 + β 2 ‖W‖ 2 ;

[0026] where α 1 , α 2 , α 3 are weight coefficients, ‖·‖ 1 and ‖·‖ 2 are the l 1 norm and l 2 norm of the matrix respectively, β 1 , β 2 are regularization coefficients;

[0027] Reconstruction error Prediction error Lifted state error

[0028] Among them, S represents the predicted time step, and τ is the decay factor, which is used to weight the prediction error and the improved state error.

[0029] In one embodiment, the deep Koopman network adopts an autoencoder structure.

[0030] In one embodiment, the mapping of the state of the robot to the linear lifting space by the lifting function forms a data matrix, specifically including:

[0031] According to the lifting function and the control input sequence set U and the trajectory set X, the data matrix Z = [z 1 , z 2 , …, z p of the lifting space is obtained, where is the corresponding state of the robot state x k in the lifting space, and p is the length of the data matrix.

[0032] In one embodiment, the obtaining of the projection matrix and the matrix of the reduced-dimensional linear evolution equation through dimensionality reduction processing specifically includes:

[0033] Mapping the data matrix to the reduced-dimensional space through the projection matrix P:

[0034] y k = P T z k ;

[0035] z k is the corresponding state of the robot state x k in the lifting space, y k is the state after the high-dimensional linear space state z k is projected into the reduced-dimensional space;

[0036] Remapping from the reduced-dimensional space back to the original space:

[0037]

[0038] is the state remapped from the reduced-dimensional space state y k back to the high-dimensional linear space;

[0039] Based on this, the matrix corresponding to the reduced-dimensional linear evolution equation is obtained The reduced-dimensional linear evolution equation is:

[0040]

[0041] In one of the embodiments, a linear model is constructed with the linear evolution equation matrix as a constraint to predict the control optimization problem. The parameters are adjusted through simulation, and a safety controller with the safety region equilibrium point as the target is designed, specifically including:

[0042] The control optimization problem is:

[0043]

[0044] where M is the length of the prediction window, is the desired state; x - , x + and u - , u + are the upper and lower bounds of the state constraint and the control constraint respectively; Q and R are cost matrices used to adjust the effect of the linear model predictive control.

[0045] In one of the embodiments, the state of the robot is collected in real time, the observed state of the robot is calculated by the lifting function and the projection matrix, and the state error is predicted based on the nominal model composed of the linear evolution equation matrix to obtain the residual matrix and update the nominal model, specifically including:

[0046] The state x k of the current robot is sampled in real time, and the observed state is obtained through the lifting function and the projection matrix P; The estimated state u k corresponding to x k is calculated according to the current nominal model;

[0047] The observed state error Δy is obtained through k , input into the online update network, and trained according to the designed online update network loss to obtain the residual matrices ΔA and ΔB, and update the nominal model:

[0048]

[0049] In one of the embodiments, the online update network loss is L adapt :

[0050] L adapt = L ε + γ 1 ‖ΔA‖ 1 + γ 2 ‖ΔB‖ 1 + γ 3 ‖ΔA‖ 2 + γ 4 ‖ΔB‖ 2

[0051] Among them, γ 1 , γ 2 , γ 3 , γ 4 is a regularization parameter; the loss term L ε quantifies the influence of the observation error:

[0052]

[0053] Δy k represents the observation state error, which is obtained from obtained.

[0054] Compared with the prior art, the beneficial technical effects of the present invention are:

[0055] 1. The present invention proposes an online update method for a robot safety controller based on deep Koopman, which can be applied to complex scenarios such as the existence of model differences during the transfer of the reinforcement learning policy from simulation to the real machine and the dynamic environment with disturbances in the physical world. It is of great significance for improving the safety performance of robot reinforcement learning and significantly enhances the applicable scenarios and practicality of safety reinforcement learning.

[0056] 2. The present invention proposes a model dimensionality reduction method, which avoids subjectively selecting the dimensionality of the high-dimensional promotion space. At the same time, it can improve the online update calculation efficiency of the safety controller, perform safety control more efficiently, and is applicable to various general robots.

[0057] 3. The present invention proposes a new deep Koopman loss function, which considers more model matching factors and can effectively improve the accuracy of learning and the effect of model prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is the flowchart of the method in the embodiment of the present invention;

[0059] Figure 2 is the schematic diagram of the offline training process of the robot safety controller based on deep Koopman in the embodiment of the present invention;

[0060] Figure 3 is the flowchart of the online update method of the robot safety controller based on deep Koopman in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0061] The following is a detailed description of a preferred embodiment of the present invention with reference to the accompanying drawings.

[0062] The present invention improves the security during the transfer of the reinforcement learning strategy by online updating the robot safety controller of the deep Koopman network, and further improves the reinforcement learning security guarantee mechanism based on the safety region restriction. The model dimensionality reduction method in the present invention avoids the difficulty of subjectively selecting the dimensionality of the high-dimensional lifting space, and at the same time can improve the online update calculation efficiency of the safety controller.

[0063] As Figure 1 shown, an online update method for a robot reinforcement learning safety controller based on deep Koopman in the present invention includes the following steps:

[0064] S1: Randomly generate a control input sequence of the robot in the simulation environment, and collect the corresponding state set of the robot.

[0065] S2: Input the control input sequence and the state of the robot into the deep Koopman network for training, optimize the parameters of the deep Koopman network through a loss function including reconstruction error, prediction error, and lifted state error, and obtain a lifting function from the state space where the robot state is located to the linear lifting space and the Koopman operator space evolution matrix.

[0066] S3: Map the state data to the linear lifting space based on the lifting function to form a data matrix, and obtain a projection matrix and a reduced-dimensional linear evolution equation matrix through dimensionality reduction processing.

[0067] S4: Construct a linear model with the linear evolution equation matrix as a constraint, predict the control optimization problem, adjust the parameters through simulation, and design a safety controller with the safety region equilibrium point as the target.

[0068] S5: Transfer the reinforcement learning strategy in the offline simulation environment to the actual robot as the initial strategy, and deploy the safety controller; obtain the control input at the current time step through the reinforcement learning strategy, and the robot executes the control input and interacts with the environment.

[0069] S6: When the safety controller determines that the current robot state is safe, update the reinforcement learning strategy: collect the state of the robot in real time, calculate the observed state of the robot through the lifting function and the projection matrix, predict the state error based on the nominal model composed of the linear evolution equation matrix, obtain the residual matrix, and update the nominal model.

[0070] S7: When the safety controller determines that the current robot state is unsafe, start the safety controller to solve the control input until it returns to a safe state, and then update the reinforcement learning strategy.

[0071] In one embodiment, randomly generating a control input sequence of the robot in the simulation environment and collecting the corresponding state set of the robot in step S1 specifically includes:

[0072] Randomly generate a control input sequence to form a set of control input sequences \(U = \{U 1 , U 2 , \ldots, U i , \ldots, U n \}\), where \(U i \) is the \(u\)-th control input sequence of the robot, and \(U i =\{i 1 , i 2 , \ldots, i k , \ldots, i T \}\), and \(i k \) is the \(k\)-th control input in \(U i \).

[0073] Collect a set of \(n\) trajectories of the robot \(X = \{X 1 , X 2 , \ldots, X i , \ldots, X n \}\), where \(X i \) is the set of robot states in the \(i\)-th trajectory, and \(X i =\{x 1 , x 2 , \ldots, x k , \ldots, x T \}\), and \(x k \) is the \(k\)-th robot state in \(X i \), and \(T\) is the length of the trajectory.

[0074] In one embodiment, obtaining the lifting function from the state space where the robot state is located to the linear lifting space and the evolution matrix of the Koopman operator space in step S2 specifically includes:[[]]

[0075] The state of the robot in the linear lifting space is defined as where \(x k \) is the \(k\)-th robot state, and the lifting function f N is composed of a fully connected deep neural network; the state estimate of the original state space reconstructed based on the linear lifting space is where the evolution matrix \(C = [I\ 0]\); the evolution equation of the Koopman operator space can be expressed as:

[0076]

[0077] The evolution matrices \(A, B\) are the weights of the deep neural network, and a fully connected neuron with a linear activation function is used to simulate the linear Koopman dynamics;

[0078] After the training is completed, the deep Koopman network will simultaneously obtain the lifting function and the evolution matrices A, B, and C of the Koopman operator space.

[0079] In one embodiment, optimizing the parameters of the deep Koopman network through a loss function that includes reconstruction error, prediction error, and lifted state error in step S2 specifically includes:

[0080] The loss function loss of the deep Koopman network is as follows:

[0081] loss = α 1 L rec + α 2 L pred + α 3 L lift + β 1 ‖W‖ 1 + β 2 ‖W‖ 2 ;

[0082] where α 1 , α 2 , α 3 are weight coefficients, ‖·‖ 1 and ‖·‖ 2 are the l 1 norm and l 2 norm of the matrix respectively, β 1 , β 2 are regularization coefficients;

[0083] Reconstruction error Prediction error Lifted state error

[0084] where S represents the predicted time step, and τ is the decay factor used to weight the prediction error and the lifted state error.

[0085] Specifically, the deep Koopman network adopts an autoencoder structure.

[0086] Since the dimension of the linear space of the Koopman operator is infinite, but in practical applications, it can only be truncated to a finite dimension, which is closely related to the accuracy and computational speed of the model. The higher the dimension, the more accurate the model description, but the corresponding computational speed will be slower. Selecting an appropriate dimension for the linear lifting space is also a challenge. To reduce the difficulty in selecting the output dimension of the lifting function, in the offline training part, the dimension is uniformly lifted to a sufficiently high dimension. Since this operation is completed offline, the requirements for computational time are not strict. Subsequent operations need to be performed in real time, and dimensionality reduction will be adopted to reduce the computational time cost.

[0087] In one embodiment, mapping the state data to a linear lifting space based on a lifting function in step S3 to form a data matrix specifically includes:

[0088] According to the lifting function and the control input sequence set U and the trajectory set X, the data matrix Z = [z 1 , z 2 , …, z p of the lifting space is obtained, where is the corresponding state of x k in the lifting space, and p is the length of the data matrix.

[0089] In one embodiment, obtaining a projection matrix and a reduced-dimensional linear evolution equation matrix through dimensionality reduction processing in step S3 specifically includes:

[0090] Mapping the data matrix to a reduced-dimensional space through the projection matrix P: y k = P T z k ;

[0091] Remapping from the reduced-dimensional space back to the original space:

[0092] Based on this, the matrix corresponding to the reduced-dimensional linear evolution equation is obtained The reduced-dimensional linear evolution equation is:

[0093]

[0094] Specifically, since the output layer dimension of the deep neural network is very high, there is a problem of too high time cost in real-time calculation. To solve this difficulty, balance the calculation time and the model accuracy, and improve the calculation efficiency, a dimensionality reduction method is adopted to simplify the model in this step.

[0095] In one embodiment, constructing a linear model with the linear evolution equation matrix as a constraint, predicting a control optimization problem, adjusting parameters through simulation, and designing a safety controller with the safety region equilibrium point as the target in step S4 specifically includes:

[0096] The control optimization problem is:

[0097]

[0098] where M is the length of the prediction window, is the desired state; x - , x + and u - , u + are the upper and lower bounds of the state constraint and the control constraint respectively; Q, R are cost matrices used to adjust the effect of the linear model predictive control.

[0099] Specifically, for the design of a safety controller for a linear model, model predictive control is adopted, which can be described as a constrained optimization problem solved online. According to the definition of the safety region, this optimization problem takes the locally asymptotically stable equilibrium point x * as the target point, and uses the linear model after dimensionality reduction processing, the control input range, and the state range as constraint conditions. By solving the optimization problem, the control input is obtained to achieve safety correction. To adjust the predictive control parameters of this optimization model, the initial state can be randomly given multiple times for solution, and the solved control input is applied to the simulation robot system, and the relevant parameters are optimized according to its correction performance. After solving the optimal U sequence, only the first control input is input into the robot system.

[0100] In one embodiment, the state of the robot is collected in real time in step S6, the observed state of the robot is calculated by the lifting function and the projection matrix, and the nominal model prediction state error formed by the linear evolution equation matrix is used to obtain the residual matrix and update the nominal model, specifically including:

[0101] The state x of the current robot is sampled in real time k , and the observed state is obtained through the lifting function and the projection matrix P. According to the current nominal model , the estimated state k corresponding to x is calculated as the control input;

[0102] Through , the observed state error Δy k is obtained and input into the online update network. According to the designed online update network loss, training is performed to obtain the residual matrices ΔA and ΔB, and update the nominal model:

[0103]

[0104] In one embodiment, the online update network loss is L adapt :

[0105] L adapt = L ε + γ 1 ‖ΔA‖ 1 + γ 2 ‖ΔB‖ 1 + γ 3 ‖ΔA‖ 2 + γ 4 ‖ΔB‖ 2 ;

[0106] Among them, γ1 , γ 2 , γ 3 , γ 4 is the regularization parameter; the loss term L ε quantifies the impact of the observation error:

[0107]

[0108] The online update network corrects the deviation between the nominal model obtained offline and the real system by improving the Koopman model online in real time. During online update, only networks A and B are updated, and the boosting function is not updated The evolved matrix network structure for this online update is the same as that in step S3, and is composed of a linear activation function and a fully connected neural network, and the residual matrices ΔA and ΔB are learned

[0109] For the offline training process of the safety controller of the present invention, refer to Figure 2 ; for the online update method of the safety controller of the present invention, refer to Figure 3 .

[0110] In one of the embodiments, a drone is selected as the robotic system in the example of the present invention. The safety region based on the region of attraction (ROA) is defined as: D = {x 0 ∈ X|lim t→∞ Φ(t; x 0 ) = 0}, where X represents the state space of the original drone, x 0 is the current state of the drone, and Φ(t; x 0 ) represents the motion trajectory of the drone starting from the current state x safe under the given safety control π 0 . If the given safety control π safe (x) can balance the state, then the current state is within the safety region. The present invention will focus on discussing the design method of the safety controller to make it applicable to complex scenarios such as the model differences during the transfer of the reinforcement learning strategy from simulation to the real machine and the dynamic environment with disturbances in the physical world. When the current state of the drone is within the safety region D, the current state is considered safe, and the reinforcement learning control strategy π RL (x) is continued to be trained. Otherwise, the current state is considered dangerous, and the safety controller π safe (x) is adopted to control the current state back inside the safety region. The safety reinforcement learning framework is as follows:

[0111]

[0112] where u represents the control strategy, p represents the safety probability threshold, and P(x ∈ D) represents the safety probability of the current state x

[0113] To obtain the initial safety controller, in the simulation, a set of control input sequence sets U = {U 1 , U 2 , …, U i , …, U n} are randomly generated and input into the UAV system in sequence to collect UAV data. A set of n trajectories X = {X 1 , X 2 , …, X i , …, X n} is collected, where U i is the i-th control input sequence, and U i = {u 1 , u 2 , …, u k , …, u T}. X i is the state set of the i-th trajectory, and X i = {x 1 , x 2 , …, x k , …, x T}. Here, in the present invention, n = 125 and T = 600 are specifically set, and the trajectory sampling interval is 0.01 s. The control input sequence set U and the trajectory set X are divided into a training set and a test set according to a ratio of 0.8:0.2 and input into the constructed deep Koopman network.

[0114] The deep Koopman network adopts an autoencoder structure, and the specific network parameters are shown in Table 1. The evolution equation of the high-dimensional linear Koopman operator space can be expressed as:

[0115]

[0116] where f N consists of a fully connected deep neural network, where C = [I 0]. Note that the evolution matrices A and B are the weights of the neural network, and the linear Koopman dynamics are simulated using fully connected neurons with linear activation functions. The deep Koopman loss function is designed as follows:

[0117] loss = α 1 L rec + α 2 L pred + α 3 L lift + β 1 ‖W‖ 1 + β 2 ‖W‖ 2

[0118] Among them, α 1 , α 2 , α 3 are weight coefficients, ‖·‖ 1 and ‖·‖ 2 are the l 1 -norm and l 2 -norm of the matrix respectively, β 1 , β 2 are its regularization coefficients, and the corresponding parts are respectively:

[0119]

[0120] Here, S = 15 represents the predicted time step, τ = 0.9 is the decay factor, which is used to weight the prediction error and the error of the lifting state. When the training of the deep Koopman network is completed, the lifting function as well as the evolution matrices A, B, C of the Koopman operator space of the UAV will be obtained simultaneously. The specific parameter settings are shown in Table 1.

[0121] Table 1 Koopman autoencoder network

[0122]

[0123] According to the obtained lifting function as well as the control input sequence set U and the trajectory set X, the data matrix Z = [z 1 , z 2 , …, z p of the lifting space is obtained. The slices in the UAV state evolution process in Z are temporally correlated. Here, p = 1000 is selected as the length of the data matrix. Perform model reduction processing of proper orthogonal decomposition (POD) on Z, and determine the number of main modes to be 18 according to the accuracy requirements. Map the original system to the reduced-order system through the projection matrix P ∈ R 40×18 :

[0124] y k = P T z k ;

[0125] Remap the reduced-order system back to the original system:

[0126]

[0127] Based on this, the matrix corresponding to the lower-dimensional linear evolution equation is obtained

[0128] Therefore, the lower-dimensional linear evolution equation is expressed as:

[0129]

[0130] Combined with A, B, C and the lifting function Finally, a linear evolution equation with a lower dimension is obtained After obtaining the above model, parameter tuning of model predictive control is carried out in the simulation. Given the initial state randomly, the constrained optimization problem solved online is as follows:

[0131]

[0132]

[0133] Among them, the specific parameters are shown in Table 2

[0134] Table 2 Model Predictive Control

[0135]

[0136] Using this model and control method as the UAV safety controller, and starting the UAV reinforcement learning strategy learning. The focus of the present invention is not on the reinforcement learning strategy training, so only the trajectory tracking task training is simply carried out. The safety controller is used for safety estimation. Here, the safety estimation method can be selected arbitrarily. The present invention uses the simplest grid method for estimation. Since it is not the focus, it will not be elaborated too much. When the reinforcement learning strategy training is completed, the strategy transfer is started

[0137] Reinforcement learning strategy π RL (x) is transferred to the actual UAV, and the safety controller is deployed. Using the model obtained in the simulation as the nominal model. While interacting with the environment, the actual state of the current UAV is sampled in real time Through and the projection matrix P to obtain the observed state According to the nominal model, the estimated state is calculated Through The observed state error Δy is obtained k , and input into the online update network

[0138] The update network ΔA, ΔB, which has the same structure as A, B, C, is composed of a linear activation function and a fully connected neural network. The corresponding loss function loss is designed as:

[0139] L adapt = L ε + γ 1 ‖ΔA‖ 1 + γ 2 ‖ΔB‖ 1 + γ 3 ‖ΔA‖ 2 + γ4 ‖ΔB‖ 2 ;

[0140] where γ 1 , γ 2 , γ 3 , γ 4 are regularization parameters, which are generally set to be larger than β 1 , β 2 . For specific value settings, please refer to

[0141] Table 3. L ε quantifies the impact of observation errors and is specifically described as follows:

[0142]

[0143] Train according to the designed loss to obtain the online updated model residual matrices ΔA and ΔB, update the nominal model, and let the new nominal model be

[0144] Table 3 Online Update of Network Parameters

[0145]

[0146] In each interaction, use the safety controller to judge the current state. When it is safe, continue with the reinforcement learning policy transfer to explore the UAV trajectory tracking strategy. When the safety controller judges that the current state is unsafe, according to the current corrected UAV nominal model and adopt model predictive control to solve for the control input u k , which acts on the UAV system to achieve safe control. At the same time, continuously perform online updates on the current nominal model so that the nominal model gradually approaches the true model. Through the adjustment of the model, continuously optimize the safety controller so that it can adapt to the dynamic environment with model differences and disturbances in the physical world, realize the continuous expansion of the safe area, and increase the scope of exploration of the reinforcement learning strategy.

[0147] It should be understood that although the steps in the flowchart of the accompanying drawings of the specification are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings of the specification may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same moment, but can be executed at different moments, and the execution order of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least a part of other steps or steps or stages in other steps.

[0148] It will be apparent to those skilled in the art that the present invention is not limited to the details of the above-described exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, in all respects, the embodiments should be considered exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description. Accordingly, all changes that fall within the meaning and scope of the equivalent elements of the claims are intended to be embraced within the present invention, and any reference signs in the claims should not be construed as limiting the claims involved.

[0149] In addition, it should be understood that although this specification is described in terms of embodiments, not every embodiment only contains an independent technical solution. This narrative manner of the specification is only for clarity. Those skilled in the art should regard the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for online updating of robot reinforcement learning safety controller based on deep Koopman, characterized in that: include: Randomly generate a control input sequence for the robot in a simulation environment and collect the corresponding state set of the robot; The control input sequence and the robot's state set are input into the deep Koopman network for training. The parameters of the deep Koopman network are optimized through a loss function including reconstruction error, prediction error and boosted state error, and the boosting function from the state space where the robot state is located to the linear boosting space and the Koopman operator space evolution matrix are obtained. Based on the lifting function, the robot's state is mapped to the linear lifting space to form a data matrix, and the projection matrix and the linear evolution equation matrix after dimensionality reduction are obtained through dimensionality reduction processing; Construct a linear model with the linear evolution equation matrix as a constraint, predict the control optimization problem, adjust the parameters through simulation, and design a safety controller with the balance point in the safety area as the target; Migrate the reinforcement learning strategy in the offline simulation environment to the actual robot as the initial strategy and deploy a safety controller; The control input of the current time step is obtained through the reinforcement learning strategy, and the robot executes the control input and interacts with the environment; When the safety controller determines that the current robot state is safe, the reinforcement learning strategy is updated: the robot state is collected in real time, the robot's observation state is calculated through the lifting function and the projection matrix, the state error is predicted based on the nominal model composed of the linear evolution equation matrix, the residual matrix is ​​obtained, and the online nominal model is updated; When the safety controller determines that the current robot state is unsafe, the safety controller is started to solve the control input until it returns to a safe state, and then the reinforcement learning strategy is updated.

2. According to claim 1, a method for online updating of a robot reinforcement learning safety controller based on deep Koopman is characterized in that: The randomly generating a control input sequence of the robot in the simulation environment and collecting a state set corresponding to the robot specifically includes: Randomly generate a control input sequence to form a control input sequence set U = {U1, U2, ..., U i ,…,U n },U i is the robot’s ith control input sequence, U i ={u1,u2,…,u k ,…,u T },u k For U i The kth control input in ; The set of n trajectories of the robot is X = {X1, X2, ..., X i ,…,X n }, X i is the set of robot states in the ith trajectory, X i ={x1,x2,…,x k ,…,x T }, x k For X i The kth robot state in .

3. According to claim 1, a method for online updating of a robot reinforcement learning safety controller based on deep Koopman, characterized in that: The step of obtaining the lifting function and the Koopman operator space evolution matrix from the state space where the robot state is located to the linear lifting space specifically includes: The state of the robot in the linear promotion space is defined as Among them, x k is the kth robot state, the improvement function f N It consists of a fully connected deep neural network, I represents the identity matrix, and the dimension of I is the same as x k The dimension is the same as that of ; the state estimate of the original state space reconstructed according to the linear lifting space is Among them, the evolution matrix C = [I0]; the evolution equation of the Koopman operator space can be expressed as: It represents the prediction of the state at time k+1 through the linear evolution equation in a high-dimensional linear space; the evolution matrices A and B are the weights of the deep neural network, and the fully connected neurons with linear activation functions are used to simulate the linear Koopman dynamics; After training, the deep Koopman network will also get a boost function And the evolution matrices A, B, C of the Koopman operator space.

4. According to claim 3, a method for online updating of a robot reinforcement learning safety controller based on deep Koopman is characterized in that: The optimization of the deep Koopman network parameters by a loss function including reconstruction error, prediction error and improved state error specifically includes: The loss function of the deep Koopman network is as follows: loss=α1L rec +α2L pred +α3L lift +β1‖W‖1+β2‖W‖2; Among them, α1, α2, α3 are weight coefficients, ‖·‖1 and ‖·‖2 are the l1 norm and l2 norm of the matrix respectively, β1, β2 are regularization coefficients; Reconstruction error Prediction Error Improved state error Among them, S represents the prediction time step, and τ is the decay factor used to weight the prediction error and the improved state error.

5. According to claim 1, a method for online updating of a robot reinforcement learning safety controller based on deep Koopman, characterized in that: The deep Koopman network adopts an autoencoder structure.

6. The method for online updating of a robot reinforcement learning safety controller based on deep Koopman according to claim 1, characterized in that: Mapping the robot state to the linear lifting space based on the lifting function to form a data matrix specifically includes: According to the lifting function As well as the control input sequence set U and trajectory set X, we get the data matrix Z = [z1,z2,…,z p ],in is the robot's state x k In the corresponding state of the lifting space, p is the length of the data matrix.

7. The method for online updating of a robot reinforcement learning safety controller based on deep Koopman according to claim 1, characterized in that: The projection matrix and the linear evolution equation matrix after dimensionality reduction are obtained by dimensionality reduction processing, specifically including: Map the data matrix into the reduced dimensional space through the projection matrix P: y k =P T z k ; z k is the robot's state x k In the corresponding state of the lifting space, y k is the high-dimensional linear space state z k The state projected into the reduced dimensionality space; Remap from the reduced dimension space back to the original space: is the state y from the reduced dimension space k Remap back to the state of the high-dimensional linear space; Based on this, the matrix corresponding to the linear evolution equation after dimensionality reduction is obtained The linear evolution equation after dimensionality reduction is:

8. The method for online updating of a robot reinforcement learning safety controller based on deep Koopman according to claim 1, characterized in that: The method of constructing a linear model with the linear evolution equation matrix as a constraint, predicting the control optimization problem, adjusting parameters through simulation, and designing a safety controller with the balance point of the safety area as the target specifically includes: The control optimization problem is: Where M is the length of the prediction window, is the desired state; x - ,x + and u - ,u + are the upper and lower bounds of the state constraints and control constraints respectively; Q, R are cost matrices used to adjust the effect of linear model predictive control.

9. The method for online updating of a robot reinforcement learning safety controller based on deep Koopman according to claim 1, characterized in that: The real-time acquisition of the robot state, the calculation of the robot's observed state through the lifting function and the projection matrix, the prediction of the state error based on the nominal model composed of the linear evolution equation matrix, the acquisition of the residual matrix and the update of the linear nominal model specifically include: Real-time sampling of the current robot state x k , by lifting the function And the projection matrix P to get the observation state According to the current nominal model Calculate the estimated state u k For x k The corresponding control input; pass Get the observed state error Δy k , input into the online update network, train according to the designed online update network loss, obtain the residual matrix ΔA, ΔB, and update the nominal model:

10. The method for online updating of a robot reinforcement learning safety controller based on deep Koopman according to claim 9, characterized in that: The online update network loss is L adapt : L adapt =L ε +γ1‖ΔA‖1+γ2‖ΔB‖1+γ3‖ΔA‖2+γ4‖ΔB‖2 Among them, γ1,γ2,γ3,γ4 are regularization parameters; the loss term L ε The effect of observation error is quantified: Δy k represents the observation state error, which is given by get.

Images