Robot safety training method, device and equipment based on incremental reinforcement learning

Abstract

The present application relates to a robot safety training method, device and equipment based on incremental reinforcement learning. The method comprises: using historical data to estimate the unknown model information in the nonlinear system of the robot, constructing an augmented incremental system based on the linear incremental system, and constructing a penalty function to characterize the robot's requirements for completing tasks, and obtaining a value function for evaluating the robot's performance according to the penalty function after the kth step; according to the value function, constructing a Q-function, and using a pre-constructed model to guide the incremental reinforcement learning algorithm to iteratively solve the Q-function online, and obtaining the optimal control strategy and the optimal error estimation strategy, as well as the robot model, and using the robot model learned online to perform forward prediction to promote reinforcement learning online training, and based on the robot model learned online, constructing a quadratic constraint optimization to fine-tune the robot's optimal control strategy, so as to achieve online safety learning. This method can achieve robot safety training.

Description

Technical Field

[0001] The present application relates to the field of robot control technology, and in particular to a robot safety training method, device and equipment based on incremental reinforcement learning. Background Art

[0002] With the continuous development of science and technology, robots are increasingly used in various fields. However, there are still many challenges in the control and learning process of robots. On the one hand, conventional reinforcement learning algorithms have the problem of low sample utilization. This makes it difficult to learn directly on robot hardware, because the cost of collecting data in actual operation of robots is high, and low sample utilization may lead to a long and inefficient learning process. In order to solve this problem, it is necessary to design an efficient learning algorithm that can make full use of historical data, improve sample utilization, reduce the amount of calculation, and drive nonlinear robots to complete the given tasks through online interactive learning of linear strategies. On the other hand, the safety of the robot body and the surrounding environment is crucial during operation. When training on robot hardware, a safe, stable and continuous training process must be ensured. At present, there is a lack of effective online safety learning strategies, and it is impossible to achieve stable training while ensuring the safety of the robot. Therefore, it is necessary to design an online safety learning strategy that can use incremental models to describe the robot's motion characteristics, solve the safety strategy online through constraint optimization, and ensure the safety of the robot body and the surrounding environment. Summary of the invention

[0003] Based on this, it is necessary to provide a robot safety training method, device and equipment based on incremental reinforcement learning to address the above technical problems.

[0004] A robot safety training method based on incremental reinforcement learning, the method comprising:

[0005] The unknown model information in the nonlinear system of the robot is estimated by using historical data, and the linear incremental system of the robot is constructed; the linear incremental system includes: a control input incremental term and a linearization error term;

[0006] An augmented incremental system is constructed based on the linear incremental system, and a penalty function is constructed to characterize the requirements for the robot to complete the task;

[0007] According to the penalty function after the kth step, the value function used to evaluate the robot performance is obtained;

[0008] According to the value function, a Q-function is constructed, and a pre-constructed model is used to guide an incremental reinforcement learning algorithm to iteratively solve the Q-function online to obtain an optimal control strategy and an optimal error estimation strategy, as well as a robot model, and the robot model learned online is used for forward prediction to obtain additional data to promote online reinforcement learning training;

[0009] During the online learning process, based on the robot model learned online, a quadratic constraint optimization is constructed to fine-tune the robot's optimal control strategy to achieve online safe learning.

[0010] In one of the embodiments, the nonlinear system of the robot is described as follows:

[0011] ;

[0012] in, , Respectively represent the state input and control input of the robot; , denote the state matrix and input matrix respectively;

[0013] Multiply both sides of the expression of the nonlinear system by the constant matrix B to obtain the expression for separating the unknown model information:

[0014] ;

[0015] in, Contains all unknown model information;

[0016] Use historical data to predict unknown model information The estimate is:

[0017] ;

[0018] Combining the expressions, the linear increment system is obtained as follows:

[0019] ;

[0020] in, represents the linearization error term, To control the input increment term, For the unit array.

[0021] In one embodiment, it further includes: constructing an augmented incremental system based on the linear incremental system as follows:

[0022] ;

[0023] in, .

[0024] In one embodiment, the control strategy corresponding to the control input increment term and the estimation strategy corresponding to the linearization error term are constructed respectively. and ;

[0025] The penalty function constructed according to the control strategy and the estimation strategy is:

[0026] ;

[0027] Among them, Q represents the state weight matrix, R represents the input weight matrix, Represents the coefficient corresponding to the estimation error strategy.

[0028] In one embodiment, it also includes:

[0029] According to the value function, the Q-function is constructed as:

[0030] ;

[0031] The value function can be expressed as:

[0032] ;

[0033] P is a symmetric positive definite matrix;

[0034] The Q-function can be further simplified as:

[0035] ;

[0036] To facilitate algorithm development, the Q-function is expressed as:

[0037] ;

[0038] in, ; ;

[0039] By solving, and The optimal strategy matrix and the optimal estimation error strategy matrix are obtained as follows:

[0040] ;

[0041] ;

[0042] according to , we can get the Bellman equation of the Q-function as:

[0043] ;

[0044] According to the formula , we can get:

[0045] ;

[0046] Will Simplified to:

[0047] ;

[0048] ;

[0049] ;

[0050] ;

[0051] Iterative learning through model-guided incremental reinforcement learning algorithm , in The iterative process is:

[0052] ;

[0053] ;

[0054] From the estimated Extract the model information of the robot model from the matrix:

[0055] ;

[0056] According to the model information, forward prediction is performed, and the data generated by the forward prediction is used to construct Matrix and Matrix,facilitates efficient online training of robot policies.

[0057] In one embodiment, the process further includes: during the online learning process, constructing a quadratic constraint optimization as:

[0058] ;

[0059] in, represents the input constraint set; represents the incremental input constraint set; is the control obstacle function that describes the edge of the obstacle.

[0060] A robot safety training device based on incremental reinforcement learning, the device comprising:

[0061] A linear system building module is used to estimate unknown model information in the nonlinear system of the robot using historical data to build a linear incremental system of the robot; the linear incremental system includes: a control input incremental term and a linearization error term;

[0062] A penalty function construction module, used to construct an augmented incremental system based on the linear incremental system, and construct a penalty function to characterize the requirements of the robot to complete the task;

[0063] A strategy solving module is used to obtain a value function for evaluating the robot performance according to the penalty function after the kth step; a Q-function is constructed according to the value function, and a pre-constructed model is used to guide the incremental reinforcement learning algorithm to iteratively solve the Q-function online to obtain an optimal control strategy and an optimal error estimation strategy, as well as a robot model, and the robot model learned online is used for forward prediction to obtain additional data to promote online reinforcement learning training;

[0064] The online safety learning module is used to construct a quadratic constraint optimization to fine-tune the optimal control strategy of the robot based on the robot model learned online during the online learning process, so as to achieve online safety learning.

[0065] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0066] The unknown model information in the nonlinear system of the robot is estimated by using historical data, and the linear incremental system of the robot is constructed; the linear incremental system includes: a control input incremental term and a linearization error term;

[0067] An augmented incremental system is constructed based on the linear incremental system, and a penalty function is constructed to characterize the requirements for the robot to complete the task;

[0068] According to the penalty function after the kth step, the value function used to evaluate the robot performance is obtained;

[0069] According to the value function, a Q-function is constructed, and a pre-constructed model is used to guide an incremental reinforcement learning algorithm to iteratively solve the Q-function online to obtain an optimal control strategy and an optimal error estimation strategy, as well as a robot model, and the robot model learned online is used for forward prediction to obtain additional data to promote online reinforcement learning training;

[0070] During the online learning process, based on the robot model learned online, a quadratic constraint optimization is constructed to fine-tune the robot's optimal control strategy to achieve online safe learning.

[0071] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0072] The unknown model information in the nonlinear system of the robot is estimated by using historical data, and the linear incremental system of the robot is constructed; the linear incremental system includes: a control input incremental term and a linearization error term;

[0073] An augmented incremental system is constructed based on the linear incremental system, and a penalty function is constructed to characterize the requirements for the robot to complete the task;

[0074] According to the penalty function after the kth step, the value function used to evaluate the robot performance is obtained;

[0075] According to the value function, a Q-function is constructed, and a pre-constructed model is used to guide an incremental reinforcement learning algorithm to iteratively solve the Q-function online to obtain an optimal control strategy and an optimal error estimation strategy, as well as a robot model, and the robot model learned online is used for forward prediction to obtain additional data to promote online reinforcement learning training;

[0076] During the online learning process, based on the robot model learned online, a quadratic constraint optimization is constructed to fine-tune the robot's optimal control strategy to achieve online safe learning.

[0077] The above-mentioned robot safety training method, device, computer equipment and storage medium based on incremental reinforcement learning use historical data to estimate the unknown model information in the robot nonlinear system, construct a linear incremental system, construct an augmented incremental system, and design an estimation strategy, a control strategy and a penalty function. On the one hand, an efficient incremental reinforcement learning algorithm is designed using historical data and a linear incremental system, and a linear strategy is driven by online interactive learning to complete a given task with low computational complexity, solving the problem of low sample utilization of conventional reinforcement learning algorithms and learning directly on robot hardware; on the other hand, the optimal estimation strategy matrix and the optimal control strategy matrix are obtained by constructing a value function and a Q-function, and a robot model is obtained, and the robot strategy is fine-tuned by constructing constraint optimization in the online learning process, realizing online safety learning, ensuring the safety of the robot body and the surrounding environment, and realizing stable and continuous training on the robot hardware. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 1 is a flow chart of a robot safety training method based on incremental reinforcement learning in one embodiment;

[0079] Figure 2 is a structural block diagram of a robot safety training device based on incremental reinforcement learning in one embodiment;

[0080] Figure 3 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0081] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0082] In one embodiment, Figure 1As shown, a robot safety training method based on incremental reinforcement learning is provided, comprising the following steps:

[0083] Step 102: Use historical data to estimate unknown model information in the nonlinear system of the robot, and construct a linear incremental system of the robot.

[0084] The linear incremental system includes: control input incremental term and linearization error term.

[0085] Historical data refers to various information recorded by the robot during its past operation. It may include state variables such as the robot's position, speed, acceleration, joint angle, torque, and control instructions applied to the robot, such as the motor's drive voltage and current. It can be obtained through the robot's own sensors and the recording function of the control system, such as position sensors, speed sensors, etc. In a specific embodiment, the operation record of an industrial robot arm on a production line in the past, including the joint angle changes and corresponding motor control signals each time it grabs an object, constitutes part of the historical data.

[0086] In order to complete the given task safely and autonomously, robots in the real world must ensure that the given task performance is achieved under the premise of satisfying the following state constraints and input constraints at any time:

[0087] ;

[0088] ;

[0089] in, , Respectively represent the state constraint set and the input constraint set; is the obstacle function, and its value greater than zero indicates that the robot is in a safe state; Represents input The jth component of represents the limit value of the jth input component. It is difficult for existing technologies to accurately model the motion characteristics of robots in the real world. The lack of accurate robot models poses a great challenge to designing strategies that drive robots to complete given tasks safely and autonomously.

[0090] In this embodiment, historical data is used to estimate the unknown model information, so that the estimation of the unknown model information is offset by solving the linear error in the subsequent linear incremental system.

[0091] Step 104: construct an augmented incremental system based on the linear incremental system, and construct a penalty function to characterize the requirements for the robot to complete the task.

[0092] In this embodiment, augmentation refers to expanding and improving the original linear incremental system, specifically considering the linearization error term as system interference. The augmented incremental system can more comprehensively reflect the characteristics of the actual robot system.

[0093] The penalty function evaluates the cost of the system in the control process by comprehensively considering the system state, control input, and the interference caused by the state.

[0094] Step 106, obtaining a value function for evaluating the performance of the robot according to the penalty function after the kth step.

[0095] The value function accumulates the cost of using the estimation strategy and the control strategy after k steps.

[0096] Step 108, construct a Q-function based on the value function, use the pre-constructed model to guide the incremental reinforcement learning algorithm to iteratively solve the Q-function online, obtain the optimal control strategy and the optimal error estimation strategy, as well as the robot model, and use the online learned robot model to perform forward prediction to obtain additional data to promote the online training of reinforcement learning.

[0097] Step 110, during the online learning process, based on the robot model learned online, a quadratic constraint optimization is constructed to fine-tune the optimal control strategy of the robot, so as to achieve online safety learning.

[0098] In the above-mentioned robot safety training method based on incremental reinforcement learning, the unknown model information in the robot nonlinear system is estimated by using historical data, a linear incremental system is constructed, the linearized error term is used as the system interference to construct an augmented incremental system, and the estimation strategy, control strategy and penalty function are designed. On the one hand, an efficient incremental reinforcement learning algorithm is designed by using historical data and a linear incremental system, and the linear strategy is driven by online interactive learning to complete the given task with low computational complexity, which solves the problem of low sample utilization of conventional reinforcement learning algorithms and direct learning on robot hardware; on the other hand, the optimal estimation strategy matrix and the optimal control strategy matrix are obtained by constructing a value function and a Q-function to obtain a robot safety motion model, and a safety constraint model is constructed in the online learning process to optimize the robot strategy, realize online safety learning, ensure the safety of the robot body and the surrounding environment, and realize stable and continuous training on the robot hardware.

[0099] In one embodiment, the nonlinear system of the robot is described as:

[0100] ;

[0101] in, , Respectively represent the state input and control input of the robot; , Represent the state matrix and input matrix respectively; multiply both sides of the expression of the nonlinear system by the constant matrix B, and the expression of independent unknown model information is obtained:

[0102] ;

[0103] in, Contains all unknown model information; uses historical data to predict unknown model information The estimate is:

[0104] ;

[0105] Combining the expressions, the linear increment system is obtained as follows:

[0106] ;

[0107] in, represents the linearization error term, To control the input increment term, For the unit array.

[0108] In another embodiment, the augmented incremental system is constructed based on the linear incremental system as follows:

[0109] ;

[0110] in, .

[0111] In one embodiment, the control strategy corresponding to the control input increment term and the estimation strategy corresponding to the linearization error term are constructed respectively. and ; The penalty function is constructed according to the control strategy and the estimation strategy:

[0112] ;

[0113] Among them, Q represents the state weight matrix, R represents the input weight matrix, Represents the coefficient corresponding to the estimation error strategy. represents the state penalty term, represents the input penalty term, represents the linear error penalty term.

[0114] In this embodiment, in the control system design, the optimal control strategy is found by minimizing this penalty function. Minimizing the penalty function means that in the control process, the system state must be close to the ideal state, the control input must be reasonable, and the interference caused by the system state must be considered and suppressed, so as to achieve stable and efficient operation of the system. For example, in robot motion control, by minimizing such a penalty function, the robot can accurately reach the target position (state control) while the control input is stable (avoiding violent control), and can effectively deal with interference caused by changes in its own state (such as friction caused by joint movement).

[0115] In one embodiment, according to the penalty function after the kth step, the value function for evaluating the robot performance is obtained as follows:

[0116] .

[0117] The value function represents the cost of the cumulative penalty function starting from the kth step.

[0118] In one embodiment, according to the value function, the Q-function is constructed as:

[0119] ;

[0120] The Q-function is calculated from the penalty function and the value function at step k+1. It is used to evaluate the value of taking the control input increment in the current state and taking into account the current interference. It comprehensively considers the immediate cost of the current action and the accumulated cost in the future.

[0121] The value function simplifies to:

[0122] ;

[0123] P It is expressed as a symmetric positive definite matrix; the Q-function is further simplified as:

[0124] ;

[0125] To facilitate algorithm development, the Q-function is expressed as:

[0126] ;

[0127] in, ; ;

[0128] By solving, and The optimal strategy matrix and the optimal estimation error strategy matrix are obtained as follows:

[0129] ;

[0130] ;

[0131] according to , we can get the Bellman equation of the Q-function as:

[0132] ;

[0133] According to the formula , we can get:

[0134] ;

[0135] Will Simplified to:

[0136] ;

[0137] ;

[0138] ;

[0139] ;

[0140] Iterative learning through model-guided incremental reinforcement learning algorithm , in The iteration process is:

[0141] ;

[0142] ;

[0143] From the estimated Extract the model information of the robot model from the matrix:

[0144] ;

[0145] The extracted model information has two main functions: one is to achieve online security learning in the subsequent quadratic constraint optimization process; the other is to generate virtual data for training in forward prediction, as shown in Algorithms 1 and 2.

[0146] Algorithm 1: Model-guided secure incremental reinforcement learning algorithm:

[0147] Input: Iteration index: , output: robot control strategy:

[0148] 1.if , then

[0149] 2.if , then

[0150] 3. Evaluate the strategy based on dataset D:

[0151] 4. ;

[0152] 5.end if

[0153] 6. Strategy improvement:

[0154] 7. ,in ;

[0155] 8. ,in ;

[0156] 9. Model extraction:

[0157] 10. ;

[0158] 11. Safety calibration:

[0159] 12. ;

[0160] 13. Application Strategy On the robot

[0161] 14. Data Looping to add to dataset middle

[0162] 15. Model forward prediction:

[0163] 16. Model forward prediction generates data set ;

[0164] 17. ;

[0165] 18.end if

[0166] Algorithm 2: Model forward prediction algorithm:

[0167] enter: ;

[0168] Output: ;

[0169] 1. = {}

[0170] 2.for , do the following

[0171] 3. ;

[0172] 4. ;

[0173] 5. ;

[0174] 6. Add to middle

[0175] 7. end

[0176] In one embodiment, during the online learning process, the quadratic constraint optimization is constructed as:

[0177] ;

[0178] in, represents the input constraint set; represents the incremental input constraint set; is the control obstacle function that describes the edge of the obstacle.

[0179] Among them, the first state constraint restricts the robot to learn in a safe area, which is a constraint designed based on the control obstacle function; the second input constraint ensures that the final input acting on the robot does not exceed its executable range; the third input constraint ensures the amplitude of the incremental input to ensure stable learning.

[0180] The solution of the above-mentioned constraint optimization process relies on model information for constraint satisfaction checking. This application uses the above-mentioned linear increment system for constraint satisfaction checking. is the unit matrix, so it can be obtained from the iterative solution Extract the build increment system from the matrix Required model information , thus the description form of the robot motion characteristics can be obtained to check whether the safety constraints are satisfied, thereby realizing online safety learning.

[0181] In summary of the above embodiments, the present invention is to design an efficient incremental reinforcement learning algorithm to solve the problem of low sample utilization of conventional reinforcement learning algorithms and learning directly on robot hardware. Specifically, by using historical data to construct an equivalent linear incremental model of a nonlinear robot, it is possible to design an incremental reinforcement learning algorithm that drives the nonlinear robot to complete a given task through online interactive learning of linear strategies with low computational complexity. Second, an online safety learning strategy is designed to ensure the safety of the robot body and the surrounding environment, and to achieve stable and continuous training on the robot hardware. Specifically, the designed incremental reinforcement learning obtains the task strategy and the incremental model through parallel learning, and uses the incremental model to describe the robot's motion characteristics to facilitate the online solution of the safety strategy through constraint optimization.

[0182] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0183] In one embodiment, Figure 2 As shown, a robot safety training device based on incremental reinforcement learning is provided, including: a linear system construction module 202, a penalty function construction module 204, a strategy solving module 206 and an online safety learning module 208.

[0184] A linear system building module 202 is used to estimate unknown model information in the nonlinear system of the robot using historical data, and to build a linear incremental system of the robot; the linear incremental system includes: a control input incremental term and a linearization error term;

[0185] A penalty function construction module 204 is used to construct an augmented incremental system based on the linear incremental system, and construct a penalty function to represent the requirements of the robot to complete the task;

[0186] The strategy solving module 206 is used to obtain a value function for evaluating the robot performance according to the penalty function after the kth step; construct a Q-function according to the value function, and use a pre-built model to guide the incremental reinforcement learning algorithm to iteratively solve the Q-function online to obtain an optimal control strategy and an optimal error estimation strategy, as well as a robot model, and use the online learned robot model to perform forward prediction to obtain additional data to promote online reinforcement learning training;

[0187] The online safety learning module 208 is used to construct a quadratic constraint optimization to fine-tune the optimal control strategy of the robot based on the robot model learned online during the online learning process, so as to achieve online safety learning. In one embodiment, the linear system construction module 202 is also used for the nonlinear system description of the robot:

[0188] ;

[0189] in, , Respectively represent the state input and control input of the robot; , denote the state matrix and input matrix respectively;

[0190] Multiply both sides of the expression of the nonlinear system by the constant matrix B to obtain the expression for separating the unknown model information:

[0191] ;

[0192] in, Contains all unknown model information;

[0193] Use historical data to predict unknown model information The estimate is:

[0194] ;

[0195] Combining the expressions, the linear increment system is obtained as follows:

[0196] ;

[0197] in, represents the linearization error term, is the control input increment term, and A is the unit matrix.

[0198] In one embodiment, the penalty function construction module 204 is further configured to construct an augmented incremental system based on the linear incremental system as follows:

[0199] ;

[0200] in, .

[0201] In one embodiment, the penalty function building module 204 is further used to build a control strategy corresponding to the control input increment term and an estimation strategy corresponding to the linearization error term, respectively: and ; The penalty function is constructed according to the control strategy and the estimation strategy:

[0202] ;

[0203] Among them, Q represents the state weight matrix, R represents the input weight matrix, Represents the coefficient corresponding to the estimation error strategy.

[0204] In one embodiment, the strategy solving module 206 is further used to obtain a value function for evaluating the robot performance according to the penalty function after the kth step:

[0205] .

[0206] In one embodiment, the strategy solving module 206 is further configured to construct a Q-function according to the value function:

[0207] ;

[0208] The value function can be expressed as:

[0209] ;

[0210] P is a symmetric positive definite matrix;

[0211] The Q-function can be further simplified as:

[0212] ;

[0213] To facilitate algorithm development, the Q-function is expressed as:

[0214] ;

[0215] in, ; ;

[0216] By solving, and The optimal strategy matrix and the optimal estimation error strategy matrix are obtained as follows:

[0217] ;

[0218] ;

[0219] according to , we can get the Bellman equation of the Q-function as:

[0220] ;

[0221] According to the formula , we can get:

[0222] ;

[0223] Will Simplified to:

[0224] ;

[0225] ;

[0226] ;

[0227] ;

[0228] Iterative learning through model-guided incremental reinforcement learning algorithm , in The iteration process is:

[0229] ;

[0230] ;

[0231] From the estimated Extract the model information of the robot model from the matrix:

[0232] ;

[0233] According to the model information, forward prediction is performed, and the data generated by the forward prediction is used to construct Matrix and Matrix,facilitates efficient online training of robot policies.

[0234] In one embodiment, the online safety learning module 208 is also used in the online learning process to construct a quadratic constraint optimization as:

[0235] ;

[0236] in, represents the input constraint set; represents the incremental input constraint set; is the control obstacle function that describes the edge of the obstacle.

[0237] For the specific limitations of the robot safety training device based on incremental reinforcement learning, please refer to the limitations of the robot safety training method based on incremental reinforcement learning above, which will not be repeated here. Each module in the above-mentioned robot safety training device based on incremental reinforcement learning can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0238] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 3As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a robot safety training method based on incremental reinforcement learning is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0239] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0240] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.

[0241] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.

[0242] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0243] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0244] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A robot safety training method based on incremental reinforcement learning, characterized in that: The method comprises: The unknown model information in the nonlinear system of the robot is estimated by using historical data, and the linear incremental system of the robot is constructed; the linear incremental system includes: a control input incremental term and a linearization error term; An augmented incremental system is constructed based on the linear incremental system, and a penalty function is constructed to characterize the requirements for the robot to complete the task; According to the penalty function after the kth step, the value function used to evaluate the robot performance is obtained; According to the value function, a Q function is constructed, and a pre-constructed model is used to guide an incremental reinforcement learning algorithm to iteratively solve the Q function online to obtain an optimal control strategy and an optimal error estimation strategy, as well as a robot model, and the robot model learned online is used for forward prediction to obtain additional data to promote online reinforcement learning training; During the online learning process, based on the robot model learned online, a quadratic constraint optimization is constructed to fine-tune the optimal control strategy of the robot, thereby achieving online safe learning; The linear incremental system corresponding to the nonlinear robot is constructed using historical data, including: The nonlinear system of the robot is described as: ； in, , Respectively represent the state input and control input of the robot; , denote the state matrix and input matrix respectively; Multiply both sides of the expression of the nonlinear system by the constant matrix B to obtain the expression that separates the unknown model information: ； in, Contains all unknown model information; Use historical data to predict unknown model information The estimate is: ； Combining the expressions, the linear increment system is obtained as follows: ； in, represents the linearization error term, To control the input increment term, is the unit matrix; Constructing an augmented incremental system based on the linear incremental system includes: The augmented incremental system constructed based on the linear incremental system is: ； in, ; Construct a penalty function, including: The control strategy corresponding to the control input increment term and the estimation strategy corresponding to the linearization error term are constructed as follows: and ; The penalty function constructed according to the control strategy and the estimation strategy is: ； Among them, Q represents the state weight matrix, R represents the input weight matrix, represents the coefficient corresponding to the estimation error strategy; According to the penalty function after the kth step, the value function used to evaluate the robot performance is obtained, including: According to the penalty function after the kth step, the value function used to evaluate the robot performance is: ； According to the value function, a Q function is constructed, and a pre-constructed model is used to guide an incremental reinforcement learning algorithm to iteratively solve the Q function online, so as to obtain an optimal control strategy and an optimal error estimation strategy, as well as a robot model, including: According to the value function, the Q function is constructed as: ； The value function can be expressed as: ； P is a symmetric positive definite matrix; The Q function can be further simplified as: ； To facilitate algorithm development, the Q function is expressed as: ； in, ; ； By solving, and The optimal strategy matrix and the optimal estimation error strategy matrix are obtained as follows: ； ； according to , we can get the Bellman equation of the Q function as: ； According to the formula , we can get: ； Will Simplified to: ； ； ； ； Iterative learning through model-guided incremental reinforcement learning algorithm , in The iteration process is: ； ； ； From the estimated Extract the model information of the robot model from the matrix: ； According to the model information, forward prediction is performed, and the data generated by the forward prediction is used to construct Matrix and Matrix,facilitates efficient online training of robot policies.

2. The robot safety training method based on incremental reinforcement learning according to claim 1 is characterized in that: During the online learning process, based on the robot model learned online, a quadratic constraint optimization is constructed to fine-tune the optimal control strategy of the robot to achieve online safety learning, including: During the online learning process, the quadratic constraint optimization is constructed as: ； in, represents the input constraint set; represents the incremental input constraint set; is the control obstacle function that describes the edge of the obstacle.

3. A robot safety training device based on incremental reinforcement learning, used to implement the robot safety training method based on incremental reinforcement learning according to any one of claims 1 to 2, characterized in that: The device comprises: A linear system building module is used to estimate unknown model information in the nonlinear system of the robot using historical data to build a linear incremental system of the robot; the linear incremental system includes: a control input incremental term and a linearization error term; A penalty function construction module, used to construct an augmented incremental system based on the linear incremental system, and construct a penalty function to characterize the requirements of the robot to complete the task; A strategy solving module is used to obtain a value function for evaluating the robot performance according to the penalty function after the kth step; a Q function is constructed according to the value function, and a pre-built model is used to guide the incremental reinforcement learning algorithm to iteratively solve the Q function online to obtain an optimal control strategy and an optimal error estimation strategy, as well as a robot model, and the robot model learned online is used for forward prediction to obtain additional data to promote online reinforcement learning training; The online safety learning module is used to construct a quadratic constraint optimization to fine-tune the optimal control strategy of the robot based on the robot model learned online during the online learning process, so as to achieve online safety learning.

4. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 2 are implemented.

Images