Robot safety planning control method and system based on risk-sensitive reinforcement learning

Abstract

The present application relates to a robot safety planning and control method and system based on risk-sensitive reinforcement learning. The method includes: constructing a robot deep incremental model based on robot data; the robot safety planning and control architecture includes a planning layer and a control layer; designing a reference trajectory in the planning layer with the robot deep incremental model as a constraint; determining the upper and lower bounds of the tracking error in the control layer based on expert experience, and using the upper and lower bounds of the tracking error to construct a buffer for collision detection; constructing a risk-sensitive function based on the buffer size in the control layer, and constructing an optimal value function based on the risk-sensitive function to characterize the reference trajectory with optimal energy and specific performance; solving the optimal value function to obtain the input for tracking on the robot, and driving the robot to safely complete the given task. The use of this method can improve the safety and accuracy of the robot during the given task.

Description

Technical Field

[0001] The present application relates to the technical field of robot safety planning and control, and in particular to a robot safety planning and control method and system based on risk-sensitive reinforcement learning. Background Art

[0002] Facing the diverse tasks in the real world, robots are expected to perform the given tasks safely with optimal performance and high precision. Due to the inherent complexity of the above tasks, the problem is usually divided and conquered, and the planning layer and the control layer work together to solve the problem. The planning layer usually uses numerical optimization algorithms to solve nonlinear constraint problems to obtain the reference trajectory of the robot to complete the given task. However, in practice, whether the robot can complete the task as expected ultimately depends on whether the control layer can efficiently execute the reference trajectory of the planning layer. Therefore, in practice, whether the robot can complete the given task safely with high precision depends on the close connection between the planning layer and the control layer, which complement each other. However, the existing planning layer and control layer schemes are usually designed separately. The planning algorithm usually assumes that its planning results can be perfectly executed by the control layer, and the control strategy assumes that the planning layer can provide a reference trajectory with high executableness for the robot. Summary of the invention

[0003] Based on this, it is necessary to provide a robot safety planning control method and system based on risk-sensitive reinforcement learning that can improve the safety and accuracy of the robot in completing a given task in response to the above-mentioned technical problems.

[0004] A robot safety planning and control method based on risk-sensitive reinforcement learning, the method comprising:

[0005] Acquire robot data; build a robot deep incremental model based on robot data; build a robot safety planning and control architecture; the robot safety planning and control architecture includes a planning layer and a control layer;

[0006] In the planning layer, the reference trajectory is designed with the robot's depth increment model as a constraint condition; the upper and lower bounds of the tracking error in the control layer are determined based on expert experience, and the buffer zone for collision detection is constructed using the upper and lower bounds of the tracking error;

[0007] At the control layer, a risk-sensitive function is constructed based on the buffer size, and an optimal value function is constructed based on the risk-sensitive function to characterize the reference trajectory tracked with optimal energy and specific performance. The optimal value function is solved to obtain the input acting on the robot for tracking, driving the robot to safely complete the given task.

[0008] A robot safety planning and control system based on risk-sensitive reinforcement learning, the system comprising:

[0009] The robot safety planning and control architecture design module is used to obtain robot data; build a robot deep incremental model based on robot data; build a robot safety planning and control architecture; the robot safety planning and control architecture includes a planning layer and a control layer;

[0010] The planning module is used to design reference trajectories at the planning layer using the robot's depth increment model as a constraint condition; the upper and lower bounds of the tracking error at the control layer are determined based on expert experience, and the buffer zone for collision detection is constructed using the upper and lower bounds of the tracking error;

[0011] The control module is used to construct a risk-sensitive function based on the buffer size at the control layer, and to construct an optimal value function based on the risk-sensitive function to characterize the tracking of the reference trajectory with optimal energy and specific performance; the optimal value function is solved to obtain the input acting on the robot for tracking, so as to drive the robot to complete the given task safely.

[0012] The above-mentioned robot safety planning and control method and system based on risk-sensitive reinforcement learning first constructs a robot deep incremental model based on robot data and robot physical structure knowledge. The robot deep incremental model is the common basis of the planning layer and the control layer. The planning layer uses the deep incremental model as a constraint when designing the reference trajectory, which makes the trajectory planned by the planning layer fit the actual physical characteristics of the robot from the beginning. During the operation, the control layer will also adjust the control strategy based on this shared data-driven model. Based on the robot deep incremental model, this application proposes a robot safety operation framework that is compatible with the planning and control algorithms. The control layer tracks the reference trajectory of the planning layer with characteristic performance. At the same time, the planning algorithm designs the buffer used for safety inspection in the planning algorithm with the limit capacity of the control strategy, and shares the data-driven model with the control layer to ensure that the planned trajectory conforms to the physical characteristics of the robot and does not exceed the execution capability of the control layer. The planning layer and the control layer share the data-driven model, so that both parties can understand the actual state of the robot in real time. During the entire task execution process, the planning layer can adjust the planning of the reference trajectory according to the actual execution status fed back by the control layer (obtaining relevant data through the shared model). At the same time, the control layer can also use the shared model to better adjust the tracking strategy based on the updated planning information of the planning layer. This data sharing and collaborative working mechanism can dynamically adjust the work of the planning layer and the control layer to ensure that their results can match each other at all stages of task execution. Together, they can better drive the robot to safely complete complex tasks, effectively improving safety and accuracy throughout the entire given task process, and overcoming the problems of insufficient safety and accuracy caused by the independent design of the two in existing solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1A schematic flow chart of a robot safety planning and control method based on risk-sensitive reinforcement learning in one embodiment;

[0014] Figure 2 The present invention is a structural block diagram of a robot safety planning and control system based on risk-sensitive reinforcement learning in one embodiment. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0016] In one embodiment, Figure 1 As shown, a robot safety planning and control method based on risk-sensitive reinforcement learning is provided, comprising the following steps:

[0017] Step 102, obtain robot data; construct a robot deep incremental model based on the robot data; construct a robot safety planning and control architecture; the robot safety planning and control architecture includes a planning layer and a control layer.

[0018] Acquire robot data and build a deep incremental model of the robot. This deep incremental model of the robot is the common basis for the planning layer and the control layer. The planning layer uses the deep incremental model as a constraint when designing the reference trajectory, which makes the trajectory planned by the planning layer fit the actual physical characteristics of the robot from the beginning. During the operation, the control layer will also adjust the control strategy based on this shared data-driven model. For example, when constructing risk-sensitive functions and optimal value functions, the error dynamics model (whose construction also depends on the deep incremental model) will play a role, so that the operation of the control layer is closely related to the actual physical state of the robot. This construction method based on a common model provides a unified basic framework that matches the results of the planning layer and the control layer and conforms to the actual situation of the robot.

[0019] The deep incremental model can more accurately reflect the state changes of the robot and its characteristics under different working conditions, so that the design of the planning layer and the control layer is not a fantasy that is divorced from reality, but a targeted construction based on the real physical performance of the robot, laying a solid foundation for improving safety and accuracy.

[0020] Step 104, design a reference trajectory at the planning layer using the robot depth increment model as a constraint condition; determine the upper and lower bounds of the tracking error at the control layer based on expert experience, and use the upper and lower bounds of the tracking error to construct a buffer for collision detection.

[0021] In the planning layer, the reference trajectory is designed with the robot's deep incremental model as a constraint condition. This means that the planned trajectory fully considers the robot's own physical characteristics and actual capability limitations, avoids planning a trajectory that does not conform to the robot's actual execution possibility, and ensures from the source that the planned trajectory is executed by the robot with greater confidence, which helps to improve the accuracy of task execution. At the same time, the planning algorithm designs the buffer used for safety inspection in the planning algorithm based on the ability of the control strategy (specific tracking performance). This means that the planning layer fully considers the upper and lower bounds of the tracking error of the control layer during the planning process. The planning layer does not generate reference trajectories in isolation, but sets a buffer according to the tracking performance that the control layer can achieve, which serves as the basis for safety inspection. In this way, the planned trajectory is within the tracking capability of the control layer, avoiding the disconnection between the planning results and the control execution capabilities, and ensuring the matching of the two results from the planning perspective.

[0022] By introducing augmented states to construct an error dynamics model, the deviation between the planned trajectory and the actual execution status can be quantified in real time, which facilitates the timely discovery and correction of deviations in the subsequent control process, further ensuring that the robot moves along the desired trajectory and improving the accuracy and safety of execution.

[0023] Step 106, construct a risk-sensitive function based on the buffer size at the control layer, and construct an optimal value function based on the risk-sensitive function to characterize the reference trajectory tracking with optimal energy and specific performance; solve the optimal value function to obtain the input acting on the robot for tracking, and drive the robot to safely complete the given task.

[0024] Based on the buffer, a risk-sensitive function is constructed. The risk-sensitive function is used to construct an optimal value function that represents the reference trajectory tracked with optimal energy and specific performance. Then, the optimal value function is solved to obtain the input for tracking on the robot. This allows the control layer to track the reference trajectory given by the planning layer in the best way while considering safety (reflected by the risk-sensitive function), ensuring the accuracy and efficiency of execution, so that the robot can move as much as possible along the planned trajectory and avoid excessive energy consumption and other situations that are not conducive to task execution. The control layer tracks the reference trajectory of the planning layer by constructing a risk-sensitive function and an optimal value function that tracks with specific performance and optimal energy. When constructing the risk-sensitive function, it is based on the buffer constructed by the planning layer considering physical characteristics. The planning algorithm designs the buffer used for safety checks in the planning algorithm based on the ability of the control strategy (specific tracking performance), and shares the data-driven model with the control layer, which ensures that the planning layer and the control layer are not isolated from each other. The planning layer and the control layer share the data-driven model, which enables both parties to understand the actual status of the robot in real time. During the entire task execution process, the planning layer can adjust the planning of the reference trajectory based on the actual execution feedback from the control layer (obtaining relevant data through the shared model). At the same time, the control layer can also use the shared model to better adjust the tracking strategy based on the updated planning information of the planning layer. This data sharing and collaborative working mechanism can dynamically adjust the work of the planning layer and the control layer to ensure that their results can match each other at all stages of task execution. Together, they can better drive the robot to safely complete complex tasks, effectively improving safety and accuracy throughout the entire given task process, and overcoming the problems of insufficient safety and accuracy caused by the independent design of the two in existing solutions.

[0025] The above-mentioned robot safety planning and control method and system based on risk-sensitive reinforcement learning first constructs a robot deep incremental model based on robot data and robot physical structure knowledge. The robot deep incremental model is the common basis of the planning layer and the control layer. The planning layer uses the deep incremental model as a constraint when designing the reference trajectory, which makes the trajectory planned by the planning layer fit the actual physical characteristics of the robot from the beginning. During the working process, the control layer will also adjust the control strategy based on this shared data-driven model. On the basis of the robot deep incremental model, this application proposes a robot safety operation framework that is compatible with the planning and control algorithms. The control layer tracks the reference trajectory of the planning layer with characteristic performance. At the same time, the planning algorithm designs the buffer used for safety inspection in the planning algorithm with the ability of the control strategy, and shares the data-driven model with the control layer to ensure that the planned trajectory conforms to the physical characteristics of the robot and does not exceed the execution capability of the control layer. The planning layer and the control layer share the data-driven model, so that both parties can understand the actual state of the robot in real time. During the entire task execution process, the planning layer can adjust the planning of the reference trajectory according to the actual execution status fed back by the control layer (obtaining relevant data through the shared model). At the same time, the control layer can also use the shared model to better adjust the tracking strategy based on the planning information updated by the planning layer. This data sharing and collaborative working mechanism can dynamically adjust the work of the planning layer and the control layer, ensuring that their results match each other at all stages of task execution. Working together, they can better drive the robot to safely complete complex tasks, effectively improving safety and accuracy throughout the entire given task process, and overcoming the problems of insufficient safety and accuracy in existing solutions due to the independent design of the two.

[0026] In one embodiment, a robot deep incremental model is constructed based on robot data and robot physical structure knowledge, including:

[0027] The robot's incremental model is calculated based on the robot state-input information and incremental matrix at two adjacent time steps;

[0028] With the goal of minimizing the estimation error, the incremental matrix is ​​parameterized by using a deep neural network. Based on a pre-constructed dataset, the optimal incremental matrix is ​​obtained through offline learning from a supervised learning perspective. Then, a robot deep incremental model is constructed according to the incremental matrix parameterized by the deep neural network.

[0029] In one embodiment, the incremental model of the robot is calculated based on the robot state-input information and the incremental matrix at two adjacent time steps, including:

[0030] The incremental model of the robot is calculated based on the robot state-input information and incremental matrix of two adjacent time steps:

[0031] ;

[0032] in, represents the difference between the system state derivatives at two adjacent sampling moments, Represents the value of the robot state derivative at the last sampling moment, represents the estimation error, For incremental input, represents the estimation error, Represents the unknown model information of the current step, represents an estimate of the unknown model information, Indicates the robot status, represents the increment matrix, represents the value of the input information u at the last sampling time, Represents the sampling time of the robot system.

[0033] In one embodiment, a robot deep incremental model is constructed according to an incremental matrix parameterized by a deep neural network, including:

[0034] The robot deep incremental model is constructed based on the incremental matrix parameterized by the deep neural network:

[0035] ;

[0036] in, Represents the value of the robot state derivative at the last sampling moment, represents the incremental matrix for parameterizing the deep neural network, Incremental input.

[0037] In a specific embodiment, the process of constructing the robot depth incremental model is as follows.

[0038] The robot's motion characteristics can be described as

[0039] ;

[0040] in, Indicates the robot status. represents the state matrix, represents the input matrix, Represents external environmental interference.

[0041] In order to separate the unknown model information in the robot model, the incremental matrix is ​​introduced , and multiplying both sides of the above formula by its pseudo-inverse, we can get

[0042] ;

[0043] in, Contains all unknown model information, express The pseudo-inverse.

[0044] For high sampling frequency systems, considering that the difference between two adjacent steps is very small, this application uses the historical data of the previous step to estimate the unknown model information of the current step, which is specifically expressed as:

[0045] ;

[0046] in, Indicates unknown model information The estimated value of Indicates the robot status x Information of the last sampling moment, It represents the information of the last sampling time of the robot control input u, and L is the sampling time of the robot system.

[0047] Combining the above results, the incremental model of the robot can be obtained as follows:

[0048] ;

[0049] in, represents the difference between the system state derivatives at two adjacent sampling moments, Represents the value of the robot state derivative at the last sampling moment, represents the estimation error, Incremental input.

[0050] In addition, in order to determine the increment matrix that can fully describe the robot's kinematic characteristics , with the goal of minimizing the estimation error, using the deep network to parameterize the incremental matrix , and is obtained from offline learning from a supervised learning perspective based on a pre-built dataset The specific implementation method is described as follows

[0051] ;

[0052] in, represents the parameterized incremental matrix of the deep neural network, Representation dataset Middle i Incremental input, Representation dataset Middle i The increment of the robot state derivative. Representation variables The Euclidean norm of .

[0053] Finally, the corresponding depth increment system can be obtained as

[0054] ;

[0055] Its equivalent form is

[0056] ;

[0057] The deep incremental system constructed in this part will subsequently serve the development of planning algorithms and control strategies.

[0058] In one embodiment, building a risk-sensitive function based on a buffer zone includes:

[0059] The risk sensitivity function based on the buffer zone is constructed as

[0060] ;

[0061] in, , , The tracking error i Quantity, Indicates the expected input i Components, specific performance functions The specific expression is ,

[0062] , , , and is a hyperparameter closely related to the buffer boundary, Indicates time, , is the risk sensitivity coefficient to be designed, Representation parameters The square value of Representation parameters The square value of .

[0063] In one embodiment, constructing an optimal value function representing tracking a reference trajectory with energy optimality and specific performance using a risk-sensitive function includes:

[0064] The optimal value function that characterizes the reference trajectory with optimal energy and specific performance is constructed using the risk-sensitive function:

[0065] ;

[0066] in, represents differential incremental input, represents the augmented state, represents the effect function, represents the risk sensitivity function, represents the control weight matrix.

[0067] In a specific embodiment, according to the reference trajectory surrounded by the buffer zone provided by the planning layer And the corresponding This application develops a specific performance optimal tracking control strategy to ensure high-precision tracking of the above reference trajectory while ensuring that the robot does not cross the buffer zone (i.e., does not enter the dangerous area) under the worst tracking performance, ensuring that the robot safely performs the given task. The specific details are as follows:

[0068] Assume that the reference trajectory provided by the planning layer satisfies Describes the dynamics of the reference trajectory. The introduction of is convenient for the subsequent explanation of the control strategy principle, and the actual control strategy does not need Display format.

[0069] To achieve high performance tracking reference trajectory , introduce the augmented state , construct the error dynamics as

[0070] ;

[0071] in, is the state matrix corresponding to the error dynamics, is the input matrix corresponding to the error dynamics, is the differential increment input, and T is the transpose sign of the vector or matrix.

[0072] The value function that represents the energy optimality of tracking a specific performance is constructed as

[0073] ;

[0074] The effect function is ,in The risk sensitivity function proposed in this patent and constructed based on the specific performance function is specifically expressed as follows:

[0075] ;

[0076] in, , , specific performance function The specific expression is

[0077] ;

[0078] The tracking error i Quantity, represents the i-th component of the expected input, , , , is a hyperparameter closely related to the buffer boundary, Indicates time, , is the risk sensitivity coefficient to be designed. R is the control weight matrix.

[0079] In specific performance functions, Tracking Error The maximum value, considering The actual range of motion of the unmanned system is determined, so that The value of is used as a reference for constructing the buffer zone in the planning layer. Characterizes the value of tracking error tracking convergence, using The accuracy of the unmanned system during stable operation can be quantified, where Representation variables The absolute value of .

[0080] Finally, the optimal value function of the above problem is

[0081] .

[0082] In one embodiment, solving the optimal value function to obtain an input for tracking on the robot includes:

[0083] Define the Hamiltonian function as

[0084] The optimal value function satisfies the following HJB equation:

[0085] .

[0086] According to the optimality condition and HJB equation, the explicit solution of the optimal control is obtained as:

[0087] ;

[0088] in, represents the partial differential of the optimal value function;

[0089] The Hamilton-Jacobi-Bellman equation is difficult to directly solve to obtain the partial differential of the value function , so this application seeks its approximate solution and approximates the optimal value function as

[0090] ;

[0091] in, is the weight matrix, is the activation function, is the approximation error;

[0092] Then the partial differential form of the optimal value function after approximation is

[0093] ;

[0094] in, Representation variables For status The partial differential of

[0095] Substituting the partial differential form of the optimal value function approximated into the HJB equation, the Lyaounov equation is obtained:

[0096] ;

[0097] in, is the residual term;

[0098] definition , Rewrite the Lyaounov equation to get the rewritten Lyaounov equation:

[0099] ;

[0100] in, Representation weight An estimated value of

[0101] definition The estimated error is in the form of

[0102] .

[0103] The weights are updated according to the designed neural network weight update rate to optimize the estimated error, and the estimated weights are used to construct an incremental strategy. The input for tracking on the robot at the current time step is calculated based on the incremental strategy and the input value of the previous time step:

[0104] ;

[0105] in, represents the incremental strategy, The expected incremental input corresponding to the reference trajectory, Represents the input value at the previous time step.

[0106] In a specific embodiment, the definition , , the above formula can be written as

[0107] ;

[0108] In the actual operation process, the value function estimated by the evaluator network is

[0109] ;

[0110] Its corresponding partial differential form is

[0111] .

[0112] According to the above estimated values, the estimated form of the above Lyaounov equation can be obtained as

[0113] ;

[0114] definition The estimated error is in the form of

[0115] .

[0116] Update the designed neural network weight update rate to optimize the estimation error ,include:

[0117] The designed neural network weight update rate is

[0118] ;

[0119] in, , , Update the hyperparameters for the chosen weights, Indicates the size of all experience pools, Represents the first element in the experience pool A quantity.

[0120] In one embodiment, constructing an incremental strategy using the estimated weights includes:

[0121] The incremental strategy constructed with the estimated weights is

[0122] ;

[0123] in, represents the control weight matrix, represents the input matrix corresponding to the error dynamics, Represents the activation function For status The partial differential of Representation weight The superscript T indicates the transpose operation.

[0124] It should be understood that although Figure 1The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0125] In one embodiment, Figure 2 As shown, a robot safety planning and control system based on risk-sensitive reinforcement learning is provided, including: a robot safety planning and control architecture design module 202, a planning module 204 and a control module 206, wherein:

[0126] A robot safety planning and control architecture design module 202 is used to obtain robot data; construct a robot deep incremental model based on the robot data; and construct a robot safety planning and control architecture; the robot safety planning and control architecture includes a planning layer and a control layer;

[0127] The planning module 204 is used to design a reference trajectory at the planning layer using the robot depth increment model as a constraint condition; determine the upper and lower bounds of the tracking error at the control layer based on expert experience, and use the upper and lower bounds of the tracking error to construct a buffer for collision detection;

[0128] The control module 206 is used to construct a risk-sensitive function based on the buffer size at the control layer, and to construct an optimal value function based on the risk-sensitive function to characterize the tracking of the reference trajectory with optimal energy and specific performance; solve the optimal value function to obtain the input for tracking on the robot, and drive the robot to safely complete the given task.

[0129] For the specific definition of a robot safety planning and control system based on risk-sensitive reinforcement learning, please refer to the definition of a robot safety planning and control method based on risk-sensitive reinforcement learning above, which will not be repeated here. Each module in the above-mentioned robot safety planning and control system based on risk-sensitive reinforcement learning can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0130] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0131] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A robot safety planning and control method based on risk-sensitive reinforcement learning, characterized in that: The method comprises: Acquire robot data; construct a robot deep incremental model based on the robot data; construct a robot safety planning and control architecture; the robot safety planning and control architecture includes a planning layer and a control layer; In the planning layer, a reference trajectory is designed with the robot depth increment model as a constraint condition; the upper and lower bounds of the tracking error in the control layer are determined based on expert experience, and a buffer zone for collision detection is constructed using the upper and lower bounds of the tracking error; A risk-sensitive function is constructed in the control layer based on the buffer size, and an optimal value function that characterizes tracking of a reference trajectory with optimal energy and specific performance is constructed based on the risk-sensitive function; the optimal value function is solved to obtain an input for tracking on the robot, thereby driving the robot to safely complete a given task.

2. The method according to claim 1, characterized in that Constructing a robot deep incremental model according to the robot data, including: The robot's incremental model is calculated based on the robot state-input information and incremental matrix at two adjacent time steps; With the goal of minimizing the estimation error, the incremental matrix is ​​parameterized by a deep neural network. Based on a pre-constructed dataset, the optimal incremental matrix is ​​obtained through offline learning from a supervised learning perspective. The robot deep incremental model is constructed using the incremental matrix parameterized by a deep neural network.

3. The method according to claim 2, characterized in that The incremental model of the robot is calculated based on the robot state-input information and incremental matrix of two adjacent time steps, including: The incremental model of the robot is calculated based on the robot state-input information and incremental matrix of two adjacent time steps: ； in, represents the difference between the system state derivatives at two adjacent sampling moments, represents the value of the robot state derivative at the last sampling moment, For incremental input, represents the estimation error, Represents the unknown model information of the current step, represents an estimate of the unknown model information, Indicates the robot status, represents the increment matrix, represents the value of the input information u at the last sampling time, represents the sampling time of the robot system, Indicates time.

4. The method according to claim 2, characterized in that: The robot deep incremental model is constructed according to the incremental matrix parameterized by the deep neural network, including: The robot deep incremental model is constructed based on the incremental matrix parameterized by the deep neural network: ； in, represents the value of the robot state derivative at the last sampling moment, represents the incremental matrix for parameterizing the deep neural network, For incremental input, Indicates time.

5. The method according to claim 1, characterized in that Construct a risk-sensitive function based on the buffer zone, including: The risk sensitivity function based on the buffer zone is constructed as ； in, , , The tracking error i Quantity, Indicates the expected input i Components, specific performance functions The specific expression is , , , , ,and is a hyperparameter closely related to the buffer boundary, Indicates time, , is the risk sensitivity coefficient to be designed, Representation parameters The square value of Representation parameters The square value of .

6. The method according to claim 1, characterized in that The risk-sensitive function is used to construct the optimal value function that represents the reference trajectory with optimal energy and specific performance, including: The optimal value function that characterizes the reference trajectory with optimal energy and specific performance is constructed using the risk-sensitive function: ； in, represents differential incremental input, represents the augmented state, represents the effect function, represents the risk sensitivity function, represents the control weight matrix.

7. The method according to claim 6, characterized in that The optimal value function is solved to obtain the input acting on the robot for tracking, including: The optimal value function satisfies the following HJB equation: ； According to the optimality condition and HJB equation, the explicit solution of the optimal control is obtained as: ； in, represents the partial differential of the optimal value function; The optimal value function is approximated as ； in, is the weight matrix, is the activation function, is the approximation error; Then the partial differential form of the optimal value function after approximation is ； in, Representation variables For status The partial differential of Substituting the partial differential form of the optimal value function approximated into the HJB equation, the Lyaounov equation is obtained: ； in, is the residual term; definition , Rewrite the Lyaounov equation to get the rewritten Lyaounov equation: ； in, Representation weight An estimated value of definition The estimated error is in the form of ； The weights are updated according to the designed neural network weight update rate to optimize the estimated error, and the estimated weights are used to construct an incremental strategy. The input for tracking on the robot at the current time step is calculated based on the incremental strategy and the input value of the previous time step: ； in, represents the incremental strategy, The expected incremental input corresponding to the reference trajectory, Represents the input value at the previous time step.

8. The method according to claim 7, characterized in that The estimated error is optimized based on the designed neural network weight update rate, including: The designed neural network weight update rate is in, , , Update the hyperparameters for the chosen weights, Indicates the size of all experience pools, , Represents the first element in the experience pool A quantity.

9. The method according to claim 7, characterized in that: Construct an incremental strategy with the estimated weights, including: The incremental strategy constructed with the estimated weights is ； in, represents the control weight matrix, represents the input matrix corresponding to the error dynamics, Represents the activation function For status The partial differential of Representation weight The superscript T indicates the transpose operation.

10. A robot safety planning and control system based on risk-sensitive reinforcement learning, characterized in that: The system comprises: A robot safety planning and control architecture design module is used to obtain robot data; construct a robot deep incremental model based on the robot data and robot physical structure knowledge; and construct a robot safety planning and control architecture; the robot safety planning and control architecture includes a planning layer and a control layer; A planning module is used to design a reference trajectory at the planning layer using the robot depth increment model as a constraint condition; determine the upper and lower bounds of the tracking error at the control layer based on expert experience, and construct a buffer zone using the upper and lower bounds of the tracking error; The control module is used to construct a risk-sensitive function based on the buffer in the control layer, and use the risk-sensitive function to construct an optimal value function that characterizes the tracking of the reference trajectory with optimal energy and specific performance; solve the optimal value function to obtain the input acting on the robot for tracking, and drive the robot to safely complete the given task.

Images