A smooth reinforcement learning method and device for robot intelligent control

Abstract

The present invention provides a smooth reinforcement learning method and device for robot intelligent control, relating to the field of robot intelligent control technology. The method comprises: obtaining environmental observation sample information of the robot at the current moment; constructing a smooth reinforcement learning control model based on the current sample information; constructing a reward function for the control model based on the sample information; constructing constraints for the control model using the robot's discrete-time kinematic equations; inputting the sample information, reward function, and constraints into the control model for online training to obtain the robot's smooth control action at the current moment; constructing an objective function for the control model; updating the control model based on the objective function to obtain an updated control model; obtaining observation sample information for the next moment based on the smooth control action, and inputting this information into the updated control model to obtain the robot's smooth control action at the next moment. Using this method, a robot can complete smooth control tasks with high real-time and high precision.

Description

Technical Field

[0001] The present invention relates to the technical field of robot intelligent control, and in particular to a smooth reinforcement learning method and device for robot intelligent control. Background Art

[0002] A robot is a programmed, controlled machine capable of performing tasks autonomously. Based on their application, they can be categorized into industrial robots used in manufacturing, medical robots used in healthcare, and mobile robots with mobility and the ability to perform tasks autonomously, such as driverless cars and drones. Intelligent robot control has enormous potential for reducing production costs, minimizing workplace safety incidents, and improving productivity.

[0003] Modern robots are typically equipped with sensors, actuators, and complex control systems, enabling them to perceive the environment, process information, and autonomously perform actions according to pre-set tasks. Existing robot control methods have the following main drawbacks: On the one hand, some typical control methods, such as proportional-integral-derivative control and linear quadratic regulation control, although simple in structure, are unable to handle complex system constraints or ignore the nonlinear characteristics of the system, making it difficult to achieve ideal control effects in practical applications. On the other hand, for predictive control methods, the controller needs to solve the optimal control sequence for each sampling period online based on the prediction model and constraints. When the solution difficulty increases and computing resources are insufficient, the solution speed cannot meet the real-time control requirements of the controller. The above control methods are essentially solving the optimal solution of the optimization control problem online under certain constraints and rules, which makes it difficult to adapt to the more efficient and intelligent control needs of future robots.

[0004] In recent years, with the rapid development of artificial intelligence and machine learning technologies, learning-based decision-making schemes, represented by reinforcement learning, have shown great potential in the field of robotic control. Reinforcement learning can learn an optimal policy through autonomous interaction with the environment, thereby maximizing future cumulative rewards. Reinforcement learning offers the advantages of strong adaptability, autonomous learning, and the ability to handle high-dimensional state-action spaces and nonlinear problems. Its offline-to-online solution model also makes reinforcement learning methods highly efficient. However, control strategies derived from reinforcement learning suffer from severe motion jitter in practical robotic applications. This jitter can accelerate wear of mechanical components, create safety hazards, and negatively impact user experience. Generally, motion jitter can be attributed to the surface roughness of the neural network that serves as the policy carrier and input perturbations caused by sensor measurement noise. This jitter problem has, to a certain extent, hindered the further application of reinforcement learning in intelligent robotic control. Summary of the Invention

[0005] To address the jitter in existing robot neural network decision-making and control technologies, as well as the difficulty in finding the optimal solution to optimization control problems online under certain constraints, and the difficulty in adapting to future demands for more efficient and intelligent robot control, the present invention provides a smooth reinforcement learning method and device for robot intelligent control. The technical solution is as follows:

[0006] In one aspect, a smooth reinforcement learning method for robot intelligent control is provided. The method is implemented by a smooth reinforcement learning device for robot intelligent control, and the method includes:

[0007] S1. Obtain the current robot's environmental observation sample information;

[0008] S2. Constructing a smooth reinforcement learning control model based on the current robot's environmental observation sample information;

[0009] S3. Constructing a reward function of the control model based on the current robot's environmental observation sample information; and constructing constraint conditions of the control model using the robot's discrete-time kinematic equations.

[0010] S4, inputting the current robot's environmental observation sample information, the reward function, and the constraint conditions into the control model for online training to obtain a smooth control action of the robot at the current moment;

[0011] S5. Constructing an objective function of the control model; performing iterative update calculations on the control model according to the objective function to obtain an updated smooth reinforcement learning control model;

[0012] S6. Obtain observation sample information at the next moment based on the smooth control action of the robot at the current moment; input the observation sample information at the next moment into the updated control model to obtain the smooth control action of the robot at the next moment.

[0013] On the other hand, a smooth reinforcement learning device for robot intelligent control is provided, which is applied to a smooth reinforcement learning method for robot intelligent control, and the device includes:

[0014] The first acquisition unit is used to obtain the environment observation sample information of the robot at the current moment;

[0015] A first construction unit is configured to construct a smooth reinforcement learning control model based on the environment observation sample information of the robot at the current moment;

[0016] A second construction unit is configured to construct a reward function of the control model based on the environment observation sample information of the robot at the current moment; and to construct the constraint conditions of the control model using the discrete-time kinematic equation of the robot;

[0017] A second acquisition unit is configured to input the current robot's environmental observation sample information, the reward function, and the constraint conditions into the control model for online training to obtain a smooth control action of the robot at the current moment;

[0018] A third acquisition unit is configured to construct an objective function of the control model; perform iterative update calculations on the control model according to the objective function to obtain an updated smooth reinforcement learning control model;

[0019] The fourth acquisition unit is used to obtain the observation sample information at the next moment based on the smooth control action of the robot at the current moment; input the observation sample information at the next moment into the updated control model to obtain the smooth control action of the robot at the next moment.

[0020] On the other hand, a smooth reinforcement learning device for robot intelligent control is provided, comprising: a processor; a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned smooth reinforcement learning methods for robot intelligent control is implemented.

[0021] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned smooth reinforcement learning methods for robot intelligent control.

[0022] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0023] The embodiment of the present invention first obtains the environmental observation sample information of the robot at the current moment; secondly, constructs a smooth reinforcement learning control model based on the environmental observation sample information of the robot at the current moment; constructs a reward function of the control model based on the environmental observation sample information of the robot at the current moment; adopts the discrete-time kinematic equation of the robot to construct the constraint conditions of the control model; inputs the environmental observation sample information, reward function and constraint conditions of the robot at the current moment into the control model for online training to obtain the smooth control action of the robot at the current moment; constructs the objective function of the control model; updates the control model iteratively according to the objective function to obtain an updated smooth reinforcement learning control model; finally, obtains the observation sample information at the next moment based on the smooth control action of the robot at the current moment; inputs the observation sample information at the next moment into the updated control model to obtain the smooth control action of the robot at the next moment.

[0024] The embodiment of the present invention proposes a smooth reinforcement learning control model for robot intelligent control. The core architecture of the model is the actor-critic-damper (ACD) architecture. This architecture is based on the actor-critic (AC) architecture of the existing mainstream reinforcement learning algorithm. By adding a damper module, a new smooth reinforcement learning control architecture is formed, which is particularly oriented to robot intelligent control. Among them, the execution module in the execution-critic-damper architecture corresponds to the traditional non-smooth reinforcement learning strategy, the damper module is responsible for providing the damping strength related to the state and converting the non-smooth action into a smooth action, and the evaluation module is responsible for evaluating the smooth strategy and the non-smooth strategy. Under the assumption that a desired reference trajectory or reference point is given, the machine The robot's own sensors obtain the robot's various state quantities at the current moment, and use this to construct the objective functions of the execution module, damping module and evaluation module, and design the corresponding offline training loss function; the robot's own state and environmental information are used as inputs to the execution module and damping module, and non-smooth actions and smooth actions are used as outputs, converting the online optimization problem into an offline solution of the network parameters of each module; when applied online, the controller directly outputs an approximate smoothing control strategy from the execution module network and the damping module network according to the robot state and surrounding environment information at each moment, and the strategy is executed by the lower-level control program to obtain the robot state at the next moment, realizing online and high-real-time robot smooth control; in addition, the execution-evaluation-damper architecture has controllable smoothing capabilities, and the expected smoothing multiple can be set in advance before training to reduce the time spent on parameter adjustment.

[0025] The embodiment of the present invention proposes an execution module with a stable structure, which can focus on optimizing the original non-smooth strategy, and the damping module focuses on optimizing the smooth strategy; the execution-evaluation-damping architecture has the ability to adaptively adjust the smoothing capability, which can reduce the time spent on parameter adjustment; the embodiment of the present invention is suitable for industrial robots, medical robots and unmanned vehicles, and has the advantages of high computational efficiency, high precision and memory saving, breaking through the bottleneck of motion jitter in the existing reinforcement learning application in robot control. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 This is a flow chart of a smooth reinforcement learning method for robot intelligent control provided by an embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram of the overall process flow of a smooth reinforcement learning method for robot intelligent control provided by an embodiment of the present invention;

[0029] Figure 3 Schematic diagram of a wheeled robot trajectory tracking and collision avoidance scenario provided by an embodiment of the present invention;

[0030] Figure 4 2. It is a schematic diagram comparing the motion fluctuations of the smooth reinforcement learning method for robot intelligent control provided by an embodiment of the present invention;

[0031] Figure 5 Schematic diagram of trajectory tracking and collision avoidance effect of the smooth reinforcement learning method for robot intelligent control provided by an embodiment of the present invention;

[0032] Figure 6 This is a block diagram of a smooth reinforcement learning device for robot intelligent control provided by an embodiment of the present invention;

[0033] Figure 7 This is a schematic diagram of the structure of a smooth reinforcement learning device for robot intelligent control provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0035] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0036] In the embodiments of the present invention, the terms "image" and "picture" may be used interchangeably. It should be noted that, when the distinction between them is not emphasized, their intended meanings are the same. The terms "of," "corresponding," and "corresponding" may be used interchangeably. It should be noted that, when the distinction between them is not emphasized, their intended meanings are the same.

[0037] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0038] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0039] The embodiment of the present invention provides a smooth reinforcement learning method for robot intelligent control, which can be implemented by a smooth reinforcement learning device for robot intelligent control, and the smooth reinforcement learning device for robot intelligent control can be a terminal or a server. Figure 1 The flow chart of the smooth reinforcement learning method for robot intelligent control shown in FIG. 1 may include the following steps:

[0040] S1. Obtain the robot's current environmental observation sample information.

[0041] Optionally, the specific implementation of S1 may include S11-S12:

[0042] S11, obtaining the current robot state information, the current surrounding environment information, and the current robot navigation information;

[0043] The robot's own coordinate system is established with its position at time t as the origin, and the robot's current state information is obtained based on the sensor. The current state information includes: the robot's horizontal and vertical positions, longitudinal speed, and heading angle at time t.

[0044] Among them, in the robot-vehicle coordinate system, the surrounding environment information at time t is obtained, and the surrounding environment information includes: the horizontal and vertical positions, longitudinal speed and heading angle of the nearest obstacle in the environment where the robot is located at time t.

[0045] Among them, in the robot-vehicle coordinate system, the navigation information of the robot at time t is obtained, that is, the expected trajectory point information of the robot movement. The expected trajectory point information includes multiple reference points, where each reference point information includes: the position information of the expected reference point, the expected speed information and the heading angle information.

[0046] S12: Combine the robot state information, the surrounding environment information and the robot navigation information at the current moment to obtain the robot's environmental observation sample information at the current moment.

[0047] In a feasible implementation, the robot state information, surrounding environment information, and navigation information obtained at time t are combined into the robot's environmental observation sample information, which can be expressed by the following formula (1):

[0048]

[0049] in, Represents the k-dimensional environment observation sample information of the robot at the current time t; Represents the m-dimensional state information of the robot at the current time t; Represents the b-dimensional environment information around the robot at the current time t; represents the p-dimensional navigation information of the robot at the current time t; represents a real number; wherein, k=m+b+p, k, m, b and p represent the dimensions of the vector.

[0050] S2. Build a smooth reinforcement learning control model based on the robot's current environmental observation sample information.

[0051] Among them, reinforcement learning includes environment, agent, state, observation and action. That is, the agent interacts with the environment based on the current state and action to obtain the state of the agent at the next moment. At the same time, the reward is calculated based on the state and action of the agent, and the relevant objective function is constructed using the reward. The obtained objective function is used to improve the agent's strategy and finally maximize the total reward.

[0052] In an embodiment of the present invention, the environment can be a system that performs state deduction on objects therein based on laws such as physical laws, kinematics, and geometric constraints, and is composed of intelligent agents, obstacles, and map constraints. The intelligent agent is an object element in the environment that can act autonomously, receive observations, and give actions, such as an autonomous driving vehicle. The state is the basic attribute of an element in the environment, such as the vehicle's position, orientation, and speed. The observation is a feature obtained by encoding the state of other elements in the environment from the perspective of the intelligent agent's given state. The action is the output of the intelligent agent, which is determined by the intelligent agent's strategy. The deduction is to iteratively apply the state transition model given an action sequence and an initial state of the object, continuously obtain the state of each step, and finally obtain a state sequence as output, wherein the state transition model is defined on the object, and given the state and action of the object at the current moment, the output object will transfer to the state at the next moment, which is determined by laws such as kinematic constraints. The reward is used to represent the evaluation of the state and action of the element in the environment at the current time step, and is a scalar. The cumulative reward is the sum of the rewards at all time steps starting from the current state until the end of the deduction.

[0053] Among them, constructing a robot smooth control model under the reinforcement learning framework is to let the robot interact with the environment to learn an optimal smooth control strategy, so as to maximize the objective function obtained by executing the smooth control strategy from the initial moment; the robot smooth control model includes a policy network and a state transition model. The policy network outputs the optimal smooth action in the current state based on the input vehicle state and environmental information; the state transition model outputs the state of the robot at the next moment given the state and action of the robot at the current moment, which is determined by laws such as kinematic constraints.

[0054] Among them, the core of the smooth reinforcement learning control model for robot intelligent control lies in the construction of the execution-evaluation-damping architecture; the execution-evaluation-damping architecture is suitable for the current mainstream execution-evaluation architecture reinforcement learning algorithm; the three modules in the execution-evaluation-damping architecture can be approximated using currently commonly used neural network functions, including multi-layer perceptron MLP, convolutional neural network CNN and recurrent neural network RNN, etc.; the embodiment of the present invention takes the multi-layer perceptron MLP as an example, and the execution module, evaluation module and damping module in the smooth reinforcement learning control model for wheeled robots all use the multi-layer perceptron as the approximate function, the hidden layer size of the three modules is 256×256, and the activation function of the three modules is all Relu function.

[0055] Optionally, the smooth reinforcement learning control model includes: an execution module, a damper module and an evaluation module; wherein,

[0056] Execution module, used to correspond to traditional non-smooth reinforcement learning strategies;

[0057] The damper module is used to provide state-dependent damping strength and transform non-smooth motion into smooth motion;

[0058] The evaluation module is used to evaluate the evaluation values ​​obtained by the smoothing strategy and the evaluation values ​​obtained by the non-smoothing strategy.

[0059] S3. Construct the reward function of the control model based on the robot's current environmental observation sample information; use the robot's discrete-time kinematic equations to construct the constraints of the control model.

[0060] Optionally, S3 adopts the discrete time kinematic equation of the robot to construct the constraint conditions of the control model, including: adopting the discrete time kinematic equation of the robot as the motion constraint of the control model and using the control component amplitude constraint as the control constraint;

[0061] Among them, the reward function of the S3 control model is the quadratic weighted sum of state tracking error, collision avoidance distance and action.

[0062] The motion constraint can be expressed by the following formula (2):

[0063] X k+1 =AX k +Bu k (2)

[0064] The control constraint can be expressed by the following formula (3):

[0065] a≤u k ≤b(3)

[0066] Among them, A is the state matrix, B is the input matrix, X k represents the robot state at time k, X k+1 represents the robot state at time k+1; u k is the smooth control action of the robot at time k, where the smooth control action includes velocity increment and angular velocity increment; a = [-0.2, -0.1] is the lower limit of the control action amplitude; b = [0.2, 0.1] is the upper limit of the action amplitude.

[0067] In a feasible implementation, the reward function can be expressed by the following formula (4):

[0068]

[0069] Where r represents the reward function; represents the state tracking error; represents the collision avoidance distance; u represents the action; Q represents the state utility matrix; R represents the action utility matrix. In mathematical form, Q and R are both diagonal positive definite matrices.

[0070] Among them, the state tracking error satisfies the following formula (5):

[0071]

[0072] Among them, x t represents the horizontal coordinate of the robot at time t; y t represents the vertical coordinate of the robot at time t; Indicates the heading angle; Represents the horizontal coordinate of the expected trajectory point at time t; represents the ordinate of the expected trajectory point at time t; represents the heading angle of the desired trajectory point at time t.

[0073] The collision avoidance distance can be obtained by calculating the relative distance between the robot and the nearest obstacle at the current moment. Assuming that the obstacle position is Collision avoidance distance Satisfy the following formula (6)

[0074]

[0075] in, Indicates the horizontal coordinate of the obstacle; Indicates the vertical coordinate of the obstacle.

[0076] S4: Input the current robot's environmental observation sample information, reward function, and constraint conditions into the control model for online training to obtain the smooth control action of the robot at the current moment.

[0077] Optionally, S4 obtains the smooth control action of the robot at the current moment, including:

[0078] S41, processing the current robot's environmental observation sample information through the execution module to obtain the current non-smoothing strategy;

[0079] S42: Processing the current robot's environmental observation sample information through the damping module, outputting the current real-time damping value for each dimensional control action, and obtaining the current smoothing strategy through gradient normalization.

[0080] Among them, the gradient normalization satisfies formula (7):

[0081]

[0082] in, represents the control action of the i-th dimension of the smoothing strategy output; represents the i-th dimension control action output by the original non-smooth strategy; Represents the i-th dimension damping signal output by the damping module; is the gradient vector; ||·|| represents the two-norm of the gradient vector, and ∈ is a small positive number.

[0083] S43: Input the current non-smooth strategy and the current smooth strategy into the evaluation module for processing to obtain the smooth control action of the robot at the current moment.

[0084] S5. Construct an objective function of the control model; perform iterative calculations on the control model based on the objective function to obtain an updated smooth reinforcement learning control model.

[0085] In a feasible implementation, the robot smooth reinforcement learning control model adopts a value distribution flexible execution-evaluation reinforcement learning algorithm, wherein the algorithm adopts an actor-critic structure to learn independent value distribution functions and policy functions.

[0086] Optionally, the objective function of the control model of S5 includes: an original value distribution objective function and a smoothed value distribution objective function.

[0087] Among them, the original value distribution objective function can be expressed by the following formula (8)-formula (11):

[0088]

[0089] Among them, Q is the objective value function of the robot smooth reinforcement learning control model, is the environmental observation sample information at any time in the infinite time domain; u is the vehicle action, θ is the parameter to be optimized in the parameterized target value network, and B is the experience pool, which is used to store the training samples generated by the interaction between the strategy and the environment or the data samples generated by human expert teaching. r is the reward function, is the Bellman self-consistent operator, Represents the k-dimensional environment observation sample information of the robot at time t+1;

[0090]

[0091] in, is a parameterized policy network, that is, at any time Get the action u of the vehicle, ω is the parameter to be optimized in the parameterized strategy, and u′ represents the action of another strategy network π ω′ Output action;

[0092]

[0093] Among them, Z is the cumulative return generated by the flexible strategy π, which is the state-action return, denoted as γ is the discount factor; is the probability density function of the flexible Z value.

[0094] In a feasible implementation, the smoothing value distribution objective function can be expressed by the following formula (12) to formula (15):

[0095]

[0096] Among them, Q is the smoothed target value function of the robot smooth reinforcement learning control model, is the environmental observation sample information at any time in the infinite time domain; u s is the smoothing action of the vehicle, λ is the parameter to be optimized in the parameterized smoothing target value network, and B is the experience pool, which is used to store the training samples generated by the interaction between the smoothing strategy and the environment or the data samples generated by human experts. r is the reward function, is the Bellman self-consistent operator;

[0097]

[0098] in, is a parameterized smoothing strategy, that is, the smoothing action u of the vehicle is obtained from the observation samples at any time s ω is the parameter to be optimized in the parameterized strategy, and u′ represents the parameter of another strategy network. Smooth action of output, is the damping value output by the damping module for the i-th dimension, and K is the damping value output by the damping module, which can be used to approximate the Lipschitz value of the network; is the parameter to be optimized in the parameterized damping module; Z is based on the flexible strategy The cumulative return generated is the state-action return, denoted as

[0099] In a feasible implementation, when the non-smooth policy evaluation module and the smooth policy evaluation module in the robot smooth reinforcement learning control model are updated independently, the update target can be obtained by the above formula (8) and formula (12) respectively, and the corresponding module parameters are updated independently; when the non-smooth policy evaluation module and the smooth policy evaluation module share the same set of parameters, that is, λ = θ, λ represents the evaluation network parameters corresponding to the non-smooth policy network, and θ represents the evaluation network parameters corresponding to the smooth policy network; the evaluation module update target can be obtained by adding formula (8) and formula (12), that is, Evaluation module outputs simultaneously and in is the value function corresponding to the non-smooth strategy, is the value function corresponding to the smoothing strategy.

[0100] The execution module update objective function can be expressed by the following formula (16):

[0101]

[0102] Among them, J π (ω) represents the update objective function of the execution module; Q is given by the non-smooth value distribution function, and α is the policy entropy coefficient;

[0103] The execution module update rule satisfies the following formula (17):

[0104]

[0105] Among them, β c is the learning rate, is the given policy entropy target value.

[0106] The damping module update objective function can be expressed by the following formula (18):

[0107]

[0108] in, represents the target function updated by the damping module; Q is given by the smooth value distribution function, α is the policy entropy coefficient, and the policy entropy coefficient update rule is the same as formula (11), is the damping signal regularization term, and κ is the balancing weight.

[0109] In a feasible implementation, the balance weight can be used to adjust the damping strength output by the damping module. Adjusting the balance weight parameter will take a lot of time, so the embodiment of the present invention constructs a model smoothing ability adjustment mechanism; optionally, the obtained non-smooth strategy and smooth strategy at time t are interacted with the environment to obtain the non-smooth action sequence and smooth action sequence from time t to time t+n, and the corresponding action fluctuation index is calculated, wherein the adaptive adjustment of the balance weight can follow the following formula (19):

[0110]

[0111] Among them, κ′ is the updated balance weight, β d is the learning rate, f(ξ) represents the action fluctuation index corresponding to the non-smooth action sequence from time t to time t+n; f(ξ s ) represents the action fluctuation index corresponding to the smooth action sequence from time t to time t+n; ξ T Indicates the given desired smoothing factor.

[0112] In a feasible implementation, Figure 2Shown is the specific implementation process of iteratively solving the robot smooth reinforcement learning control model.

[0113] Optionally, performing iterative calculations on the control model according to the objective function in S5 includes:

[0114] S51, giving relevant initial conditions and parameter initialization;

[0115] In a feasible implementation, the initial state information of the given robot includes the robot's position, speed and heading angle; the environmental information includes the position, speed and heading angle of the surrounding dynamic or static obstacles; the system control frequency, target position and expected running speed are given by humans or the upper-level trajectory planning module; wherein, the upper-level trajectory planning module is used to plan possible future driving trajectories, which is a conventional technical means and will not be elaborated on in the present invention.

[0116] Among them, the initialization time t; the initialization experience pool B, which is used to store the training samples generated by the interaction between the strategy and the environment; the initialization evaluation network parameters θ, execution network parameters ω, damping network parameters φ and strategy entropy coefficient α; the initialization target network parameters θ'←θ, ω'←ω, φ'←φ; give a suitable learning rate β a , β b , β c , β d , τ and error ∈ = 1e-4; initialize the number of iterations k = 0; give the expected smoothing factor ξ T =2;

[0117] S52, obtaining the observation sample at the current time t;

[0118] In a feasible implementation, the robot state information, surrounding environment information, and navigation information obtained at time t are spliced ​​into the robot's environmental observation sample information

[0119] S53, sampling and calculating the reward function according to the obtained observation samples;

[0120] In a feasible implementation, the robot's smooth control action u obtained at time t interacts with the environment to obtain the robot state and surrounding environment information corresponding to time t+1 and calculate the reward function r; the robot state information, surrounding environment information and navigation information obtained at time t+1 are spliced ​​into the robot's environmental observation sample information The sample Deposit into experience pool;

[0121] S54, solving the evaluation network objective function value according to the reward function;

[0122] In a feasible implementation, multiple experience samples sampled from the experience replay pool are used as the initial values ​​of the forward solution process, and the evaluation network shown in formula (8) and formula (12) is used to calculate the objective function value respectively, providing the initial value for the backward solution evaluation network gradient and the update parameter;

[0123] S55, updating the evaluation network parameters according to the evaluation network objective function value;

[0124] In a feasible implementation, the objective function value obtained by solving is used to calculate the gradient, and after averaging the gradients of the evaluation network calculated in batches, the network parameters are subjected to gradient descent. The update rule of the non-smoothed evaluation network in the Kth iteration can be expressed by the following formulas (20) and (21):

[0125]

[0126] in, Represents the corresponding value function of the non-smooth evaluation network Gradient term: θ K+1 represents the updated non-smoothed evaluation network parameters; θ K represents the parameters of the non-smoothed evaluation network before updating; β a represents the learning rate;

[0127] The update rule of the smoothing evaluation network in the Kth iteration can be expressed by the following formulas (22) and (23):

[0128]

[0129] in, Represents the value function corresponding to the smooth evaluation network Gradient term; λ K+1 represents the updated smoothing evaluation network parameters; λ K represents the smoothing evaluation network parameters before updating;

[0130] S56, calculating the gradient of the objective function corresponding to the execution network, and updating the execution network parameters through back propagation;

[0131] In a feasible implementation, when K is a positive integer multiple of m=2, formula (16) is used to batch calculate the objective function and solve the policy gradient corresponding to the execution network. The averaged policy gradient is used to perform gradient descent on the execution network parameters. The update rule of the execution network in the Kth iteration can be expressed by the following formula (24):

[0132]

[0133] Among them, ω K+1represents the updated execution module network parameters; β b Indicates the network learning rate corresponding to the execution module; ω K Indicates the execution module network parameters before updating;

[0134] S57, calculating the gradient of the objective function corresponding to the damping network, and updating the damping network parameters through back propagation;

[0135] In a feasible implementation, when K is a positive integer multiple of m=2, formula (18) is used to batch calculate the objective function and solve the corresponding gradient of the damping network. The averaged gradient is used to perform gradient descent on the damping network parameters. The update rule of the damping network in the Kth iteration can be expressed by the following formula (25):

[0136]

[0137] in, represents the updated damping module network parameters; β c Represents the learning rate of the damping module network; Represents the damping module network parameters before updating;

[0138] S58. Update the policy entropy coefficient according to the policy entropy coefficient update rule;

[0139] In a feasible implementation, when K is an integer multiple of m=2 which is a positive integer, formula (17) is used for updating;

[0140] S59, updating the target value network and strategy network parameters according to the target evaluation network parameters before updating, the target execution network parameters before updating, and the target damping network parameters before updating;

[0141] In a feasible implementation, when K is an integer multiple of m=2 which is a positive integer, the update is performed according to the following formula (26)-formula (28):

[0142] θ' K+1 =-(1-τ)θ K ′+τθ K (26)

[0143] ω′ K+1 =-(1-τ)ω K ′+τω K (27)

[0144]

[0145] Where τ represents the delay update factor; θ K ′ represents the target evaluation network parameters before updating; θ′ K+1Represents the updated target evaluation network parameters; ω K ′ represents the target execution network parameters before updating; ω′ K+1 Represents the updated target execution network parameters; represents the target damping network parameters before updating; Represents the updated target damping network parameters;

[0146] S60: When it is determined that the network parameter update difference is less than a given threshold, the result is determined to be converged, and the optimal parameterization strategy is output.

[0147] In a feasible implementation, when |J π (ω K+1 )-J π (ω K )|>1e-4, it indicates that the strategy has not converged yet, and the solution is returned to initialization; otherwise, it indicates that the strategy has converged within the error range, and the output result is the optimal parameterized strategy.

[0148] S6. Obtain observation sample information at the next moment based on the smooth control action of the robot at the current moment; input the observation sample information at the next moment into the updated control model to obtain the smooth control action of the robot at the next moment.

[0149] In one feasible implementation, the parameterized smooth policy network obtained through offline training is used as the output of the robot's smooth reinforcement learning control model. When applied online, the smooth policy network can directly output approximate action quantities based on the robot's current state and environmental information.

[0150] Among them, the embodiment of the present invention adopts a value distribution flexible execution-evaluation reinforcement learning algorithm, and adopts a parameterized strategy network obtained by offline training to directly output the optimal control quantity of the robot according to the current state of the robot and environmental information, so that the robot can realize the change of spatial state, and continuously output the control quantity according to the robot state and environmental information, so as to realize the robot's high real-time and high-precision smooth control task.

[0151] Among them, such as Figure 3 This is a schematic diagram of a wheeled robot trajectory tracking and collision avoidance scenario provided by an embodiment of the present disclosure; the embodiment of the present invention uses the optimal parameterized strategy network learned after 190,000 iterations of the robot smooth control model to perform simulation, controlling the wheeled robot to v T =0.4m / s, and the given reference curve y = 0 is tracked. The obtained results are compared with those of the traditional non-smoothing control method, namely the distributed soft actor critic (DSAC), under the same prediction time domain, reference trajectory, and reference speed. Figure 4and Figure 5 As shown, the results show that the actor-critic-damper (ACD) architecture method proposed in the embodiment of the present invention can accurately track both the position and velocity quantities during the entire trajectory tracking and collision avoidance process, and can achieve a controllable smoothing effect and effective smoothing of the action.

[0152] The embodiment of the present invention first obtains the environmental observation sample information of the robot at the current moment; secondly, constructs a smooth reinforcement learning control model based on the environmental observation sample information of the robot at the current moment; constructs a reward function of the control model based on the environmental observation sample information of the robot at the current moment; adopts the discrete-time kinematic equation of the robot to construct the constraint conditions of the control model; inputs the environmental observation sample information, reward function and constraint conditions of the robot at the current moment into the control model for online training to obtain the smooth control action of the robot at the current moment; constructs the objective function of the control model; updates the control model iteratively according to the objective function to obtain an updated smooth reinforcement learning control model; finally, obtains the observation sample information at the next moment based on the smooth control action of the robot at the current moment; inputs the observation sample information at the next moment into the updated control model to obtain the smooth control action of the robot at the next moment.

[0153] The embodiment of the present invention proposes a smooth reinforcement learning control model for robot intelligent control. The core architecture of the model is the actor-critic-damper (ACD) architecture. This architecture is based on the actor-critic (AC) architecture of the existing mainstream reinforcement learning algorithm. By adding a damper module, a new smooth reinforcement learning control architecture is formed, which is particularly oriented to robot intelligent control. Among them, the execution module in the execution-critic-damper architecture corresponds to the traditional non-smooth reinforcement learning strategy, the damper module is responsible for providing the damping strength related to the state and converting the non-smooth action into a smooth action, and the evaluation module is responsible for evaluating the smooth strategy and the non-smooth strategy. Under the assumption that a desired reference trajectory or reference point is given, the machine The robot's own sensors obtain the robot's various state quantities at the current moment, and use this to construct the objective functions of the execution module, damping module and evaluation module, and design the corresponding offline training loss function; the robot's own state and environmental information are used as inputs to the execution module and damping module, and non-smooth actions and smooth actions are used as outputs, converting the online optimization problem into an offline solution of the network parameters of each module; when applied online, the controller directly outputs an approximate smoothing control strategy from the execution module network and the damping module network according to the robot state and surrounding environment information at each moment, and the strategy is executed by the lower-level control program to obtain the robot state at the next moment, realizing online and high-real-time robot smooth control; in addition, the execution-evaluation-damper architecture has controllable smoothing capabilities, and the expected smoothing multiple can be set in advance before training to reduce the time spent on parameter adjustment.

[0154] The embodiment of the present invention proposes an execution module with a stable structure, which can focus on optimizing the original non-smooth strategy, and the damping module focuses on optimizing the smooth strategy; the execution-evaluation-damping architecture has the ability to adaptively adjust the smoothing capability, which can reduce the time spent on parameter adjustment; the embodiment of the present invention is suitable for industrial robots, medical robots and unmanned vehicles, and has the advantages of high computational efficiency, high precision and memory saving, breaking through the bottleneck of motion jitter in the existing reinforcement learning application in robot control.

[0155] Figure 6 This is a block diagram of a smooth reinforcement learning device for robot intelligent control according to an exemplary embodiment. The device is used for a smooth reinforcement learning method for robot intelligent control. Figure 6 The device includes a first acquisition unit 310, a first construction unit 320, a second construction unit 330, a second acquisition unit 340, a third acquisition unit 350, and a fourth acquisition unit 360.

[0156] The first acquisition unit 310 is used to obtain the environment observation sample information of the robot at the current moment;

[0157] A first construction unit 320 is configured to construct a smooth reinforcement learning control model based on the current environment observation sample information of the robot;

[0158] The second construction unit 330 is configured to construct a reward function of the control model based on the current environment observation sample information of the robot; and to construct the constraint conditions of the control model using the discrete-time kinematic equations of the robot;

[0159] The second acquisition unit 340 is configured to input the current robot's environmental observation sample information, the reward function, and the constraint conditions into the control model for online training to obtain a smooth control action of the robot at the current moment;

[0160] The third acquisition unit 350 is used to construct an objective function of the control model; perform iterative calculations on the control model according to the objective function to obtain an updated smooth reinforcement learning control model;

[0161] The fourth acquisition unit 360 is used to obtain the observation sample information at the next moment based on the smooth control action of the robot at the current moment; input the observation sample information at the next moment into the updated control model to obtain the smooth control action of the robot at the next moment.

[0162] Optionally, the first acquiring unit 310 is configured to:

[0163] Obtain the current robot status information, current surrounding environment information, and current robot navigation information;

[0164] The robot's current state information, the current surrounding environment information, and the current robot navigation information are spliced ​​together to obtain the robot's current environmental observation sample information.

[0165] Optionally, the smooth reinforcement learning control model includes: an execution module, a damper module and an evaluation module; wherein,

[0166] The execution module is used to correspond to the traditional non-smooth reinforcement learning strategy;

[0167] The damper module is used to provide a state-dependent damping strength to convert a non-smooth motion into a smooth motion;

[0168] The evaluation module is used to evaluate the evaluation value obtained by the smoothing strategy and the evaluation value obtained by the non-smoothing strategy.

[0169] Optionally, the use of the discrete-time kinematic equation of the robot to construct the constraint conditions of the control model includes: using the discrete-time kinematic equation of the robot as the motion constraint of the control model, and using the control component amplitude constraint as the control constraint.

[0170] The reward function of the control model is a quadratic weighted sum of state tracking error, collision avoidance distance, and action.

[0171] Optionally, obtaining the smooth control action of the robot at the current moment includes:

[0172] The execution module processes the robot's current environmental observation sample information to obtain the current non-smoothing strategy;

[0173] The damping module processes the robot's current environmental observation sample information, outputs the current real-time damping value for each dimensional control action, and obtains the current smoothing strategy through gradient normalization.

[0174] The non-smooth strategy at the current moment and the smooth strategy at the current moment are input into the evaluation module for processing to obtain the smooth control action of the robot at the current moment.

[0175] Optionally, the objective function of the control model includes: an original value distribution objective function and a smoothed value distribution objective function.

[0176] Optionally, performing iterative calculations on the control model according to the objective function includes:

[0177] Given relevant initial conditions and parameter initialization;

[0178] Get the observation sample at the current time t;

[0179] Sampling and calculating reward functions based on the obtained observation samples;

[0180] Solve the evaluation network objective function value based on the reward function;

[0181] Update the evaluation network parameters according to the evaluation network objective function value;

[0182] Calculate the gradient of the objective function corresponding to the execution network and update the execution network parameters through backpropagation;

[0183] Calculate the gradient of the objective function corresponding to the damping network and update the damping network parameters through back propagation;

[0184] Update the policy entropy coefficient according to the policy entropy coefficient update rule;

[0185] Update the target value network and strategy network parameters according to the target evaluation network parameters before updating, the target execution network parameters before updating, and the target damping network parameters before updating;

[0186] When the difference in network parameter updates is judged to be less than a given threshold, the results are judged to be converged and the optimal parameterization strategy is output.

[0187] The embodiment of the present invention first obtains the environmental observation sample information of the robot at the current moment; secondly, constructs a smooth reinforcement learning control model based on the environmental observation sample information of the robot at the current moment; constructs a reward function of the control model based on the environmental observation sample information of the robot at the current moment; adopts the discrete-time kinematic equation of the robot to construct the constraint conditions of the control model; inputs the environmental observation sample information, reward function and constraint conditions of the robot at the current moment into the control model for online training to obtain the smooth control action of the robot at the current moment; constructs the objective function of the control model; updates the control model iteratively according to the objective function to obtain an updated smooth reinforcement learning control model; finally, obtains the observation sample information at the next moment based on the smooth control action of the robot at the current moment; inputs the observation sample information at the next moment into the updated control model to obtain the smooth control action of the robot at the next moment.

[0188] The embodiment of the present invention proposes a smooth reinforcement learning control model for robot intelligent control. The core architecture of the model is the actor-critic-damper (ACD) architecture. This architecture is based on the actor-critic (AC) architecture of the existing mainstream reinforcement learning algorithm. By adding a damper module, a new smooth reinforcement learning control architecture is formed, which is particularly oriented to robot intelligent control. Among them, the execution module in the execution-critic-damper architecture corresponds to the traditional non-smooth reinforcement learning strategy, the damper module is responsible for providing the damping strength related to the state and converting the non-smooth action into a smooth action, and the evaluation module is responsible for evaluating the smooth strategy and the non-smooth strategy. Under the assumption that a desired reference trajectory or reference point is given, the machine The robot's own sensors obtain the robot's various state quantities at the current moment, and use this to construct the objective functions of the execution module, damping module and evaluation module, and design the corresponding offline training loss function; the robot's own state and environmental information are used as inputs to the execution module and damping module, and non-smooth actions and smooth actions are used as outputs, converting the online optimization problem into an offline solution of the network parameters of each module; when applied online, the controller directly outputs an approximate smoothing control strategy from the execution module network and the damping module network according to the robot state and surrounding environment information at each moment, and the strategy is executed by the lower-level control program to obtain the robot state at the next moment, realizing online and high-real-time robot smooth control; in addition, the execution-evaluation-damper architecture has controllable smoothing capabilities, and the expected smoothing multiple can be set in advance before training to reduce the time spent on parameter adjustment.

[0189] The embodiment of the present invention proposes an execution module with a stable structure, which can focus on optimizing the original non-smooth strategy, and the damping module focuses on optimizing the smooth strategy; the execution-evaluation-damping architecture has the ability to adaptively adjust the smoothing capability, which can reduce the time spent on parameter adjustment; the embodiment of the present invention is suitable for industrial robots, medical robots and unmanned vehicles, and has the advantages of high computational efficiency, high precision and memory saving, breaking through the bottleneck of motion jitter in the existing reinforcement learning application in robot control.

[0190] Figure 7 Schematic diagram of a smooth reinforcement learning device for robot intelligent control provided by an embodiment of the present invention. Figure 7 As shown, the smooth reinforcement learning device for robot intelligent control may include the above Figure 6 Optionally, the smooth reinforcement learning device 410 for robot intelligent control may include a first processor 2001 .

[0191] Optionally, the smooth reinforcement learning device 410 for robot intelligent control may further include a memory 2002 and a transceiver 2003 .

[0192] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0193] The following combination Figure 7 The components of the smooth reinforcement learning device 410 for intelligent robot control are described in detail:

[0194] The first processor 2001 is the control center of the smooth reinforcement learning device 410 for intelligent robot control, and can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), or application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs) or one or more field programmable gate arrays (FPGAs).

[0195] Optionally, the first processor 2001 can perform various functions of the smooth reinforcement learning device 410 for robot intelligent control by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0196] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 7 CPU0 and CPU1 are shown in FIG.

[0197] In a specific implementation, as an embodiment, the smooth reinforcement learning device 410 for intelligent control of robots may also include multiple processors, such as Figure 7 1 and 2. The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0198] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled by the first processor 2001 for execution. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0199] Alternatively, the memory 2002 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001 or exist independently and access the memory 2002 through the interface circuit ( Figure 7 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0200] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0201] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 7 (not shown separately in the figure). The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0202] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or can exist independently and communicate with the first processor 2001 through the interface circuit ( Figure 7 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0203] It should be noted that Figure 7 The structure of the smooth reinforcement learning device 410 for intelligent control of robots shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0204] In addition, the technical effects of the smooth reinforcement learning device 410 for robot intelligent control can refer to the technical effects of the smooth reinforcement learning method for robot intelligent control described in the above method embodiment, and will not be repeated here.

[0205] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0206] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0207] The above embodiments can be implemented in whole or in part through software, hardware (such as circuits), firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired method (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, or magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0208] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0209] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0210] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0211] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0212] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0213] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.

[0214] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0215] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0216] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0217] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A smooth reinforcement learning method for robot intelligent control, characterized in that: The method comprises: S1. Obtain the current robot's environmental observation sample information; The step of obtaining the robot's current environmental observation sample information includes: S11, obtaining the current robot state information, the current surrounding environment information, and the current robot navigation information; S12, combining the robot state information at the current moment, the surrounding environment information at the current moment, and the robot navigation information at the current moment to obtain the robot's environmental observation sample information at the current moment; S2. Constructing a smooth reinforcement learning control model based on the current robot's environmental observation sample information; The smooth reinforcement learning control model includes: an execution module, a damper module and an evaluation module; wherein, The execution module is used to correspond to the traditional non-smooth reinforcement learning strategy; The damper module is used to provide a state-dependent damping strength to convert a non-smooth motion into a smooth motion; The evaluation module is used to evaluate the evaluation value obtained by the smoothing strategy and the evaluation value obtained by the non-smoothing strategy; S3. Constructing a reward function of the control model based on the current robot's environmental observation sample information; and constructing constraint conditions of the control model using the robot's discrete-time kinematic equations. Wherein, the step S3 adopts the discrete time kinematic equation of the robot to construct the constraint conditions of the control model, including: adopting the discrete time kinematic equation of the robot as the motion constraint of the control model, and using the control component amplitude constraint as the control constraint; The reward function of the control model of S3 is the quadratic weighted sum of the state tracking error, the collision avoidance distance, and the action; S4, inputting the current robot's environmental observation sample information, the reward function, and the constraint conditions into the control model for online training to obtain a smooth control action of the robot at the current moment; The step S4 of obtaining the smooth control action of the robot at the current moment includes: S41, processing the current robot's environmental observation sample information through the execution module to obtain the current non-smoothing strategy; S42: Processing the current robot's environmental observation sample information through the damping module, outputting the current real-time damping value for each dimensional control action, and obtaining the current smoothing strategy through gradient normalization. S43, inputting the current non-smooth strategy and the current smooth strategy into an evaluation module for processing to obtain a smooth control action of the robot at the current moment; S5. Constructing an objective function of the control model; performing iterative update calculations on the control model according to the objective function to obtain an updated smooth reinforcement learning control model; S6. Obtain observation sample information at the next moment based on the smooth control action of the robot at the current moment; input the observation sample information at the next moment into the updated control model to obtain the smooth control action of the robot at the next moment.

2. The smooth reinforcement learning method for robot intelligent control according to claim 1, characterized in that: The objective function of the control model of S5 includes: an original value distribution objective function and a smoothed value distribution objective function.

3. The smooth reinforcement learning method for robot intelligent control according to claim 1, characterized in that: The step S5 of performing iterative calculations on the control model according to the objective function includes: S51, giving relevant initial conditions and parameter initialization; S52, get the current Observe samples at all times; S53, sampling and calculating the reward function according to the obtained observation samples; S54, solving the evaluation network objective function value according to the reward function; S55, updating the evaluation network parameters according to the evaluation network objective function value; S56, calculating the gradient of the objective function corresponding to the execution network, and updating the execution network parameters through back propagation; S57, calculating the gradient of the objective function corresponding to the damping network, and updating the damping network parameters through back propagation; S58. Update the policy entropy coefficient according to the policy entropy coefficient update rule; S59, updating the target value network and strategy network parameters according to the target evaluation network parameters before updating, the target execution network parameters before updating, and the target damping network parameters before updating; S60: When it is determined that the network parameter update difference is less than a given threshold, the result is determined to be converged, and the optimal parameterization strategy is output.

4. A smooth reinforcement learning device for robot intelligent control, wherein the smooth reinforcement learning device for robot intelligent control is used to implement the smooth reinforcement learning method for robot intelligent control according to any one of claims 1 to 3, characterized in that: The device comprises: The first acquisition unit is used to obtain the environment observation sample information of the robot at the current moment; A first construction unit is configured to construct a smooth reinforcement learning control model based on the environment observation sample information of the robot at the current moment; A second construction unit is configured to construct a reward function of the control model based on the environment observation sample information of the robot at the current moment; and to construct the constraint conditions of the control model using the discrete-time kinematic equation of the robot; A second acquisition unit is configured to input the current robot's environmental observation sample information, the reward function, and the constraint conditions into the control model for online training to obtain a smooth control action of the robot at the current moment; A third acquisition unit is configured to construct an objective function of the control model; perform iterative update calculations on the control model according to the objective function to obtain an updated smooth reinforcement learning control model; The fourth acquisition unit is used to obtain the observation sample information at the next moment based on the smooth control action of the robot at the current moment; input the observation sample information at the next moment into the updated control model to obtain the smooth control action of the robot at the next moment.

5. A smooth reinforcement learning device for robot intelligent control, characterized in that: The smooth reinforcement learning device for robot intelligent control includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, which can be called by a processor to execute the method according to any one of claims 1 to 3.

Images