Smooth reinforcement learning method and device for robot intelligent control

Abstract

The invention provides a smooth reinforcement learning method and device for robot intelligent control, and relates to the technical field of robot intelligent control. The method comprises the following steps: acquiring environment observation sample information of a robot at the current moment; constructing a smooth reinforcement learning control model according to the current sample information; constructing a reward function of the control model according to the sample information; constructing constraint conditions of the control model by adopting a discrete time kinematics equation of the robot; inputting the sample information, the reward function and the constraint condition into a control model for online training to obtain a smooth control action of the robot at the current moment; constructing a target function of the control model; updating the control model according to the target function to obtain an updated control model; and obtaining observation sample information at the next moment according to the smooth control action, and inputting the observation sample information into the updated control model to obtain the smooth control action of the robot at the next moment. According to the invention, the robot can complete a high-real-time and high-precision smooth control task.

Description

Technical Field

[0001] The present invention relates to the technical field of robot intelligent control, and in particular to a smooth reinforcement learning method and device for robot intelligent control. Background Art

[0002] A robot is a machine device that can automatically perform tasks through programming control. According to its use, it can be divided into industrial robots used in industrial production and manufacturing, medical robots used in the medical field, and mobile robots with mobility and autonomous task execution capabilities, such as driverless cars and drones. Realizing intelligent control of robots has great potential in reducing enterprise production costs, reducing production safety accidents, and improving production efficiency.

[0003] Modern robots are usually equipped with sensors, actuators and complex control systems, which enable them to perceive the environment, process information and autonomously perform corresponding actions according to pre-set tasks. The existing robot control methods mainly have the following defects: On the one hand: some typical control methods such as proportional-integral-differential control and linear quadratic regulation control, although simple in structure, cannot handle complex system constraints or ignore the nonlinear characteristics of the system, so it is often difficult to achieve ideal control effects in practical applications. On the other hand: for predictive control methods, the controller needs to solve the optimal control sequence for each sampling period online based on the prediction model and constraints; when the difficulty of solving increases and the computing resources are insufficient, the speed of solving is difficult to meet the real-time control requirements of the controller. The above control method is essentially to solve the optimal solution of the optimization control problem online under certain constraint rules, which is difficult to adapt to the more efficient and intelligent control needs of future robots.

[0004] In recent years, with the rapid development of artificial intelligence and machine learning technologies, learning-based decision-making schemes represented by reinforcement learning have shown great potential in the field of robot control. Reinforcement learning can learn an optimal strategy through autonomous interaction with the environment, thereby maximizing future cumulative rewards. Reinforcement learning has the advantages of strong adaptability, autonomous learning, processing high-dimensional state-action space and nonlinear problems, and its offline solution and online application mode also makes the reinforcement learning method have a high online solution efficiency. However, the control strategy obtained by reinforcement learning has serious motion jitter problems in the actual application of robots. The existence of motion jitter problems will accelerate the wear of mechanical parts, cause safety hazards and affect the user experience. Generally speaking, the cause of motion jitter can be attributed to the rough surface characteristics of the neural network as a strategy carrier and the input disturbance caused by the sensor measurement noise contained in the neural network input. The existence of the motion jitter problem has, to a certain extent, restricted the further application of reinforcement learning in the intelligent control of robots. Summary of the invention

[0005] In order to solve the technical problems of motion jitter existing in the existing robot neural network decision-making control technology and the difficulty in finding the optimal solution to the optimization control problem online under certain constraint rules, and the difficulty in adapting to the more efficient and intelligent control needs of future robots, the embodiment of the present invention provides a smooth reinforcement learning method and device for robot intelligent control. The technical solution is as follows:

[0006] On the one hand, a smooth reinforcement learning method for robot intelligent control is provided, the method is implemented by a smooth reinforcement learning device for robot intelligent control, and the method includes:

[0007] S1. Obtain the current robot's environmental observation sample information;

[0008] S2. Constructing a smooth reinforcement learning control model based on the environmental observation sample information of the robot at the current moment;

[0009] S3. Constructing a reward function of the control model based on the environment observation sample information of the robot at the current moment; and constructing constraint conditions of the control model using the discrete time kinematic equation of the robot;

[0010] S4, inputting the current robot's environmental observation sample information, the reward function and the constraint condition into the control model for online training to obtain a smooth control action of the robot at the current moment;

[0011] S5, constructing the objective function of the control model; performing update iterative calculation on the control model according to the objective function to obtain an updated smooth reinforcement learning control model;

[0012] S6. Obtain observation sample information at the next moment according to the smooth control action of the robot at the current moment; input the observation sample information at the next moment into the updated control model to obtain the smooth control action of the robot at the next moment.

[0013] On the other hand, a smooth reinforcement learning device for robot intelligent control is provided, which is applied to a smooth reinforcement learning method for robot intelligent control, and the device comprises:

[0014] The first acquisition unit is used to acquire the environment observation sample information of the robot at the current moment;

[0015] A first construction unit is used to construct a smooth reinforcement learning control model according to the environmental observation sample information of the robot at the current moment;

[0016] The second construction unit is used to construct a reward function of the control model according to the environment observation sample information of the robot at the current moment; and to construct the constraint conditions of the control model using the discrete time kinematic equation of the robot;

[0017] A second acquisition unit is used to input the environment observation sample information of the robot at the current moment, the reward function and the constraint condition into the control model for online training to obtain a smooth control action of the robot at the current moment;

[0018] A third acquisition unit is used to construct an objective function of the control model; update and iterate the control model according to the objective function to obtain an updated smooth reinforcement learning control model;

[0019] The fourth acquisition unit is used to obtain the observation sample information at the next moment according to the smooth control action of the robot at the current moment; input the observation sample information at the next moment into the updated control model to obtain the smooth control action of the robot at the next moment.

[0020] On the other hand, a smooth reinforcement learning device for robot intelligent control is provided, the smooth reinforcement learning device for robot intelligent control comprising: a processor; a memory, the memory storing computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned smooth reinforcement learning methods for robot intelligent control is implemented.

[0021] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned smooth reinforcement learning methods for robot intelligent control.

[0022] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0023] The embodiment of the present invention first obtains the environmental observation sample information of the robot at the current moment; secondly, constructs a smooth reinforcement learning control model according to the environmental observation sample information of the robot at the current moment; constructs a reward function of the control model according to the environmental observation sample information of the robot at the current moment; adopts the discrete-time kinematic equation of the robot to construct the constraint conditions of the control model; inputs the environmental observation sample information, reward function and constraint conditions of the robot at the current moment into the control model for online training to obtain the smooth control action of the robot at the current moment; constructs the objective function of the control model; updates the control model iteratively according to the objective function to obtain the updated smooth reinforcement learning control model; finally, according to the smooth control action of the robot at the current moment, obtains the observation sample information at the next moment; inputs the observation sample information at the next moment into the updated control model to obtain the smooth control action of the robot at the next moment.

[0024] The embodiment of the present invention proposes a smooth reinforcement learning control model for robot intelligent control. The core architecture of the model is an actor-critic-damper (ACD) architecture. The architecture is based on the actor-critic (AC) architecture of the existing reinforcement learning algorithm. By adding a damper module, a new smooth reinforcement learning control architecture is formed, which is particularly oriented to robot intelligent control. Among them, the execution module in the execution-evaluation-damper architecture corresponds to the traditional non-smooth reinforcement learning strategy, the damper module is responsible for providing the damping strength related to the state and converting the non-smooth action into a smooth action, and the evaluation module is responsible for evaluating the smooth strategy and the non-smooth strategy. Under the assumption that a desired reference trajectory or reference point is given, the machine The robot's own sensors obtain the robot's various state quantities at the current moment, and use them to construct the objective functions of the execution module, damping module and evaluation module, and design the corresponding offline training loss function; the robot's own state and environmental information are used as the input of the execution module and damping module, and non-smooth actions and smooth actions are used as outputs, and the online optimization problem is converted into an offline solution of the network parameters of each module; when applied online, the controller directly outputs an approximate smoothing control strategy from the execution module network and the damping module network according to the robot state and surrounding environment information at each moment, and the strategy is executed by the lower-level control program to obtain the robot state at the next moment, thereby realizing online and high-real-time robot smooth control; in addition, the execution-evaluation-damper architecture has controllable smoothing capabilities, and the expected smoothing multiples can be set in advance before training to reduce the time spent on parameter adjustment.

[0025] The embodiment of the present invention proposes an execution module with a stable structure, which can focus on optimizing the original non-smooth strategy, and the damping module can focus on optimizing the smooth strategy; the execution-evaluation-damping architecture has the ability to adaptively adjust the smoothing capability, which can reduce the time spent on parameter adjustment; the embodiment of the present invention is suitable for industrial robots, medical robots and unmanned vehicles, and has the advantages of high computing efficiency, high precision and memory saving, breaking through the bottleneck of motion jitter in the existing reinforcement learning application in robot control. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 It is a flow chart of a smooth reinforcement learning method for robot intelligent control provided by an embodiment of the present invention;

[0028] Figure 2 It is a schematic diagram of the overall process flow of a smooth reinforcement learning method for robot intelligent control provided by an embodiment of the present invention;

[0029] Figure 3 is a schematic diagram of a wheeled robot trajectory tracking and collision avoidance scenario provided by an embodiment of the present invention;

[0030] Figure 4 It is a schematic diagram for comparing the motion fluctuations of the smooth reinforcement learning method for robot intelligent control provided by an embodiment of the present invention;

[0031] Figure 5 It is a schematic diagram of the trajectory tracking and collision avoidance effect of the smooth reinforcement learning method for robot intelligent control provided by an embodiment of the present invention;

[0032] Figure 6 It is a block diagram of a smooth reinforcement learning device for robot intelligent control provided by an embodiment of the present invention;

[0033] Figure 7 It is a structural schematic diagram of a smooth reinforcement learning device for robot intelligent control provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0035] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0036] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference between them is not emphasized, the meanings they intend to express are the same.

[0037] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0038] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0039] The embodiment of the present invention provides a smooth reinforcement learning method for robot intelligent control, which can be implemented by a smooth reinforcement learning device for robot intelligent control, and the smooth reinforcement learning device for robot intelligent control can be a terminal or a server. Figure 1 The flow chart of the smooth reinforcement learning method for robot intelligent control shown in FIG. 1 may include the following steps:

[0040] S1. Obtain the robot's current environmental observation sample information.

[0041] Optionally, the specific implementation of S1 may include S11-S12:

[0042] S11, obtaining the current state information of the robot, the current surrounding environment information and the current navigation information of the robot;

[0043] The robot's own coordinate system is established with the robot's position at time t as the origin, and the robot's current state information is obtained according to the sensor. The current state information includes: the robot's horizontal and vertical positions, longitudinal speed and heading angle at time t.

[0044] Among them, in the robot self-vehicle coordinate system, the surrounding environment information at time t is obtained, and the surrounding environment information includes: the horizontal and vertical positions, longitudinal speed and heading angle of the robot's environment at time t and the nearest obstacle to the robot.

[0045] Among them, in the robot self-vehicle coordinate system, the navigation information of the robot at time t is obtained, that is, the expected trajectory point information of the robot movement, the expected trajectory point information includes multiple reference points, and each reference point information includes: the position information of the expected reference point, the expected speed information and the heading angle information.

[0046] S12: Combine the robot state information at the current moment, the surrounding environment information at the current moment, and the robot navigation information at the current moment to obtain the robot's environmental observation sample information at the current moment.

[0047] In a feasible implementation, the robot state information, surrounding environment information and navigation information acquired at time t are spliced ​​into the robot's environmental observation sample information, which can be expressed by the following formula (1):

[0048]

[0049] in, Represents the k-dimensional environment observation sample information of the robot at the current time t; Represents the m-dimensional state information of the robot at the current time t; Represents the b-dimensional environment information around the robot at the current time t; Represents the p-dimensional navigation information of the robot at the current time t; represents a real number; wherein, k=m+b+p, k, m, b and p represent the dimensions of the vector.

[0050] S2. Construct a smooth reinforcement learning control model based on the robot’s current environmental observation sample information.

[0051] Among them, reinforcement learning includes environment, agent, state, observation and action. That is, the agent interacts with the environment based on the current state and action to obtain the state of the agent at the next moment. At the same time, the reward is calculated based on the state and action of the agent, and the relevant objective function is constructed using the reward. The obtained objective function is used to improve the agent's strategy, and finally the total reward is maximized.

[0052] In the embodiment of the present invention, the environment can be a system for state deduction of objects therein based on laws such as physical laws, kinematics and geometric constraints, and is composed of intelligent agents, obstacles and map constraints; the intelligent agent is an object element in the environment that can act autonomously, accept observations and give actions, such as an autonomous driving vehicle; the state is the basic attribute of the elements in the environment, such as the vehicle position, orientation and speed; the observation is the feature obtained by encoding the state of other elements in the environment from the perspective of the given state of the intelligent agent; the action is the output of the intelligent agent, that is, it is determined by the strategy of the intelligent agent; the deduction is to iteratively apply the state transition model given the action sequence and the initial state of the object, continuously obtain the state of each step, and finally obtain the state sequence as the output, wherein the state transition model is defined on the object, given the state and action of the object at the current moment, the output object will transfer to the state at the next moment, which is determined by laws such as kinematic constraints; the reward is used to represent the evaluation of the state and action of the elements in the environment at the current time step, which is a scalar; the cumulative reward is the sum of the rewards at all time steps starting from the current state until the end of the deduction.

[0053] Among them, building a robot smooth control model under the reinforcement learning framework is to let the robot interact with the environment to learn an optimal smooth control strategy, so that the objective function obtained by executing the smooth control strategy from the initial moment is maximized; the robot smooth control model includes a policy network and a state transition model. The policy network outputs the optimal smooth action in the current state based on the input vehicle state and environmental information; the state transition model outputs the state of the robot at the next moment given the state and action of the robot at the current moment, which is determined by laws such as kinematic constraints.

[0054] Among them, the core of the smooth reinforcement learning control model for robot intelligent control lies in the construction of an execution-evaluation-damping architecture; the execution-evaluation-damping architecture is suitable for the current mainstream execution-evaluation architecture reinforcement learning algorithm; the three modules in the execution-evaluation-damping architecture can be approximated using currently commonly used neural network functions, including multi-layer perceptron MLP, convolutional neural network CNN and recurrent neural network RNN, etc.; the embodiment of the present invention takes the multi-layer perceptron MLP as an example, and the execution module, evaluation module and damping module in the smooth reinforcement learning control model for wheeled robots all use the multi-layer perceptron as the approximate function, the hidden layer scale of the three modules is 256×256, and the activation function of the three modules all selects the Relu function.

[0055] Optionally, the smooth reinforcement learning control model includes: an execution module, a damper module and an evaluation module; wherein,

[0056] Execution module, used to correspond to the traditional non-smooth reinforcement learning strategy;

[0057] The damper module is used to provide state-dependent damping strength to transform non-smooth motion into smooth motion;

[0058] The evaluation module is used to evaluate the evaluation value obtained by the smoothing strategy and the evaluation value obtained by the non-smoothing strategy.

[0059] S3. Construct the reward function of the control model based on the robot's environmental observation sample information at the current moment; use the robot's discrete-time kinematic equation to construct the constraint conditions of the control model.

[0060] Optionally, S3 adopts the discrete time kinematic equation of the robot to construct the constraint conditions of the control model, including: adopting the discrete time kinematic equation of the robot as the motion constraint of the control model and taking the control component amplitude constraint as the control constraint;

[0061] Among them, the reward function of the control model of S3 is the quadratic weighted sum of state tracking error, collision avoidance distance and action.

[0062] The motion constraint can be expressed by the following formula (2):

[0063] X k+1 =AX k +Bu k (2)

[0064] The control constraint can be expressed by the following formula (3):

[0065] a≤u k ≤b(3)

[0066] Among them, A is the state matrix, B is the input matrix, X k represents the robot state at time k, X k+1 represents the robot state at time k+1; u k is the smooth control action of the robot at time k, where the smooth control action includes velocity increment and angular velocity increment; a = [-0.2, -0.1] is the lower limit of the control action amplitude; b = [0.2, 0.1] is the upper limit of the action amplitude.

[0067] In a feasible implementation, the reward function can be expressed by the following formula (4):

[0068]

[0069] Where r represents the reward function; represents the state tracking error; represents the collision avoidance distance; u represents the action; Q represents the state utility matrix; R represents the action utility matrix. In mathematical form, Q and R are both diagonal positive definite matrices.

[0070] Among them, the state tracking error satisfies the following formula (5):

[0071]

[0072] Among them, x t represents the horizontal coordinate of the robot at time t; y t represents the vertical coordinate of the robot at time t; Indicates the heading angle; Represents the horizontal coordinate of the expected trajectory point at time t; Represents the ordinate of the expected trajectory point at time t; represents the heading angle of the desired trajectory point at time t.

[0073] The collision avoidance distance can be obtained by calculating the relative distance between the robot and the nearest obstacle at the current moment. Assuming that the obstacle is located at Collision avoidance distance Satisfies the following formula (6)

[0074]

[0075] in, Indicates the horizontal coordinate of the obstacle; Indicates the vertical coordinate of the obstacle.

[0076] S4: Input the current robot's environmental observation sample information, reward function and constraint conditions into the control model for online training to obtain the smooth control action of the robot at the current moment.

[0077] Optionally, S4 obtains the smooth control action of the robot at the current moment, including:

[0078] S41, processing the environment observation sample information of the robot at the current moment through the execution module to obtain the non-smooth strategy at the current moment;

[0079] S42, processing the environment observation sample information of the robot at the current moment through the damping module, outputting the real-time damping value at the current moment for each dimensional control action, and obtaining the smoothing strategy at the current moment through gradient normalization processing;

[0080] Among them, the gradient normalization satisfies formula (7):

[0081]

[0082] in, represents the i-th dimension control action of the smoothing strategy output; represents the i-th dimension control action of the original non-smooth strategy output; represents the i-th dimension damping signal output by the damping module; is the gradient vector; ||·|| represents the binary norm of the gradient vector, and ∈ is a small positive number.

[0083] S43, inputting the current non-smooth strategy and the current smooth strategy into the evaluation module for processing to obtain the smooth control action of the robot at the current moment.

[0084] S5. Construct an objective function of the control model; perform iterative calculations on the control model according to the objective function to obtain an updated smooth reinforcement learning control model.

[0085] In a feasible implementation, the robot smooth reinforcement learning control model adopts a value distribution flexible execution-evaluation reinforcement learning algorithm, wherein the algorithm adopts an actor-critic structure to learn independent value distribution functions and policy functions.

[0086] Optionally, the objective function of the control model of S5 includes: an original value distribution objective function and a smoothed value distribution objective function.

[0087] Among them, the original value distribution objective function can be expressed by the following formula (8)-formula (11):

[0088]

[0089] Among them, Q is the objective value function of the robot smooth reinforcement learning control model, is the sample information of environmental observation at any time in the infinite time domain; u is the action of the vehicle, θ is the parameter to be optimized in the parameterized target value network, and B is the experience pool, which is used to store the training samples generated by the interaction between the strategy and the environment or the data samples generated by the teaching of human experts. r is the reward function, is a Bellman self-consistent operator, Represents the k-dimensional environment observation sample information of the robot at time t+1;

[0090]

[0091] in, is a parameterized strategy network, that is, Get the action u of the vehicle, ω is the parameter to be optimized in the parameterized strategy, and u′ represents the action of the vehicle by another strategy network π ω′ Output action;

[0092]

[0093] Among them, Z is the cumulative return generated based on the flexible strategy π, which is the state-action return, denoted as γ is the discount factor; is the probability density function of the flexible Z value.

[0094] In a feasible implementation, the smoothing value distribution objective function can be expressed by the following formula (12)-formula (15):

[0095]

[0096] Among them, Q is the smoothed target value function of the robot smooth reinforcement learning control model, is the environmental observation sample information at any time in the infinite time domain; u s is the smoothing action of the vehicle, λ is the parameter to be optimized in the parameterized smoothing target value network, and B is the experience pool, which is used to store the training samples generated by the interaction between the smoothing strategy and the environment or the data samples generated by the teaching of human experts. r is the reward function, is the Bellman self-consistent operator;

[0097]

[0098] in, is a parameterized smoothing strategy, that is, the smoothing action u of the vehicle is obtained from the observation samples at any time s ω is the parameter to be optimized in the parameterized strategy, and u′ represents the parameter of another strategy network. Smooth action of output, is the damping value output by the damping module for the i-th dimension, K is the damping value output by the damping module, which can be used to approximate the Lipschitz value of the network; is the parameter to be optimized in the parameterized damping module; Z is based on the flexible strategy The cumulative return generated is the state-action return, denoted as

[0099] In a feasible implementation, when the non-smooth policy evaluation module and the smooth policy evaluation module in the robot smooth reinforcement learning control model are updated independently, the update target can be obtained by the above formula (8) and formula (12) respectively, and the corresponding module parameters are updated independently; when the non-smooth policy evaluation module and the smooth policy evaluation module share the same set of parameters, that is, λ=θ, λ represents the evaluation network parameters corresponding to the non-smooth policy network, and θ represents the evaluation network parameters corresponding to the smooth policy network; the update target of the evaluation module can be obtained by adding formula (8) and formula (12), that is, Evaluation module outputs simultaneously and in is the value function corresponding to the non-smoothing strategy, is the value function corresponding to the smoothing strategy.

[0100] The execution module update objective function can be expressed by the following formula (16):

[0101]

[0102] Among them, J π (ω) represents the update objective function of the execution module; Q is given by the non-smooth value distribution function, and α is the policy entropy coefficient;

[0103] The execution module update rule satisfies the following formula (17):

[0104]

[0105] Among them, β c is the learning rate, is the given policy entropy target value.

[0106] Among them, the damping module update objective function can be expressed by the following formula (18):

[0107]

[0108] in, represents the damping module update objective function; Q is given by the smoothing value distribution function, α is the policy entropy coefficient, and the policy entropy coefficient update rule is the same as formula (11), is the damping signal regularization term, and κ is the balance weight.

[0109] In a feasible implementation, the balance weight can be used to adjust the damping strength output by the damping module. Adjusting the balance weight parameter will take a lot of time, so the embodiment of the present invention constructs a model smoothing ability adjustment mechanism; optionally, the obtained non-smooth strategy and smooth strategy at time t interact with the environment to obtain the non-smooth action sequence and smooth action sequence from time t to time t+n, and calculate the corresponding action fluctuation index, wherein the adaptive adjustment of the balance weight can follow the following formula (19):

[0110]

[0111] Among them, κ′ is the updated balance weight, β d is the learning rate, f(ξ) represents the action fluctuation index corresponding to the non-smooth action sequence from time t to time t+n; f(ξ s ) represents the action fluctuation index corresponding to the smooth action sequence from time t to time t+n; ξ T Indicates a given desired smoothing factor.

[0112] In a feasible implementation, Figure 2Shown is the specific implementation process of iteratively solving the robot smooth reinforcement learning control model.

[0113] Optionally, performing iterative calculation on the control model according to the objective function in S5 includes:

[0114] S51, giving relevant initial conditions and parameter initialization;

[0115] In a feasible implementation, the initial state information of a given robot includes the robot's position, speed and heading angle; the environment information includes the position, speed and heading angle of surrounding dynamic or static obstacles; the system control frequency, target position and expected running speed are given by humans or an upper-level trajectory planning module; wherein the upper-level trajectory planning module is used to plan possible future driving trajectories, which is a conventional technical means and will not be elaborated in the present invention.

[0116] Among them, the initialization time is t; the experience pool B is initialized to store the training samples generated by the interaction between the strategy and the environment; the evaluation network parameters θ, the execution network parameters ω, the damping network parameters φ and the strategy entropy coefficient α are initialized; the target network parameters θ'←θ, ω′←ω, φ'←φ are initialized; and a suitable learning rate β is given. a , β b , β c , β d , τ and error ∈ = 1e-4; initialize the number of iterations k = 0; give the expected smoothing factor ξ T =2;

[0117] S52, obtaining the observation sample at the current time t;

[0118] In a feasible implementation, the robot state information, surrounding environment information, and navigation information obtained at time t are spliced ​​into the robot's environmental observation sample information

[0119] S53, sampling and calculating a reward function according to the obtained observation samples;

[0120] In a feasible implementation, the smooth control action u of the robot obtained at time t interacts with the environment to obtain the robot state and surrounding environment information corresponding to time t+1 and calculate the reward function r; the robot state information, surrounding environment information and navigation information obtained at time t+1 are spliced ​​into the robot's environmental observation sample information The sample Deposit into experience pool;

[0121] S54, solving the objective function value of the evaluation network according to the reward function;

[0122] In a feasible implementation, multiple experience samples sampled from the experience replay pool are used as initial values ​​of the forward solution process, and the evaluation network shown in formula (8) and formula (12) is used to calculate the objective function value respectively, so as to provide initial values ​​for the evaluation network gradient and update parameters of the backward solution;

[0123] S55, updating the evaluation network parameters according to the evaluation network objective function value;

[0124] In a feasible implementation, the objective function value obtained by solving is used to calculate the gradient, and after averaging the gradients of the evaluation network calculated in batches, the network parameters are subjected to gradient descent; wherein, the update rule of the non-smooth evaluation network in the Kth iteration can be expressed by the following formula (20) and formula (21):

[0125]

[0126] in, Represents the corresponding value function of the non-smooth evaluation network The gradient term: θ K+1 represents the updated non-smoothed evaluation network parameters; θ K represents the network parameters of the non-smoothed evaluation network before updating; β a represents the learning rate;

[0127] Among them, the update rule of the smoothing evaluation network in the Kth iteration can be expressed by the following formula (22) and formula (23):

[0128]

[0129] in, Represents the value function corresponding to the smooth evaluation network Gradient term; λ K+1 represents the updated smoothing evaluation network parameters; λ K represents the smoothing evaluation network parameters before updating;

[0130] S56, calculating the gradient of the objective function corresponding to the execution network, and updating the execution network parameters through back propagation;

[0131] In a feasible implementation, when K is a positive integer multiple of m=2, formula (16) is used to batch calculate the objective function and solve the policy gradient corresponding to the execution network: The averaged policy gradient is used to perform gradient descent on the execution network parameters; the update rule of the execution network in the Kth iteration can be expressed by the following formula (24):

[0132]

[0133] Among them, ω K+1represents the updated execution module network parameters; β b Indicates the network learning rate corresponding to the execution module; ω K Indicates the execution module network parameters before updating;

[0134] S57, calculating the gradient of the damping network corresponding to the objective function, and updating the damping network parameters through back propagation;

[0135] In a feasible implementation, when K is a positive integer multiple of m=2, formula (18) is used to batch calculate the objective function and solve the corresponding gradient of the damping network: The averaged gradient is used to perform gradient descent on the damping network parameters. The update rule of the damping network in the Kth iteration can be expressed by the following formula (25):

[0136]

[0137] in, represents the updated damping module network parameters; β c Represents the learning rate of the damping module network; Represents the damping module network parameters before updating;

[0138] S58, updating the strategy entropy coefficient according to the strategy entropy coefficient update rule;

[0139] In a feasible implementation, when K is an integer multiple of m=2 which is a positive integer, formula (17) is used for updating;

[0140] S59, updating the target value network and strategy network parameters according to the target evaluation network parameters before updating, the target execution network parameters before updating, and the target damping network parameters before updating;

[0141] In a feasible implementation, when K is an integer multiple of m=2 which is a positive integer, the update is performed according to the following formula (26)-formula (28):

[0142] θ' K+1 =-(1-τ)θ K ′+τθ K (26)

[0143] ω′ K+1 =-(1-τ)ω K ′+τω K (27)

[0144]

[0145] Where τ represents the delay update factor; θ K ′ represents the target evaluation network parameters before updating; θ′ K+1Represents the updated target evaluation network parameters; ω K ′ represents the target execution network parameters before updating; ω′ K+1 Represents the updated target execution network parameters; represents the target damping network parameters before updating; Represents the updated target damping network parameters;

[0146] S60: When it is determined that the network parameter update difference is less than a given threshold, the result is determined to be converged, and an optimal parameterization strategy is output.

[0147] In one feasible implementation, when |J π (ω K+1 )-J π (ω K )|>1e-4, it indicates that the strategy has not converged yet, and returns to initialization to continue solving; otherwise, it indicates that the strategy has converged within the error range, and the output result is the optimal parameterized strategy.

[0148] S6. Obtain observation sample information at the next moment according to the smooth control action of the robot at the current moment; input the observation sample information at the next moment into the updated control model to obtain the smooth control action of the robot at the next moment.

[0149] In a feasible implementation, the parameterized smooth policy network obtained through offline training is used as the output of the robot smooth reinforcement learning control model. When applied online, the smooth policy network can directly output the approximate action amount according to the current state of the robot and environmental information.

[0150] Among them, the embodiment of the present invention adopts a value distribution flexible execution-evaluation reinforcement learning algorithm, and adopts a parameterized strategy network obtained by offline training. It directly outputs the optimal control amount of the robot according to the current state of the robot and environmental information, so that the robot can realize the change of spatial state, and continuously outputs the control amount according to the robot state and environmental information, so as to realize the robot's high real-time and high-precision smooth control task.

[0151] Among them, Figure 3 Schematic diagram of a wheeled robot trajectory tracking and collision avoidance scenario provided by an embodiment of the present disclosure; the embodiment of the present invention uses the optimal parameterized strategy network learned after 190,000 iterations of the robot smooth control control model to perform simulation, and controls the wheeled robot to v T = 0.4 m / s, the desired speed tracks the given reference curve y = 0, and the obtained results are compared with those of the traditional non-smoothing control method, i.e., the distributed soft actor critic (DSAC), under the same prediction time domain, reference trajectory, and reference speed. Figure 4and Figure 5 As shown, the results show that the actor-critic-damper (ACD) architecture method proposed in the embodiment of the present invention can accurately track both the position and velocity quantities in the entire trajectory tracking and collision avoidance process, and can achieve a controllable smoothing effect and effective smoothing of the action.

[0152] The embodiment of the present invention first obtains the environmental observation sample information of the robot at the current moment; secondly, constructs a smooth reinforcement learning control model according to the environmental observation sample information of the robot at the current moment; constructs a reward function of the control model according to the environmental observation sample information of the robot at the current moment; adopts the discrete-time kinematic equation of the robot to construct the constraint conditions of the control model; inputs the environmental observation sample information, reward function and constraint conditions of the robot at the current moment into the control model for online training to obtain the smooth control action of the robot at the current moment; constructs the objective function of the control model; updates the control model iteratively according to the objective function to obtain the updated smooth reinforcement learning control model; finally, according to the smooth control action of the robot at the current moment, obtains the observation sample information at the next moment; inputs the observation sample information at the next moment into the updated control model to obtain the smooth control action of the robot at the next moment.

[0153] The embodiment of the present invention proposes a smooth reinforcement learning control model for robot intelligent control. The core architecture of the model is an actor-critic-damper (ACD) architecture. The architecture is based on the actor-critic (AC) architecture of the existing reinforcement learning algorithm. By adding a damper module, a new smooth reinforcement learning control architecture is formed, which is particularly oriented to robot intelligent control. Among them, the execution module in the execution-evaluation-damper architecture corresponds to the traditional non-smooth reinforcement learning strategy, the damper module is responsible for providing the damping strength related to the state and converting the non-smooth action into a smooth action, and the evaluation module is responsible for evaluating the smooth strategy and the non-smooth strategy. Under the assumption that a desired reference trajectory or reference point is given, the machine The robot's own sensors obtain the robot's various state quantities at the current moment, and use them to construct the objective functions of the execution module, damping module and evaluation module, and design the corresponding offline training loss function; the robot's own state and environmental information are used as the input of the execution module and damping module, and non-smooth actions and smooth actions are used as outputs, and the online optimization problem is converted into an offline solution of the network parameters of each module; when applied online, the controller directly outputs an approximate smoothing control strategy from the execution module network and the damping module network according to the robot state and surrounding environment information at each moment, and the strategy is executed by the lower-level control program to obtain the robot state at the next moment, thereby realizing online and high-real-time robot smooth control; in addition, the execution-evaluation-damper architecture has controllable smoothing capabilities, and the expected smoothing multiples can be set in advance before training to reduce the time spent on parameter adjustment.

[0154] The embodiment of the present invention proposes an execution module with a stable structure, which can focus on optimizing the original non-smooth strategy, and the damping module can focus on optimizing the smooth strategy; the execution-evaluation-damping architecture has the ability to adaptively adjust the smoothing capability, which can reduce the time spent on parameter adjustment; the embodiment of the present invention is suitable for industrial robots, medical robots and unmanned vehicles, and has the advantages of high computing efficiency, high precision and memory saving, breaking through the bottleneck of motion jitter in the existing reinforcement learning application in robot control.

[0155] Figure 6 is a block diagram of a smooth reinforcement learning device for robot intelligent control according to an exemplary embodiment, wherein the device is used for a smooth reinforcement learning method for robot intelligent control. Figure 6 The device includes a first acquisition unit 310, a first construction unit 320, a second construction unit 330, a second acquisition unit 340, a third acquisition unit 350 and a fourth acquisition unit 360. Wherein:

[0156] The first acquisition unit 310 is used to acquire the environment observation sample information of the robot at the current moment;

[0157] A first construction unit 320 is used to construct a smooth reinforcement learning control model according to the environment observation sample information of the robot at the current moment;

[0158] The second construction unit 330 is used to construct the reward function of the control model according to the environment observation sample information of the robot at the current moment; and to construct the constraint conditions of the control model using the discrete time kinematic equation of the robot;

[0159] The second acquisition unit 340 is used to input the environment observation sample information of the robot at the current moment, the reward function and the constraint condition into the control model for online training to obtain the smooth control action of the robot at the current moment;

[0160] The third acquisition unit 350 is used to construct the objective function of the control model; perform update iterative calculation on the control model according to the objective function to obtain an updated smooth reinforcement learning control model;

[0161] The fourth acquisition unit 360 is used to obtain the observation sample information at the next moment according to the smooth control action of the robot at the current moment; input the observation sample information at the next moment into the updated control model to obtain the smooth control action of the robot at the next moment.

[0162] Optionally, the first acquiring unit 310 is configured to:

[0163] Obtain the current robot status information, current surrounding environment information and current robot navigation information;

[0164] The robot state information at the current moment, the surrounding environment information at the current moment, and the robot navigation information at the current moment are spliced ​​to obtain the robot's environmental observation sample information at the current moment.

[0165] Optionally, the smooth reinforcement learning control model includes: an execution module, a damper module and an evaluation module; wherein,

[0166] The execution module is used to correspond to the traditional non-smooth reinforcement learning strategy;

[0167] The damper module is used to provide a damping strength related to the state to convert a non-smooth motion into a smooth motion;

[0168] The evaluation module is used to evaluate the evaluation value obtained by the smoothing strategy and the evaluation value obtained by the non-smoothing strategy.

[0169] Optionally, the discrete-time kinematic equation of the robot is used to construct the constraint conditions of the control model, including: using the discrete-time kinematic equation of the robot as the motion constraint of the control model, and using the control component amplitude constraint as the control constraint.

[0170] The reward function of the control model is a quadratic weighted sum of state tracking error, collision avoidance distance, and action.

[0171] Optionally, obtaining the smooth control action of the robot at the current moment includes:

[0172] The current non-smooth strategy is obtained by processing the current robot's environmental observation sample information through the execution module;

[0173] The robot's current environmental observation sample information is processed through the damping module, and the real-time damping value of the current moment is output for each dimensional control action. The smoothing strategy of the current moment is obtained through gradient normalization processing;

[0174] The non-smooth strategy at the current moment and the smooth strategy at the current moment are input into the evaluation module for processing to obtain the smooth control action of the robot at the current moment.

[0175] Optionally, the objective function of the control model includes: an original value distribution objective function and a smoothed value distribution objective function.

[0176] Optionally, updating and iterating the control model according to the objective function includes:

[0177] Given relevant initial conditions and parameter initialization;

[0178] Get the observation sample at the current time t;

[0179] Sampling and calculating reward function based on the obtained observation samples;

[0180] Solve the objective function value of the evaluation network according to the reward function;

[0181] Update the evaluation network parameters according to the evaluation network objective function value;

[0182] Calculate the gradient of the objective function corresponding to the execution network and update the execution network parameters through back propagation;

[0183] Calculate the gradient of the objective function corresponding to the damping network, and update the damping network parameters through back propagation;

[0184] Update the policy entropy coefficient according to the policy entropy coefficient update rule;

[0185] Update the target value network and strategy network parameters according to the target evaluation network parameters before updating, the target execution network parameters before updating, and the target damping network parameters before updating;

[0186] When the network parameter update difference is judged to be less than a given threshold, the result is judged to be converged and the optimal parameterization strategy is output.

[0187] The embodiment of the present invention first obtains the environmental observation sample information of the robot at the current moment; secondly, constructs a smooth reinforcement learning control model according to the environmental observation sample information of the robot at the current moment; constructs a reward function of the control model according to the environmental observation sample information of the robot at the current moment; adopts the discrete-time kinematic equation of the robot to construct the constraint conditions of the control model; inputs the environmental observation sample information, reward function and constraint conditions of the robot at the current moment into the control model for online training to obtain the smooth control action of the robot at the current moment; constructs the objective function of the control model; updates the control model iteratively according to the objective function to obtain the updated smooth reinforcement learning control model; finally, according to the smooth control action of the robot at the current moment, obtains the observation sample information at the next moment; inputs the observation sample information at the next moment into the updated control model to obtain the smooth control action of the robot at the next moment.

[0188] The embodiment of the present invention proposes a smooth reinforcement learning control model for robot intelligent control. The core architecture of the model is an actor-critic-damper (ACD) architecture. The architecture is based on the actor-critic (AC) architecture of the existing reinforcement learning algorithm. By adding a damper module, a new smooth reinforcement learning control architecture is formed, which is particularly oriented to robot intelligent control. Among them, the execution module in the execution-evaluation-damper architecture corresponds to the traditional non-smooth reinforcement learning strategy, the damper module is responsible for providing the damping strength related to the state and converting the non-smooth action into a smooth action, and the evaluation module is responsible for evaluating the smooth strategy and the non-smooth strategy. Under the assumption that a desired reference trajectory or reference point is given, the machine The robot's own sensors obtain the robot's various state quantities at the current moment, and use them to construct the objective functions of the execution module, damping module and evaluation module, and design the corresponding offline training loss function; the robot's own state and environmental information are used as the input of the execution module and damping module, and non-smooth actions and smooth actions are used as outputs, and the online optimization problem is converted into an offline solution of the network parameters of each module; when applied online, the controller directly outputs an approximate smoothing control strategy from the execution module network and the damping module network according to the robot state and surrounding environment information at each moment, and the strategy is executed by the lower-level control program to obtain the robot state at the next moment, thereby realizing online and high-real-time robot smooth control; in addition, the execution-evaluation-damper architecture has controllable smoothing capabilities, and the expected smoothing multiples can be set in advance before training to reduce the time spent on parameter adjustment.

[0189] The embodiment of the present invention proposes an execution module with a stable structure, which can focus on optimizing the original non-smooth strategy, and the damping module can focus on optimizing the smooth strategy; the execution-evaluation-damping architecture has the ability to adaptively adjust the smoothing capability, which can reduce the time spent on parameter adjustment; the embodiment of the present invention is suitable for industrial robots, medical robots and unmanned vehicles, and has the advantages of high computing efficiency, high precision and memory saving, breaking through the bottleneck of motion jitter in the existing reinforcement learning application in robot control.

[0190] Figure 7 is a schematic diagram of the structure of a smooth reinforcement learning device for robot intelligent control provided by an embodiment of the present invention, such as Figure 7 As shown, the smooth reinforcement learning device for robot intelligent control may include the above Figure 6 Optionally, the smooth reinforcement learning device 410 for robot intelligent control may include a first processor 2001 .

[0191] Optionally, the smooth reinforcement learning device 410 for robot intelligent control may further include a memory 2002 and a transceiver 2003 .

[0192] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0193] Combine the following Figure 7 The components of the smooth reinforcement learning device 410 for intelligent control of robots are specifically introduced:

[0194] The first processor 2001 is the control center of the smooth reinforcement learning device 410 for intelligent control of robots, and can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement an embodiment of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0195] Optionally, the first processor 2001 can perform various functions of the smooth reinforcement learning device 410 for robot intelligent control by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0196] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 7 CPU0 and CPU1 are shown in FIG.

[0197] In a specific implementation, as an embodiment, the smooth reinforcement learning device 410 for robot intelligent control may also include multiple processors, such as Figure 7 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0198] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0199] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be accessed through the interface circuit ( Figure 7 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0200] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0201] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 7 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0202] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently and communicate with the first processor 2001 through the interface circuit ( Figure 7 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0203] It should be noted that Figure 7 The structure of the smooth reinforcement learning device 410 for robot intelligent control shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0204] In addition, the technical effects of the smooth reinforcement learning device 410 for robot intelligent control can refer to the technical effects of the smooth reinforcement learning method for robot intelligent control described in the above method embodiment, which will not be repeated here.

[0205] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0206] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0207] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0208] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0209] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0210] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0211] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0212] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0213] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0214] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0215] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0216] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0217] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A smooth reinforcement learning method for robot intelligent control, characterized in that: The method comprises: S1. Obtain the current robot's environmental observation sample information; S2. Constructing a smooth reinforcement learning control model based on the environmental observation sample information of the robot at the current moment; S3. Constructing a reward function of the control model based on the environment observation sample information of the robot at the current moment; and constructing constraint conditions of the control model using the discrete time kinematic equation of the robot; S4, inputting the current robot's environmental observation sample information, the reward function and the constraint condition into the control model for online training to obtain a smooth control action of the robot at the current moment; S5, constructing the objective function of the control model; performing update iterative calculation on the control model according to the objective function to obtain an updated smooth reinforcement learning control model; S6. Obtain observation sample information at the next moment according to the smooth control action of the robot at the current moment; input the observation sample information at the next moment into the updated control model to obtain the smooth control action of the robot at the next moment.

2. The smooth reinforcement learning method for robot intelligent control according to claim 1, characterized in that: The step of obtaining the current robot's environmental observation sample information includes: S11, obtaining the current state information of the robot, the current surrounding environment information and the current navigation information of the robot; S12: Combine the robot state information at the current moment, the surrounding environment information at the current moment, and the robot navigation information at the current moment to obtain the robot's environmental observation sample information at the current moment.

3. The smooth reinforcement learning method for robot intelligent control according to claim 1, characterized in that: The smooth reinforcement learning control model includes: an execution module, a damper module and an evaluation module; wherein, The execution module is used to correspond to the traditional non-smooth reinforcement learning strategy; The damper module is used to provide a damping strength related to the state to convert a non-smooth motion into a smooth motion; The evaluation module is used to evaluate the evaluation value obtained by the smoothing strategy and the evaluation value obtained by the non-smoothing strategy.

4. The smooth reinforcement learning method for robot intelligent control according to claim 1, characterized in that: The step S3 adopts the discrete time kinematic equation of the robot to construct the constraint conditions of the control model, including: adopting the discrete time kinematic equation of the robot as the motion constraint of the control model, and taking the control component amplitude constraint as the control constraint; The reward function of the control model of S3 is a quadratic weighted sum of state tracking error, collision avoidance distance and action.

5. The smooth reinforcement learning method for robot intelligent control according to claim 3, characterized in that: The step S4 of obtaining the smooth control action of the robot at the current moment includes: S41, processing the environment observation sample information of the robot at the current moment through the execution module to obtain the non-smooth strategy at the current moment; S42, processing the environment observation sample information of the robot at the current moment through the damping module, outputting the real-time damping value at the current moment for each dimensional control action, and obtaining the smoothing strategy at the current moment through gradient normalization processing; S43, inputting the non-smooth strategy at the current moment and the smooth strategy at the current moment into the evaluation module for processing to obtain the smooth control action of the robot at the current moment.

6. The smooth reinforcement learning method for robot intelligent control according to claim 1, characterized in that: The objective function of the control model of S5 includes: an original value distribution objective function and a smoothed value distribution objective function.

7. The smooth reinforcement learning method for robot intelligent control according to claim 1, characterized in that: The step S5 of performing iterative calculation on the control model according to the objective function includes: S51, giving relevant initial conditions and parameter initialization; S52, obtaining the observation sample at the current time t; S53, sampling and calculating a reward function according to the obtained observation samples; S54, solving the objective function value of the evaluation network according to the reward function; S55, updating the evaluation network parameters according to the evaluation network objective function value; S56, calculating the gradient of the objective function corresponding to the execution network, and updating the execution network parameters through back propagation; S57, calculating the gradient of the damping network corresponding to the objective function, and updating the damping network parameters through back propagation; S58, updating the strategy entropy coefficient according to the strategy entropy coefficient update rule; S59, updating the target value network and strategy network parameters according to the target evaluation network parameters before updating, the target execution network parameters before updating, and the target damping network parameters before updating; S60: When it is determined that the network parameter update difference is less than a given threshold, the result is determined to be converged, and an optimal parameterization strategy is output.

8. A smooth reinforcement learning device for robot intelligent control, the smooth reinforcement learning device for robot intelligent control is used to implement the smooth reinforcement learning method for robot intelligent control as claimed in any one of claims 1 to 7, characterized in that: The device comprises: The first acquisition unit is used to acquire the environment observation sample information of the robot at the current moment; A first construction unit is used to construct a smooth reinforcement learning control model according to the environmental observation sample information of the robot at the current moment; The second construction unit is used to construct a reward function of the control model according to the environment observation sample information of the robot at the current moment; and to construct the constraint conditions of the control model using the discrete time kinematic equation of the robot; A second acquisition unit is used to input the environment observation sample information of the robot at the current moment, the reward function and the constraint condition into the control model for online training to obtain a smooth control action of the robot at the current moment; A third acquisition unit is used to construct an objective function of the control model; update and iterate the control model according to the objective function to obtain an updated smooth reinforcement learning control model; The fourth acquisition unit is used to obtain the observation sample information at the next moment according to the smooth control action of the robot at the current moment; input the observation sample information at the next moment into the updated control model to obtain the smooth control action of the robot at the next moment.

9. A smooth reinforcement learning device for robot intelligent control, characterized in that: The smooth reinforcement learning device for robot intelligent control includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 7.

Images