Robot system control method and device based on model-free regression reinforcement learning

Abstract

The present invention relates to the technical field of model-free regression reinforcement learning, and in particular to a robot system control method and device based on model-free regression reinforcement learning. The method comprises: performing system transformation on a robot nonlinear system, searching for admissible control as input data, collecting system information, including state information at different times, basis function values ​​of an execution-criteria network, etc.; and calculating key column vectors and performance index functions using the collected information. Iterative regression information is used to derive iterative rules for auxiliary regression constant weights and constant weights of an integrated execution-criteria network, and the robot system is optimally controlled based on the optimal execution-criteria network value, so that the robot system can grasp objects. The present invention obtains an optimal weight vector by collecting robot system signals, continuously iterating the weights of a neural network with discount parameters, and obtaining an optimal decision control method, thereby obtaining an optimal control decision for the robot system, and grasping objects through the robot system.

Description

Technical Field

[0001] The present invention relates to the technical field of model-free regression reinforcement learning, and in particular to a robot system control method and device based on model-free regression reinforcement learning. Background Art

[0002] Robot control has important applications in the industrial field, service field, autonomous driving and drone fields, etc. The HJB equation is designed to describe the optimal decision-making process in the robot dynamic system, and the solution to the optimal decision-making problem of the robot dynamic system often depends on the solution of the HJB equation. Reinforcement learning can handle high-dimensional state space, overcome the "curse of dimensionality" problem of traditional methods, and can learn in real time in the environment and adapt to the dynamic changes of the system, such as references 1 and 2 [Frank L. Lewis, Draguna Vrabie, Kyriakos G. Vamvoudakis. "Reinforcement Learning and FeedbackControl: Using Natural Decision Methods to Design Optimal AdaptiveControllers". IEEE Control Systems Magazine, vol. 32, no. 6, pp. 76-105, Dec.2012, Modares, Hamidreza, Frank L. Lewis. "Optimal Tracking Control ofNonlinear Partially-Unknown Constrained-Input Systems Using IntegralReinforcement Learning". Automatica vol. 50, no. 7, pp. 1780-1792, 2014]. Therefore, reinforcement learning is a key technology for calculating high-dimensional partial differential equations and has been widely used in multi-agent systems, robotic systems, and drone systems.

[0003] In order to deal with the optimal control problem of nonlinear robot systems, researchers have proposed a series of reinforcement learning methods, among which model-free reinforcement learning has significant advantages in solving the optimal control problem of nonlinear systems under model uncertainty. Reference 3 [Hamidreza Modares, Frank L. Lewis, Wei Kang, Ali Davoudi. “Optimal Synchronization of Heterogeneous Nonlinear Systems With Unknown Dynamics”.IEEE Transactions on Automatic Control, vol. 63, no. 1, pp. 117-131, Jan.2018] considers the optimal output synchronization problem of systems with unknown nonlinear dynamics. First, a discounted performance index function is defined for the agent, and an augmented HJB equation is derived to find its minimum value. Then, the proposed optimal control protocol ensures that the synchronization error of all agents converges locally and asymptotically to zero. Finally, the model-free reinforcement learning technique is used to learn the optimal control protocol for each agent without any agent dynamic information, thereby achieving the collaborative goal. Reference 4 [Zhongyang Ming, Huaguang Zhang, Juan Zhang, Xiangpeng Xie. “A Novel Actor–Critic–Identifier Architecture for Nonlinear Multiagent Systems with GradientDescent Method”. Automatica, vol. 155, pp. 111128, 2023] proposes an online adaptive reinforcement learning method for the infinite-time optimal consensus control problem of continuous-time unknown nonlinear systems. First, an execution-critic-identification network architecture is proposed to approximate the HJB equation. This architecture is simpler than most known architectures. Then, a neural network adjustment law based on the gradient descent method is designed. Finally, the adaptive control technology based on the Lyapunov method proves the convergence of the proposed model-free reinforcement learning algorithm. However, the method in reference 3 will face the existence of singular solutions in the process of solving the optimal control protocol, and the parameters are not easy to adjust; reference 4 needs to meet the continuous excitation condition, and the increase in the number of neural networks increases the complexity of the algorithm calculation. Most importantly, the convergence speed of the above two methods has a certain room for improvement. Therefore, how to construct an optimal control method for a robot system based on model-free reinforcement learning is one of the core issues that technicians in this field urgently need to solve. Summary of the invention

[0004] In order to solve the technical problem in the prior art that it is impossible to quickly obtain an effective control strategy for a robot system in nonlinear robot system control, an embodiment of the present invention provides a robot system control method and device based on model-free regression reinforcement learning. The technical solution is as follows:

[0005] On the one hand, a robot system control method based on model-free regression reinforcement learning is provided, characterized in that the method includes:

[0006] S1. construct a 2-DOF nonlinear robot system and convert the 2-DOF nonlinear robot system; construct an execution-critique network of the converted robot system, and initialize the number of iterations of the constant weight update of the execution-critique network to 1; wherein the 2-DOF nonlinear robot system is a robot system for an object grasping task;

[0007] S2, obtaining permissible input control data, and using the permissible input control data as an input signal of the converted robot system;

[0008] S3, execute the permissible input control data into the converted robot system, collect the converted robot system state signal data; calculate the information data of the execution-judgment network based on the permissible input control data and the robot system state signal data; calculate the performance index function value of the converted robot system based on the permissible input control data; wherein the robot system state signal data is the position, joint angle and speed information of the robot system in the grasping task;

[0009] S4, constructing the HJB equation, listing the key column vectors of the HJB equation based on the information data of the execution-judgment network, and integrating the constant weights of the execution-judgment network; calculating the auxiliary regression constant weights of the execution-judgment network based on the key column vectors and the performance indicator function values;

[0010] S5. Obtain a constant weight iteration rule of the execution-critique network according to the auxiliary regression constant weight, the key column vector, and the constant weight of the integrated execution-critique network; add 1 to the number of iterations, and calculate a new constant weight of the execution-critique network based on the constant weight iteration rule of the execution-critique network;

[0011] S6. Preset a constant weight error threshold to determine whether the constant weight error of two adjacent iterative execution-judgment networks meets the set constant weight error threshold. If so, stop the iteration to obtain the optimal execution-judgment network value, and optimally control the robot system based on the optimal execution-judgment network value to enable the robot system to grasp the object; otherwise, repeat S2-S5 until the constant weight error threshold is met.

[0012] Optionally, in S1, a 2-DOF nonlinear robot system is constructed and the 2-DOF nonlinear robot system is converted; an execution-criteria network of the converted robot system is constructed, and the number of iterations of updating the constant weights of the execution-criteria network is initialized to 1, including:

[0013] The 2-DOF nonlinear robot system is constructed according to the following formula (1):

[0014] (1)

[0015] in, is the angular position of the robot system, and , represents the first-order derivative of the angular position, represents the second derivative of the angular position; is the angular velocity of the robot system, is the inertia matrix, are the centrifugal and Coriolis force matrices, is the input torque, It is input interference;

[0016] The 2-DOF nonlinear robot system is converted into the standard form of the nonlinear system by mathematical derivation as shown in the following formula (2):

[0017] (2)

[0018] in, Indicates the status of the robot system; Derivatives representing the state of the robot system; Represents the system drift dynamic model information, and It is the system input dynamic model information, is the iterative control strategy related to the input torque at the iteration number s; is the iterative control strategy related to the input disturbance at the iteration number s;

[0019] Constructing an execution-criteria network of the converted robot system; wherein the execution-criteria network includes an execution network and a critique network; the execution network includes two neural networks, namely a first neural network and a second neural network; and the critique network includes a neural network;

[0020] The number of iterations for updating the constant weights of the initial execution-criteria network is .

[0021] Optionally, in S2, obtaining the permissible input control data and using the permissible input control data as the input signal of the converted robot system includes:

[0022] Acquire the allowable input control data; the allowable input control data is the data that makes the converted robot system stable and the performance index function limited; the allowable input control data includes: input torque and input interference ;

[0023] The permissible input control data is used as an input signal of the converted robot system.

[0024] Optionally, in S3, executing the allowable input control data into the converted robot system, collecting the state signal data of the converted robot system; calculating the information data of the execution-judgment network based on the allowable input control data and the state signal data of the robot system; and calculating the performance indicator function value of the converted robot system based on the allowable input control data, including:

[0025] Execution allows input of control data into the converted robot system, collecting Moment and System status signal data at all times and , where the time interval ;

[0026] Get the execution-criticism network basis function information, including: Basis functions at time , judging network Basis functions at time , execute the basis function of the first neural network in the network and the basis functions of the second neural network in the execution network ;

[0027] Wherein, p1 represents the number of neurons in the neural network of the evaluation network; p2 represents the number of neurons in the first neural network of the execution network; p3 represents the number of neurons in the second neural network of the execution network;

[0028] Based on the execution network basis function information, the approximate values ​​of the two neural networks in the execution network are calculated as follows: and :

[0029] (3)

[0030] (4)

[0031] in, and are the constant weights of the two neural networks in the execution network respectively; m is the dimension of the input torque u in the control input; is the dimension of the input disturbance d in the control input;

[0032] Calculate the approximation value of the two neural networks in the execution network under the iteration number s and the difference of the corresponding allowable input control, and simplify it to get: and ;

[0033] Based on the basis function information of the judgment network, the approximation value of the judgment network is calculated as follows: :

[0034] (5)

[0035] in, is the constant weight of the evaluation network; T represents transposition;

[0036] Based on the allowable input control data, the performance index function value of the converted robot system is calculated as follows: :

[0037] (6)

[0038] in, , represents a positive diagonal matrix; represents a positive discount parameter, represents the elements in the matrix R; represents a positive symmetric matrix; represents the square of the L2-gain parameter.

[0039] Optionally, in S4, a HJB equation is constructed, key column vectors of the HJB equation are listed based on the information data of the execution-critique network, and constant weights of the execution-critique network are integrated, including:

[0040] Construct the HJB equation. Based on the information data of the execution-criteria network, the key column vectors of the HJB equation are listed according to the following formula (7): :

[0041] (7)

[0042] in, and ,

[0043] The constant weight of the execution-criteria network is integrated according to the following formula (8):

[0044] (8)

[0045] in, is the constant weight of the integrated executive-critic network; is the constant weight of the first neural network in the execution network The first column of Represents the constant weight of the first neural network in the execution network The mth column of Represents the constant weight of the second neural network in the execution network Column 1 of Represents the constant weight of the second neural network in the execution network No. List.

[0046] Optionally, in S4, based on the key column vector and the performance indicator function value, the auxiliary regression constant weight of the execution-judgment network is calculated, including:

[0047] Define discount parameters , let the discount parameter Greater than 0 and less than 1;

[0048] As shown in the following formula (9), based on the key column vector And the performance indicator function value , combined with the discount parameters , calculate the auxiliary regression constant weight :

[0049] (9)

[0050] in, Indicates the number of iterations from 1 to The sequence value of .

[0051] Optionally, in S6, a constant weight error threshold is preset, and judging whether the constant weight error of the execution-criteria network of two adjacent iterations satisfies the set constant weight error threshold includes:

[0052] Preset constant weight error threshold ɛ;

[0053] Calculate the constant weight error of the execution-criteria network between two consecutive iterations ;

[0054] Determine whether the constant weight error of the execution-criteria network of two adjacent iterations is less than the preset constant weight error threshold ɛ.

[0055] Optionally, best execution-judgment network value, including:

[0056] Output the optimal execution-critique network value through the execution-critique network, including: , , On the other hand, a robot system control device based on model-free regression reinforcement learning is provided, and the device is applied to a robot system control method based on model-free regression reinforcement learning, and the device comprises:

[0057] A system conversion module is used to construct a 2-DOF nonlinear robot system and convert the 2-DOF nonlinear robot system; construct an execution-criteria network of the converted robot system, and initialize the number of iterations of the constant weight update of the execution-criteria network to 1; wherein the 2-DOF nonlinear robot system is a robot system for object grasping tasks;

[0058] An input data acquisition module, used for acquiring permissible input control data, and using the permissible input control data as an input signal of the converted robot system;

[0059] The numerical calculation module is used to execute the permissible input control data into the converted robot system and collect the state signal data of the converted robot system; based on the permissible input control data and the state signal data of the robot system, calculate the information data of the execution-judgment network; based on the permissible input control data, calculate the performance index function value of the converted robot system; wherein the state signal data of the robot system is the position, joint angle and speed information of the robot system in the grasping task;

[0060] The constant weight calculation module is used to construct the HJB equation, list the key column vectors of the HJB equation based on the information data of the execution-judgment network, and integrate the constant weights of the execution-judgment network; based on the key column vectors and the performance indicator function value, calculate the auxiliary regression constant weights of the execution-judgment network;

[0061] An iteration module is used to obtain a constant weight iteration rule of the execution-critique network according to the auxiliary regression constant weight, the key column vector and the constant weight of the integrated execution-critique network; add 1 to the number of iterations, and calculate a new constant weight of the execution-critique network based on the constant weight iteration rule of the execution-critique network;

[0062] The robot control module is used to preset a constant weight error threshold and determine whether the constant weight error of two adjacent iterative execution-judgment networks meets the set constant weight error threshold. If so, the iteration is stopped to obtain the optimal execution-judgment network value, and the robot system is optimally controlled based on the optimal execution-judgment network value; otherwise, S2-S5 are repeatedly executed until the constant weight error threshold is met.

[0063] On the other hand, a robot system control device based on model-free regression reinforcement learning is provided, and the robot system control device based on model-free regression reinforcement learning includes: a processor; a memory, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned robot system control methods based on model-free regression reinforcement learning is implemented.

[0064] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned robot system control methods based on model-free regression reinforcement learning.

[0065] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0066] In the embodiment of the present invention, a robot system control method and device based on model-free reinforcement learning are proposed to overcome the problems of difficulty and slow solution of HJB equations in the optimal control problem of model-free robot systems, and to deal with the problem of slow convergence of the algorithm by introducing discount parameters. In addition, a neural network is used to simulate the execution-judgment structure. At the same time, by collecting certain signals (angular position, angular velocity, basis function) of the robot system, the weights of the neural network with discount parameters are continuously iterated to obtain the optimal weight vector, thereby obtaining the solution of the HJB equation of the robot system, obtaining the optimal decision control method, obtaining the optimal control decision of the robot system, and grasping the object through the robot system. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0068] Figure 1 is a flow chart of a robot system control method based on model-free regression reinforcement learning provided by an embodiment of the present invention;

[0069] Figure 2 1 is a diagram of a method for solving the Hamilton-Jacobi-Bellman equation based on model-free regression reinforcement learning provided by an embodiment of the present invention;

[0070] Figure 3 It is a block diagram of a robot system control device based on model-free regression reinforcement learning provided by an embodiment of the present invention;

[0071] Figure 4It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0072] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0073] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0074] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0075] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0076] The embodiment of the present invention provides a robot system control method based on model-free regression reinforcement learning, which can be implemented by a robot system control device based on model-free regression reinforcement learning, and the robot system control device based on model-free regression reinforcement learning can be a terminal or a server. Figure 1 The flowchart of the robot system control method based on model-free regression reinforcement learning is shown in FIG. The processing flow of the method may include the following steps:

[0077] S1. Construct a 2-DOF nonlinear robot system and transform the 2-DOF nonlinear robot system; construct an execution-critique network of the transformed robot system, and initialize the number of iterations of the constant weight update of the execution-critique network to 1.

[0078] In a feasible implementation, the 2-DOF nonlinear robot system is a robot system used for precise tasks within a small range, such as grasping, positioning, and assembling objects.

[0079] The 2-DOF nonlinear robot system is constructed according to the following formula (1):

[0080] (1)

[0081] in, is the angular position of the robot system, and , represents the first-order derivative of the angular position, represents the second derivative of the angular position; is the angular velocity of the robot system, is the inertia matrix, are the centrifugal and Coriolis force matrices, is the input torque, It is input interference;

[0082] The 2-DOF nonlinear robot system is converted into the following standard nonlinear system form through mathematical derivation:

[0083] (2)

[0084] in, Indicates the status of the robot system; Derivatives representing the state of the robot system; Represents the system drift dynamic model information, and It is the system input dynamic model information, is the iterative control strategy related to the input torque at the iteration number s; is the iterative control strategy related to the input disturbance at the iteration number s;

[0085] Constructing an Actor-Critic Network of the converted robot system; wherein the Actor-Critic Network includes an Actor Network and a Critic Network, wherein the Actor Network includes two neural networks and the Critic Network includes one neural network;

[0086] The number of iterations for updating the constant weights of the initial execution-criteria network is .

[0087] S2. Acquire permissible input control data, and use the permissible input control data as an input signal of the converted robot system.

[0088] In a feasible implementation manner, in S2, obtaining the permissible input control data and using the permissible input control data as the input signal of the converted robot system includes:

[0089] Acquire the allowable input control data; the allowable input control data is the data that makes the converted robot system stable and the performance index function limited; the allowable input control data includes: input torque and input interference ;

[0090] The permissible input control data is used as the input signal of the system after the 2-DOF robot system is converted.

[0091] S3. Execute the permissible input control data into the converted robot system, collect the state signal data of the converted robot system, calculate the information data of the execution-judgment network based on the input signal and the robot system state signal data; calculate the performance indicator function value of the converted robot system based on the permissible input control data.

[0092] In a feasible implementation, the robot system state signal data is the position, joint angle and speed information of the robot system in the grasping task;

[0093] Execution allows input of control data into the converted robot system, collecting Moment and System status signal data at all times and , where the time interval ;

[0094] Get the basis function information of the execution-criteria network: the criterion network is Basis functions at time , judging network Basis functions at time , execute the basis function of the first neural network in the network and the basis functions of the second neural network in the execution network ;

[0095] Wherein, p1 represents the number of neurons in the neural network of the evaluation network; p2 represents the number of neurons in the first neural network of the execution network; p3 represents the number of neurons in the second neural network of the execution network;

[0096] Based on the execution network basis function information, calculate the approximate values ​​of the two neural networks in the execution network and :

[0097] (3)

[0098] (4)

[0099] in, and are the constant weights of the two neural networks in the execution network respectively; m is the dimension of the input torque u in the control input; is the dimension of the input disturbance d in the control input;

[0100] Calculate the approximate values ​​of the two neural networks in the execution network under the iteration number s and the difference of the corresponding allowable input control and simplify them to obtain: and ;

[0101] Based on the judgment network basis function information, calculate the judgment network approximation value :

[0102] (5)

[0103] in, is the constant weight of the evaluation network; T represents transposition;

[0104] Calculate the performance index function value of the converted robot system based on the admissible input control data :

[0105] (6)

[0106] in, , represents a positive diagonal matrix; represents a positive discount parameter, represents the elements in the matrix R; represents a positive symmetric matrix; represents the square of the L2-gain parameter.

[0107] In one possible implementation, the control input and The neural network approximation value is

[0108]

[0109]

[0110]

[0111] in, is the constant weight of the judging network, and is the constant weight of the execution network, and The control inputs are and The dimension of and is the execution neural network designed and It is to judge the neural network.

[0112] Finally, calculate the performance index function value , the specific operations are as follows:

[0113]

[0114] in, is the discount factor, , , .

[0115] yes -Gain parameters must meet the following conditions:

[0116]

[0117] in, , The performance output signal is as follows:

[0118] .

[0119] S4. Construct the HJB equation, list the key column vectors of the HJB equation based on the information data of the execution-judgment network, and integrate the constant weights of the execution-judgment network; calculate the auxiliary regression constant weights of the execution-judgment network based on the key column vectors and the performance indicator function values.

[0120] In a feasible implementation, based on the information data of the execution-judgment network, the key column vector of the HJB equation is listed according to the following formula (7): :

[0121] (7)

[0122] in, and ,

[0123] The constant weight of the execution-criteria network is integrated according to the following formula (8):

[0124] (8)

[0125] in, is the constant weight of the integrated executive-critic network; is the constant weight of the first neural network in the execution network The first column of Represents the constant weight of the first neural network in the execution network The mth column of Represents the constant weight of the second neural network in the execution network Column 1 of Represents the constant weight of the second neural network in the execution network No. List.

[0126] Optionally, in S4, based on the key column vector and the performance indicator function value, the auxiliary regression constant weight of the execution-judgment network is calculated, including:

[0127] Define discount parameters , let the discount parameter Greater than 0 and less than 1;

[0128] Based on discount parameters , the key column vector And the performance indicator function value , calculate the auxiliary regression constant weight :

[0129] (9)

[0130] in, Indicates the number of iterations from 1 to The sequence value of .

[0131] In a feasible implementation, based on the literature [Adam L. Bruce, Ankit Goel, Dennis S.Bernstein. “Convergence and Consistency of Recursive Least Squares with Variable-Rate Forgetting”. Automatica, vol. 119, pp. 109052.][ Islam SyedAseem Ul, Dennis S. Bernstein. “Recursive Least Squares for Real-Time Implementation”. IEEE Control Systems Magazine vol. 39, no. 3, pp. 82-85,2019.], the auxiliary regression constant weight is further obtained. Iteration rules:

[0132] .

[0133] S5. According to the auxiliary regression constant weight, the key column vector and the constant weight of the integrated execution-critique network, the constant weight iteration rule of the execution-critique network is obtained; the number of iterations is increased by 1, and a new constant weight is calculated based on the constant weight iteration rule of the execution-critique network.

[0134] In a feasible implementation, the auxiliary regression constant weights are obtained by , the key column vector and the constant weights of the integrated execution-criteria network , and get its iteration rules, as follows:

[0135] .

[0136] S6. Preset a constant weight error threshold, determine whether the constant weight error of two adjacent iterative execution-judgment networks meets the set constant weight error threshold. If so, stop the iteration, obtain the optimal execution-judgment network value, and optimally control the robot system based on the optimal execution-judgment network value to enable the robot system to grasp the object; otherwise, repeat steps S2-S5 until the constant weight error threshold is met.

[0137] In a feasible implementation, in S6, a constant weight error threshold is preset, and judging whether the constant weight error of the execution-criteria network of two adjacent iterations satisfies the set constant weight error threshold includes:

[0138] Preset constant weight error threshold ɛ;

[0139] Calculate the constant weight error of the execution-criteria network between two consecutive iterations ;

[0140] Determine whether the constant weight error of the execution-criteria network of two adjacent iterations is less than the preset constant weight error threshold ɛ.

[0141] In a feasible implementation, the optimal execution-judgment network value includes:

[0142] Output the optimal execution-criteria network value through the execution-criteria network, including: , , .

[0143] In a feasible implementation, Figure 2 It is a diagram of a method for solving the Hamilton-Jacobi-Bellman equation based on model-free regression reinforcement learning provided by an embodiment of the present invention.

[0144] In an embodiment of the present invention, the present invention proposes a robot system control method and device based on model-free reinforcement learning. Compared with the existing model-free reinforcement learning algorithm, the advantage of this method is that it does not need to meet the continuous excitation condition and solves the problem of difficult adjustment parameters, effectively avoiding the defect of using the least squares method to easily obtain singular solutions; by collecting certain signals (angular position, angular velocity, basis function) of the robot system, the neural network weights with discount parameters are continuously iterated to obtain the optimal weight vector, thereby obtaining the solution of the HJB equation of the robot system and obtaining the optimal decision control method. At the same time, the introduction of discount parameters and auxiliary regression constant weights effectively solves key problems such as slow algorithm convergence speed. In addition, by continuously iterating the auxiliary regression constant weights and the neural network constant weights, the optimal execution-judgment network value and the solution of the HJB equation of the nonlinear robot system are obtained, the optimal control decision of the robot system is obtained, and the object is grasped by the robot system.

[0145] Figure 3 is a block diagram of a robot system control device 300 based on model-free regression reinforcement learning according to an exemplary embodiment. The device 300 is used for a robot system control method based on model-free regression reinforcement learning. Figure 3 , the device comprises. Wherein:

[0146] The system conversion module 310 is used to construct a 2-DOF nonlinear robot system and convert the 2-DOF nonlinear robot system; construct an execution-criteria network of the converted robot system, and initialize the number of iterations of the constant weight update of the execution-criteria network to 1; wherein the 2-DOF nonlinear robot system is a robot system for object grasping tasks

[0147] A data acquisition module 320, for acquiring permissible input control data, and using the permissible input control data as an input signal of the converted robot system;

[0148] The numerical calculation module 330 is used to execute the permissible input control data into the converted robot system, collect the converted robot system state signal data; calculate the information data of the execution-judgment network based on the permissible input control data and the robot system state signal data; calculate the performance index function value of the converted robot system based on the permissible input control data; wherein the robot system state signal data is the position, joint angle and speed information of the robot system in the grasping task;

[0149] The constant weight calculation module 340 is used to construct the HJB equation, list the key column vectors of the HJB equation based on the information data of the execution-judgment network, and integrate the constant weights of the execution-judgment network; calculate the auxiliary regression constant weights of the execution-judgment network based on the key column vectors and the performance indicator function value;

[0150] The iteration module 350 is used to obtain the constant weight iteration rule of the execution-critique network according to the auxiliary regression constant weight, the key column vector and the constant weight of the integrated execution-critique network; add 1 to the number of iterations, and calculate the new constant weight of the execution-critique network based on the preset constant weight iteration rule of the execution-critique network;

[0151] The robot control module 360 ​​is used to preset a constant weight error threshold, determine whether the constant weight error of two adjacent iterative execution-judgment networks meets the set constant weight error threshold, and if so, stop the iteration to obtain the optimal execution-judgment network value, and optimally control the robot system based on the optimal execution-judgment network value so that the robot system can grasp the object; otherwise, repeat steps S2-S5 until the constant weight error threshold is met.

[0152] Optionally, the system conversion module 310 is used to construct a 2-DOF nonlinear robot system according to the following formula (1):

[0153] (1)

[0154] in, is the angular position of the robot system, and , represents the first-order derivative of the angular position, represents the second derivative of the angular position; is the angular velocity of the robot system, is the inertia matrix, are the centrifugal and Coriolis force matrices, is the input torque, It is input interference;

[0155] The 2-DOF nonlinear robot system is converted into the following standard nonlinear system form through mathematical derivation:

[0156] (2)

[0157] in, Indicates the status of the robot system; Derivatives representing the state of the robot system; Represents the system drift dynamic model information, and It is the system input dynamic model information, is the iterative control strategy related to the input torque at the iteration number s; is the iterative control strategy related to the input disturbance at the iteration number s;

[0158] Constructing an execution-criteria network of the converted robot system; wherein the execution-criteria network includes an execution network and a critique network, the execution network includes two neural networks, and the critique network includes one neural network;

[0159] The number of iterations for updating the constant weights of the initial execution-criteria network is .

[0160] Optionally, the data acquisition module 320 is used to acquire permissible input control data; the permissible input control data is data that makes the converted robot system stable and the performance index function limited; the permissible input control data includes: input torque and input interference ;

[0161] The permissible input control data is used as the input signal of the system after the 2-DOF robot system is converted.

[0162] Optionally, the numerical calculation module 330 is used to execute the permissible input control data into the converted robot system, collect Moment and System status signal data at all times and , where the time interval ;

[0163] Get the basis function information of the execution-criteria network: the criterion network is Basis functions at time , judging network Basis functions at time , execute the basis function of the first neural network in the network and the basis functions of the second neural network in the execution network ;

[0164] Wherein, p1 represents the number of neurons in the neural network of the evaluation network; p2 represents the number of neurons in the first neural network of the execution network; p3 represents the number of neurons in the second neural network of the execution network;

[0165] Based on the execution network basis function information, calculate the approximate values ​​of the two neural networks in the execution network and :

[0166] (3)

[0167] (4)

[0168] in, and are the constant weights of the two neural networks in the execution network respectively; m is the dimension of the input torque u in the control input; is the dimension of the input disturbance d in the control input;

[0169] Calculate the approximate values ​​of the two neural networks in the execution network under the iteration number s and the difference of the corresponding allowable input control and simplify them to obtain: and ;

[0170] Based on the judgment network basis function information, calculate the judgment network approximation value :

[0171] (5)

[0172] in, is the constant weight of the evaluation network; T represents transposition;

[0173] Calculate the performance index function value of the converted robot system based on the admissible input control data :

[0174] (6)

[0175] in, , represents a positive diagonal matrix; represents a positive discount parameter, represents the elements in the matrix R; represents a positive symmetric matrix; represents the square of the L2-gain parameter.

[0176] Optionally, the constant weight calculation module 340 is used to construct the HJB equation, and lists the key column vectors of the HJB equation according to the following formula (7) based on the information data of the execution-judgment network: :

[0177] (7)

[0178] in, and ,

[0179] The constant weight of the execution-criteria network is integrated according to the following formula (8):

[0180] (8)

[0181] in, is the constant weight of the integrated executive-critic network; is the constant weight of the first neural network in the execution network The first column of Represents the constant weight of the first neural network in the execution network The mth column of Represents the constant weight of the second neural network in the execution network Column 1 of Represents the constant weight of the second neural network in the execution network No. List.

[0182] Optionally, the constant weight calculation module 340 is used to define the discount parameter , let the discount parameter Greater than 0 and less than 1;

[0183] Based on discount parameters , key column vector And the performance indicator function value , calculate the auxiliary regression constant weight :

[0184] (9).

[0185] in, Indicates the number of iterations from 1 to The sequence value of .

[0186] Optionally, the robot control module 360 ​​is used to preset a constant weight error threshold ɛ;

[0187] Calculate the constant weight error of the execution-criteria network between two consecutive iterations ;

[0188] Determine whether the constant weight error of the execution-criteria network of two adjacent iterations is less than the preset constant weight error threshold ɛ.

[0189] Optionally, best execution-judgment network value, including:

[0190] Output the optimal execution-criteria network value through the execution-criteria network, including: , , .

[0191] The present invention provides a robot system control device based on model-free reinforcement learning to overcome the problems of difficulty and slow solution of HJB equations in the optimal control problem of model-free robot systems, and to deal with the problem of slow algorithm convergence by introducing discount parameters. In addition, a neural network is used to simulate the execution-judgment structure. At the same time, by collecting certain signals (angular position, angular velocity, basis function) of the robot system, the weights of the neural network with discount parameters are continuously iterated to obtain the optimal weight vector, thereby obtaining the solution of the HJB equation of the robot system, obtaining the optimal decision control method, obtaining the optimal control decision of the robot system, and grasping the object through the robot system.

[0192] Figure 4 is a schematic diagram of the structure of a robot system control device based on model-free regression reinforcement learning provided by an embodiment of the present invention, such as Figure 4 As shown, the robot system control device based on model-free regression reinforcement learning may include the above Figure 3 The robot system control device based on model-free regression reinforcement learning is shown. Optionally, the robot system control device based on model-free regression reinforcement learning 410 may include a first processor 2001.

[0193] Optionally, the robot system control device 410 based on model-free regression reinforcement learning may further include a memory 2002 and a transceiver 2003 .

[0194] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0195] Combine the following Figure 4 The components of the robot system control device 410 based on model-free regression reinforcement learning are specifically introduced:

[0196] The first processor 2001 is the control center of the robot system control device 410 based on model-free regression reinforcement learning, which can be a processor or a general term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs).

[0197] Optionally, the first processor 2001 can perform various functions of the robot system control device 410 based on model-free regression reinforcement learning by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0198] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 are shown in FIG.

[0199] In a specific implementation, as an embodiment, the robot system control device 410 based on model-free regression reinforcement learning may also include multiple processors, such as Figure 4 The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0200] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled to be executed by the first processor 2001. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0201] Optionally, the memory 2002 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001, or may exist independently, and may be controlled by the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0202] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0203] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 4 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0204] Optionally, the transceiver 2003 may be integrated with the first processor 2001, or may exist independently, and may be controlled by the interface circuit ( Figure 4 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0205] It should be noted that Figure 4 The structure of the robot system control device 410 based on model-free regression reinforcement learning shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0206] In addition, the technical effects of the robot system control device 410 based on model-free regression reinforcement learning can refer to the technical effects of the robot system control method based on model-free regression reinforcement learning described in the above method embodiment, and will not be repeated here.

[0207] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0208] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0209] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable sensors. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0210] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0211] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0212] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0213] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0214] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0215] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention.

[0216] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A robot system control method based on model-free regression reinforcement learning, characterized in that: The method comprises: S1, constructing a 2-DOF nonlinear robot system, and converting the 2-DOF nonlinear robot system; constructing an execution-critique network of the converted robot system, and initializing the number of iterations of updating the constant weight of the execution-critique network to 1; wherein the 2-DOF nonlinear robot system is a robot system for an object grasping task; S2, obtaining permissible input control data, and using the permissible input control data as an input signal of the converted robot system; S3, execute the allowable input control data into the converted robot system, collect the converted robot system state signal data; calculate the information data of the execution-judgment network based on the allowable input control data and the robot system state signal data; calculate the performance index function value of the converted robot system based on the allowable input control data; wherein the robot system state signal data is the position, joint angle and speed information of the robot system in the grasping task; S4, constructing an HJB equation, listing the key column vectors of the HJB equation based on the information data of the execution-judgment network, and integrating the constant weights of the execution-judgment network; calculating the auxiliary regression constant weights of the execution-judgment network based on the key column vectors and the performance indicator function value; S5. Obtain a constant weight iteration rule of the execution-critique network according to the auxiliary regression constant weight, the key column vector, and the constant weight of the integrated execution-critique network; add 1 to the number of iterations, and calculate a new constant weight of the execution-critique network based on the constant weight iteration rule of the execution-critique network; S6. Preset a constant weight error threshold, determine whether the constant weight error of two adjacent iterative execution-judgment networks meets the set constant weight error threshold. If so, stop the iteration, obtain the optimal execution-judgment network value, and optimally control the robot system based on the optimal execution-judgment network value to enable the robot system to grasp the object; otherwise, repeat S2-S5 until the constant weight error threshold is met.

2. The robot system control method based on model-free regression reinforcement learning according to claim 1, characterized in that: In S1, a 2-DOF nonlinear robot system is constructed and the 2-DOF nonlinear robot system is converted; an execution-critique network of the converted robot system is constructed, and the number of iterations of updating the constant weights of the execution-critique network is initialized to 1, including: The 2-DOF nonlinear robot system is constructed according to the following formula (1): （1） in, is the angular position of the robot system, and , represents the first-order derivative of the angular position, represents the second derivative of the angular position; is the angular velocity of the robot system, is the inertia matrix, are the centrifugal and Coriolis force matrices, is the input torque, It is input interference; The 2-DOF nonlinear robot system is converted into a standard nonlinear system form as shown in the following formula (2) through mathematical derivation: （2） in, Indicates the status of the robot system; Derivatives representing the state of the robot system; Represents the system drift dynamic model information, and It is the system input dynamic model information, is the iterative control strategy related to the input torque at the iteration number s; is the iterative control strategy related to the input disturbance at the iteration number s; Constructing an execution-criteria network of the converted robot system; wherein the execution-criteria network includes an execution network and a critique network; the execution network includes two neural networks, namely a first neural network and a second neural network; and the critique network includes a neural network; Initialize the constant weight update iteration number of the execution-criteria network as .

3. The robot system control method based on model-free regression reinforcement learning according to claim 1, characterized in that: In the step S2, the permissible input control data is obtained and the permissible input control data is used as the input signal of the converted robot system; including: Acquire the permissible input control data; the permissible input control data is the data that makes the converted robot system stable and the performance index function limited; the permissible input control data includes: input torque and input interference ; The permissible input control data is used as an input signal of the converted robot system.

4. The robot system control method based on model-free regression reinforcement learning according to claim 3 is characterized in that: In S3, the permissible input control data is executed into the converted robot system, and the converted robot system state signal data is collected; based on the permissible input control data and the robot system state signal data, information data of the execution-judgment network is calculated; Calculating the performance indicator function value of the converted robot system based on the allowable input control data includes: Execution of the permissible input control data into the converted robot system, collecting Moment and System status signal data at all times and , where the time interval ; Get the execution-criticism network basis function information, including: Basis functions at time , judging network Basis functions at time , execute the basis function of the first neural network in the network and the basis functions of the second neural network in the execution network ; Wherein, p1 represents the number of neurons in the neural network of the evaluation network; p2 represents the number of neurons in the first neural network of the execution network; p3 represents the number of neurons in the second neural network of the execution network; Based on the execution network basis function information, the approximate values ​​of the two neural networks in the execution network are calculated as follows: and : （3） （4） in, and are the constant weights of the two neural networks in the execution network respectively; m is the dimension of the input torque u in the control input; is the dimension of the input disturbance d in the control input; Calculate the approximation value of the two neural networks in the execution network under the iteration number s and the difference of the corresponding allowable input control, and simplify it to get: and ; Based on the basis function information of the judgment network, the approximation value of the judgment network is calculated as follows: : （5） in, is the constant weight of the evaluation network; T represents transposition; Based on the allowable input control data, the performance index function value of the converted robot system is calculated as follows: : （6） in, , represents a positive diagonal matrix; represents a positive discount parameter, represents the elements in the matrix R; represents a positive symmetric matrix; represents the square of the L2-gain parameter.

5. The robot system control method based on model-free regression reinforcement learning according to claim 4 is characterized in that: In S4, the HJB equation is constructed, the key column vectors of the HJB equation are listed based on the information data of the execution-judgment network, and the constant weights of the execution-judgment network are integrated, including: Construct the HJB equation, based on the information data of the execution-criteria network, and list the key column vectors of the HJB equation according to the following formula (7): : （7） in, and , The constant weight of the execution-criteria network is integrated according to the following formula (8): （8） in, is the constant weight of the integrated executive-critic network; is the constant weight of the first neural network in the execution network The first column of Represents the constant weight of the first neural network in the execution network The mth column of Represents the constant weight of the second neural network in the execution network Column 1 of Represents the constant weight of the second neural network in the execution network No. List.

6. The robot system control method based on model-free regression reinforcement learning according to claim 5, characterized in that: In S4, based on the key column vector and the performance indicator function value, the auxiliary regression constant weight of the execution-judgment network is calculated, including: Define discount parameters , let the discount parameter Greater than 0 and less than 1; As shown in the following formula (9), based on the key column vector And the performance indicator function value , combined with the discount parameters , calculate the auxiliary regression constant weight : （9） in, Indicates the number of iterations from 1 to The sequence value of .

7. The robot system control method based on model-free regression reinforcement learning according to claim 6, characterized in that: In S6, a constant weight error threshold is preset, and judging whether the constant weight error of two adjacent iterative execution-judgment networks satisfies the set constant weight error threshold includes: Preset constant weight error threshold ɛ; Calculate the constant weight error of the execution-criteria network between two consecutive iterations ; It is determined whether the constant weight error of the execution-criteria network of two adjacent iterations is less than the preset constant weight error threshold ɛ.

8. The robot system control method based on model-free regression reinforcement learning according to claim 6, characterized in that: The optimal execution-judgment network value includes: Outputting the optimal execution-judgment network value through the execution-judgment network includes: , , .

9. A robot system control device based on model-free regression reinforcement learning, wherein the robot system control device based on model-free regression reinforcement learning is used to implement the robot system control method based on model-free regression reinforcement learning as claimed in any one of claims 1 to 8, characterized in that: The device comprises: A system conversion module is used to construct a 2-DOF nonlinear robot system and convert the 2-DOF nonlinear robot system; construct an execution-critique network of the converted robot system, and initialize the number of iterations of the constant weight update of the execution-critique network to 1; wherein the 2-DOF nonlinear robot system is a robot system for object grasping tasks; An input data acquisition module, used for acquiring permissible input control data, and using the permissible input control data as an input signal of the converted robot system; A numerical calculation module is used to execute the allowable input control data into the converted robot system and collect the converted robot system state signal data; based on the allowable input control data and the robot system state signal data, calculate the information data of the execution-judgment network; based on the allowable input control data, calculate the performance index function value of the converted robot system; wherein the robot system state signal data is the position, joint angle and speed information of the robot system in the grasping task; A constant weight calculation module is used to construct an HJB equation, list the key column vectors of the HJB equation based on the information data of the execution-judgment network, and integrate the constant weights of the execution-judgment network; based on the key column vectors and the performance indicator function value, calculate the auxiliary regression constant weights of the execution-judgment network; An iteration module is used to obtain a constant weight iteration rule of the execution-critique network according to the auxiliary regression constant weight, the key column vector and the constant weight of the integrated execution-critique network; add 1 to the number of iterations, and calculate a new constant weight of the execution-critique network based on the constant weight iteration rule of the execution-critique network; The robot control module is used to preset a constant weight error threshold, determine whether the constant weight error of two adjacent iterative execution-judgment networks meets the set constant weight error threshold, and if so, stop the iteration to obtain the optimal execution-judgment network value, and optimally control the robot system based on the optimal execution-judgment network value to enable the robot system to grasp the object; otherwise, repeat S2-S5 until the constant weight error threshold is met.

10. A robot system control device based on model-free regression reinforcement learning, characterized in that: The robot system control device of the model-free regression reinforcement learning includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 8 is implemented.

Images