Robot data-driven control method and device based on iterative algorithm solving

Abstract

The application relates to a robot data-driven control method and device based on an iterative algorithm. The method comprises the following steps: constructing a control strategy of a robot dynamics model according to a linear gain matrix to be optimized and a dimension-up characteristic function; the dimension-up characteristic function is used for mapping an original state space of the robot to a high-dimensional characteristic space in a Koopman operator framework, so that a nonlinear optimization problem of the control strategy is converted into a high-dimensional linear optimization problem; a Q function is defined; the Q function is converted; and data samples of z=(s, u) are used to iteratively calculate an updated value h of the matrix in a data-driven manner i The linear gain matrix is calculated, so that the control strategy of the robot dynamics model is obtained. The method can avoid introducing model errors and can calculate a more optimal control strategy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, in particular to a robot data-driven control method and device based on iterative algorithm solving. BACKGROUND

[0002] Currently, the control method widely used in the industrial field is model predictive control (MPC), which has attracted attention due to its theoretical development and wide application in industrial objects. Most classic MPC algorithms assume that the given model is generated a priori, in which case the identification process can be ignored. In fact, due to the presence of noisy data sets or the use of unreasonable assumption structures, the identification problem of an accurate model description, especially for unknown nonlinear dynamics, is quite tricky. From the perspective of identification, the uncertainty of modeling may also lead to conservative and degenerative control performance of robust MPC or tube-based MPC.

[0003] Currently, a typical model predictive control method is a control method based on Koopman operator. However, this control method is to obtain model parameters in a data-driven manner, and a controller is designed on this basis to solve the problem. This method inevitably introduces model errors, so the controller designed therefrom may be suboptimal. SUMMARY

[0004] Therefore, it is necessary to provide a robot data-driven control method and device based on iterative algorithm solving to avoid model errors in view of the above technical problems.

[0005] A robot data-driven control method based on iterative algorithm solving, the method comprising:

[0006] According to the linear gain matrix to be optimized and the dimension-up feature function, a control strategy of a robot dynamics model is constructed; the dimension-up feature function is to map the original state space of the robot to a high-dimensional feature space in the framework of the Koopman operator, so as to convert the nonlinear optimization problem of the control strategy into a high-dimensional linear optimization problem;

[0007] In the linear space, the Q function is defined as:

[0008] Q(z k )=l(z k )+γV(As k +Bu k +w k )

[0009] where z=(s, u), s represents a simplified representation of the lifted feature function Ψ(x), u represents the control policy, l(z k ) = |u k | 2 + |s k | 2 , A, B, C are model parameters for solving the high-dimensional feature space, w k is a model error, γ represents a discount factor, and V(·) represents a state value function.

[0010] According to the Q function, define where q = n ψ +m, n ψ represents the spatial dimension of the lifted feature function, m represents the dimension of the control policy, is the vectorization of the matrix related to .

[0011] The data sample of z=(s, u) is used in a data-driven manner to iteratively calculate the update value h i of the matrix , and the linear gain matrix is calculated to obtain the control policy of the robot dynamics model.

[0012] In one of the embodiments, it further comprises: constructing a lifted feature function Ψ(x) using a set of basis functions, and the expression of the lifted feature function Ψ(x) is:

[0013]

[0014] where c i represents the core of the basis function, and σ i represents the width of the basis function.

[0015] In one of the embodiments, it further comprises: constructing a lifted feature function Ψ(x) using a multi-layer neural network, and the expression of the lifted feature function Ψ(x) is:

[0016]

[0017] where L represents the number of network layers, and the output of the jth hidden layer is described as:

[0018] g j (x) = σ j (W j g j-1 (y j-1 )+b j )

[0019] where y j-1 is the output of the (j-1)th network. denotes a weight matrix, σ j denotes an activation function, b j denotes a bias term.

[0020] In one embodiment, the method further comprises: transforming the nonlinear optimization problem of the control policy into a high-dimensional linear optimization problem as:

[0021]

[0022] wherein, the high-dimensional linear optimization problem satisfies the model constraints under high-dimension as:

[0023]

[0024] s + denotes a next state of s, w and v both denote model errors.

[0025] In one embodiment, the method further comprises: simplifying the Q function as:

[0026]

[0027] wherein, Ξ su (P k ) = γA T P k B, Ξ ss (P k ) = γB T P k A, Ξ ss (P k ) = R + γA T P k B.

[0028] In one embodiment, the method further comprises: defining

[0029] At the i-th iteration, the update value h i of the matrix is calculated by least square method as:

[0030]

[0031] The update value h i of the matrix is calculated using the data samples of z = (s, u), and the linear gain matrix is calculated.

[0032] In one embodiment, the method further comprises: setting a data set of data samples z = (s, u) as recorded as Define ΔZ = Z - γZ + , and require ΔZΔZ T to be full rank, define the control policy as:

[0033]

[0034] where d is a zero-mean noise.

[0035] In one embodiment, the method further comprises updating the value h i The calculation formula of h

[0036] is: i T -1 ΔZY

[0037] where

[0038] A robot data-driven control device based on an iterative algorithm, the device comprising:

[0039] A transformation module for constructing a control policy of a robot dynamics model according to a linear gain matrix to be optimized and an up-dimensional feature function; the up-dimensional feature function is to map the original state space of the robot to a high-dimensional feature space in the framework of the Koopman operator, so as to convert the nonlinear optimization problem of the control policy into a high-dimensional linear optimization problem;

[0040] A Q function definition module for defining a Q function as:

[0041] Q(z k ) = l(z k ) + γV(As k + Bu k + w k )

[0042] where z = (s, u), s represents a simplified representation of the up-dimensional feature function Ψ(x), and u represents the control policy, l(z k ) = |u k | 2 + |s k | 2 , A, B, and C are model parameters for solving the high-dimensional feature space, w k is a model error, γ represents a discount factor, and V(·) represents a state value function;

[0043] A Q function transformation module for defining according to the Q function q = n ψ + m, n ψ ​​a space dimension of the lifted feature function, m represents a dimension of the control policy, is a vectorization of a matrix related to ;

[0044] a control policy calculation module, configured to iteratively calculate an updated value h of the matrix i using data samples of z=(s, u) in a data-driven manner.

[0045] A computer device, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:

[0046] constructing a control policy of a robot dynamics model according to a linear gain matrix to be optimized and a lifted feature function; the lifted feature function is mapping an original state space of the robot to a high-dimensional feature space in a Koopman operator framework, so as to convert a nonlinear optimization problem of the control policy into a high-dimensional linear optimization problem;

[0047] in a linear space, defining a Q function as:

[0048] Q(z k ) = l(z k ) + γV(As k + Bu k + w k )

[0049] wherein z=(s, u), s represents a simplified representation of the lifted feature function Ψ(x), u represents the control policy, l(z k ) = |u k | 2 + |s k | 2 , A, B, C are all model parameters for solving the high-dimensional feature space, w k is a model error, γ represents a discount factor, and V(·) represents a state value function;

[0050] according to the Q function, defining wherein q = n ψ + m, n ψ represents a space dimension of the lifted feature function, m represents a dimension of the control policy, is a vectorization of a matrix related to ;

[0051] iteratively calculating the matrix an updated value h i The linear gain matrix is calculated to obtain a control strategy of the robot dynamics model.

[0052] The robot data-driven control method and device based on the iterative algorithm solve the model parameters in the Koopman operator framework, and directly calculate the linear gain matrix by using an iterative solving method on the basis of establishing a Q function, thereby avoiding model errors and obtaining a more optimal control strategy. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 FIG. 1 is a flowchart of a robot data-driven control method based on an iterative algorithm in an embodiment;

[0054] Figure 2 FIG. 1 is a structural block diagram of a robot data-driven control device based on an iterative algorithm in an embodiment;

[0055] Figure 3 FIG. 1 is a structural block diagram of a robot data-driven control device based on an iterative algorithm in an embodiment; DETAILED DESCRIPTION

[0056] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not used to limit the present application.

[0057] In an embodiment, as shown in FIG. 1, a robot data-driven control method based on an iterative algorithm is provided, including the following steps: Figure 1

[0058] Step 102, constructing a control strategy of the robot dynamics model according to the linear gain matrix to be optimized and the dimension-raising characteristic function.

[0059] The dimension-raising characteristic function is to map the original state space of the robot to a high-dimensional feature space in the Koopman operator framework, so as to convert the nonlinear optimization problem of the control strategy into a high-dimensional linear optimization problem;

[0060] Specifically, the typical robot dynamics model is:

[0061] x k+1 =f(x k ,u k )

[0062] wherein, and are state and control variables, and k is a discrete time pointer.

[0063] ​The control objective is to:

[0064] From the initial state find a control policy u k = u(x k ) that minimizes the following performance index:

[0065]

[0066] where Q1, R > 0, γ ∈ [0, 1] is a discount factor.

[0067] In one embodiment, a set of basis functions is used to construct the lifted feature function Ψ(x), which is expressed as:

[0068]

[0069] where c i represents the core of the basis function, and σ i represents the width of the basis function. The basis function can be a Gaussian kernel function.

[0070] In another embodiment, a multi-layer neural network is used to construct the lifted feature function Ψ(x), which is expressed as:

[0071]

[0072] where L represents the number of network layers, and the output of the jth hidden layer is described as:

[0073] g j (x) = σ j (W j g j-1 (y j-1 ) + b j )

[0074] where y j-1 is the output of the (j-1)th network layer, represents the weight matrix, σ j represents the activation function, and b j represents the bias term.

[0075] In one specific embodiment, the nonlinear optimization problem of the control policy is converted into a high-dimensional linear optimization problem as:

[0076]

[0077] where The high-dimensional linear optimization problem satisfies the model constraints under high dimension as:

[0078]

[0079] s + represents the next state of s, and w and v both represent model errors.

[0080] The prior art method is to obtain model parameters A, B and C in a data-driven manner, and then design a controller to solve the problem. This method inevitably introduces the above-mentioned model errors w and v, so the controller designed therefrom can be suboptimal. The present application does not need to identify A, B and C, but directly solves K through data to avoid the generation of model errors, so that a more optimal control strategy can be obtained.

[0081] Step 104, defining a Q function in a linear space.

[0082] The Q function is:

[0083] Q(z k )=l(z k )+γV(As k +Bu k +w k )

[0084] wherein z=(s,u), s represents a simplified representation of an ascending dimension feature function Ψ(x), and u represents the control strategy, l(z k )=|u k | 2 +|s k | 2 , A, B and C are all model parameters for solving the high-dimensional feature space, w k is a model error, γ represents a discount factor, and V(·) represents a state value function.

[0085] Step 106, defining

[0086] wherein q=n ψ +m, n ψ represents the space dimension of the ascending dimension feature function, m represents the dimension of the control strategy, is a vectorization of a matrix related to .

[0087] Step 108, using data samples of z=(s,u) in a data-driven manner to iteratively calculate an updated value h i of the matrix , calculating the linear gain matrix, and obtaining the control strategy of the robot dynamics model.

[0088] In the above robot data-driven control method based on iterative algorithm, the model parameters in the Koopman operator framework do not need to be identified, but the linear gain matrix is directly calculated by using the iterative solving method on the basis of establishing the Q function, so that the model error can be avoided, and a more optimal control strategy can be obtained.

[0089] In one embodiment, the Q function is simplified as:

[0090]

[0091] wherein, Ξ su (P k )=γA T P k B, Ξ ss (P k )=γB T P k A, Ξ ss (P k )=R+γA T P k B.

[0092] On this basis, define wherein

[0093] q=n ψ +m, n ψ represents the space dimension of the up-dimensional characteristic function, m represents the dimension of the control strategy, is the vectorization of the matrix related to The element corresponding to is and the element corresponding to z i z j , wherein i≠j, is constructed as

[0094] Then, in order to solve the parameters K in the control strategy, an initial condition is given The value of is updated in the observable space through the iterative algorithm, so that gradually approaches its optimal value, and the optimal control strategy can be derived.

[0095] Further, define

[0096] At the i-th iteration, the update value h i of is calculated by the least square method as:

[0097]

[0098] iteratively computing a matrix using the data samples z = (s, u) of an updated value h i , computing the linear gain matrix.

[0099] Then, for all s k within the compact set Ω, set the data set of data samples z = (s, u) as denote Define ΔZ = Z - γZ + and require ΔZΔZ T to be full rank, define the control policy as:

[0100]

[0101] where d is a zero-mean noise.

[0102] Specifically, the formula for computing the updated value h i is:

[0103] h i = (ΔZΔZ T ) -1 ΔZY

[0104] where,

[0105] The above process can be performed offline or can be learned and deployed simultaneously.

[0106] It should be understood that, although each step in the flowchart of Figure 1 is shown in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps in may include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0107] In one embodiment, as shown in Figure 2 , a robot data-driven control device based on iterative algorithm solution is provided, comprising a transformation module 202, a Q function definition module 204, a Q function transformation module 206, and a control policy calculation module 208, wherein:

[0108] The transformation module 202 is configured to construct a control strategy of the robot dynamics model according to a linear gain matrix to be optimized and an enhanced dimensionality feature function, the enhanced dimensionality feature function being used to map an original state space of the robot to a high-dimensional feature space in a Koopman operator framework, so as to convert a nonlinear optimization problem of the control strategy into a high-dimensional linear optimization problem.

[0109] The Q function definition module 204 is configured to define a Q function as follows in the linear space:

[0110] Q(z k )=l(z k )+γV(As k +Bu k +w k )

[0111] wherein z=(s,u), s represents a simplified representation of the enhanced dimensionality feature function Ψ(x), and u represents the control strategy, l(z k )=|u k | 2 +|s k | 2 , A, B and C are model parameters for solving the high-dimensional feature space, w k is a model error, γ represents a discount factor, and V(·) represents a state value function.

[0112] The Q function transformation module 206 is configured to define according to the Q function. q=n ψ +m, n ψ represents a space dimensionality of the enhanced dimensionality feature function, m represents a dimensionality of the control strategy, is a vectorization of a matrix related to .

[0113] The control strategy calculation module 208 is configured to iteratively calculate an updated value h i of the matrix using data samples of z=(s,u) in a data-driven manner, calculate the linear gain matrix, and obtain the control strategy of the robot dynamics model.

[0114] The specific limitations of the robot data-driven control device based on iterative algorithm solving can refer to the limitations of the robot data-driven control method based on iterative algorithm solving in the above, which will not be repeated here. Each module in the robot data-driven control device based on iterative algorithm solving can be realized by software, hardware and their combination in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so that the processor calls and executes the operations corresponding to each module.

[0115] In one embodiment, a computer device, which can be a terminal, is provided, and an internal structure diagram of the computer device can be as shown in Figure 3 The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is configured to communicate with external terminals through network connection. The computer program is executed by the processor to implement a robot data-driven control method based on iterative algorithm solving. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0116] Those skilled in the art can understand that Figure 3 The structure shown in the above

[0117] In one embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method in the above embodiments.

[0118] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, databases, or other media in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0119] The technical features of the above embodiments can be combined in any way. In order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0120] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of the patent of the present application should be subject to the appended claims.

Claims

1. A robot data-driven control method based on an iterative algorithm solution, characterized by, The method comprises: According to the linear gain matrix to be optimized and the dimensionality elevation characteristic function, a control strategy of a robot dynamics model is constructed; the dimensionality elevation characteristic function is used to map the original state space of the robot to a high-dimensional feature space in the framework of a Koopman operator, so as to convert a nonlinear optimization problem of the control strategy into a high-dimensional linear optimization problem; converting the nonlinear optimization problem of the control strategy into the high-dimensional linear optimization problem comprises: Converting the nonlinear optimization problem of the control strategy into the high-dimensional linear optimization problem is: wherein, The high-dimensional linear optimization problem satisfies the model constraint under high-dimension as: s + represents the next state of s, w and v both represent model errors; In the linear space, the Q function is defined as: Q(z k ) = l(z k ) + γV(As k + Bu k + w k ) where z = (s, u), s represents a simplified representation of the high-dimensional feature function Ψ(x), u represents the control policy, r(z k ) = |u k | 2 + |s k | 2 , A, B, C are model parameters for solving the high-dimensional feature space, w k is a model error, γ represents a discount factor, and V(·) represents a state value function; According to the Q function, define where q = n ψ + m, n ψ denotes the spatial dimension of the lifted feature function, m denotes the dimension of the control policy, is the vectorization of the matrix related to ​ iteratively computing the matrix h using data samples of z = (s, u) in a data-driven manner i computing the linear gain matrix h to obtain a control policy u for the robot dynamics model​ iteratively computing the matrix h i computation of the linear gain matrix, comprising: Definitions At the i-th iteration, the update value h of the least squares method is calculated i is: iteratively computing the matrix of updated values h i , computing the linear gain matrix; The method further comprises: A data set of data samples z = (s, u) is provided It is noted that ΔZ = Z - γZ is defined + and that ΔZΔZ T is full rank, the control policy is defined as: Wherein d is a zero-mean noise; The method further comprises: Update value h i The calculation formula is: h i = (ΔZ ΔZ T ) -1 ΔZY wherein, 2. The method of claim 1, wherein, The step of constructing the dimensionality elevation characteristic function comprises: The dimensionality elevation characteristic function Ψ(x) is constructed using a set of basis functions, and an expression of the dimensionality elevation characteristic function Ψ(x) is: where c i represents the core of the basis function, σ i represents the width of the basis function.

3. The method of claim 1, wherein, The step of constructing the dimensionality elevation characteristic function comprises: The dimensionality elevation characteristic function Ψ(x) is constructed using a multilayer neural network, and an expression of the dimensionality elevation characteristic function Ψ(x) is: Wherein L represents the number of network layers, and the output of the jth hidden layer is described as: g j (x) = σ j (W j g j-1 (y j-1 )+b j ) where y j-1 is the output of the j-1 layer network, denotes a weight matrix, σ j denotes an activation function, b j denotes a bias term.

4. The method of claim 1, wherein, The method further comprises: simplifying the Q function as: wherein Ξ su (P k ) = γA T P k B, Ξ ss (P k ) = γB T P k A, Ξ ss (P k ) = R + γA T P k B.

5. A robot data-driven control apparatus based on an iterative algorithm solution, characterized by, The device for implementing the robot data-driven control method based on an iterative algorithm solution according to any one of claims 1 to 4 comprises: A conversion module is configured to construct a control strategy of a robot dynamics model according to a linear gain matrix to be optimized and a dimensionality elevation characteristic function; the dimensionality elevation characteristic function is used to map the original state space of the robot to a high-dimensional feature space in the framework of a Koopman operator, so as to convert a nonlinear optimization problem of the control strategy into a high-dimensional linear optimization problem; A Q function definition module is configured to define a Q function as: Q(z k ) = l(z k ) + γV(As k + Bu k + w k ) where z = (s, u), s represents a simplified representation of the high-dimensional feature function Ψ(x), u represents the control policy, l(z k ) = |u k | 2 + |s k | 2 , A, B, C are model parameters for solving the high-dimensional feature space, w k is a model error, γ represents a discount factor, and V(·) represents a state value function. a Q-function transformation module configured to define wherein q = n ψ +m, n ψ denotes the spatial dimension of the lifted feature function, m denotes the dimension of the control policy, is a vectorization of the matrix related to ​ a control policy computation module configured to iteratively compute an update value h of a matrix H = (s, u) using data samples of z = (s, u) in a data-driven manner i , and compute the linear gain matrix H to obtain a control policy of a robot dynamics model.​ 6.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-5 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 4.

Images