Robot data-driven control method and device based on strategy iteration solution

By using the dimensionality-upgraded feature function and data-driven iterative calculation method in the Koopman operator framework, the control strategy of the robot dynamics model is transformed, and the control performance degradation caused by model error in the existing technology is solved, and better control is achieved Strategy and robustness.

CN118672199BActive Publication Date: 2024-11-22NAT UNIV OF DEFENSE TECH
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411176928.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-26
Publication Date
2024-11-22
Estimated Expiration
2044-08-26

AI Technical Summary

Technical Problem

Existing model prediction control methods are prone to introduce model errors when facing noise data and unknown nonlinear dynamics, resulting in conservative and degraded control performance, especially in the identification and control of unknown nonlinear systems, which are difficult to effectively solve.

Method used

Through the up-dimensional feature function based on the Koopman operator framework, the control strategy of the robot dynamics model is transformed from nonlinear optimization problems to high-dimensional linear optimization problems, and the Q function is defined in linear space, and the linear gain iteratively is calculated in a data-driven manner. Matrix avoids the generation of model errors and thus obtains a better control strategy.

Benefits of technology

This method can effectively avoid model errors, improve the optimization of control strategies, provide more efficient robot dynamic model control, and is suitable for robust control of unknown nonlinear systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118672199B_ABST
    Figure CN118672199B_ABST
Patent Text Reader

Abstract

The present application relates to a robot data-driven control method and device based on policy iteration solution. The method comprises: constructing a control strategy of a robot dynamics model according to a linear gain matrix to be optimized and a dimension-raising characteristic function; the dimension-raising characteristic function maps the original state space of the robot to a high-dimensional characteristic space in a Koopman operator framework, thereby converting the nonlinear optimization problem of the control strategy into a high-dimensional linear optimization problem, defining a Q function in a linear space, and using the Q function to transform, in a data-driven manner, using the data sample of #imgabs0# to iteratively calculate the updated value #imgabs2# of the matrix #imgabs1#, and calculating the linear gain matrix, thereby obtaining the control strategy of the robot dynamics model. The use of this method can avoid the introduction of model errors and can calculate a better control strategy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of robot control technology, and in particular to a robot data-driven control method and device based on strategy iteration solution. Background Art

[0002] At present, the most widely used control method in the industrial field is Model Predictive Control (MPC), which has attracted attention due to its theoretical development and wide application in industrial objects. Most classic MPC algorithms assume that the given model is generated a priori, in which case the identification process can be ignored. In fact, due to the possible existence of noisy data sets or the use of unreasonable assumptions, the identification problem of accurate model description, especially for unknown nonlinear dynamics, is more difficult. Considering the uncertainty of modeling from the perspective of identification, robust MPC or tube-based MPC may also lead to conservative and degraded control performance. Tube-based MPC is an enhanced MPC strategy specifically designed to deal with system uncertainty. It adds a protection "tube" to the traditional MPC, allowing the system to remain in a safe working range when facing external interference or model uncertainty.

[0003] At present, a typical model predictive control method is a control method based on the Koopman operator. However, this control method obtains model parameters in a data-driven manner and designs a controller to solve the problem on this basis. This method will inevitably introduce model errors, so the controller designed in this way may be suboptimal. Summary of the invention

[0004] Based on this, it is necessary to provide a robot data-driven control method and device based on strategy iteration solution that avoids model errors in order to address the above technical problems.

[0005] A robot data-driven control method based on policy iteration solution, the method comprising:

[0006] A control strategy for the robot dynamics model is constructed according to the linear gain matrix to be optimized and the dimension-raising characteristic function; the dimension-raising characteristic function maps the original state space of the robot to a high-dimensional characteristic space in the Koopman operator framework, thereby converting the nonlinear optimization problem of the control strategy into a high-dimensional linear optimization problem;

[0007] In linear space, the Q function is defined as:

[0008] ;

[0009] in, , Represents the dimension-raising feature function A simplified representation of represents the control strategy, , All of them are for solving the model parameters of the high-dimensional feature space. is the model error, represents the discount factor, represents the state value function;

[0010] According to the Q function, we define ,in , represents the spatial dimension of the dimensionality-raising feature function, represents the dimension of the control strategy, is with Related Matrix Vectorization of

[0011] Use in a data-driven way The data sample iteratively calculates the matrix Update value of , calculate the linear gain matrix to obtain the control strategy of the robot dynamics model.

[0012] In one embodiment, the method further includes: constructing a dimension-raising feature function using a set of basis functions , dimension-raising feature function The expression is:

[0013] ;

[0014] in, represents the core of the basis function, Represents the width of the basis function.

[0015] In one embodiment, it also includes: constructing a dimension-raising feature function using a multi-layer neural network , dimension-raising feature function The expression is:

[0016] ;

[0017] in, Indicates the number of network layers, The output of the hidden layer is described as:

[0018] ;

[0019] in, It is The output of the layer network, represents the weight matrix, represents the activation function, Represents the bias term.

[0020] In one embodiment, the nonlinear optimization problem of the control strategy is converted into a high-dimensional linear optimization problem:

[0021] ;

[0022] in, ; The high-dimensional linear optimization problem satisfies the model constraints in high dimensions:

[0023] ;

[0024] express The next state, and Both represent model errors.

[0025] In one embodiment, the Q function is simplified to:

[0026] ;

[0027] in, , .

[0028] In one embodiment, it also includes: defining ;

[0029] In the At the iteration, the least squares method is used to calculate Update value of for:

[0030] ;

[0031] Use the The data sample iteratively calculates the matrix The updated value of , calculates the linear gain matrix.

[0032] In one embodiment, the method further includes: setting a data sample The data set is ,remember , in order to Full rank, the control strategy is defined as:

[0033] ;

[0034] in, is zero mean noise.

[0035] In one embodiment, it also includes: updating the value The calculation formula is:

[0036] ;

[0037] in, ;

[0038] By iterative least squares update for:

[0039] ;

[0040] in, represents the intermediate square matrix in the update process, whose row and column dimensions are the same as The dimensions of the vector columns are consistent;

[0041] A robot data-driven control device based on policy iteration solution, the device comprising:

[0042] A conversion module is used to construct a control strategy for the robot dynamics model according to the linear gain matrix to be optimized and the dimension-raising characteristic function; the dimension-raising characteristic function maps the original state space of the robot to a high-dimensional characteristic space in the Koopman operator framework, thereby converting the nonlinear optimization problem of the control strategy into a high-dimensional linear optimization problem;

[0043] The Q function definition module is used to define the Q function in a linear space as:

[0044] ;

[0045] in, , Represents the dimension-raising feature function A simplified representation of represents the control strategy, , All of them are for solving the model parameters of the high-dimensional feature space. is the model error, represents the discount factor, represents the state value function;

[0046] A Q function conversion module is used to define, according to the Q function, ,in , represents the spatial dimension of the dimensionality-raising feature function, represents the dimension of the control strategy, is with Related Matrix Vectorization of

[0047] Control strategy calculation module for use in a data-driven manner The data sample iteratively calculates the matrix Update value of , calculate the linear gain matrix to obtain the control strategy of the robot dynamics model.

[0048] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0049] A control strategy for the robot dynamics model is constructed according to the linear gain matrix to be optimized and the dimension-raising characteristic function; the dimension-raising characteristic function maps the original state space of the robot to a high-dimensional characteristic space in the Koopman operator framework, thereby converting the nonlinear optimization problem of the control strategy into a high-dimensional linear optimization problem;

[0050] Define the Q function as:

[0051] ;

[0052] in, , Represents the dimension-raising feature function A simplified representation of represents the control strategy, , All of them are for solving the model parameters of the high-dimensional feature space. is the model error, represents the discount factor, represents the state value function;

[0053] According to the Q function, we define ,in , represents the spatial dimension of the dimensionality-raising feature function, represents the dimension of the control strategy, is with Related Matrix Vectorization of

[0054] Use in a data-driven way The data sample iteratively calculates the matrix Update value of , calculate the linear gain matrix to obtain the control strategy of the robot dynamics model.

[0055] The above-mentioned robot data-driven control method and device based on strategy iteration solution first establishes a control strategy of the robot dynamics model , and then determine the high-dimensional linear optimization problem. The traditional way to solve this high-dimensional linear optimization problem is to solve the model parameters , but this approach will inevitably introduce model errors and Therefore, the optimal control strategy cannot be achieved. However, the present invention does not need to identify the model parameters in the Koopman operator framework. Instead, when solving high-dimensional linear optimization problems, a Q function is established in a linear space and used in a data-driven manner. The linear gain matrix is ​​directly calculated by iterative solution of the data samples K , thereby avoiding model errors and obtaining better control strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A schematic flow chart of a robot data-driven control method based on policy iteration solution in one embodiment;

[0057] Figure 2 A structural block diagram of a robot data-driven control device based on policy iteration solution in one embodiment;

[0058] Figure 3 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] In one embodiment, Figure 1 As shown, a robot data-driven control method based on policy iteration solution is provided, comprising the following steps:

[0061] Step 102: construct a control strategy for the robot dynamics model according to the linear gain matrix to be optimized and the dimension-increased characteristic function.

[0062] The dimension-raising feature function maps the original state space of the robot to a high-dimensional feature space in the Koopman operator framework, thereby transforming the nonlinear optimization problem of the control strategy into a high-dimensional linear optimization problem.

[0063] Specifically, the typical robot dynamics model is:

[0064] ;

[0065] in, and are state and control variables, is a discrete time pointer.

[0066] The control objectives are:

[0067] From the initial state , find a control strategy The following performance indicators can be minimized:

[0068] ;

[0069] in is the discount factor.

[0070] In one embodiment, a set of basis functions is used to construct the dimension-raising feature function. , dimension-raising feature function The expression is:

[0071] ;

[0072] in, represents the core of the basis function, Indicates the width of the basis function. The basis function can be a Gaussian kernel function.

[0073] In another embodiment, a multi-layer neural network is used to construct a dimension-increasing feature function. , dimension-raising feature function The expression is:

[0074] ;

[0075] in, Indicates the number of network layers, The output of the hidden layer is described as:

[0076] ;

[0077] in, It is The output of the layer network, represents the weight matrix, represents the activation function, Represents the bias term.

[0078] In a specific embodiment, the nonlinear optimization problem of the control strategy is transformed into a high-dimensional linear optimization problem:

[0079] ;

[0080] in, ; The high-dimensional linear optimization problem satisfies the model constraints in high dimensions:

[0081] ;

[0082] express The next state, and Both represent model errors.

[0083] The existing method is to find the model parameters in a data-driven way. , and design a controller based on this to solve the problem. This method will inevitably introduce the above model error , so the controller designed in this way may be suboptimal. The present invention does not require identification , but directly solve it through data , avoiding model errors, thus obtaining a better control strategy.

[0084] Step 104, define a Q function in a linear space.

[0085] The Q function is:

[0086] ;

[0087] in, , Represents the dimension-raising feature function A simplified representation of represents the control strategy, , All of them are for solving the model parameters of the high-dimensional feature space. is the model error, represents the discount factor, Represents a state-value function.

[0088] Step 106, based on the Q function, define .

[0089] in , represents the spatial dimension of the dimensionality-raising feature function, represents the dimension of the control strategy, is with Related Matrix Vectorization of .

[0090] Step 108: Use in a data-driven manner The data sample iteratively calculates the matrix Update value of , calculate the linear gain matrix to obtain the control strategy of the robot dynamics model.

[0091] In the above-mentioned robot data-driven control method based on policy iteration solution, the control strategy of the robot dynamics model is first established. , and then determine the high-dimensional linear optimization problem. The traditional way to solve this high-dimensional linear optimization problem is to solve the model parameters , but this approach will inevitably introduce model errors and Therefore, the optimal control strategy cannot be achieved. However, the present invention does not need to identify the model parameters in the Koopman operator framework. Instead, when solving high-dimensional linear optimization problems, a Q function is established in a linear space and used in a data-driven manner. The linear gain matrix is ​​directly calculated by iterative solution of the data samples K , thereby avoiding model errors and obtaining better control strategies.

[0092] In one embodiment, the Q function is simplified to:

[0093] ;

[0094] in, , .

[0095] On this basis, the definition ,in , , represents the spatial dimension of the dimensionality-raising feature function, represents the dimension of the control strategy, is with Related Matrix The vectorization of The elements of , and corresponds to ,in , then it is constructed as .

[0096] Then, in order to solve the parameters in the control strategy , given an initial condition , updated in the observable space through an iterative algorithm The value of It asymptotically approaches its optimal value and the optimal control strategy can be derived.

[0097] Furthermore, define ;

[0098] In the At the iteration, the least squares method is used to calculate Update value of for:

[0099] ;

[0100] use The data sample iteratively calculates the matrix The updated value of , calculates the linear gain matrix.

[0101] Then, for all within , set the data sample The data set is ,remember , in order to Full rank, the control strategy is defined as:

[0102] ;

[0103] in, is zero mean noise.

[0104] Specifically, update the value The calculation formula is:

[0105] ;

[0106] in, ;

[0107] By iterative least squares update for:

[0108] ;

[0109] in, represents the intermediate square matrix in the update process, whose row and column dimensions are the same as The dimensions of the vector columns are consistent;

[0110] The above process can be performed offline or by learning and deploying at the same time.

[0111] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0112] In one embodiment, Figure 2 As shown, a robot data-driven control device based on policy iteration solution is provided, including: a conversion module 202, a Q function definition module 204, a Q function conversion module 206 and a control strategy calculation module 208, wherein:

[0113] The conversion module 202 is used to construct a control strategy for the robot dynamics model according to the linear gain matrix to be optimized and the dimension-raising characteristic function; the dimension-raising characteristic function maps the original state space of the robot to a high-dimensional characteristic space in the Koopman operator framework, thereby converting the nonlinear optimization problem of the control strategy into a high-dimensional linear optimization problem;

[0114] The Q function definition module 204 is used to define the Q function in the linear space as:

[0115] ;

[0116] in, , Represents the dimension-raising feature function A simplified representation of represents the control strategy, All of them are for solving the model parameters of the high-dimensional feature space. is the model error, represents the discount factor, Represents a state-value function.

[0117] The Q function conversion module 206 is used to define, according to the Q function, ,in , represents the spatial dimension of the dimensionality-raising feature function, represents the dimension of the control strategy, is with Related Matrix Vectorization of

[0118] Control strategy calculation module 208, used in a data-driven manner The data sample iteratively calculates the matrix Update value of , calculate the linear gain matrix to obtain the control strategy of the robot dynamics model.

[0119] For the specific definition of the robot data-driven control device based on policy iteration solution, please refer to the definition of the robot data-driven control method based on policy iteration solution in the above text, which will not be repeated here. Each module in the above-mentioned robot data-driven control device based on policy iteration solution can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0120] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a robot data-driven control method based on policy iteration solution is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0121] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0122] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.

[0123] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0124] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0125] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A robot data-driven control method based on policy iteration solution, characterized in that: The method comprises: According to the linear gain matrix to be optimized K and the dimension-increasing feature function , construct the control strategy of the robot dynamics model ; The dimension-raising feature function The original state space of the robot is mapped to a high-dimensional feature space in the Koopman operator framework, thereby converting the nonlinear optimization problem of the control strategy into a high-dimensional linear optimization problem; The control strategy u The nonlinear optimization problem is transformed into a high-dimensional linear optimization problem: in, , , ; The high-dimensional linear optimization problem satisfies the model constraints in high dimensions: express The next state, and Both represent model errors, All of them are for solving the model parameters of the high-dimensional feature space; In linear space, the Q function is defined as: in, , Represents the dimension-raising feature function A simplified representation of , is the model error, represents the discount factor, represents the state value function, k Represents a discrete time pointer; According to the Q function, we define ,in , , represents the spatial dimension of the dimensionality-raising feature function, represents the dimension of the control strategy, is with Related Matrix Vectorization of Use in a data-driven way The data sample iteratively calculates the matrix Update value of , calculate the linear gain matrix K , and thus obtain the control strategy of the robot dynamics model u ; Use in a data-driven way The data sample iteratively calculates the matrix Update value of , calculate the linear gain matrix K ,include: definition ; At the i+1th iteration, the least squares method is used to calculate Update value of for: Use the The data sample iteratively calculates the matrix The updated value of , calculates the linear gain matrix.

2. The method according to claim 1, characterized in that The steps of constructing the dimension-raising feature function include: Constructing dimensionality-increasing feature functions using a set of basis functions , dimension-raising feature function The expression is: in, represents the core of the basis function, Represents the width of the basis function.

3. The method according to claim 1, characterized in that The steps of constructing the dimension-raising feature function include: Constructing dimensional feature functions using multi-layer neural networks , dimension-raising feature function The expression is: in, L Indicates the number of network layers, j The output of the hidden layer is described as: in, It is j -The output of the 1-layer network, represents the weight matrix, represents the activation function, Represents the bias term.

4. The method according to claim 1, characterized in that: The method further comprises: simplifying the Q function to: in, , , , , .

5. The method according to claim 1, characterized in that The method further comprises: Setting up the data sample The data set is ,remember , in order to Full rank, the control strategy is defined as: in, d is zero mean noise.

6. The method according to claim 5, characterized in that The method further comprises: Update Value The calculation formula is: in, ; By iterative least squares update for: in, represents the intermediate square matrix in the update process, whose row and column dimensions are the same as The dimensions of the vector columns are consistent.

7. A robot data-driven control device based on policy iteration solution, used to implement the robot data-driven control method based on policy iteration solution according to any one of claims 1 to 6, characterized in that: The device comprises: The conversion module is used to construct the control strategy of the robot dynamics model according to the linear gain matrix to be optimized and the dimension-raising characteristic function; the dimension-raising characteristic function maps the original state space of the robot to the high-dimensional characteristic space in the Koopman operator framework, thereby converting the nonlinear optimization problem of the control strategy into a high-dimensional linear optimization problem; the control strategy u The nonlinear optimization problem is transformed into a high-dimensional linear optimization problem: in, , , ; The high-dimensional linear optimization problem satisfies the model constraints in high dimensions: express The next state, and Both represent model errors, All of them are for solving the model parameters of the high-dimensional feature space; The Q function definition module is used to define the Q function in a linear space as: in, , Represents the dimension-raising feature function A simplified representation of represents the control strategy, , is the model error, represents the discount factor, represents the state value function; A Q function conversion module is used to define, according to the Q function, ,in , , represents the spatial dimension of the dimensionality-raising feature function, represents the dimension of the control strategy, is with Related Matrix Vectorization of Control strategy calculation module for use in a data-driven manner The data sample iteratively calculates the matrix Update value of , calculate the linear gain matrix to obtain the control strategy u of the robot dynamics model.

Citation Information

Patent Citations

  • Mobile robot trajectory tracking control method based on strategy iteration

    CN112965487A

  • Baxter mechanical arm trajectory tracking control method based on reinforcement learning

    CN113199477A

  • Baxter mechanical arm intelligent optimization control method based on value iteration

    CN113290554A

  • Robot control method, device and equipment and storage medium

    CN118034038A

  • Device and method for controlling a robot

    EP3904973A1