A model-free inverse reinforcement learning method fusing q-learning and sum-of-squares relaxation

Abstract

The application provides a model-free inverse reinforcement learning method combining Q learning and sum of squares relaxation, and relates to the field of optimal control design. The application fixes the cost function parameters, converts the policy evaluation problem into a semi-definite programming with SOS constraint, and obtains the policy update by solving the convex optimization problem, so that the HJB equation is no longer directly solved, and the calculation time is reduced. The application represents the Q function and the state cost function as a polynomial form, and on this basis, introduces the SOS relaxation and semi-definite programming, so that the parameter update can be completed only by relying on the state, control input and instant cost data collected during operation, and the explicit system dynamics model is not needed, and the model-free online inverse reinforcement learning control of the continuous-time nonlinear system can be realized. The application represents the state cost function as a polynomial quadratic form, identifies the cost parameters by using the expert demonstration data, so that the obtained cost function approximates the expert implicit cost function in a given function family, and high-quality policy imitation is realized.

Description

Technical Field

[0001] This invention relates to the field of optimal control design, specifically a model-free inverse reinforcement learning method that integrates Q-learning and sum-of-squares relaxation. Background Technology

[0002] In traditional optimal control design, a suitable cost function is usually pre-constructed, and the optimal control law is obtained by solving the Hamilton-Jacobi-Bellman (HJB) equation or related optimality conditions. However, in practical engineering, the cost function is often implicit in the "expert control strategy" and is difficult to design directly. To address this, Inverse Reinforcement Learning (IRL) is introduced. First, expert demonstration data is used to identify the equivalent implicit cost function from the expert control strategy. Then, based on this, the weights of each component are adjusted according to actual optimization objectives such as energy consumption, safety margin, and comfort. This allows for the acquisition of an optimal control strategy that better meets engineering requirements while inheriting the expert control experience.

[0003] Most mainstream IRLs currently adopt a "two-layer structure": the outer layer is responsible for updating the cost function parameters, while the inner layer needs to re-approximate the optimal control problem after each cost parameter update (for example, by repeatedly training a neural network to approximate the HJB equation). This frequently triggers high-cost calculations throughout the iteration process, resulting in a significant increase in the overall computational burden and time overhead.

[0004] To alleviate the difficulty of directly solving the HJB equations, the Sum-of-Squares (SOS) method has been introduced into optimal control and adaptive dynamic programming in recent years. Its basic idea is to construct Lyapunov functions and their derivative terms on a polynomial basis, transforming the closed-loop stability test into a positive semidefinite constraint on the Gram matrix. This allows for the use of SOS-constrained semidefinite programming (SDP) to replace the direct solution of the HJB equations, obtaining a closed-loop stable suboptimal control strategy with relatively low computational cost while ensuring stability. However, most existing model-free frameworks based on SOS relaxation remain at the offline solution level, making it difficult to continuously update the strategy using real-time data during system operation, and thus failing to achieve truly online learning and adaptive optimization.

[0005] On the other hand, Q-learning, as a typical model-free adaptive dynamic programming method, only relies on state-action-reward samples collected during interaction with the environment to iteratively update the Q-function (action-value function), without requiring an explicit system dynamics model. Existing research has extended Q-learning to continuous-time systems and applied it to optimal control and robust control design to improve online learning capabilities and sample utilization efficiency. Therefore, organically combining the model-free online learning advantages of Q-learning with the stability guarantee capability of SOS relaxation is an important approach to achieving optimal control of continuous-time nonlinear systems through free inverse reinforcement learning.

[0006] Generally speaking, existing technologies can be broadly categorized as follows:

[0007] 1. Inverse reinforcement learning method based on HJB: The cost function is recovered through expert demonstration, but each update of the cost parameters requires re-approximate solution of HJB, which has a large computational load;

[0008] 2. Optimal control method based on SOS relaxation: It uses SOS and semidefinite programming to achieve suboptimal control under stability constraints, but it usually still relies on the system model or complete trajectory and cannot achieve model-free online learning.

[0009] 3. Model-free adaptive dynamic programming method based on Q-learning: It focuses on online policy optimization and value function approximation. Although there are some studies on continuous-time systems, they are mainly concentrated on forward optimal control and robust control design. There are no public reports on the systematic combination of it with inverse reinforcement learning and SOS relaxation framework for model-free inverse reinforcement learning optimal control.

[0010] Existing technologies still have the following main problems in inverse reinforcement learning optimal control of continuous-time nonlinear systems:

[0011] 1. Two-tier architecture has high computational overhead.

[0012] Traditional IRL methods typically employ a two-layer structure: an outer layer for cost parameter updates and an inner layer for HJB (Hardware-Judges-Brains) solution. Each time the outer layer updates the cost function parameters, the inner layer needs to re-approximate the optimal control problem (e.g., through repeated iterations via a neural network). This results in the high-cost numerical solution being triggered multiple times during the outer layer iterations, leading to a significant accumulation of computational burden and high time consumption, making it difficult to meet real-time requirements.

[0013] 2. The SOS relaxation framework is difficult to apply online without a model.

[0014] Existing optimal control and adaptive dynamic programming methods based on SOS mostly require explicit system dynamics models or complete state trajectory information. In real industrial scenarios, system models are often partially unknown or difficult to model accurately, making online learning based solely on measurement data more in line with engineering practice; however, existing SOS frameworks lack system support for "model-free, online IRL".

[0015] 3. The SOS constraint in high-dimensional systems faces feasibility and numerical stability issues.

[0016] In high-dimensional state spaces, requiring the Lyapunov function or related polynomial to be positive definite / non-negative over the global working domain to make the SOS constraint feasible often leads to shrinkage of the feasible region, increased conservatism, and numerical instability; this further limits the application of the SOS method in complex nonlinear systems. Summary of the Invention

[0017] To address the shortcomings of existing technologies, the present invention aims to propose a model-free inverse reinforcement learning method that integrates Q-learning and sum-of-squares relaxation, comprising:

[0018] Step 1: Using the piezoelectric cantilever beam system as an expert system, construct the dynamic state equation of the expert system and obtain the demonstration dataset of the expert system. The demonstration dataset includes multiple sampling times, as well as the state variables and control inputs at each sampling time.

[0019] Step 2: Based on the dynamic state equation of the expert system, construct the dynamic state equation of the learning system, and then define the instantaneous cost of the learning system;

[0020] Step 3: Construct based on the real-time cost of the learning system Functions and based on polynomial basis function vector pairs The function is approximated by parameterization to obtain a parameterized approximation. The function partitions the polynomial basis function vector to obtain the partitioned basis function vector;

[0021] Step 4: According to Function definition: Lyapunov operator;

[0022] Step 5: Based on the demonstration dataset, parameterized approximation The function, the partitioned basis function vector, and the Lyapunov operator are subjected to multiple iterations to find the optimal control input.

[0023] Optionally, step 1 involves constructing the dynamic state equations of the expert system, including:

[0024] The dynamic state equations of the expert system are constructed as follows:

[0025] ;

[0026] Among them, state variables ,in, For displacement, For speed, This is the piezoelectric output voltage. Indicates matrix transpose. for The time derivative, for The time derivative, for The time derivative; Indicates the linear stiffness coefficient; Indicates the equivalent mass of the vibration system; Indicates the cubic nonlinear stiffness coefficient; Indicates the linear damping coefficient; Indicates the nonlinear damping coefficient; Indicates the electromechanical coupling coefficient; For control input; This represents the equivalent capacitance of a piezoelectric element; Indicates the external load resistance or equivalent resistance;

[0027] The dynamic state equations of the expert system are transformed to obtain the dynamic state equations, which are expressed as follows:

[0028] ;

[0029] in, for The time derivative, This represents the drift dynamics term of the expert system; This represents the control input gain function.

[0030] Optionally, step 2 specifically includes:

[0031] Based on the dynamic state equation of the expert system, the dynamic state equation of the learning system is constructed as follows:

[0032] ;

[0033] in, To learn the state vector of the system, For the control input of the learning system, for The time derivative; In the dynamic state equations of expert systems Consistent;

[0034] Define the instant cost of the learning system , is represented as:

[0035] ;

[0036] in, It is the state cost function of the learning system. It is a positive definite weight matrix.

[0037] Optionally, step 3 specifically includes:

[0038] Introducing value functions and its gradient And define the Hamiltonian function. , is represented as:

[0039] ;

[0040] in, This means that the Hamiltonian function is defined as the formula on the right side of the equals sign;

[0041] Learning systems based on Hamiltonian functions The function is represented as:

[0042] ;

[0043] To facilitate the subsequent establishment of a learnable linear parameterization form, The function is divided into state cost term, control input cost term, drift mechanics term, control input coupling term, and value function term, specifically represented as follows:

[0044] ;

[0045] ;

[0046] ;

[0047] ;

[0048] ;

[0049] ;

[0050] in, Represents the state cost term; Indicates the control input cost term; Represents the drift motion mechanical terms; Indicates the control input coupling term; Represents the value function term;

[0051] Within the working domain comprised of all combinations of state and control inputs, polynomial basis function vectors are selected, and based on these vectors... Perform approximate parameterization to obtain a parameterized approximation. The function is represented as:

[0052] ;

[0053] in, The parameter vector to be identified, These are the parameter vectors to be identified corresponding to the state cost term, the control input cost term, the drift mechanics term, and the control input coupling term, respectively. This is the vector of parameters to be identified corresponding to the value function term;

[0054] For basis function vectors The partitioning process yields the basis function vectors, represented as follows:

[0055] ;

[0056] in, For the state cost term The corresponding polynomial basis function vector; To control input cost terms The corresponding polynomial basis function vector; To the system drift mechanics terms The corresponding polynomial basis function vector; For coupling with control input terms The corresponding polynomial basis function vector; For the AND-valued function term The corresponding polynomial basis function vector.

[0057] Optionally, the Lyapunov operator described in step 4 Represented as:

[0058] ;

[0059] in, Let be the coefficient vector of the polynomial on the basis vectors. ; It represents a polynomial or monomial basis vector consisting of state variables and control inputs, with a total degree not exceeding 2d.

[0060] Optionally, step 5 specifically includes:

[0061] Step 5.1: Set the outer loop iteration count k and the inner loop iteration count i, let k=0, i=1; set the initial control input. Set the inner loop control convergence threshold Convergence threshold of outer loop parameters Initialize the parameter vector to be identified. ;

[0062] Step 5.2: During the inner loop iteration process, based on the Lyapunov operator and parameterized approximation... The function and the partitioned basis function vectors, and the inner loop control convergence threshold. By performing multiple inner loop iterations, the optimal control input for the inner loop is obtained. and the identified parameter vector ;

[0063] Step 5.3: According to and calculate Specifically, this is achieved through the following formula:

[0064] ;

[0065] in, for The identified parameter vector corresponding to the value function term in the value function term. express Basis functions of the time-value function;

[0066] Step 5.4: Based on the partitioned basis function vectors, the example dataset, and the parameterized approximation function, , Perform multiple outer loop iterations to obtain the outer loop's th iteration. The parameter vector corresponding to the state cost term in the next iteration Get the value In ,exist At that time, As the optimal control input equivalent to expert control; in At that time, let k+1, and... In Replace with Return to step 5.2.

[0067] Optionally, step 5.2 specifically includes:

[0068] Step 5.2.1: Let the expressions for the Lyapunov operator in the i-th iteration of the inner loop and the k-th iteration of the outer loop be: ,in, Let be the coefficient vector of the polynomials in the basis vectors for the i-th iteration of the inner loop and the k-th iteration of the outer loop. Define the control inputs for the i-th iteration of the inner loop and the k-th iteration of the outer loop. For the sum of squares constraint, define at the same time For the sum of squares constraint, where, Let be the vector of parameters to be identified in the (i-1)th iteration of the inner loop and the kth iteration of the outer loop. The vectors to be identified are the inner loop's i-th iteration and the outer loop's k-th iteration; using... For the sum of squares constraint and Solving the objective function yields the following results: The value;

[0069] The optimization objective function is expressed as:

[0070] ;

[0071] ;

[0072] Among them, c Q For intermediate parameters, Indicates the working domain. It is a weighted function;

[0073] Step 5.2.2: Based on the partitioned basis function vectors and parameterized approximations The function yields the equation for the control input coupling term, expressed as:

[0074] ;

[0075] in, for The identified parameter vector corresponding to the control input coupling term in the input vector. Let i be the gradient of the value function when the inner loop iteration number is i and the outer loop iteration number is k.

[0076] The strategy update formula is obtained and expressed as:

[0077] ;

[0078] in, This serves as the control input for the (i+1)th iteration of the inner loop and the kth iteration of the outer loop;

[0079] By performing an equivalent transformation on the equations for the control input coupling terms, we obtain the equations for state-control separation, which are expressed as follows:

[0080] ;

[0081] in, For control basis functions, Indicates the Kronecker product;

[0082] Based on the state-control separation equation, the policy update formula is transformed, and the transformed policy update formula is used to calculate... The value of ; where the transformed policy update formula is expressed as:

[0083] ;

[0084] Step 5.2.3: In At that time, As the optimal control input of the inner loop ,Will As the identified parameter vector ;exist When i is reached, increment i by 1 and return to step 5.2.1.

[0085] Optionally, in step 5.4, based on the partitioned basis function vectors, the example dataset, and the parameterized approximation... function, , Perform multiple outer loop iterations to obtain the outer loop's th iteration. The parameter vector corresponding to the state cost term in the next iteration The values ​​include:

[0086] Using the demonstration dataset to divide the time period The collected data is used to construct a state cost update function, which is represented as follows:

[0087] ;

[0088] in, For the outer ring The state cost function for the next iteration Central sampling time for demonstration dataset The control input, Let be the value function of the outer loop at time t+T for the kth iteration;

[0089] Based on the partitioned basis function vectors and parameterized approximations The function yields the equation for the state cost term, expressed as:

[0090] ;

[0091] in, for The identified parameter vector corresponding to the state cost term in the data;

[0092] The sample dataset and use Control basis functions Represented as:

[0093] ;

[0094] ;

[0095] ;

[0096] in, As the control input for sampling time t in the demonstration dataset, and These respectively represent the use of control basis functions After and The weight vector, for The identified parameter vector corresponding to the control input coupling term in the text;

[0097] according to Constructing the control intermediate matrix Specifically, this is achieved through the following formula:

[0098] ;

[0099] Combining the equations of state cost terms and the control intermediate matrix The state cost update function is transformed to obtain the transformed state cost update function, which is expressed as:

[0100] ;

[0101] in, For the outer ring The parameter vector corresponding to the state cost term in the next iteration. for The basis function values ​​at time t;

[0102] Define performance index parameters based on the equation of the state cost term. , is represented as:

[0103] ;

[0104] Using gradient descent, based on right Update and get The value of is achieved through the following formula:

[0105] ;

[0106] ;

[0107] in, For adaptive step size, The adjustment coefficient for the adaptive step size. To prevent the denominator from being zero or an excessively small positive number, This is a data matrix or a regression matrix.

[0108] The beneficial effects of adopting the above technical solution are as follows:

[0109] 1. Significantly reduces computational complexity and resource consumption.

[0110] Existing inverse reinforcement learning methods often employ a two-layer structure: an outer layer that updates the cost function and an inner layer that repeatedly solves the HJB equation. The inner layer typically requires solving large-scale nonlinear equations or training deep neural networks, which is computationally intensive, consumes a lot of memory, and is difficult to meet real-time requirements.

[0111] This invention transforms the policy evaluation problem into a semidefinite programming problem with SOS constraints, under the premise of fixed cost function parameters. Policy updates are obtained by solving a convex optimization problem, eliminating the need for direct solution of the HJB equations. Simulations show that, under conditions of similar control performance, the single-cycle computation time and total computation time of the inner iteration of this invention are significantly lower than traditional methods, making it advantageous for implementation on resource-constrained devices such as industrial controllers.

[0112] 2. Enables online inverse reinforcement learning with a flexible model, eliminating the need for an explicit system model.

[0113] Traditional SOS-based optimal control methods mostly rely on accurate system dynamics models, requiring pre-defined analytical forms and parameters of the controlled object. This makes them difficult to apply to complex objects with incomplete models or those exhibiting time-varying or uncertain factors. While existing model-free SOS frameworks mitigate the reliance on accurate models to some extent, they typically only offer offline solutions and cannot utilize real-time data to learn and update control strategies and cost functions online during system operation. This invention, based on Q-learning, unifies the Q-function and state cost function into polynomial form. Furthermore, it introduces SOS relaxation and semidefinite programming, enabling parameter updates solely based on state, control input, and real-time cost data collected during operation. This achieves model-free online inverse reinforcement learning control of continuous-time nonlinear systems without requiring an explicit system dynamics model.

[0114] 3. Capable of uncovering and adjusting the implicit cost function of experts.

[0115] This invention represents the state cost function as a polynomial quadratic form, uses expert demonstration data to identify cost parameters, and makes the obtained cost function approximate the expert implicit cost function within a given family of functions. The corresponding control strategy is basically consistent with the expert strategy in terms of state response and control energy, thus achieving high-quality strategy imitation.

[0116] Building upon this foundation, while retaining the structural form of the expert cost function, the weight coefficients of each state variable and control variable can be quantitatively adjusted according to actual engineering needs. For example, penalties can be strengthened for indicators such as energy consumption, overshoot, safety margin, actuator saturation, or ride comfort, achieving a trade-off between multiple objectives such as "performance, energy consumption, and safety constraints." By simply modifying the weights, optimized cost functions and corresponding control strategies adapted to different operating conditions and control objectives can be quickly obtained without redesigning the overall algorithm, facilitating engineering deployment and subsequent maintenance. Attached Figure Description

[0117] Figure 1 This is a flowchart illustrating a model-free inverse reinforcement learning method that integrates Q-learning and sum-of-squares relaxation in an embodiment of the present invention.

[0118] Figure 2 This is a flowchart illustrating another model-free inverse reinforcement learning method that integrates Q-learning and sum-of-squares relaxation in an embodiment of the present invention.

[0119] Figure 3 This is the convergence graph of the inner loop algorithm control in this embodiment of the invention;

[0120] Figure 4 This is a three-dimensional image showing the change of the Lyapunov function V value with iteration during the inner-loop strategy evaluation process in this embodiment of the invention.

[0121] Figure 5 This is a schematic diagram of the convergence process of the outer loop state cost function parameters in an embodiment of the present invention. Detailed Implementation

[0122] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0123] To address the problems existing in the prior art, this invention provides a model-free inverse reinforcement learning method that integrates Q-learning and sum-of-squares relaxation, achieving the following objectives without relying on an explicit model of the system:

[0124] To address the challenges of multi-objective and multi-constraint requirements in practical engineering, such as energy consumption, safety, and comfort, traditional optimal control requires the pre-construction of cost functions and struggles to adaptively adjust them based on operating conditions. This paper introduces inverse reinforcement learning to extract equivalent implicit cost functions from expert control strategies. Based on these, the weights of each component are adjusted according to actual needs, achieving adaptive optimization design of the cost function while inheriting expert experience.

[0125] To address the problem that traditional inverse reinforcement learning often employs a two-layer optimization architecture of "outer layer cost function update + inner layer HJB equation solution," which leads to high-cost HJB numerical solutions being repeatedly triggered during iteration, resulting in large computational loads and difficulty in engineering implementation, we introduce the SOS relaxation framework. This framework transforms the inner-layer optimal control problem into a semidefinite programming problem with SOS constraints, and obtains a suboptimal control law that meets stability requirements, thereby significantly reducing the computational complexity and resource consumption of the inner-layer solution.

[0126] To address the limitations of existing model-free SOS relaxation methods, which struggle to utilize runtime data for online learning and are limited to offline design, this paper organically combines the model-free adaptive dynamic programming concept of Q-learning with SOS relaxation. This approach updates policy and cost parameters based solely on sampled data such as state, control input, and immediate cost, enabling online policy optimization and model-free inverse reinforcement learning-based optimal control for continuous-time nonlinear systems. The effectiveness of this method in policy convergence speed, cost function identification accuracy, and closed-loop control performance is verified in a simulation system.

[0127] To enable those skilled in the art to better understand the present invention, the following detailed description of the model-free inverse reinforcement learning optimal control method combining Q-learning and sum-of-squares (SOS) relaxation framework is provided in conjunction with specific embodiments and simulation experiments.

[0128] Consider a vibration energy recovery system (such as a piezoelectric cantilever beam system) with nonlinear damping and stiffness characteristics, which can be used for structural health monitoring or self-powering of wireless sensing nodes. Expert systems have been carefully designed to achieve an optimal balance between energy recovery and oscillation suppression in broadband vibration environments. However, the precise cost function upon which this system operates (balancing displacement constraints, voltage output, control energy consumption, etc.) is unknown to learners (such as newly deployed or similar systems requiring adaptive adjustment).

[0129] The challenge for learners lies in the difficulty of accurately modeling the nonlinear dynamics of the system (such as cubic stiffness and nonlinear damping), and the potential for variations in environmental vibration characteristics. The learner's goal is to infer the implicit cost function of the expert system by observing its state response (displacement, velocity, output voltage) under typical excitations and the corresponding optimal control inputs (such as adjustable resistors or switching actions in a shunt circuit), without establishing an accurate explicit dynamic model, and ultimately generate a control strategy that can reproduce its optimal performance.

[0130] Specifically, this invention provides a model-free inverse reinforcement learning method that integrates Q-learning and sum-of-squares relaxation, combining... Figure 1 and Figure 2 This may include the following steps:

[0131] A model-free inverse reinforcement learning optimal control method combining Q-learning and a sum-of-squares relaxation framework, characterized by comprising:

[0132] Step 1: Using the piezoelectric cantilever beam system as an expert system, construct the dynamic state equation of the expert system and obtain the demonstration dataset of the expert system. The demonstration dataset includes multiple sampling times, as well as the state variables and control inputs at each sampling time.

[0133] Specifically, the dynamic state equations of the expert system are constructed as follows:

[0134] ;

[0135] Among them, state variables ,in, For displacement, For speed, This is the piezoelectric output voltage. Indicates matrix transpose. for The time derivative, for The time derivative, for The time derivative; Indicates the linear stiffness coefficient; Indicates the equivalent mass of the vibration system; This represents the cubic nonlinear stiffness coefficient, used to describe the nonlinear stiffness characteristics in elastic restoring forces. Indicates the linear damping coefficient; This represents the nonlinear damping coefficient, used to describe velocity-dependent or displacement-dependent nonlinear energy dissipation effects. This represents the electromechanical coupling coefficient, used to characterize the energy conversion relationship between mechanical vibration and voltage output; The control input is the optimal force (or equivalent control voltage) applied to the actuator. This represents the equivalent capacitance of a piezoelectric element; Indicates the external load resistance or equivalent resistance; For the cubic stiffness term, This is a nonlinear damping term;

[0136] The dynamic state equations of the expert system are transformed to obtain the dynamic state equations, which are expressed as follows:

[0137] ;

[0138] in, for The time derivative, This represents the drift dynamics term of an expert system, which is the nonlinear dynamic determined by the system's own mass, stiffness, damping, and electromechanical coupling relationship in the absence of control input. This represents the control input gain function, used to describe the control input. Impact on changes in system state;

[0139] Furthermore, define instant cost :

[0140] ;

[0141] in, It is a state-related cost term of the expert system and a positive definite function over the working domain (the physical range of values ​​for state and control variables allowed for the safe and stable operation of the expert system). As a symmetric positive definite matrix, this instantaneous cost form ensures that, while satisfying the optimality condition, the analytical optimal control law and HJB condition are easily derived. Therefore, the cumulative cost of the expert system is defined as:

[0142] ;

[0143] During the operation of the piezoelectric cantilever beam system, the state variables and control inputs at each moment are collected to generate a demonstration dataset.

[0144] Specifically, a mature, high-performance controller is deployed as an expert controller in the actual system or high-fidelity simulation platform. This expert controller outputs expert-optimal (or engineering-obtainable near-optimal) control commands in real time. Subsequently, under expert optimal control The system operates under the action of sampling period. Discrete sampling is performed on the system to obtain the timestamp sequence. (in At each sampling time, the corresponding expert state and expert control input are recorded to form a demonstration dataset: The meanings of each symbol are as follows: The sampling point number, ; This indicates the total number of sampling points (i.e., the number of data entries). Indicates the first The timestamp of each sampling moment; Indicates at the sampling time The acquired state vector of the expert system (corresponding to the one defined in S1) wait); Indicates at the sampling time The expert-applied control inputs (corresponding to the control variables in S1) This dataset is used for subsequent inverse reinforcement learning steps to make the learned control policy behave in a way that approximates the expert demonstration trajectory.

[0145] Step 2: Based on the dynamic state equation of the expert system, construct the dynamic state equation of the learning system, and then define the instantaneous cost of the learning system;

[0146] To facilitate the subsequent derivation of the HJB conditions, the construction of the SOS optimization problem, and the design of the control strategy to be learned, the same controlled object during the learning phase is referred to as the learning system, i.e., the expert system and the learning system. Equally, specifically, based on the dynamic state equations of the expert system, the dynamic state equations of the learning system are constructed, expressed as:

[0147] ;

[0148] in, To learn the state vector of the system, For the control input of the learning system, for The time derivative; In the dynamic state equations of expert systems They are the same, the only difference being that S1-S3 use expert control input. This step introduces learner-controlled input, which needs to be learned through study. ; These are the control variables that this invention aims to solve, with the goal of achieving safe and stable performance within the working domain and approximating expert performance; the initial conditions are... And assume (The origin is the equilibrium point) so that stability and optimality analysis can be performed later based on the value function and HJB conditions.

[0149] Define the instant cost of the learning system , is represented as:

[0150] ;

[0151] in, It is the state cost function of the learning system, that is, the state-related cost term of the learning system, and it is a positive definite function in the working domain. It is a positive definite weight matrix;

[0152] Therefore, the cumulative cost of the learning system Defined as:

[0153] ;

[0154] Among them, the learning system can learn its current cost function. However, the expert cost function that can explain expert behavior is unknown. (or its parameters).

[0155] Hypothesis 1: The learning system can observe the demonstration dataset obtained by the expert system. .

[0156] Assumption 2: Learners are not required to have explicit knowledge of the system dynamics. The analytical expression can be learned and solved by relying solely on the sampled data.

[0157] Step 3: Construct based on the real-time cost of the learning system Functions and based on polynomial basis function vector pairs The function is approximated by parameterization to obtain a parameterized approximation. The function partitions the polynomial basis function vector to obtain the partitioned basis function vector;

[0158] To transform the HJB conditions for continuous-time optimal control into a form usable for data-driven learning, a value function is introduced. and its gradient And define the Hamiltonian function. , is represented as:

[0159] ;

[0160] in, This means that the Hamiltonian function is defined as the formula on the right side of the equals sign;

[0161] in, Represents the state vector The optimal cumulative cost (value function) for starting. express For the state vector gradient vector, Let Hamiltonian function be used to represent the state. Apply control Real-time cost With the rate of change of the function along the system dynamics The sum is used in HJB conditions for... Minimize to obtain optimal control.

[0162] Learning systems based on Hamiltonian functions The function is represented as:

[0163] ;

[0164] To facilitate the subsequent establishment of a learnable linear parameterization form, The function is divided into state cost term, control input cost term, drift mechanics term, control input coupling term, and value function term, specifically represented as follows:

[0165] ;

[0166] ;

[0167] ;

[0168] ;

[0169] ;

[0170] ;

[0171] in, This represents the state cost term associated with the learning system's state, used to evaluate the degree to which the current state deviates from the desired equilibrium point or the target state; This represents the control input cost term, used to measure the energy consumption of control or the intensity of control action; The drift kinetics term represents the contribution of the drift kinetics of the learning system to the rate of change of the value function, reflecting the impact of the nonlinear dynamics of the learning system itself on long-term performance indicators in the absence of control input. This represents the control input coupling term, which indicates the contribution of the control input to the rate of change of the value function through the input channel, reflecting the direct impact of the control strategy on the system state evolution and performance optimization. The value function term, i.e. the value function term corresponding to the next time step or the state vector, is used to describe the cumulative performance cost in the subsequent evolution process starting from the current state.

[0172] In the working domain consisting of all combinations of state and control inputs, select polynomial basis function vectors. (number of times) (Based on polynomial basis function vector pairs) Perform approximate parameterization to obtain a parameterized approximation. The function is represented as:

[0173] ;

[0174] in, The parameter vector to be identified, These are the parameter vectors to be identified corresponding to the state cost term, the control input cost term, the drift mechanics term, and the control input coupling term, respectively. The parameter vector to be identified for the value function term; parameter vector The results will be obtained in subsequent steps based on the sampled data and through SOS relaxation optimization constraint estimation.

[0175] according to The structure of each component term in the polynomial basis function vector The division is represented as follows:

[0176] ;

[0177] in, For the state cost term The corresponding polynomial basis function vector; To control input cost terms The corresponding polynomial basis function vector; To the system drift mechanics terms The corresponding polynomial basis function vector; For coupling with control input terms The corresponding polynomial basis function vector; For the AND-valued function term The corresponding polynomial basis function vector;

[0178] Step 4: According to Function definition: Lyapunov operator;

[0179] Among them, the Lyapunov operator Represented as:

[0180] ;

[0181] in, Let be the coefficient vector of the polynomial on the basis vectors. ; Represents a polynomial or monomial basis vector consisting of state variables and control inputs, with a total degree not exceeding 2d;

[0182] To be Defined as an SOS constraint, it must satisfy:

[0183] ;

[0184] ;

[0185] in, This represents the highest total degree of the polynomial (for vectors / matrices, it represents the maximum degree of each component / element). This is based on the previously defined... ,Know The order of the polynomial is .

[0186] Furthermore, the Lyapunov operator SOS constraint can be written as:

[0187] ;

[0188] in, Represents state variables With control input Construct a basis vector of polynomials or monomials with a total degree not exceeding d. express It is a positive semi-definite matrix, thus guaranteeing The nonnegativity of .

[0189] Step 5: Based on the demonstration dataset, parameterized approximation The function, the partitioned basis function vector, and the Lyapunov operator are subjected to multiple iterations to find the optimal control input;

[0190] To ensure the feasibility of the strategy evaluation and improvement process and the stability of the system, the following assumptions are set:

[0191] Assumption 3: There exists a continuously differentiable function , belongs to set:

[0192] ;

[0193] in, yes The set of continuously differentiable functions of the first order. This represents a candidate Lyapunov function used to characterize the system state. A measure of energy or stability relative to the equilibrium point. (Set) It is to satisfy And for any have The set of positive definite candidate Lyapunov functions, and the corresponding control laws. This makes Lyapunov's inequality:

[0194] ;

[0195] satisfy

[0196] ;

[0197] This assumption guarantees that at least one control strategy exists to stabilize the system.

[0198] Assumption 4: Exists This makes the Hamiltonian equation hold:

[0199] ;

[0200] in, This represents the optimal value function, which describes how the system, under the optimal control strategy, changes from its current state. The minimum cumulative performance index corresponding to the process from start to equilibrium. It not only satisfies the positive definiteness condition, but also satisfies the Hamilton-Jacobi-Bellman equation. The Hamiltonian is:

[0201] ;

[0202] The optimal control law can then be expressed as:

[0203] ;

[0204] Step 5.1: Set the outer loop iteration count k and the inner loop iteration count i, let k=0, i=1; set the initial control input. Set the inner loop control convergence threshold Convergence threshold of outer loop parameters Initialize the parameter vector to be identified. ;

[0205] Furthermore, the present invention can also be used for... Function initialization is represented as:

[0206] ;

[0207] in, This indicates that the inner loop iteration count is 1 and the outer loop iteration count is 0. function, This is the value function when the inner loop iteration count is 1 and the outer loop iteration count is 0. for The gradient; where Assumption 3 must be satisfied to ensure system stability;

[0208] Step 5.2: During the inner loop iteration process, based on the Lyapunov operator and parameterized approximation... The function and the partitioned basis function vectors, and the inner loop control convergence threshold. By performing multiple inner loop iterations, the optimal control input for the inner loop is obtained. and the identified parameter vector ;

[0209] Step 5.2.1: Let the expressions for the Lyapunov operator in the i-th iteration of the inner loop and the k-th iteration of the outer loop be: ,in, Let be the coefficient vector of the polynomials in the basis vectors for the i-th iteration of the inner loop and the k-th iteration of the outer loop. Define the control inputs for the i-th iteration of the inner loop and the k-th iteration of the outer loop. For the sum of squares constraint, define at the same time For the sum of squares constraint, where, Let be the vector of parameters to be identified in the (i-1)th iteration of the inner loop and the kth iteration of the outer loop. The vectors to be identified are the inner loop's i-th iteration and the outer loop's k-th iteration; using... For the sum of squares constraint and Solving the objective function yields the following results: The value of ; where, in obtaining The value can be calculated afterward. The value of the function.

[0210] It should be noted that when i=1, only the following method is used. To constrain the sum of squares, the objective function is solved. When i=2, the following approach is used: For the sum of squares constraint and Solve the optimization objective function with the sum of squares constraint.

[0211] The optimization objective function is expressed as:

[0212] ;

[0213] ;

[0214] Among them, c Q For intermediate parameters, Indicates the working domain. The weighting function is used to adjust the importance distribution of the optimization objective within the working domain. Introducing the weighting function ensures the feasibility and numerical stability of the sum-of-squares optimization problem and is a key factor in achieving the solvability of the problem.

[0215] in, and The sum-of-squares (SOS) constraint means that both constraints can be expressed as a polynomial sum of squares, thus restricting their nonnegativity. While minimizing the objective function, the other two constraints must also be satisfied. This means that, given a basis function structure, optimization of the parameter vector is achieved. This makes the approximate state-action value function The goal is to minimize the weighted integral within the working domain. This objective is used to approximate the minimum cumulative performance index corresponding to the optimal value function. Two SOS constraints are used to ensure the feasibility and stability of the optimization process. The first SOS constraint... It is SOS, used to guarantee the state cost term of the Lyapunov operator. The nonnegativity of the SOS polynomial. Since the SOS polynomial is always nonnegative, this constraint ensures that the state cost function satisfies the positive definite or semi-positive definite requirement within the working domain, thus guaranteeing that the optimization problem has reasonable performance evaluation significance. The second SOS constraint. It is SOS, used to ensure that the value function obtained in the current iteration is approximately no greater than the result of the previous iteration, i.e. This constraint reflects the monotonicity of the policy improvement process, ensuring that the performance index does not increase after each iteration, thus guaranteeing that the inner-loop policy evaluation and improvement process updates towards a better direction. Therefore, the optimization objective is... The two SOS constraints are used to find smaller approximate function parameters, while the two SOS constraints guarantee the non-negativity of the state cost term and the monotonic improvement between adjacent iterations, respectively.

[0216] Step 5.2.2: Based on the partitioned basis function vectors and parameterized approximations The function yields the equation for the control input coupling term, expressed as:

[0217] ;

[0218] in, for The identified parameter vector corresponding to the control input coupling term in the input vector. Let i be the gradient of the value function when the inner loop iteration number is i and the outer loop iteration number is k.

[0219] The strategy update formula is obtained and expressed as:

[0220] ;

[0221] in, This serves as the control input for the (i+1)th iteration of the inner loop and the kth iteration of the outer loop;

[0222] By performing an equivalent transformation on the equations for the control input coupling terms, we obtain the equations for state-control separation, which are expressed as follows:

[0223] ;

[0224] in, For control basis functions, This represents the Kronecker product; it should be noted that, after obtaining... Then the parameters on both sides of the equation can be calculated to obtain specific values;

[0225] Based on the state-control separation equation, the policy update formula is transformed, and the transformed policy update formula is used to calculate... The value of ; where the transformed policy update formula is expressed as:

[0226] ;

[0227] Step 5.2.3: In At that time, As the optimal control input of the inner loop ,Will As the identified parameter vector ;exist When i is reached, increment i by 1 and return to step 5.2.1;

[0228] Step 5.3: According to and ,calculate and Specifically, this is achieved through the following formula:

[0229] ;

[0230] ;

[0231] in, for The identified parameter vector corresponding to the value function term in the value function term. express The basis function values ​​of the time-valued function;

[0232] Step 5.4: Based on the partitioned basis function vectors, the example dataset, and the parameterized approximation function, , Perform multiple outer loop iterations to obtain the outer loop's th iteration. The parameter vector corresponding to the state cost term in the next iteration Get the value In ,exist At that time, As the optimal control input equivalent to expert control; in At that time, let k+1, and... In Replace with Return to step 5.2.

[0233] Using the demonstration dataset to divide the time period The collected data is used to construct a state cost update function, which is represented as follows:

[0234] ;

[0235] in, For the outer ring The state cost function for the next iteration Central sampling time for demonstration dataset The control input, Let be the value function of the outer loop at time t+T for the kth iteration;

[0236] Based on the partitioned basis function vectors and parameterized approximations The function yields the equation for the state cost term, expressed as:

[0237] ;

[0238] in, for The identified parameter vector corresponding to the state cost term in the data;

[0239] The sample dataset and use Control basis functions Represented as:

[0240] ;

[0241] ;

[0242] ;

[0243] in, As the control input for sampling time t in the demonstration dataset, and These respectively represent the use of control basis functions After and The weight vector, for The identified parameter vector corresponding to the control input coupling term in the text;

[0244] according to Constructing the control intermediate matrix Specifically, this is achieved through the following formula:

[0245] ;

[0246] Combining the equations of state cost terms and the control intermediate matrix The state cost update function is transformed to obtain the transformed state cost update function, which is expressed as:

[0247] ;

[0248] in, For the outer ring The parameter vector corresponding to the state cost term in the next iteration. for The basis function values ​​at time t;

[0249] Define performance index parameters based on the equation of the state cost term. , is represented as:

[0250] ;

[0251] Using gradient descent, based on right Update and get The value of is achieved through the following formula:

[0252] ;

[0253] ;

[0254] in, For adaptive step size, This is the adjustment coefficient for the adaptive step size, which can also be understood as the learning rate factor. It controls the magnitude of each gradient descent update. The larger the value, the faster the parameters update, but excessively large values ​​may lead to oscillations or non-convergence. The smaller the value, the more stable the updates, but the slower the convergence. To prevent the denominator from being zero or an excessively small positive number, a very small constant is generally chosen. This ensures... It is always greater than zero, thus avoiding numerical instability; This is a data matrix or regression matrix composed of sampled data and basis functions. It reflects the influence of the collected data on the unknown parameter l1 within the current time period. It is the error term under the current parameter estimation;

[0255] To prove the theoretical feasibility and stability of the above combined inner and outer loop algorithm, the following theorems and derivations are given in conjunction with Assumption 3 and Assumption 4.

[0256] Theorem 1: (Stability of Closed-Loop Systems)

[0257] For any state With control law , if the sequence From the aforementioned The inner loop strategy is generated by the iterative algorithm, then the closed loop system... It is asymptotically stable globally at the origin; and The function value decreases with the number of iterations, that is:

[0258] ;

[0259] in, This represents the Q-function value obtained in the i-th iteration of the inner loop. This represents the Q-function value obtained in the (i+1)th iteration of the inner loop. This represents the optimal Q-function value that converges within the inner loop. Therefore, due to the monotone bounded theorem, we have... hour .

[0260] Proof: By the definition of policy iteration, we can obtain:

[0261] ;

[0262] Right now .in, Indicates the inner ring number The value function obtained in the second iteration Indicates the inner ring number The cost function obtained in the second iteration Indicates the inner ring number The gradient of the value function obtained in the next iteration. Zhengding Since the radial direction is bounded, the closed-loop system is asymptotically stable globally at the origin. Integrating along the trajectory, we get:

[0263] ;

[0264] Explain the control law Feasible. Combining the property of monotonic convergence, we have... .

[0265] Q.E.D.

[0266] Theorem 2: (Convergence of outer loops)

[0267] If the algorithm converges, then the learner satisfies , . ( (This refers to the equivalent state cost parameter of the expert system)

[0268] Proof: After the inner loop converges, the state-value weight matrix satisfies:

[0269] .

[0270] Substitute into expert optimal control :

[0271] .

[0272] when hour, Monotonically increasing and bounded. With , .

[0273] therefore Converging to the equivalent weight matrix .

[0274] Q.E.D.

[0275] Theorem 3: (Global asymptotic stability)

[0276] When the algorithm is used to mimic the optimal control of an expert system, the learner system is globally asymptotically stable (GAS) under each inner and outer loop iteration.

[0277] prove:

[0278] 1. Inner loop: It has been proven to be fixed by Theorem 1. The closed-loop system is stable at all times;

[0279] 2. Outer loop: The rate of change of the Lyapunov function of the system is:

[0280] .

[0281] because The system remains Lyapunov stable in each iteration, and the closed-loop state converges asymptotically.

[0282] In summary, regardless of whether it's the strategy evaluation (inner loop) or parameter update (outer loop) phase, the system always satisfies:

[0283] ;

[0284] Therefore, the learner system is globally asymptotically stable (GAS) under all iterations.

[0285] Q.E.D.

[0286] Next, we will conduct implementation examples and simulation verification:

[0287] I. System Modeling and Parameter Setting

[0288] To verify the effectiveness of the method of this invention, a second-order model of a nonlinear continuous-time power system is considered as the dynamic equation of the expert system, and its state equation can be expressed as:

[0289] ;

[0290] Among them, state variables For inductor current, For capacitor voltage, control variable The voltage signal applied to the converter. System parameter settings are as follows:

[0291] Table 1 System Parameter Settings

[0292]

[0293] Define immediate cost ,in , .

[0294] The state and control trajectory of the expert system under the optimal control law were collected. In the experiment, a linear optimal control law was used.

[0295] ;

[0296] The system samples within the time domain [0,10] to obtain expert sample data. This data will be used as target behavior data in the subsequent inverse reinforcement learning phase.

[0297] II. Simulation of Inner Layer Strategy Iterative Control

[0298] Scholar systems share the same dynamic structure as expert systems, and scholar systems can obtain state-control pairs through interaction with the environment without requiring an explicit system model. Under this setting, the system's control law is obtained by solving a semidefinite programming problem based on SOS relaxation. Each iteration updates the policy using a Gaussian kernel function, and the convergence condition is met.

[0299] The event will end at that time.

[0300] To verify the effectiveness of this algorithm, the initial state of the system is sampled from a random distribution. A polynomial approximation of the Q-function is performed using the basis function space, with the maximum order being... .

[0301] Simulation results show that after approximately four policy updates, the control law converges to a stable form. At this point, the system can effectively adjust its state to the equilibrium point while maintaining low energy consumption, such as... Figure 3 As shown.

[0302] III. Convergence Analysis of Control Input and Cost Function

[0303] To further analyze the convergence performance of the control strategy, the value function was plotted. The three-dimensional evolution surface, such as Figure 4 As shown, it can be observed that as the number of iterations increases, the surface of the value function gradually shrinks, exhibiting a gradual optimization process of control performance.

[0304] IV. Simulation of Outer Layer Inverse Optimal Control

[0305] In the outer optimization stage, an inverse optimal controller is introduced to identify unknown state cost weight parameters in the expert control strategy. .

[0306] Expert systems employ linear feedback control laws. The algorithm is then projected onto a basis function space containing second-order terms. The goal of the outer optimization is to minimize the behavioral bias between the learner's policy and the expert's policy, thereby estimating reasonable state cost parameters.

[0307] Parameter learning is achieved using gradient descent, where the parameter vector is adaptively updated along the negative gradient direction based on the behavioral residual. When the Euclidean norm of the parameter changes in two consecutive iterations is lower than... The algorithm terminates when the time is right.

[0308] like Figure 5 As shown, the 2-norm of the state cost parameter gradually decreases during the iteration process and eventually converges to a stable value, indicating that the outer optimization stage can effectively identify the optimal cost function structure and achieve the inverse optimal control objective.

[0309] In summary, the key points of this invention are:

[0310] 1. Deep coupling of Q-learning and SOS relaxation enables model-free online optimal control.

[0311] This invention organically combines the model-free adaptive dynamic programming concept of Q-learning with the Lyapunov stability constraints provided by SOS relaxation within a continuous time frame. Without explicitly solving the HJB equations or relying on an explicit system dynamic model, it achieves optimal control through semidefinite programming with SOS constraints, enabling policy evaluation and improvement while maintaining model freedom, stability guarantees, and online updates.

[0312] 2. A unified algorithm framework of "inner policy iteration + outer state cost identification" is proposed for the two-layer joint iteration of IRL: the inner layer solves the policy using SOS and Q-learning under a fixed cost function; the outer layer uses expert demonstration data to iteratively update the state cost parameters, realizing "cost recovery + policy imitation" in the sense of inverse reinforcement learning.

[0313] 3. The unified modeling approach based on polynomial bases uses polynomial bases to represent Q-functions, value functions, and state cost functions, enabling SOS constraints, semidefinite programming, and gradient updates to be performed in the same polynomial space, which facilitates overall implementation and theoretical analysis.

[0314] 4. Theoretical guarantees of closed-loop stability and convergence: By constructing the Lyapunov operator and its SOS constraint, theorems on closed-loop stability and parameter convergence are given in the inner and outer layers respectively, and it is further proved that the entire joint iteration process maintains global asymptotic stability at each step.

[0315] 5. Model Freedom and Online Learning Capability: Learners do not need explicit system model functions. They can update policies and cost parameters online by relying only on state-control-reward data sampled from interactions with the environment and a small number of structural assumptions. This is particularly suitable for complex and difficult-to-model engineering systems.

[0316] Compared with existing inverse reinforcement learning methods based on solving the HJB equation and traditional SOS optimal control methods, this invention has the following beneficial effects and advantages:

[0317] 1. Significantly reduce computational complexity and resource consumption. Existing inverse reinforcement learning often adopts a two-layer structure of updating the cost function in the outer layer and repeatedly solving the HJB equation in the inner layer. The inner layer usually requires large-scale nonlinear equation solving or deep neural network training, which is computationally intensive, consumes a lot of memory, and is difficult to meet real-time requirements.

[0318] This invention transforms the policy evaluation problem into a semidefinite programming problem with SOS constraints, under the premise of fixed cost function parameters. Policy updates are obtained by solving a convex optimization problem, eliminating the need for direct solution of the HJB equations. Simulations show that, under conditions of similar control performance, the single-cycle computation time and total computation time of the inner iteration of this invention are significantly lower than traditional methods, making it advantageous for implementation on resource-constrained devices such as industrial controllers.

[0319] 2. Model-Free Online Inverse Reinforcement Learning Without Explicit System Model: Traditional SOS-based optimal control methods mostly rely on accurate system dynamics models, requiring pre-defined analytical forms and parameters of the controlled object. This makes them difficult to apply to complex objects with incomplete models or those involving time-varying or uncertain factors. While existing model-free SOS frameworks reduce reliance on accurate models to some extent, they typically only allow offline solutions and cannot utilize real-time data to learn and update control strategies and cost functions online during system operation. This invention, based on Q-learning, unifies the representation of the Q-function and state cost function into polynomial form. Furthermore, it introduces SOS relaxation and semidefinite programming, enabling parameter updates solely based on state, control input, and real-time cost data collected during operation. This achieves model-free online inverse reinforcement learning control of continuous-time nonlinear systems without requiring an explicit system dynamics model.

[0320] 3. Capable of mining and adjusting expert hidden cost functions: This invention represents the state cost function as a polynomial quadratic form, identifies cost parameters using expert demonstration data, and makes the obtained cost function approximate the expert hidden cost function within a given family of functions. The corresponding control strategy is basically consistent with the expert strategy in terms of state response and control energy, achieving high-quality strategy imitation.

[0321] Building upon this foundation, while retaining the structural form of the expert cost function, the weight coefficients of each state variable and control variable can be quantitatively adjusted according to actual engineering needs. For example, penalties can be strengthened for indicators such as energy consumption, overshoot, safety margin, actuator saturation, or ride comfort, achieving a trade-off between multiple objectives such as "performance, energy consumption, and safety constraints." By simply modifying the weights, optimized cost functions and corresponding control strategies adapted to different operating conditions and control objectives can be quickly obtained without redesigning the overall algorithm, facilitating engineering deployment and subsequent maintenance.

[0322] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A model-free inverse reinforcement learning method that integrates Q-learning and sum-of-squares relaxation, characterized in that, include: Step 1: Using the piezoelectric cantilever beam system as an expert system, construct the dynamic state equation of the expert system and obtain the demonstration dataset of the expert system. The demonstration dataset includes multiple sampling times, as well as the state variables and control inputs at each sampling time. Step 2: Based on the dynamic state equation of the expert system, construct the dynamic state equation of the learning system, and then define the instantaneous cost of the learning system; Step 3: Construct based on the real-time cost of the learning system Functions and based on polynomial basis function vector pairs The function is approximated by parameterization to obtain a parameterized approximation. The function partitions the polynomial basis function vector to obtain the partitioned basis function vector; Step 4: According to Function definition: Lyapunov operator; Step 5: Based on the demonstration dataset, parameterized approximation The function, the partitioned basis function vector, and the Lyapunov operator are subjected to multiple iterations to find the optimal control input.

2. The model-free inverse reinforcement learning method integrating Q-learning and sum-of-squares relaxation as described in claim 1, characterized in that, Step 1 involves constructing the dynamic state equations of the expert system, including: The dynamic state equations of the expert system are constructed as follows: ； Among them, state variables ,in, For displacement, For speed, This is the piezoelectric output voltage. Indicates matrix transpose. for The time derivative, for The time derivative, for The time derivative; Indicates the linear stiffness coefficient; Indicates the equivalent mass of the vibration system; Indicates the cubic nonlinear stiffness coefficient; Indicates the linear damping coefficient; Indicates the nonlinear damping coefficient; Indicates the electromechanical coupling coefficient; For control input; This represents the equivalent capacitance of a piezoelectric element; Indicates the external load resistance or equivalent resistance; The dynamic state equations of the expert system are transformed to obtain the dynamic state equations, which are expressed as follows: ； in, for The time derivative, Represents the drift dynamics term of the expert system; This represents the control input gain function.

3. The model-free inverse reinforcement learning method integrating Q-learning and sum-of-squares relaxation as described in claim 2, characterized in that, Step 2 specifically includes: Based on the dynamic state equation of the expert system, the dynamic state equation of the learning system is constructed as follows: ； in, To learn the state vector of the system, For the control input of the learning system, for The time derivative; In the dynamic state equations of expert systems Consistent; Define the instant cost of the learning system , represented as: ； in, It is the state cost function of the learning system. It is a positive definite weight matrix.

4. The model-free inverse reinforcement learning method integrating Q-learning and sum-of-squares relaxation as described in claim 3, characterized in that, Step 3 specifically includes: Introducing value functions and its gradient And define the Hamiltonian function. , represented as: ； in, This means that the Hamiltonian function is defined as the formula on the right side of the equals sign; Learning systems based on Hamiltonian functions The function is represented as: ； To facilitate the subsequent establishment of a learnable linear parameterization form, The function is divided into state cost term, control input cost term, drift mechanics term, control input coupling term, and value function term, specifically represented as follows: ； ； ； ； ； ； in, Represents the state cost term; Indicates the control input cost term; Represents the drift motion mechanical terms; Indicates the control input coupling term; Represents the value function term; Within the working domain comprised of all combinations of state and control inputs, polynomial basis function vectors are selected, and based on these vectors... Perform approximate parameterization to obtain a parameterized approximation. The function is represented as: ； in, The parameter vector to be identified, These are the parameter vectors to be identified corresponding to the state cost term, the control input cost term, the drift mechanics term, and the control input coupling term, respectively. This is the vector of parameters to be identified corresponding to the value function term; For basis function vectors The partitioning process yields the basis function vectors, represented as follows: ； in, For the state cost term The corresponding polynomial basis function vector; To control input cost terms The corresponding polynomial basis function vector; To match the system's drift mechanics terms The corresponding polynomial basis function vector; For coupling with control input terms The corresponding polynomial basis function vector; For the AND-valued function term The corresponding polynomial basis function vector.

5. The model-free inverse reinforcement learning method integrating Q-learning and sum-of-squares relaxation as described in claim 4, characterized in that, The Lyapunov operator described in step 4 Represented as: ； in, Let be the coefficient vector of the polynomial on the basis vectors. ; It represents a polynomial or monomial basis vector consisting of state variables and control inputs, with a total degree not exceeding 2d.

6. The model-free inverse reinforcement learning method integrating Q-learning and sum-of-squares relaxation as described in claim 5, characterized in that, Step 5 specifically includes: Step 5.1: Set the outer loop iteration count k and the inner loop iteration count i, let k=0, i=1; set the initial control input. Set the inner loop control convergence threshold Convergence threshold of outer loop parameters Initialize the parameter vector to be identified. ; Step 5.2: During the inner loop iteration process, based on the Lyapunov operator and parameterized approximation... The function and the partitioned basis function vectors, and the inner loop control convergence threshold. By performing multiple inner loop iterations, the optimal control input for the inner loop is obtained. and the identified parameter vector ; Step 5.3: According to and calculate Specifically, this is achieved through the following formula: ； in, for The identified parameter vector corresponding to the value function term in the value function term. express The basis functions of the time-value function; Step 5.4: Based on the partitioned basis function vectors, the example dataset, and the parameterized approximation function, , Perform multiple outer loop iterations to obtain the outer loop's th iteration. The parameter vector corresponding to the state cost term in the next iteration Get the value In ,exist At that time, As the optimal control input equivalent to expert control; in At that time, let k+1, and... In Replace with Return to step 5.

2.

7. The model-free inverse reinforcement learning method integrating Q-learning and sum-of-squares relaxation as described in claim 6, characterized in that, Step 5.2 specifically includes: Step 5.2.1: Let the expressions for the Lyapunov operator in the i-th iteration of the inner loop and the k-th iteration of the outer loop be: ,in, Let be the coefficient vector of the polynomials in the basis vectors for the i-th iteration of the inner loop and the k-th iteration of the outer loop. Define the control inputs for the i-th iteration of the inner loop and the k-th iteration of the outer loop. For the sum of squares constraint, define at the same time For the sum of squares constraint, where, Let be the vector of parameters to be identified in the (i-1)th iteration of the inner loop and the kth iteration of the outer loop. The vectors to be identified are the inner loop's i-th iteration and the outer loop's k-th iteration; using... For the sum of squares constraint and Solving the objective function yields the following results: The value; The optimization objective function is expressed as: ； ； Among them, c Q For intermediate parameters, Indicates the working domain. It is a weighted function; Step 5.2.2: Based on the partitioned basis function vectors and parameterized approximations The function yields the equation for the control input coupling term, expressed as: ； in, for The identified parameter vector corresponding to the control input coupling term in the input vector. Let i be the gradient of the value function when the inner loop iteration number is i and the outer loop iteration number is k. The strategy update formula is obtained and expressed as: ； in, This serves as the control input for the (i+1)th iteration of the inner loop and the kth iteration of the outer loop; By performing an equivalent transformation on the equations for the control input coupling terms, we obtain the equations for state-control separation, which are expressed as follows: ； in, For control basis functions, Indicates the Kronecker product; Based on the state-control separation equation, the policy update formula is transformed, and the transformed policy update formula is used to calculate... The value of ; where the transformed policy update formula is expressed as: ； Step 5.2.3: In At that time, As the optimal control input of the inner loop ,Will As the identified parameter vector ;exist When i is reached, increment i by 1 and return to step 5.2.

1.

8. The model-free inverse reinforcement learning method integrating Q-learning and sum-of-squares relaxation according to claim 7, characterized in that, In step 5.4, based on the partitioned basis function vectors, the example dataset, and the parameterized approximation... function, , Perform multiple outer loop iterations to obtain the outer loop's th iteration. The parameter vector corresponding to the state cost term in the next iteration The values ​​include: Using the demonstration dataset to divide the time period The collected data is used to construct a state cost update function, which is represented as follows: ； in, For the outer ring The state cost function for the next iteration Central sampling time for demonstration dataset The control input, Let be the value function of the outer loop at time t+T for the kth iteration; Based on the partitioned basis function vectors and parameterized approximations The function yields the equation for the state cost term, expressed as: ； in, for The identified parameter vector corresponding to the state cost term in the data; The sample dataset and use Control basis functions Represented as: ； ； ； in, As the control input for sampling time t in the demonstration dataset, and These respectively represent the use of control basis functions After and The weight vector, for The identified parameter vector corresponding to the control input coupling term in the text; according to Constructing the control intermediate matrix Specifically, this is achieved through the following formula: ； Combining the equations of state cost terms and the control intermediate matrix The state cost update function is transformed to obtain the transformed state cost update function, which is expressed as: ； in, For the outer ring The parameter vector corresponding to the state cost term in the next iteration. for The basis function values ​​at time t; Define performance index parameters based on the equation of the state cost term. , represented as: ； Using gradient descent, based on right Update and get The value of is achieved through the following formula: ； ； in, For adaptive step size, The adjustment coefficient for the adaptive step size. To prevent the denominator from being zero or an excessively small positive number, This is a data matrix or a regression matrix.

Images