Adaptive control method based on reinforcement learning and radial basis function neural network

Abstract

The invention relates to the technical field of information technology, in particular to a self-adaptive control method based on reinforcement learning and a radial basis function neural network, which comprises the following steps: establishing a state equation of a nonlinear system with unknown time-varying uncertainty, and defining a system state variable and a control target; under a reinforcement learning framework, a control strategy and a value function are initialized, initial network parameters are set, and the upper definite bound of uncertainty is preliminarily estimated; estimating the uncertainty of the system in real time by using the radial basis function neural network, and approaching the upper definite bound of the uncertainty by adjusting the parameters of the radial basis function neural network; based on the uncertainty of estimation, performing iterative updating on the control strategy, and in each iteration, optimizing the control strategy by using a reinforcement learning algorithm; and the control strategy is adjusted in real time, control strategy parameters are optimized based on the latest uncertainty estimation value, and a final stable control strategy is obtained through convergence analysis. According to the method, the problem that the upper bound of uncertainty needs to be known in advance is effectively solved, the robustness and the control precision of the system are improved, the dynamic adaptive capacity is high, the control strategy design is simplified, and the stability and the response efficiency of the system are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of information technology technology, and in particular to an adaptive control method and system based on reinforcement learning and radial basis function neural network. Background Art

[0002] In modern industry, aerospace, robotics and other fields, the stability and robustness of control systems are crucial. Robust control technology can maintain the stability of system performance in the face of system parameter perturbations and external interference. It is an important method to ensure the reliability of control systems. However, many practical systems have unknown time-varying uncertainties, which makes robust control design extremely complex.

[0003] Traditional control methods usually rely on accurate knowledge of the system model when dealing with these uncertainties, and obtain the optimal control law by solving the Hamilton-Jacobi-Bellman equations. However, due to the nonlinearity of the system, analytical solutions are often difficult to obtain. In addition, reinforcement learning, as a machine learning method that does not require a preset model, obtains feedback through interaction with the environment to optimize the control strategy, and has shown the potential to solve complex control problems. However, existing value iteration and policy iteration methods face the challenge of describing the upper bound of uncertainty when dealing with uncertainty. Using too wide an upper bound will reduce system performance, and it is difficult to accurately describe the upper bound. Summary of the invention

[0004] The present invention provides an adaptive control method and system based on reinforcement learning and radial basis function neural network, thereby effectively solving the problems pointed out in the background technology.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] Adaptive control methods based on reinforcement learning and radial basis function neural networks, including:

[0007] Establish state equations for nonlinear systems with unknown time-varying uncertainties, and define system state variables and control objectives;

[0008] In the reinforcement learning framework, the control strategy and value function are initialized, the initial network parameters are set, and the supremum of uncertainty is preliminarily estimated;

[0009] Using a radial basis function neural network to estimate the uncertainty of the system in real time, and by adjusting the parameters of the radial basis function neural network, approaching the supremum of the uncertainty;

[0010] Based on the estimated uncertainty, the control strategy is iteratively updated, and in each iteration, the control strategy is optimized using a reinforcement learning algorithm;

[0011] The control strategy is adjusted in real time, the control strategy parameters are optimized based on the latest uncertainty estimation value, and the final stable control strategy is obtained through convergence analysis.

[0012] Furthermore, the state equation is:

[0013] x(t)=f(x(t))+g(x(t))u(t)+Δf(x(t))

[0014] Where x(t) is the system state, f(x(t)) is the natural state of the system itself, g(x(t)) is the control input gain function of the system, u(t) is the control input, and Δf(x(t)) is the unknown time-varying uncertainty.

[0015] Furthermore, the iterative steps of the initial strategy include:

[0016] Initializing the control strategy, value function and initial neural network parameters;

[0017] Select the initial state and set the learning rate, and use the current policy to sample a series of state and action pairs;

[0018] The value function and control strategy are updated according to the reinforcement learning framework until the control strategy and value function converge.

[0019] Furthermore, the uncertainty estimation step includes:

[0020] The radial basis function neural network is used to estimate the uncertainty of the system in real time to obtain an estimated value of the uncertainty.

[0021] The parameters of the radial basis function neural network are adjusted to approximate the supremum of uncertainty, thereby obtaining an approximation value of the supremum of uncertainty.

[0022] Furthermore, the control strategy is optimized using reinforcement learning algorithms, including:

[0023] defining an objective function of the current strategy, and calculating a value of the objective function based on the system state and the uncertainty of the estimation;

[0024] The reinforcement learning algorithm is used to calculate the gradient of the policy, and the control policy parameters are adjusted according to the gradient information to minimize the objective function.

[0025] Furthermore, the reinforcement learning algorithm is a policy iteration algorithm, and the expression of policy improvement is:

[0026]

[0027] Among them, π i is the current strategy, π i+1(x) is the improved policy function, r(x,u) is the immediate reward, Y is the discount factor, P(x′|x,u) is the state transition probability, Following the current strategy π in state x′ i The expected total return that can be obtained.

[0028] Furthermore, the activation function of the radial basis function neural network is a Gaussian radial basis function, which is expressed as:

[0029] φ(r)=exp(-β||xc i || 2 )

[0030] Among them, φ is the activation function, β is the adjustment parameter, x is the input vector, c i is the center vector.

[0031] Furthermore, the reinforcement learning algorithm is an ACE reinforcement learning algorithm, and the approximate formula of the radial basis function neural network of the ACE reinforcement learning algorithm is:

[0032]

[0033] in, represents the approximation of the value function at the i-th iteration time t, is the weight vector at the i-th iteration, σ(V ci x(t)) is the activation function σ acting on the feature vector V ci The result of x(t), V ci x(t) is the input state x(t) through the feature map V ci The result after conversion, represents the control input approximation at the i-th iteration time t, is the weight vector at the i-th iteration, σ(V ai x(t)) is the activation function σ acting on the feature vector V ai The result of x(t), V ai x(t) is the input state x(t) through the feature map V ai The result after conversion, L i represents the cumulative value from t+b at the i-th iteration, is the weight vector of the i-th iteration time τ, σ(V ei x) is the activation function σ acting on the feature vector V ei The result of x, V ei x is the input state x through the feature map V ei The result of the conversion, τ, is the integral variable from t to t+b.

[0034] Furthermore, the real-time adjustment control strategy includes:

[0035] In each control cycle, the current state of the system and control input data are collected;

[0036] Calculate new control input and value functions using current weight parameters;

[0037] The current weight parameters are used to generate new state and control input data to adjust the control strategy.

[0038] Furthermore, the method further comprises:

[0039] During the control strategy iteration process, dynamically evaluate the robustness index of the current strategy under different uncertainty conditions, wherein the robustness index includes the stability margin and the fluctuation of the control input;

[0040] Through the system simulation process, different types and levels of uncertainty are actively injected to simulate the changes in the actual operating environment and obtain simulation results;

[0041] According to the robustness evaluation results and simulation results, the control strategy parameters are adjusted in real time, and the control strategy is optimized using the feedback loop.

[0042] The technical solution of the present invention can achieve the following technical effects:

[0043] The present invention effectively solves the problem of needing to know the upper bound of uncertainty in advance, improves system robustness and control accuracy, has strong dynamic adaptive capability, simplifies control strategy design, and enhances system stability and response efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 It is a flow chart of an adaptive control method based on reinforcement learning and radial basis function neural network;

[0046] Figure 2 Schematic diagram of the process of iterative steps of the initial strategy;

[0047] Figure 3 It is a flowchart of the uncertainty estimation step;

[0048] Figure 4 Schematic diagram of the process of optimizing the control strategy using reinforcement learning algorithm;

[0049] Figure 5 A flow chart for adjusting the control strategy in real time;

[0050] Figure 6 A schematic diagram of the subsequent process of the adaptive control method based on reinforcement learning and radial basis function neural network; DETAILED DESCRIPTION

[0051] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0053] Embodiment 1

[0054] like Figure 1 As shown, the present invention provides an adaptive control method based on reinforcement learning and radial basis function neural network, the method comprising:

[0055] S1: Establish the state equation of a nonlinear system with unknown time-varying uncertainties, and define the system state variables and control objectives;

[0056] Specifically, a nonlinear system with unknown time-varying uncertainty means that the dynamic characteristics of the system contain nonlinear behavior and there are uncertainties that vary with time and cannot be predetermined. By establishing the state equation of the system, clarifying the dynamic characteristics and potential uncertainties of the system, accurately describing the overall behavior of the system, and defining the control objectives, this step provides the necessary mathematical model foundation for the subsequent control strategy design and optimization, so that the control process can be strictly analyzed and implemented in theory, thereby ensuring that the control system can operate effectively in a complex and changing environment.

[0057] S2: In the reinforcement learning framework, initialize the control strategy and value function, set the initial network parameters, and preliminarily estimate the supremum of uncertainty;

[0058] Specifically, by initializing the control strategy and value function under the reinforcement learning framework, setting the initial network parameters, and preliminarily estimating the supremum of uncertainty, an initial optimization foundation is established for the entire control system, providing a starting point for subsequent iterative updates and real-time adjustments, so that the control system can consider uncertainty factors from the beginning and gradually approach the optimal control strategy, thereby improving the adaptability and stability of the system.

[0059] S3: Use radial basis function neural network to estimate the uncertainty of the system in real time, and adjust the parameters of radial basis function neural network to approach the supremum of uncertainty;

[0060] Specifically, the radial basis function neural network (RBFNN) is an artificial neural network that is particularly suitable for function approximation, classification and control problems. The characteristic of RBFNN is that it uses radial basis functions as activation functions. It usually consists of three layers: input layer, hidden layer and output layer. By using the radial basis function neural network (RBFNN) to estimate the uncertainty in the system in real time and dynamically adjust the parameters of the neural network to approach the supremum of uncertainty, the uncertainty characteristics of the system can be accurately captured, ensuring that the uncertainty that changes over time can be accurately and timely estimated and responded to during the operation of the system, and the accuracy and robustness of the control strategy can be improved. By gradually approaching the supremum of uncertainty, the system can effectively control the system without knowing the upper bound of uncertainty in advance, so that it can maintain stable and efficient operation.

[0061] S4: Iteratively update the control strategy based on the estimated uncertainty, and in each iteration, use the reinforcement learning algorithm to optimize the control strategy;

[0062] Specifically, the control strategy is iteratively updated based on the estimated uncertainty, and the reinforcement learning algorithm is used to optimize the control strategy in each iteration to achieve adaptive control and enhanced robustness of the system. In a dynamic environment, the control strategy is gradually adjusted to continuously adapt to the current system state and uncertainty conditions, ensuring that the control strategy can be optimized in real time, thereby improving the performance and stability of the system. Using the reinforcement learning algorithm, the system can autonomously learn and improve the control strategy, find the optimal control input, and enable the system to achieve the best control effect, ensuring that it can still operate stably and efficiently under various uncertainty conditions.

[0063] S5: Adjust the control strategy in real time, optimize the control strategy parameters based on the latest uncertainty estimates, and obtain the final stable control strategy through convergence analysis.

[0064] Specifically, during the operation of the system, the control strategy is dynamically adjusted according to the latest uncertainty estimates so that it can adapt to changes in the actual environment and maintain the robustness and stability of the system. The convergence analysis ensures the reliability of the optimization process and ensures that the control strategy can reach a stable state after repeated iterations and adjustments, avoiding instability caused by frequent changes.

[0065] As a preferred embodiment of the above, the state equation is:

[0066] x(t)=f(x(t))+g(x(t))u(t)+Δf(x(t))

[0067] Where x(t) is the system state, f(x(t)) is the natural state of the system itself, g(x(t)) is the control input gain function of the system, u(t) is the control input, and Δf(x(t)) is the unknown time-varying uncertainty.

[0068] Specifically, the equation integrates the intrinsic behavior of the system (described by f(x(t))), the influence of external control input (described by g(x(t))u(t)), and unpredictable disturbances (described by Δf(x(t))), and fully reflects the dynamic characteristics of the system. The state equation of the system is the basis for designing the control strategy. By understanding the role of each part, an effective control input u(t) can be designed to achieve the desired control target. The term Δf(x(t)) clearly points out the unknown time-varying uncertainty in the system. By considering this part of uncertainty in the control strategy, the robustness and adaptability of the system in actual operation can be improved, and the stability of the system under different conditions can be ensured. The reinforcement learning algorithm and radial basis function neural network (RBFNN) use this state equation to estimate and adjust uncertainty in real time. By iteratively optimizing the control strategy, the system can adapt to the changing environment and maintain efficient and stable operation.

[0069] As a preferred embodiment of the above, Figure 2 As shown, the iterative steps of the initial strategy include:

[0070] A10: Initialize control strategy, value function and initial neural network parameters;

[0071] A20: Select the initial state and set the learning rate, and use the current strategy to sample a series of state and action pairs;

[0072] A30: Update the value function and control strategy according to the reinforcement learning framework until the control strategy and value function converge.

[0073] Specifically, first, set the initial control strategy, which is the control scheme of the system in the initial state. Usually, a simple linear strategy or random strategy can be selected to start the exploration process. Then, initialize the value function, which is used to evaluate the value of different states. This can be initially set to zero or a small random value so that it can be gradually updated during the iteration process. In addition, initialize the parameters of the radial basis function neural network (RBFNN), including the center, width and connection weight of the hidden layer nodes, where the center position is usually determined by a clustering algorithm (such as K-means), and the width parameter and weight can be set to random values; select the initial state of the system, which will serve as the starting point for the control strategy iteration to ensure that the system is optimized from the known initial conditions. Then, set the learning rate to control the step size of the parameter update in each iteration. The choice of the learning rate affects the speed and stability of the convergence of the strategy and value function; under the current control strategy, sample a series of states and corresponding action pairs by simulating or actually running the system, and record the state transition and reward. In order to balance exploration and utilization, the ε-greedy strategy can be adopted, that is, the current optimal action is selected most of the time, but random actions are selected in a certain proportion of the time to explore new strategies. An experience replay pool is formed through sampling to collect enough state and action pair data for subsequent strategy and value function updates; the value function and control strategy are iteratively updated using the reinforcement learning algorithm. First, the value function is updated using the time difference method to calculate the value of the current state and adjust it according to the reward of the new observation and the estimated value of the next state. The output of the RBFNN is adjusted to approach the new value function by adjusting the parameters of the RBFNN; then, the control strategy is updated using the policy gradient method to adjust the policy parameters so that the probability distribution of selecting actions in a given state is better, and the changes in the value function and control strategy are monitored. When their changes are less than the preset threshold, the strategy and value function are considered to have converged and the iterative process stops.

[0074] As a preferred embodiment of the above, Figure 3 As shown, the uncertainty estimation steps include:

[0075] B10: Use radial basis function neural network to estimate the uncertainty of the system in real time and obtain the estimated value of the uncertainty

[0076] B20: By adjusting the parameters of the radial basis function neural network, the supremum of uncertainty is approximated and the approximate value of the supremum of uncertainty is obtained.

[0077] Specifically, the radial basis function neural network calculates and outputs an estimate of the system uncertainty by inputting the current system state and control input. In this process, the hidden layer nodes of the RBFNN use radial basis functions to perform nonlinear transformations on the input data to capture the complex relationship between the system state and uncertainty. This real-time estimation can dynamically reflect the changes in the uncertainty of the system and provide more accurate estimates. Next, by adjusting the parameters of the radial basis function neural network, such as the center position, width and output layer weights of the hidden layer nodes, the output of the RBFNN gradually approaches the supremum of uncertainty. The purpose of adjusting these parameters is to optimize the fitting ability of the neural network so that it can more accurately estimate and approximate the supremum of uncertainty of the system. In this process, optimization algorithms such as gradient descent can be used to continuously adjust the network parameters according to error feedback, gradually reduce the gap between the estimated value and the actual supremum of uncertainty, and obtain a more accurate approximation of the supremum of uncertainty.

[0078] As a preferred embodiment of the above, Figure 4 As shown in Figure 1, the control strategy is optimized using reinforcement learning algorithm, including:

[0079] C10: Define the objective function of the current strategy and calculate the objective function value based on the system state and estimated uncertainty;

[0080] C20: Use reinforcement learning algorithm to calculate the gradient of the policy, and adjust the control policy parameters according to the gradient information to minimize the objective function.

[0081] Specifically, first, the optimization goal of the current control strategy is clarified. The objective function should comprehensively consider the stability of the system, the energy consumption of the control input, and the robustness to uncertainty. The objective function can be formalized as the weighted sum of the system state error and the control input to measure the overall performance of the control strategy. The purpose of this step is to clarify the optimization direction and provide a basis for subsequent calculations. After defining the objective function, the specific value of the objective function is calculated according to the current system state and the uncertainty estimated by the radial basis function neural network. The reinforcement learning algorithm is used to calculate the gradient of the objective function with respect to the control strategy parameters. The policy gradient indicates how to adjust the policy parameters to optimize the direction and size of the objective function value under the current policy parameter settings. Using the policy gradient information calculated in the previous step, the gradient descent method or other optimization methods are applied to update the control strategy parameters. By gradually iteratively adjusting the policy parameters, the objective function value is gradually reduced to achieve the optimization of the control strategy. After each iteration, the system continuously adjusts the strategy according to the feedback until the optimization goal is achieved.

[0082] As a preference of the above embodiment, the reinforcement learning algorithm is a policy iteration algorithm, and the expression of the policy improvement is:

[0083]

[0084] Among them, π i is the current strategy, π i+1 (x) is the improved policy function, r(xu) is the immediate reward, Y is the discount factor, P(x′|x,u) is the state transition probability, Following the current strategy π in state x′ i The expected total return that can be obtained.

[0085] Specifically, the reason for using the policy iteration algorithm is that it combines policy evaluation and policy improvement, and ensures that each iteration can make improvements on the current basis by alternately optimizing the control strategy step by step. The algorithm can effectively utilize the information of the current strategy, gradually approach the optimal strategy, has a clear optimization direction and convergence, is suitable for complex nonlinear systems and uncertainty problems, and has good flexibility and scalability. It should be noted that other common algorithms can also be used as the reinforcement learning algorithm of this embodiment, but the policy iteration algorithm should be the preferred solution.

[0086] As a preference for the above embodiment, the activation function of the radial basis function neural network is a Gaussian radial basis function, expressed as:

[0087] φ(r)=exp(-β||xc i || 2 )

[0088] Among them, φ is the activation function, β is the adjustment parameter, x is the input vector, c i is the center vector.

[0089] Specifically, the Gaussian radial basis function is a commonly used activation function used to convert the input vector into the activation value of the hidden layer node. It has the characteristics of smoothness, continuity and differentiability, and is very suitable for capturing nonlinear relationships. The adjustment parameter β determines the expansion range of the activation function. When the β value is large, the response range of the activation function is narrow, and it only responds significantly to the input vector x that is close to the center vector ci; when the β value is small, the response range of the activation function is wide, and it can respond to the input vector x that is farther away. The input vector x is the state or feature data of the current system. It is input into the radial basis function neural network, and the nonlinear transformation is performed through the activation function of the hidden layer node to extract the important features of the input data. The center vector ci is the center point of the Gaussian radial basis function. Each hidden layer node has a center vector, which represents the position of the node in the input space. The center vector is usually determined by a clustering algorithm so that the hidden layer nodes can cover different areas of the input space. By using Gaussian radial basis function as the activation function, the radial basis function neural network can effectively capture the nonlinear characteristics in the input data and improve the estimation accuracy of system uncertainty. This method enables the neural network to have stronger adaptability and robustness when dealing with complex nonlinear systems, which helps to achieve precise adaptive control.

[0090] As a preferred embodiment of the above, the reinforcement learning algorithm is the ACE reinforcement learning algorithm, and the approximate formula of the radial basis function neural network of the ACE reinforcement learning algorithm is:

[0091]

[0092] in, represents the approximation of the value function at the i-th iteration time t, is the weight vector at the i-th iteration, σ(V ci x(t)) is the activation function σ acting on the feature vector V ci The result of x(t), V ci x(t) is the input state x(t) through the feature map V ci The result after conversion, represents the control input approximation at the i-th iteration time t, is the weight vector at the i-th iteration, σ(V ai x(t)) is the activation function σ acting on the feature vector V ai The result of x(t), V ai x(t) is the input state x(t) through the feature map V ai The result after conversion, L i represents the cumulative value from t+b at the i-th iteration, is the weight vector of the i-th iteration time τ, σ(V eix) is the activation function σ acting on the feature vector V ei The result of x, V ei x is the input state x through the feature map V ei The result of the conversion, τ, is the integral variable from t to t+b.

[0093] Specifically, in the ACE reinforcement learning algorithm, the value function, control input and cumulative value are approximated by the radial basis function neural network. At the i-th iteration time t, the value function is approximated by multiplying the activation value obtained by feature mapping the input state by the weight vector; the control input is approximated by a similar method, and the cumulative value is obtained by multiplying and integrating the activation value of the input state after feature mapping and the corresponding weight vector in the integration process from t to t+b. These processes achieve effective approximation of the uncertainty and dynamic behavior of complex nonlinear systems by gradually adjusting the weight vector and feature mapping.

[0094] As a preferred embodiment of the above, Figure 5 As shown, the control strategy is adjusted in real time, including:

[0095] D10: In each control cycle, collect the current state of the system and control input data;

[0096] D20: Calculate new control input and value function using current weight parameters;

[0097] D30: Use the current weight parameters to generate new state and control input data and adjust the control strategy.

[0098] Specifically, in each control cycle, the system will monitor and collect the current state and control input data in real time. These data include the real-time state variables of the system and the corresponding control signals, providing dynamic information of the current system operation. Based on the collected current state and control input data, the radial basis function neural network is used to calculate new control inputs and value functions using the existing weight parameters. This calculation process includes inputting the current state into the feature mapping function, obtaining the activation value through the activation function, and then multiplying it with the weight parameter to obtain the new control input and value function. Using the current control strategy, simulate or actually run the system to generate new data. By comparing the newly generated data with the target value, adjust the control strategy parameters and feature mapping to optimize the control strategy so that it can better cope with the uncertainty and dynamic changes of the system in the next control cycle.

[0099] As a preferred embodiment of the above, Figure 6 As shown, the method also includes:

[0100] E10: During the control strategy iteration process, dynamically evaluate the robustness indicators of the current strategy under different uncertainty conditions. The robustness indicators include stability margin and fluctuation of control input.

[0101] E20: Actively inject different types and levels of uncertainty through the system simulation process to simulate the changes in the actual operating environment and obtain simulation results;

[0102] E30: Adjust control strategy parameters in real time based on robustness evaluation results and simulation results, and optimize control strategy using feedback loops.

[0103] Specifically, during the iteration of the control strategy, the robustness indicators of the current strategy under different uncertainty conditions are dynamically evaluated. The robustness indicators include the stability margin of the system and the fluctuation of the control input. The stability margin measures the ability of the system to remain stable when disturbed, while the fluctuation of the control input reflects the smoothness of the response of the control strategy to system changes. The system simulation process is used to actively inject different types and levels of uncertainty into the system to simulate changes in the actual operating environment. These uncertainties may include parameter changes, external disturbances, sensor noise, etc. In this way, simulation results are generated to evaluate the performance of the control strategy under different uncertainty conditions. According to the robustness evaluation results obtained from the dynamic evaluation and simulation process, the control strategy parameters are adjusted in real time. By analyzing these results, the weak links of the control strategy are identified and targeted optimization is carried out. The feedback loop is used to continuously optimize the control strategy to ensure that it can effectively cope with uncertainties and changes in the actual operating environment.

[0104] Although the present application has been described in conjunction with specific features and embodiments thereof, it is obvious that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the drawings are merely exemplary illustrations of the present application as defined herein and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application is intended to include these modifications and variations.

Claims

1. An adaptive control method based on reinforcement learning and radial basis function neural network, characterized in that: include: Establish state equations for nonlinear systems with unknown time-varying uncertainties, and define system state variables and control objectives; In the reinforcement learning framework, the control strategy and value function are initialized, the initial network parameters are set, and the supremum of uncertainty is preliminarily estimated; Using a radial basis function neural network to estimate the uncertainty of the system in real time, and by adjusting the parameters of the radial basis function neural network, approaching the supremum of the uncertainty; Based on the estimated uncertainty, the control strategy is iteratively updated, and in each iteration, the control strategy is optimized using a reinforcement learning algorithm; The control strategy is adjusted in real time, the control strategy parameters are optimized based on the latest uncertainty estimation value, and the final stable control strategy is obtained through convergence analysis.

2. The adaptive control method based on reinforcement learning and radial basis function neural network according to claim 1, characterized in that: The state equation is: x(t)=f(x(t))+g(x(t))u(t)+Δf(x(t)) Where x(t) is the system state, f(x(t)) is the natural state of the system itself, g(x(t)) is the control input gain function of the system, u(t) is the control input, and Δf(x(t)) is the unknown time-varying uncertainty.

3. The adaptive control method based on reinforcement learning and radial basis function neural network according to claim 1, characterized in that: The iterative steps of the initial strategy include: Initializing the control strategy, value function and initial neural network parameters; Select the initial state and set the learning rate, and use the current policy to sample a series of state and action pairs; The value function and control strategy are updated according to the reinforcement learning framework until the control strategy and value function converge.

4. The adaptive control method based on reinforcement learning and radial basis function neural network according to claim 1, characterized in that: The uncertainty estimation steps include: The radial basis function neural network is used to estimate the uncertainty of the system in real time to obtain an estimated value of the uncertainty. The parameters of the radial basis function neural network are adjusted to approximate the supremum of uncertainty, thereby obtaining an approximation value of the supremum of uncertainty.

5. The adaptive control method based on reinforcement learning and radial basis function neural network according to claim 1, characterized in that: Optimize control strategies using reinforcement learning algorithms, including: defining an objective function of the current strategy, and calculating a value of the objective function based on the system state and the uncertainty of the estimation; The reinforcement learning algorithm is used to calculate the gradient of the policy, and the control policy parameters are adjusted according to the gradient information to minimize the objective function.

6. The adaptive control method based on reinforcement learning and radial basis function neural network according to claim 1, characterized in that: The reinforcement learning algorithm is a policy iteration algorithm, and the expression of policy improvement is: Among them, π i is the current strategy, π i+1 (x) is the improved policy function, r(xu) is the immediate reward, γ is the discount factor, P(x′|x,u) is the state transition probability, Following the current strategy π in state x′ i The expected total return that can be obtained.

7. The adaptive control method based on reinforcement learning and radial basis function neural network according to claim 6, characterized in that: The activation function of the radial basis function neural network is a Gaussian radial basis function, expressed as: φ(r)=exp(-β||x-c i || 2 ) Among them, φ is the activation function, β is the adjustment parameter, x is the input vector, c i is the center vector.

8. The adaptive control method based on reinforcement learning and radial basis function neural network according to claim 1, characterized in that: The reinforcement learning algorithm is the ACE reinforcement learning algorithm, and the approximate formula of the radial basis function neural network of the ACE reinforcement learning algorithm is: in, represents the approximation of the value function at the i-th iteration time t, is the weight vector at the i-th iteration, σ(V ci x(t)) is the activation function σ acting on the feature vector V ci The result of x(t), V ci x(t) is the input state x(t) through the feature map V ci The result after conversion, represents the control input approximation at the i-th iteration time t, is the weight vector at the i-th iteration, σ(V ai x(t)) is the activation function σ acting on the feature vector V ai The result of x(t), V ai x(t) is the input state x(t) through the feature map V ai The result after conversion, L i represents the cumulative value from t+b at the i-th iteration, is the weight vector of the i-th iteration time τ, σ(V ei x) is the activation function σ acting on the feature vector V ei The result of x, V ei x is the input state x through the feature map V ei The result of the conversion, τ, is the integral variable from t to t+b.

9. The adaptive control method based on reinforcement learning and radial basis function neural network according to claim 1, characterized in that: The real-time adjustment control strategy includes: In each control cycle, the current state of the system and control input data are collected; Calculate new control input and value functions using current weight parameters; The current weight parameters are used to generate new state and control input data to adjust the control strategy.

10. The adaptive control method based on reinforcement learning and radial basis function neural network according to claim 1, characterized in that: The method further comprises: During the control strategy iteration process, dynamically evaluate the robustness index of the current strategy under different uncertainty conditions, wherein the robustness index includes the stability margin and the fluctuation of the control input; Through the system simulation process, different types and levels of uncertainty are actively injected to simulate the changes in the actual operating environment and obtain simulation results; According to the robustness evaluation results and simulation results, the control strategy parameters are adjusted in real time, and the control strategy is optimized using the feedback loop.

Images