Nonlinear pulse control method, device and equipment based on hybrid reinforcement learning

Abstract

The invention relates to the field of nonlinear system optimal control based on artificial intelligence, in particular to a nonlinear pulse control method, device and equipment based on hybrid reinforcement learning, and the method comprises the steps: obtaining the state data of a controlled system in real time; the continuous identifier learns nonlinear dynamics of the system online, and judges whether a pulse control moment is reached or not; a CNN calculation value function is pulsed, and the quality of a current control strategy is judged; the pulse ANN outputs an optimal pulse control signal to drive the system to execute; on the basis of a gradient approximation mechanism, a key basis is provided for weight updating of the pulse ANN; and loop iteration is carried out until the system state converges, and a stable control target is achieved. The problem that a nonlinear controlled system lacks an accurate mathematical model is solved; the problem of high energy consumption caused by continuous output of control signals in traditional continuous control is avoided; and meanwhile, the pulse ANN weight is updated by using a gradient approximate value, so that the problem that the gradient between control input and state transition in a nonlinear system is difficult to analyze and calculate is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of optimal control of nonlinear systems based on artificial intelligence, and in particular to a nonlinear impulsive control method, device and equipment based on hybrid reinforcement learning. BACKGROUND

[0002] In the fields of industrial process control, aerospace, robotics and Internet of Things, there are a large number of controlled objects with limited energy consumption, limited communication bandwidth or only capable of control at discrete time points. For example, satellite attitude control, unmanned aerial vehicle trajectory correction, intermittent production chemical reactor temperature control, and battery-powered Internet of Things node scheduling. The common feature of these scenarios is that they cannot be continuously controlled frequently, and must use impulsive control, that is, a transient control force or energy pulse is applied at specific discrete time points. However, existing advanced control methods based on reinforcement learning are designed for continuous control scenarios and cannot be directly applied to the above-mentioned impulsive control scenarios, resulting in low automation level in energy-constrained fields. The present patent is proposed to overcome this practical engineering technical problem.

[0003] First, existing research mainly focuses on applying reinforcement learning methods (such as the Actor-Critic framework) to continuous control systems. This type of method uses Critic Neural Networks (CNN) to approximate the value function, and Actor Neural Networks (ANN) to learn the optimal control policy, thereby solving the continuous optimal control problem of nonlinear systems. However, as mentioned earlier, these methods usually assume that the system can be continuously controlled and are not suitable for energy-constrained or only impulsive control scenarios.

[0004] Second, existing research uses Identifier Neural Networks (INN) to learn unknown dynamics and uses the identified model for predictive control or adaptive control. This type of method can handle systems with difficult modeling to some extent, but does not combine impulsive control with reinforcement learning and only stays within the continuous control framework.

[0005] Finally, for linear systems, there are optimal solutions for impulsive control, such as analytical derivation based on Riccati equations or state transition matrices. However, when the system is nonlinear, these analytical solutions are not available and existing methods cannot be easily extended to general nonlinear systems. In addition, related research often requires the exact system model to be known and cannot handle model-unknown or complex systems. SUMMARY

[0006] To solve the problem that current reinforcement learning-based methods mainly face continuous control scenarios and are difficult to be directly applied to pulse control systems, thereby resulting in poor effects in energy consumption limited or only discrete time control occasions, the application provides a nonlinear pulse control method, device and equipment based on hybrid reinforcement learning.

[0007] In a first aspect, the application provides a nonlinear pulse control method based on hybrid reinforcement learning: S1: real-time acquisition of running state data of a nonlinear controlled system, the running state data comprising real-time state vectors of the nonlinear controlled system in continuous time; S2: starting a continuous identifier, online learning unknown nonlinear dynamics of the nonlinear controlled system based on the running state data, and outputting an approximate model of the nonlinear dynamics; S3: determining whether the current time is a preset pulse control time, if not, returning to step S1 to continuously acquire the running state data and drive the continuous identifier to update the approximate model; if yes, executing step S4; S4: starting a pulse critic network CNN, based on the current pulse time control state vector before the running state data, the approximate model output by the continuous identifier, the value function of the current control strategy of the nonlinear controlled system is evaluated, the current value function is obtained, and the network weight of the pulse CNN is updated through the time difference TD error; S5: starting a pulse actor network ANN, based on the current pulse time control state vector before, the current value function and the approximate model, calculating and outputting the optimal pulse control amount, sending the optimal pulse control amount to the actuator of the nonlinear controlled system, and driving the nonlinear controlled system to execute pulse control to update the system state; S6: calculating a Jacobian matrix based on the network parameters of the continuous identifier, combining the time interval of the current pulse time and the last pulse time, and a unit matrix to construct a gradient approximation value; S7: updating the network weight of the pulse ANN based on the gradient approximation value, returning to step S1 to realize the cyclic pulse control of the nonlinear controlled system.

[0008] By adopting the technical scheme, the problem of lack of accurate mathematical model of the nonlinear controlled system is solved by acquiring system state data in real time and combining the continuous identifier to learn the nonlinear dynamics online; the high energy consumption problem caused by continuous output of control signals in traditional continuous control is avoided by judging the pulse control time and only starting the pulse CNN and the pulse ANN at the preset time; meanwhile, the gradient approximation value is used to update the pulse ANN weight, solving the problem of difficulty in analytical calculation of the gradient between the control input and the state transition in the nonlinear system, and finally realizing the cyclic pulse control, which significantly reduces the control energy consumption while ensuring the control accuracy and stability of the nonlinear controlled system, and is suitable for scenes with limited energy consumption or only discrete control.

[0009] Optionally, in step S2, the continuous identifier is an identifier neural network INN; the continuous identifier updates a weight matrix of the INN online based on the running state data and a preset adaptive learning rate, and the INN outputs an approximate model of the nonlinear dynamics through multiplication operation of the weight matrix and a preset first nonlinear activation function.

[0010] By adopting the technical scheme, the continuous identifier is specifically defined as the identifier neural network INN, and the INN weight matrix is updated through the adaptive learning rate, and the approximate model is output in combination with the first nonlinear activation function, so that the continuous identifier can more accurately online approximate the unknown dynamics of the nonlinear controlled system, the accuracy and real-time performance of the approximate model are improved, more reliable dynamics basis is provided for subsequent value function evaluation of the pulse CNN and optimal control amount calculation of the pulse ANN, and the rationality of the control strategy and the stability of the control effect are further ensured.

[0011] Optionally, in step S3, whether the current time is the pulse control time includes that a time difference between the current time and a previous pulse control time reaches a preset time interval, or a deviation between a state vector before the current pulse time control and the target state reaches a preset deviation threshold.

[0012] Optionally, in step S4, when the pulse CNN performs value function evaluation, a preset hybrid performance index is further combined; the hybrid performance index includes a terminal state penalty term, a pulse control cost penalty term and a free flight state drift penalty term; the terminal state penalty term is used to punish the deviation between the terminal state of the nonlinear controlled system at the end of control and the target state, the pulse control cost penalty term is used to punish the size of the optimal pulse control amount and the deviation of the state vector before the current pulse time control, and the free flight state drift penalty term is used to punish the state deviation of the nonlinear controlled system between adjacent two pulse control times.

[0013] By adopting the technical scheme, the mixed performance index including the terminal state penalty term, the pulse control cost penalty term and the free flight state drift penalty term is introduced into the value function evaluation of the pulse CNN, so that the value function evaluation can comprehensively consider the terminal accuracy, the control energy consumption and the state stability of the free flight period in the control process, the problem of paying more attention to the accuracy or the stability than the energy consumption is avoided, the balance between the control accuracy, the energy saving and the state stability of the nonlinear controlled system is achieved, and the comprehensive performance of the control strategy is improved.

[0014] Optionally, in step S5, when the pulse ANN calculates the optimal pulse control amount, the multiplication operation of the weight matrix of the pulse ANN and the preset second nonlinear activation function is performed, and the input of the multiplication operation is the state vector before control at the current pulse moment.

[0015] By adopting the technical scheme, the multiplication operation of the weight matrix of the pulse ANN and the second nonlinear activation function is performed to calculate the optimal pulse control amount, and the state vector before control at the current pulse moment is taken as the input, so that the pulse ANN can fully fit the nonlinear correlation between the control input and the state in the nonlinear controlled system, the defect that the linear calculation cannot adapt to the nonlinear system is avoided, the output optimal pulse control amount can meet the system state correction requirement, and the waste of energy consumption caused by too large control amount or the insufficient correction caused by too small control amount is avoided, and the accuracy of the pulse control is improved.

[0016] Optionally, in step S6, the specific process of constructing the gradient approximation value includes: taking the sum of the product of the time interval and the Jacobian matrix and the unit matrix as an intermediate matrix, and sequentially multiplying N intermediate matrices obtained by sampling the intermediate matrix N times in the time interval from the current pulse moment to the previous pulse moment, to obtain the gradient approximation value, where N is an integer greater than or equal to 1.

[0017] Optionally, the nonlinear controlled system is any one of a satellite attitude control system, an unmanned aerial vehicle path correction system, a chemical reactor temperature control system or a battery-powered Internet of Things node scheduling system.

[0018] Optionally, when the nonlinear controlled system is a satellite attitude control system, the state vector includes an attitude angle and an angular velocity of the satellite; when the nonlinear controlled system is a third-order nonlinear system, the state vector includes a position, a speed and an acceleration of the third-order nonlinear system.

[0019] In a second aspect, the application provides a nonlinear impulsive control device based on hybrid reinforcement learning; the device comprises various modules for executing the nonlinear impulsive control method based on hybrid reinforcement learning in the first aspect or any possible implementation manner of the first aspect: The nonlinear impulsive control device based on hybrid reinforcement learning comprises: a data acquisition module for acquiring running state data of a nonlinear controlled system in real time, wherein the running state data comprises a real-time state vector of the nonlinear controlled system in continuous time; a continuous identifier for learning unknown nonlinear dynamics of the nonlinear controlled system based on the running state data, and outputting an approximate model of the nonlinear dynamics; a judgment module for judging whether the current time is a preset impulsive control time, if not, returning to step S1 to continuously acquire the running state data and drive the continuous identifier to update the approximate model; if yes, executing step S4; a pulse critic network CNN for performing value function evaluation on a current control strategy of the nonlinear controlled system based on a current pulse time control previous state vector in the running state data, the approximate model output by the continuous identifier, obtaining a current value function, and updating network weights of the pulse CNN through a time difference TD error; a pulse actor network ANN for calculating and outputting an optimal impulsive control amount based on the current pulse time control previous state vector, the current value function and the approximate model, sending the optimal impulsive control amount to an actuator of the nonlinear controlled system, and driving the nonlinear controlled system to perform impulsive control to update the system state; a gradient approximation calculation module for calculating a Jacobian matrix based on network parameters of the continuous identifier, combining a time interval between the current pulse time and a previous pulse time, and a unit matrix to construct a gradient approximation value; an iterative update module for updating network weights of the pulse ANN based on the gradient approximation value to realize cyclic impulsive control of the nonlinear controlled system.

[0020] In a third aspect, the application provides a computer device comprising a processor, a memory and a communication bus for realizing communication connection between the processor and the memory, and the processor is configured to execute a computer program stored in the memory to realize the nonlinear impulsive control method based on hybrid reinforcement learning according to any one of the above aspects.

[0021] In a fourth aspect, the present application also provides a computer readable storage medium storing a computer program; the computer program can be executed by a processor to implement the nonlinear pulse control method based on hybrid reinforcement learning as described above.

[0022] In a fifth aspect, the present application also provides a computer program product comprising a computer program, which can be executed by a processor to implement the nonlinear pulse control method based on hybrid reinforcement learning as described above.

[0023] The present application has at least the following beneficial technical effects: By acquiring system state data in real time and combining the continuous identifier to learn the nonlinear dynamics online, the problem of lack of accurate mathematical model of the nonlinear controlled system is solved; by judging the pulse control time to start the pulse CNN and the pulse ANN only at the preset time, the high energy consumption problem caused by continuous output of control signals in traditional continuous control is avoided; at the same time, the gradient approximation value is used to update the pulse ANN weight, solving the problem that the gradient between the control input and the state transition in the nonlinear system is difficult to calculate analytically, and finally realizing the cyclic pulse control, which significantly reduces the control energy consumption while ensuring the control accuracy and stability of the nonlinear controlled system, and is suitable for scenes with limited energy consumption or only discrete control. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 A nonlinear pulse control method flowchart provided by an embodiment of the present application; Figure 2 A hybrid reinforcement learning architecture diagram provided by an embodiment of the present application; Figure 3 Another nonlinear pulse control method flowchart provided by an embodiment of the present application; Figure 4 A state convergence diagram of a damped pendulum system provided by an embodiment of the present application; Figure 5 A weight change diagram of a pulse CNN provided by an embodiment of the present application; Figure 6 A state convergence diagram of a third-order nonlinear system provided by an embodiment of the present application; Figure 7 A pulse scheme and continuous scheme controller comparison and energy consumption comparison diagram provided by an embodiment of the present application; Figure 8 A nonlinear pulse control device structure diagram provided by an embodiment of the present application; Figure 9 A computer device structure diagram provided by an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application and not to limit the present application.

[0026] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to be limiting of the present application. As used in the specification and the appended claims of the present application, the singular forms "a," "an" and "the" are intended to include plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or" as used herein refer to and encompass any or all possible combinations of one or more of the listed items. The term "exemplary" means "serving as an example, instance, or illustration," and any embodiment described as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. The terms "first," "second," and the like are used only to describe the objects and do not imply relative importance or a quantity of the indicated technical features. Thus, features defined with "first," "second" can explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise specified.

[0027] At present, there are still many deficiencies in the optimal control of nonlinear systems in the related art. The method based on reinforcement learning mainly faces continuous control scenarios, and it is difficult to be directly applied to pulse control systems, thereby resulting in poor effect in the case where energy consumption is limited or control can only be applied at discrete time. The traditional pulse optimal control method mainly depends on the analytical solution of the linear system, and for nonlinear systems, a complete mathematical model is required, but in actual application, it is often difficult to obtain an accurate model, which limits its applicability. Meanwhile, in the nonlinear pulse control problem, the strategy update of reinforcement learning depends on the gradient information of the system state transition, and the existing method lacks effective means to obtain the gradient, which makes it difficult to guarantee the convergence and optimization of the control law. In addition, the continuous control method needs to apply control input frequently, which is energy-consuming, and the existing research lacks the idea of improving the control method from the energy consumption angle, which is difficult to meet the energy saving requirement. More importantly, although some neural network-based control methods perform well in simulation, they lack systematic stability analysis, which makes it difficult to ensure the convergence and reliability in actual application. In view of this, the nonlinear pulse control method, device and equipment based on hybrid reinforcement learning provided by the embodiments of the present application can solve the above technical problems.

[0028] Reference Figure 1 A nonlinear pulse control method based on hybrid reinforcement learning mainly includes the following steps: S1: real-time acquisition of the running state data of the nonlinear controlled system, wherein the running state data includes a real-time state vector of the nonlinear controlled system in continuous time; S2: starting a continuous identifier, online learning unknown nonlinear dynamics of the nonlinear controlled system based on the running state data, and outputting an approximate model of the nonlinear dynamics; Optionally, in step S2, the continuous identifier is an identifier neural network INN; the continuous identifier updates a weight matrix of the INN online based on the running state data and a preset adaptive learning rate, and the INN outputs the approximate model of the nonlinear dynamics through multiplication operation of the weight matrix and a preset first nonlinear activation function.

[0029] S3: judging whether the current time is a preset pulse control time, if not, returning to step S1 to continuously acquire the running state data and drive the continuous identifier to update the approximate model; if yes, executing step S4; Optionally, in step S3, judging whether the current time is the pulse control time includes that a time difference between the current time and a last pulse control time reaches a preset time interval, or a deviation between a state vector before the current pulse time control and the target state reaches a preset deviation threshold.

[0030] S4: starting a pulse critic network CNN, performing value function evaluation on a current control strategy of the nonlinear controlled system based on a state vector before the current pulse time control in the running state data, the approximate model output by the continuous identifier, obtaining a current value function, and updating network weights of the pulse CNN through a time difference TD error; Optionally, in step S4, when performing the value function evaluation, the pulse CNN also combines a preset hybrid performance index; the hybrid performance index includes a terminal state penalty term, a pulse control cost penalty term and a free flight state drift penalty term; the terminal state penalty term is used to punish a deviation between a terminal state at the end of control of the nonlinear controlled system and the target state, the pulse control cost penalty term is used to punish a size of the optimal pulse control amount and a deviation of the state vector before the current pulse time control, and the free flight state drift penalty term is used to punish a state deviation of the nonlinear controlled system between adjacent two pulse control times.

[0031] S5: starting a pulse actor network ANN, calculating and outputting an optimal pulse control amount based on the state vector before the current pulse time control, the current value function and the approximate model, sending the optimal pulse control amount to an execution mechanism of the nonlinear controlled system, driving the nonlinear controlled system to perform pulse control to update the system state; Optionally, in step S5, when the pulse ANN calculates the optimal pulse control amount, the calculation is realized by a product operation of a weight matrix of the pulse ANN and a preset second nonlinear activation function, and an input of the product operation is the current pulse moment control state vector.

[0032] S6: Calculate a Jacobian matrix based on network parameters of the continuous identifier, and combine a time interval between the current pulse moment and a previous pulse moment and a unit matrix to construct a gradient approximation value. Optionally, in step S6, the specific process of constructing the gradient approximation value includes: taking a sum of the time interval and a product of the Jacobian matrix as an intermediate matrix, and sequentially multiplying N intermediate matrices obtained by sampling the intermediate matrix N times in a time interval between the current pulse moment and the previous pulse moment, to obtain the gradient approximation value, where N is an integer greater than or equal to 1.

[0033] S7: Update network weights of the pulse ANN based on the gradient approximation value, and return to step S1 to realize cyclic pulse control of the nonlinear controlled system.

[0034] Optionally, the nonlinear controlled system is any one of a satellite attitude control system, an unmanned aerial vehicle path correction system, a chemical reactor temperature control system, or a battery-powered Internet of Things node scheduling system.

[0035] Optionally, when the nonlinear controlled system is a satellite attitude control system, the state vector includes attitude angles and angular velocities of the satellite; and when the nonlinear controlled system is a three-order nonlinear system, the state vector includes position, velocity and acceleration of the three-order nonlinear system.

[0036] The following describes a specific technical solution of the above nonlinear pulse control method by taking a satellite attitude damper as a typical pulse control scenario. The satellite needs to maintain stability through attitude adjustment in space, but its thruster fuel is limited, and therefore, a pulse working mode must be adopted to save energy. Existing reinforcement learning is mostly applied to continuous control systems, and the embodiment of the present application is specially designed for such a pulse control scenario, an Actor-Critic framework is designed, and the control input is only updated at discrete pulse moments, so that the control performance is guaranteed while the control frequency and energy consumption are significantly reduced, thereby meeting the dual needs of energy saving and efficient control.

[0037] As Figure 2As shown, the embodiment provides a nonlinear impulsive control system based on hybrid reinforcement learning, mainly including a hybrid reinforcement learning controller deployed in an onboard computer, and a pose sensor. The control process mainly includes state acquisition, model identification, impulsive control decision, gradient support, and iterative optimization. For details, please refer to Figure 3 As shown: Step S301: Real-time acquisition of state data of the controlled system; First, the real-time running state of the nonlinear system is collected through the sensor, and the current status of the control object is determined to provide basic data for subsequent modeling and control.

[0038] Taking the satellite attitude control scene as an example, the key states of the satellite are measured by the attitude sensor, such as the attitude angle and the angular velocity The two parameters are integrated into the system state vector , and the state data (including the impulsive control moment and the free flight moment between two impulses) are continuously output to enable the identifier and the controller to grasp the system dynamics in real time and avoid blind control.

[0039] In this embodiment, the state of the control system is represented as: Among them, represents the derivative of the system state with respect to time (state change rate), reflecting the change trend of the system state with time; represents the system state vector; represents an unknown / complex nonlinear dynamic model, describing the natural motion law of the system without control input (non-impulsive moment), such as the comprehensive influence of the earth's gravitational gradient and the star's magnetic coupling torque on the satellite motion; represents the system state vector before the control at the i-th impulsive moment, i.e., the initial state of the impulsive control; represents the system state vector after the control at the i-th impulsive moment, i.e., the termination state of the impulsive control, triggered by the control input ; is a pre-defined control input matrix (dimension: “state vector dimension x control input dimension”), describing the influence channel of the control input on the system state; represents the instantaneous control input (impulse amount) to be applied at the discrete moment , for example, the satellite scene is the applied thruster impulse thrust (unit: Newton-second).

[0040] Step S302: Online learning of the nonlinear dynamics of the system by using a continuous identifier INN; It is worth noting that the main difficulty of the impulse control based reinforcement learning scheme compared with the traditional continuous control is that in order to update the actor network, it is necessary to know "how will the attitude change at the next moment if the thrust size is changed a little", that is This gradient information; in linear systems, this term can be directly simplified by the state transition matrix, but it is difficult to be analytically obtained in nonlinear systems. Therefore, the identifier neural network is introduced in the embodiment to model the complex system dynamics and serve the gradient approximation mechanism. The structure of the continuous INN is as follows: wherein, represents the approximate nonlinear dynamic model output by the continuous INN (identifier neural network), which is used to online approximate the unknown real dynamics f(x(t)) (such as the complex space dynamics of a satellite, the gravity-friction dynamics of a damped pendulum), and provides the system rule basis for the control strategy. represents the online real-time estimated weight matrix of the continuous INN (dimension: activation function output dimension x state vector dimension), which is updated with time t; is the nonlinear activation function (such as Sigmoid, ReLU, etc.) of the continuous INN, the input is the real-time system state x(t), and the output is the nonlinear transformed feature vector.

[0041] Based on the system at all times (including impulse time and free flight period), the network continuously updates the INN using the sampled data of the attitude angle , the angular velocity and the adaptive rate, online learns and approximates the unknown space dynamics model (such as how the satellite will deviate at the next step under a certain attitude), and the output is a real-time updated dynamics model , which is used for the learning of the impulse controller and the value function network in the online framework.

[0042] Through the adaptive update rule, the weights of the continuous identifier are continuously adjusted , so that the output approximate model is closer and closer to the real system rule.

[0043] Step S303: judging whether the impulse control time is reached; Unlike traditional continuous control, the embodiment only performs control at discrete impulse time t i , such as when the satellite attitude deviation exceeds the threshold or reaches the preset time interval, so as to avoid continuous energy consumption.

[0044] Specifically, if the current time (non-pulse moment), the system is in free flight state, no control is applied, only the continuous identifier continues to collect state, update model; if the current moment (pulse moment): start the follow-up control evaluation, execution, etc.

[0045] To depict the system performance and control cost, save fuel (control energy consumption) while quickly and stably stabilizing the attitude, the following continuous-pulse hybrid performance index is defined in the embodiment: , wherein, represents the system performance index at the kth pulse moment; represents the terminal state penalty function, input

[0046] is the system state vector at the terminal moment (such as the attitude angle and angular velocity of the satellite at the end of control), which is used to punish the deviation of the terminal state from the target state, and ensure that the system is aligned with the target (such as the satellite attitude returning to zero) at the end of control. represents the control cost penalty function at the ith pulse moment, the input is the pulse control input and the state before control , which is used to punish the size of each pulse thrust (directly related to fuel consumption) and the attitude error when the thrust is applied; represents the state drift penalty function of the ith pulse interval

[0047] , i.e. the free flight period between two pulses); the input is the real-time system state x(t) in the free flight period; which is used to punish the state deviation (such as satellite attitude drift and damping swing angle deviation) between two pulses, and ensure that the system will not deviate too much from the target without control. Unlike traditional methods, the embodiment proposes the following innovative mechanisms for hybrid optimal performance index: that is, a value function conforming to the performance index is designed at each pulse moment, and a pulse Actor-Critic mechanism is designed accordingly. Specifically, a pulse CNN is used to approximate the value function at each pulse point, and the weight update is corrected through temporal-difference (TD) error. This design ensures that the value function can accurately reflect the pros and cons of the strategy, thereby providing a basis for strategy improvement; Step S304: the pulse CNN (critic network) calculates the value function to judge the pros and cons of the current control strategy;

[0048] At the pulse moment t i , the pulse CNN is started to evaluate the pros and cons of the follow-up control strategy under the current state, and provide a reference for control force adjustment. Specifically, the pulse CNN receives the initial state at the current pulse moment (the state before applying control); Based on the preset performance indicators , including terminal attitude deviation penalty, pulse thrust energy consumption penalty, and free flight drift penalty, the value function is approximately calculated The larger the value is, the better the subsequent control can achieve low energy consumption and stable attitude under the current state; The weights of the CNN are corrected through the time difference (TD) error, so that the evaluation of the value function is more and more accurate. For example, the actual energy consumption is higher than expected last time, so the weight is adjusted to make the judgment more accurate. Thus, the pulse thrust output by the subsequent ANN provides an optimization basis.

[0049] Step S305: The pulse ANN (actor network) outputs the optimal pulse control signal to drive the system to execute; At the pulse moment t i , combined with the model of the identifier and the evaluation result of the CNN, the pulse ANN (actor network) is started, and the appropriate pulse control strength is calculated and output.

[0050] Specifically, the pulse ANN receives the current state , the evaluation value of the CNN , and the approximate model output by the continuous identifier ; Based on the preset network structure (including weights and activation functions ), the optimal pulse control amount is calculated; Based on the optimal pulse control amount , a control signal is generated and sent to the actuator (such as a satellite thruster) to apply a transient pulse. The system state will jump from to , and B is the control input matrix. Thus, the system state (such as the attitude of a satellite deviating from the target value) is accurately corrected without wasting energy.

[0051] In this embodiment, the pulse ANN is used to output the optimal pulse controller at the pulse moment, and the modeling is as follows: Wherein, is the estimated weight matrix of the pulse ANN (actor network) at the i-th pulse moment, T represents the matrix transpose symbol, and is converted from a column matrix to a row matrix to ensure that the dimensions match (matrix multiplication is possible) with the output of the activation function ; For nonlinear activation functions (such as ReLU, tanh, etc.), at each pulse control time... The network is based on the currently measured attitude state. Calculate and estimate the optimal pulse thrust in real time. By combining the gradient approximation information provided by the continuous identifier with the optimization error correction mechanism, the ANN actor network can achieve online iterative updates of the weights, thereby continuously approximating the optimal impulse control law. This design not only solves the problem of unavailable gradient information in nonlinear systems but also ensures the convergence and stability of the learning process.

[0052] Step S306: Based on the gradient approximation mechanism, provide an optimization basis for the weight update of the pulsating ANN.

[0053] In nonlinear systems, the effect of changes in control force on subsequent states, i.e., the gradient, cannot be directly calculated. To ensure that the control strategy of the pulse ANN can be continuously optimized, an approximate gradient needs to be constructed using the model provided by the continuous identifier.

[0054] In this embodiment, the following gradient approximation mechanism is designed: in, This is an approximation of the gradient. It is the identity matrix. For the i-th pulse interval [ ti−1 ,t i The time interval, The Jacobian matrix is ​​based on the INN weights and activation function. This represents the system state vector at the j-th sampling time within the i-th pulse interval; By using this approximation mechanism, it is possible to construct a pair The estimated value is used to input the gradient information into the impulse ANN, and the weights of the replacement ANN are adjusted. This breakthrough overcomes the bottlenecks imposed on Actor-Critic design by nonlinearity and impulse characteristics.

[0055] Step S307: Iterate repeatedly until the system state converges and the stable control objective is achieved.

[0056] Repeat steps S401-S406 to continuously optimize the control strategy: the identifier continuously updates the model to accurately model complex dynamics; the pulse CNN and ANN update the weights at each pulse moment to make the evaluation more accurate and the control energy consumption lower; until the system state meets the preset target, such as the satellite attitude angle θ approaching 0, the angular velocity ω approaching 0, and the energy consumption maintained at a low level, the control process stops to ensure that the control method can be implemented.

[0057] Finally, this embodiment integrates the above ideas into a unified hybrid reinforcement learning framework: the pulsed Actor-Critic module is responsible for state value function evaluation and pulse controller output at pulse time; while the continuously updated INN module continuously identifies the entire nonlinear dynamics and provides key transition gradient information to the Actor network through the approximation mechanism. The three form an information interaction and collaborative learning relationship, such as Figure 2 The continuous identifier, pulse CNN and pulse ANN in the middle form the overall hybrid learning architecture. This framework realizes optimal pulse control learning under unknown nonlinear dynamics, reduces control energy consumption and execution frequency while ensuring system stability. Through the collaborative work of the components, the invention enables the satellite to learn an optimal pulse control law online, which can use the least fuel (fewer pulse times and smaller thrust each time) and quickly stabilize the attitude.

[0058] The superiority of the proposed hybrid reinforcement learning architecture is illustrated by two simulation experiments. First, the damped pendulum system is taken as the controlled object (its dynamic model is equivalent to the damped pendulum system), and the standard dynamic model is: where, denotes the swing angle of the damped pendulum, denotes the angular velocity of the damped pendulum, is the acceleration of gravity, is the pendulum length of the damped pendulum, is the damping coefficient. Define the state vector and apply the pulse control input in the angular velocity equation, then the system can be written as a state space model: where, denotes the derivative of the system state vector x with respect to time, x denotes the system state vector of the damped pendulum, denotes the first component of the state vector x, i.e. the swing angle θ; denotes the second component of the state vector x, i.e. the angular velocity ω; u denotes the pulse control input, i.e. the control torque applied to the damped pendulum.

[0059] In this system, if the traditional continuous control method is used, the control torque needs to be applied frequently, which not only consumes a lot of energy, but also is not practical in embedded platforms or energy-limited scenarios. The nonlinear pulse control method based on hybrid reinforcement learning proposed in this embodiment only calculates and applies control input at discrete pulse time, avoiding the energy consumption problem caused by continuous control. Using this scheme to apply pulse control to the system, Figure 4The state evolution of the system under pulse control is shown, and the system state converges after a certain time, which illustrates the effectiveness of the control scheme.

[0060] Meanwhile, the pulse-type CNN network proposed in this embodiment can approximate the value function corresponding to the performance index during the learning process and update iteratively through the time difference error. Figure 5 The change of the CNN network weight with time is shown, and it can be observed that the weight converges to a stable value eventually, which indicates that the critic network can accurately evaluate the pros and cons of the control strategy and provide an effective basis for the policy improvement of the actor network. This result not only verifies the feasibility of the Actor-Critic architecture in the pulse control scenario, but also reflects that the mixed characteristics proposed in this embodiment can maintain a stable learning process in actual operation.

[0061] To better illustrate the applicability of the present scheme to nonlinear models, this embodiment also illustrates the nonlinear pulse control based on mixed reinforcement learning provided by the present scheme for a nonlinear system based on a third-order theoretical model. Specifically, a three-dimensional system state vector is defined for the nonlinear system as and a time-varying disturbance is introduced The state space modeling of the system is as follows: In the formula, denotes the derivative of the third-order system state vector x with respect to time, , , denotes the three components of the state vector x, for example the basic state (such as position); denotes the rate of change of (the speed); denotes the rate of change of (the acceleration); denotes the time-varying disturbance term, which is an external disturbance (such as wind, noise, load fluctuation, etc.) that changes with time t, making the system not self-stable and having to rely on control input u for stability; denotes the control input matrix B; u denotes the pulse control input, i.e., the instantaneous control quantity applied to the third-order system.

[0062] The system has obvious nonlinear characteristics. Due to the presence of the disturbance term, the system itself does not have self-stability and must rely on external control input for stable operation. Traditional continuous control methods usually require high-frequency control input to offset the disturbance in such scenarios, which significantly increases energy consumption and places higher requirements on system communication bandwidth. By using the nonlinear pulse control method based on mixed reinforcement learning proposed in this embodiment, the system state can be stably converged within a limited time. Figure 6The state trajectory evolution of the third-order nonlinear system under the embodiment is shown, and it can be seen that all state components gradually decay to zero, verifying the effectiveness and robustness of the control law under nonlinear and disturbance interference.

[0063] To further verify the advantages of the present scheme, the present embodiment is compared with the classical continuous control type Actor-Critic reinforcement learning method, and the results are shown in Figure 7 Figure 7 The upper subgraph of FIG. 1 intuitively shows the controllers of the two methods, and the controller proposed in the present embodiment is a pulse control, which can be seen in the green subgraph in the subgraph; Figure 7 The lower subgraph of FIG. 1 quantifies the comparison of the energy consumption inputs of the two methods, and it can be seen that the total energy consumption reduction effect of the scheme of the present embodiment is significant, and it can be more flexibly applied to various communication-limited and resource-energy-limited systems.

[0064] 1. The present embodiment combines continuous system identification technology with pulse Actor-Critic method according to the characteristics of pulse control system, and proposes a unique hybrid reinforcement learning architecture. The architecture can update the controller at discrete pulse time, and identify unknown nonlinear dynamics in continuous interval, thereby overcoming the limitation of traditional reinforcement learning which is only applicable to continuous control and cannot effectively handle pulse control problems; 2. The present embodiment introduces the energy-efficient pulse control idea into the reinforcement learning control framework, and systematically analyzes and designs for nonlinear systems. Compared with traditional continuous control methods, the scheme can reduce control frequency and energy consumption while still guaranteeing control performance and convergence, forming a complete nonlinear pulse optimal control solution; 3. In view of the problem that the Actor network in the hybrid architecture relies on gradient data in the update process, and the nonlinear system is difficult to provide directly, the present embodiment designs an approximation mechanism based on the weights of the identifier neural network, which is used to calculate the sensitivity between pulse input and state transition. The mechanism effectively solves the key technical problems in the online implementation of the hybrid architecture, and guarantees the correctness and convergence of the policy update direction.

[0065] On the basis of the above-mentioned nonlinear pulse control method based on hybrid reinforcement learning, the present embodiment further provides a nonlinear pulse control device based on hybrid reinforcement learning, which can be used to realize the steps of the above-mentioned method.

[0066] Referring to Figure 8 , the nonlinear pulse control device based on hybrid reinforcement learning comprises: A data acquisition module 81 is configured to acquire real-time operating state data of a nonlinear controlled system, wherein the operating state data comprises a real-time state vector of the nonlinear controlled system in continuous time; ​a continuous identifier 82, configured to learn unknown nonlinear dynamics of the nonlinear controlled system based on the running state data, and output an approximate model of the nonlinear dynamics; a judgment module 83, configured to judge whether a current time is a preset pulse control time, if not, return to step S1 to continuously acquire the running state data and drive the continuous identifier to update the approximate model; if yes, execute step S4; a CNN pulse critic network 84, configured to perform value function evaluation on a current control strategy of the nonlinear controlled system based on a current pulse time control state vector in the running state data, the approximate model output by the continuous identifier, obtain a current value function, and update network weights of the pulse CNN through a time difference TD error; an ANN pulse actor network 85, configured to calculate and output an optimal pulse control amount based on the current pulse time control state vector, the current value function and the approximate model, and send the optimal pulse control amount to an actuator of the nonlinear controlled system to drive the nonlinear controlled system to perform pulse control to update a system state; a gradient approximation calculation module 86, configured to calculate a Jacobian matrix based on network parameters of the continuous identifier, and construct a gradient approximation value in combination with a time interval between the current pulse time and a previous pulse time and a unit matrix; an iterative update module 87, configured to update network weights of the pulse ANN based on the gradient approximation value, to realize cyclic pulse control of the nonlinear controlled system.

[0067] The various changes and specific examples in the method provided in the above embodiments are also applicable to the nonlinear pulse control device of the present embodiment. Through the foregoing detailed description of the nonlinear pulse control method, those skilled in the art can clearly understand the implementation method of the device in the present embodiment. For the sake of brevity of the description, the implementation method of the device will not be described in detail here.

[0068] In order to better execute the program of the above method, the present embodiment further provides a computer device, as shown in the accompanying drawings, Figure 9 The computer device includes a processor 97, a memory 92, and a communication bus 93 for realizing communication connection between the processor 97 and the memory 92.

[0069] The computer device can be implemented in various forms, including mobile phones, tablets, palmtop computers, notebook computers and desktop computers, etc.

[0070] The memory can be configured to store instructions, programs, codes, code sets or instruction sets. The memory can include a program storage area and a data storage area. The program storage area can store instructions for implementing an operating system, instructions for at least one function, and instructions for implementing the method provided in the above embodiments, etc. The data storage area can store data involved in the method provided in the above embodiments, etc.

[0071] Optionally, the memory is a read-only memory (ROM), a random access memory (RAM), an electrically erasable programmable read-only memory (EEPROM), an optical disk (including a compact disc read-only memory (CD-ROM), a compressed disk, a laser disk, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer, but is not limited to this. The memory exists independently and is connected to the processor through a communication bus, or the memory is integrated with the processor.

[0072] The processor can include one or more processing cores. The processor calls data stored in the memory by running or executing instructions, programs, code sets or instruction sets stored in the memory, and performs various functions and processes data of the present application. The processor can be at least one of an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller and a microprocessor. It can be understood that, for different devices, the electronic devices used to implement the functions of the above processor can also be others, and the embodiments of the present application are not limited specifically.

[0073] The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus can be divided into an address bus, a data bus, a control bus, or the like. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0074] In an optional embodiment, the computer device can further include a communication interface (not shown) for communication with other devices.

[0075] Embodiments of the present application provide a computer readable storage medium, for example, including: a U disk, a mobile hard disk, a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes. The computer readable storage medium stores a computer program capable of being loaded by a processor and executing the method of the above embodiments.

[0076] Embodiments of the present application also provide a computer program product, which includes a computer program tangibly embodied on a computer readable medium, the computer program containing program codes for executing any of the methods of the embodiments of the present application. The computer program can be downloaded and installed on the network, and / or installed from a removable medium (such as a disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like).

[0077] The above embodiments are only used to introduce the technical solutions of the present application in detail, but the above embodiment descriptions are only used to help understand the method of the present application and its core idea, and should not be understood as a limitation of the present application. Those skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A nonlinear impulse control method based on hybrid reinforcement learning, characterized in that, The method includes: S1: Real-time acquisition of the operating status data of the nonlinear controlled system, wherein the operating status data includes the real-time state vector of the nonlinear controlled system over a continuous time period; S2: Start the continuous identifier, learn the unknown nonlinear dynamics of the nonlinear controlled system online based on the operating state data, and output an approximate model of the nonlinear dynamics; S3: Determine whether the current time is the preset pulse control time. If not, return to step S1 to continuously acquire the running status data and drive the continuous identifier to update the approximate model; if yes, execute step S4. S4: Start the Pulse Commentator Network (CNN), and based on the current pulse moment control state vector in the running state data and the approximate model output by the continuous identifier, evaluate the current control strategy of the nonlinear controlled system by the value function, obtain the current value function, and update the network weights of the Pulse CNN through the time difference TD error; S5: Start the pulse actor network (ANN), calculate and output the optimal pulse control quantity based on the current pulse moment control state vector, the current value function and the approximate model, and send the optimal pulse control quantity to the actuator of the nonlinear controlled system to drive the nonlinear controlled system to perform pulse control to update the system state; S6: Calculate the Jacobian matrix based on the network parameters of the continuous identifier, and construct an approximate gradient value by combining the time interval between the current pulse moment and the previous pulse moment and the identity matrix. S7: Update the network weights of the pulse ANN based on the gradient approximation, and return to step S1 to realize the cyclic pulse control of the nonlinear controlled system.

2. The method as described in claim 1, characterized in that, In step S2, the continuous identifier is an identifier neural network (INN); the continuous identifier updates the weight matrix of the INN online based on the running state data and a preset adaptive learning rate; the INN outputs an approximate model of the nonlinear dynamics by multiplying the weight matrix with a preset first nonlinear activation function.

3. The method as described in claim 1, characterized in that, In step S3, determining whether the current time is the pulse control time includes: the time difference between the current time and the previous pulse control time reaches a preset time interval, or the deviation between the current pulse control state vector and the target state reaches a preset deviation threshold.

4. The method as described in claim 1, characterized in that, In step S4, when the pulse CNN performs value function evaluation, it also incorporates a preset hybrid performance index. The hybrid performance index includes a terminal state penalty term, a pulse control cost penalty term, and a free-flight state drift penalty term. The terminal state penalty term is used to penalize the deviation between the terminal state and the target state at the end of the control of the nonlinear controlled system. The pulse control cost penalty term is used to penalize the magnitude of the optimal pulse control quantity and the deviation of the state vector before control at the current pulse moment. The free-flight state drift penalty term is used to penalize the state deviation of the nonlinear controlled system between two adjacent pulse control moments.

5. The method as described in claim 1, characterized in that, In step S5, when the pulse ANN calculates the optimal pulse control quantity, it is achieved by multiplying the weight matrix of the pulse ANN with a preset second nonlinear activation function. The input of the multiplication operation is the state vector before control at the current pulse moment.

6. The method as described in claim 1, characterized in that, In step S6, the specific process of constructing the gradient approximation includes: taking the sum of the product of the time interval, the Jacobian matrix, and the identity matrix as an intermediate matrix, and sequentially multiplying the N intermediate matrices obtained by sampling the intermediate matrix N times within the time interval between the current pulse time and the previous pulse time to obtain the gradient approximation, where N is an integer greater than or equal to 1.

7. The method according to any one of claims 1-6, characterized in that, The nonlinear controlled system can be any one of a satellite attitude control system, a UAV trajectory correction system, a chemical reactor temperature control system, or a battery-powered Internet of Things node scheduling system.

8. The method as described in claim 7, characterized in that, When the nonlinear controlled system is a satellite attitude control system, the state vector includes the satellite's attitude angle and angular velocity; when the nonlinear controlled system is a third-order nonlinear system, the state vector includes the position, velocity, and acceleration of the third-order nonlinear system.

9. A nonlinear impulse control device based on hybrid reinforcement learning, characterized in that, include: The data acquisition module is used to acquire the operating status data of the nonlinear controlled system in real time, the operating status data including the real-time state vector of the nonlinear controlled system over a continuous time period; A continuous identifier is used to learn the unknown nonlinear dynamics of the nonlinear controlled system online based on the operating state data, and output an approximate model of the nonlinear dynamics; The judgment module is used to determine whether the current time is the preset pulse control time. If not, it returns to step S1 to continuously acquire the running status data and drive the continuous identifier to update the approximate model; if yes, it executes step S4. The Pulse Commentator Network (CNN) is used to evaluate the current control strategy of the nonlinear controlled system based on the current pulse moment pre-control state vector in the running state data and the approximate model output by the continuous identifier, to obtain the current value function, and to update the network weights of the Pulse CNN through the time-difference (TD) error. An ANN (Application Not Responsible Network) is used to calculate and output the optimal pulse control quantity based on the current pulse moment, the current value function, and the approximate model. The optimal pulse control quantity is then sent to the actuator of the nonlinear controlled system to drive the nonlinear controlled system to perform pulse control to update the system state. The gradient approximation calculation module is used to calculate the Jacobian matrix based on the network parameters of the continuous identifier, and construct an approximate gradient value by combining the time interval between the current pulse moment and the previous pulse moment and the identity matrix. An iterative update module is used to update the network weights of the pulse ANN based on the gradient approximation value, so as to realize the cyclic pulse control of the nonlinear controlled system.

10. A computer device, characterized in that, It includes a processor, a memory, and a communication bus, wherein the communication bus is used to establish a communication connection between the processor and the memory, and the processor is used to execute a computer program stored in the memory to implement the nonlinear impulse control method based on hybrid reinforcement learning as described in any one of claims 1-7.

Images