A Safety Control Method and System for Unmanned Vehicles Based on Derivative-Aware Reinforcement Learning
By combining derivative-aware Actor-Critic reinforcement learning with a steady-state priority scheduler, the stability and performance issues of the vehicle trajectory tracking control system under DoS attacks are solved. Steady-state error protection and transient optimization are achieved during the attack, improving the robustness and accuracy of vehicle trajectory tracking control.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- QILU UNIVERSITY OF TECHNOLOGY (SHANDONG ACADEMY OF SCIENCES)
- Filing Date
- 2026-06-01
- Publication Date
- 2026-06-30
AI Technical Summary
Existing technologies in vehicle trajectory tracking control systems face challenges such as denial-of-service attacks that prevent the controller from acquiring measurement data, leading to tracking degradation and system instability. Furthermore, reinforcement learning methods may degrade control performance during attacks, lack derivative sensing and learning gating mechanisms, and are unable to achieve transient performance optimization and steady-state accuracy protection.
A derivative-aware Actor-Critic reinforcement learning method is adopted, combined with a steady-state priority scheduler, to design a finite-time switching state observer and a safety backstepping controller. The transient tracking performance is optimized during the intervals between DoS attacks, and the reinforcement learning authority is decayed in the steady-state region to protect the steady-state error accuracy of the safety subject control.
Significant improvements in the stability and tracking performance of the vehicle trajectory tracking system under DoS attacks were achieved. Transient oscillations were suppressed by derivative sensing to avoid learning contamination and ensure steady-state error accuracy and the safety and robustness of the controller.
Smart Images
Figure CN122308059A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of vehicle safety control technology, specifically to a method and system for unmanned vehicle safety control based on derivative perception reinforcement learning. Background Technology
[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.
[0003] Vehicle trajectory tracking control systems, as a type of networked second-order nonlinear motion system, are widely used in fields such as intelligent transportation and autonomous driving. These systems achieve data interaction between sensors and actuators through network communication, but they also face a serious threat from denial-of-service (DoS) attacks. DoS attacks block communication channels, preventing the controller from acquiring sensor measurement data during the attack, potentially causing significant tracking degradation or even system instability.
[0004] For the safety control problem of disturbed second-order nonlinear systems, existing research mainly employs observer-based robust control, event-triggered mechanisms, or fuzzy / neural network compensation methods. These methods can maintain bounded tracking performance under DoS attacks and external disturbances, but their controllers, primarily tuned based on robustness inequalities, still have the following limitations: (1) Transient performance and energy efficiency are not explicitly optimized online. The control gain of existing security control methods is usually designed conservatively offline and cannot be adaptively adjusted according to the real-time operating status, resulting in unsatisfactory transient tracking performance and control energy consumption during attack intervals.
[0005] (2) The theoretical separation between reinforcement learning enhancement and baseline safety agent control is unclear. Although Actor-Critic reinforcement learning has shown strong optimization capabilities in nonlinear adaptive optimal control, it is still impossible to strictly separate the reinforcement learning enhancement channel from the safety agent control in the context of safety control. Therefore, it is impossible to ensure that reinforcement learning does not compromise safety guarantees while providing an interpretable performance optimization objective.
[0006] (3) Learning updates during an attack may be corrupted. System measurements are unavailable during a DoS attack. If reinforcement learning weight updates continue during this time, it will lead to learning based on erroneous data, which will worsen the control performance. Existing reinforcement learning control methods lack learning gating mechanisms for active attack periods.
[0007] (4) Lack of derivative-aware performance metrics. The standard reinforcement learning cost function only includes the state error term and does not consider the penalty for the rate of change of the state, which makes the controller unable to effectively constrain the rapid oscillations in the transient process and limits the ability to safely improve the reinforcement learning permission space. Summary of the Invention
[0008] To address the aforementioned issues, this disclosure proposes a safety control method and system for unmanned vehicles based on derivative-aware reinforcement learning. It proposes a hierarchical control architecture that integrates a derivative-aware Actor-Critic reinforcement learning enhancement channel and a steady-state priority scheduler on the basis of robust backstepping control. The steady-state priority scheduler attenuates the reinforcement learning authority in the steady-state region to protect the steady-state error accuracy of the safety subject control.
[0009] According to some embodiments, the present disclosure adopts the following technical solutions: Safety control methods for unmanned vehicles based on derivative-aware reinforcement learning include: Based on the dynamic characteristics of unmanned vehicles, a three-degree-of-freedom disturbed vehicle horizontal plane dynamic model and a DoS attack model are established, and the vehicle horizontal plane dynamic model is described as a disturbed second-order nonlinear model. Design a finite-time switching state observer and a safe backstepping controller based on a disturbed second-order nonlinear model; The parameter errors and external disturbances observed by the switching state observer are input to the safety backstepping controller. Based on the safety robust backstepping control, a derivative-aware Actor-Critic reinforcement learning enhancement channel and a steady-state priority scheduler are integrated. During the interval of the DoS attack model, reinforcement learning is used to enhance the online optimization of transient tracking performance. The steady-state priority scheduler decays the reinforcement learning authority in the steady-state region to protect the steady-state error accuracy of the safety main control.
[0010] According to some embodiments, the present disclosure adopts the following technical solutions: The autonomous vehicle safety control system based on derivative-aware reinforcement learning includes: The model building module is used to establish a three-degree-of-freedom disturbed vehicle horizontal plane dynamic model and a DoS attack model based on the dynamic characteristics of unmanned vehicles, and to describe the vehicle horizontal plane dynamic model as a disturbed second-order nonlinear model. The controller building module is used to design a finite-time switching state observer and a safe backstepping controller based on a disturbed second-order nonlinear model. The control module is used to input the parameter errors and external disturbances observed by the switching state observer to the safety backstepping controller. Based on the safety robust backstepping control, it integrates a derivative-aware Actor-Critic reinforcement learning enhancement channel and a steady-state priority scheduler. During the interval of the DoS attack model, it enhances the online optimization of transient tracking performance through reinforcement learning. The steady-state priority scheduler decays the reinforcement learning authority in the steady-state region to protect the steady-state error accuracy of the safety main control.
[0011] According to some embodiments, the present disclosure adopts the following technical solutions: A computer program product includes a computer program that, when executed by a processor, implements the aforementioned autonomous vehicle safety control method based on derivative-aware reinforcement learning.
[0012] According to some embodiments, the present disclosure adopts the following technical solutions: A non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the aforementioned unmanned vehicle safety control method based on derivative-aware reinforcement learning.
[0013] According to some embodiments, the present disclosure adopts the following technical solutions: An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the unmanned vehicle safety control method based on derivative perception reinforcement learning.
[0014] Compared with the prior art, the beneficial effects of this disclosure are as follows: This disclosure presents a safety control method for unmanned vehicles based on derivative-aware reinforcement learning. It designs a finite-time switching state observer and a safety backstepping controller, strictly decoupling the robust safety control entity from the reinforcement learning enhancement channel. Lyapunov stability analysis relies solely on the ISS properties of the control layer and the boundedness of the reinforcement learning channel, without requiring reinforcement learning to solve the exact HJB (Hamilton-Jacobi-Bellman) equations, thus achieving a theoretical separation between safety assurance and performance optimization.
[0015] This disclosed method for autonomous vehicle safety control based on derivative-aware reinforcement learning introduces a state derivative penalty term on top of the standard state error cost, making the Critic residual sensitive to rapid state changes and suppressing transient oscillations. This allows for a safe increase in reinforcement learning authority (from 0.80 to 2.00), forming a monotonically improving ablation chain.
[0016] The safety control method for unmanned vehicles based on derivative-aware reinforcement learning disclosed herein decays the learning update rate to near zero during the active period of a DoS attack, avoiding pollution of weight updates based on erroneous measurement data, and ensuring that the reinforcement learning channel can continue to learn correctly after communication is restored.
[0017] This disclosed method for safety control of unmanned vehicles based on derivative-aware reinforcement learning automatically decays reinforcement learning privileges and restores baseline privileges when the tracking error approaches zero, thus protecting the steady-state error accuracy of the safety control subject. The method allows for the complete shutdown of steady-state reinforcement learning while retaining minor reinforcement learning corrections to improve disturbance suppression.
[0018] The safety control method for unmanned vehicles based on derivative-aware reinforcement learning disclosed herein provides mathematical interpretability for the reinforcement learning layer. During the attack free period and when the projection operator is not activated, the update laws of Critic and Actor are equivalent to the regularized gradient descent flow of the explicit derivative-aware surrogate loss function.
[0019] The present invention discloses a safety control method for unmanned vehicles based on derivative-aware reinforcement learning. The stability theorem provides an expression for the actual limit of the tracking error. This limit is monotonic with respect to the permission envelope of reinforcement learning, providing clear theoretical guidance for controller parameter tuning. Attached Figure Description
[0020] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.
[0021] Figure 1 This is a comparison diagram of the three-degree-of-freedom output tracking trajectories of four controllers according to embodiments of this disclosure under a DoS attack; wherein, the gray shaded area represents the DoS attack zone; Figure 2 This is a comparison chart of the aggregate tracking errors of four controllers according to embodiments of this disclosure; Figure 3 This is a comparison diagram of the control input signals of four controllers according to embodiments of this disclosure; Figure 4 This is a comparison diagram of the position and heading observer states of the AC-Derivative controller according to an embodiment of this disclosure with the actual states; Figure 5 The following is a time evolution diagram of the adaptive filter gain, scheduler parameters, and Bellman residual of the AC-Derivative controller according to an embodiment of this disclosure; Figure 6 The following is a time evolution diagram of the Actor, Critic, and auxiliary neural network weight norms of three reinforcement learning controllers according to embodiments of this disclosure; Figure 7 This is a comparison chart of tracking error and control energy under a DoS duration attack according to an embodiment of this disclosure; Figure 8 This is a comparison chart of tracking error and control energy under the continuous disturbance amplitude of the embodiments of this disclosure; Figure 9 The statistical results of 24 Monte Carlo randomization experiments in this embodiment of the present disclosure; Figure 10 Monte Carlo pairwise difference analysis for embodiments of this disclosure. Detailed Implementation
[0022] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.
[0023] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.
[0024] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0025] Terminology Explanation DoS (Denial of Service) attacks are a type of cyberattack that blocks the communication channel between sensors and actuators, causing the control system to be unable to obtain system measurement information during the attack, thereby causing tracking performance degradation or even instability.
[0026] Backstepping is a recursive nonlinear control design method based on Lyapunov stability theory, which achieves system stability by stepping through the design of virtual control inputs.
[0027] Actor-Critic (AC) reinforcement learning is an online adaptive optimization method in which the Critic (evaluation network) evaluates the value function of the current policy, and the Actor (execution network) outputs the control policy. The two work together to update the performance metrics.
[0028] RBFNN (Radial Basis Function Neural Network) is a general approximator used to approximate unknown nonlinear functions. It consists of a linear combination of Gaussian functions.
[0029] Finite-time observers are a class of state estimators that can converge to the true state in a finite time and provide fast state recovery during the interval.
[0030] Input State Stability (ISS) is a concept of stability for nonlinear systems that guarantees that the system state remains bounded under bounded input disturbances.
[0031] Example 1 One embodiment of this disclosure provides a safety control method for unmanned vehicles based on derivative-aware reinforcement learning, the method steps of which include: Step 1: Based on the dynamic characteristics of unmanned vehicles, establish a three-degree-of-freedom disturbed vehicle horizontal plane dynamic model and a DoS attack model, and describe the vehicle horizontal plane dynamic model as a disturbed second-order nonlinear model; Step 2: Design a finite-time switching state observer and a safe backstepping controller based on the disturbed second-order nonlinear model; Step 3: Input the parameter errors and external disturbances observed by the switching state observer to the safety backstepping controller. On the basis of safety robust backstepping control, integrate derivative-aware Actor-Critic reinforcement learning enhancement channel and steady-state priority scheduler. During the interval of the DoS attack model, enhance the online optimization of transient tracking performance through reinforcement learning. The steady-state priority scheduler decays the reinforcement learning authority in the steady-state region to protect the steady-state error accuracy of the safety main control.
[0032] As one embodiment, this disclosure proposes a hierarchical control architecture to address the security tracking control problem of a vehicle trajectory tracking system under a DoS attack. Based on secure and robust backstepping control, this architecture integrates a derivative-aware Actor-Critic reinforcement learning enhancement channel and a steady-state priority scheduler to achieve the following objectives: (1) Ensure that all closed-loop signals are consistent and eventually bounded (UUB) under DoS attacks and external disturbances; (2) Enhance the online transient tracking performance through reinforcement learning during attack intervals; (3) Avoid learning pollution during attacks by using DoS-aware learning gating; (4) By using a steady-state priority scheduler to attenuate the reinforcement learning authority in the steady-state region, the steady-state error accuracy of the safety subject control is protected. The specific implementation process is as follows: Step 1: Establish a three-degree-of-freedom disturbed vehicle trajectory tracking system model and a DoS attack model.
[0033] This disclosure considers a horizontal plane dynamics model for a three-degree-of-freedom disturbed vehicle trajectory tracking system:
[0034]
[0035] in, p =[x,y,ψ] T This refers to the vehicle's position and heading angle in the inertial coordinate system. x , y These represent the longitudinal and lateral positions, respectively, with ψ being the heading angle. υ =[ v x , v y , r ] TFor the velocity in the vehicle coordinate system, v x , v y These are the longitudinal and lateral velocities, respectively. r This refers to the yaw rate; J ( ψ M is the rotation matrix from the vehicle coordinate system to the inertial coordinate system; C ( υ ) is the Coriolis / eccentricity matrix. D ( υ ) is the damping matrix, τ To control the input or driving torque vector, w It is a bounded external perturbation vector.
[0036] make The system is described as a disturbed second-order nonlinear system model as follows:
[0037]
[0038]
[0039] in These represent the system position, velocity, control input, and output, respectively. For unknown smooth nonlinear dynamics, It is a bounded external disturbance.
[0040] Furthermore, a DoS attack model is established, including: The DoS attack range is denoted as The attack free zone is .set up for Number of times the DoS switch is toggled. Assume the DoS attack meets the following duration and frequency constraints:
[0041]
[0042] Where H(τ,t) represents the set of active DoS attack times within the time interval [τ,t], |H(τ,t)| is the total duration of this set; n(τ,t) is the number of DoS on / off switching times within [τ,t]; ι, τ is a non-negative constant, representing the initial margins for attack duration and switching frequency, respectively; T≥1 is the average attack duration constraint parameter, and τ_d>0 is the average dwell time or switching frequency constraint parameter; τ and t are the start and end times of the statistics, respectively. This represents removing the portion of set H(τ,t) from the time interval [τ,t]. It represents the difference between sets.
[0043] Using RBFNN (Radial Basis Function Neural Network) to approximate unknown nonlinear terms:
[0044] in For the basis function vector, For the ideal weight matrix, This is the bounded approximation error.
[0045] Given a bounded reference signal Define tracking error .
[0046] Step 2: Design a finite-time switching state observer and a safe backstepping controller based on the disturbed second-order nonlinear model.
[0047] (1) Design a finite-time switching state observer:
[0048]
[0049] in, , These are the estimated values for position and velocity, respectively. u To control the input, ( , ) is the RBFNN for unknown nonlinear terms v The estimate, Inject terms into the observer that switch with the DoS state ρ; ρ =0 represents the active period of the DoS attack H(0,∞), and ρ=1 represents the free period of the attack Δ(0,∞).
[0050] Define the observer output error ,in This indicates the available measurement value (the last valid output is retained during the DoS period).
[0051] Furthermore, the active attack period ( Using linear injection terms:
[0052] in To design the gain matrix.
[0053] Attack Freedom Period ( Using finite-time nonlinear injection:
[0054] in:
[0055] in, h ∈(0.8,1) is the finite-time convergent power exponent. >0 and >0 represents the nonlinear injection gain, which adjusts the weights of the position error term and the velocity correction term, respectively. >0 represents the global gain of the finite-time injection term. For any a>0 and z=[z1,z2,z3] T , For component-wise sign-power vectors:
[0056] in, (·) is the sign function. z j For vectors z The j One portion, a The sign power exponent; the above definition guarantees that the injected term has continuous finite-time convergence correction capability for errors of different magnitudes.
[0057] The switching observer provides a state estimate that converges in finite time during the attack-free period and maintains a linearly stable state estimate during the attack-active period.
[0058] (2) Design a safety backstepping controller: Define backstep coordinates:
[0059]
[0060]
[0061]
[0062] in .
[0063] Design an adaptive nonlinear filter:
[0064] in, z 2= θ 2 ρ is the filter error or second-order backstepping auxiliary error; ρ2 is the filter time constant, which determines the convergence speed; β 12 To adaptively compensate for gain,β 22 This is a regularization parameter used to avoid the denominator being in the range of 1 / 2 digits. z 2. Degenerates when approaching zero; 12 , 22 φ is the adaptive compensation gain vector for the filter, used to adjust the second-order squared error term compensation and positive bias compensation, respectively; f >0 represents the robust compensation constant; 13 represents a three-dimensional all-one vector; [·] [q] 、|·|、⊙ represent component exponentiation, absolute value, and Hadamard product, respectively.
[0065] The filter-side adaptive law for each axis Designed as follows:
[0066]
[0067] in γ 12 >0 and γ 22 >0 are respectively 12,i , 22,i Adaptive update gain, λ 12 >0 is 12,i The leakage or forgetting coefficient; =sqrt( β 22 / ( β 12 1)) is the error trigger threshold. i ∈{1,2,3} is the axial index. When | z 2,i |< Updates are paused periodically to reduce steady-state chattering.
[0068] Furthermore, the baseline safety control law is designed as follows: Virtual control law: ; Physical control laws: ; RBFNN Weight Adaptive Update Law: ; in, To determine based on DoS status ρ Available backstep position error for switching between actual tracking error and estimated tracking error; = θ 2 represents the second-order backstepping error; For baseline safety control input; , Positive control gain; For RBFNN estimation terms with unknown nonlinear dynamics, This is the weight estimation matrix. A vector of basis functions; Inject items to switch observers; The output derivative of the filter; and These are the RBFNN weight learning gain matrix and leakage matrix, respectively. This baseline security controller forms the basis of the security robust main control of this invention, and its Lyapunov analysis guarantees the ISS (Input State Stability) property under DoS attacks.
[0069] Step 3: Input the parameter errors and external disturbances observed by the switching state observer to the safety backstepping controller. On the basis of safety robust backstepping control, integrate derivative-aware Actor-Critic reinforcement learning enhancement channel and steady-state priority scheduler. During the interval of the DoS attack model, enhance the online optimization of transient tracking performance through reinforcement learning. The steady-state priority scheduler decays the reinforcement learning authority in the steady-state region to protect the steady-state error accuracy of the safety main control.
[0070] Step 301: Reinforcement learning enhances input channel design; To prioritize reinforcement learning during transient processes while maintaining steady-state accuracy and robustness, the total control input is designed as follows:
[0071] in:
[0072]
[0073] Steady-state priority scheduler Defined as:
[0074]
[0075] in, u This is the main control input. For baseline safety control input, To enhance learning and increase input, The bounded normalization strategy output by the Actor. For state-related effective reinforcement learning permissions; The baseline control scaling factor. To enhance tracking status; >0 represents the upper limit of reinforcement learning permissions. Scaling the lower bound of the baseline >0 represents the scheduler radius. ≥1 represents the scheduler power. The minimum scheduling value is ∈[0,1). , >0 is and Weights in the weighted state norm. hour, The reinforcement learning channel is strongly attenuated, and the baseline permission is restored to close to 1, thereby protecting the steady-state error accuracy.
[0076] Let the augmented tracking state be Define the local channel state for each axis. .
[0077] Step 302: Design of derivative-aware performance index; Define a derivative-aware performance index to explicitly constrain the rate of change of state:
[0078] in, For the first i Axial local channel status , ∈R^{3×3} is a positive definite weight matrix, which penalizes the state error and the rate of change of state, respectively; >0 represents the energy weight for reinforcement learning control. For the first i Axis reinforcement learning enhances input. τ This is the time variable for integration. The penalty for rapid local state changes makes the Critic residual sensitive to the rate of state change, thereby suppressing transient oscillations and allowing for safe enhancement of reinforcement learning permissions.
[0079] Step 303: Axial Critic-Actor parameterization; Select local feature vectors The Critic channel approximation is:
[0080] The Actor channel is parameterized using a bounded strategy:
[0081] in, , , ∈ These are the estimated vectors for the Critic weight, Actor weight, and auxiliary Actor weight on the i-th axis, respectively. The dimension of the feature vector. For local feature vectors, >0 represents the auxiliary strategy channel weight coefficient, and tanh(·) is used to ensure the policy output. Bounded.
[0082] Step 304: Bellman residuals and DoS-aware update law: Define local Bellman residuals:
[0083] Design a DoS-aware learning gating system:
[0084] in, (t) represents the DoS-aware learning gating coefficient, which is set to 1 during the attack freedom period and 1 during the attack active period. 0 significantly reduces the learning update rate; It is a small constant close to zero, used to avoid erroneous measurement data from contaminating Critic, Actor, and auxiliary weight updates during DoS.
[0085] For each axis The update laws for Critic, Actor, and auxiliary weights are as follows:
[0086] in, , , For a pre-selected compact convex set, Proj is a projection operator used to ensure that the weights are bounded; , , It is a positive definite learning gain matrix; , This refers to weight leakage or regularization coefficients; , To assist in weight decay gain; For local Bellman residuals, The derivative of the eigenvector, For the first i Second-order backstepping error of the axis; (t) represents the DoS perceptual learning gating coefficient; [3] represents the element-wise cubic power. This represents the element-wise cubic calculation of the auxiliary Actor weight estimation vector, used to provide a stronger nonlinear decay to suppress weight drift when the auxiliary weights are large.
[0087] During the attack-free period and when the projection operator is not activated, the above update laws for Critic and Actor (evaluator and executor) are equivalent to the gradient descent flow of the regularized surrogate loss function as follows:
[0088] in The derivative is aware of the cost of operation.
[0089] Step 305: Discrete-time implementation: sampling step size The controller uses the following axial feature vector:
[0090] Strategy Channel:
[0091] The state derivative and characteristic derivative are estimated using finite difference:
[0092] Component-wise saturation of the final control input:
[0093] Where k∈{1,2,3} is the control channel index. For the first k Axis actual control input, and The first k Axis baseline safety control component and reinforcement learning enhancement component; The baseline scaling factor after scheduling. >0 is the upper limit for controlling saturation. (·) denotes a component-wise saturation function, limiting the control input to [ , ]Inside.
[0094] Step 4: Stability Analysis and Simulation Verification Step 401: Switch to ISS-Lyapunov (Lyapunov based on input state stability) stability analysis; Define the security subject control aggregation state:
[0095] in, The security subject controls the aggregated state vector; e p = p ande v = v These represent the position and velocity observation errors, respectively; vec(·) denotes matrix vectorization; , These represent the RBFNN weight estimation errors on the position and velocity sides, respectively. 12 , 22 This represents the adaptive gain estimation error of the filter.
[0096] Construct piecewise differentiable Lyapunov functions ,satisfy:
[0097] in, c b lower bound and c b The upper bounds are all positive integers. Give the Lyapunov functions respectively. Relative to the aggregation state norm The lower and upper bounds guarantee It is equivalent to the energy of the closed-loop error.
[0098] The following conditions are met during the attack freedom period and the attack activity period, respectively:
[0099]
[0100] in, >0 indicates the Lyapunov decay rate during the attack-free period. ≥0 indicates the potential growth rate during a period of active DoS attack. >0 represents the gain coefficient of the reinforcement learning enhancement input with respect to the Lyapunov derivative of the subject. It is the combined residual constant composed of external disturbances, neural network approximation errors, and the upper bound of observation errors.
[0101] During DoS switching : .in t j For the first j DoS state transition moments ≥1 is the upper bound of the Lyapunov function jump amplification at the moment of switching.
[0102] By reinforcement learning channel boundedness (projection operator and ensure ), combined with DoS duration / frequency constraints, if there exists Make:
[0103] in, >0 represents the exponential decay margin, and T and These constraints are derived from the DoS duration constraint and the switching frequency constraint, respectively. The switching jump amplification constant is required; the above inequality requires that the decay capability during the attack free period is sufficient to offset the effects of the attack active period growth and the switching jump.
[0104] Then all closed-loop signals are uniformly bounded eventually, and the main control Lyapunov function satisfies:
[0105] in , Specifically, the practical limit of the tracking error is:
[0106] in, Let the closed-loop exponential decay rate be determined by the DoS constraint and the Lyapunov inequality. To switch the amplification factor for DoS, The upper bound of the combined residual caused by perturbation, approximation error, and bounded input of reinforcement learning. c b The lower bound is V b The lower bound constant, This represents the position tracking error.
[0107] This field's understanding of reinforcement learning permission envelopes Monotonically increasing, but due to the steady-state priority scheduler in The reinforcement learning channel will decay to In reality, the contribution of steady-state reinforcement learning is far less than the theoretical worst-case boundary.
[0108] Step 402: Simulation verification; The inertial matrix of a three-degree-of-freedom omnidirectional ground vehicle is:
[0109] The Coriolis matrix and the damping matrix are as follows:
[0110]
[0111] The external disturbance is:
[0112] Reference output signal:
[0113] DoS attack timeline: s.
[0114] Initial conditions: , .
[0115] The simulation comparison of the four controllers (AC: Actor-Critic) is shown in Table 1.
[0116] Table 1 Simulation comparison of four controllers
[0117] Note: AC-Error: Only error-driven Actor-Critic augmented control; AC-Derivative-Const: Has AC and derivative terms, but scheduling is off and parameters are fixed; AC-Derivative: Has AC and derivative terms, RL gain and baseline weights change with the error state.
[0118] Furthermore, to quantitatively compare the tracking performance, steady-state accuracy, and control energy consumption of different controllers, the following evaluation index is defined. Let the total simulation time be... The steady-state evaluation begins at the time when The tracking error is The heading error is The control input is .in, This represents the overall tracking error throughout the entire simulation process. This represents the tracking error during the steady-state phase. This indicates the control of energy consumption. This represents the heading error index. Specifically, it is defined as:
[0119]
[0120]
[0121]
[0122] The comparison results are shown in Table 2 below.
[0123] Table 2 Comparison Results
[0124] As shown in Table 2, the tracking error of the system gradually decreases as the controller functionality is enhanced, indicating that each additional control module brings effective improvement. The baseline controller did not incorporate reinforcement learning; the first improvement method, AC-Error, only incorporated error-based reinforcement learning, thus reducing the overall tracking error by 9.8%; the second improvement method, AC-Derivative-Const, further incorporated derivative-aware cost, enabling the controller to focus not only on the magnitude of the error but also on its rate of change, resulting in a 13.0% reduction in the overall tracking error; the complete method, AC-Derivative, added a steady-state priority scheduling mechanism, allowing reinforcement learning to play a greater role in the transient phase and automatically weaken in the steady-state phase, ultimately reducing the overall tracking error by 33.7%. This demonstrates that derivative-aware cost can suppress overshoot and oscillations in the transient process, allowing the system to increase the reinforcement learning control authority within a safe range, thereby achieving better tracking performance. Robustness verification includes DoS duration scans ( ), Disturbance amplitude scan ( ) and 24 Monte Carlo randomization experiments. Monte Carlo results: AC-Derivative mean It decreased from 3.98 to 2.09 (an improvement of 47%).
[0125] Example 2 One embodiment of this disclosure uses an omnidirectional ground vehicle as an example, and is verified through simulation using MATLAB R2025a software. Specific parameters are selected as follows: Inertia matrix : kg, kg, kg·m, kg·m².
[0126] Baseline control gain: , .
[0127] Observer gain: , , , .
[0128] RBFNN parameters: , , , .
[0129] Filter parameters: , , .
[0130] Reinforcement learning cost matrix: , , .
[0131] Reinforcement learning adaptive gain: , , , , .
[0132] Scheduler parameters: , , , , .
[0133] Control saturation limit: Simulation step size s, simulation duration s.
[0134] Results Explanation: For example Figure 1 As shown, the tracking trajectories of the four controllers are almost indistinguishable under a DoS attack, confirming that reinforcement learning enhances the tracking capabilities of controls that do not compromise security. A brief transient occurs during the active DoS attack period but recovers within one attack cycle.
[0135] like Figure 2 As shown, the aggregate tracking error norm peaks during the DoS attack and decays after the attack ends. The reinforcement learning-based augmentation controller exhibits significantly lower error during the attack interval.
[0136] like Figure 3 As shown, the longitudinal input in the control input channel appears close to [a certain value] during the transient period. The saturation pulses are smoother in both the lateral and directional channels. The control trajectory of the reinforcement learning controller closely matches the baseline control trajectory, confirming the bounded authority design.
[0137] like Figure 4 As shown, the position and heading observer states converge to the true state within 2 seconds.
[0138] like Figure 5 As shown, the time evolution of the adaptive filter gain, scheduler parameters, and Bellman residuals verifies the expected behavior of the scheduler, which remains close to 1 in the transient state (full reinforcement learning privileges) and decays rapidly after the error decreases.
[0139] like Figure 6 As shown, the norms of all Actor, Critic, and auxiliary NN weights remain uniformly bounded within the projection set.
[0140] like Figure 7-8 As shown, DoS attack and disturbance tests demonstrate that the AC-Derivative controller consistently outperforms the baseline under all test conditions.
[0141] like Figure 9-10 As shown, the statistical results of 24 Monte Carlo randomization experiments confirmed the statistical significance of the improvement.
[0142] Example 3 One embodiment of this disclosure provides a safety control system for unmanned vehicles based on derivative-aware reinforcement learning, including: The model building module is used to establish a three-degree-of-freedom disturbed vehicle horizontal plane dynamic model and a DoS attack model based on the dynamic characteristics of unmanned vehicles, and to describe the vehicle horizontal plane dynamic model as a disturbed second-order nonlinear model. The controller building module is used to design a finite-time switching state observer and a safe backstepping controller based on a disturbed second-order nonlinear model. The control module is used to input the parameter errors and external disturbances observed by the switching state observer to the safety backstepping controller. Based on the safety robust backstepping control, it integrates a derivative-aware Actor-Critic reinforcement learning enhancement channel and a steady-state priority scheduler. During the interval of the DoS attack model, it enhances the online optimization of transient tracking performance through reinforcement learning. The steady-state priority scheduler decays the reinforcement learning authority in the steady-state region to protect the steady-state error accuracy of the safety main control.
[0143] Example 4 One embodiment of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned unmanned vehicle safety control method based on derivative-aware reinforcement learning.
[0144] Example 5 One embodiment of this disclosure provides a non-transitory computer-readable storage medium for storing computer instructions. When these computer instructions are executed by a processor, they implement the unmanned vehicle safety control method based on derivative-aware reinforcement learning.
[0145] Example 6 One embodiment of this disclosure provides an electronic device, including a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the unmanned vehicle safety control method based on derivative perception reinforcement learning.
[0146] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0147] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0148] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.
Claims
1. A method for safe control of an unmanned vehicle based on derivative-aware reinforcement learning, characterized in that, include: Based on the dynamic characteristics of unmanned vehicles, a three-degree-of-freedom disturbed vehicle horizontal plane dynamic model and a DoS attack model are established, and the vehicle horizontal plane dynamic model is described as a disturbed second-order nonlinear model. Design a finite-time switching state observer and a safe backstepping controller based on a disturbed second-order nonlinear model; The parameter errors and external disturbances observed by the switching state observer are input to the safety backstepping controller. Based on the safety robust backstepping control, a derivative-aware Actor-Critic reinforcement learning enhancement channel and a steady-state priority scheduler are integrated. During the interval of the DoS attack model, reinforcement learning is used to enhance the online optimization of transient tracking performance. The steady-state priority scheduler decays the reinforcement learning authority in the steady-state region to protect the steady-state error accuracy of the safety main control. 2.The derivative-aware reinforcement learning based safety control method for an unmanned vehicle according to claim 1, wherein, Based on the dynamic characteristics of unmanned vehicles, a three-degree-of-freedom disturbed vehicle horizontal plane dynamic model is established, and the vehicle horizontal plane dynamic model is described as a disturbed second-order nonlinear model, including: Consider the horizontal plane dynamics model of a three-degree-of-freedom disturbed vehicle trajectory tracking system: in, p =[ x , y , ψ ] T This refers to the vehicle's position and heading angle in the inertial coordinate system. x , y These are the vertical and horizontal positions, respectively. ψ For heading angle; υ =[ v x , v y , r ] T For the velocity in the vehicle coordinate system, v x , v y These are the longitudinal and lateral velocities, respectively. r This refers to the yaw rate; J ( ψ () is the rotation matrix from the vehicle coordinate system to the inertial coordinate system; M For the vehicle inertia matrix, C ( υ ) is the Coriolis / eccentricity matrix. D ( υ ) is the damping matrix, τ To control the input or driving torque vector, w Let the external perturbation vector be bounded. make Based on the model, the system is described as a disturbed second-order nonlinear system as follows: in These represent the system position, velocity, control input, and output, respectively. For unknown smooth nonlinear dynamics, It is a bounded external disturbance.
3. The unmanned vehicle safety control method based on derivative-aware reinforcement learning as described in claim 1, characterized in that, Establish a DoS attack model, including: The DoS attack range is denoted as The attack free zone is ,set up for The number of times the DoS switch is toggled, assuming the DoS attack meets the following duration and frequency constraints: in , , It is a known constant.
4. The unmanned vehicle safety control method based on derivative-aware reinforcement learning as described in claim 1, characterized in that, The design of the safe backstepping controller based on the disturbed second-order nonlinear model includes: Define the backstep coordinates; Design an adaptive nonlinear filter; The baseline safety control law is designed as follows: Virtual control law: Physical control laws: RBFNN Weight Adaptive Update Law: in, The backstep position error available in DoS state. This is the second-order backstep error. For baseline safety control input; , Positive control gain; v This is the weight estimation matrix for the RBFNN. For the basis function vector, For unknown nonlinear dynamic estimation terms; To switch observer injection items, The output derivative of the filter; , These are the positive definite learning gain matrix and the leakage matrix, respectively.
5. The unmanned vehicle safety control method based on derivative-aware reinforcement learning as described in claim 1, characterized in that, The integration of derivative-aware Actor-Critic reinforcement learning enhancement channels and a steady-state priority scheduler on the basis of safe and robust backstepping control includes: To leverage reinforcement learning during transient processes while maintaining steady-state accuracy and robustness, the total control input is designed as follows: in: Steady-state priority scheduler Defined as: in, u This is the main control input. For baseline safety control input, To enhance learning and increase input, The bounded normalization strategy output by the Actor. For state-related effective reinforcement learning permissions; The baseline control scaling factor. To enhance tracking status; >0 represents the upper limit of reinforcement learning permissions. ∈(0,1] is the lower bound of the baseline scaling. For the scheduler radius, For the scheduler power, The minimum scheduling value is ∈[0,1). The weights in the weighted state norm; when hour, The reinforcement learning channel is strongly attenuated, and the baseline permission is restored to close to 1, thereby protecting the steady-state error accuracy.
6. The unmanned vehicle safety control method based on derivative-aware reinforcement learning as described in claim 5, characterized in that, Define a derivative-aware performance index to explicitly constrain the rate of change of state: in, For the first i Local channel status of the shaft. , ∈R^{3×3} is a positive definite weight matrix, which penalizes the state error and the rate of change of state, respectively; >0 represents the energy weight for reinforcement learning control. ,i For the first i Axis reinforcement learning enhances input. τ For integration time; The penalty for rapid local state changes makes the Critic residual sensitive to the rate of state change, thereby suppressing transient oscillations and allowing for safe enhancement of reinforcement learning permissions.
7. A safety control system for unmanned vehicles based on derivative-aware reinforcement learning, characterized in that, include: The model building module is used to establish a three-degree-of-freedom disturbed vehicle horizontal plane dynamic model and a DoS attack model based on the dynamic characteristics of unmanned vehicles, and to describe the vehicle horizontal plane dynamic model as a disturbed second-order nonlinear model. The controller building module is used to design a finite-time switching state observer and a safe backstepping controller based on a disturbed second-order nonlinear model. The control module is used to input the parameter errors and external disturbances observed by the switching state observer to the safety backstepping controller. Based on the safety robust backstepping control, it integrates a derivative-aware Actor-Critic reinforcement learning enhancement channel and a steady-state priority scheduler. During the interval of the DoS attack model, it enhances the online optimization of transient tracking performance through reinforcement learning. The steady-state priority scheduler decays the reinforcement learning authority in the steady-state region to protect the steady-state error accuracy of the safety main control.
8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the unmanned vehicle safety control method based on derivative-aware reinforcement learning as described in any one of claims 1-6.
9. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement the unmanned vehicle safety control method based on derivative-aware reinforcement learning as described in any one of claims 1-6.
10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the unmanned vehicle safety control method based on derivative perception reinforcement learning as described in any one of claims 1-6.