A small robot dynamic balance control system based on DDPG

Abstract

The application relates to the technical field of intelligent robot control, in particular to a small robot dynamic balance control system based on DDPG, which comprises a sensor and a preprocessing module, accurate and reliable state data are provided through denoising and standardization processing, and the accuracy of subsequent processing is enhanced; an ESO observer module estimates system state and total disturbance in real time, and the adaptability to complex environments and anti-interference performance are improved; an information fusion and time sequence alignment module ensures information synchronization and accuracy through fusion processing and multi-dimensional state vector construction, guarantees the timeliness of control decisions, and improves overall control effect; a DDPG strategy network and a safety layer module are combined with strategy optimization and multi-layer safety constraints, and continuous and safe control actions are output, so that the stability and safety of control are improved; and an actuator interface module realizes control instruction transmission, forms a closed loop feedback, adjusts the control strategy in real time, and improves the dynamic balance capability and overall stability of the system.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot control technology, and in particular to a dynamic balance control system for a small robot based on DDPG. Background Technology

[0002] Small robots (such as bipedal robots, quadrupedal robots, wheeled robots, etc.) are characterized by lightweight, high mobility, and multiple constraints. Achieving dynamic balance control in complex terrain and disturbed environments faces many core technical challenges.

[0003] Traditional PID control relies on precise mathematical models, and its performance deteriorates sharply when faced with parameter perturbations, unmodeled dynamics, and external disturbances, failing to guarantee dynamic equilibrium control. Adaptive control, while capable of handling uncertainty, suffers from high computational complexity, making it difficult to meet real-time control requirements. Simple control methods, while meeting real-time requirements, lack robustness. Model predictive control (MPC) has advantages in constraint handling, but its computational burden is heavy, making it difficult to achieve high-frequency real-time control. Furthermore, in complex environments, traditional control methods may produce unsafe or abrupt control actions, threatening robot safety.

[0004] However, existing technologies that use ESO or DDPG alone have certain limitations and cannot effectively solve the problem of dynamic balance control for small robots. Therefore, it is necessary to combine the two to achieve better control performance. To this end, this invention proposes a dynamic balance control system for small robots based on DDPG. Summary of the Invention

[0005] The purpose of this invention is to solve the problems in the background art by proposing a dynamic balance control system for small robots based on DDPG.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A dynamic balance control system for a small robot based on DDPG includes: a sensor and preprocessing module, an ESO observer module, an information fusion and timing alignment module, a DDPG policy network and security layer module, and an actuator interface module.

[0008] The sensor and preprocessing module is used to collect the state data of the small robot and preprocess the state data.

[0009] The ESO observer module is used to perform system state estimation and total disturbance estimation based on preprocessed state data and a state extension model. The total disturbance includes unmodeled dynamics, parameter perturbations, and external disturbances.

[0010] The information fusion and timing alignment module is used to fuse the output system state estimate and total disturbance estimate, and perform timing alignment operation to generate a fused state vector.

[0011] The DDPG policy network and security layer module is used to generate control action vectors by outputting continuous action vectors through the DDPG policy network based on the fused state vectors and applying multi-layer security constraints and disturbance feedforward compensation mechanisms; wherein, the DDPG policy network includes an Actor network and a Critic network.

[0012] The actuator interface module is used to transmit control motion vectors to the actuators of the small robot, control the small robot to perform corresponding actions, and update the status data in real time to form a closed-loop control.

[0013] Optionally, the status data of the small robot includes IMU data, encoder data, and force sensor data.

[0014] Optionally, the ESO observer module, based on the preprocessed state data, uses a state extension model to perform system state estimation and total disturbance estimation, including the following steps:

[0015] S21. The nth-order nonlinear system model describing the dynamics of a small robot is extended to an n+1th-order state extension model, where the newly added n+1th state variable is defined as the total disturbance including unmodeled dynamics, parameter perturbations and external disturbances.

[0016] S22. Based on the n+1 order state expansion model, the gain of the ESO observer is designed using the bandwidth parameterization method, and the gain vector is obtained. , For the gain vector, These are the components of the gain vector. Let represent the transpose of the vector; where the ESO observer poles are configured as follows: In the formula, Let Laplace be a complex variable. Here are the bandwidth parameters for the ESO observer, used to adjust the observation speed, and the gain components satisfy... This forms a continuous-time system state and total disturbance observer;

[0017] S23. Discretize the continuous-time ESO observer using the forward Euler method, converting it into a discrete-time ESO observer executable by a digital controller; use the preprocessed state data as input to drive the discrete-time ESO observer, and estimate and output all system states and total disturbances online in real time.

[0018] Optionally, the process of discretizing the continuous-time ESO observer in S23 includes:

[0019] S231. Based on the state-space equation of the continuous-time ESO observer, a fixed sampling period of the digital controller is set, and the forward Euler method is used to discretize the system matrix and the output matrix to obtain the discrete-time system matrix and the output matrix.

[0020] S232. Based on the discrete-time system matrix and output matrix, and according to the continuous-time gain vector determined by the bandwidth parameter of the ESO observer, the gain matrix of the discrete-time ESO observer is calculated through the discretization mapping relationship.

[0021] S233. Construct a complete discrete-time ESO observer state update equation using the discrete-time system matrix, output matrix, and discrete gain matrix; perform stability verification by calculating the eigenvalues ​​of the system matrix of this state update equation.

[0022] S234. The preprocessed state data is used as input with the sampling period to drive the calibrated discrete-time ESO observer, which iteratively executes the state update equation and outputs the system state estimation sequence and the total disturbance estimation sequence in discrete time.

[0023] Optionally, the information fusion and timing alignment module fuses the output system state estimate and total disturbance estimate, and performs timing alignment to generate the fused state vector. The process includes:

[0024] S31. Low-pass filtering is applied to the real-time output system state estimate and total disturbance estimate to suppress high-frequency observation noise and obtain smoothed system state estimate and total disturbance estimate.

[0025] S32. Based on the smoothed system state estimate and the total disturbance estimate, calculate the observation uncertainty index to quantify the confidence level of the discrete-time ESO observer at the current time.

[0026] S33. Generate a constraint mask vector based on the physical structure and dynamic constraints of the small robot;

[0027] S34. The smoothed system state estimate, the smoothed total disturbance estimate, the observation uncertainty index, and the constraint mask vector are concatenated and integrated to construct a multi-dimensional fused state vector.

[0028] S35. For the fused state vector, the runtime alignment algorithm synchronizes the running frequency of the discrete-time ESO observer and the DDPG policy network through fixed delay line initialization, data caching and interpolation prediction.

[0029] Optionally, the process by which the DDPG policy network and security layer module output continuous action vectors based on the fused state vectors includes:

[0030] S41. Input the fused state vector into the Actor network of the DDPG policy network; wherein the Actor network generates an initial continuous action vector by forward propagation with at least two hidden layers and mapping through the tanh activation function of the output layer.

[0031] S42. Input the fused state vector and the initial continuous action vector together into the Critic network of the DDPG policy network. The Critic network calculates and outputs the corresponding Q value, which is used to evaluate the long-term expected return of the initial action in the current state.

[0032] S43. Using the Q-value, the parameters of the Actor network and the Critic network are optimized and updated through the deep deterministic policy gradient algorithm; at the same time, a soft update mechanism is adopted to slowly update the target network parameters according to the update formula.

[0033] S44. During the training process, a multi-objective reward function is constructed, and the weights of each reward in the multi-objective reward function are dynamically adjusted using the sliding window averaging method based on the obtained observation uncertainty index and the progress of the training phase.

[0034] Optionally, the process of dynamically adjusting the weights of each reward using the sliding window averaging method in S44 includes:

[0035] S441. Set a fixed-length sliding time window to cache the recent observation uncertainty index sequence and the corresponding sub-reward value sequence;

[0036] S442. At each training step, calculate the average value of the observation uncertainty index within the sliding window, and use it as the evaluation benchmark for the current observation quality.

[0037] S443. Dynamically adjust the reward weights of the multi-objective reward function based on the evaluation benchmark;

[0038] S444. Apply the adjusted new weight vector to the calculation of the multi-objective reward function.

[0039] Optionally, the DDPG policy network and security layer module apply multi-layer security constraints and disturbance feedforward compensation mechanisms to generate control action vectors. The process includes:

[0040] S45. For the initial continuous action vector generated by the Actor network, apply multi-layer safety constraints in sequence, including action boundary checks, constraint projection, smoothing, and emergency stop checks, and correct out-of-bounds or abrupt actions and output them as control action vectors.

[0041] S46. Obtain the total disturbance value estimated in real time from the discrete-time ESO observer module, and multiply it by the preset feedforward gain matrix to calculate the feedforward compensation amount.

[0042] S47. Add the control action vector output by the multi-layer safety constraints to the feedforward compensation amount to synthesize the final total control quantity, and output it to the actuator interface module.

[0043] Optionally, S45 may apply multi-layered safety constraints, including motion boundary checks, constraint projection, smoothing, and emergency stop checks, as follows:

[0044] S451. The steps of motion boundary checking include: comparing the initial continuous motion vector with the physical position, speed and torque limits of the actuator to identify the out-of-bounds motion components;

[0045] S452. The steps of constrained projection include: for out-of-bounds motion components, use boundary saturation processing or scaling algorithm to project them back into the feasible motion space to generate the projected motion vector.

[0046] S453. The smoothing process includes: performing a difference calculation on the projected action vector at the current moment and the final output control action vector at the previous moment; if the rate of change exceeds the preset smoothing threshold, a first-order low-pass filter is used to smooth the action vector to suppress sudden action changes.

[0047] S454. The emergency stop check steps include: real-time monitoring of the small robot's attitude angle; if the attitude angle exceeds the safety threshold, the current motion vector is ignored and forcibly replaced with a predefined emergency stop action sequence.

[0048] Optionally, the system also employs a three-stage training strategy, including:

[0049] During the ESO offline tuning phase, the observer parameters are calibrated based on historical data.

[0050] During the joint training phase of ESO and DDPG, policy optimization is performed in a simulation environment;

[0051] In the offline-online combined deployment phase, the trained model is transferred to an actual small robot platform.

[0052] Compared with existing technologies, the beneficial effects of this invention are as follows: By collecting IMU data, encoder data, and force sensor data, and performing low-pass filtering, data denoising, and standardization preprocessing, the data quality is effectively improved, providing an accurate and reliable foundation of state information for subsequent modules; by employing a state expansion model for system state and total disturbance estimation, unmodeled dynamics, parameter perturbations, and external disturbances can be estimated in real time and accurately, enhancing the system's adaptability to complex environments; by fusing and aligning the system state estimation and total disturbance estimation, a timestamped fused state vector is generated, ensuring that the information received by the DDPG policy network is synchronous and accurate, improving the timeliness and accuracy of control decisions; by combining the continuous action vector output by the DDPG policy network and applying multi-layered safety constraints and disturbance feedforward compensation, both the continuity and optimality of actions are guaranteed, as well as the safety of control actions, preventing the system from going out of control due to external disturbances or internal faults; by accurately transmitting safe control action vectors to the actuators, precise control of the small robot is achieved, and real-time feedback of state data forms a closed-loop control, further improving the system's stability and dynamic balance capability. Attached Figure Description

[0053] Figure 1 This is a block diagram of a dynamic balance control system for a small robot based on DDPG proposed in this invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Reference Figure 1 A dynamic balance control system for a small robot based on DDPG is proposed. The system includes a sensor and preprocessing module, an ESO observer module, an information fusion and timing alignment module, a DDPG policy network and security layer module, and an actuator interface module.

[0056] The sensor and preprocessing module is used to collect the state data of the small robot and preprocess the state data; the state data of the small robot includes IMU data, encoder data and force sensor data.

[0057] Specifically, IMU data is acquired by using an inertial measurement unit (which includes a three-axis gyroscope and a three-axis accelerometer) fixed to the torso of a small robot. This data is used to directly measure the robot's three-axis angular velocity and three-axis acceleration in the body coordinate system, which is the core basis for estimating the robot's body attitude (pitch angle, roll angle) and angular velocity.

[0058] Photoelectric or magnetic encoders installed on the motors of each joint are used to collect encoder data. This data is used to measure the rotation angle and angular velocity of each joint, thereby obtaining the limb configuration (such as the swing angle of the legs) and movement speed of the small robot.

[0059] Force sensor data is acquired by force or torque sensors installed on the robot's feet or joints. This data is used to measure the contact force between the feet and the ground or the output torque of the joints. It can sense external interaction forces, determine the support phase and the swing phase, and is the key to achieving a stable gait.

[0060] Preprocessing includes low-pass filtering, data denoising, and data standardization. A low-pass filter is used to process the raw data, with its cutoff frequency set according to the main motion frequency of the small robot (e.g., 1-10 Hz). This aims to filter out high-frequency measurement noise (such as sensor electronic noise and vibration noise) while retaining low-frequency useful signals that reflect the robot's actual motion. A sliding window midpoint filter is used to remove transient pulse interference (outliers) from the data. Simultaneously, by setting reasonable physical ranges (e.g., joint angle limits and angular velocity limits), obviously erroneous data exceeding these ranges are identified and removed. Data from different sensors, with varying dimensions and numerical ranges, are uniformly mapped to a similar scale to accelerate the convergence of subsequent neural network training and improve numerical stability.

[0061] The ESO observer module is used to perform system state estimation and total disturbance estimation based on preprocessed state data and a state extension model. The total disturbance includes unmodeled dynamics, parameter perturbations, and external disturbances.

[0062] The information fusion and timing alignment module is used to fuse the output system state estimate and total disturbance estimate, and perform timing alignment operation to generate a fused state vector.

[0063] The DDPG policy network and security layer module is used to generate control action vectors by outputting continuous action vectors through the DDPG policy network based on the fused state vectors and applying multi-layer security constraints and disturbance feedforward compensation mechanisms; wherein, the DDPG policy network includes an Actor network and a Critic network.

[0064] The actuator interface module is used to transmit control motion vectors to the actuators of the small robot, control the small robot to perform corresponding actions, and update the status data in real time to form a closed-loop control.

[0065] It should be further explained that, in the specific implementation process, the ESO observer module, based on the preprocessed state data, uses a state extension model to perform system state estimation and total disturbance estimation, which includes:

[0066] S21. Extend the nth-order nonlinear system model describing the dynamics of a small robot to an n+1th-order state extension model, where the newly added n+1th state variable is defined as the total disturbance including unmodeled dynamics, parameter perturbations, and external disturbances; n is the original order of the nonlinear system model and is a positive integer.

[0067] S22. Based on the n+1 order state expansion model, the gain of the ESO observer is designed using the bandwidth parameterization method, and the gain vector is obtained. , For the gain vector, These are the components of the gain vector. Let represent the transpose of the vector; where the ESO observer poles are configured as follows: In the formula, Let Laplace be a complex variable. Here are the bandwidth parameters for the ESO observer, used to adjust the observation speed, and the gain components satisfy... To construct a continuous-time system state and total disturbance observer, where C(n,2) is a binomial coefficient, representing the number of combinations of choosing 2 from n elements, and its calculation formula is C(n,2)=n(n-1) / 2;

[0068] S23. Discretize the continuous-time ESO observer using the forward Euler method, converting it into a discrete-time ESO observer executable by a digital controller; use the preprocessed state data as input to drive the discrete-time ESO observer, and estimate and output all system states and total disturbances online in real time.

[0069] In S23, the process of discretizing the continuous-time ESO observer includes:

[0070] S231. Based on the state-space equation of the continuous-time ESO observer, set the fixed sampling period of the digital controller. The forward Euler method is used to discretize the system matrix A and the output matrix D to obtain the discrete-time system matrix. With output matrix The state-space equation of the continuous-time ESO observer is directly derived from the established n+1 order state extension model. This model treats the total disturbance as an extended state, and its dynamic characteristics are usually modeled as first-order or higher-order differential forms. Based on this model, by combining the original state of the system with the extended state (total disturbance) to form a new state vector, and according to the input-output relationship of the system, a standard continuous-time state-space equation expression can be established, which provides a mathematical model basis for subsequent discretization and observer gain design.

[0071] S232, System Matrix Based on Discrete Time With output matrix And based on the bandwidth parameters of the ESO observer The determined continuous-time gain vector By discretizing the mapping relationship, the gain matrix of the discrete-time ESO observer is calculated. ;

[0072] S233, Utilizing Discrete-Time System Matrices Output matrix and discrete gain matrix Construct the complete discrete-time ESO observer state update equation; calculate the system matrix of this state update equation. The eigenvalues ​​are used to perform stability checks to ensure that the magnitudes of all eigenvalues ​​are less than 1, i.e., they are located within the unit circle in the Z-domain. In discrete system analysis, the unit circle in the Z-domain is the stability boundary. If all eigenvalues ​​of the discrete system matrix are located inside the unit circle (i.e., their magnitudes are all less than 1), it indicates that the discrete system is stable and its state response will not diverge. This stability check ensures that the designed discrete-time ESO observer can operate stably and reliably on the digital controller.

[0073] S234. The preprocessed state data is sampled at a period of... As input, it drives a validated discrete-time ESO observer to iteratively execute the state update equation and outputs a discrete-time system state estimation sequence and a total disturbance estimation sequence.

[0074] It should be further explained that, in the specific implementation process, the information fusion and timing alignment module fuses the output system state estimate and total disturbance estimate, and performs timing alignment operations to generate the fused state vector. The process includes:

[0075] S31. Low-pass filtering is applied to the real-time output system state estimate and total disturbance estimate to suppress high-frequency observation noise and obtain smoothed system state estimate and total disturbance estimate. The low-pass filtering process includes: S311. First-order Butterworth low-pass filters are selected according to the different dynamic characteristics of the system state estimate and total disturbance estimate. The cutoff frequency fc is set according to the dominant frequency fd of the small robot's body dynamics. The specific relationship is fc=k×fd, where k is a scaling factor with a value range of 1.5 to 3.0 to ensure that noise introduced by sensor noise and high-frequency fluctuations of the observer is effectively filtered out without causing significant phase lag that would affect the real-time performance of the control.

[0076] S312. When the system starts up, the filter state is initialized and the initial output is set to the first input observation value to avoid startup transient impact; after each discrete-time ESO observer outputs a new estimate, it is input to the corresponding system state estimation filter and total disturbance estimation filter respectively.

[0077] S313. After the filter is completed, the smoothed system state estimate and total disturbance estimate are output. At the same time, the rate of change of the signal before and after filtering is calculated in real time. If the rate of change of the signal before and after filtering exceeds the preset change threshold, it is determined that abnormal filtering or data jump may occur, and a flag bit is generated for subsequent quality assessment.

[0078] S32. Based on the smoothed system state estimate and the total disturbance estimate, calculate the observation uncertainty index to quantify the estimation confidence of the discrete-time ESO observer at the current moment. The calculation process of the observation uncertainty index includes: S321. In each control cycle, compare the smoothed system state estimate (especially directly measurable states, such as joint angles, fuselage angular velocity, etc.) output by the discrete-time ESO observer with the preprocessed corresponding state measurement values ​​output by the sensor and preprocessing module, calculate the difference, and form the real-time residual.

[0079] S322. Set a sliding time window of length N to cache the real-time residual data of the most recent N time steps; calculate the variance of the residual sequence within the sliding window. The formula for calculating variance is: ,in For the first in the sliding window The residual value at each time point, where μ is the average value of the residuals within the window;

[0080] S323, calculate the variance The uncertainty index U is converted into an observational index through a pre-defined mapping function; this mapping function is set to be a monotonically increasing function, for example... ,in The variance threshold is preset based on the system noise level and performance requirements. When the variance is less than the preset variance threshold, the observation uncertainty index U is approximately linearly related to the variance. When the variance exceeds the preset variance threshold, the observation uncertainty index U is saturated and limited to 1.0, indicating that the observation confidence is extremely low. The range of the observation uncertainty index U is [0,1]. The larger the value, the higher the observation uncertainty and the lower the confidence.

[0081] S33. Based on the physical structure and dynamic constraints of the small robot, generate a constraint mask vector to identify the feasible operational range of each state variable in subsequent control; wherein, the process of generating the constraint mask vector includes: S331. In advance, establish a constraint rule base according to the specific configuration of the small robot (such as bipedal, quadrupedal, wheeled legged) and the physical performance of its actuators; the rule base includes: upper and lower limits of joint angles, limits of joint angular velocity, safety boundaries of body pitch angle and roll angle, and maximum output torque of actuators;

[0082] S332. In each control cycle, read the smoothed system state estimate, including joint angles, angular velocities, fuselage attitude angles, etc.; compare each state variable with the corresponding safety boundary in the rule base.

[0083] S333. Based on the comparison results, generate a constraint mask vector with the same dimension as the system state vector. The constraint mask vector consists of 0 and 1: if a state variable is within its safety boundary, the value at the corresponding position in the mask vector is 1, indicating that the state is controllable and can be used for policy network decision-making; if a state variable is close to or exceeds its safety boundary (for example, reaching 95% of the boundary), the value at the corresponding position is set to 0, indicating that the state is close to the danger zone and should be masked or given a very high penalty weight in subsequent control decisions to avoid the control action from pushing it further to the limit.

[0084] Understandably, this constraint mask is not fixed and can be dynamically loaded according to the current task mode of the small robot (such as walking, running, standing) to achieve adaptive safety constraints.

[0085] S34. The smoothed system state estimate, the smoothed total disturbance estimate, the observation uncertainty index, and the constraint mask vector are concatenated and integrated to construct a multi-dimensional fused state vector.

[0086] S35. For the fused state vector, a runtime alignment algorithm is used. Through fixed delay line initialization, data caching, and interpolation prediction, the operating frequencies of the discrete-time ESO observer and the DDPG policy network are synchronized to ensure that the fused state vector finally output to the DDPG policy network is strictly aligned with the control cycle of the DDPG policy network in terms of timestamp. The specific process of the timing alignment algorithm includes: S351. Based on the single calculation processing time of the discrete-time ESO observer and the control cycle of the DDPG policy network, a certain data transmission delay is calculated. Based on this data transmission delay, a first-in-first-out data buffer with a fixed length is initialized. This buffer is used to temporarily store the timestamped fused state vector output by the ESO observer module.

[0087] S352. At the end of the running cycle of each ESO observer module, the currently generated fusion state vector is bound to the system high-precision timestamp corresponding to the generation of the vector to form a timestamped data unit, and the data unit is pushed into the end of the first-in-first-out data buffer for storage.

[0088] S353. When the DDPG policy network reaches its predetermined control period, the high-precision timestamp of the current moment is obtained; the two historical fusion state vector data units that are closest in time to the current DDPG control moment are retrieved and extracted from the data buffer; using the timestamps of these two historical data units and their corresponding fusion state vector values ​​as known data points, a new fusion state vector that is completely aligned with the current control moment of the DDPG policy network is calculated and derived using a linear interpolation algorithm.

[0089] S354. Check the timestamp span of the two historical fused state vector data units used for interpolation calculation and calculate their time difference. If the time difference is less than the preset maximum allowable interpolation time interval threshold, the data quality of this interpolation prediction operation is determined to be reliable, and the new fused state vector obtained through interpolation is output to the DDPG policy network as its decision basis. If the time difference exceeds the preset maximum allowable interpolation time interval threshold, it is determined that the effective data has been lost or the delay is too high, the interpolation quality is unreliable, the interpolation result is discarded, and the data loss security handling mechanism is triggered. At the same time, the value of the observation uncertainty index is actively increased to warn the downstream module that the current system's perception reliability has decreased.

[0090] It should be further explained that, in the specific implementation process, the DDPG policy network and the security layer module are based on the fused state vector. The process of generating control action vectors by outputting continuous action vectors through the DDPG policy network and applying multi-layer security constraints and disturbance feedforward compensation mechanisms includes:

[0091] S41. The fused state vector is input into the Actor network of the DDPG policy network; wherein, the Actor network generates an initial continuous action vector through forward propagation containing at least two hidden layers and mapping by the tanh activation function of the output layer; specifically, S411, the fused state vector is fed as the data of the input layer into the first hidden layer of the Actor network, which contains 512 neurons and uses the ReLU activation function for nonlinear transformation to initially extract state features;

[0092] S412. The output of the first hidden layer is passed to the second hidden layer, which contains 256 neurons and also uses the ReLU activation function to perform a higher level of abstraction of state features, understand the complex mapping relationship between state and action, and gradually form a control strategy.

[0093] S413. The output of the second hidden layer is finally passed to the output layer. The number of neurons in the output layer strictly corresponds to the control dimension of the small robot actuator. This layer uses the hyperbolic tangent function as the activation function to normalize and restrict the output value of each neuron to the continuous interval [-1, 1], thereby directly generating a normalized initial continuous motion vector that matches the physical interface of the actuator. Each component in this vector corresponds to a control command, such as the target torque or speed of the joint motor.

[0094] S42. The fused state vector and the initial continuous action vector are input together into the Critic network of the DDPG policy network. The Critic network calculates and outputs the corresponding Q value, which is used to evaluate the long-term expected return of the initial action in the current state. The core of the long-term expected return is to guide the policy to learn how to maintain dynamic equilibrium rather than just pursuing instantaneous stability. Specifically, S421. The Critic network receives two inputs at the same time: the fused state vector from the information fusion and temporal alignment module and the initial continuous action vector generated by the Actor network. The first layer of the network is responsible for concatenating these two vectors to form an extended-dimensional state-action joint feature vector, which serves as the basis for evaluation.

[0095] S422. The concatenated joint feature vector is fed into the hidden layer of the Critic network (e.g., using a two-layer structure with 512 and 256 neurons, and the ReLU activation function). The hidden layer is responsible for learning the complex features of the combination of state and action, and evaluating the immediate effect and potential long-term impact of the action in the current state.

[0096] S423. After the nonlinear transformation of the hidden layer, the features are passed to the output layer of the Critic network. The output layer contains only one neuron and uses a linear activation function to output a real value, namely the Q value. The Q value represents the expected estimate of the future cumulative discount reward that can be obtained by performing the initial action in the current fusion state and then following the current optimal policy. Its core value is to provide direction for policy optimization: a higher Q value indicates that the action is not only effective at present, but also helps the small robot maintain dynamic equilibrium and stability in the long term. It is a key signal to guide policy learning toward the optimal.

[0097] S43. Using the Q-value, the parameters of the Actor network and Critic network are optimized and updated using a deep deterministic policy gradient algorithm. Simultaneously, a soft update mechanism is employed to slowly update the target network parameters according to the update formula. Specifically, S431. The Q-value calculated by the current Critic network, along with the actual reward obtained through environmental interaction and the Q-value of the next state calculated by the target Critic network, constitute the temporal difference error target. The mean squared error loss function is used to calculate the difference between the predicted value and the target value of the current Critic network. The stochastic gradient descent algorithm is used to minimize this loss and update the parameters of the Critic network to improve the accuracy of its value assessment.

[0098] S432. Using the updated Critic network, calculate the gradient of the Q-value of the Actor network's output action relative to that action; according to the deterministic policy gradient theorem, propagate the gradient back to the Actor network along the direction that increases the Q-value using the chain rule, and update its parameters; the purpose of this step is to adjust the policy so that when the Actor network encounters similar states in the future, it can output actions that can obtain higher Q-values ​​(i.e., better long-term returns).

[0099] S433. After completing the parameter update of the main network, perform a soft update operation on the target network; update the parameter sets of the Actor main network respectively. The parameter set of the Critic main network According to the updated formula and It is slowly updated to the corresponding target network;

[0100] In the formula, This represents the updated set of parameters for the Actor target network. This represents the updated set of parameters for the Critic target network. This represents the set of Actor target network parameters before the update; This represents the set of Critic target network parameters before the update. The soft update coefficient is a positive number much smaller than 1. The purpose of this mechanism is to ensure that the target Q value changes slowly, thereby greatly improving the stability of the entire reinforcement learning training process and avoiding oscillations and divergences in value estimation.

[0101] S44. During training, a multi-objective reward function R is constructed. Based on the obtained observation uncertainty index and the progress of the training phase, the weights of each reward in the multi-objective reward function are dynamically adjusted using the sliding window averaging method to adaptively guide policy learning. The expression of the multi-objective reward function is R = wt × Rtra + wd × Rdis + ws × Rsta + wo × Robs. In the formula, Rtra is the attitude tracking reward, which is negatively correlated with the error between the actual attitude of the small robot and the target attitude (i.e., the negative value of the Euclidean norm between the actual attitude vector of the small robot and the target attitude vector); Rdis is the disturbance resistance reward, which is negatively correlated with the total disturbance amplitude estimated by the ESO observer (i.e., the negative value of the L2 norm of the total disturbance value estimated by the ESO observer module); Rsta is the stability reward, which gives a large negative reward when the small robot is close to overturning (returns 0 when the small robot's attitude angle does not exceed the overturning critical angle, and returns a large negative reward value when it exceeds it); Robs is the observation quality reward, which is negatively correlated with the observation uncertainty index; wt, wd, ws, and wo are the reward weights.

[0102] In S44, the process of dynamically adjusting the weights of each reward using the sliding window averaging method includes:

[0103] S441. Set a fixed-length sliding time window to cache the recent observation uncertainty index sequence and the corresponding sub-reward value sequence; wherein, the observation uncertainty index sequence is generated by the information fusion and time series alignment module at each step, and the sub-reward value sequence is calculated by the small robot after interacting with the environment according to the reward function definition;

[0104] S442. At each training step, calculate the average value of the observation uncertainty index within the sliding window, and use it as the evaluation benchmark for the current observation quality.

[0105] S443. Dynamically adjust the reward weights of the multi-objective reward function according to the evaluation benchmark: when the average value is higher than the high confidence threshold, the observation quality is judged to be poor, the weight wt of the attitude tracking reward Rtra is reduced, and the weight ws of the stability reward Rsta is increased; when the average value is lower than the low confidence threshold, the opposite operation is performed; the high confidence threshold and the low confidence threshold are pre-calibrated in the system before training through a large number of simulation experiments.

[0106] S444. Apply the adjusted new weight vector to the calculation of the multi-objective reward function to guide the DDPG policy network to perform more targeted policy optimization under the current observation quality.

[0107] S45. For the initial continuous action vector generated by the Actor network, apply multi-layer safety constraints in sequence, including action boundary checks, constraint projection, smoothing, and emergency stop checks, and correct out-of-bounds or abrupt actions and output them as control action vectors.

[0108] In S45, the application of multi-layered safety constraints, including motion boundary checks, constraint projection, smoothing, and emergency stop checks, specifically includes:

[0109] S451. The steps of motion boundary checking include: comparing the initial continuous motion vector with the physical position, velocity and torque limit of the actuator to identify the out-of-bounds motion components; wherein, the torque limit is determined based on the actuator's product data sheet and the dynamics analysis of the small robot.

[0110] S452. The steps of constrained projection include: for out-of-bounds motion components, use boundary saturation processing or scaling algorithm to project them back into the feasible motion space to generate the projected motion vector; wherein, boundary saturation processing directly sets the out-of-bounds components as boundary values, and scaling algorithm proportionally shrinks the entire motion vector until all components do not exceed the boundaries.

[0111] S453. The smoothing process includes: performing a difference calculation on the projected motion vector at the current moment and the final output control motion vector at the previous moment; if the rate of change exceeds a preset smoothing threshold, a first-order low-pass filter is used to smooth the motion vector to suppress sudden changes in motion; wherein, the rate of change is the Euclidean norm of the difference between the components of the two motion vectors; the smoothing threshold is set based on the maximum acceleration capability of the actuator.

[0112] S454. The emergency stop check steps include: real-time monitoring of the small robot's attitude angles (including pitch and roll angles). If the attitude angles exceed the safety threshold, the current motion vector is ignored and forcibly replaced with a predefined emergency stop action sequence. The safety threshold is set according to the static stability angle of the small robot's mechanical structure. The emergency stop action sequence is a series of preset action commands that make all joint motors output zero torque or drive the small robot to slowly retract to a safe static configuration.

[0113] S46. Obtain the real-time estimated total disturbance value from the discrete-time ESO observer module. And multiply by a preset feedforward gain matrix. The feedforward compensation amount is calculated. The feedforward gain matrix is ​​identified through a system dynamics model or pre-set using a controller loop shaping method.

[0114] S47. The control action vector output by the multi-layer safety constraints With feedforward compensation Add them together to synthesize the final total control quantity. (in ), and output to the actuator interface module.

[0115] It should be further explained that, in the specific implementation process, the system also adopts a three-stage training strategy, including:

[0116] During the ESO offline tuning phase, the observer parameters are calibrated based on historical data.

[0117] The goal of this phase is to obtain a high-performance, stable discrete-time ESO observer before the actual deployment of the DDPG strategy. The specific process is as follows: H1. In the simulation environment or using an actual small robot platform, apply a series of wideband excitation control signals (such as sweep frequency signals, pseudo-random binary sequences), and simultaneously record the preprocessed sensor data (system output y) and the corresponding control commands (system input u) to form a dataset. ,in For continuous time variables;

[0118] H2, Based on collected data A preliminary n-order nonlinear model describing the dynamics of a small robot is fitted using a system identification algorithm (such as the least squares method). Based on this model, an n+1-order state extension model is constructed, and its discrete-time form (i.e., a discrete-time ESO observer) is preliminarily designed according to the sampling period of the digital controller. The initial bandwidth parameter of the observer is... The initial settings are based on the desired observation speed;

[0119] H3. With the goal of minimizing observation error, the key parameters of ESO (mainly bandwidth parameters) are analyzed. Optimize the dataset: The input is fed into a discrete-time ESO observer to calculate the root mean square error between the estimated system state and the true state (or high-precision measurement); the system is then automatically adjusted using optimization algorithms (such as gradient descent or Bayesian optimization). This process continues until the observation difference of the discrete-time ESO observer converges to a minimum. This process ensures that the discrete-time ESO observer can achieve optimal estimation accuracy and speed when dealing with the intrinsic dynamics of the system.

[0120] H4. Run the tuned discrete-time ESO observer in various untrained test scenarios to verify the accuracy of its state and perturbation estimation and its own numerical stability, ensuring that it can be used as a reliable state perceiver for subsequent stages.

[0121] During the joint training phase of ESO and DDPG, policy optimization is performed in a simulation environment;

[0122] This stage is conducted in a simulation environment and aims to train the DDPG policy network to learn complex balance control policies using a tuned discrete-time ESO observer. The specific process is as follows: V1. Construct a high-fidelity physical simulation environment for a small robot to simulate the robot's dynamics, sensor noise, and various complex terrains (such as flat ground, slopes, and irregular road surfaces) and external disturbances (such as lateral thrust).

[0123] V2. Randomly initialize the Actor and Critic network parameters of DDPG; integrate the fixed parameters of the discrete-time ESO observer tuned in the first stage with the DDPG policy network and simulation environment into a closed-loop training framework;

[0124] V3a, the intelligent agent (small robot) interacts in the simulation environment according to the current strategy (with exploration noise); at each discrete simulation time step, the discrete-time ESO observer receives preprocessed simulation sensor data and outputs the discrete-time system state estimate and total disturbance estimate; the information fusion module processes them into a fused state vector. , For discrete time steps; Actor network is based on Output Action After the environment performs this action, it transitions to a new state. And generate rewards ; experience Store in the experience replay pool;

[0125] V3b randomly samples a small batch of empirical data from the experience replay pool; the Critic network updates its parameters based on the temporal difference error to more accurately evaluate the long-term value of state-action pairs; the Actor network updates its parameters along the policy gradient direction provided by the Critic network to output actions that can obtain higher expected rewards.

[0126] V3c, Target Network Soft Update: Slowly update the target network parameters to stabilize the training process;

[0127] V4. During training, the weights of each item in the multi-objective reward function are dynamically adjusted based on the observation uncertainty index provided by the ESO observer. For example, when the observation uncertainty is high, the system will automatically reduce the requirements for high-precision tracking and instead focus more on maintaining basic stability.

[0128] V5. Periodically evaluate the trained policy in a fixed set of test scenarios. When the performance metrics (such as average survival time and average reward) no longer improve significantly and stabilize at a high level, the training is considered to have converged.

[0129] In the offline-online combined deployment phase, the trained model is transferred to the actual small robot platform;

[0130] After the offline-online combined deployment phase, the system enters an online adaptive fine-tuning mode. In this mode, the system continuously calculates the absolute error between the actual attitude angle and the target attitude angle of the small robot at a fixed control cycle as the actual attitude error, and obtains the observation uncertainty index in real time. When the actual attitude error exceeds 50% of the overturning angle safety threshold determined based on the physical structure of the small robot, or when the observation uncertainty index exceeds half of the difference between the maximum and minimum values ​​obtained in the joint training phase, the system automatically triggers a gradient update with a small learning rate for the bias term of the output layer of the Critic network in the DDPG policy network, while keeping the parameters of the Actor network and other networks frozen.

[0131] It should be understood that determining B based on A does not mean determining B solely based on A; it also means determining B based on A and / or other information.

[0132] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0133] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dynamic balance control system for a small robot based on DDPG, characterized in that: It includes a sensor and preprocessing module, an ESO observer module, an information fusion and timing alignment module, a DDPG policy network and security layer module, and an actuator interface module; Sensor and preprocessing module: Collects the status data of the small robot and preprocesses the status data; ESO Observer Module: Based on the preprocessed state data, the system state estimation and total disturbance estimation are performed using a state extension model. The ESO observer uses a state extension model that treats the total disturbance as an extended state to observe the total disturbance, which includes unmodeled dynamics, parameter perturbations, and external disturbances, and outputs the system state estimate and the total disturbance estimate. The information fusion and timing alignment module fuses the output system state estimate and total disturbance estimate, performs timing alignment, and generates a fused state vector. The fusion process includes low-pass filtering the system state estimate and total disturbance estimate to obtain smooth values, calculating the observation uncertainty index, generating a constraint mask vector, and then concatenating them into a fused state vector. DDPG Policy Network and Security Layer Module: Based on the fused state vector, the DDPG policy network outputs continuous action vectors and applies multi-layer security constraints and disturbance feedforward compensation mechanisms to generate control action vectors; among which, the multi-layer security constraints include action boundary checks, constraint projection, smoothing processing, and emergency stop checks. Actuator Interface Module: Transmits control motion vectors to the actuators of the miniature robot, controls the miniature robot to perform corresponding actions, and updates status data in real time.

2. The dynamic balance control system for a small robot based on DDPG according to claim 1, characterized in that, The status data of a small robot includes IMU data, encoder data, and force sensor data.

3. The dynamic balance control system for a small robot based on DDPG according to claim 1, characterized in that, The ESO observer module performs system state estimation and total disturbance estimation using a state extension model based on preprocessed state data. The process includes: S21. The nth-order nonlinear system model describing the dynamics of a small robot is extended to an n+1th-order state extension model, where the newly added n+1th state variable is defined as the total disturbance including unmodeled dynamics, parameter perturbations and external disturbances. S22. Based on the (n+1)th order state expansion model, the gain of the ESO observer is designed using the bandwidth parameterization method, and the gain vector L = [l1, l2, ..., l n ] T L is the gain vector, l is the component of the gain vector, and T represents the transpose of the vector; where the poles of the ESO observer are configured as s = -ω0, -2ω0, ..., -nω0, where s is a Laplace complex variable, ω0 is the bandwidth parameter of the ESO observer used to adjust the observation speed, and the gain components satisfy l1 = nω0. …, To form a continuous-time system state and total disturbance observer; S23. Discretize the continuous-time ESO observer using the forward Euler method, converting it into a discrete-time ESO observer executable by a digital controller; use the preprocessed state data as input to drive the discrete-time ESO observer, and estimate and output all system states and total disturbances online in real time.

4. The dynamic balance control system for a small robot based on DDPG according to claim 3, characterized in that, The process of discretizing the continuous-time ESO observer in S23 includes: S231. Based on the state-space equation of the continuous-time ESO observer, a fixed sampling period of the digital controller is set, and the forward Euler method is used to discretize the system matrix and the output matrix to obtain the discrete-time system matrix and the output matrix. S232. Based on the discrete-time system matrix and output matrix, and according to the continuous-time gain vector determined by the bandwidth parameter of the ESO observer, the gain matrix of the discrete-time ESO observer is calculated through the discretization mapping relationship. S233. Construct a complete discrete-time ESO observer state update equation using the discrete-time system matrix, output matrix, and discrete gain matrix; perform stability verification by calculating the eigenvalues ​​of the system matrix of this state update equation. S234. The preprocessed state data is used as input with the sampling period to drive the calibrated discrete-time ESO observer, which iteratively executes the state update equation and outputs the system state estimation sequence and the total disturbance estimation sequence in discrete time.

5. A dynamic balance control system for a small robot based on DDPG according to claim 1, characterized in that, The information fusion and timing alignment module fuses the output system state estimate and total disturbance estimate, and performs timing alignment to generate the fused state vector. The process includes: S31. Perform low-pass filtering on the real-time output system state estimate and total disturbance estimate to obtain smoothed system state estimate and total disturbance estimate. S32. Based on the smoothed system state estimate and the total disturbance estimate, calculate the observation uncertainty index to quantify the confidence level of the discrete-time ESO observer at the current time. S33. Generate a constraint mask vector based on the physical structure and dynamic constraints of the small robot; S34. The smoothed system state estimate, the smoothed total disturbance estimate, the observation uncertainty index, and the constraint mask vector are concatenated and integrated to construct a multi-dimensional fused state vector. S35. For the fused state vector, the runtime alignment algorithm synchronizes the running frequency of the discrete-time ESO observer and the DDPG policy network through fixed delay line initialization, data caching and interpolation prediction.

6. A dynamic balance control system for a small robot based on DDPG according to claim 1, characterized in that, The DDPG policy network and security layer module are based on fused state vectors. The process of outputting continuous action vectors through the DDPG policy network includes: S41. Input the fused state vector into the Actor network of the DDPG policy network; wherein the Actor network generates an initial continuous action vector by forward propagation with at least two hidden layers and mapping through the tanh activation function of the output layer. S42. Input the fused state vector and the initial continuous action vector together into the Critic network of the DDPG policy network. The Critic network calculates and outputs the corresponding Q value, which is used to evaluate the long-term expected return of the initial action in the current state. S43. Using the Q-value, the parameters of the Actor network and the Critic network are optimized and updated through the deep deterministic policy gradient algorithm; at the same time, a soft update mechanism is adopted to slowly update the target network parameters according to the update formula. S44. During the training process, a multi-objective reward function is constructed, and the weights of each reward in the multi-objective reward function are dynamically adjusted using the sliding window averaging method based on the obtained observation uncertainty index and the progress of the training phase.

7. A dynamic balance control system for a small robot based on DDPG according to claim 6, characterized in that, The process of dynamically adjusting the weights of each reward using the sliding window averaging method in S44 includes: S441. Set a fixed-length sliding time window to cache the recent observation uncertainty index sequence and the corresponding sub-reward value sequence; S442. At each training step, calculate the average value of the observation uncertainty index within the sliding window, and use it as the evaluation benchmark for the current observation quality. S443. Dynamically adjust the reward weights of the multi-objective reward function based on the evaluation benchmark; S444. Apply the adjusted new weight vector to the calculation of the multi-objective reward function.

8. A dynamic balance control system for a small robot based on DDPG according to claim 1, characterized in that, The process by which the DDPG policy network and security layer module apply multi-layered security constraints and disturbance feedforward compensation mechanisms to generate control action vectors includes: S45. For the initial continuous action vector generated by the Actor network, apply multi-layer safety constraints in sequence, including action boundary checks, constraint projection, smoothing, and emergency stop checks, and correct out-of-bounds or abrupt actions and output them as control action vectors. S46. Obtain the total disturbance value estimated in real time from the discrete-time ESO observer module, and multiply it by the preset feedforward gain matrix to calculate the feedforward compensation amount. S47. Add the control action vector output by the multi-layer safety constraints to the feedforward compensation amount to synthesize the final total control quantity, and output it to the actuator interface module.

9. A dynamic balance control system for a small robot based on DDPG according to claim 8, characterized in that, The S45 imposes multiple layers of safety constraints, including motion boundary checks, constraint projection, smoothing, and emergency stop checks. S451. The steps of motion boundary checking include: comparing the initial continuous motion vector with the physical position, speed and torque limits of the actuator to identify the out-of-bounds motion components; S452. The steps of constrained projection include: for out-of-bounds motion components, use boundary saturation processing or scaling algorithm to project them back into the feasible motion space to generate the projected motion vector. S453. The smoothing process includes: performing a difference calculation on the projected action vector at the current moment and the final output control action vector at the previous moment; if the rate of change exceeds the preset smoothing threshold, a first-order low-pass filter is used to smooth the action vector. S454. The emergency stop check steps include: real-time monitoring of the small robot's attitude angle; if the attitude angle exceeds the safety threshold, the current motion vector is ignored and forcibly replaced with a predefined emergency stop action sequence.

10. A dynamic balance control system for a small robot based on DDPG according to claim 1, characterized in that, The system also employs a three-stage training strategy, including: During the ESO offline tuning phase, the observer parameters are calibrated based on historical data. During the joint training phase of ESO and DDPG, policy optimization is performed in a simulation environment; In the offline-online combined deployment phase, the trained model is transferred to an actual small robot platform.

Images