A robot motion control method and system based on cerebellum reinforcement learning
By constructing a dual-pathway architecture that combines the main strategy channel and the cerebellar compensation channel, along with a multimodal prediction error learning mechanism, the adaptability and learning efficiency issues of existing robot motion control methods in dynamic environments are solved, enabling the robot to respond quickly to environmental changes and continuously optimize.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SINARD DIGITAL TECH (SHANGHAI) CO LTD
- Filing Date
- 2026-03-05
- Publication Date
- 2026-05-22
AI Technical Summary
Existing robot motion control methods suffer from problems such as rigid environmental adaptability, low relearning efficiency, contradiction between model accuracy and adaptability, and underutilization of bio-inspired potential when migrating from simulation to the real world. They are particularly difficult to cope with changes in dynamic environments.
A dual-pathway architecture is constructed, with the main strategy channel and the cerebellar compensation channel running in parallel. The cerebellar compensation channel provides real-time fine-tuning, and the prediction error learning mechanism combined with multimodal sensor information enables rapid self-calibration. Furthermore, efficient learning is achieved through a hierarchical learning system with dual loops.
This enables robots to respond quickly to sudden changes in the environment, improves the stability and efficiency of the learning process, ensures the macroscopic optimality and rapid adaptability of the strategy, and achieves continuous optimization in the robot's motion control process.
Smart Images
Figure CN121756368B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot control technology, and in particular to a robot motion control method and system based on cerebellar reinforcement learning. Background Technology
[0002] In the fields of robot motion control and adaptive learning, deep reinforcement learning technology has become the mainstream paradigm for solving complex skill learning problems. Among them, model-free reinforcement learning methods, such as proximal policy optimization algorithms, have achieved significant success in robot simulation learning because they do not require the pre-establishment of an accurate environmental dynamics model and can directly handle high-dimensional state inputs from sensors. They are widely used in training complex skills such as bipedal walking, quadrupedal running and jumping, and robotic arm grasping. The industry generally follows a "simulation first, deployment later" process, that is, training a high-performance policy network in a physical simulation environment, and then directly deploying the trained fixed model to the physical robot for operation.
[0003] However, the aforementioned existing technical approaches reveal several fundamental limitations in practical applications, especially during the migration from simulation to the real world and long-term operation: First, their environmental adaptability is rigid and insufficient. Once training is complete, the parameters of the policy network are fixed, making it difficult to effectively cope with dynamic uncertainties such as continuous or sudden changes in ground material, external interference, or sudden changes in its own load in the real world. Second, relearning and adaptation are inefficient. When environmental changes exceed the policy's generalization ability, traditional solutions often require the re-collection of large amounts of data and fine-tuning from scratch or the entire network. This process is time-consuming, labor-intensive, and inefficient in data utilization. Third, there is a contradiction between model accuracy and adaptability. Model-based methods are limited by the accuracy of the model itself, and model errors are unavoidable in dynamic environments. Pure model-free methods, while avoiding model dependence, have a slow learning process and lack the ability to quickly adapt to real-time changes. Fourth, the potential of bio-inspiration has not been fully realized. Existing solutions mostly use the cerebellum model as a static or predefined compensator, only utilizing its feedforward control capabilities, without deeply integrating and leveraging its core error-driven, online, and rapid learning biological learning characteristics. Summary of the Invention
[0004] To address the aforementioned technical problems, this invention provides a robot motion control method and system based on cerebellar reinforcement learning, comprising:
[0005] Obtain the current environment state vector and input it in parallel into the main policy channel and the cerebellum compensation channel;
[0006] The cerebellar compensation channel queries through an internal lookup table mechanism, outputs compensation action vectors, and the main strategy channel calculates through forward propagation to output basic action vectors.
[0007] The basic motion vector and the compensated motion vector are combined to obtain the composite motion vector, and the motion commands for robot movement are generated based on the composite motion vector.
[0008] After the robot executes motion vectors based on motion commands, it returns a reward signal and determines the real environmental state vector of the new environment in which the robot is located, as well as the state transition data generated by the robot's interaction with the new environment.
[0009] Based on the current environmental state vector and the synthesized action vector, a prediction is made to obtain the environmental prediction state vector, and the sensory prediction error is determined based on the environmental prediction state vector and the real environmental state vector.
[0010] The original parameters of the cerebellar compensation channel are updated based on sensory prediction error, and the original policy parameters of the main policy channel are optimized based on reward signals and state transition data.
[0011] Furthermore, the cerebellar compensation channel performs queries through an internal lookup table mechanism to output compensation action vectors, including:
[0012] The cerebellar compensation channel receives the current environmental state vector and performs linear scaling and discretization quantization on each dimension of the current environmental state vector, mapping the current environmental state vector from a continuous space to a discrete index space.
[0013] Multiple overlapping quantization grids are used to cover the discretized quantization state. For each grid, the quantization state is converted into an integer coordinate vector by adding a preset offset and rounding down.
[0014] The integer coordinate vector corresponding to each quantized grid is calculated using a preset hash function to obtain the calculation results. The calculation results are then mapped from a preset weight table to obtain one or more unique memory address indices.
[0015] Read the weight values corresponding to the activated memory address indices, and perform aggregation operations on these weight values to obtain the output compensation action vector.
[0016] Furthermore, the basic action vector, calculated by the main policy channel through forward propagation, includes:
[0017] Determine the current environment state vector and perform standardization preprocessing on the current environment state vector;
[0018] The preprocessed current environment state vector is input into a fully connected hidden layer network, and the input current environment state vector is linearly weighted and summed layer by layer based on each hidden layer of the fully connected hidden layer network;
[0019] After the output of the last hidden layer is passed to the output layer of the fully connected hidden layer network, the output layer calculates the basic action vector for each action vector dimension through linear transformation and a preset nonlinear mapping function.
[0020] Furthermore, the process of synthesizing the base motion vector and the compensated motion vector to obtain a synthesized motion vector, and generating motion commands for robot movement based on the synthesized motion vector, includes:
[0021] The basic motion vector and the compensated motion vector are added together to obtain the synthetic motion vector, and motion commands for robot movement are generated based on the synthetic motion vector.
[0022] Furthermore, the step of controlling the robot to execute motion vectors based on action commands and returning a reward signal, and determining the real environmental state vector of the new environment in which the robot is located and the state transition data generated by the robot's interaction with the new environment, includes:
[0023] After the robot performs a movement action based on the motion command, the robot moves to a new environment and returns a reward signal;
[0024] Determine the true environmental state vector of the robot in the new environment, and determine the state transition data generated by the interaction between the robot and the new environment;
[0025] Interaction data is constructed based on the returned reward signals and state transition data, and the interaction data is stored in the experience buffer of the main policy channel.
[0026] Furthermore, the step of predicting based on the current environmental state vector and the synthesized action vector to obtain an environmental prediction state vector, and determining the sensory prediction error based on the environmental prediction state vector and the real environmental state vector, includes:
[0027] A pre-defined prediction model is determined, and the prediction model makes predictions based on the current environmental state vector and the synthesized action vector to obtain the environmental prediction state vector.
[0028] The sensory prediction error is obtained by subtracting the predicted environmental state vector from the actual environmental state vector.
[0029] Furthermore, the update of the original parameters of the cerebellar compensation channel based on sensory prediction error includes:
[0030] Based on the sensory prediction error, locate the memory address index activated by the sensory prediction error from the preset weight table;
[0031] The weight change is determined based on the magnitude of the sensory prediction error and the preset fast learning rate, and the weight value corresponding to the activated memory address index is adjusted and updated based on the weight change.
[0032] Furthermore, the optimization of the original policy parameters of the main policy channel based on reward signals and state transition data includes:
[0033] Interaction data is periodically collected from the experience buffer of the main policy channel. An optimization objective function is constructed based on the interaction data, and the value evaluation network in the main policy channel is optimized using the optimization objective function.
[0034] The environmental state value of the robot is evaluated based on the optimized value evaluation network to obtain the environmental state value evaluation value of the robot, and an advantage function is constructed based on the environmental state value evaluation value of the robot.
[0035] A proxy objective function is constructed based on the advantage function and the near-end policy optimization pruning algorithm, and the main policy network in the main policy channel is optimized based on the proxy objective function.
[0036] Furthermore, the near-end strategy optimization pruning algorithm is as follows:
[0037] Calculate the number of historical actions selected by the new policy and the old policy in the main policy network respectively, and calculate the ratio between the number of historical actions selected by the new policy and the number of historical actions selected by the old policy to obtain the probability ratio;
[0038] The probability ratio is restricted based on a preset clipping function, limiting it to the range immediately adjacent to 1.0.
[0039] This invention also provides a robot motion control system based on cerebellar reinforcement learning, comprising:
[0040] The acquisition module is used to acquire the current environment state vector and input the current environment state vector into the main policy channel and the cerebellum compensation channel in parallel.
[0041] The output module is used by the cerebellar compensation channel to query through the internal lookup table mechanism, output the compensation action vector, and the main strategy channel to calculate through forward propagation and output the basic action vector.
[0042] The synthesis module is used to synthesize the basic motion vector and the compensated motion vector to obtain the synthesized motion vector, and to generate motion commands for robot movement based on the synthesized motion vector;
[0043] The determination module is used to control the robot to execute motion vectors based on motion commands and return a reward signal, and to determine the real environmental state vector of the new environment in which the robot is located and the state transition data generated by the robot's interaction with the new environment;
[0044] The prediction module is used to make predictions based on the current environmental state vector and the synthesized action vector to obtain the environmental prediction state vector, and to determine the sensory prediction error based on the environmental prediction state vector and the real environmental state vector.
[0045] The optimization module is used to update the original parameters of the cerebellar compensation channel based on sensory prediction error, and to optimize the original policy parameters of the main policy channel based on reward signals and state transition data.
[0046] Compared with existing technologies, the robot motion control method and system based on cerebellar reinforcement learning, as described in this embodiment of the invention, have the following advantages:
[0047] This invention achieves functional decoupling of advanced decision-making and rapid adaptation by constructing a dual-pathway architecture in parallel with the main strategy channel and the cerebellar compensation channel. The main strategy channel focuses on long-term task planning and outputs basic action instructions, while the cerebellar compensation channel is dedicated to handling dynamic changes in the environment and providing real-time fine-tuning. This collaborative architecture ensures both the macro-optimality of the strategy and the robot's ability to respond quickly to sudden changes.
[0048] This invention innovatively establishes a prediction error learning mechanism based on multimodal sensor information. This mechanism integrates visual, force and other multi-dimensional information to calculate the difference between the expected state and the actual state in real time, providing the cerebellar compensation channel with richer and more timely learning guidance than traditional reward signals. It can quickly identify changes in environmental features and achieve accurate online self-calibration.
[0049] This invention constructs a dual-loop hierarchical learning system. The high-frequency loop achieves rapid dynamic compensation through a feedforward method, while the low-frequency loop ensures long-term strategy optimization through an optimization algorithm. Through a multi-timescale collaborative mechanism, it can achieve both rapid response to sudden environmental changes and stable convergence of the learning process, thus realizing the unity of rapid adaptation and continuous optimization in robot motion control. Attached Figure Description
[0050] Figure 1 This is a schematic diagram of the process structure of the robot motion control method based on cerebellar reinforcement learning in an embodiment of the present invention;
[0051] Figure 2 This is a schematic diagram of the composition of the robot motion control system based on cerebellar reinforcement learning in an embodiment of the present invention. Detailed Implementation
[0052] The specific embodiments of this application will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.
[0053] like Figure 1As shown in the embodiments of this application, a robot motion control method based on cerebellar reinforcement learning is provided, including: S100: acquiring the current environment state vector and inputting the current environment state vector in parallel to the main strategy channel and the cerebellar compensation channel; S200: the cerebellar compensation channel queries through an internal lookup table mechanism, outputs a compensation action vector, and the main strategy channel calculates through forward propagation to output a basic action vector; S300: synthesizing the basic action vector and the compensation action vector to obtain a synthesized action vector, and generating a motion command for robot movement based on the synthesized action vector; S400: controlling the robot to execute the motion action vector based on the motion command and returning a reward signal, and determining the real environment state vector of the new environment in which the robot is located and the state transition data generated by the robot's interaction with the new environment; S500: predicting based on the current environment state vector and the synthesized action vector to obtain a predicted environment state vector, and determining the sensory prediction error based on the predicted environment state vector and the real environment state vector; S600: updating the original parameters of the cerebellar compensation channel based on the sensory prediction error, and optimizing the original strategy parameters of the main strategy channel based on the reward signal and the state transition data.
[0054] Furthermore, this invention achieves functional decoupling of advanced decision-making and rapid adaptation by constructing a dual-pathway architecture in parallel with the main strategy channel and the cerebellar compensation channel. The main strategy channel focuses on long-term task planning and outputs basic action commands, while the cerebellar compensation channel specifically handles dynamic environmental changes and provides real-time fine-tuning. This collaborative architecture ensures both the macro-optimality of the strategy and the robot's rapid response to sudden changes. This invention innovatively establishes a prediction error learning mechanism based on multimodal sensor information. This mechanism integrates visual, force, and other multi-dimensional information to calculate the difference between the expected and actual states in real time, providing the cerebellar compensation channel with richer and more timely learning guidance than traditional reward signals. It can quickly identify changes in environmental features and achieve accurate online self-calibration. This invention constructs a dual-loop hierarchical learning system. The high-frequency loop achieves rapid dynamic compensation through a feedforward approach, while the low-frequency loop ensures long-term strategy optimization through optimization algorithms. The multi-timescale collaborative mechanism achieves both rapid response to environmental changes and stable convergence of the learning process, realizing the unity of rapid adaptation and continuous optimization in robot motion control.
[0055] In the embodiments of this application, a robot motion control method based on cerebellar reinforcement learning is provided. The cerebellar compensation channel queries through an internal lookup table mechanism to output a compensation action vector. The method includes: the cerebellar compensation channel receiving the current environment state vector and performing linear scaling and discretization quantization processing on each dimension of the current environment state vector, mapping the current environment state vector from a continuous space to a discrete index space; covering the discretized quantized state with multiple overlapping quantization grids, and for each grid, converting the quantized state into an integer coordinate vector by adding a preset offset and rounding down; calculating the integer coordinate vector corresponding to each quantization grid using a preset hash function to obtain the calculation results, and mapping the calculation results from a preset weight table to obtain one or more unique memory address indices; reading the weight values corresponding to the activated memory address indices, and performing aggregation operations on these weight values to obtain the output compensation action vector.
[0056] Specifically, each dimension of the input continuous state vector is linearly scaled and discretized, transforming it from a continuous numerical space into a finite discrete level, thus significantly compressing the representation space. Multiple overlapping quantization grids are used to cover the discretized state, with each grid independently mapping the discrete state to an integer coordinate vector by adding a specific offset and rounding. Each coordinate vector is calculated using a preset hash function, and its output serves as a memory address index pointing to a large but practical weight table. This step cleverly solves the curse of dimensionality problem that may be encountered when directly looking up the table. The weight values corresponding to all activated grids are read, and these local weights are merged through aggregation operations (such as averaging) to generate a smooth and continuous output compensation action vector. The entire process is essentially based on non-parametric table lookup and weighted summation using state indexing, resulting in extremely low computational complexity and ensuring millisecond-level real-time response speed, perfectly meeting the high-frequency requirements of control loops. By using hash mapping to compress the high-dimensional state space into a fixed-size weight table, the memory consumption problem of traditional table lookup methods that grows exponentially with dimensionality is avoided, enabling high-dimensional state processing with limited memory. The design of multiple overlapping grids ensures that adjacent input states activate some of the same weight units. This local generalization characteristic allows the output to exhibit a smooth transition to small changes in the input state, avoiding output jumps and improving control stability. The structure of CMAC determines that each query only activates a very small portion of the global weight table. The learning signal based on the prediction error only needs to update these few activated weights. This is a highly localized and computationally efficient online learning process, enabling the cerebellum module to immediately self-correct based on new experiences.
[0057] In the embodiments of this application, a robot motion control method based on cerebellar reinforcement learning is provided. The method for calculating and outputting basic action vectors through forward propagation by the master policy channel includes: determining the current environment state vector and performing standardized preprocessing on the current environment state vector; inputting the preprocessed current environment state vector into a fully connected hidden layer network, and performing a layer-by-layer linear weighted summation of the input current environment state vector based on each hidden layer of the fully connected hidden layer network; after the output of the last hidden layer is passed to the output layer of the fully connected hidden layer network, the output layer calculates the basic action vector by linear transformation and a preset nonlinear mapping function for each dimension of the action vector.
[0058] Specifically, the current environmental state vector is standardized and preprocessed to scale the data of each dimension to a similar numerical range, which can significantly improve the stability and convergence speed of subsequent neural network training. The preprocessed state vector is fed into a fully connected hidden layer network, in which the data undergoes nonlinear transformation layer by layer. Each hidden layer performs a linear weighted summation of the input vector, and then passes it through a nonlinear activation function to gradually extract and combine high-order abstract features from the state information. The output of the last hidden layer is passed to the output layer, which performs another linear transformation and usually calculates each action dimension through a carefully designed pre-defined nonlinear mapping function to finally generate the basic action vector. This step enables the network to automatically learn and extract complex features and abstract patterns highly relevant to the task from high-dimensional, raw state inputs through multiple nonlinear hidden layers, giving it excellent generalization ability and the ability to handle new states that are similar but not seen during training. The output layer, through a nonlinear mapping function, can directly output smooth, continuous multidimensional motion vectors, which are very suitable for controlling multiple joints of a robot. Unlike the fast-responding cerebellar compensation channel, the learning and updating of the main policy channel is relatively slow and based on batch data, focusing on learning and solidifying those universal and effective macro-behavioral strategies and long-term plans.
[0059] In an embodiment of this application, a robot motion control method based on cerebellar reinforcement learning is provided. The method involves synthesizing a basic motion vector and a compensation motion vector to obtain a synthetic motion vector, and generating motion commands for robot motion based on the synthetic motion vector. The method includes: adding the basic motion vector and the compensation motion vector to obtain a synthetic motion vector, and generating motion commands for robot motion based on the synthetic motion vector.
[0060] Specifically, the basic action vector (from the main policy network) and the compensation action vector (from the cerebellum module) are directly added together. The result is the synthetic action vector that drives the robot to execute, and is ultimately converted into specific joint torques or position commands. In this step, the vector addition is a linear superposition operation, which keeps the outputs of the two channels physically independent. The weight updates of the main policy network are driven entirely by reinforcement learning signals aimed at maximizing long-term rewards, focusing on the "strategic intent" of the learning task; while the weight updates of the cerebellum module are driven by instantaneous sensory prediction errors, focusing on learning how to accurately achieve the intent or compensate for perturbations. The two do not interfere with each other, fundamentally avoiding gradient conflicts and policy oscillations that may occur when a single network learns multiple objectives.
[0061] In an embodiment of this application, a robot motion control method based on cerebellar reinforcement learning is provided. The method involves controlling the robot to execute a motion vector based on action commands, returning a reward signal, and determining the real environmental state vector of the new environment in which the robot is located, as well as state transition data generated by the robot's interaction with the new environment. The method includes: controlling the robot to execute a motion action based on action commands, then transferring the robot to a new environment and returning a reward signal; determining the real environmental state vector of the new environment in which the robot is located, and determining the state transition data generated by the robot's interaction with the new environment; constructing interaction data based on the returned reward signal and the state transition data, and storing the interaction data in the experience buffer of the main strategy channel.
[0062] Specifically, after the robot completes a movement based on a synthetic action, the environment changes accordingly. The system observes the new environmental state vector and obtains a scalar reward signal from the environment. This signal quantitatively evaluates the immediate quality of the previous action in the previous environmental state. The core information generated in this interaction process—the current state, the action performed, the reward obtained, and the next state—is encapsulated into a complete state transition data tuple. This tuple is then stored as an interaction data item in the experience buffer dedicated to the main policy channel. This step decouples interaction and learning, breaking the temporal correlation of data. The experience buffer acts as a data repository, breaking the continuous data stream with strong temporal correlation generated by real-time interaction. When the main policy performs batch learning, a batch of historical experiences can be randomly sampled from the buffer. This effectively breaks the temporal correlation between data and greatly reduces the problems of high gradient estimation variance and training instability caused by sequence correlation in deep reinforcement learning training. After each experience is stored in the buffer, it can be repeatedly sampled and learned in subsequent policy updates, greatly improving the utilization rate of valuable interaction data. It overcomes the problem of low sample efficiency caused by the "use and discard" of online policy gradient methods and is a key technology to solve the "data hunger" problem in robot reinforcement learning.
[0063] In an embodiment of this application, a robot motion control method based on cerebellar reinforcement learning is provided. The method involves predicting the environment based on the current environmental state vector and the synthesized action vector to obtain an environmental prediction state vector, and determining the sensory prediction error based on the environmental prediction state vector and the real environmental state vector. The method includes: determining a pre-set preset prediction model, and using the preset prediction model to predict the environment based on the current environmental state vector and the synthesized action vector to obtain the environmental prediction state vector; and subtracting the environmental prediction state vector from the real environmental state vector to obtain the sensory prediction error.
[0064] Specifically, the model receives the current environmental state vector and the synthetic action vector actually performed by the machine as input; the model predicts the predicted state vector that the environment should be in at the next moment under ideal or expected conditions; this predicted value is directly subtracted from the real environmental state vector actually observed by the sensor, and the difference is defined as the sensor prediction error. Unlike the sparse and delayed reward signals in traditional reinforcement learning, sensory prediction error is a dense, instantaneous, and high-dimensional signal that directly and quantitatively characterizes the instantaneous deviation between the robot's internal cognitive model of itself and its environment and the external real world, providing precise guidance for rapid adaptation. This error signal does not require complex value function backtracking or policy gradient calculation and can be directly used to drive the online learning rules of the cerebellum module, enabling the cerebellum module to focus on learning and compensating for local dynamic characteristics or perturbations that lead to inaccurate predictions, achieving millisecond-level parameter fine-tuning. This step achieves dual sensitivity to model errors and external perturbations. Prediction errors have two main sources: one is the inaccuracy of the internal prediction model itself (model error), and the other is unexpected changes in the environment (external perturbations, such as uneven ground or external forces). For the cerebellum module's learning, this mechanism does not distinguish between error sources; its unified goal is to adjust compensation actions to reduce this error, thereby adapting to both model defects and dynamic changes in the real environment.
[0065] In an embodiment of this application, a robot motion control method based on cerebellar reinforcement learning is provided. The method for updating the original parameters of the cerebellar compensation channel based on sensory prediction error includes: locating the memory address index activated by the sensory prediction error from a preset weight table based on the sensory prediction error; determining the weight change amount based on the magnitude of the sensory prediction error and a preset fast learning rate; and adjusting and updating the weight value corresponding to the activated memory address index based on the weight change amount.
[0066] Specifically, the specific weights affected by the current state and responsible for this error are identified; this is achieved by backtracking the memory address indices activated by the state during forward computation; based on the magnitude of the error vector and a preset fast learning rate, the required weight change Δw for each activated weight unit is calculated using the following formula:
[0067] Δw=η*e t / k,
[0068] Where Δw is the weight change, η is the preset fast learning rate, and e t Let k be the total number of CMAC units activated in the current state, representing the sensory prediction error. This change is added to the original weight value stored at the corresponding memory address index, completing a rapid parameter update. This step achieves true millisecond-level online adaptation. The entire update process involves only simple arithmetic operations on a small number of activated weights, resulting in extremely low computational overhead. It can be completed within a single control cycle, enabling the cerebellar module to achieve rapid learning and adjustment similar to a reflex arc, immediately integrating new experiences into its behavior. The structure of CMAC ensures that each update modifies only a very small portion of the global weight table, avoiding catastrophic interference. Learning a new state does not significantly damage the memory of other learned states because the weights are distributed and locally represented. Each interaction experience is directly and accurately used to adjust the parameters most relevant to the current situation, resulting in extremely high sample efficiency. The update rule directly aims to minimize the sensory prediction error; from a control theory perspective, this is equivalent to learning an inverse force online. By learning from a model or perturbation observer, the cerebellum module adjusts its weights so that its output compensation action precisely offsets the unmodeled dynamics or external perturbations that cause prediction errors. This update mechanism perfectly utilizes the local generalization and hash index characteristics of CMAC. Overlapping grids ensure that similar states will activate some of the same weights, enabling smooth generalization of learning. The hash table makes weight addressing and updating highly efficient, making the entire cerebellum module an adaptive filter optimized for fast, incremental, nonlinear function approximation. In effect, the continuously updated cerebellum weight table is equivalent to a dynamically adjustable feedforward controller gain, which automatically adjusts according to real-time errors, enabling the system to maintain high-precision tracking performance and anti-interference capability when facing changing environmental dynamics.
[0069] In embodiments of this application, a robot motion control method based on cerebellar reinforcement learning is provided. The optimization of the original policy parameters of the main policy channel based on reward signals and state transition data includes: periodically collecting interaction data from the experience buffer of the main policy channel; constructing an optimization objective function based on the interaction data; optimizing the value evaluation network in the main policy channel using the optimization objective function; evaluating the robot's environmental state value based on the optimized value evaluation network to obtain the robot's environmental state value assessment value; constructing an advantage function based on the robot's environmental state value assessment value; constructing a surrogate objective function based on the advantage function and a proximal policy optimization pruning algorithm; and optimizing the main policy network in the main policy channel based on the surrogate objective function.
[0070] Specifically, a batch of interaction data is periodically sampled from a buffer, and an optimization objective function is constructed based on this data to evaluate the quality of a state. This objective function is then used to update the value evaluation network, which learns to accurately estimate the expected cumulative reward that can be obtained from a given state under a given policy. The optimized value network is then used to evaluate the state and calculate the advantage function. The advantage function quantifies the relative merit of performing a specific action in a particular state compared to the average action in that state, and serves as the core guiding signal for policy optimization. A proximal policy optimization pruning algorithm is used, guided by the advantage function, to construct a proxy objective function. This function, through a pruning mechanism, explicitly limits the range of changes between the old and new policies in a single update, thereby ensuring the stability of the update. By maximizing this constrained proxy objective, the parameters of the main policy network can be safely and efficiently adjusted towards a better direction. This step, by rigidly constraining the update step size, effectively prevents policy performance collapse caused by excessively large single updates, a common risk in traditional policy gradient methods. This makes the learning process exceptionally stable, allowing for more aggressive optimization using larger batches of data and higher learning rates, thereby accelerating convergence and improving final performance. Batch sampling from the experience buffer enables data reuse, improving sample efficiency. The introduction of the value evaluation network significantly reduces the volatility of policy gradient estimation by providing more accurate and lower-variance advantage estimation, making the update direction of the main policy network more definite and reliable. Unlike cerebellar learning based on instantaneous prediction errors, the optimization drive in this process comes entirely from the reward signal; the calculation of the advantage function and the agent objective function is essentially... The evaluation and optimization of actions contribute to long-term cumulative returns, driving the main policy network beyond instantaneous reactions to learn the sequential decision-making and macro-planning capabilities required to achieve complex goals. The cerebellum module provides the main policy network with a smoother and more predictable state transition experience by rapidly compensating for dynamic uncertainty, making the value network's estimation of state value more accurate and the calculation of the advantage function more reliable. This further reduces the difficulty and variance of main policy learning, forming a positive cycle. The rapid adaptation of the cerebellum makes main policy learning more stable, while the continuous improvement of the main policy defines new and more challenging "ideal actions" for the cerebellum to accurately track, forming a "slow-thinking" strategic decision-making layer. The entire optimization process is periodic and batch-based, decoupled from the real-time control loop.
[0071] In the embodiments of this application, a robot motion control method based on cerebellar reinforcement learning is provided. The proximal policy optimization pruning algorithm is as follows: calculate the number of historical actions selected by the new policy and the old policy in the main policy network respectively, and calculate the ratio between the number of historical actions selected by the new policy and the number of historical actions selected by the old policy to obtain the probability ratio; restrict the probability ratio based on a preset pruning function, and restrict the probability ratio to the interval immediately adjacent to 1.0.
[0072] Specifically, for each historical state-action pair in the data, the algorithm calculates the probability of the new policy (the policy network to be updated) and the old policy (the policy network before the update) choosing that action, and calculates the probability ratio between the two. This ratio intuitively reflects the degree of change in the preference of the new and old policies for that historical decision. A ratio close to 1 indicates a small change, a ratio much greater than 1 indicates that the new policy is significantly more inclined to take that action, and a ratio much less than 1 indicates that the new policy is significantly more averse to it. By introducing a preset pruning function with pruning parameter ε, the function forces the probability ratio to the boundary value of the interval [1-ε, 1+ε] when it falls outside the interval, thus ensuring that the ratio finally used for gradient calculation is strictly limited to a narrow range close to 1.0. This seemingly simple pruning operation in this step brings a revolutionary improvement in learning stability, preventing catastrophic policy updates and performance crashes. In traditional policy gradient pruning, an unfortunate, excessively large update can lead to a sharp deterioration of the policy, the collection of low-quality data, and a vicious cycle, resulting in performance crashes. PPO pruning completely eliminates this situation by hard-limiting the maximum change in policy behavior in a single update, making the training process exceptionally smooth and reliable. The pruning operation mathematically approximates the enforcement of a trust region constraint, ensuring that the new policy does not deviate too far from the old policy. That is, the algorithm only "trusts" the validity of the first-order approximation in the region near the old policy, allowing the algorithm to boldly perform larger step-size optimizations within this trust region while avoiding catastrophic consequences caused by going out of bounds, achieving a balance between robustness and learning efficiency.
[0073] like Figure 2 As shown in the embodiments of this application, a robot motion control system based on cerebellar reinforcement learning is provided, comprising: an acquisition module for acquiring the current environment state vector and inputting the current environment state vector in parallel to the main strategy channel and the cerebellar compensation channel; an output module for the cerebellar compensation channel to query through an internal lookup table mechanism, outputting a compensation action vector, and the main strategy channel to calculate through forward propagation and outputting a basic action vector; a synthesis module for synthesizing the basic action vector and the compensation action vector to obtain a synthesized action vector, and generating a motion command for robot movement based on the synthesized action vector; a determination module for controlling the robot to execute the motion action vector based on the motion command and returning a reward signal, and determining the real environment state vector of the new environment in which the robot is located and the state transition data generated by the robot's interaction with the new environment; a prediction module for predicting based on the current environment state vector and the synthesized action vector to obtain a predicted environment state vector, and determining the sensory prediction error based on the predicted environment state vector and the real environment state vector; and an optimization module for updating the original parameters of the cerebellar compensation channel based on the sensory prediction error, and optimizing the original strategy parameters of the main strategy channel based on the reward signal and the state transition data.
[0074] In summary, this invention provides a robot motion control method and system based on cerebellar reinforcement learning, comprising: acquiring the current environmental state vector and inputting it in parallel to a main strategy channel and a cerebellar compensation channel; the main strategy channel outputting basic actions based on long-term task objectives, while the cerebellar compensation channel outputting compensation actions to cope with real-time dynamics through an efficient query mechanism; synthesizing the basic action vector and the compensation action vector into a combined action vector to drive the robot; feeding back the latest data after the robot drives the motion action vector, and determining the sensory prediction error based on the latest data; updating the original parameters of the cerebellar compensation channel based on the sensory prediction error, and optimizing the original strategy parameters of the main strategy channel based on the latest data. This invention achieves functional decoupling of high-level decision-making and rapid adaptation by constructing a parallel dual-pathway architecture, and the hierarchical learning system of the dual loop achieves the unity of rapid adaptation and continuous optimization.
[0075] Finally, it should be noted that those skilled in the art can obviously make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims and their equivalents, this invention also intends to include these modifications and variations.
Claims
1. A robot motion control method based on cerebellar reinforcement learning, characterized in that, include: Obtain the current environment state vector and input it in parallel into the main policy channel and the cerebellum compensation channel; The cerebellar compensation channel queries through an internal lookup table mechanism, outputs compensation action vectors, and the main strategy channel calculates through forward propagation to output basic action vectors. The basic motion vector and the compensated motion vector are combined to obtain the composite motion vector, and the motion commands for robot movement are generated based on the composite motion vector. After the robot executes motion vectors based on motion commands, it returns a reward signal and determines the real environmental state vector of the new environment in which the robot is located, as well as the state transition data generated by the robot's interaction with the new environment. Based on the current environmental state vector and the synthesized action vector, a prediction is made to obtain the environmental prediction state vector, and the sensory prediction error is determined based on the environmental prediction state vector and the real environmental state vector. The original parameters of the cerebellar compensation channel are updated based on sensory prediction error, and the original policy parameters of the main policy channel are optimized based on reward signals and state transition data.
2. The robot motion control method based on cerebellar reinforcement learning according to claim 1, characterized in that, The cerebellar compensation channel performs queries through an internal lookup table mechanism to output compensation action vectors, including: The cerebellar compensation channel receives the current environmental state vector and performs linear scaling and discretization quantization on each dimension of the current environmental state vector, mapping the current environmental state vector from a continuous space to a discrete index space. Multiple overlapping quantization grids are used to cover the discretized quantization state. For each grid, the quantization state is converted into an integer coordinate vector by adding a preset offset and rounding down. The integer coordinate vector corresponding to each quantized grid is calculated using a preset hash function to obtain the calculation results. The calculation results are then mapped from a preset weight table to obtain one or more unique memory address indices. Read the weight values corresponding to the activated memory address indices, and perform aggregation operations on these weight values to obtain the output compensation action vector.
3. The robot motion control method based on cerebellar reinforcement learning according to claim 1, characterized in that, The basic action vector, calculated by the main policy channel through forward propagation, includes: Determine the current environment state vector and perform standardization preprocessing on the current environment state vector; The preprocessed current environment state vector is input into a fully connected hidden layer network, and the input current environment state vector is linearly weighted and summed layer by layer based on each hidden layer of the fully connected hidden layer network; After the output of the last hidden layer is passed to the output layer of the fully connected hidden layer network, the output layer calculates the basic action vector for each action vector dimension through linear transformation and a preset nonlinear mapping function.
4. The robot motion control method based on cerebellar reinforcement learning according to claim 1, characterized in that, The process of synthesizing the base motion vector and the compensated motion vector to obtain the synthesized motion vector, and generating motion commands for robot movement based on the synthesized motion vector, includes: The basic motion vector and the compensated motion vector are added together to obtain the synthetic motion vector, and motion commands for robot movement are generated based on the synthetic motion vector.
5. The robot motion control method based on cerebellar reinforcement learning according to claim 1, characterized in that, The process of controlling the robot to execute motion vectors based on action commands and returning a reward signal, and determining the true environmental state vector of the new environment in which the robot is located and the state transition data generated by the robot's interaction with the new environment, includes: After the robot performs a movement based on motion commands, it moves to a new environment and returns a reward signal. Determine the true environmental state vector of the robot in the new environment, and determine the state transition data generated by the interaction between the robot and the new environment; Interaction data is constructed based on the returned reward signals and state transition data, and the interaction data is stored in the experience buffer of the main policy channel.
6. The robot motion control method based on cerebellar reinforcement learning according to claim 1, characterized in that, The process of predicting based on the current environmental state vector and the synthesized action vector to obtain an environmental prediction state vector, and determining the sensory prediction error based on the environmental prediction state vector and the actual environmental state vector, includes: A pre-defined prediction model is determined, and the prediction model makes predictions based on the current environmental state vector and the synthesized action vector to obtain the environmental prediction state vector. The sensory prediction error is obtained by subtracting the predicted environmental state vector from the actual environmental state vector.
7. The robot motion control method based on cerebellar reinforcement learning according to claim 1, characterized in that, The updating of the original parameters of the cerebellar compensation channel based on sensory prediction error includes: Based on the sensory prediction error, locate the memory address index activated by the sensory prediction error from the preset weight table; The weight change is determined based on the magnitude of the sensory prediction error and the preset fast learning rate, and the weight value corresponding to the activated memory address index is adjusted and updated based on the weight change.
8. The robot motion control method based on cerebellar reinforcement learning according to claim 1, characterized in that, The optimization of the original policy parameters of the main policy channel based on reward signals and state transition data includes: Interaction data is periodically collected from the experience buffer of the main policy channel. An optimization objective function is constructed based on the interaction data, and the value evaluation network in the main policy channel is optimized using the optimization objective function. The environmental state value of the robot is evaluated based on the optimized value evaluation network to obtain the environmental state value evaluation value of the robot, and an advantage function is constructed based on the environmental state value evaluation value of the robot. A proxy objective function is constructed based on the advantage function and the near-end policy optimization pruning algorithm, and the main policy network in the main policy channel is optimized based on the proxy objective function.
9. A robot motion control method based on cerebellar reinforcement learning according to claim 8, characterized in that, The near-end strategy optimization pruning algorithm is as follows: Calculate the number of historical actions selected by the new policy and the old policy in the main policy network respectively, and calculate the ratio between the number of historical actions selected by the new policy and the number of historical actions selected by the old policy to obtain the probability ratio; The probability ratio is restricted based on a preset clipping function, limiting it to the range immediately adjacent to 1.
0.
10. A robot motion control system based on cerebellar reinforcement learning, characterized in that, include: The acquisition module is used to acquire the current environment state vector and input the current environment state vector into the main policy channel and the cerebellum compensation channel in parallel. The output module is used by the cerebellar compensation channel to query through the internal lookup table mechanism, output the compensation action vector, and the main strategy channel to calculate through forward propagation and output the basic action vector. The synthesis module is used to synthesize the basic motion vector and the compensated motion vector to obtain the synthesized motion vector, and to generate motion commands for robot movement based on the synthesized motion vector; The determination module is used to control the robot to execute motion vectors based on motion commands and return a reward signal, and to determine the real environmental state vector of the new environment in which the robot is located and the state transition data generated by the robot's interaction with the new environment; The prediction module is used to make predictions based on the current environmental state vector and the synthesized action vector to obtain the environmental prediction state vector, and to determine the sensory prediction error based on the environmental prediction state vector and the real environmental state vector. The optimization module is used to update the original parameters of the cerebellar compensation channel based on sensory prediction error, and to optimize the original policy parameters of the main policy channel based on reward signals and state transition data.