A deep learning-based robot cerebellum reinforcement learning method and system

Abstract

The application relates to the technical field of robot control, in particular to a robot cerebellum reinforcement learning method and system based on deep learning. The method comprises the following steps: establishing a basic control model based on a preset simulation environment, generating a training data packet according to the running monitoring data of the robot, generating an iteration instruction of the basic control model according to the training data packet, and generating a running control model according to the iteration result; acquiring environment perception data, judging whether to generate a learning instruction of the running control model according to the environment perception data; the basic control model is built based on a dual collaborative architecture of an adaptive feedforward network and an error correction network, can be embedded between an upper motion planner and a bottom driver of any robot, and the basic control model is iteratively optimized through real physical interaction data of the robot, so that the model quickly adapts to the dynamic characteristics of the robot, and training loss and safety risks are reduced.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and in particular to a deep learning-based robot cerebellar reinforcement learning method and system. Background Technology

[0002] Traditional robot motion control relies on model-based control methods (such as PID and MPC), which require accurate dynamic models and struggle to cope with unstructured environments such as terrain changes and load disturbances. Deep reinforcement learning has been introduced for end-to-end policy learning, but it suffers from problems such as low sample efficiency, large policy jitter, and difficulties in deployment.

[0003] Currently, the concept of the "cerebellum" in the field of robotics remains largely metaphorical, lacking an algorithmic framework that explicitly mimics the cerebellum's "adaptive feedforward + fast correction" neural circuits. Furthermore, policy networks trained purely with DRL exhibit poor interpretability and high parameter tuning costs; they fail to fully utilize the cerebellum's "prediction-correction" mechanism, resulting in insufficient stability of robots in high-speed, high-precision tasks; and existing methods lack sufficient fusion of multimodal sensory information (proprioception + vision + force perception), making it difficult to achieve agile movements under partial observation. Summary of the Invention

[0004] The purpose of this application is to provide a deep learning-based reinforcement learning method and system for robot cerebellum to solve the above-mentioned technical problems, aiming to improve the training efficiency and operational control accuracy of robot cerebellum.

[0005] In some embodiments of this application, a basic control model is built based on a dual collaborative architecture of adaptive feedforward network and error correction network. This model can be embedded between the upper-level motion planner and the lower-level actuator of any robot. The basic control model is optimized and iterated through the robot's real physical interaction data, enabling the model to quickly adapt to the robot's dynamic characteristics and reducing training losses and safety risks.

[0006] In some embodiments of this application, through scenario-based and incremental learning, when a robot encounters a completely new scenario after long-term deployment, it can find the anchor scenario through the difference value and quickly adapt to the new scenario by adjusting local parameters. This avoids the forgetting of existing skills or system instability that may be caused by global adjustments, and improves the control accuracy and robustness of the operation control model.

[0007] In some embodiments of this application, a deep learning-based reinforcement learning method for the cerebellum of a robot is provided, including:

[0008] A basic control model is established based on a preset simulation environment, and training data packages are generated based on the robot's operation monitoring data.

[0009] The training data package generates iterative instructions for the basic control model, and the iteration results generate the running control model.

[0010] Acquire environmental perception data and determine whether to generate learning instructions for the operation control model based on the environmental perception data;

[0011] The basic control model includes a feedforward sub-model and an initial error model.

[0012] In some embodiments of this application, the establishment of the basic control model includes:

[0013] A control network is constructed based on the robot's structural parameters, and the control network includes multiple control points;

[0014] Multiple basic scenarios were set up based on historical operational needs;

[0015] Establish a sequence of basic scenarios A, A=(a1, a2, ..., ai, ..., an), where ai is the i-th basic scenario; n is the number of basic scenarios;

[0016] Based on the basic scenario sequence A, ai is sequentially set as the target basic scenario;

[0017] The initial mapping actions of the control network in the target basic scenario are generated based on the simulation environment.

[0018] Establish a predictive sub-model for the target basic scenario;

[0019] The initial mapping actions of the control network in each basic scenario are generated sequentially, and the feedforward sub-model is generated based on all the initial mapping actions;

[0020] Obtain the prediction sub-models for each basic scenario, and generate an initial error model based on all the prediction sub-models;

[0021] The basic control model is generated based on the feedforward sub-model and the initial error model.

[0022] In some embodiments of this application, generating the training data package includes:

[0023] Deploy the basic control model into the robot;

[0024] The feedforward sub-model acquires the primary environmental data collected by the robot;

[0025] Generate actions to be performed and data identification tags based on primary environmental data;

[0026] The initial error model generates expected error parameters based on primary environmental data and the actions to be performed;

[0027] Generate compensation instructions based on the expected error parameters;

[0028] Generate a first-level execution action based on the compensation instruction and the action to be executed;

[0029] Obtain the actual error parameters;

[0030] Generate a record sub-package of the first-level execution action, and generate a single monitoring feedback package based on the record sub-package and data identification tags;

[0031] Training data packets are generated based on all monitoring feedback packets.

[0032] In some embodiments of this application, the generation of the action to be performed and the data identification tag includes:

[0033] Similarity values ​​are generated based on primary environmental data and various basic scenarios;

[0034] The scene corresponding to the maximum value among all similarity values ​​is designated as the first-level scene.

[0035] Set the initial mapping action corresponding to the first-level scene as the action to be executed;

[0036] Set the scene label of the first-level scene as the data recognition label.

[0037] In some embodiments of this application, the generation of the operation control model includes:

[0038] Based on the basic scenario sequence A, ai is sequentially set as the scenario to be trained;

[0039] Generate associated data packets for the training scenario based on the training data packets;

[0040] The iteration parameters of the initial mapping action and the initial error model in the training scenario are set according to the associated data packets;

[0041] Based on the iteration results, a first-level prediction model for the training scenario and a first-level mapping action of the control network in the training scenario are generated.

[0042] The first-level mapping actions of the control network in each basic scenario are generated sequentially, and a first-level feedforward model is generated based on all the first-level mapping actions.

[0043] Obtain the first-level prediction model for each basic scenario, and generate a first-level error model based on all the first-level prediction models;

[0044] The operation control model is generated based on the first-level error model and the first-level feedforward model.

[0045] In some embodiments of this application, the step of determining whether to generate learning instructions for the operation control model includes:

[0046] Incremental scenarios are established based on environmental perception data;

[0047] Generate the difference values ​​between the incremental scene and each base scene, and select the minimum value b among all the difference values. min ;

[0048] Preset difference threshold B1;

[0049] If b min >B1, generates Level 1 learning instructions;

[0050] The first-level learning instructions include:

[0051] Set minimum value b min The corresponding basic scenario is the anchor scenario of the incremental scenario;

[0052] Multiple control points to be optimized are selected based on the anchoring scenario;

[0053] Obtain monitoring feedback packages for incremental scenarios;

[0054] Generate first-level mapping actions and first-level prediction models for incremental scenarios based on all control points to be optimized and monitoring feedback packages.

[0055] In some embodiments of this application, the selection of multiple control points to be optimized includes:

[0056] Generate the dependency values ​​of the anchored scenario on each control point in the control network;

[0057] Preset dependency threshold C1;

[0058] If C1>ci (i=1,2…m), set the i-th control point as the control point to be optimized in the target anchoring scene;

[0059] Where ci is the dependency value of the anchored scene on the i-th control point; m is the number of control points in the control network.

[0060] In some embodiments of this application, a deep learning-based robotic cerebellar reinforcement learning system is provided, comprising:

[0061] The simulation unit is used to preset the simulation environment and establish a basic control model.

[0062] The basic control model includes: a feedforward sub-model and an initial error model;

[0063] The central control unit is used to deploy the basic control model into the robot;

[0064] The central control unit includes:

[0065] The perception module is used to generate training data packages based on the robot's operational monitoring data;

[0066] The sensing module is also used to acquire environmental sensing data;

[0067] The first learning module is used to generate iterative instructions for the basic control model based on the training data package, and to generate the running control model based on the iteration results.

[0068] The second learning module is used to determine whether to generate learning instructions for the operation control model based on environmental perception data.

[0069] In some embodiments of this application, the simulation unit includes:

[0070] The first simulation module is used to construct a control network based on the robot's structural parameters, and the control network includes multiple control points.

[0071] The second simulation module is used to set up multiple basic scenarios based on historical operational requirements;

[0072] Establish a sequence of basic scenarios A, A=(a1, a2, ..., ai, ..., an), where ai is the i-th basic scenario; n is the number of basic scenarios;

[0073] Based on the basic scenario sequence A, ai is sequentially set as the target basic scenario;

[0074] The initial mapping actions of the control network in the target basic scenario are generated based on the simulation environment;

[0075] Establish a predictive sub-model for the target basic scenario;

[0076] The initial mapping actions of the control network in each basic scenario are generated sequentially, and the feedforward sub-model is generated based on all the initial mapping actions;

[0077] Obtain the prediction sub-models for each basic scenario, and generate an initial error model based on all the prediction sub-models;

[0078] The basic control model is generated based on the feedforward sub-model and the initial error model.

[0079] In some embodiments of this application, the sensing module is further configured to:

[0080] Deploy the basic control model into the robot;

[0081] The feedforward sub-model acquires the primary environmental data collected by the robot;

[0082] Generate actions to be performed and data identification tags based on primary environmental data;

[0083] The initial error model generates expected error parameters based on primary environmental data and the actions to be performed;

[0084] Generate compensation instructions based on the expected error parameters;

[0085] Generate a first-level execution action based on the compensation instruction and the action to be executed;

[0086] Obtain the actual error parameters;

[0087] Generate a record sub-package of the first-level execution action, and generate a single monitoring feedback package based on the record sub-package and data identification tags;

[0088] Generate training data packets based on all monitoring feedback packets;

[0089] This includes generating actions to be performed and data identification tags, including:

[0090] Similarity values ​​are generated based on primary environmental data and various basic scenarios;

[0091] The scene corresponding to the maximum value among all similarity values ​​is designated as the first-level scene.

[0092] Set the initial mapping action corresponding to the first-level scene as the action to be executed;

[0093] Set the scene label of the first-level scene as the data recognition label.

[0094] Compared with existing technologies, the beneficial effects of the deep learning-based robot cerebellar reinforcement learning method and system proposed in this application are as follows:

[0095] The basic control model is built based on a dual collaborative architecture of adaptive feedforward network and error correction network. It can be embedded between the upper-level motion planner and the lower-level actuator of any robot. The basic control model is optimized and iterated through the robot's real physical interaction data, so that the model can quickly adapt to the robot's dynamic characteristics, reduce training loss and safety risks.

[0096] By using scenario-based and incremental learning, when robots encounter entirely new scenarios after long-term deployment, they can find anchor scenarios through difference values ​​and quickly adapt to new scenarios by adjusting local parameters. This avoids the loss of existing skills or system instability that may be caused by global adjustments, thereby improving the control accuracy and robustness of the operation control model. Attached Figure Description

[0097] Figure 1 This is a flowchart illustrating a deep learning-based reinforcement learning method for the cerebellum of a robot, as described in a preferred embodiment of this application. Detailed Implementation

[0098] The specific embodiments of this application will be described in further detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate this application, but are not intended to limit the scope of this application.

[0099] In the description of this application, it should be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0100] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0101] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0102] like Figure 1 As shown, a preferred embodiment of this application provides a deep learning-based reinforcement learning method for the cerebellum of a robot, comprising:

[0103] S101: Establish a basic control model based on a preset simulation environment, and generate training data packages based on the robot's operation monitoring data;

[0104] S102: Generate iterative instructions for the basic control model based on the training data package, and generate the running control model based on the iteration results;

[0105] S103: Acquire environmental perception data and determine whether to generate learning instructions for the operation control model based on the environmental perception data;

[0106] The basic control model includes: a feedforward sub-model and an initial error model.

[0107] Specifically, a simulation environment is constructed by collecting the robot's historical operation data. The simulation environment includes various basic terrains and corresponding disturbance parameters (e.g., straight travel on flat ground, turning, obstacle avoidance, uphill, and rugged terrain).

[0108] Allow the robot to conduct extensive random or guided exploration in a simulated environment. Simultaneously record all data (environmental perception (e.g., event flow simulation), robot state (joint angles, speed), executed motion commands, and reward or error signals calculated by the simulator (e.g., fall penalty, goal achievement reward).

[0109] Specifically, establishing a basic control model includes:

[0110] A control network is constructed based on the robot's structural parameters, and the control network includes multiple control points;

[0111] Multiple basic scenarios were set up based on historical operational needs;

[0112] Establish a sequence of basic scenarios A, A=(a1, a2, ..., ai, ..., an), where ai is the i-th basic scenario; n is the number of basic scenarios;

[0113] Based on the basic scenario sequence A, ai is sequentially set as the target basic scenario;

[0114] The initial mapping actions of the control network in the target basic scenario are generated based on the simulation environment;

[0115] Establish a predictive sub-model for the target basic scenario;

[0116] The initial mapping actions of the control network in each basic scenario are generated sequentially, and the feedforward sub-model is generated based on all the initial mapping actions;

[0117] Obtain the prediction sub-models for each basic scenario, and generate an initial error model based on all the prediction sub-models;

[0118] The basic control model is generated based on the feedforward sub-model and the initial error model.

[0119] Specifically, each movable node is set as a control point (e.g., joint, walking drive node) based on the robot's structural parameters.

[0120] Specifically, multiple basic scenes are generated based on the basic terrain contained in the simulation environment. Each basic scene represents a basic terrain, and the robot is in an unloaded state in each basic scene.

[0121] Specifically, the robot is allowed to conduct a large number of random or guided explorations in a simulated environment. All data is recorded at the same time, thereby generating multiple sets of terrain-action data pairs. The collected terrain-action data is used to train an adaptive feedforward network in a reinforcement learning manner (maximizing cumulative reward) to generate initial mapped actions (i.e., the angle, speed and torque output by each control point in the current scene) corresponding to each basic scene.

[0122] Specifically, all recorded data includes: environmental perception (such as event flow simulation), robot state (joint angles, speed), executed motion commands, and calculated reward or error signals (such as fall penalty, goal achievement reward).

[0123] Specifically, prediction sub-models for each basic sub-scene are generated based on the error prediction network. The prediction sub-model can output the expected error value (such as the expected posture error and trajectory deviation) when the robot performs the initial mapping action corresponding to the current basic scene.

[0124] Understandably, in the above embodiments, a basic control model is constructed by establishing a simulation environment, avoiding high-risk trial and error in the real environment, thus reducing hardware consumption and safety risks. Simultaneously, the basic control model is built based on a dual collaborative architecture of an adaptive feedforward network and an error correction network. The feedforward part is responsible for rapid response, while the error compensation part focuses on accuracy adjustment. Their collaborative work ensures both response speed and improved control accuracy.

[0125] In a preferred embodiment of this application, generating a training data package includes:

[0126] Deploy the basic control model into the robot;

[0127] The feedforward sub-model acquires the primary environmental data collected by the robot;

[0128] Generate actions to be performed and data identification tags based on primary environmental data;

[0129] The initial error model generates expected error parameters based on primary environmental data and the actions to be performed;

[0130] Generate compensation instructions based on the expected error parameters;

[0131] Generate a first-level execution action based on the compensation instruction and the action to be executed;

[0132] Obtain the actual error parameters;

[0133] Generate a record sub-package of the first-level execution action, and generate a single monitoring feedback package based on the record sub-package and data identification tags;

[0134] Training data packets are generated based on all monitoring feedback packets.

[0135] Specifically, after deploying the basic control model into the robot, the robot first performs simple tasks, namely, tasks corresponding to various basic scenarios. Simultaneously, by acquiring primary environmental data collected by the robot, it determines which specific basic scenario the robot is currently in, and encapsulates all relevant data generated during execution to create a single monitoring feedback package. By executing multiple simple tasks, a large number of monitoring feedback packages are accumulated.

[0136] Specifically, the relevant data includes expected error parameters and actual error parameters, where the actual error parameters can be collected through built-in body sensors (IMU, encoder, force sensor).

[0137] Specifically, the compensation instruction refers to the real-time fine-tuning of the action to be executed based on the expected error parameters.

[0138] Specifically, generating actions to be performed and data identification tags includes:

[0139] Similarity values ​​are generated based on primary environmental data and various basic scenarios;

[0140] The scene corresponding to the maximum value among all similarity values ​​is designated as the first-level scene.

[0141] Set the initial mapping action corresponding to the first-level scene as the action to be executed;

[0142] Set the scene label of the first-level scene as the data recognition label.

[0143] Specifically, based on the primary environmental data, the robot determines which basic scene it belongs to based on its current terrain environment. The similarity value between the primary environmental data and that scene is set to 1, and the similarity value between the primary environmental data and the remaining basic scenes is set to 0.

[0144] Specifically, by generating data identification tags, it is convenient for various basic scenarios to quickly filter related data during optimization and iteration.

[0145] Specifically, generating an operational control model includes:

[0146] Based on the basic scenario sequence A, ai is sequentially set as the scenario to be trained;

[0147] Generate associated data packets for the training scenario based on the training data packets;

[0148] The iteration parameters of the initial mapping action and the initial error model in the training scenario are set according to the associated data packets;

[0149] Based on the iteration results, a first-level prediction model for the training scenario and a first-level mapping action of the control network in the training scenario are generated.

[0150] The first-level mapping actions of the control network in each basic scenario are generated sequentially, and a first-level feedforward model is generated based on all the first-level mapping actions.

[0151] Obtain the first-level prediction model for each basic scenario, and generate a first-level error model based on all the first-level prediction models;

[0152] The operation control model is generated based on the first-level error model and the first-level feedforward model.

[0153] Specifically, the data identification label of the associated data packet is all the monitoring feedback packets corresponding to the scene label of the scene to be trained.

[0154] Specifically, the initial mapping action and initial error model in the training scenario are optimized iteratively using associated data packets. This involves: setting an iteration objective (minimizing the prediction error, reducing the prediction error caused by the initial mapping action, and improving the prediction accuracy of the prediction sub-model); establishing a loss function (calculating the difference between the prediction error of the prediction sub-model and the actual measurement error after execution (e.g., mean squared error) to determine whether the prediction sub-model's prediction is accurate and whether the initial mapping action led to a predictable, small actual error); calculating the gradient using the loss function and updating the parameters of the initial mapping action and prediction sub-model; and generating a first-level prediction model and a first-level mapping action based on the update results.

[0155] It is understood that in the above embodiments, the basic control model is built based on the dual collaborative architecture of adaptive feedforward network and error correction network. It can be embedded between the upper-level motion planner and the lower-level actuator of any robot, and the basic control model is optimized and iterated through the robot's real physical interaction data, so that the model can quickly adapt to the dynamic characteristics of the robot and reduce training loss and safety risks.

[0156] In a preferred embodiment of this application, determining whether to generate learning instructions for the operation control model includes:

[0157] Incremental scenarios are established based on environmental perception data;

[0158] Generate the difference values ​​between the incremental scene and each base scene, and select the minimum value b among all the difference values. min ;

[0159] Preset difference threshold B1;

[0160] If b min >B1, generates Level 1 learning instructions;

[0161] The first-level learning instructions include:

[0162] Set minimum value b min The corresponding basic scenario is the anchor scenario of the incremental scenario;

[0163] Multiple control points to be optimized are selected based on the anchoring scenario;

[0164] Obtain monitoring feedback packages for incremental scenarios;

[0165] Generate first-level mapping actions and first-level prediction models for incremental scenarios based on all control points to be optimized and monitoring feedback packages.

[0166] Specifically, when a robot encounters a completely new scenario (such as unknown terrain or new load) during long-term deployment, an incremental scenario is established based on the environmental perception data collected by the robot.

[0167] Specifically, based on the robot load and terrain environment in the incremental scene, a similarity score is generated with each basic scene. The greater the similarity score, the smaller the corresponding difference score. The mapping relationship between the two can be set according to historical parameters.

[0168] Specifically, the difference threshold can be set based on historical parameters. If the minimum value among all difference values ​​(i.e., the most similar base scenario) exceeds the preset difference threshold, it indicates that the current incremental scenario is a completely new scenario. The operation control model then precisely controls the robot's actions in the current incremental scenario, requiring timely optimization of the operation control model.

[0169] Specifically, control points to be optimized refer to control points whose operating parameters can be adjusted.

[0170] Specifically, in incremental scenarios, the robot first executes the first-level mapping action and the first-level prediction model corresponding to the anchor scenario, and collects relevant data in real time to generate corresponding monitoring feedback packages.

[0171] Specifically, by using the data in the monitoring feedback package, the parameters of each control point to be optimized in the first-level mapping action of the anchored scene are adjusted to generate the first-level mapping action corresponding to the current incremental scene. At the same time, the first-level prediction model in the anchored scene is optimized to generate the first-level prediction model of the incremental scene.

[0172] Specifically, multiple control points to be optimized are selected, including:

[0173] Generate the dependency values ​​of the anchored scenario on each control point in the control network;

[0174] Preset dependency threshold C1;

[0175] If C1>ci (i=1,2…m), set the i-th control point as the control point to be optimized in the target anchoring scene;

[0176] Where ci is the dependency value of the anchoring scene on the i-th control point; m is the number of control points in the control network.

[0177] Specifically, the corresponding dependency value is set according to the degree of interference of each control point in the control network with the execution of the first-level mapping action in the anchoring scenario. The greater the degree of interference (that is, the greater the increase in the actual error of the first-level mapping action when the parameters of the current control point are adjusted), the greater the corresponding dependency value. The mapping relationship between the two can be set according to historical parameters.

[0178] Specifically, the dependency threshold can be set based on historical parameters. This means that if the dependency value of a control point is greater than the preset dependency threshold, adjusting the operating parameters of the current control point will cause the system to become unstable.

[0179] It is understandable that in the above embodiments, through scenario-based and incremental learning, when the robot encounters a completely new scenario after long-term deployment, it can find the anchor scenario through the difference value and quickly adapt to the new scenario by adjusting the local parameters. This avoids the forgetting of existing skills or system instability that may be caused by global adjustments, and improves the control accuracy and robustness of the operation control model.

[0180] In another preferred embodiment of a deep learning-based robot cerebellar reinforcement learning method based on any of the above preferred embodiments, this preferred embodiment provides a deep learning-based robot cerebellar reinforcement learning system, comprising:

[0181] The simulation unit is used to preset the simulation environment and establish a basic control model.

[0182] The basic control model includes: a feedforward sub-model and an initial error model;

[0183] The central control unit is used to deploy the basic control model into the robot;

[0184] The central control unit includes:

[0185] The perception module is used to generate training data packages based on the robot's operational monitoring data;

[0186] The perception module is also used to acquire environmental perception data;

[0187] The first learning module is used to generate iterative instructions for the basic control model based on the training data package, and to generate the running control model based on the iteration results.

[0188] The second learning module is used to determine whether to generate learning instructions for the operation control model based on environmental perception data.

[0189] In a preferred embodiment of this application, the simulation unit includes:

[0190] The first simulation module is used to construct a control network based on the robot's structural parameters. The control network includes multiple control points.

[0191] The second simulation module is used to set up multiple basic scenarios based on historical operational requirements;

[0192] Establish a sequence of basic scenarios A, A=(a1, a2, ..., ai, ..., an), where ai is the i-th basic scenario; n is the number of basic scenarios;

[0193] Based on the basic scenario sequence A, ai is sequentially set as the target basic scenario;

[0194] The initial mapping actions of the control network in the target basic scenario are generated based on the simulation environment.

[0195] Establish a predictive sub-model for the target basic scenario;

[0196] The initial mapping actions of the control network in each basic scenario are generated sequentially, and the feedforward sub-model is generated based on all the initial mapping actions;

[0197] Obtain the prediction sub-models for each basic scenario, and generate an initial error model based on all the prediction sub-models;

[0198] The basic control model is generated based on the feedforward sub-model and the initial error model.

[0199] In a preferred embodiment of this application, the sensing module is further configured to:

[0200] Deploy the basic control model into the robot;

[0201] The feedforward sub-model acquires the primary environmental data collected by the robot;

[0202] Generate actions to be performed and data identification tags based on primary environmental data;

[0203] The initial error model generates expected error parameters based on primary environmental data and the actions to be performed;

[0204] Generate compensation instructions based on the expected error parameters;

[0205] Generate a first-level execution action based on the compensation instruction and the action to be executed;

[0206] Obtain the actual error parameters;

[0207] Generate a record sub-package of the first-level execution action, and generate a single monitoring feedback package based on the record sub-package and data identification tags;

[0208] Generate training data packets based on all monitoring feedback packets;

[0209] This includes generating actions to be performed and data identification tags, including:

[0210] Similarity values ​​are generated based on primary environmental data and various basic scenarios;

[0211] The scene corresponding to the maximum value among all similarity values ​​is designated as the first-level scene.

[0212] Set the initial mapping action corresponding to the first-level scene as the action to be executed;

[0213] Set the scene label of the first-level scene as the data recognition label.

[0214] According to the first concept of this application, a basic control model is built based on a dual collaborative architecture of adaptive feedforward network and error correction network. It can be embedded between the upper-level motion planner and the lower-level actuator of any robot. The basic control model is optimized and iterated through the robot's real physical interaction data, so that the model can quickly adapt to the dynamic characteristics of the robot and reduce training loss and safety risks.

[0215] According to the second concept of this application, through scenario-based and incremental learning, when a robot encounters a completely new scenario after long-term deployment, it can find the anchor scenario through the difference value and quickly adapt to the new scenario by adjusting the local parameters. This avoids the forgetting of existing skills or system instability that may be caused by global adjustments, and improves the control accuracy and robustness of the operation control model.

[0216] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and substitutions can be made without departing from the technical principles of this application, and these improvements and substitutions should also be considered within the scope of protection of this application.

Claims

1. A deep learning-based reinforcement learning method for the cerebellum of a robot, characterized in that, include: A basic control model is established based on a preset simulation environment, and training data packages are generated based on the robot's operation monitoring data. The training data package generates iterative instructions for the basic control model, and the iteration results generate the running control model. Acquire environmental perception data and determine whether to generate learning instructions for the operation control model based on the environmental perception data; The basic control model includes: a feedforward sub-model and an initial error model; The establishment of the basic control model includes: A control network is constructed based on the robot's structural parameters, and the control network includes multiple control points; Each movable node is set as a control point based on the robot's structural parameters; Multiple basic scenarios were set up based on historical operational needs; Establish a sequence of basic scenarios A, A=(a1, a2, ..., ai, ..., an), where ai is the i-th basic scenario; n is the number of basic scenarios; Based on the basic scenario sequence A, ai is sequentially set as the target basic scenario; The initial mapping actions of the control network in the target basic scenario are generated based on the simulation environment. Establish a predictive sub-model for the target basic scenario; The initial mapping actions of the control network in each basic scenario are generated sequentially, and the feedforward sub-model is generated based on all the initial mapping actions; Obtain the prediction sub-models for each basic scenario, and generate an initial error model based on all the prediction sub-models; The basic control model is generated based on the feedforward sub-model and the initial error model.

2. The deep learning-based robot cerebellar reinforcement learning method as described in claim 1, characterized in that, The generation of training data packets includes: Deploy the basic control model into the robot; The feedforward sub-model acquires the primary environmental data collected by the robot; Generate actions to be performed and data identification tags based on primary environmental data; The initial error model generates expected error parameters based on primary environmental data and the actions to be performed; Generate compensation instructions based on the expected error parameters; Generate a first-level execution action based on the compensation instruction and the action to be executed; Obtain the actual error parameters; Generate a record sub-package of the first-level execution action, and generate a single monitoring feedback package based on the record sub-package and data identification tags; Training data packets are generated based on all monitoring feedback packets.

3. The deep learning-based robot cerebellar reinforcement learning method as described in claim 2, characterized in that, The generation of the action to be executed and the data identification tag includes: Similarity values ​​are generated based on primary environmental data and various basic scenarios; The scene corresponding to the maximum value among all similarity values ​​is designated as the first-level scene. Set the initial mapping action corresponding to the first-level scene as the action to be executed; Set the scene label of the first-level scene as the data recognition label.

4. The deep learning-based robot cerebellar reinforcement learning method as described in claim 3, characterized in that, The generated operation control model includes: Based on the basic scenario sequence A, ai is sequentially set as the scenario to be trained; Generate associated data packets for the training scenario based on the training data packets; The iteration parameters of the initial mapping action and the initial error model in the training scenario are set according to the associated data packets; Based on the iteration results, a first-level prediction model for the training scenario and a first-level mapping action of the control network in the training scenario are generated. The first-level mapping actions of the control network in each basic scenario are generated sequentially, and a first-level feedforward model is generated based on all the first-level mapping actions. Obtain the first-level prediction model for each basic scenario, and generate a first-level error model based on all the first-level prediction models; The operation control model is generated based on the first-level error model and the first-level feedforward model.

5. The deep learning-based robot cerebellar reinforcement learning method as described in claim 4, characterized in that, The determination of whether to generate a learning instruction for the operation control model includes: Incremental scenarios are established based on environmental perception data; Generate the difference values ​​between the incremental scene and each base scene, and select the minimum value b among all the difference values. min ; Preset difference threshold B1; If b min >B1, generates Level 1 learning instructions; The first-level learning instructions include: Set minimum value b min The corresponding basic scenario is the anchor scenario of the incremental scenario; Multiple control points to be optimized are selected based on the anchoring scenario; Obtain monitoring feedback packages for incremental scenarios; Generate first-level mapping actions and first-level prediction models for incremental scenarios based on all control points to be optimized and monitoring feedback packages.

6. The deep learning-based robot cerebellar reinforcement learning method as described in claim 5, characterized in that, The selection of multiple control points to be optimized includes: Generate the dependency values ​​of the anchored scenario on each control point in the control network; Preset dependency threshold C1; If C1>ci (i=1,2…m), set the i-th control point as the control point to be optimized in the target anchoring scene; Where ci is the dependency value of the anchoring scene on the i-th control point; m is the number of control points in the control network.

7. A deep learning-based robot cerebellar reinforcement learning system, employing the deep learning-based robot cerebellar reinforcement learning method according to any one of claims 1-6, characterized in that, include: The simulation unit is used to preset the simulation environment and establish a basic control model. The basic control model includes: a feedforward sub-model and an initial error model; The central control unit is used to deploy the basic control model into the robot; The central control unit includes: The perception module is used to generate training data packages based on the robot's operational monitoring data; The sensing module is also used to acquire environmental sensing data; The first learning module is used to generate iterative instructions for the basic control model based on the training data package, and to generate the running control model based on the iteration results. The second learning module is used to determine whether to generate learning instructions for the operation control model based on environmental perception data; The simulation unit includes: The first simulation module is used to construct a control network based on the robot's structural parameters, and the control network includes multiple control points. Each movable node is set as a control point based on the robot's structural parameters; The second simulation module is used to set up multiple basic scenarios based on historical operational requirements; Establish a sequence of basic scenarios A, A=(a1, a2, ..., ai, ..., an), where ai is the i-th basic scenario; n is the number of basic scenarios; Based on the basic scenario sequence A, ai is sequentially set as the target basic scenario; The initial mapping actions of the control network in the target basic scenario are generated based on the simulation environment; Establish a predictive sub-model for the target basic scenario; The initial mapping actions of the control network in each basic scenario are generated sequentially, and the feedforward sub-model is generated based on all the initial mapping actions; Obtain the prediction sub-models for each basic scenario, and generate an initial error model based on all the prediction sub-models; The basic control model is generated based on the feedforward sub-model and the initial error model.

8. The deep learning-based robotic cerebellar reinforcement learning system as described in claim 7, characterized in that, The sensing module is also used for: Deploy the basic control model into the robot; The feedforward sub-model acquires the primary environmental data collected by the robot; Generate actions to be performed and data identification tags based on primary environmental data; The initial error model generates expected error parameters based on primary environmental data and the actions to be performed; Generate compensation instructions based on the expected error parameters; Generate a first-level execution action based on the compensation instruction and the action to be executed; Obtain the actual error parameters; Generate a record sub-package of the first-level execution action, and generate a single monitoring feedback package based on the record sub-package and data identification tags; Generate training data packets based on all monitoring feedback packets; This includes generating actions to be performed and data identification tags, including: Similarity values ​​are generated based on primary environmental data and various basic scenarios; The scene corresponding to the maximum value among all similarity values ​​is designated as the first-level scene. Set the initial mapping action corresponding to the first-level scene as the action to be executed; Set the scene label of the first-level scene as the data recognition label.

Images