Robust reinforcement learning control method and device based on prediction error feedback

Abstract

The present application relates to the field of robot control technology, and in particular to a robust reinforcement learning control method and device based on prediction error feedback, wherein the method comprises: simplifying the structure of a target robot into a single rigid body; predicting the operating state of the single rigid body according to the output instruction of the reinforcement learning strategy; obtaining the actual operating state of the target robot when executing the output instruction, and calculating the prediction error between the predicted operating state and the actual operating state; using the prediction error and the actual operating state as feedback signals of the reinforcement learning strategy, and the reinforcement learning strategy outputs instructions based on the feedback signal and the task control instruction. Thus, the present application solves the problems that the related technologies face in dealing with nonlinear and unstructured disturbance environments, are limited by the uncertainty of the prediction model, and are difficult to adjust the manually designed disturbance estimator.

Description

Technical Field

[0001] The present application relates to the field of robot control technology, and in particular to a robust reinforcement learning control method and device based on prediction error feedback. Background Art

[0002] Maintaining stable and precise motion of robots in complex and disturbed environments is a key requirement in many applications, such as maintaining stable flight of quadcopters in wind fields and carrying loads on different terrains by quadruped robots. This challenge is called robust control or adaptive control and has been widely studied. Classical adaptive control methods compensate for dynamic environmental changes by adjusting control parameters in real time, usually based on mathematical models of the system and disturbances; while robust control methods use more informative control inputs to enhance robustness, such as IMC (Internal Model Control). Although these methods can ensure the stability of the system, they rely on accurate modeling of the system dynamics, which is very difficult for nonlinear systems and environments with unstructured disturbances.

[0003] In contrast, in recent years, reinforcement learning (RL) methods have demonstrated remarkable success in the continuous control of various robotic forms, including skill learning for robotic arms, high-speed flight control for quadrotors, and adaptive walking for quadrupeds on variable terrain. By randomizing dynamic and environmental parameters during large-scale training, RL control methods generalize across diverse conditions and environments, enabling robots to operate reliably in unknown and complex scenarios.

[0004] There has recently been a trend toward combining model-based control with RL neural control to leverage the strengths of both approaches. A representative framework incorporates predicted future states or estimated disturbances as additional input signals to obtain richer information to guide model-free RL policies. For example, a predictive encoder was trained via unsupervised contrastive learning to estimate the next state in latent space; a hand-designed L1 adaptive law was used to explicitly estimate applied force disturbances, resulting in a robust trajectory tracking controller capable of handling sudden wind events.

[0005] While this combined model-based control approach with RL holds great promise, it faces significant challenges. Predictive models exhibit significant uncertainty when the training data fails to capture the actual dynamics, while hand-crafted disturbance estimators require precise parameter tuning, limiting their applicability in complex systems and making them difficult to adapt to dynamic changes. Summary of the Invention

[0006] The present application provides a robust reinforcement learning control method and device based on prediction error feedback to address the challenges faced by related technologies in dealing with nonlinear and unstructured disturbance environments, the limitations of the uncertainty of the prediction model, and the difficulty of adjusting the manually designed disturbance estimator.

[0007] The first aspect of the present application provides a robust reinforcement learning control method based on prediction error feedback, comprising the following steps: simplifying the structure of a target robot into a single rigid body; predicting the operating state of the single rigid body according to the output instructions of the reinforcement learning strategy; obtaining the actual operating state of the target robot when executing the output instructions, and calculating the prediction error between the predicted operating state and the actual operating state; using the prediction error and the actual operating state as feedback signals of the reinforcement learning strategy, and the reinforcement learning strategy outputs instructions according to the feedback signal and the task control instructions.

[0008] Optionally, the structure of the target robot includes a main body part, a limb part and an actuator.

[0009] Optionally, simplifying the structure of the target robot into a single rigid body includes: ignoring the preset state of the actuator or limb and retaining the main body of the target robot; and treating the main body of the target robot as a single rigid body.

[0010] Optionally, the operating state includes position, speed, attitude and angular velocity, wherein:

[0011] The prediction formula for position is:

[0012]

[0013] Among them, p k is the position; v k is the speed; Δt is the time interval; a k is the acceleration instruction at the current moment;

[0014] The speed prediction formula is:

[0015]

[0016] The update formula of the posture is:

[0017]

[0018] Among them, q k is the attitude expressed by quaternion; represents the multiplication of quaternions; Δθ is the angle change, ω k is the angular velocity; is the angular acceleration command; q ΔθTo generate the rotation quaternion based on Δθ, the exponential mapping method is used for calculation, and the calculation formula is:

[0019]

[0020] The update formula for angular velocity is:

[0021]

[0022] Optionally, the prediction error is calculated as:

[0023]

[0024] The predicted operating status is The actual running status is x k+1 =[p k+1 ,q k+1 , v k+1 ,ω k+1 ]; is the position prediction error; is the speed prediction error; is the angular velocity prediction error; For posture differences; To predict quaternion and the real quaternion q k+1 The angle difference between them.

[0025] Optionally, the target robot is trained using a reinforcement learning algorithm that approximates policy gradients.

[0026] Optionally, the target robot is a real robot or a simulated robot.

[0027] The second aspect of the present application provides a robust reinforcement learning control device based on prediction error feedback, including: a simplification module, used to simplify the structure of the target robot into a single rigid body; a first output module, used to predict the operating state of the single rigid body according to the output instructions of the reinforcement learning strategy; a calculation module, used to obtain the actual operating state of the target robot when executing the output instructions, and calculate the prediction error between the predicted operating state and the actual operating state; a second output module, used to use the prediction error and the actual operating state as feedback signals of the reinforcement learning strategy, and the reinforcement learning strategy outputs instructions according to the feedback signal and the task control instructions.

[0028] The third aspect of the present application provides a robot, comprising: a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein the processor executes the program to implement the robust reinforcement learning control method based on prediction error feedback of the first aspect.

[0029] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program or instruction stored thereon. When the computer program or instruction is executed, the robust reinforcement learning control method based on prediction error feedback of the first aspect is implemented.

[0030] Therefore, this application has the following beneficial effects:

[0031] The embodiment of the present application simplifies the structure of the target robot into a single rigid body, predicts the operating state of the single rigid body based on the output instructions of the reinforcement learning strategy, and simultaneously obtains the actual operating state of the target robot when executing the output instructions. The prediction error between the predicted operating state and the actual operating state is calculated, and the prediction error and the actual operating state are used as feedback signals of the reinforcement learning strategy. The reinforcement learning strategy outputs instructions based on the feedback signal and the task control instructions, effectively improving the robustness and generalization ability of the strategy, avoiding manual adjustment of complex parameters, and having high practicality and scalability. Thus, it solves the challenges faced by related technologies in handling nonlinear and unstructured disturbance environments, being limited by the uncertainty of the prediction model, and the difficulty of adjusting the manually designed disturbance estimator.

[0032] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0034] Figure 1 Flowchart of a robust reinforcement learning control method based on prediction error feedback provided according to an embodiment of the present application;

[0035] Figure 2 A schematic diagram of improving the robustness of a control strategy by calculating the error between a predicted action and an actual action according to an embodiment of the present application;

[0036] Figure 3 A schematic diagram of a closed-loop "neural internal model control" framework for calculating the next state of a subject using an explicit prediction model according to one embodiment of the present application;

[0037] Figure 4 This is an example diagram of a robust reinforcement learning control device based on prediction error feedback provided according to an embodiment of the present application;

[0038] Figure 5 Schematic diagram of the structure of a robot provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] The following describes in detail embodiments of the present application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0040] The following describes a robust reinforcement learning control method and device based on prediction error feedback of an embodiment of the present application with reference to the accompanying drawings. In view of the challenges faced by the related technologies mentioned in the above background technology in handling nonlinear and unstructured disturbance environments, the problems of being limited by the uncertainty of the prediction model and the difficulty of adjusting the manually designed disturbance estimator, the present application provides a robust reinforcement learning control method based on prediction error feedback, in which the structure of the target robot is simplified into a single rigid body, and the operating state of the single rigid body is predicted according to the output instruction of the reinforcement learning strategy, and the actual operating state of the target robot when executing the output instruction is obtained at the same time, the prediction error between the predicted operating state and the actual operating state is calculated, and the prediction error and the actual operating state are used as the feedback signal of the reinforcement learning strategy. The reinforcement learning strategy outputs instructions according to the feedback signal and the task control instruction, effectively improving the robustness and generalization ability of the strategy, avoiding manual adjustment of complex parameters, and having high practicality and scalability. Thus, the problems faced by the related technologies in handling nonlinear and unstructured disturbance environments, being limited by the uncertainty of the prediction model and the difficulty of adjusting the manually designed disturbance estimator are solved.

[0041] Specifically, Figure 1 A flowchart of a robust reinforcement learning control method based on prediction error feedback provided in an embodiment of the present application.

[0042] like Figure 1 As shown in FIG, the robust reinforcement learning control method based on prediction error feedback includes the following steps:

[0043] In step S101 , the structure of the target robot is simplified into a single rigid body.

[0044] Among them, a single rigid body refers to a single rigid body that is not connected by components such as joints, motors or hinges. That is, the robot model is simplified, the movement of joints and motors is ignored, and it is only considered as a whole rigid body.

[0045] It is understandable that the embodiment of the present application simplifies the structure of the target robot into a single rigid body, that is, it assumes that all components of the target robot are completely rigid, ignores the movement of joints and motors, and only considers it as a whole rigid body.

[0046] In the embodiment of the present application, the target robot is a real robot or a simulated robot.

[0047] Among them, real robots refer to physical robots that exist in physical space and are composed of hardware, including mechanical structures, sensors, actuators and other electronic components; simulated robots are robot models created in a computer simulation environment, which imitate the behavior and characteristics of real robots through software.

[0048] It is understandable that the target robot in the embodiment of the present application can be a physical robot existing in a physical space, or it can be a robot model created in a computer simulation environment.

[0049] In the embodiment of the present application, the structure of the target robot includes a main body, limbs and an actuator.

[0050] It can be understood that the target robot structure of the embodiment of the present application is divided into three parts, namely the main body part, the limb part and the actuator.

[0051] In an embodiment of the present application, the structure of the target robot is simplified into a single rigid body, including: ignoring the preset state of the actuator or limb, retaining the main body of the target robot; and treating the main body of the target robot as a single rigid body.

[0052] The preset state of the actuator or limb is a state of a pre-set position, angle, or other parameters, which can be static or dynamic, depending on the application requirements and is not specifically limited here.

[0053] It can be understood that when the embodiment of the present application simplifies the structure of the target robot into a single rigid body, it is necessary to identify the main body, limbs and actuators of the target robot, ignore the preset state of the actuator or limb, retain the main body, and treat the main body as a single rigid body. This simplified processing method can significantly reduce the need for complex dynamic modeling.

[0054] In an embodiment of the present application, the target robot is trained using a reinforcement learning algorithm with approximate policy gradient.

[0055] Among them, the reinforcement learning algorithm with approximate policy gradient is a method used to train target robots to make decisions in complex environments. By first defining the task objectives and reward functions, the robot is allowed to repeatedly try actions in the environment. Through parallel large-scale environmental sampling, the robot can quickly explore different strategies and learn efficiently, significantly accelerating the training process. At the same time, the reinforcement learning algorithm with approximate policy gradient supports randomizing parameters, allowing the robot to learn in a variety of training scenarios, thereby improving the generalization ability and robustness of the strategy in practical applications, which not only ensures the stability of learning, but also is suitable for efficient training of complex tasks.

[0056] It can be understood that the target robot of the embodiment of the present application utilizes a reinforcement learning algorithm that approximates policy gradients. By first defining the task objectives and reward functions, the robot is allowed to repeatedly try actions in the environment. Through environmental sampling, the robot can quickly explore different strategies and learn efficiently, significantly accelerating the training process, improving the generalization ability and robustness of the strategy in practical applications, and ensuring the stability of learning, which is suitable for efficient training of complex tasks.

[0057] In step S102 , the operating state of the single rigid body is predicted according to the output instruction of the reinforcement learning strategy.

[0058] In the embodiment of the present application, the operating state includes position, speed, attitude and angular velocity, wherein:

[0059] The prediction formula for position is:

[0060]

[0061] Among them, p k is the position; v k is the speed; Δt is the time interval; a k is the acceleration instruction at the current moment;

[0062] The speed prediction formula is:

[0063]

[0064] The update formula of the posture is:

[0065]

[0066] Among them, q k is the attitude expressed by quaternion; represents the multiplication of quaternions; Δθ is the angle change, ω k is the angular velocity; is the angular acceleration command; q Δθ To generate the rotation quaternion based on Δθ, the exponential mapping method is used for calculation, and the calculation formula is:

[0067]

[0068] The update formula for angular velocity is:

[0069]

[0070] In step S103, the actual operating state of the target robot when executing the output instruction is obtained, and the prediction error between the predicted operating state and the actual operating state is calculated.

[0071] The actual operating state refers to the actual state reached by the robot after executing output commands in the actual environment, which is usually directly measured by sensors such as encoders, IMUs, and cameras. The predicted operating state is derived based on Newton's laws of motion and the kinematics of single rigid bodies, which will be described in detail below and will not be repeated here.

[0072] It can be understood that the embodiments of the present application can directly measure the actual operating state of the target robot when executing the output instruction through sensors such as encoders, IMUs, cameras, etc., and calculate the prediction error between the predicted operating state and the actual operating state based on Newton's laws of motion and the single rigid body kinematic formula. The calculation method will be described in detail below and will not be repeated here.

[0073] In the embodiment of the present application, the calculation formula of the prediction error is:

[0074]

[0075] The predicted operating status is The actual running status is x k+1 =[p k+1 ,q k+1 , v k+1 ,ω k+1 ]; is the position prediction error; v k+1 is the speed prediction error; is the angular velocity prediction error; For posture differences; To predict quaternion and the real quaternion q k+1 The angle difference between them.

[0076] In step S104, the prediction error and the actual operating state are used as feedback signals of the reinforcement learning strategy, and the reinforcement learning strategy outputs instructions according to the feedback signal and the task control instructions.

[0077] It can be understood that the embodiment of the present application inputs the prediction error together with the actual operating status as feedback signals into the reinforcement learning strategy. The reinforcement learning algorithm is trained based on these feedback signals and given task control instructions using model-free reinforcement learning algorithms such as approximate policy gradients.

[0078] According to the robust reinforcement learning control method based on prediction error feedback proposed in the embodiment of the present application, the structure of the target robot is simplified into a single rigid body, and the operating state of the single rigid body is predicted according to the output instructions of the reinforcement learning strategy. At the same time, the actual operating state of the target robot when executing the output instructions is obtained, and the prediction error between the predicted operating state and the actual operating state is calculated. The prediction error and the actual operating state are used as feedback signals of the reinforcement learning strategy. The reinforcement learning strategy outputs instructions according to the feedback signal and the task control instructions, which effectively improves the robustness and generalization ability of the strategy, avoids manual adjustment of complex parameters, and has high practicality and scalability.

[0079] The following is a further description of the robust reinforcement learning control method based on prediction error feedback through a specific embodiment. Figure 2 As shown, the prediction error may be caused by system modeling mismatch and / or external disturbances. The robustness of the control strategy can be improved by calculating the error between the predicted action and the actual action. This embodiment proposes a general framework that bridges model-based control and reinforcement learning control, named neural internal model control, as shown in Figure 3 As shown in Figure 2, the framework consists of two parts: a simplified prediction model and a reinforcement learning control strategy combined with prediction error feedback, as follows:

[0080] 1. Simplify the prediction model

[0081] The core of this embodiment is to design a simplified and efficient prediction model to detect and compensate for system deviations and disturbances. To reduce modeling complexity, this embodiment models the robot as a single rigid body and predicts only the state of the robot's main body, ignoring the specific states of actuators or limbs. This approach significantly reduces the need for complex dynamic modeling and is particularly suitable for robots equipped with cameras and inertial measurement units (IMUs), leveraging existing visual inertial odometry technology to obtain stable state estimates.

[0082] In order to effectively predict the next state of the robot body, this embodiment derives the predicted values ​​of the robot's posture and speed based on Newton's laws of motion and the single rigid body kinematics formula. Assume that the current state of the body is where p k is the position, q k is the attitude (quaternion representation), v k is the speed, ω k is the angular velocity and the time interval is Δt. The next state prediction is as follows:

[0083] 1) Position and velocity prediction:

[0084] According to Newton's laws, the predicted position and velocity are as follows:

[0085]

[0086] where a k is the acceleration command at the current moment.

[0087] 2) Angle change calculation:

[0088] Calculate the change in angle using angular velocity and angular acceleration:

[0089]

[0090] in is the angular acceleration command.

[0091] 3) Attitude update (quaternion):

[0092] Generate the rotation quaternion q according to Δθ Δθ , using the exponential mapping method:

[0093]

[0094] Then update the posture to:

[0095]

[0096] Where ∈ represents the floating point calculation precision of the computer, Represents a quaternion multiplication operation.

[0097] 4) Angular velocity update:

[0098] Predict the angular velocity based on the angular acceleration command:

[0099]

[0100] Finally, the predicted state for the next step is:

[0101]

[0102] Through the above-formulated method, the simplified model proposed in the embodiment of the present application can efficiently predict the next state, providing a basis for subsequent error feedback and control strategy optimization.

[0103] 2. Reinforcement learning control strategy combined with prediction error feedback

[0104] After obtaining the predicted state, this embodiment calculates the difference between the predicted value and the actual system response to form an explicit prediction error, which is further introduced into the reinforcement learning strategy through a feedback mechanism as an effective feedback signal, thereby improving the adaptability to complex dynamic environments.

[0105] The following is a section describing how to calculate the prediction error with specific formulas:

[0106] Assume that the next predicted state is:

[0107]

[0108] in is the predicted position, is the predicted pose (quaternion representation), is the predicted speed, is the predicted angular velocity; the actual subject state is:

[0109] x k+1 =[p k+1 ,q k+1 , v k+1 ,ω k+1 ].

[0110] Based on the difference between the predicted value and the true value, the prediction error is defined as:

[0111]

[0112] in: is the position prediction error; is the speed prediction error; is the angular velocity prediction error. The attitude difference is calculated by cosine distance: in To predict quaternion and the real quaternion q k+1 The angle difference between them.

[0113] The prediction error vector calculated by the above formula is This is fed back into the control strategy, providing an accurate and dynamic error signal, thereby enhancing the robustness and adaptability of the control system. In addition to receiving prediction error feedback, the control strategy also receives the robot's state information and task control instructions as input, and is trained using model-free reinforcement learning algorithms such as Proximal Policy Gradient (PPO).

[0114] Next, a robust reinforcement learning control device based on prediction error feedback proposed in accordance with an embodiment of the present application will be described with reference to the accompanying drawings.

[0115] Figure 4 It is a block diagram of a robust reinforcement learning control device based on prediction error feedback according to an embodiment of the present application.

[0116] like Figure 4 As shown, the robust reinforcement learning control device 10 based on prediction error feedback includes: a simplification module 201 , a first output module 202 , a calculation module 203 and a second output module 204 .

[0117] Among them, the simplification module 201 is used to simplify the structure of the target robot into a single rigid body; the first output module 202 is used to predict the operating state of the single rigid body according to the output instructions of the reinforcement learning strategy; the calculation module 203 is used to obtain the actual operating state of the target robot when executing the output instructions, and calculate the prediction error between the predicted operating state and the actual operating state; the second output module 204 is used to use the prediction error and the actual operating state as feedback signals of the reinforcement learning strategy, and the reinforcement learning strategy outputs instructions according to the feedback signal and the task control instructions.

[0118] In the embodiment of the present application, the structure of the target robot includes a main body, limbs and an actuator.

[0119] In an embodiment of the present application, the simplification module 201 is further used to: simplify the structure of the target robot into a single rigid body, including ignoring the preset state of the actuator or limb and retaining the main body of the target robot; and treat the main body of the target robot as a single rigid body.

[0120] In the embodiment of the present application, the operating state includes position, speed, attitude and angular velocity, wherein:

[0121] The prediction formula for position is:

[0122]

[0123] Among them, p k is the position; v k is the speed; Δt is the time interval; a k is the acceleration instruction at the current moment;

[0124] The speed prediction formula is:

[0125]

[0126] The update formula of the posture is:

[0127]

[0128] Among them, q k is the attitude expressed by quaternion; represents the multiplication of quaternions; Δθ is the angle change, ω k is the angular velocity; is the angular acceleration command; q Δθ To generate the rotation quaternion based on Δθ, the exponential mapping method is used for calculation, and the calculation formula is:

[0129]

[0130] The update formula for angular velocity is:

[0131]

[0132] In the embodiment of the present application, the calculation formula of the prediction error is:

[0133]

[0134] The predicted operating status is The actual running status is x k+1 =[p k+1 ,q k+1 , v k+1 ,ω k+1 ]; is the position prediction error; is the speed prediction error; is the angular velocity prediction error; For posture differences; To predict quaternion and the real quaternion q k+1 The angle difference between them.

[0135] In an embodiment of the present application, the target robot is trained using a reinforcement learning algorithm with approximate policy gradient.

[0136] In the embodiment of the present application, the target robot is a real robot or a simulated robot.

[0137] It should be noted that the above explanation of the embodiment of the robust reinforcement learning control method based on prediction error feedback is also applicable to the robust reinforcement learning control device based on prediction error feedback in this embodiment, and will not be repeated here.

[0138] According to the robust reinforcement learning control device based on prediction error feedback proposed in the embodiment of the present application, the structure of the target robot is simplified into a single rigid body, and the operating state of the single rigid body is predicted according to the output instructions of the reinforcement learning strategy. At the same time, the actual operating state of the target robot when executing the output instructions is obtained, and the prediction error between the predicted operating state and the actual operating state is calculated. The prediction error and the actual operating state are used as feedback signals of the reinforcement learning strategy. The reinforcement learning strategy outputs instructions according to the feedback signal and the task control instructions, which effectively improves the robustness and generalization ability of the strategy, avoids manual adjustment of complex parameters, and has high practicality and scalability.

[0139] Figure 5 This is a schematic diagram of the structure of a robot provided in an embodiment of the present application. The robot may include:

[0140] Memory 301 , processor 302 , and computer programs stored in the memory 301 and executable on the processor 302 .

[0141] When the processor 302 executes the program, the robust reinforcement learning control method based on prediction error feedback provided in the above embodiment is implemented.

[0142] Furthermore, the robot also includes:

[0143] The communication interface 303 is used for communication between the memory 301 and the processor 302 .

[0144] The memory 301 is used to store computer programs that can be run on the processor 302 .

[0145] The memory 301 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0146] If the memory 301, processor 302, and communication interface 303 are implemented independently, the communication interface 303, memory 301, and processor 302 can be connected to each other via a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0147] Optionally, in a specific implementation, if the memory 301, the processor 302 and the communication interface 303 are integrated on a chip, the memory 301, the processor 302 and the communication interface 303 can communicate with each other through an internal interface.

[0148] The processor 302 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0149] An embodiment of the present application also provides a computer-readable storage medium having a computer program or instruction stored thereon. When the computer program or instruction is executed, the above-mentioned robust reinforcement learning control method based on prediction error feedback is implemented.

[0150] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0151] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0152] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing a custom logical function or process step, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed in a different order than shown or discussed, including performing functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0153] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, the steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement the method: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0154] Those skilled in the art will understand that all or part of the steps carried out in the method for implementing the above-mentioned embodiment can be completed by instructing the relevant hardware through a program, and the above-mentioned program can be stored in a computer-readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0155] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A robust reinforcement learning control method based on prediction error feedback, characterized in that: The following steps are involved: Simplify the structure of the target robot into a single rigid body; The operating state of the single rigid body is predicted according to the output instruction of the reinforcement learning strategy, where the operating state includes position, velocity, posture and angular velocity, wherein the prediction formula of the position is: in, for location; for speed; is the time interval; is the acceleration instruction at the current moment; The prediction formula for the speed is: The update formula of the posture is: ， in, is the attitude expressed by quaternion; Represents quaternion multiplication operation; is the angle change, , is the angular velocity; is the angular acceleration instruction; Based on Generate rotation quaternion and use exponential mapping method to calculate. The calculation formula is: The updating formula of the angular velocity is: ； The actual running state of the target robot when executing the output instruction is obtained, and the prediction error between the predicted running state and the actual running state is calculated. The calculation formula of the prediction error is: The predicted operating status is ; The actual operating status is ; is the position prediction error; is the speed prediction error; is the angular velocity prediction error; For posture differences; To predict quaternion and real quaternions The angle difference between The prediction error and the actual operating state are used as feedback signals of the reinforcement learning strategy, and the reinforcement learning strategy outputs instructions according to the feedback signal and the task control instruction.

2. The robust reinforcement learning control method based on prediction error feedback according to claim 1, characterized in that: The structure of the target robot includes a main body, limbs and an actuator.

3. The robust reinforcement learning control method based on prediction error feedback according to claim 2, characterized in that: The method of simplifying the structure of the target robot into a single rigid body includes: Ignoring the preset state of the actuator or limb and retaining the main body of the target robot; The main body of the target robot is used as the single rigid body.

4. The robust reinforcement learning control method based on prediction error feedback according to claim 1, characterized in that: The target robot is trained using a reinforcement learning algorithm with approximate policy gradient.

5. The robust reinforcement learning control method based on prediction error feedback according to any one of claims 1 to 4, characterized in that: The target robot is a real robot or a simulated robot.

6. A robust reinforcement learning control device based on prediction error feedback, characterized in that: The device is used to implement the robust reinforcement learning control method based on prediction error feedback according to any one of claims 1 to 5, comprising: A simplification module is used to simplify the structure of the target robot into a single rigid body; The first output module is configured to predict the operating state of the single rigid body according to the output instruction of the reinforcement learning strategy, wherein the operating state includes position, velocity, posture, and angular velocity, wherein the prediction formula of the position is: in, for location; for speed; is the time interval; is the acceleration instruction at the current moment; The prediction formula for the speed is: The update formula of the posture is: ， in, is the attitude expressed by quaternion; Represents quaternion multiplication operation; is the angle change, , is the angular velocity; is the angular acceleration instruction; Based on Generate rotation quaternion and use exponential mapping method to calculate. The calculation formula is: The updating formula of the angular velocity is: ； The calculation module is used to obtain the actual operating state of the target robot when executing the output instruction, and calculate the prediction error between the predicted operating state and the actual operating state. The calculation formula of the prediction error is: The predicted operating status is ; The actual operating status is ; is the position prediction error; is the speed prediction error; is the angular velocity prediction error; For posture differences; To predict quaternion and real quaternions The angle difference between The second output module is used to use the prediction error and the actual operating state as feedback signals of the reinforcement learning strategy, and the reinforcement learning strategy outputs instructions according to the feedback signal and the task control instruction.

7. A robot, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the robust reinforcement learning control method based on prediction error feedback according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed, the robust reinforcement learning control method based on prediction error feedback according to any one of claims 1 to 5 is implemented.

Images