Robust reinforcement learning control method and device based on prediction error feedback

Abstract

The invention relates to the technical field of robot control, in particular to a robust reinforcement learning control method and device based on prediction error feedback, and the method comprises the steps: simplifying the structure of a target robot into a single rigid body; predicting the operation state of the single rigid body according to an output instruction of the reinforcement learning strategy; obtaining a real operation state when the target robot executes the output instruction, and calculating a prediction error between the predicted operation state and the real operation state; and the prediction error and the real operation state are used as feedback signals of a reinforcement learning strategy, and the reinforcement learning strategy outputs an instruction according to the feedback signals and the task control instruction. Therefore, the problems that in the prior art, processing of nonlinear and unstructured disturbance environments faces challenges and is limited by uncertainty of a prediction model, and adjustment difficulty of manual design of a disturbance estimator is caused are solved.

Description

Technical Field

[0001] The present application relates to the field of robot control technology, and in particular to a robust reinforcement learning control method and device based on prediction error feedback. Background Art

[0002] Achieving stable and precise motion of robots in complex and disturbed environments is a key requirement in many applications, such as maintaining stable flight of quadcopters in wind fields and carrying loads on different terrains for quadruped robots. This challenge is known as the robust control or adaptive control problem and has been widely studied. Classic adaptive control methods compensate for dynamic environmental changes by adjusting control parameters in real time, usually based on mathematical models of the system and disturbances; while robust control methods use more informative control inputs to enhance robustness, such as IMC (Internal Model Control). Although these methods can ensure the stability of the system, they rely on accurate modeling of the system dynamics, which is very difficult for nonlinear systems and environments with unstructured disturbances.

[0003] In contrast, in recent years, RL (Reinforcement Learning) methods have shown remarkable results in the continuous control of various robot forms, including skill learning of robotic arms, high-speed flight control of quadrotors, and adaptive walking of quadruped robots on variable terrains. During large-scale training, randomized dynamics and environmental parameters enable RL control methods to generalize under different conditions and environments, allowing robots to operate reliably in unknown and complex scenarios.

[0004] Recently, there has been a trend to combine model-based control with RL neural control in order to leverage the strengths of both control approaches. A representative framework is to use predicted future states or estimated disturbances as additional input signals to obtain richer information to guide model-free RL policies. For example, a predictive encoder is trained through unsupervised contrastive learning to estimate the next state in the latent space; a hand-designed L1 adaptive law is used to explicitly estimate the applied force disturbance, thereby constructing a robust trajectory tracking controller that can cope with sudden wind fields.

[0005] Although this control method combining models and RL has good prospects, it faces significant challenges. The prediction model shows large uncertainty when the training data cannot cover the actual dynamic changes, and the manually designed disturbance estimator requires precise parameter adjustment, which limits its application in complex systems and makes it difficult to adapt to dynamic changes. Summary of the invention

[0006] The present application provides a robust reinforcement learning control method and device based on prediction error feedback to solve the challenges faced by related technologies in dealing with nonlinear and unstructured disturbance environments, the limitations of the uncertainty of the prediction model, and the difficulty of adjusting the manually designed disturbance estimator.

[0007] The first aspect of the present application provides a robust reinforcement learning control method based on prediction error feedback, comprising the following steps: simplifying the structure of a target robot into a single rigid body; predicting the operating state of the single rigid body according to the output instructions of a reinforcement learning strategy; obtaining the actual operating state of the target robot when executing the output instructions, and calculating the prediction error between the predicted operating state and the actual operating state; using the prediction error and the actual operating state as feedback signals of the reinforcement learning strategy, and the reinforcement learning strategy outputs instructions according to the feedback signal and the task control instructions.

[0008] Optionally, the structure of the target robot includes a main body part, a limb part and an actuator.

[0009] Optionally, simplifying the structure of the target robot into a single rigid body includes: ignoring the preset state of the actuator or limb and retaining the main body of the target robot; and treating the main body of the target robot as a single rigid body.

[0010] Optionally, the operating state includes position, speed, attitude and angular velocity, wherein:

[0011] The prediction formula for position is:

[0012]

[0013] Among them, p k is the position; v k is the speed; Δt is the time interval; a k is the acceleration instruction at the current moment;

[0014] The speed prediction formula is:

[0015]

[0016] The update formula of the posture is:

[0017]

[0018] Among them, q k is the attitude expressed by quaternion; represents the multiplication of quaternions; Δθ is the angle change, ω k is the angular velocity; is the angular acceleration command; q ΔθIn order to generate the rotation quaternion according to Δθ, the exponential mapping method is used for calculation, and the calculation formula is:

[0019]

[0020] The update formula of angular velocity is:

[0021]

[0022] Optionally, the prediction error is calculated as:

[0023]

[0024] The predicted operating status is The actual running status is x k+1 =[p k+1 ,q k+1 , v k+1 ,ω k+1 ]; is the position prediction error; is the speed prediction error; is the angular velocity prediction error; For posture differences; To predict the quaternion and the real quaternion q k+1 The angle difference between them.

[0025] Optionally, the target robot is trained using a reinforcement learning algorithm that approximates policy gradients.

[0026] Optionally, the target robot is a real robot or a simulated robot.

[0027] The second aspect of the present application provides a robust reinforcement learning control device based on prediction error feedback, including: a simplification module, used to simplify the structure of a target robot into a single rigid body; a first output module, used to predict the operating state of the single rigid body according to the output instructions of the reinforcement learning strategy; a calculation module, used to obtain the actual operating state of the target robot when executing the output instructions, and calculate the prediction error between the predicted operating state and the actual operating state; a second output module, used to use the prediction error and the actual operating state as feedback signals of the reinforcement learning strategy, and the reinforcement learning strategy outputs instructions according to the feedback signal and the task control instructions.

[0028] The third aspect of the present application provides a robot, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the robust reinforcement learning control method based on prediction error feedback of the first aspect.

[0029] The fourth aspect of the present application provides a computer-readable storage medium having a computer program or instruction stored thereon. When the computer program or instruction is executed, the robust reinforcement learning control method based on prediction error feedback of the first aspect is implemented.

[0030] Therefore, this application includes the following beneficial effects:

[0031] The embodiment of the present application simplifies the structure of the target robot into a single rigid body, and predicts the operating state of the single rigid body according to the output instruction of the reinforcement learning strategy, and obtains the real operating state of the target robot when executing the output instruction, calculates the prediction error between the predicted operating state and the real operating state, and uses the prediction error and the real operating state as the feedback signal of the reinforcement learning strategy. The reinforcement learning strategy outputs instructions according to the feedback signal and the task control instruction, which effectively improves the robustness and generalization ability of the strategy, avoids manual adjustment of complex parameters, and has high practicality and scalability. Therefore, it solves the challenges faced by the related technology in dealing with nonlinear and unstructured disturbance environments, is limited by the uncertainty of the prediction model, and is difficult to adjust the manually designed disturbance estimator.

[0032] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0034] Figure 1 A flowchart of a robust reinforcement learning control method based on prediction error feedback provided according to an embodiment of the present application;

[0035] Figure 2 A schematic diagram of improving the robustness of a control strategy by calculating the error between a predicted action and an actual action according to an embodiment of the present application;

[0036] Figure 3 A schematic diagram of a closed-loop "neural internal model control" framework for calculating the next state of a subject using an explicit prediction model according to an embodiment of the present application;

[0037] Figure 4 This is an example diagram of a robust reinforcement learning control device based on prediction error feedback provided according to an embodiment of the present application;

[0038] Figure 5 The present invention is a schematic diagram of the structure of a robot provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0040] The following describes the robust reinforcement learning control method and device based on prediction error feedback of the embodiment of the present application with reference to the accompanying drawings. In view of the challenges faced by the related technologies mentioned in the above background technology when dealing with nonlinear and unstructured disturbance environments, limited by the uncertainty of the prediction model, and the difficulty of adjusting the manually designed disturbance estimator, the present application provides a robust reinforcement learning control method based on prediction error feedback, in which the structure of the target robot is simplified to a single rigid body, and the running state of the single rigid body is predicted according to the output instruction of the reinforcement learning strategy, and the real running state of the target robot when executing the output instruction is obtained at the same time, the prediction error between the predicted running state and the real running state is calculated, and the prediction error and the real running state are used as the feedback signal of the reinforcement learning strategy, and the reinforcement learning strategy outputs instructions according to the feedback signal and the task control instruction, which effectively improves the robustness and generalization ability of the strategy, avoids manual adjustment of complex parameters, and has high practicality and scalability. Thus, the challenges faced by the related technologies when dealing with nonlinear and unstructured disturbance environments, limited by the uncertainty of the prediction model, and the difficulty of adjusting the manually designed disturbance estimator are solved.

[0041] Specifically, Figure 1 A flowchart of a robust reinforcement learning control method based on prediction error feedback provided in an embodiment of the present application.

[0042] like Figure 1 As shown, the robust reinforcement learning control method based on prediction error feedback includes the following steps:

[0043] In step S101 , the structure of the target robot is simplified into a single rigid body.

[0044] Among them, a single rigid body refers to a single rigid body that is not connected by components such as joints, motors or hinges, that is, the robot model is simplified, the movement of joints and motors is ignored, and it is only considered as a whole rigid body.

[0045] It is understandable that the embodiment of the present application simplifies the structure of the target robot into a single rigid body, that is, it is assumed that all components of the target robot are completely rigid, ignoring the movements of joints and motors, etc., and only considering it as an integral rigid body.

[0046] In the embodiment of the present application, the target robot is a real robot or a simulated robot.

[0047] Among them, real robots refer to those physical robots that exist in physical space and are composed of hardware, including mechanical structures, sensors, actuators and other electronic components; simulated robots are robot models created in a computer simulation environment, which use software to imitate the behavior and characteristics of real robots.

[0048] It can be understood that the target robot in the embodiment of the present application can be a physical robot existing in the physical space, or it can be a robot model created in a computer simulation environment.

[0049] In the embodiment of the present application, the structure of the target robot includes a main body part, a limb part and an actuator.

[0050] It can be understood that the target robot structure of the embodiment of the present application is divided into three parts, namely a main body part, a limb part and an actuator.

[0051] In an embodiment of the present application, the structure of the target robot is simplified into a single rigid body, including: ignoring the preset state of the actuator or limb and retaining the main body of the target robot; and treating the main body of the target robot as a single rigid body.

[0052] Among them, the preset state of the actuator or limb is a state of a preset position, angle, or other parameters, which can be static or dynamic, depending on the application requirements and is not specifically limited here.

[0053] It can be understood that when the embodiment of the present application simplifies the structure of the target robot into a single rigid body, it is necessary to identify the main body, limbs and actuators of the target robot, ignore the preset state of the actuator or limb, retain the main body, and treat the main body as a single rigid body. This simplified processing method can significantly reduce the need for complex dynamic modeling.

[0054] In an embodiment of the present application, the target robot is trained using a reinforcement learning algorithm that approximates policy gradients.

[0055] Among them, the reinforcement learning algorithm of approximate policy gradient is a method for training target robots to make decisions in complex environments. By first defining the task objectives and reward functions, the robot is allowed to repeatedly try actions in the environment. Through parallel large-scale environmental sampling, the robot can quickly explore different strategies and learn efficiently, which significantly accelerates the training process. At the same time, the reinforcement learning algorithm of approximate policy gradient supports randomizing parameters, allowing the robot to learn in a variety of training scenarios, thereby improving the generalization ability and robustness of the strategy in practical applications, which not only ensures the stability of learning, but also is suitable for efficient training of complex tasks.

[0056] It can be understood that the target robot of the embodiment of the present application utilizes a reinforcement learning algorithm of approximate policy gradient. By first defining the task objectives and reward functions, the robot is allowed to repeatedly try actions in the environment. Through environmental sampling, the robot can quickly explore different strategies and learn efficiently, which significantly accelerates the training process, improves the generalization ability and robustness of the strategy in practical applications, and ensures the stability of learning, which is suitable for efficient training of complex tasks.

[0057] In step S102, the operating state of the single rigid body is predicted according to the output instruction of the reinforcement learning strategy.

[0058] In the embodiment of the present application, the operating state includes position, speed, attitude and angular velocity, among which:

[0059] The prediction formula for position is:

[0060]

[0061] Among them, p k is the position; v k is the speed; Δt is the time interval; a k is the acceleration instruction at the current moment;

[0062] The speed prediction formula is:

[0063]

[0064] The update formula of the posture is:

[0065]

[0066] Among them, q k is the attitude expressed by quaternion; represents the multiplication of quaternions; Δθ is the angle change, ω k is the angular velocity; is the angular acceleration command; q Δθ In order to generate the rotation quaternion according to Δθ, the exponential mapping method is used for calculation, and the calculation formula is:

[0067]

[0068] The update formula of angular velocity is:

[0069]

[0070] In step S103, the actual operating state of the target robot when executing the output instruction is obtained, and the prediction error between the predicted operating state and the actual operating state is calculated.

[0071] Among them, the actual operating state refers to the actual state reached by the robot after executing the output command in the actual environment, which is usually directly measured by sensors such as encoders, IMUs, cameras, etc.; the predicted operating state is derived based on Newton's laws of motion and monomer rigid body kinematics, which will be described in detail below and will not be repeated here.

[0072] It can be understood that the embodiments of the present application can directly measure the actual operating state of the target robot when executing output instructions through sensors such as encoders, IMUs, cameras, etc., and calculate the prediction error between the predicted operating state and the actual operating state based on Newton's laws of motion and the monomer rigid body kinematics formula. The calculation method will be described in detail below and will not be repeated here.

[0073] In the embodiment of the present application, the calculation formula of the prediction error is:

[0074]

[0075] The predicted operating status is The actual running status is x k+1 =[p k+1 ,q k+1 , v k+1 ,ω k+1 ]; is the position prediction error; v k+1 is the speed prediction error; is the angular velocity prediction error; For posture differences; To predict the quaternion and the real quaternion q k+1 The angle difference between them.

[0076] In step S104, the prediction error and the actual operating state are used as feedback signals of the reinforcement learning strategy, and the reinforcement learning strategy outputs instructions according to the feedback signal and the task control instruction.

[0077] It can be understood that the embodiment of the present application inputs the prediction error together with the actual operating status as feedback signals into the reinforcement learning strategy, and the reinforcement learning algorithm is trained based on these feedback signals and given task control instructions using model-free reinforcement learning algorithms such as approximate policy gradients.

[0078] According to the robust reinforcement learning control method based on prediction error feedback proposed in the embodiment of the present application, the structure of the target robot is simplified into a single rigid body, and the operating state of the single rigid body is predicted according to the output instructions of the reinforcement learning strategy. At the same time, the actual operating state of the target robot when executing the output instructions is obtained, and the prediction error between the predicted operating state and the actual operating state is calculated. The prediction error and the actual operating state are used as feedback signals of the reinforcement learning strategy. The reinforcement learning strategy outputs instructions according to the feedback signal and the task control instructions, which effectively improves the robustness and generalization ability of the strategy, avoids manual adjustment of complex parameters, and has high practicality and scalability.

[0079] The following is a further description of the robust reinforcement learning control method based on prediction error feedback through a specific embodiment. Figure 2 As shown, the prediction error may be caused by system modeling mismatch and / or external disturbance. The robustness of the control strategy can be improved by calculating the error between the predicted action and the actual action. This embodiment proposes a general framework that bridges model-based control and reinforcement learning control, named neural internal model control, such as Figure 3 As shown in Figure 1, the framework consists of two parts: a simplified prediction model and a reinforcement learning control strategy combined with prediction error feedback, as follows:

[0080] 1. Simplify the prediction model

[0081] The core of this embodiment is to design a simplified and efficient prediction model to detect and compensate for system deviations and disturbances. To reduce the modeling complexity, this embodiment models the robot as a single rigid body and only predicts the state of the robot's main body, while ignoring the specific state of the actuators or limbs. In this way, this embodiment can significantly reduce the need for complex dynamic modeling, especially for robots equipped with cameras and inertial measurement units (IMUs), using existing visual inertial odometry technology to obtain stable state estimates.

[0082] In order to effectively predict the next state of the robot body, this embodiment derives the predicted values ​​of the robot's position and speed based on Newton's laws of motion and the single rigid body kinematics formula. Assume that the current state of the body is where p k is the position, q k is the attitude (quaternion representation), v k is the speed, ω k is the angular velocity, and the time interval is Δt. The next state prediction is as follows:

[0083] 1) Position and velocity prediction:

[0084] According to Newton's laws, the predicted position and velocity are as follows:

[0085]

[0086] where a k is the acceleration command at the current moment.

[0087] 2) Angle change calculation:

[0088] Calculate the change in angle using angular velocity and angular acceleration:

[0089]

[0090] in is the angular acceleration command.

[0091] 3) Attitude update (quaternion):

[0092] Generate the rotation quaternion q according to Δθ Δθ , using the exponential mapping method:

[0093]

[0094] Then update the posture to:

[0095]

[0096] Where ∈ represents the floating point calculation accuracy of the computer, Represents a quaternion multiplication operation.

[0097] 4) Angular velocity update:

[0098] Predict the angular velocity based on the angular acceleration command:

[0099]

[0100] Finally, the predicted state for the next step is:

[0101]

[0102] Through the above-formulated method, the simplified model proposed in the embodiment of the present application can efficiently predict the next state, providing a basis for subsequent error feedback and control strategy optimization.

[0103] 2. Reinforcement learning control strategy combined with prediction error feedback

[0104] After obtaining the predicted state, this embodiment calculates the difference between the predicted value and the actual system response to form an explicit prediction error, which is further introduced into the reinforcement learning strategy through a feedback mechanism as an effective feedback signal, thereby improving the adaptability to complex dynamic environments.

[0105] The following is a section describing how to calculate the prediction error with specific formulas:

[0106] Assume that the next predicted subject state is:

[0107]

[0108] in is the predicted position, is the predicted attitude (quaternion representation), is the predicted speed, is the predicted angular velocity; the actual subject state is:

[0109] x k+1 =[p k+1 ,q k+1 , v k+1 ,ω k+1 ].

[0110] Based on the difference between the predicted value and the true value, the prediction error is defined as:

[0111]

[0112] in: is the position prediction error; is the speed prediction error; is the angular velocity prediction error. The attitude difference is calculated by the cosine distance: in To predict the quaternion and the real quaternion q k+1 The angle difference between them.

[0113] The prediction error vector calculated by the above formula is The prediction error is fed back to the control strategy, providing an accurate and dynamic error signal, thereby enhancing the robustness and adaptability of the control system. In addition to receiving the prediction error feedback, the control strategy also receives the robot's state information and task control instructions as input, and is trained using model-free reinforcement learning algorithms such as Proximal Policy Gradient (PPO).

[0114] Next, a robust reinforcement learning control device based on prediction error feedback proposed in accordance with an embodiment of the present application is described with reference to the accompanying drawings.

[0115] Figure 4 It is a block diagram of a robust reinforcement learning control device based on prediction error feedback according to an embodiment of the present application.

[0116] like Figure 4 As shown, the robust reinforcement learning control device 10 based on prediction error feedback includes: a simplification module 201 , a first output module 202 , a calculation module 203 and a second output module 204 .

[0117] Among them, the simplification module 201 is used to simplify the structure of the target robot into a single rigid body; the first output module 202 is used to predict the operating state of the single rigid body according to the output instructions of the reinforcement learning strategy; the calculation module 203 is used to obtain the actual operating state of the target robot when executing the output instructions, and calculate the prediction error between the predicted operating state and the actual operating state; the second output module 204 is used to use the prediction error and the actual operating state as feedback signals of the reinforcement learning strategy, and the reinforcement learning strategy outputs instructions according to the feedback signal and the task control instructions.

[0118] In the embodiment of the present application, the structure of the target robot includes a main body part, a limb part and an actuator.

[0119] In an embodiment of the present application, the simplification module 201 is further used to: simplify the structure of the target robot into a single rigid body, including ignoring the preset state of the actuator or limbs and retaining the main body of the target robot; and treat the main body of the target robot as a single rigid body.

[0120] In the embodiment of the present application, the operating state includes position, speed, attitude and angular velocity, among which:

[0121] The prediction formula for position is:

[0122]

[0123] Among them, p k is the position; v k is the speed; Δt is the time interval; a k is the acceleration instruction at the current moment;

[0124] The speed prediction formula is:

[0125]

[0126] The update formula of the posture is:

[0127]

[0128] Among them, q k is the attitude expressed by quaternion; represents the multiplication of quaternions; Δθ is the angle change, ω k is the angular velocity; is the angular acceleration command; q Δθ In order to generate the rotation quaternion according to Δθ, the exponential mapping method is used for calculation, and the calculation formula is:

[0129]

[0130] The update formula of angular velocity is:

[0131]

[0132] In the embodiment of the present application, the calculation formula of the prediction error is:

[0133]

[0134] The predicted operating status is The actual running status is x k+1 =[p k+1 ,q k+1 , v k+1 ,ω k+1 ]; is the position prediction error; is the speed prediction error; is the angular velocity prediction error; For posture differences; To predict the quaternion and the real quaternion q k+1 The angle difference between them.

[0135] In an embodiment of the present application, the target robot is trained using a reinforcement learning algorithm that approximates policy gradients.

[0136] In the embodiment of the present application, the target robot is a real robot or a simulated robot.

[0137] It should be noted that the aforementioned explanation of the embodiment of the robust reinforcement learning control method based on prediction error feedback is also applicable to the robust reinforcement learning control device based on prediction error feedback of this embodiment, and will not be repeated here.

[0138] According to the robust reinforcement learning control device based on prediction error feedback proposed in the embodiment of the present application, the structure of the target robot is simplified into a single rigid body, and the operating state of the single rigid body is predicted according to the output instructions of the reinforcement learning strategy. At the same time, the actual operating state of the target robot when executing the output instructions is obtained, and the prediction error between the predicted operating state and the actual operating state is calculated. The prediction error and the actual operating state are used as feedback signals of the reinforcement learning strategy. The reinforcement learning strategy outputs instructions according to the feedback signal and the task control instructions, which effectively improves the robustness and generalization ability of the strategy, avoids manual adjustment of complex parameters, and has high practicality and scalability.

[0139] Figure 5 A schematic diagram of the structure of a robot provided in an embodiment of the present application. The robot may include:

[0140] A memory 301 , a processor 302 , and a computer program stored in the memory 301 and executable on the processor 302 .

[0141] When the processor 302 executes the program, the robust reinforcement learning control method based on prediction error feedback provided in the above embodiment is implemented.

[0142] Furthermore, the robot also includes:

[0143] The communication interface 303 is used for communication between the memory 301 and the processor 302 .

[0144] The memory 301 is used to store computer programs that can be run on the processor 302 .

[0145] The memory 301 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0146] If the memory 301, the processor 302 and the communication interface 303 are implemented independently, the communication interface 303, the memory 301 and the processor 302 can be connected to each other through a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0147] Optionally, in a specific implementation, if the memory 301, the processor 302 and the communication interface 303 are integrated on a chip, the memory 301, the processor 302 and the communication interface 303 can communicate with each other through an internal interface.

[0148] The processor 302 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0149] An embodiment of the present application also provides a computer-readable storage medium having a computer program or instruction stored thereon. When the computer program or instruction is executed, the above-mentioned robust reinforcement learning control method based on prediction error feedback is implemented.

[0150] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0151] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0152] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0153] It should be understood that the various parts of the present application can be implemented in hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, the steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0154] A person of ordinary skill in the art may understand that all or part of the steps carried by the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the above-mentioned program may be stored in a computer-readable storage medium, which, when executed, includes one of the steps of the method embodiment or a combination thereof.

[0155] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A robust reinforcement learning control method based on prediction error feedback, characterized in that: The following steps are involved: Simplify the structure of the target robot into a single rigid body; Predicting the operating state of the monomer rigid body according to the output instruction of the reinforcement learning strategy; Acquiring the actual running state of the target robot when executing the output instruction, and calculating the prediction error between the predicted running state and the actual running state; The prediction error and the actual operating state are used as feedback signals of the reinforcement learning strategy, and the reinforcement learning strategy outputs instructions according to the feedback signal and the task control instruction.

2. The robust reinforcement learning control method based on prediction error feedback according to claim 1, characterized in that: The structure of the target robot includes a main body part, a limb part and an actuator.

3. The robust reinforcement learning control method based on prediction error feedback according to claim 2, characterized in that: The method of simplifying the structure of the target robot into a single rigid body comprises: Ignore the preset state of the actuator or limb and retain the main body of the target robot; The main body of the target robot is used as the single rigid body.

4. The robust reinforcement learning control method based on prediction error feedback according to claim 1, characterized in that: The operating state includes position, speed, attitude and angular velocity, wherein: The prediction formula for the position is: Among them, p k is the position; v k is the speed; Δt is the time interval; a k is the acceleration instruction at the current moment; The prediction formula for the speed is: The update formula of the posture is: Among them, q k is the attitude expressed by quaternion; represents the multiplication of quaternions; Δθ is the angle change, ω k is the angular velocity; is the angular acceleration command; q Δθ In order to generate the rotation quaternion according to Δθ, the exponential mapping method is used for calculation, and the calculation formula is: The updating formula of the angular velocity is:

5. The robust reinforcement learning control method based on prediction error feedback according to claim 4, characterized in that: The calculation formula of the prediction error is: The predicted operating status is The actual running status is x k+1 =[p k+1 ,q k+1 ,v k+1 ,ω k+1 ]; is the position prediction error; is the speed prediction error; is the angular velocity prediction error; For posture differences; To predict the quaternion and the real quaternion q k+1 The angle difference between them.

6. The robust reinforcement learning control method based on prediction error feedback according to claim 1, characterized in that: The target robot is trained using a reinforcement learning algorithm with approximate policy gradient.

7. The robust reinforcement learning control method based on prediction error feedback according to any one of claims 1 to 6, characterized in that: The target robot is a real robot or a simulated robot.

8. A robust reinforcement learning control device based on prediction error feedback, characterized in that: include: A simplification module is used to simplify the structure of the target robot into a single rigid body; A first output module, used for predicting the operating state of the single rigid body according to the output instruction of the reinforcement learning strategy; A calculation module, used for obtaining the actual running state of the target robot when executing the output instruction, and calculating the prediction error between the predicted running state and the actual running state; The second output module is used to use the prediction error and the actual operating state as feedback signals of the reinforcement learning strategy, and the reinforcement learning strategy outputs instructions according to the feedback signal and the task control instruction.

9. A robot, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the robust reinforcement learning control method based on prediction error feedback as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed, the robust reinforcement learning control method based on prediction error feedback as described in any one of claims 1 to 7 is implemented.

Images