Training of reinforcement learning model and robot control method, device and equipment

Abstract

The application provides a training method, device and equipment of a reinforcement learning model and a robot control method, which are used to improve the walking stability of a robot. The training method of the reinforcement learning model comprises the following steps: in response to receiving a first speed instruction, determining a first body observation; inputting the first speed instruction and the first body observation into a to-be-trained encoder to obtain first auxiliary decision information; the first auxiliary decision information comprises a first estimated linear velocity and a first tensor; inputting the first auxiliary decision information into a to-be-trained policy network to obtain a first action, and updating a first simulation state based on the first action to obtain a second simulation state; inputting the first action and the first auxiliary decision information into a to-be-trained decoder to obtain a first predicted state; and updating first parameters of the to-be-trained encoder and second parameters of the to-be-trained policy network based on a difference between the second simulation state and the first predicted state to obtain a target reinforcement learning model comprising a target encoder and a target policy network.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and in particular to a method, apparatus and equipment for training reinforcement learning models and controlling robots. Background Technology

[0002] For the motion control of quadrupedal wheeled robots, reinforcement learning (RL) models are typically used. During the training phase of the RL model, simulations are usually performed in a physical simulation environment to train the policy network within the RL model, enabling it to output control information to guide the quadrupedal wheeled robot's movement. Because quadrupedal wheeled robots need to navigate different terrains, their motion strategies must be adjusted accordingly for different environments. For example, on flat ground, they may use wheels without lifting their legs, while on uneven ground, they may lift their legs to move forward. However, the policy network in the RL model often lacks adaptability to environmental changes such as terrain variations, leading to gait instability and insufficient motion stability in the robot. Summary of the Invention

[0003] Therefore, it is necessary to provide a method, apparatus, and device for training reinforcement learning models and controlling robots to address the aforementioned technical problems.

[0004] In a first aspect, embodiments of this application provide a method for training a reinforcement learning model, wherein the reinforcement learning model to be trained includes a trainable encoder, a trainable decoder, and a trainable policy network connected to the trainable encoder and the trainable decoder respectively, and the method includes:

[0005] In response to receiving the first velocity command, the first body observation is determined;

[0006] The first velocity command and the first body observation are input into the encoder to be trained to obtain first auxiliary decision information; wherein, the first auxiliary decision information includes a first estimated linear velocity and a first tensor;

[0007] The first auxiliary decision information is input into the policy network to be trained to obtain a first action, and the first simulation state is updated based on the first action to obtain a second simulation state.

[0008] The first action and the first auxiliary decision information are input into the decoder to be trained to obtain the first prediction state;

[0009] Based on the difference between the second simulation state and the first prediction state, the first parameters of the encoder to be trained and the second parameters of the policy network to be trained are updated to obtain a target reinforcement learning model containing a target encoder and a target policy network.

[0010] In some embodiments, updating the first parameters of the encoder to be trained and the second parameters of the policy network to be trained based on the difference between the second simulation state and the first prediction state to obtain a target reinforcement learning model containing a target encoder and a target policy network includes:

[0011] Based on the acquired privileged observations, historical ontology observations, and first ontology observations at the first time step, the first loss value of the preset reconstruction loss function and the first reward value of the preset reward function are determined.

[0012] Based on the first loss value, the first parameter is updated to obtain the target encoder, and based on the first reward value, the second parameter is updated to obtain the target policy network;

[0013] The decoder loss function in the reconstruction loss function is set according to the difference between the first predicted state and the second simulation state, and the first predicted state and the second simulation state correspond to the same time step.

[0014] The reward function includes an instruction tracking reward function corresponding to the first velocity instruction, and / or a foot height reward function corresponding to the simulated terrain information in the privileged observation.

[0015] In some embodiments, the reconstruction loss function further includes an encoder loss function; the encoder loss function is obtained by the following method:

[0016] Determine the simulated linear velocity of the simulated object;

[0017] The encoder loss function is determined based on the difference between the first estimated linear velocity and the simulated linear velocity.

[0018] In some embodiments, the reconstruction loss function The expression is as follows: ; Encoder loss function, Let be the decoder loss function; where, ; To simulate linear velocity, The first estimated linear velocity; ; This represents the second simulation state at the second time step. This is the first predicted state at the second time step. The preset balance coefficient, This is the first tensor at the first time step. This represents the simulation state at historical time steps prior to the first time step.

[0019] In some embodiments, the first tensor includes foot contact force information; the first predicted state corresponds to a preset dynamic equation; wherein the dynamic equation characterizes the correspondence between the first predicted state, the second predicted state, the first action, and the foot contact force information;

[0020] The second predicted state, the first action, and the foot contact force information all correspond to the first time step, and the second time step corresponding to the first predicted state is adjacent to the first time step.

[0021] In some embodiments, the instruction tracking reward function is used to generate a reward based on the difference between the simulated linear velocity in the simulation environment and the first velocity instruction.

[0022] The instruction tracking reward function is expressed as follows: ; in, ; ; For linear velocity reward, As a bonus for angular velocity, The weight of the linear velocity reward, The weight of the angular velocity reward; The linear velocity in the first speed command. For linear velocity in privileged observations, The turning speed in the first speed command. The turning speed is the one observed in the privileged setting.

[0023] In some embodiments, the simulated terrain types in the simulated terrain information include flat terrain and elevated terrain. The height difference between the highest and lowest points in the flat terrain is less than a first height threshold, and the height of the elevated platform in the elevated terrain is greater than a second height threshold.

[0024] The target policy network is trained using the following methods:

[0025] The simulated terrain information is determined through the privileged observation.

[0026] Determine the reward function corresponding to the simulated terrain type; the reward function includes a basic reward function and a high-platform reward function corresponding to the high-platform terrain.

[0027] Based on the basic reward function and / or the high-platform reward function, a real-time reward corresponding to the first action is generated;

[0028] Based on the real-time reward corresponding to the first action, the second parameter in the policy network to be trained is updated to obtain the target policy network.

[0029] Secondly, embodiments of this application provide a robot control method based on a reinforcement learning model, comprising:

[0030] Receive control commands and determine a second body observation; wherein the control commands include a second velocity command, and the second body observation includes the second velocity command;

[0031] In the target reinforcement learning model, the second ontology observation is input into the target encoder to obtain the second auxiliary decision information; wherein, the target reinforcement learning model includes a target encoder and a target policy network, and the second auxiliary decision information includes a second estimated linear velocity and a second tensor;

[0032] The second auxiliary decision information is input into the target policy network to obtain a second action; wherein the second action includes the robot's motion information.

[0033] Thirdly, embodiments of this application provide a training apparatus for a reinforcement learning model, wherein the reinforcement learning model to be trained includes a trainable encoder, a trainable decoder, and a trainable policy network connected to the trainable encoder and the trainable decoder respectively, and the method includes:

[0034] The observation unit is used to determine the first body observation in response to receiving the first velocity command;

[0035] The encoding unit is configured to input the first velocity command and the first ontology observation into the encoder to be trained to obtain first auxiliary decision information; wherein, the first auxiliary decision information includes a first estimated linear velocity and a first tensor;

[0036] A state unit is used to input the first auxiliary decision information into the policy network to be trained to obtain a first action, and update the first simulation state based on the first action to obtain a second simulation state.

[0037] The decoding unit is used to input the first action and the first auxiliary decision information into the decoder to be trained to obtain the first prediction state;

[0038] An update unit is used to update the first parameters of the encoder to be trained and the second parameters of the policy network to be trained based on the difference between the second simulation state and the first prediction state, so as to obtain a target reinforcement learning model containing a target encoder and a target policy network.

[0039] In some embodiments, the update unit is specifically used to determine a first loss value of a preset reconstruction loss function and a first reward value of a preset reward function based on the acquired privileged observations, historical ontology observations, and first ontology observations at a first time step; based on the first loss value, update the first parameters to obtain the target encoder, and based on the first reward value, update the second parameters to obtain the target policy network; wherein, the decoder loss function in the reconstruction loss function is set according to the difference between the first predicted state and the second simulation state, and the first predicted state and the second simulation state correspond to the same time step; the reward function includes an instruction tracking reward function corresponding to the first velocity command, and / or a foot height reward function corresponding to the simulated terrain information in the privileged observations.

[0040] In some embodiments, the reconstruction loss function further includes an encoder loss function; the reconstruction loss unit is specifically used to determine the encoder loss function, determine the simulation linear velocity of the simulation object, and determine the encoder loss function based on the difference between the first estimated linear velocity and the simulation linear velocity.

[0041] In some embodiments, the reconstruction loss function The expression is as follows: ; Encoder loss function, Let be the decoder loss function; where, ; To simulate linear velocity, The first estimated linear velocity; ; This is the second simulation state. This is the first predicted state. The preset balance coefficient, For the first tensor, This represents the simulation state at historical time steps prior to the first time step.

[0042] In some embodiments, the first tensor includes foot contact force information; the first predicted state corresponds to a preset dynamic equation; wherein the dynamic equation characterizes the correspondence between the first predicted state, the second predicted state, the first action, and the foot contact force information; the second predicted state, the first action, and the foot contact force information all correspond to a first time step, and the second time step corresponding to the first predicted state is adjacent to the first time step.

[0043] In some embodiments, the simulated terrain types in the simulated terrain information include flat terrain and elevated terrain. The elevation difference between the highest and lowest points in the flat terrain is less than a first height threshold, and the height of the elevated platforms in the elevated terrain is greater than a second height threshold. The update unit is further configured to determine the simulated terrain information through the privileged observation; determine a reward function corresponding to the simulated terrain type; the reward function includes a basic reward function and an elevated reward function corresponding to the elevated terrain; generate a real-time reward corresponding to the first action based on the basic reward function and / or the elevated reward function; and update the second parameter in the policy network to be trained based on the real-time reward corresponding to the first action to obtain the target policy network.

[0044] Fourthly, embodiments of this application provide a robot control device based on a reinforcement learning model, comprising:

[0045] The instruction unit is configured to receive control instructions and determine a second body observation; wherein the control instructions include a second speed instruction, and the second body observation includes the second speed instruction;

[0046] An auxiliary unit is used in the target reinforcement learning model to input the second ontology observation into the target encoder to obtain second auxiliary decision information; wherein the target reinforcement learning model includes a target encoder and a target policy network, and the second auxiliary decision information includes a second estimated linear velocity and a second tensor;

[0047] The action unit inputs the second auxiliary decision information into the target policy network to obtain a second action; wherein the second action includes the robot's motion information.

[0048] Fifthly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and for running on the processor, characterized in that the processor executes the computer program to implement the methods described in the first aspect and any embodiment, or the methods described in the second aspect and any embodiment.

[0049] In a sixth aspect, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the methods described in the first aspect and any embodiment, or the second aspect and any embodiment.

[0050] In a seventh aspect, embodiments of this application provide a computer program product, including a computer program, characterized in that, when the computer program is executed by a processor, it implements the methods described in the first aspect and any embodiment, or the second aspect and any embodiment.

[0051] This application provides a training method for a reinforcement learning model. By embedding a trainable encoder and a trainable decoder within the reinforcement learning model to be trained, the output of the trainable encoder assists the training policy network in making a first action that better matches the simulation environment, thereby improving the motion performance of the simulated robot. This auxiliary decision-making information compensates for the simulation object's inability to accurately calculate linear velocity and the ground or obstacles its feet contact. Furthermore, the training method provided in this application, through joint training of the trainable encoder and decoder, guides the trainable encoder to output more accurate auxiliary decision-making information. This efficiently trains a target reinforcement learning model with high generalization capabilities, improving the robot's walking stability when used for robot control.

[0052] Other features and advantages of the invention will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not intended to limit this disclosure. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0054] Figure 1 This is a flowchart illustrating the training method of a reinforcement learning model in one embodiment; Figure 2 This is a schematic diagram of the information flow between the encoder, decoder and policy network to be trained in a training reinforcement model in one embodiment. Figure 3 This is a schematic diagram of a phased training method in one embodiment; Figure 4 This is a schematic diagram of the structure of a reinforcement learning model to be trained in one embodiment. Figure 5 This is a flowchart illustrating a robot control method based on a reinforcement learning model in one embodiment; Figure 6 This is a schematic diagram illustrating how a simulated robot, controlled by a trained target reinforcement learning model, completes a high-platform crossing in one embodiment. Figure 7This is a schematic diagram illustrating the movement of a wheeled robot in the physical world based on a target reinforcement learning model, as shown in one embodiment. Figure 8 This is a structural block diagram of a training device for a reinforcement learning model in one embodiment; Figure 9 This is a structural block diagram of a robot control device based on a reinforcement learning model in one embodiment; Figure 10 This is a diagram of the internal structure of an electronic device in one embodiment. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0056] It should be noted that the illustrations provided in this embodiment are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show components relevant to the present invention and are not drawn according to the actual number, shape, and size of components in implementation. In actual implementation, the form, quantity, and proportion of each component can be arbitrarily changed, and the component layout may be more complex. The structures, proportions, sizes, etc., depicted in the accompanying drawings are only for illustrative purposes to aid those skilled in the art and are not intended to limit the implementation conditions of the present invention. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in proportions, or adjustments to size, without affecting the effects and objectives of the present invention, should still fall within the scope of the technical content disclosed in the present invention. Furthermore, the terms such as "upper," "lower," "left," "right," "middle," and "one" used in this specification are only for clarity of description and are not intended to limit the scope of the present invention. Changes or adjustments to their relative relationships, without substantially altering the technical content, should also be considered within the scope of the present invention.

[0057] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the document does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0058] The definitions used herein, such as the terms “having,” “may have,” “including,” or “may include,” indicate the presence of the corresponding function, operation, element, etc., and do not limit the presence of one or more other functions, operations, elements, etc.

[0059] In this embodiment of the invention, prefixes such as "first" and "second" are used merely to distinguish different descriptive objects and do not limit the position, order, priority, quantity, or content of the described objects. The use of ordinal numbers and other prefixes to distinguish descriptive objects in this application does not constitute a limitation on the described objects. For statements regarding the described objects, please refer to the claims or the context of the embodiments. The use of such prefixes should not constitute unnecessary limitations. Furthermore, in the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0060] To improve the robustness of reinforcement learning models in controlling robot movement, this application provides a method for training such a model. This reinforcement learning model is particularly suitable for controlling the movement of wheeled robots. Furthermore, due to its good generalization ability, it can reliably and promptly control the robot to adjust its motion strategy, improving the robot's motion stability in different terrains and its environmental adaptability. A detailed description follows; please refer to the provided text. Figure 1 :

[0061] Step 101: In response to receiving the first velocity command, determine the first body observation.

[0062] The first ontology observation includes ontology observation at the historical time step and ontology observation at the first time step.

[0063] Specifically, the aforementioned first velocity command can be the first velocity command received at the first time step. This first velocity command includes a first linear velocity command and a first angular velocity command. The first velocity command can be understood according to the desired linear velocity command. Specifically, the first linear velocity command includes the linear velocities in the x and y directions on the plane. The first angular velocity command includes the yaw angular velocity (ω) about the vertical direction (z-axis) of the aircraft body.

[0064] Furthermore, in this embodiment, the observation information is divided into ontology observation and privileged observation. Therefore, the ontology observation at the historical time step and the ontology observation at the first time step correspond to the robot's own observation information at different time steps (moments).

[0065] In one possible implementation, the aforementioned ontology observation includes ontology state. This ontology state can include joint positions, joint velocities, simulated angular velocities, and the gravity projection corresponding to the simulated object of the robot. Furthermore, ontology observation may also include gait phase rhythm. Gait phase rhythm refers to the phase position of each leg relative to a common time reference during the periodic movement of a multi-legged robot. This phase is typically represented by a variable that continuously varies within the range [0, 1) or [0, 2π), describing the cyclical state of the foot between the "support phase" (ground contact) and the "swing phase" (lifting and moving). By coordinating the phase differences between the four legs, different gait patterns (such as walk, trot, pace, gallop) can be generated.

[0066] The aforementioned ontological observations can be determined by the robot based on its onboard sensors or chips. These sensors could be, for example, IMUs (Inertial Measurement Units). The chip could be used to receive a first velocity command. Privileged observations, on the other hand, are obtained based on user input. Therefore, during the training phase of the reinforcement learning model, these ontological observations can be acquired through simulated objects.

[0067] In one possible implementation, the aforementioned historical time steps correspond to a preset historical time window. For example, if the preset time window is 5 time steps, then the first time step is denoted as t, and the historical time steps are denoted as {t-1, t-2, t-3, t-4, t-5}.

[0068] Furthermore, the privileged observations corresponding to the aforementioned ontological observations may include at least one of the following: simulated linear velocity, foot height, fuselage height scan lattice, foot height scan lattice, foot friction force, and foot contact force. In the embodiments of this application, the privileged observations mainly serve as supervisory signals during the training phase of the reinforcement learning model, and can be input by the user through the commentator network in the reinforcement learning model to be trained.

[0069] Step 102: Input the first speed command and the first body observation into the encoder to be trained to obtain the first auxiliary decision information.

[0070] The first auxiliary decision-making information includes the first estimated linear velocity and the first tensor.

[0071] Specifically, the first estimated linear velocity can be understood as the linear velocity that the encoder to be trained expects the simulated object to actually reach in the next time step, based on the first velocity command and the first body observation.

[0072] Simultaneously, the encoder to be trained generates a first tensor based on the ontology observations from historical time steps in the first ontology observation and the ontology observation at the current moment (i.e., the first time step). This first tensor can be viewed as an implicit state estimate. It includes at least foot contact force information. This foot contact force information is used to characterize the interaction between the foot of the simulated robot and the ground, thus reflecting the ground environment.

[0073] The encoder to be trained can be an MLP (Multilayer Perceptron).

[0074] Step 103: Input the first auxiliary decision information into the policy network to be trained to obtain the first action, and update the first simulation state based on the first action to obtain the second simulation state.

[0075] Specifically, the first action can serve as an action command for the simulated object. The simulated object moves according to the first action at the first time step, updating the first simulation state at the first time step and the second simulation state at the second time step. It can be understood that the first and second time steps are two adjacent time steps.

[0076] Step 104: Input the first action and the first auxiliary decision information into the decoder to be trained to obtain the first predicted state.

[0077] Specifically, the first predicted state is the state of the simulated object after the decoder to be trained performs the first action, so the first predicted state corresponds to the second simulation state mentioned above.

[0078] That is, the first predicted state and the second simulation state correspond to the same time step.

[0079] Step 105: Based on the difference between the second simulation state and the first prediction state, update the first parameters of the encoder to be trained and the second parameters of the policy network to be trained to obtain a target reinforcement learning model containing the target encoder and the target policy network.

[0080] Specifically, the decoder to be trained can form an encoder-decoder architecture with the aforementioned encoder to be trained. Thus, the encoder and decoder to be trained are trained together. Both can have their parameters updated using a preset reconstruction loss function.

[0081] In one possible implementation, the first parameter can be updated by the reconstruction loss function corresponding to the encoder to be trained, and the second parameter can be updated by the reward function and value network corresponding to the policy network to be trained, until a preset condition is met, thereby obtaining the target encoder and the target policy network, and thus obtaining the target reinforcement learning model.

[0082] The above-mentioned preset conditions may include at least one of the following: (1) the reconstruction loss function converges, the loss function of the value network converges, and the fluctuation value of the reward function is less than the first stability threshold; (2) the iteration step is greater than the preset iteration threshold; (3) the function value of the reconstruction loss function is less than the first loss threshold, the loss function value of the value network is less than the second loss threshold, and the value of the reward function is greater than the third loss threshold.

[0083] In the training method of the reinforcement learning model described in steps 101-105 of this application embodiment, the first auxiliary decision information output by the encoder to be trained is used as the input of the policy network to be trained. This enables the policy network to achieve more accurate motion self-perception, compensating for the inability to measure the linear velocity of the simulated object and robot during operation. Furthermore, the contact force information carried by the first tensor can indirectly reflect feature information such as terrain type, thus effectively enhancing the adaptability of the policy network to different terrains. This results in a more accurate first action output, improving the robustness of the target reinforcement learning model and the generalization ability of the simulated object when facing different terrains.

[0084] In one possible implementation, the target reinforcement learning model is generated as follows: First, based on the acquired privileged observations, historical ontology observations, and first ontology observations at the first time step, a first loss value for a pre-defined reconstruction loss function and a first reward value for a pre-defined reward function are determined. This first reward value is the sum of all reward values ​​in the pre-defined reward function. This first reward value can be positive or negative. Then, the first parameters are updated using the first loss value to obtain the target encoder, and the second parameters are updated using the first reward value to obtain the target policy network.

[0085] Among them, the decoder loss function L in the reconstruction loss function decoder Based on the difference between the first predicted state and the second simulation state, both the first predicted state and the second simulation state correspond to the same time step. For example, the second time step is t+1. Then the first predicted state can be represented as... The second simulation state can be represented as .

[0086] The reward function includes a command tracking reward function corresponding to the same velocity command, and / or a foot height reward function corresponding to the simulated terrain information in the privileged observation. The simulated terrain information may include simulated terrain types, such as flat terrain and elevated terrain. In flat terrain, the height difference between the highest and lowest points is less than a first height threshold. Flat terrain includes flat ground and steps. The height of flat ground is 0, and the height of steps is less than the first height threshold. In elevated terrain, the height of the platform is greater than a second height threshold.

[0087] Both the first and second height thresholds are empirical values. Optionally, the second height threshold is greater than the first height threshold. Or, the first height threshold is equal to the second height threshold. For example, both the first and second height thresholds are 20cm. Or, the first height threshold is 10cm and the second height threshold is 20cm. Then, the height of each step can be, for example, 10cm or 20cm, and the height of the platform can be, for example, 50cm. In this embodiment, a step means that a platform with alternating low vertical and horizontal surfaces forms a stepped and continuous surface, gradually increasing in height. A platform means a single vertically steep plane with a sudden change in height and a discontinuous terrain structure.

[0088] Furthermore, the reconstruction loss function will be explained in detail below: The reconstruction loss function may also include an encoder loss function; the reconstruction loss function can be obtained as follows: First, the simulated linear velocity generated by the simulation environment in the reinforcement learning model to be trained. This simulated linear velocity is actually the actual linear velocity of the simulated object when performing the first action in the simulation environment. Then, based on the difference between the aforementioned first estimated linear velocity and the simulated linear velocity, the encoder loss function can be determined. For example, the mean square error between the first estimated linear velocity and the simulated linear velocity can be determined as the encoder loss function.

[0089] Thus, through the above embodiments, the decoder and encoder to be trained can be considered as a whole. During training, by reconstructing the decoder loss function in the loss function, the difference between the first predicted state and the second simulated state is reduced, thereby updating the parameters in the decoder to be trained, improving the accuracy of the first predicted state output. Simultaneously, by reducing the difference between the first estimated linear velocity and the simulated linear velocity, the accuracy of the encoder output is improved. The following, in conjunction with... Figure 2 Further examples illustrating the reconstruction loss function and the relationship between the encoder and decoder to be trained are provided below: Please refer to Figure 2 According to the first speed command v c and the first entity observation o t The encoder to be trained generates a model containing the first estimated linear velocity v. t and the first tensor z t The first auxiliary decision information. This first auxiliary decision information is input into the policy network to be trained, and is used by the policy network to be trained based on z. t The implicit information about foot contact force generates a more generalized first action a. t Foot contact force information reflects the physical information of the interaction between the simulated object's foot and the terrain. This occurs when the simulated object performs the first action, a. t Subsequently, the first simulation state in the simulation environment is updated to form the second simulation state. .

[0090] To improve the accuracy and convergence rate of the encoder output, the first auxiliary decision information is input into the decoder to be trained, so that the decoder outputs the first predicted state s. t+1 .

[0091] Therefore, please continue to refer to [the relevant documentation / reference]. Figure 2 By predicting the first state s t+1 and the second simulation state Input the decoder loss function in the reconstruction loss function, and input the first estimated linear velocity v t and the simulated linear velocity generated by the simulation environment The input is fed into the encoder loss function in the reconstruction loss function to calculate the generated reconstruction loss value L. mlp The reconstruction loss value is then used for backpropagation to optimize the first parameters of the encoder to be trained, thus obtaining the target encoder.

[0092] In one possible implementation, the above reconstruction loss function The encoder loss function and decoder loss function The sum of these. Then the reconstruction loss function. It can be represented as: .

[0093] in, ; To simulate linear velocity, The first estimated linear velocity; ; This represents the second simulation state at the second time step. This is the first predicted state at the second time step. The preset balance coefficient, This is the first tensor at the first time step. This second simulation state represents the simulation state at a historical time step prior to the first time step. It is obtained by updating the first simulation state in the simulation environment of the reinforcement learning model to be trained after the simulated object performs the first action.

[0094] In this way, the encoder loss function and decoder loss function are used to jointly train the decoder to be trained and the encoder to be trained. By introducing KL (Kullback-Leibler divergence) into the decoder loss function, the problem of overfitting of the encoder and decoder to be trained and the decline in generalization ability during training is avoided.

[0095] In one possible implementation, the first tensor includes foot contact force information. This foot contact force information reflects the physical information of the interaction between the simulated object's (robot's) foot and the terrain. Exemplarily, the foot contact force information may include at least one of foot contact force and foot friction force. The first tensor may also include the body state of the simulated object. Exemplarily, the body state includes the simulated object's position information, attitude information, and angular velocity information.

[0096] Based on this, the target reinforcement learning model trained in the embodiments of this application can implicitly obtain terrain information through the first tensor. Therefore, when the trained target reinforcement learning model is applied to a wheeled robot without a visual sensor, it can show obvious advantages.

[0097] To further accelerate the convergence of the reconstruction loss function and improve the training efficiency of the encoder and decoder, one possible implementation is to guide the encoder to output the first tensor containing foot contact force information in the following way, while ensuring its accuracy:

[0098] The first predicted state output by the decoder to be trained is mapped to a preset dynamic equation, which is then embedded into the reconstruction loss function. This dynamic equation characterizes the correspondence between the first predicted state, the second predicted state, the first action, and the foot contact force information. Specifically, the second predicted state, the first action, and the foot contact force information all correspond to the first time step, and the second time step corresponding to the first predicted state is adjacent to the first time step. The first time step precedes the second time step; for example, if the first time step can be represented as t, then the second time step can be represented as t+1.

[0099] The aforementioned pre-defined dynamic equations can be expressed as follows: ;in, This is the first predicted state. This is the second predicted state. To control the input, and the first action a t equal, The system matrix A represents the foot contact force information, and A, B, and E represent the system matrices corresponding to the second predicted state, the second predicted state, and the foot contact force information, respectively. This system matrix is ​​also obtained through the first tensor. In other words, the system matrix corresponds to the first tensor.

[0100] Therefore, during the training process, as the accuracy of the predicted state output by the decoder to be trained (i.e. the first and second predicted states of two adjacent time steps) improves, it guides the encoder to be trained to output a more accurate first tensor; that is, it guides the encoder to be trained to compress more accurate foot contact force information into the first tensor and enhance its expressive power.

[0101] In other words, the encoder and decoder to be trained are trained by reconstructing the loss function and implicitly learn the motion mapping corresponding to the aforementioned preset dynamic equations, which enables the accuracy and expressive power of the first auxiliary decision information output by the encoder to be trained, especially the first tensor, to be efficiently improved.

[0102] The aforementioned pre-defined dynamic equations Foot contact force information This is a vector. In one possible implementation, the above foot contact force information includes foot contact force and foot friction force; then the foot contact force information corresponding one-to-one with the foot is represented as follows: ;in, Here, n represents the total number of feet corresponding to the simulated robot. For example, for a quadruped robot, n=4. The meaning of the foot index is: the... The serial number of each foot.

[0103] Correspondingly, the contact force vector at any foot tip It can be decomposed into a normal vector and a tangential component, as shown in one embodiment: ;in, This represents the normal component on the foot contact surface, corresponding to the normal contact force containing the foot's rebound force. and The tangential component represents the frictional force on the foot contact surface, and the frictional force in the x-direction and y-direction in the plane represents the frictional force, respectively.

[0104] To further improve training efficiency, one possible implementation is to obtain simulated foot contact force information through privileged observation. Since privileged observation is introduced through the critic network in the reinforcement learning model to be trained, the foot contact force information in the first tensor can be reduced through the critic network in the reinforcement learning model to be trained. The difference between the simulated foot contact force information and the actual foot contact force information is used to quickly improve the accuracy of the foot contact force in the first tensor, thereby accelerating the training efficiency of the policy network to be trained.

[0105] Furthermore, since the simulated terrain types include flat terrain and elevated terrain, the preset reward functions correspond to the simulated terrain types. These preset reward functions include a base reward function used for training on both flat terrain and elevated terrain, and an elevated terrain reward function used only for elevated terrain. In the flat terrain, the height difference between the highest and lowest points is less than a first height threshold, and in the elevated terrain, the height of the elevated platform is greater than a second height threshold. The first and second height thresholds can be equal or unequal. For example, both the first and second height thresholds can be 50 cm. Alternatively, the first height threshold can be 20 cm, and the second height threshold can be 50 cm.

[0106] In one possible implementation, the second parameter is updated using a preset reward function to obtain the target policy network, including:

[0107] First, the simulated terrain type is determined through privileged observation, and the reward function corresponding to that terrain type is obtained. The preset reward functions include a base reward function for flat terrain and a high-platform reward function for elevated terrain. That is, when the terrain is flat, the base reward function is the reward function corresponding to the simulated terrain. When the terrain is elevated, the weighted sum of the base reward function and the high-platform reward function is the reward function corresponding to the simulated terrain type.

[0108] Then, based on the base reward function and / or the elevated reward function, a real-time reward corresponding to the first action is generated. Finally, the second parameters in the policy network to be trained can be updated based on this real-time reward to obtain the target policy network.

[0109] In one possible implementation, the basic reward function includes at least one of the following: command tracking reward function, gait reward function, rolling posture reward function, posture stability reward function, foot contact reward function, safety constraint reward function, and energy consumption and motion smoothness reward function. The elevated platform reward function includes the aforementioned command tracking reward function and foot height reward function; furthermore, the weight of the gait reward function in the elevated platform reward function is 0, so that the simulation object can achieve lateral movement and rotation with a good gait structure in flat ground movement, and can flexibly adjust the foot landing point according to the edge position of the elevated platform during elevated platform training, avoiding the restriction of the degrees of freedom of movements such as crossing and leg lifting in elevated platform terrain.

[0110] The following is an explanation of the expression of each reward function: The instruction reward function in the above-mentioned preset reward function is used to generate a reward based on the difference between the simulation linear velocity in the simulation environment and the first velocity instruction.

[0111] The instruction trace reward function is represented as follows: ; in, ; ; For linear velocity reward, As a bonus for angular velocity, The weight of the linear velocity reward, The weight of the angular velocity reward; The linear velocity in the first speed command. For linear velocity in privileged observations, The turning speed in the first speed command. The turning speed is the one observed in the privileged setting.

[0112] The foot height reward function in the above-mentioned preset reward function Corresponding to the terrain type, this foot height reward function rewards the simulated object for leg lift. When the terrain type is flat terrain, ;in, For foot index, Indicates the first The target height at the foot tip For the first The actual height at which the foot performs the first action.

[0113] pass The conclusion is that, Where h is the target leg lift height, For smooth periodic trajectory functions, For the first The phase of a leg. When When the target leg is in the swing phase, it is in the swing phase; otherwise, the corresponding leg is in the support phase. The target leg lift height is obtained through simulated terrain information.

[0114] ; For foot index, For the first Local gait phase of a leg, The offset of each leg's gait is mod1: Perform operations on the fractional part to ensure that the phase is within the interval [0, 1).

[0115] The global gait phase is generated based on the following formula: ;in, To control the cycle, This represents the gait frequency.

[0116] The control command mask is defined as follows: Where c (command) represents the linear velocity and angular velocity in the first velocity command. (threshold) corresponds to the terrain type, so the th value for flat terrain is [1.5, 0.1, 0.1].

[0117] When the terrain type is elevated. ; ; ;

[0118] in, The preset terrain level for elevated terrain in the environment. As a landmark of the high platform, This refers to the height interval between different platform levels. The control command mask is defined as follows: Where c (command) represents the linear velocity and angular velocity in the first velocity command. (threshold) corresponds to the terrain type, so the th for high plateau terrain is [0.1, 0.1, 0.1].

[0119] Therefore, when the preset reward function includes the foot height reward function as shown above, the trained target reinforcement learning model is particularly suitable for wheel-legged robots.

[0120] In one possible implementation, the aforementioned preset reward function may further include a fuselage height reward function, a gait reward function, a rolling posture reward function, a posture stability reward function, a foot contact reward function, a safety constraint reward function, and an energy and motion smoothness reward function.

[0121] Optionally, fuselage height reward function It is expressed as follows: ;in, For the pre-configured target height value, The simulated height for the simulated object to perform the first action. This is a fixed value corresponding to the standing posture of the machine.

[0122] Optionally, gait reward function It is expressed as follows: ;in, For foot index, For the first The target trajectory point is located at the foot end in the x and y directions. For the first The simulated trajectory points of the foot performing the first action in the x and y directions. Used to distinguish between the oscillating phase and the supporting phase; when it is an oscillating phase... ;in, For the first The phase of a leg This is a smooth periodic trajectory function. When it is a support phase... The value is 0. Optionally, the above... It can be represented as: ;in, Generates based on a predefined base foot position (relative to the robot's center). This is the foot offset. This foot offset is generated based on the aircraft's movement speed.

[0123] Optionally, ; ;in, For gait frequency, This is the speed compensation coefficient.

[0124] The gait reward function described above is used to encourage the simulated object to land stably and to adopt symmetrical gait, so as to ensure the stability of the robot's movement and the aesthetics of its posture.

[0125] Optionally, the rolling posture reward function It is expressed as follows: ;in, For foot index, For the first corresponding to the foot end The angle of the joint corresponding to each foot tip For the first to be in a stable standing posture The pre-configured angle of the joint corresponding to each foot end, and the mask is the control command mask corresponding to the first velocity command. For the specific calculation method, please refer to the control command mask in the aforementioned foot height reward function, which will not be repeated here.

[0126] This rolling posture reward function is used to simulate a wheeled robot, constraining it to maintain a stable and graceful posture when moving forward on flat ground.

[0127] Optionally, the attitude stabilization reward function is used to limit the pitch and roll angle changes of the simulated object's body and penalize excessive vertical and angular velocities to maintain the attitude stability of the simulated object during motion. This attitude stabilization function then includes an attitude angle constraint function. Vertical linear velocity reward function Horizontal angular velocity reward function : ; ; .

[0128] Where g represents the projection of the gravity vector into the robot's body coordinate system. This represents the linear velocity of the simulated object in the vertical direction (z-axis). This represents the angular velocities (i.e., roll angular velocity and pitch angular velocity) of the simulated object around the x-axis and y-axis of the body coordinate system.

[0129] Optionally, a foot contact reward function It is used to penalize contact forces exceeding a preset limit, thereby preventing the simulated object from generating large and unnecessary impact forces during movement and improving the smoothness and stability of the motion.

[0130] ;in, For foot index, This is the preset maximum permissible foot contact force.

[0131] Optionally, the safety constraint reward function is used to penalize situations such as collisions or joints exceeding their range of motion, ensuring that the simulation can proceed safely while maintaining robot motion. This safety constraint reward function may include a collision reward function. and joint limit reward function It is expressed as follows: ; This represents the threshold for slight contact force; anything exceeding this threshold (i.e., 0.1N) is considered a collision.

[0132] ; Indicates the first corresponding to the foot end The angle of the joint corresponding to each foot tip , They represent the first The upper and lower limits of the angle of the joint corresponding to each foot during movement.

[0133] Optionally, the energy consumption and motion smoothness reward function balances energy consumption and motion smoothness through joint torque, joint acceleration, joint energy, and rate of change of motion. This energy consumption and motion smoothness reward function may include a joint torque reward function. Joint acceleration reward function Joint power reward function At least one of them.

[0134] ;in, For foot index, Indicates the first Output torque of the joint (N·m); Where t is the interval between two adjacent time steps; Indicates the first step The angular velocity of the joint corresponding to each foot (i.e., the current angular velocity). This indicates a time step preceding the first time step, the 1st time step. The angular velocity of the joint corresponding to each foot.

[0135] ; Indicates the first The instantaneous mechanical power of the joint corresponding to each foot end ; The first action indicated by the first time step The acceleration of the joint corresponding to each foot tip The first action indicated by the first action of the time step preceding the first time step The acceleration of the joint corresponding to each foot.

[0136] In this embodiment, the reward value calculated by each reward function can be positive or negative. When the reward value is positive and the sum of the function values ​​of each reward function is positive, it can positively encourage the reinforcement learning model to be trained. When the reward value of a certain basic reward function or high-level reward function is negative, or the sum of the function values ​​of each reward function is negative, it can suppress the reinforcement learning model to be trained, especially the corresponding policy generation mechanism in the policy network to be trained. In order to generate corresponding rewards and penalties and match the importance of the parameters to be optimized corresponding to different reward functions, this embodiment sets positive weights for the above reward functions to achieve positive encouragement, thereby encouraging the reinforcement learning model to be trained to optimize according to the larger output value of the corresponding reward function. At the same time, by setting negative weights for the above reward functions as penalties, it is used to suppress the corresponding reward function from outputting larger values ​​in the reinforcement learning model to be trained.

[0137] In one possible implementation, the weight of the instruction tracking reward function in the aforementioned preset reward function is positive. For example, the weight of the linear velocity reward... =2.5, the weight of angular velocity bonus is =1.0.

[0138] Optionally, the weights of the foot height reward function are positive. For example, the weights of the foot height reward function... .

[0139] Optionally, the weight of the fuselage height reward function is positive. For example, .

[0140] Optionally, the weights of the gait reward function are negative. For example, .

[0141] Optionally, the weight of the rolling posture reward function is negative. For example, .

[0142] Optionally, the weights of the attitude stabilization reward function are negative. For example, the weights of the attitude angle constraint function for flat terrain are... Weight of vertical linear velocity in flat terrain Weight of horizontal angular velocity bonus for flat terrain And, the weights of the attitude angle constraint function for the high-altitude terrain. Weight of vertical linear velocity in flat terrain Weight of horizontal angular velocity bonus for flat terrain .

[0143] The weight of the foot contact reward function is negative. For example, .

[0144] The weights of the safety reward function are negative. For example, the weights of the collision reward... Weight of joint limit reward .

[0145] The weights for energy consumption and motion smoothness reward functions are negative. For example, the joint torque reward function... Joint acceleration reward function Joint power reward function The weights for each item are -0.00005, -0.00000025, 0.00002, and -0.03, respectively.

[0146] In one possible implementation, the terrain type in the simulation environment is first set to flat terrain. Combined with a critic network, the policy network to be trained is then trained until the outputs of the aforementioned basic reward function and its value function meet preset conditions, resulting in an intermediate policy network. Then, the terrain type in the simulation environment is set to elevated terrain, and the intermediate policy network is trained again until the outputs of the aforementioned basic reward function, elevated reward function, and value function meet preset conditions, resulting in the target policy network. Figure 3As shown, the training of the reinforcement learning model to be trained is divided into two stages based on the simulated terrain type. Stage one involves basic flat terrain training, used to train the policy network to control the stable movement of the simulated object in flat terrain. Simultaneously, through appropriate curriculum settings, the difficulty is gradually increased, for example, the basic terrain changes from flat ground to steps, and then to variations in the number of consecutive steps, to adapt to various basic terrain types within the flat terrain, resulting in an intermediate policy network. For example, in Stage one, the flat terrain changes as follows, based on the curriculum settings: after flat ground training, it transitions to gentle slopes and inclines; then to steps, allowing the simulated object to ascend and descend a small number of steps; finally, the simulated object is trained to walk on consecutive steps; this training process yields the intermediate policy network. Alternatively, according to the preset number of levels in the curriculum settings, the difficulty is gradually increased from flat ground. For each level increase, the number of steps or the slope angle (not exceeding a preset empirical value for slope) is increased proportionally, forming a progressively increasing difficulty curriculum training.

[0147] Please continue to refer to this. Figure 3 After saving, the second stage of training, specifically for elevated terrain, is initiated. The simulated terrain type is changed to elevated terrain, and the intermediate policy network trained on the flat terrain is loaded. The basic reward function and the elevated terrain reward function are used, and the difficulty is gradually increased according to the curriculum. Elevated terrain includes upper and lower elevated terrain, and the difficulty is increased progressively according to the height of the elevated platform. For example, the number of levels is increased from 0.05m to 0.8m, creating a progressive difficulty training. Simultaneously, to maintain the robot's ability to move on flat ground, flat or gently sloping flat terrain is retained during the elevated terrain training. The final trained model is then reinforced. This phased training allows the reinforcement learning model to adaptively adjust its movement strategy based on the terrain, avoiding policy interference between different movement modes. This enables the robot to lift its legs promptly when facing elevated platforms, avoiding conflicts between flat ground movement and elevated platform movement, and improving the smoothness of the simulated object's movement.

[0148] The following is a specific embodiment that further illustrates the composition of the reinforcement learning model to be trained and its training process. Please refer to [the provided example]. Figure 4The reinforcement learning model to be trained includes a decoder to be trained, an encoder to be trained, a policy network to be trained (i.e., the actor network), a critic network, a simulation environment, a reconstruction loss function for optimizing the encoder and decoder to be trained, and a PPO (Proximal Policy Optimization Loss) module. The reconstruction loss function generates an MLP loss based on the output of the decoder to be trained and the output of the simulation environment. The PPO module calculates the advantage function using a pre-defined reward function in the simulation environment and the value estimate of the critic network, and uses the advantage function to construct a pruned policy gradient loss to update the parameters of the actor network through backpropagation.

[0149] Based on the same inventive concept, this application provides a robot control method based on a reinforcement learning model, wherein the robot can be a wheeled robot. Please refer to... Figure 5 The robot control method based on reinforcement learning models includes the following implementation steps:

[0150] Step 501: Receive control commands and determine the second body observation.

[0151] The control commands include a second velocity command. The second body observation includes a second velocity command.

[0152] The second velocity command can include linear velocity and angular velocity commands. Then, the second ontological observation can be input into the target reinforcement learning model.

[0153] This second ontological state corresponds to the first ontological state in the aforementioned training method. The second ontological state also includes the robot's gait phase rhythm, joint positions, joint velocities, historical time step movements, angular velocities, and gravity projection.

[0154] Step 502: In the target reinforcement learning model, the second ontology observation is input into the target encoder to obtain the second auxiliary decision information.

[0155] The target reinforcement learning model includes a target encoder and a target policy network.

[0156] The second auxiliary decision-making information includes the second estimated linear velocity and the second tensor.

[0157] This second auxiliary decision information is used to assist the target policy network in outputting a second action that is more consistent with the robot's actual environment (including terrain information).

[0158] Step 503: Input the second auxiliary decision information into the target policy network to obtain the second action.

[0159] The second action includes the robot's motion information.

[0160] This motion information includes motion strategies and corresponding parameters such as speed.

[0161] For example, the movement strategy can be a wheel-based movement strategy or a foot-based movement strategy.

[0162] When the movement strategy is wheeled movement, the movement information also includes the direction of movement and the speed of movement.

[0163] When the movement strategy is a foot-based movement strategy, the movement information also includes at least one of the joint angle, joint position, and joint movement rate corresponding to foot control, as well as movement direction information.

[0164] Thus, through the robot control method based on the reinforcement learning model described in steps 501-503 above, by inputting the second auxiliary decision information into the target policy network, a second action that can match the actual environment (especially the ground environment) and has greater generalization ability is obtained, thereby effectively improving the robot's motion performance. This is because the sensors on the robot itself do not support its direct detection of real-time linear velocity, and calculation is prone to extremely poor accuracy due to factors such as time delay. However, the second estimated linear velocity generated by the target encoder and inputted into the target policy network can make up for this deficiency. At the same time, in this embodiment, the robot controlled by the target reinforcement learning model does not have a visual sensor. The second tensor can effectively alleviate the problem that the robot cannot know the ground conditions contacted by its feet, resulting in difficulty in outputting a second action that matches its ground environment.

[0165] Regarding the reinforcement learning model training method described in steps 101-105 above, the trained target reinforcement learning model is applied to the control of the simulated robot, and its performance is observed. Please refer to [reference needed]. Figure 6 .like Figure 6 The simulation robot successfully crossed the platform before moving from flat ground to the platform.

[0166] Furthermore, the robot control method based on the reinforcement learning model described in steps 501-503 above was applied to a wheeled robot to test its motion performance. Please refer to [reference needed]. Figure 7 As can be seen, the wheel-legged robot first moves to a platform 0.8m high using wheels, then switches to legged mode to successfully climb the platform. Experiments demonstrate that, based on the target reinforcement learning model provided in this application embodiment, when controlling the robot, it can stably cross a platform with a height almost twice its body height, and maintain a stable motion state after crossing.

[0167] Figures 6-7 The robot's body length is approximately 0.62m, its normal standing height is approximately 0.45m, and the maximum extended length of a single leg is approximately 0.50m.

[0168] It should be understood that, although Figure 1 , Figure 5 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 , Figure 5 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0169] Based on the same inventive concept, such as Figure 8 As shown, this application provides a training apparatus for a reinforcement learning model. The reinforcement learning model to be trained includes an encoder to be trained, a decoder to be trained, and a policy network to be trained connected to the encoder and the decoder to be trained, respectively. The training apparatus includes: an observation unit 801, an encoding unit 802, a state unit 803, a decoding unit 804, and an update unit 804, wherein:

[0170] Observation unit 801 is used to determine the first body observation in response to receiving the first velocity command;

[0171] The encoding unit 802 is used to input the first velocity command and the first body observation into the encoder to be trained to obtain first auxiliary decision information; wherein, the first auxiliary decision information includes a first estimated linear velocity and a first tensor;

[0172] State unit 803 is used to input the first auxiliary decision information into the policy network to be trained to obtain a first action, and update the first simulation state based on the first action to obtain a second simulation state.

[0173] Decoding unit 804 is used to input the first action and the first auxiliary decision information into the decoder to be trained to obtain the first prediction state;

[0174] The update unit 805 is used to update the first parameters of the encoder to be trained and the second parameters of the policy network to be trained based on the difference between the second simulation state and the first prediction state, so as to obtain a target reinforcement learning model containing a target encoder and a target policy network.

[0175] Based on the same inventive concept, such as Figure 9As shown in the figure, this application provides a robot control device based on a reinforcement learning model. The control device includes: an instruction unit 901, an auxiliary unit 902, and an action unit 903, wherein:

[0176] The instruction unit 901 is used to receive control instructions and determine a second body observation; wherein the control instructions include a second speed instruction, and the second body observation includes the second speed instruction;

[0177] The auxiliary unit 902 is used to input the second ontology observation into the target encoder in the target reinforcement learning model to obtain second auxiliary decision information; wherein the target reinforcement learning model includes a target encoder and a target policy network, and the second auxiliary decision information includes a second estimated linear velocity and a second tensor;

[0178] Action unit 903 inputs the second auxiliary decision information into the target policy network to obtain a second action; wherein the second action includes the robot's motion information.

[0179] Specific limitations regarding the training device for reinforcement learning models can be found in the above section on the limitations of training methods for reinforcement learning models; similarly, specific limitations regarding the robot control device based on reinforcement learning models can be found in the above section on the limitations of robot control methods based on reinforcement learning models; these will not be repeated here. Each module in the aforementioned device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the electronic device, or stored in software in the memory of the electronic device, so that the processor can call and execute the operations corresponding to each module.

[0180] Based on the same inventive concept, please refer to Figure 10 This application also provides an electronic device. In one embodiment, the electronic device, as shown in the figure, may include a memory 1001, a communication module 1003, and one or more processors 1002.

[0181] The memory 1001 is used to store computer programs executed by the processor 1002. The memory 1001 may mainly include a program storage area and a data storage area, wherein the program storage area may store the operating system; and the data storage area may store various operation instruction sets, etc.

[0182] Memory 1001 may be volatile memory, such as random-access memory (RAM); memory 1001 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 1001 may be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1001 may be a combination of the above-described memories.

[0183] The processor 1002 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 1002 is used to implement the aforementioned reinforcement learning model training method and the reinforcement learning model-based robot control method when calling computer programs stored in the memory 1001.

[0184] The communication module 1003 is used to communicate with terminal equipment, site equipment or other network equipment.

[0185] This application embodiment does not limit the specific connection medium between the memory 1001, communication module 1003, and processor 1002. This application embodiment... Figure 10 The memory 1001 and the processor 1002 are connected via a bus 1004, and the bus 1004 is in Figure 10 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered limiting. Bus 1004 can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 10 It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0186] The memory 1001 stores a computer storage medium, which stores computer-executable instructions. These instructions are used to implement the reinforcement learning model training method and the robot control method based on the reinforcement learning model, as described in the embodiments of this application. The processor 1002 is used to execute the reinforcement learning model training method and the robot control method based on the reinforcement learning model, as described in the computer-executable instructions.

[0187] Those skilled in the art will understand that Figure 10The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0188] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, wherein when the computer program is executed by a processor, it is a training method for the aforementioned reinforcement learning model or a robot control method based on the reinforcement learning model.

[0189] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0190] Based on the same inventive concept, embodiments of this application also provide a computer program product, including a computer program, which, when executed by a processor, implements the training method of the reinforcement learning model and the robot control method based on the reinforcement learning model described above.

[0191] The program code for executing the computer program product of this application can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0192] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0193] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0194] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0195] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of user-operated steps to be executed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0196] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for training a reinforcement learning model, characterized in that, The reinforcement learning model to be trained includes an encoder to be trained, a decoder to be trained, and a policy network to be trained connected to the encoder and the decoder to be trained, respectively. The method includes: In response to receiving the first velocity command, the first body observation is determined; The first velocity command and the first body observation are input into the encoder to be trained to obtain first auxiliary decision information; wherein, the first auxiliary decision information includes a first estimated linear velocity and a first tensor; The first auxiliary decision information is input into the policy network to be trained to obtain a first action, and the first simulation state is updated based on the first action to obtain a second simulation state. The first action and the first auxiliary decision information are input into the decoder to be trained to obtain the first prediction state; Based on the difference between the second simulation state and the first prediction state, the first parameters of the encoder to be trained and the second parameters of the policy network to be trained are updated to obtain a target reinforcement learning model containing a target encoder and a target policy network.

2. The method as described in claim 1, characterized in that, The step of updating the first parameters of the encoder to be trained and the second parameters of the policy network to be trained based on the difference between the second simulation state and the first prediction state, to obtain a target reinforcement learning model containing a target encoder and a target policy network, includes: Determine the first loss value of the preset reconstruction loss function, and determine the first reward value of the preset reward function; Based on the first loss value, the first parameter is updated to obtain the target encoder, and based on the first reward value, the second parameter is updated to obtain the target policy network; The decoder loss function in the reconstruction loss function is set according to the difference between the first predicted state and the second simulation state, and the first predicted state and the second simulation state correspond to the same time step. The reward function includes an instruction tracking reward function corresponding to the first velocity instruction, and / or a foot height reward function corresponding to the simulated terrain information in the privileged observation.

3. The method as described in claim 2, characterized in that, The reconstruction loss function further includes an encoder loss function; the encoder loss function is obtained through the following method: Determine the simulation linear velocity of the simulation object; The encoder loss function is determined based on the difference between the first estimated linear velocity and the simulated linear velocity.

4. The method as described in claim 2, characterized in that, The reconstruction loss function The expression is as follows: ; Encoder loss function, The decoder loss function; in, ; To simulate linear velocity, The first estimated linear velocity; ; This represents the second simulation state at the second time step. This is the first predicted state at the second time step. The preset balance coefficient, This is the first tensor at the first time step. This represents the simulation state at historical time steps prior to the first time step.

5. The method as described in claim 1, characterized in that, The first tensor includes foot contact force information; the first predicted state corresponds to a preset dynamic equation; wherein, the dynamic equation characterizes the correspondence between the first predicted state, the second predicted state, the first action, and the foot contact force information; The second predicted state, the first action, and the foot contact force information all correspond to the first time step, and the second time step corresponding to the first predicted state is adjacent to the first time step.

6. The method as described in claim 2, characterized in that, The instruction tracking reward function is used to generate a reward based on the difference between the simulated linear velocity in the simulation environment and the first velocity instruction. The instruction tracking reward function is expressed as follows: ; in, ; ; For linear velocity reward, As a bonus for angular velocity, The weight of the linear velocity reward, The weight of the angular velocity reward; The linear velocity in the first speed command. For linear velocity in privileged observations, The turning speed in the first speed command. The turning speed is the one observed in the privileged setting.

7. The method as described in claim 2, characterized in that, The simulated terrain types in the simulated terrain information include flat terrain and elevated terrain. The height difference between the highest and lowest points in the flat terrain is less than a first height threshold, and the height of the elevated platform in the elevated terrain is greater than a second height threshold. The target policy network is trained using the following methods: The type of simulated terrain is determined through the privileged observation; Determine the reward function corresponding to the simulated terrain type; the reward function includes a basic reward function and a high-platform reward function corresponding to the high-platform terrain. Based on the basic reward function and / or the high-platform reward function, a real-time reward corresponding to the first action is generated; Based on the real-time reward corresponding to the first action, the second parameter in the policy network to be trained is updated to obtain the target policy network.

8. A robot control method based on a reinforcement learning model, characterized in that, include: Receive control commands and determine a second body observation; wherein the control commands include a second velocity command, and the second body observation includes the second velocity command; In the target reinforcement learning model, the second ontology observation is input into the target encoder to obtain the second auxiliary decision information; wherein, the target reinforcement learning model includes a target encoder and a target policy network, and the second auxiliary decision information includes a second estimated linear velocity and a second tensor; The second auxiliary decision information is input into the target policy network to obtain a second action; wherein the second action includes the robot's motion information.

9. A training device for a reinforcement learning model, characterized in that, The reinforcement learning model to be trained includes an encoder to be trained, a decoder to be trained, and a policy network to be trained connected to the encoder and the decoder to be trained, respectively. The apparatus includes: The observation unit is used to determine the first body observation in response to receiving the first velocity command; The encoding unit is configured to input the first velocity command and the first ontology observation into the encoder to be trained to obtain first auxiliary decision information; wherein, the first auxiliary decision information includes a first estimated linear velocity and a first tensor; A state unit is used to input the first auxiliary decision information into the policy network to be trained to obtain a first action, and update the first simulation state based on the first action to obtain a second simulation state. The decoding unit is used to input the first action and the first auxiliary decision information into the decoder to be trained to obtain the first prediction state; An update unit is used to update the first parameters of the encoder to be trained and the second parameters of the policy network to be trained based on the difference between the second simulation state and the first prediction state, so as to obtain a target reinforcement learning model containing a target encoder and a target policy network.

10. A robot control device based on a reinforcement learning model, characterized in that, include: The instruction unit is configured to receive control instructions and determine a second body observation; wherein the control instructions include a second speed instruction, and the second body observation includes the second speed instruction; An auxiliary unit is used to input the second ontology observation into the target encoder in the target reinforcement learning model to obtain second auxiliary decision information; wherein the target reinforcement learning model includes a target encoder and a target policy network, and the second auxiliary decision information includes a second estimated linear velocity and a second tensor; The action unit inputs the second auxiliary decision information into the target policy network to obtain a second action; wherein the second action includes the robot's motion information.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and for running on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 8.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 8.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

Images