Training method and device of double-point foot robot squatting action control model

Abstract

The present disclosure provides a training method and device for a double-point-foot robot squatting action control model, and relates to the technical field of robots. The method comprises: in the training process of the squatting action control model, configuring an instruction vector containing squatting instructions for the double-point-foot robot; the squatting instructions are scalar values randomly sampled within a preset range; in each training cycle, the current value of the squatting instructions is read, and in response to the current value meeting a threshold condition, the current training instance is identified as a squatting instance; for the squatting instance, the corresponding gait parameters are set to zero values, and a reward function is configured to obtain a positive reward when the robot knee joint is in contact with the ground and a positive reward when the robot body height is lower than a preset threshold. The present disclosure introduces the squatting instruction dimension in the training stage, so that the control model can distinguish between normal walking behavior and squatting behavior based on the instruction state, thereby having the response ability to switch the squatting action at any posture.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of robots, and relates to a training method and device for a squatting action control model of a biped robot. BACKGROUND

[0002] In the motion control system of a biped robot or a double-wheel robot, when training the robot to perform a non-periodic or posture mutation action such as squatting or prone, a target trajectory or a hard-coded controller is usually relied on. Such a method needs to manually specify a target posture and a joint trajectory, and track control is performed through a traditional control strategy, so as to realize a predefined action.

[0003] However, for a biped robot, since the support area of the biped structure is small, when performing an action such as squatting to reduce the height of the robot body, the requirement for posture stability is high, especially when multi-joint coordinated control and frequent changes in ground contact state are involved, the control difficulty is significantly improved. Based on this, a reinforcement learning framework is introduced to obtain complex motion control ability through a self-learning mechanism, but the existing training method tends to maintain a stable gait pattern for the policy network, and when performing a squatting action, it is difficult to effectively coordinate the multi-joint motion of the lower limbs and the change of the floating base posture, and it is easy to cause the floating base to lean forward or backward during the height reduction process of the robot body, causing the projection of the robot's center of mass to deviate from the support range of the foot, thereby causing support instability. As a result, the success rate of the squatting action is low, and the biped robot is prone to falling, and the falling process lacks controllability and it is difficult to form a stable or controlled failure posture.

[0004] Therefore, it is urgent to provide a new squatting control scheme for a biped robot to enable the biped robot to stably switch to a squatting action at any posture.

[0005] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0006] The present disclosure provides a training method and device for a squatting action control model of a biped robot to overcome the problem that in the related art, the policy network is easily solidified into a stable gait pattern during the reinforcement learning training process, resulting in the biped robot being able to perform squatting only at a specific posture and poor control effect.

[0007] Additional aspects and advantages of the present disclosure will be set forth in part in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present disclosure.

[0008] According to a first aspect of the present disclosure, a training method for a squatting action control model of a biped robot is provided, comprising:

[0009] In the crouching action control model training process, an instruction vector containing a crouching instruction is configured for the two-point foot robot; wherein the crouching instruction is a scalar value randomly sampled in a preset range;

[0010] In each training cycle, the current value of the crouching instruction is read, and in response to the current value meeting a threshold condition, the current training instance is identified as a crouching instance;

[0011] For the crouching instance, the corresponding gait parameters are set to zero values, and the reward function is configured to obtain a positive reward when the robot knee joint is in contact with the ground and a positive reward when the robot body height is lower than a preset threshold.

[0012] In an exemplary embodiment of the present disclosure, the reward function includes a contact reward function and a height control reward function, and the positive reward includes a first positive reward and a second positive reward;

[0013] The reward function is configured to obtain a positive reward when the robot knee joint is in contact with the ground and a positive reward when the robot body height is lower than a preset threshold, comprising:

[0014] In each training step, the first positive reward corresponding to the robot knee joint being in contact with the ground is calculated according to the contact reward function;

[0015] And the second positive reward corresponding to the robot body height being lower than the preset threshold is calculated according to the height control reward function.

[0016] In an exemplary embodiment of the present disclosure, the first positive reward corresponding to the robot knee joint being in contact with the ground is calculated according to the contact reward function, comprising:

[0017] The contact force vector of the robot knee joint and the contact force vector of the foot are obtained, and the contact force norm of the corresponding contact geometry is calculated according to each contact force vector;

[0018] When the contact force norm is greater than or equal to a preset norm threshold, the contact geometry corresponding to the contact force norm is determined as an effective contact point;

[0019] When the number of effective contact points is greater than or equal to a preset number, the first positive reward is output.

[0020] In an exemplary embodiment of the present disclosure, the first positive reward increases with the increase of the number of effective contact points.

[0021] In an exemplary embodiment of the present disclosure, the second positive reward corresponding to the robot body height being lower than the preset threshold is calculated according to the height control reward function, comprising:

[0022] The robot body height is obtained;

[0023] When the robot body height is lower than the preset threshold, calculating a square difference between the robot body height and the preset threshold, and outputting a second positive reward based on the square difference.

[0024] In an example embodiment of the present disclosure, the second positive reward increases with the decrease of the square difference.

[0025] In an example embodiment of the present disclosure, for the squat instance, the reward function is configured to mask the foot speed penalty.

[0026] In an example embodiment of the present disclosure, the method further comprises:

[0027] For the squat instance, the reward function is configured to obtain a positive reward when the robot action amplitude is less than a first amplitude and / or obtain a positive reward when the difference between the robot action amplitudes of adjacent training periods is less than a second amplitude.

[0028] In an example embodiment of the present disclosure, for the squat instance, the reward function is configured to cause the biped robot to obtain a negative reward when an abnormal termination condition is triggered, and reset the corresponding training environment.

[0029] In an example embodiment of the present disclosure, the squat instruction is a scalar value corresponding to a preset dimension in the instruction vector, and the preset range of the sampled squat instruction is [0, 1].

[0030] In an example embodiment of the present disclosure, in response to the current value satisfying a threshold condition, the current training instance is identified as a squat instance, comprising:

[0031] In response to the current value being greater than or equal to a preset scalar value, the current training instance is identified as a squat instance.

[0032] In an example embodiment of the present disclosure, the method further comprises:

[0033] In response to the current value being less than a preset scalar value, the current training instance is identified as a walking instance.

[0034] In an example embodiment of the present disclosure, the method further comprises:

[0035] The size of the preset scalar value is dynamically adjusted based on the training progress between the normal walking task and the squat task.

[0036] In an example embodiment of the present disclosure, the gait parameters include at least one of gait frequency, gait phase, gait offset, gait duration, and swing height.

[0037] According to a second aspect of the present disclosure, a biped robot squat action control method is provided, comprising:

[0038] obtaining a current observation vector and an instruction vector of the biped robot;

[0039] inputting the current observation vector and the instruction vector into a pre-trained squatting action control model, and outputting an action strategy for controlling the biped robot to squat based on the squatting action control model;

[0040] The squatting action control model is obtained according to the training method of the biped robot squatting action control model in the first aspect of the present disclosure.

[0041] In an exemplary embodiment of the present disclosure, the current observation vector of the biped robot includes at least one of a joint position difference value, a joint angular velocity, robot body posture information, body angular velocity, and an action amount in a previous control period.

[0042] In an exemplary embodiment of the present disclosure, the method further includes:

[0043] In response to the biped robot performing the squatting action, generating a motion control instruction according to a difference between a current joint position of the biped robot and a target joint position corresponding to a preset prone posture;

[0044] sending the motion control instruction to the joint driver to drive the joint driver to switch the biped robot from the current squatting posture to the preset prone posture according to the motion control instruction.

[0045] According to a third aspect of the present disclosure, a training device for a biped robot squatting action control model is provided, including:

[0046] An instruction vector configuration module is configured to configure an instruction vector containing a squatting instruction for the biped robot during training of the squatting action control model, wherein the squatting instruction is a scalar value randomly sampled within a preset range;

[0047] A squatting instance identification module is configured to read a current value of the squatting instruction in each training period, and identify a current training instance as a squatting instance in response to the current value meeting a threshold condition;

[0048] A reward function configuration module is configured to set a corresponding gait parameter to zero for the squatting instance, and configure a reward function to obtain a positive reward when the robot knee joint contacts the ground and a positive reward when the robot body height is lower than a preset threshold.

[0049] According to a fourth aspect of the present disclosure, a biped robot squatting action control device is provided, including:

[0050] An input vector acquisition module is configured to obtain a current observation vector and an instruction vector of the biped robot;

[0051] The action strategy output module is used to input the current observation vector and command vector into the pre-trained squatting action control model, and output the action strategy for controlling the squatting of the bipedal robot based on the squatting action control model.

[0052] The squatting motion control model is obtained based on the training method of the bipedal robot squatting motion control model in the first aspect of this disclosure.

[0053] According to a fifth aspect of this disclosure, an electronic device is provided, comprising:

[0054] Processor; and

[0055] A memory that stores computer-readable instructions, which, when executed by a processor, implement the method as described in the above embodiments.

[0056] According to a sixth aspect of this disclosure, a bipedal robot is provided, comprising:

[0057] Processor; and

[0058] A memory that stores computer-readable instructions, which, when executed by a processor, implement the method as described in the above embodiments.

[0059] According to a seventh aspect of this disclosure, a computer-readable storage medium is provided that stores computer program code instructions that, when invoked by a processor, cause the processor to perform the method as described in the above embodiments.

[0060] As can be seen from the above technical solution, this disclosure possesses at least one of the following advantages and positive effects:

[0061] This disclosure introduces a squatting command dimension during the training phase, enabling the control model to distinguish between normal walking and squatting behaviors based on command states, thus preventing the control model from solidifying into a single stable gait pattern during training. With the squatting command dimension, the control model can trigger the learning process of squatting behavior under different initial postures, thereby possessing the responsiveness to switch squatting actions in any posture. Furthermore, this disclosure does not rely on external posture settings or predefined trajectories. By combining squatting commands with environmental feedback for joint modeling and designing rewards for squatting behavior during training, squatting-related control actions are sufficiently strengthened during the training phase. This allows the bipedal robot to achieve more stable squatting control in multi-task environments, reducing instability and falls during squatting. Attached Figure Description

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative labor.

[0063] Figure 1 A system architecture diagram of a training method of a double-point foot robot squatting action control model and a double-point foot robot squatting action control method in the embodiments of the present disclosure is shown.

[0064] Figure 2 A flowchart of a training method of a double-point foot robot squatting action control model in the embodiments of the present disclosure is shown.

[0065] Figure 3 A flowchart of configuring a reward function related to a squatting behavior in the embodiments of the present disclosure is shown.

[0066] Figure 4 A flowchart of configuring a contact reward function in the embodiments of the present disclosure is shown.

[0067] Figure 5 A flowchart of configuring a height control reward function in the embodiments of the present disclosure is shown.

[0068] Figure 6 An execution effect diagram of a double-point foot robot squatting action control method in the embodiments of the present disclosure is shown.

[0069] Figure 7 An effect diagram of a double-point foot robot in a squatting state in the embodiments of the present disclosure is shown.

[0070] Figure 8 A block diagram of a training device of a double-point foot robot squatting action control model in the embodiments of the present disclosure is shown.

[0071] Figure 9 A block diagram of a double-point foot robot squatting action control device in the embodiments of the present disclosure is shown.

[0072] Figure 10 A double-point foot robot in the embodiments of the present disclosure is shown.

[0073] Figure 11 A structural diagram of a computer system of an electronic device suitable for implementing the embodiments of the present disclosure is shown.

[0074] Figure 12 A diagram of a computer readable storage medium in the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0075] In the description of the present disclosure, the terms "first", "second" are only used for description, not indicating relative importance or implying the number of technical features. Therefore, the "first", "second" features can be explicitly or implicitly included at least one of the features. The meaning of "a plurality of" is at least two, unless otherwise explicitly limited.

[0076] Firstly, the related terms involved in the example embodiments of the present disclosure are explained and described:

[0077] Two-point foot robot: refers to a leg-type robot that mainly forms support through two foot ends when in contact with the support plane, and the foot end contact area is limited.

[0078] Squatting action: refers to a control action of a two-point foot robot to lower the body height by flexing the lower limb joints while maintaining the support relationship between the foot end and the support plane, so as to enter a low center of gravity support posture. The squatting action is used for posture adjustment before starting and before power-off of the two-point foot robot, so as to maintain stability when initialization is not completed, control ability is limited, or there is no power. Due to the small number of support points and low stability margin of the two-point foot robot, the squatting action constitutes a basic posture control requirement that distinguishes it from wheeled robots or multi-legged robots.

[0079] Squatting instance: refers to a training sample in the training process that is marked as performing the squatting action due to the squatting instruction meeting the preset conditions. Such instances correspond to training processes that require the robot to complete behaviors related to squatting, such as lowering the body height, approaching the knee joint to the ground, or contacting the ground.

[0080] Instruction vector: refers to a set of control inputs used to drive the robot to perform different action modes in the training process, which can include scalar signals such as randomly sampled squatting instructions. The instruction vector can be used to distinguish between normal gait behavior and squatting behavior, and is the input basis for identifying squatting instances and controlling behavior switching.

[0081] Squatting instruction: a scalar instruction signal indicating whether the robot should perform the squatting action in the current training step. It can be randomly sampled within a preset range and used as part of the control input to guide the behavior differentiation in the training process.

[0082] Gait parameter: refers to a set of control variables that characterize the current gait mode or motion rhythm of the robot. In the squatting instance of the present disclosure, the gait parameter can be set to zero to stop the gait behavior, facilitating the learning of the squatting action by the two-point foot robot.

[0083] Contact reward function: refers to the reward item set according to the contact between the robot and the ground during the training process, including but not limited to the knee joint contacting the ground, the foot end reliably contacting and the like. When the robot performs the squatting behavior, the contact reward function can give positive reward to the action that meets the expected contact mode, guiding the model to learn stable squatting contact behavior.

[0084] Height control reward function: refers to the reward item set according to the relationship between the height of the robot body and the preset height threshold, used to guide the robot to reduce the body height to the target range during squatting. When the height of the robot body is lower than the preset threshold, the height control reward function can give positive reward to promote the model to learn reasonable squatting posture.

[0085] Contact force vector: refers to the force vector generated when the robot contacts the ground, used to describe the force on the foot end or knee joint at the contact point. The vector can be used to calculate the contact force norm or to determine whether contact occurs and its stability, to support contact determination, action stability analysis and the execution of related training logic in the squatting behavior.

[0086] Contact geometry: refers to the specific contact entity of the robot when contacting the ground, including the entity of the foot end contact point, the knee joint contact point or other contact parts and its position and geometry in space, used for contact force calculation and contact-related determination.

[0087] Contact force norm: refers to the norm value (such as Euclidean norm) of the contact force vector generated at a certain contact geometry, used to describe the contact force size of the contact part, which can be used for reward function design or contact stability evaluation.

[0088] Foot speed penalty: refers to the penalty item set according to the speed of the robot foot end, used to suppress unnecessary rapid swinging or excessive sliding behavior. In the squatting instance or stable support process, high foot speed may cause unstable posture, so the foot speed penalty can be used to improve the smoothness of the action.

[0089] Robot action amplitude: refers to the change amount of the joint action of the robot or the numerical amplitude of the joint target instruction. The amplitude can be used to evaluate whether the robot action is too large or unreasonable, and can be used as a penalty item in training to limit too intense or unnatural actions.

[0090] Robot action amplitude difference: refers to the difference between the amplitudes of the robot joint action at adjacent training time steps, used to measure the smoothness of the action change. A larger amplitude difference means that the action is discontinuous or unstable, which can be used for punishment in training to improve the continuity and stability of the robot action.

[0091] Abnormal termination condition: refers to the trigger condition for ending the current training sequence in advance when the robot is obviously unstable, falls down, joint over-limit or physically unreasonable behavior during the simulation training process. For example, the abnormal termination condition includes that the robot body height is lower than the safety threshold, the attitude tilt angle exceeds the limit, the contact force is abnormal, etc. Abnormal termination can avoid continuing to accumulate bad samples and improve training efficiency.

[0092] Figure 1 A system architecture diagram of the training method and the double-point foot robot squatting action control method in the embodiments of the present disclosure is shown. As shown in Figure 1 the system architecture 100 can include a terminal device 101, a robot 102, a network 103 and a server 104.

[0093] The terminal device 101 includes but is not limited to desktop computers, portable computers, smart phones and tablet computers, etc. The terminal device 101 is configured with a graphical user interface for providing visual interaction function in the training process. Through the interface, the state data of the robot 102, the training progress and the sampling distribution of the squatting instruction can be displayed in real time, and the visual configuration of the training parameters is supported. For example, the user can configure the sampling range of the squatting instruction, the squatting threshold parameter, the gait parameter zeroing strategy and the weight coefficient of the knee joint contact and body height term in the reward function through the terminal device. The terminal device 101 can be used in cooperation with the server 104 for monitoring the training process, recording the training data and analyzing the convergence of the training model.

[0094] The robot 102 can be a double-point foot robot, which is used to collect the state information of the robot itself in the training data generation stage. It at least carries inertial measurement devices, joint angle encoders and other sensors, which can be used to obtain attitude, speed, joint angle, foot contact and other data in real time. The robot 102 at least includes a main controller and a joint driver. The main controller can receive the state information collected by the actual sensor in the deployment stage, and execute the squatting action control model obtained in the training stage, so as to generate the target action for controlling the joint driver. The joint driver executes the corresponding joint action according to the action instruction output by the main controller, so that the robot can complete the squatting behavior or normal gait behavior.

[0095] The server 104 is deployed with a training module for performing training of the squatting action control model of the biped robot, and can perform the training process in a simulation environment. In the training process, the server 104 can configure an instruction vector containing squatting instructions for the biped robot in the simulation environment, and read the current value of the squatting instructions at each training cycle. When the squatting instructions meet the preset threshold condition, the server 104 identifies the training instance as a squatting instance, and performs a step parameter zeroing operation on the squatting instance to disable the normal gait mode. At the same time, the server 104 provides positive rewards for the squatting behavior in the training process according to the contact condition of the knee joint of the robot with the ground and whether the height of the robot body is lower than the preset threshold, thereby guiding the model to learn a stable and reliable squatting control strategy. After the training is completed, the server 104 can deploy the obtained squatting action control model to the robot 102 for real-time motion control of the robot in the actual environment.

[0096] The network 103 is used to provide a communication link between the terminal device 101, the robot 102 and the server 104. The network 103 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc. It should be understood that Figure 1 The number and type of terminal devices, robots, networks and servers in the system architecture 100 are only illustrative. According to the implementation needs, there can be any number and any type of terminal devices, robots, networks and servers, for example, the server can be a cloud server.

[0097] Through the cooperative operation of each part in the system architecture 100, the complete process of training phase, including training process configuration containing squatting instructions, squatting instance identification, gait parameter zeroing, reward function calculation and model parameter updating until the deployment of the control model after the training is completed, can be realized, to ensure that the robot can realize stable and reliable squatting control action in any posture.

[0098] The example embodiments of the present disclosure provide a training method of a squatting action control model of a biped robot. Referring to FIG. 1, Figure 2 The method can include the following steps S201 to S203:

[0099] In step S201, an instruction vector containing squatting instructions is configured for the biped robot in the squatting action control model training process; wherein the squatting instructions are scalar values randomly sampled in a preset range;

[0100] In step S202, the current value of the squatting instructions is read at each training cycle, and in response to the current value meeting the threshold condition, the current training instance is identified as a squatting instance;

[0101] In step S203, for the squatting instance, the corresponding gait parameters are set to zero values, and the reward function is configured to obtain positive rewards when the robot knee joint is in contact with the ground and the robot body height is lower than a preset threshold.

[0102] The training method of the double-point-foot robot squatting action control model provided by the example embodiments of the present disclosure can distinguish between normal walking behavior and squatting behavior based on the instruction state by introducing the squatting instruction dimension in the training phase, thereby having the response capability to switch the squatting action at any posture. Moreover, the present disclosure does not need to rely on external posture setting or predefined trajectory, and through the joint modeling of the instruction signal and the environment feedback and the reward function optimization, the control behavior related to squatting can be fully strengthened in the training process, thereby enabling the robot to obtain more stable squatting control effect in a multi-task environment.

[0103] In the following, the training method of the double-point-foot robot squatting action control model in the example embodiments will be described in detail.

[0104] In step S201, during the training of the squatting action control model, an instruction vector containing a squatting instruction is configured for the double-point-foot robot; wherein the squatting instruction is a scalar value randomly sampled in a preset range.

[0105] In the example embodiments of the present disclosure, during the training of the squatting action control model, first, an instruction vector for indicating different training behaviors is set for the double-point-foot robot. In order to enable the double-point-foot robot to contact different degrees of squatting intention in the training phase, the instruction vector of the double-point-foot robot can be extended in the training environment. Specifically, on the basis of the original instruction vector for controlling the behaviors such as speed and direction of the double-point-foot robot, an instruction dimension for indicating the squatting intention is added, that is, the scalar value corresponding to the preset dimension in the instruction vector is taken as the squatting instruction. For example, the instruction vector set in the training environment contains multiple dimensions, and the newly added squatting instruction dimension can be represented as commands[env_id, 3], that is, the third dimension in the instruction vector is used to identify the squatting instruction, which is used to indicate the type of action that the double-point-foot robot should focus on learning in the training process.

[0106] In order to ensure that the training data contains sufficient squatting related cases, the squatting instruction can be sampled in a random manner in a preset range, so that it presents different value levels at different time steps in the training. Wherein, the preset range is used to limit the value range that the squatting instruction can be sampled, and the range is generally set according to the control requirements of the double-point-foot robot performing the squatting action.

[0107] For example, the sampling range of the squatting instruction can be set as [0, 1], the lower limit of the value range can correspond to the normal walking state without performing the squatting action, the upper limit of the value range can correspond to the state of triggering the obvious squatting action, and the middle section is used to cover different degrees of squatting intention. By randomly sampling in the preset range, the instruction distribution formed in the training process can be more uniform, thereby covering various input situations from weak squatting intention to strong squatting intention.

[0108] Correspondingly, in the training process, when the biped robot is in the normal walking state, the value of the commands[env_id, 3] dimension is randomly sampled in the range of [0, 1], so that the biped robot can contact different intensity squatting instruction inputs in continuous training steps.

[0109] In this example, random sampling of the squatting instruction can enable the biped robot to contact various squatting instruction inputs of different intensities in the training phase, thereby covering various transition situations between the normal action state and the potential squatting state. Through this configuration mode, the action data formed in the training process can be more diverse, which is beneficial for the biped robot to gradually establish a response relationship to different squatting instruction values in subsequent training steps, and lays a foundation for realizing stable squatting action control.

[0110] In step S202, in each training cycle, the current value of the squatting instruction is read, and in response to the current value satisfying a threshold condition, the current training instance is identified as a squatting instance.

[0111] Since the squatting instruction is randomly sampled according to the preset range in the training phase, it can present different numerical levels in different training cycles. In order to distinguish between normal training behavior and squatting related training behavior, the squatting instruction can be judged by a pre-set threshold condition. For example, when the current value of the squatting instruction satisfies the threshold condition, it is considered that the training instance corresponding to the training cycle should perform the squatting action, and the training instance is identified as a squatting instance. If the current value of the squatting instruction does not satisfy the threshold condition, the training instance is considered as a training instance in the normal walking or non-squatting state.

[0112] The threshold condition can be a fixed threshold mode, an interval judgment mode, a multi-level threshold mode, etc., and the present disclosure does not limit this. For the fixed threshold mode, a certain fixed value in the preset range is set as the squat judgment threshold, and the squat instance is identified by comparing the current value of the squat instruction with the size of the squat judgment threshold. For the interval judgment mode, the squat instance is identified by judging whether the current value of the squat instruction falls within the preset squat interval. For the multi-level threshold mode, the preset range can be divided into multiple different value sections, respectively corresponding to weak squat, moderate squat and obvious squat, etc. different levels. By judging which squat section the current value of the squat instruction is in, the squat instance is identified.

[0113] In a specific embodiment, the threshold condition is set in a fixed threshold mode. Specifically, a preset scalar value can be selected in the preset sampling range of the squat instruction as the squat judgment threshold. In response to the current value being greater than or equal to the preset scalar value, it is considered that the current training instance corresponds to the squat task requirement, and the training instance is identified as a squat instance. On the contrary, in response to the current value being less than the preset scalar value, the current training instance is identified as a walking instance.

[0114] For example, when the sampling range of the squat instruction is [0, 1], 0.5 can be set as the preset scalar value. If the current value is greater than or equal to 0.5, the biped robot will enter the training process related to squat. If the current value is less than 0.5, the biped robot maintains the normal walking training process.

[0115] It should be noted that, in the actual training process, in order to take into account the distribution ratio of the normal walking task and the squat task in different training stages, the preset scalar value can also be dynamically adjusted according to the training progress, so that the squat instruction judgment mode is more in line with the requirements of the current training stage, thereby improving the stability and convergence effect of the overall training. For example, a higher preset scalar value can be used in the early training stage to reduce the triggering probability of the squat task, so that the biped robot can first master the basic walking behavior. In the later training stage, as the biped robot improves its mastery of the basic behavior, the preset scalar value can be gradually reduced, so that more training instances are identified as squat instances, thereby increasing the participation ratio of the squat task in the training process.

[0116] In addition, in order to further match the squat judgment mechanism with the real-time performance of the training process, the preset scalar value can also be adjusted according to the training state. For example, when there are situations such as unstable motion, reward decrease or slow model convergence speed in the training process, the triggering frequency of the squat task can be adjusted by increasing or decreasing the preset scalar value, so that the difficulty of the training task is consistent with the current training performance, thereby improving the overall training effect.

[0117] Through this step, different types of training instances can be automatically distinguished according to the value of the squat instruction in the training process, so that the training process can accurately identify the timing of performing the squat action without additional manual annotation, thereby providing a basis for performing specific processing on the squat instance in the subsequent step.

[0118] In step S203, for the squat instance, the corresponding gait parameter is set to zero, and the reward function is configured to give positive rewards when the robot knee joint is in contact with the ground and the robot body height is lower than a preset threshold.

[0119] For the training period identified as a squat instance, specific processing is performed to ensure that the biped robot can learn the control behavior related to the squat action in this period. Specifically, when a training instance is identified as a squat instance, the gait parameter corresponding to the training instance can be set to zero to stop the gait modulation in the normal walking mode, so that the biped robot no longer performs the swing, support or periodic gait related actions in this training period, thereby focusing on the learning of squat behavior.

[0120] In the example embodiments of the present disclosure, the gait parameters can include at least one of gait frequency, gait phase, gait offset, gait duration and swing height. The gait frequency is used to represent the repetition speed of the gait cycle. The gait phase is used to represent the relative stage of the biped robot in the gait cycle. The gait offset is used to represent the phase difference between different foot ends. The gait duration is used to describe the time length of a single support or swing period. The swing height is used to represent the height of the foot end from the ground during the swing process.

[0121] Of course, the gait parameters can also include other gait-related variables set based on specific control requirements, such as a swing bias parameter for adjusting the symmetry of left and right foot end swings, a horizontal displacement parameter for adjusting the stride size, an interpolation parameter for describing the foot end target trajectory, etc. The gait parameters can be selectively used according to the specific implementation to depict the gait characteristics of the biped robot in the walking task, and the present disclosure does not limit this.

[0122] Meanwhile, to guide the biped robot to form the correct action mode in the squat instance, a corresponding reward function can be configured in the training process.

[0123] In some example embodiments, when the robot knee joint is in contact with the ground, a positive reward can be given to gradually form a suitable support mode for the squat action of the biped robot. To further guide the biped robot to lower the body height during the squat process, the robot body height can also be monitored, and when the body height is lower than a preset threshold, a positive reward is also given, so that the biped robot gradually forms a posture that meets the requirements of the squat action in the training stage.

[0124] By configuring the reward function, the training process can clearly distinguish between normal gait behavior and squatting behavior when squatting instances occur. On the one hand, setting the gait parameters to zero can prevent the interference of normal walking gait on squatting action learning; on the other hand, the reward design based on knee joint contact and body height can effectively guide the bipedal robot to strengthen key movement patterns related to squatting during training. Thus, the bipedal robot can gradually establish stable and standardized squatting movement patterns during the training phase, laying the foundation for reliable squatting control in subsequent actual operation.

[0125] For example, the reward function includes a contact reward function and a height control reward function. The contact reward function evaluates the contact between the bipedal robot's knee joint and the ground during training and provides a first positive reward when contact occurs. The height control reward function evaluates whether the bipedal robot's height meets the requirements for a squatting posture and provides a second positive reward when the robot's height is below a preset threshold. By combining these two reward functions, the bipedal robot can be effectively guided during the training phase to gradually develop a support pattern and posture adjustment behavior that meets the requirements of a squatting action.

[0126] Specifically, refer to Figure 3 As shown, a reward function related to the squatting behavior can be configured according to steps S301 and S302:

[0127] Step S301: In each training step, calculate the first positive reward corresponding to the robot's knee joint contacting the ground according to the contact reward function.

[0128] When contact between the knee joint and the ground is detected, a first positive reward can be assigned to the training step according to the preset contact reward calculation method to encourage the bipedal robot to form a support mode suitable for the squatting process.

[0129] refer to Figure 4 As shown, step S301 can be further subdivided into steps S401 to S403 to perform a more precise analysis of the contact behavior of the bipedal robot during training, thereby ensuring that the triggering condition of the first positive reward can accurately reflect the actual requirements of the squatting action:

[0130] Step S401: Obtain the contact force vector of the robot's knee joint and the contact force vector of its foot, and calculate the contact force norm of the corresponding contact geometry based on each contact force vector.

[0131] In this step, the knee joint and at least one foot contact geometry of the robot can be pre-set as a target contact point set, and the contact force vector corresponding to the knee joint and the contact force vector of the foot are selected from the contact force data in each training step. Among them, the contact force vector can include force components in the normal direction and the tangential direction, which are used to reflect the force between the relevant contact points of the biped robot and the ground. For each contact force vector obtained, the contact force norm at the corresponding contact geometry is calculated according to each contact force component, and the contact force norm can effectively reflect whether the contact point is in a real force state with supporting significance.

[0132] Exemplarily, the contact force norm can adopt Euclidean norm, normal priority norm or other calculation methods suitable for contact force characterization, and the specific calculation method of the contact force norm can be flexibly selected according to the actual training environment, which can highlight the vertical support of the biped robot to the ground during the squatting process, and the present disclosure does not limit this.

[0133] Among them, the Euclidean norm calculates the overall force size by squaring and summing each component of the contact force vector, which can reflect the comprehensive force of the contact point. The normal priority norm gives higher weight to the normal component in the contact force vector, such as taking only the absolute value of the normal component as the contact force norm, or setting different weights between the normal component and the tangential component, so that the normal force dominates in the judgment. Using the normal priority norm can more accurately reflect the support ability of the knee joint or foot in the vertical direction during the squatting process, thereby improving the reliability of effective contact point identification.

[0134] In step S402, when the contact force norm is greater than or equal to the preset norm threshold, the contact geometry corresponding to the contact force norm is determined as an effective contact point.

[0135] The calculated contact force norm is compared with the preset norm threshold. For example, when the contact force norm is greater than or equal to the preset norm threshold, the contact geometry corresponding to the contact force norm is determined as an effective contact point. The specific value of the preset norm threshold can be set according to the actual required support force level in the squatting action of the biped robot, for example, the minimum force that can form effective support can be estimated according to the typical normal pressure that should be borne by the single leg of the biped robot during the squatting process; or the minimum effective force value that can stably reflect the real contact state can be selected as the norm threshold according to the ground friction coefficient, contact model and noise level in the simulation environment. In addition, statistical analysis can be performed on the contact force data of the knee and foot in the early training stage, and the boundary position that can distinguish light touch and stable contact, i.e. the norm threshold, is determined according to the contact force distribution characteristics; if necessary, the norm threshold can be adjusted according to the change of the training target, so that the contact judgment has more appropriate sensitivity in different training stages.

[0136] It can be understood that the preset norm threshold can be used to eliminate weak contact forces caused by sensor noise, slight touch or short slip, so as to only identify contact points with sufficient supporting effect as effective contact points, and trigger subsequent rewards based on more stable and reliable mechanical determination.

[0137] In step S403, when the number of effective contact points is greater than or equal to the preset number, a first positive reward is output.

[0138] In this step, the number of effective contact points identified in step S402 is counted. When the number of effective contact points is greater than or equal to the preset number, it is considered that the current training step meets the contact characteristics required for the squatting action, and thus a first positive reward is output.

[0139] In some embodiments, the preset number can be set to 2, indicating that at least two squat-related contact parts form effective contact with the ground in the same training step. Specifically, if the sum of the number of effective contact points between the knee joint and the ground and the number of effective contact points formed by the foot is greater than or equal to 2, it indicates that the biped robot has formed a posture with squat supporting characteristics, and thus a first positive reward can be output to encourage the biped robot to strengthen the contact mode conducive to the squatting action during the training process, further improving the stability and realizability of the squatting action.

[0140] In addition, the value of the first positive reward can also be adjusted according to the number of effective contact points. For example, when the number of effective contact points reaches the preset number, a basic value of the first positive reward is triggered. If the number of effective contact points further increases, the value of the first positive reward can be increased according to a preset reward growth rule. The reward growth rule can include, but is not limited to, a linear growth rule, a segmented growth rule or a weighted growth rule for the number of effective contact points, and the specific form of the reward growth rule is not limited in the present disclosure.

[0141] In this way, the training process can give greater rewards to postures with higher supporting stability, guiding the biped robot to form a more stable contact mode during squatting. For example, when the knee joint and the foot both form effective contact with the ground, the contact stability is usually higher than when only the knee joint contacts, and thus a higher first positive reward can be allocated to the biped robot to encourage the biped robot to further strengthen this more stable supporting mode during training.

[0142] In step S302, a second positive reward corresponding to the case where the height of the robot body is lower than the preset threshold is calculated according to a height control reward function.

[0143] In each training step, the current robot body height value is obtained. The robot body height value refers to the height of the robot body base in the vertical direction, which can be directly obtained from the height of the body base in the world coordinate system. In the presence of terrain undulations, the effective height of the body relative to the ground can be obtained by subtracting the terrain height values obtained from multiple measurement points from the base height. This height value is used to describe the height change of the overall posture of the biped robot, and is a basic parameter in height control and posture determination.

[0144] It should be noted that when the instruction vector indicates that the biped robot is in a squatting state, the height control is no longer dependent on the height of the body relative to the terrain, but the height relationship of the foot relative to the body is taken as the control basis. Specifically, the difference between the body height and the height of each foot end can be calculated to obtain the relative height of the foot end in the body coordinate sense, and the relative height is compared with a preset threshold (such as 0.4m). According to the deviation, a reward term is constructed to encourage the robot to form a stable posture with the feet raised and folded during the squatting stage. When the robot body height value is lower than the preset threshold, it is considered that the biped robot has completed the height reduction behavior conforming to the characteristics of the squatting action, and a second positive reward is output according to the height control reward function at this time to encourage the biped robot to form a stable squatting posture during the training process.

[0145] The preset threshold can be used to distinguish between normal walking state and squatting state, and only when the robot body height is lowered to a height interval sufficient to reflect the squatting intention during the training process, the second positive reward is triggered, thereby avoiding false judgment caused by slight posture fluctuations.

[0146] In addition, the height control reward function can gradually increase the reward value according to the reduction amplitude of the robot body height, so that the biped robot further strengthens the more sufficient squatting action during the training process. For example, when the body height gradually approaches the set squatting target height interval, a higher second positive reward can be output to guide the biped robot to form a stable descending motion trajectory. The specific form of the height control reward function is not limited in the present disclosure, and can be flexibly selected according to actual training needs.

[0147] Exemplarily, the robot body height can be obtained, and when the robot body height is lower than the preset threshold, it is considered that the biped robot has a squatting trend, at this time the squared difference between the robot body height and the preset threshold can be calculated, and finally the second positive reward is output based on the squared difference.

[0148] For example, the current robot body height is h , the preset threshold is , when h < 0.4m , the squared difference between the two is calculated as:

[0149]

[0150] After the square difference is calculated, a preset reward mapping mode can be used to make the second positive reward larger when the square difference is smaller, that is, the robot body height is closer to the preset threshold or further lower than the threshold. When the square difference is larger, that is, the robot body height is still much higher than the preset threshold, the second positive reward obtained is correspondingly smaller.

[0151] That is, when the second positive reward is output, the second positive reward increases with the decrease of the square difference, for example:

[0152]

[0153] wherein, k is a proportional coefficient for adjusting the reward amplitude. When the robot body height gradually approaches the preset threshold, the square term of the height difference gradually decreases, so that the corresponding second positive reward also increases, which is beneficial to prompting the biped robot to further approach the target squatting height and maintain a stable posture during the training process.

[0154] By making the second positive reward increase with the decrease of the height square difference, the incentive for sufficient squatting behavior can be strengthened during the training process, so that the biped robot gradually learns to generate a squatting posture closer to the target height. The specific mapping mode of the second positive reward is not limited in the present disclosure, and other reward forms that increase with the decrease of the height difference can be used according to different training targets to achieve the guiding effect on the squatting action.

[0155] In addition, when the robot body height is not lower than the preset threshold, the second positive reward can not be output, or the second positive reward can be set to zero, so as to avoid unnecessary incentive to the biped robot when the squatting height requirement is not reached.

[0156] In the example embodiments of the present disclosure, for the training situation identified as a squatting instance, the foot speed penalty can be shielded when configuring the reward function. Since the squatting action is usually accompanied by slight slippage or non-periodic contact change of the foot relative to the ground, if the foot speed penalty in the normal walking task is still used, it may cause unnecessary interference to the learning of the squatting action. By suspending the effect of the foot speed penalty in the squatting instance, reasonable foot speed changes generated in the squatting process can be avoided from being misjudged as undesirable behavior, so that the reward design is more in line with the dynamic characteristics of the squatting action, which helps the biped robot to stably form a natural and continuous squatting action pattern during the training process.

[0157] In some example embodiments, for the training situation identified as a squatting instance, an additional positive reward can also be set in combination with the robot action amplitude and its change when configuring the reward function.

[0158] Specifically, the action amplitude of each joint of the biped robot can be calculated in each training step, and compared with a preset first amplitude. When the action amplitude of the biped robot in a training step is less than the first amplitude, a corresponding positive reward can be output to encourage the biped robot to maintain a convergent and stable joint motion range during the squatting process, and to avoid excessive instantaneous joint action that causes posture disturbance or squatting action instability.

[0159] Further, an additional reward can also be set according to the action change between consecutive training periods. For example, the differential component of the action amplitude between adjacent training periods can be calculated, that is:

[0160]

[0161] wherein, represents the action amplitude difference, represents the action amplitude of the current training period, represents the action amplitude of the previous training period.

[0162] When the action amplitude difference is less than a preset second amplitude, it can be considered that the action change of the biped robot in consecutive training periods is relatively stable, and thus a positive reward is output to guide the squatting action to form a more continuous and natural dynamic transition.

[0163] The action difference reward can be in the form of inverse proportion or exponential to improve the sensitivity to smooth action. It should be noted that the specific threshold and reward form of the action amplitude and its difference are not limited in the present disclosure, as long as the joint action stability and action smoothness can be effectively promoted in the squatting instance, different reward design methods can be flexibly selected according to the actual training task requirements.

[0164] In some example embodiments, for the training situation identified as the squatting instance, a negative reward can also be set in configuring the reward function in combination with the abnormal termination condition, for constraining the abnormal behavior of the biped robot during the training process. By introducing the abnormal termination mechanism, the current training step can be interrupted in time when abnormal action occurs, to avoid the continuous accumulation of bad action patterns and affect the subsequent policy update.

[0165] Specifically, a plurality of abnormal termination conditions can be set in the training environment, so that the biped robot obtains a negative reward when triggering any abnormal termination condition, to punish the action pattern that causes the training failure, and immediately reset the corresponding training environment, so as to continue the learning of the squatting policy in the new training period.

[0166] The abnormal termination condition can reflect an unstable or unreasonable state of the biped robot in the squatting action. For example, when the biped robot has a serious posture deviation, joint state over-limit, body height rapid instability, or other situations inconsistent with squatting during the execution of the squatting action, it can be determined that the current training step triggers the abnormal termination condition.

[0167] The serious posture deviation refers to a body roll angle or a pitch angle exceeding a preset posture stability threshold; the joint state over-limit refers to a joint angle exceeding a maximum angle range allowed by the mechanism, or an abnormal mutation of joint speed occurring in a short time; the body height rapid instability refers to an abnormal body height drop or an abnormal body height rise in a very short time, which causes the squatting action to be unable to maintain stable support; the contact mode abnormality refers to a loss of contact of the knee joint or the foot or an abnormal slip of the contact force in the squatting stage in which support should be established; and the other situations inconsistent with the squatting task can be a body posture disturbance, an action instruction execution abnormality, or a contact force fluctuation exceeding a reasonable range.

[0168] Of course, the types of the abnormal termination condition, the determination manner, and the specific form of the negative reward are not limited in the present disclosure, and can be flexibly configured according to actual training requirements, simulation environment characteristics, or hardware platform capabilities.

[0169] When any of the above abnormal termination conditions is met, a negative reward can be immediately output for the biped robot to punish the action mode leading to the training failure. Subsequently, the training environment is synchronously reset, including but not limited to reinitializing the robot root state, resetting the joint position and speed, clearing the historical trajectory information, and resampling the instruction vector of the current training period, so that the training process can start again in a stable initial state.

[0170] By introducing the abnormal termination condition and the corresponding negative reward mechanism, the training system can effectively avoid the biped robot repeatedly entering an undesirable action region, improve the stability and convergence speed of the training process. At the same time, the mechanism can also promote the biped robot to gradually form a safer, controllable, and target-compliant squatting action mode in the squatting task.

[0171] After the training is completed, the squatting action control model obtained by the training can be deployed to an actual robot system to generate robot action instructions in a real environment, so as to realize the squatting behavior control of the biped robot under actual running conditions. Accordingly, the example embodiment of the present disclosure also provides a biped robot squatting action control method, as shown in Figure 5 The method can include the following steps S501 and S502:

[0172] Step S501: obtaining a current observation vector and an instruction vector of the biped robot.

[0173] During actual operation of the robot, an observation vector and a corresponding instruction vector of the biped robot at a current time are obtained. The observation vector is used to represent a current motion state of the robot, and the instruction vector is used to indicate a type of action expected to be performed by the robot.

[0174] In the example embodiment of the present disclosure, the current observation vector of the biped robot can include at least one of a joint position difference value, a joint angular velocity, body posture information of the robot, a body angular velocity, and an action amount in a previous control period. The joint position difference value refers to a spatial position change amount of each joint of the robot between adjacent control periods, which can be obtained by subtracting a joint position recorded in a previous control period from a current joint position, and is used to represent a position offset of the joint in consecutive control periods. The joint angular velocity refers to an angular change rate per unit time of each joint, which can be obtained in real time through a joint encoder or a related measuring device, and is used to reflect a current motion speed and a change trend of the joint. The body posture information can include a roll angle, a pitch angle, and a change thereof of the main body of the robot, and is used to describe an orientation of the posture of the robot in space. The body angular velocity is an angular motion speed of the main body of the robot around three axes, and is used to represent a rotation dynamic of the overall posture of the robot. The action amount in the previous control period refers to a control instruction executed by the robot in the previous control period, which can include at least one of a joint position instruction, a speed instruction, or a torque instruction in the previous control period, and is used to provide an action reference in consecutive time sequences to the squat action control model.

[0175] Based on the observation vector, the robot can accurately reflect a current motion state in a running environment with high real-time requirements. At the same time, the instruction vector can include a squat instruction dimension corresponding to the training stage, so that the robot can trigger a squat action or maintain normal walking behavior according to an external input in the deployment stage.

[0176] In step S502, the current observation vector and the instruction vector are input into the pre-trained squat action control model, and an action strategy for controlling the biped robot to squat is output based on the squat action control model.

[0177] After the current observation vector and the instruction vector are obtained, the two are input into the pre-trained squat action control model. The model can generate an action strategy for performing a squat action according to the current observation state and the instruction requirement. The action strategy can include a target joint position, a target joint speed, or a control torque, etc., which are used to drive the robot joints to perform squat-related actions.

[0178] In this example, the robot can automatically generate a matching control action according to the input squat instruction in an actual running environment, thereby realizing a natural and stable squat behavior.

[0179] It is understood that the squatting motion control model used in the reasoning process was trained according to the training method of the bipedal robot squatting motion control model described in detail in other embodiments of this disclosure, and will not be repeated here.

[0180] refer to Figure 6 The diagram shown illustrates the execution effect of a method for controlling the squatting motion of a bipedal robot. Figure 6 It can be seen that during the squatting action, the floating base of the bipedal robot remains within a controlled range, without any obvious forward or backward tilting. The robot's center of gravity remains within the foot support range, thus avoiding support failure due to posture instability. After squatting, the robot forms a stable low-height support posture, and the body does not directly contact the ground, with the support force still transmitted from the lower limb structure to the feet.

[0181] By implementing the squatting motion control method provided in this embodiment, the bipedal robot can achieve smooth and continuous squatting motion switching during posture changes, reducing the occurrence of instability or falls during squatting. It is suitable for posture adjustment needs in scenarios such as power-on preparation, power-off, and multi-task operation.

[0182] refer to Figure 7 The diagram shows the effect of a bipedal robot in a squatting position.

[0183] In addition, after the bipedal robot completes the squatting action based on the squatting motion control model, in order to achieve a further transition from the squatting posture to the preset prone posture, corresponding posture adjustment commands can be generated according to the current joint configuration of the bipedal robot.

[0184] For example, in response to the bipedal robot completing a squatting action, the current joint position of the bipedal robot after completing the squatting action can be obtained, and the difference between the current joint position of the bipedal robot and the target joint position corresponding to the preset prone posture can be calculated.

[0185] After calculating the joint difference, motion control commands can be generated according to a preset posture adjustment strategy, causing the robot joints to transition postures in the desired direction. These motion control commands can be joint position commands, joint velocity commands, or torque control commands, used to drive the bipedal robot to gradually approach a prone posture. This disclosure does not limit the specific form of the motion control commands, which can be flexibly determined according to the type of control system and application requirements.

[0186] Further, the motion control instruction is sent to the joint driver to drive the joint driver to drive each joint of the robot to perform a corresponding action according to the motion control instruction, so that the biped robot smoothly switches from the current squatting posture to the preset prone posture. In the transition process, the robot can maintain the necessary support stability and posture continuity through the precise control of the joint driver, ensuring the safety and controllability of the posture conversion process.

[0187] In the example embodiment of the present disclosure, a training device for a squatting action control model of a biped robot is also provided. Referring to Figure 8 As shown in the figure, the training device 800 for the squatting action control model of the biped robot includes an instruction vector configuration module 801, a squatting instance identification module 802, and a reward function configuration module 803, wherein:

[0188] The instruction vector configuration module 801 is configured to configure an instruction vector containing a squatting instruction for the biped robot during the training process of the squatting action control model; wherein the squatting instruction is a scalar value randomly sampled within a preset range;

[0189] The squatting instance identification module 802 is configured to read the current value of the squatting instruction in each training period, and in response to the current value satisfying a threshold condition, identify the current training instance as a squatting instance;

[0190] The reward function configuration module 803 is configured to set the corresponding gait parameters to zero for the squatting instance, and configure the reward function to obtain a positive reward when the robot knee joint is in contact with the ground and a positive reward when the robot body height is lower than a preset threshold.

[0191] The specific details of each module in the above-mentioned training device for the squatting action control model of the biped robot have been described in detail in the corresponding training method for the squatting action control model of the biped robot, and therefore will not be described here.

[0192] In the example embodiment of the present disclosure, a biped robot squatting action control device is also provided. Referring to Figure 9 As shown in the figure, the biped robot squatting action control device 900 includes an input vector acquisition module 901 and an action strategy output module 902, wherein:

[0193] The input vector acquisition module 901 is configured to acquire a current observation vector and an instruction vector of the biped robot;

[0194] The action strategy output module 902 is configured to input the current observation vector and the instruction vector into the pre-trained squatting action control model, and output an action strategy for controlling the biped robot to squat based on the squatting action control model;

[0195] The squatting action control model is obtained according to a training method of the double-point foot robot squatting action control model as shown in Figure 2

[0196] The specific details of the modules in the double-point foot robot squatting action control device have been described in detail in the corresponding double-point foot robot squatting action control method, and thus will not be described here.

[0197] In the example embodiments of the present disclosure, a double-point foot robot is also provided, which includes a processor and a memory, and the memory stores computer readable instructions, and the computer readable instructions are executed by the processor to implement the above method. Referring to Figure 10 A schematic diagram of a double-point foot robot is shown.

[0198] Referring to Figure 11 An electronic device capable of implementing the above method is also provided. The electronic device 1100 includes a processor 1101 and a memory 1102, and the memory 1102 stores computer readable instructions, and the computer readable instructions are executed by the processor 1101 to implement the method in the embodiments of the present disclosure.

[0199] In the example embodiments of the present disclosure, a computer readable storage medium is also provided, which stores computer program code instructions, and when the computer program code instructions are called by the processor of the robot, the robot executes the method as described in the embodiments.

[0200] Referring to Figure 12 A program product 1200 for implementing the above method according to the embodiments of the present disclosure is described, which can adopt a portable compact disc read-only memory (CD-ROM) and includes program code, and can run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited to this, and in this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in conjunction with an instruction execution system, device or apparatus.

[0201] Through the above description of the embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or a network, and includes a plurality of instructions to make a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) execute the method according to the embodiments of the present disclosure.

[0202] ​Finally, the above preferred embodiments are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail, those skilled in the art should understand that forms and details thereof can be changed without departing from the scope defined by the claims of the present application. The sizes of the drawings are not related to the specific objects, and the sizes of the objects can be arbitrarily changed.

Claims

1. A training method for a squatting motion control model of a bipedal robot, characterized in that, include: During the training of the squatting action control model, a command vector containing squatting instructions is configured for the bipedal robot; wherein, the squatting instruction is a scalar value randomly sampled within a preset range; the lower limit of the preset range represents the normal walking state without squatting, the upper limit represents the state that triggers obvious squatting, and the middle range represents different degrees of squatting intention; In each training cycle, the current value of the squat command is read, and in response to the current value satisfying the threshold condition, the current training instance is identified as a squat instance; For the squatting example, the corresponding gait parameters are set to zero, and a reward function is configured so that the robot receives a positive reward when its knee joint contacts the ground and when the robot's height is below a preset threshold.

2. The training method for the squatting motion control model of the bipedal robot according to claim 1, characterized in that, The reward function includes a contact reward function and a height control reward function, and the positive reward includes a first positive reward and a second positive reward; The configured reward function provides a positive reward when the robot's knee joint contacts the ground and when the robot's height is below a preset threshold, including: In each training step, the first positive reward corresponding to the contact between the robot's knee joint and the ground is calculated according to the contact reward function; And, according to the height control reward function, calculate the second positive reward corresponding to the robot body height being lower than the preset threshold.

3. The training method for the squatting motion control model of the dual-point-foot robot according to claim 2, characterized in that, The step of calculating the first positive reward corresponding to the robot's knee joint contacting the ground according to the contact reward function includes: Obtain the contact force vector of the robot's knee joint and the contact force vector of its foot, and calculate the contact force norm of the corresponding contact geometry based on each contact force vector; When the contact force norm is greater than or equal to a preset norm threshold, the contact geometry corresponding to the contact force norm is determined as an effective contact point. When the number of effective contact points is greater than or equal to the preset number, the first positive reward is output.

4. The training method for the squatting motion control model of the bipedal robot according to claim 3, characterized in that, The first positive reward increases with the number of effective contact points.

5. The training method for the squatting motion control model of the bipedal robot according to claim 2, characterized in that, The step of calculating the second positive reward corresponding to the robot body height being lower than the preset threshold according to the height control reward function includes: Obtain the height of the robot body; When the height of the robot body is lower than the preset threshold, the squared difference between the height of the robot body and the preset threshold is calculated, and the second positive reward is output based on the squared difference.

6. The training method for the squatting motion control model of the bipedal robot according to claim 5, characterized in that, The second positive reward increases as the squared difference decreases.

7. The training method for the squatting motion control model of the dual-point-foot robot according to claim 1, characterized in that, The method further includes: For the squatting example, the foot speed penalty is disabled when configuring the reward function.

8. The training method for the squatting motion control model of the dual-point-foot robot according to claim 1, characterized in that, The method further includes: For the squatting example, the reward function is configured such that a positive reward is obtained when the robot's motion amplitude is less than a first amplitude and / or a positive reward is obtained when the difference in robot motion amplitude between adjacent training cycles is less than a second amplitude.

9. The training method for the squatting motion control model of the bipedal robot according to claim 1, characterized in that, The method further includes: For the squatting example, the reward function is configured so that the bipedal robot receives a negative reward when an abnormal termination condition is triggered, and the corresponding training environment is reset.

10. The training method for the squatting motion control model of the bipedal robot according to claim 1, characterized in that, The squat command is a scalar value corresponding to a preset dimension in the command vector, and the preset range of the squat command obtained by sampling is [0, 1].

11. The training method for the squatting motion control model of a bipedal robot according to claim 1, characterized in that, The step of identifying the current training instance as a squatting instance in response to the current value satisfying the threshold condition includes: In response to the current value being greater than or equal to a preset scalar value, the current training instance is identified as a squatting instance.

12. The training method for the squatting motion control model of the bipedal robot according to claim 11, characterized in that, The method further includes: In response to the current value being less than the preset scalar value, the current training instance is identified as a walking instance.

13. The training method for the squatting motion control model of a bipedal robot according to any one of claims 11 or 12, characterized in that, The method further includes: The preset scalar value is dynamically adjusted based on the training progress between normal walking and squatting tasks.

14. The training method for the squatting motion control model of the bipedal robot according to claim 1, characterized in that, The gait parameters include at least one of gait frequency, gait phase, gait deviation, gait duration, and swing height.

15. A method for controlling the squatting motion of a bipedal robot, characterized in that, include: Obtain the current observation vector and command vector of the bipedal robot; The current observation vector and the command vector are input into the pre-trained squatting motion control model, and the action strategy for controlling the squatting of the bipedal robot is output based on the squatting motion control model. The squatting motion control model is obtained by the training method of the bipedal robot squatting motion control model according to any one of claims 1 to 14.

16. The method for controlling the squatting motion of a bipedal robot according to claim 15, characterized in that, The current observation vector of the bipedal robot includes at least one of the following: joint position difference, joint angular velocity, robot body posture information, body angular velocity, and motion quantity from the previous control cycle.

17. The method for controlling the squatting motion of a bipedal robot according to claim 15, characterized in that, The method further includes: In response to the bipedal robot completing the squatting action, a motion control command is generated based on the difference between the current joint position of the bipedal robot and the target joint position corresponding to the preset prone posture. The motion control command is sent to the joint actuator so that the joint actuator drives the bipedal robot to switch from the current squatting posture to the preset prone posture according to the motion control command.

18. A training device for a bipedal robot squatting motion control model, characterized in that, include: The instruction vector configuration module is used to configure an instruction vector containing a squatting instruction for a bipedal robot during the training of the squatting action control model. The squatting instruction is a scalar value randomly sampled within a preset range. The lower limit of the preset range represents the normal walking state without squatting, the upper limit represents the state that triggers a clear squatting action, and the middle range represents different degrees of squatting intention. The squatting instance recognition module is used to read the current value of the squatting instruction in each training cycle, and in response to the current value satisfying the threshold condition, to recognize the current training instance as a squatting instance; The reward function configuration module is used to set the corresponding gait parameters to zero for the squatting instance, and to configure the reward function so that the robot receives a positive reward when its knee joint contacts the ground and when the robot's height is below a preset threshold.

19. A control device for the squatting motion of a dual-point-foot robot, characterized in that, include: The input vector acquisition module is used to acquire the current observation vector and command vector of the bipedal robot; The action strategy output module is used to input the current observation vector and the instruction vector into the pre-trained squatting action control model, and output the action strategy for controlling the squatting of the bipedal robot based on the squatting action control model. The squatting motion control model is obtained by the training method of the bipedal robot squatting motion control model according to any one of claims 1 to 14.

20. An electronic device, characterized in that, include: processor; as well as A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 17.

21. A bipedal robot, characterized in that, include: processor; as well as A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 15 to 17.

22. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program code instructions that, when invoked by a processor, cause the processor to perform the method as described in any one of claims 1 to 17.

Images