Method and device for training a humanoid robot motion control policy network based on reinforcement learning

Abstract

The application provides a training method and device of a humanoid robot motion control strategy network based on reinforcement learning. When the supporting foot is in a supporting phase, the method performs ray grid sampling on a local area of a foot bottom to obtain N terrain height values; a median of the terrain height values is taken as a local reference height value, and sampling points with an absolute deviation exceeding a preset flatness threshold are counted to obtain a bad point proportion R; a reward punishment term is determined according to the bad point proportion R, written into a reward signal and used to update motion control strategy network parameters, and the punishment degree increases with the increase of the bad point proportion R or remains unchanged. The application adopts a local relative height evaluation of the foot bottom to evaluate landing quality, and can improve the distinguishing ability for stable landing and local stepping, foot palm suspension and the like.

Description

Technical Field

[0001] This invention relates to the field of robotics, and specifically to a training method and apparatus for a humanoid robot motion control strategy network based on reinforcement learning. Background Technology

[0002] When a humanoid robot walks on stairs, it needs to form stable support after its feet touch the ground. When the feet step on the edge of the step or on a local protrusion or depression on the stair surface, even if the feet are in contact with the ground, there may be problems such as the soles of the feet being partially suspended in the air, insufficient effective support area, or unstable support, which can easily cause foot slippage, body swaying, or walking instability.

[0003] To address the aforementioned issues, existing humanoid robot foot contact evaluation schemes typically employ force sensors or binary contact information to determine whether the foot is in contact with the ground. While these schemes can determine the foot's contact status, they struggle to further reflect the uniformity of distance between different points on the foot and the ground, thus failing to effectively distinguish between stable full-foot contact and unstable landing states such as half-foot suspension at the edge of a step, or localized protrusions or depressions. Some schemes also evaluate landing status through depth perception, global terrain mapping, or absolute terrain height comparison; however, these schemes are highly dependent on global terrain information and positioning results, and may be affected by map update delays and positioning errors.

[0004] With the application of reinforcement learning methods in the field of humanoid robot motion control, those skilled in the art have attempted to obtain terrain information around the feet and set a landing reward in the reward function to guide the motion control policy network to learn walking actions on stair terrain.

[0005] For example, Huayi Wang, Zirui Wang, Junli Ren, et al. published the paper "BeamDojo: Learning Agile Humanoid Locomotion on Sparse Footholds" in Proceedings of Robotics: Science and Systems, 2025, DOI: 10.15607 / RSS.2025.XXI.068, which discloses a reinforcement learning motion control framework for humanoid robots. This framework sets up multiple sampling points in the foot area, obtains the terrain height of each sampling point in the global coordinate system through ray projection, and compares the terrain height of each sampling point with a fixed tolerance threshold to count the sampling points that do not effectively contact the safe support area and construct a foothold penalty reward. For example, Yuanhang Zhang, Younggyo Seo, Juyue Chen, et al. published a paper on arXiv entitled "RPL: Learning Robust Humanoid Perceptive Locomotion on Challenging Terrains" (arXiv:2602.03002, 2026), which disclosed a humanoid robot perception and motion control framework. This framework penalizes the landing state of the robot in the edge area, partially supported, or empty support based on the coverage of the support area by a pre-constructed terrain edge mask or foot sampling points.

[0006] The above-mentioned methods can evaluate the foot's landing status based on the terrain height, terrain edges, or coverage of the support area. However, during actual robot walking, the height reference of different steps or different ground areas may change, and the positioning results may also be biased. Furthermore, as the foot gradually transitions from a full landing to partially touching the edge or hovering, its support status will continuously change. Existing foot landing evaluation results are insufficient to fully reflect the actual support status of different foot positions relative to the current supporting ground.

[0007] During reinforcement learning training, if the reward signal cannot accurately reflect the flatness of the local ground after the foot lands, the motor control strategy network will have difficulty effectively distinguishing between different landing states such as stable landing of the whole foot and landing on the edge of the ground or the foot hanging off the ground from the reward feedback. It may learn movement movements in which the foot has made contact with the ground but the actual support quality is poor.

[0008] In summary, existing foot landing evaluation methods are insufficient to accurately reflect the actual support conditions of humanoid robots at different foot positions after landing. This makes it difficult for the reward signals used in reinforcement learning training to effectively distinguish between stable landing states and landing states with poor support quality, such as localized footsteps or feet dangling in the air. Summary of the Invention

[0009] Given that existing foot landing evaluation methods are difficult to accurately reflect the actual support situation of humanoid robots at different foot positions after landing, and that the reward signals used in reinforcement learning training are difficult to effectively distinguish between stable landing states and landing states with poor support quality, this invention provides a training method and device for a humanoid robot motion control strategy network based on reinforcement learning, a humanoid robot motion control method and device based on reinforcement learning, an electronic device, a humanoid robot, and a computer-readable storage medium. The aim is to improve the ability to distinguish between different landing states by establishing a contact quality evaluation method based on the local terrain height distribution of the foot and incorporating the contact quality evaluation results into the reinforcement learning reward signal.

[0010] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be apparent from the description, or may be learned by practice of the invention.

[0011] According to one aspect of the present invention, a training method for a motion control strategy network for a humanoid robot based on reinforcement learning is provided, comprising: when the supporting foot of the humanoid robot is in the supporting phase, acquiring the terrain height values ​​of N sampling points obtained by ray mesh sampling in a local area of ​​the sole of the supporting foot, wherein the terrain height values ​​characterize the measured values ​​of the ground height at the sampling points; using the median of the terrain height values ​​of the N sampling points as the local reference height value, calculating the absolute deviation of the terrain height value of each sampling point relative to the local reference height value, and counting the number of sampling points whose absolute deviation exceeds a preset flatness threshold to obtain the bad point ratio R; determining a reward / penalty term according to the bad point ratio R, writing the reward / penalty term into the reward signal of the motion control strategy network, and updating the parameters of the motion control strategy network based on the reward signal, wherein the penalty degree of the reward / penalty term increases or remains unchanged as the bad point ratio R increases.

[0012] The process of determining reward and penalty items based on the bad pixel ratio R includes: obtaining a preset bad pixel ratio constraint range, which includes a preset lower threshold and a preset upper threshold, wherein the preset lower threshold is less than the preset upper threshold; if the bad pixel ratio R is less than or equal to the preset lower threshold, the value of the reward and penalty item is determined as the minimum penalty value; if the bad pixel ratio R is greater than the preset lower threshold and less than the preset upper threshold, a transition penalty value is determined based on the bad pixel ratio R, and the value of the reward and penalty item is determined as the transition penalty value, wherein the transition penalty value is positively correlated with the bad pixel ratio R; if the bad pixel ratio R is greater than or equal to the preset upper threshold, the value of the reward and penalty item is determined as the maximum penalty value.

[0013] The process of obtaining the preset defective pixel ratio constraint range includes: obtaining the staircase travel direction of the humanoid robot; when the staircase travel direction is upward, determining a first preset lower threshold value for the upward direction as the preset lower threshold value, and determining a first preset upper threshold value for the upward direction as the preset upper threshold value, thus obtaining the preset defective pixel ratio constraint range for the upward direction; when the staircase travel direction is downward, determining a second preset lower threshold value for the downward direction as the preset lower threshold value, and determining a second preset upper threshold value for the downward direction as the preset upper threshold value, thus obtaining the preset defective pixel ratio constraint range for the downward direction; wherein the second preset upper threshold value is less than the first preset upper threshold value, and the second preset lower threshold value is less than the first preset lower threshold value.

[0014] The process of obtaining the staircase travel direction of the humanoid robot includes: during the movement of the humanoid robot, obtaining a sequence of height information of the humanoid robot within a preset time window, the height information sequence including the height sequence of the robot's base or the height sequence of its center of mass; determining the average height change rate of the humanoid robot within the preset time window based on the height information sequence; and determining the staircase travel direction of the humanoid robot based on the average height change rate.

[0015] The method of determining the stair travel direction of the humanoid robot based on the average height change rate includes: if the average height change rate is greater than a preset upward change rate threshold, the stair travel direction is determined to be the upward direction, and the preset upward change rate threshold is greater than 0; if the average height change rate is less than a preset downward change rate threshold, the stair travel direction is determined to be the downward direction, and the preset downward change rate threshold is less than 0.

[0016] If the bad pixel ratio R is greater than the preset lower threshold and less than the preset upper threshold, then the transition penalty value is calculated based on the bad pixel ratio R and the preset exponential function, which is a monotonically increasing function.

[0017] Among them, the reward and penalty items are normalized penalty values, with a minimum penalty value of 0, a maximum penalty value of 1, and a transitional penalty value greater than 0 and less than 1.

[0018] The process of counting the number of sampling points with absolute deviations exceeding a preset flatness threshold to obtain the bad point ratio R includes: identifying sampling points with absolute deviations exceeding the preset flatness threshold as bad points and counting the number of bad points; and determining the bad point ratio R as the ratio between the number of bad points and N, where N is an integer greater than 1.

[0019] Specifically, when the humanoid robot's supporting foot is in the supporting phase, the terrain height values ​​of N sampling points obtained by ray mesh sampling in a local area of ​​the foot's sole are acquired. This includes: determining N sampling points arranged according to a preset mesh in a local area of ​​the humanoid robot's supporting foot's sole; projecting rays from each sampling point along a set projection direction toward the ground to determine the intersection point of the ray corresponding to each sampling point with the ground, setting the projection direction to be parallel to the direction of gravity; acquiring the distance between each sampling point and the corresponding intersection point, and determining the terrain height value of the corresponding sampling point based on each distance.

[0020] The reward signal, which incorporates reward and penalty terms into the motion control strategy network, includes: acquiring contact state information of the supporting foot and determining a contact state coefficient based on the contact state information; determining a mask penalty value based on the value of the reward and penalty terms and the contact state coefficient, and determining a weighted penalty value based on the mask penalty value and a preset penalty weight, wherein the preset penalty weight is less than 0; and obtaining a reward signal based on the weighted penalty value and a preset motion reward term, wherein the motion reward term is used to characterize the degree of completion of the humanoid robot's motion task.

[0021] According to another aspect of the present invention, a motion control method for a humanoid robot based on reinforcement learning is also provided. The method employs a motion control strategy network trained by the above-described training method. The method includes: acquiring state data of the humanoid robot, the state data including the current angles of each joint, the current velocities of each joint, and the robot's posture; inputting the state data into the trained motion control strategy network, and outputting control quantities for each joint from the trained motion control strategy network; and controlling the joint movements of the humanoid robot according to the control quantities for each joint, so that the humanoid robot can walk on a staircase terrain.

[0022] The state data also includes the target motion command of the humanoid robot, which includes at least one of the desired travel speed and desired travel direction. The state data is input into the trained motion control strategy network, and the trained motion control strategy network outputs the control quantities of each joint, including: inputting the state data containing the target motion command, the current angle of each joint, the current speed of each joint, and the body posture into the trained motion control strategy network, and the trained motion control strategy network outputs the control quantities of each joint corresponding to the target motion command.

[0023] The method further includes: controlling the joint movement of the humanoid robot according to the joint control quantities of the current control cycle output by the motion control strategy network within each control cycle; acquiring the updated state data of the humanoid robot after the joint movement; and inputting the updated state data as the state data of the next control cycle into the trained motion control strategy network to obtain the joint control quantities of the next control cycle.

[0024] According to another aspect of the present invention, a training apparatus for a humanoid robot motion control policy network based on reinforcement learning is also provided, the apparatus comprising: The terrain sampling module is used to obtain the terrain height values ​​of N sampling points in a local area of ​​the foot sole of the humanoid robot through ray mesh sampling when the supporting foot is in the supporting phase. The terrain height values ​​represent the measured values ​​of the ground height at the sampling points. The contact assessment module is used to calculate the absolute deviation of the terrain height value of each sampling point relative to the local reference height value, using the median of the terrain height values ​​of N sampling points as the local reference height value, and to count the number of sampling points whose absolute deviation exceeds the preset flatness threshold, thereby obtaining the bad point ratio R. The parameter update module is used to determine the reward and penalty terms based on the bad pixel ratio R, write the reward and penalty terms into the reward signal of the motion control policy network, and update the parameters of the motion control policy network based on the reward signal. The penalty degree of the reward and penalty terms increases or remains unchanged as the bad pixel ratio R increases.

[0025] According to another aspect of the present invention, a humanoid robot motion control device based on reinforcement learning is also provided, employing a motion control strategy network trained by the above-described training method, the device comprising: The state acquisition module is used to acquire the state data of the humanoid robot, including the current angle of each joint, the current speed of each joint, and the robot's posture. The strategy output module is used to input state data into the trained motion control strategy network, and the trained motion control strategy network outputs the control quantities of each joint. The drive control module is used to control the joint movement of the humanoid robot according to the control values ​​of each joint, so that the humanoid robot can walk on the staircase terrain.

[0026] According to another aspect of the present invention, an electronic device is also provided, comprising: a processor; and a memory storing computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, the training method of the humanoid robot motion control policy network based on reinforcement learning, or the humanoid robot motion control method based on reinforcement learning, described above, is implemented.

[0027] According to another aspect of the present invention, a humanoid robot is also provided, comprising: a robot body having supporting feet and a joint drive mechanism for driving joint movement of the robot body; and the aforementioned electronic device, which is controlled and connected to the joint drive mechanism for executing the aforementioned reinforcement learning-based humanoid robot motion control method.

[0028] According to another aspect of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the training method of the humanoid robot motion control policy network based on reinforcement learning, or the humanoid robot motion control method based on reinforcement learning.

[0029] As can be seen from the above technical solution, the present invention possesses at least one of the following advantages and positive effects: This invention departs from existing technologies that rely on the relationship between the absolute terrain height of foot sampling points and a fixed threshold, or on evaluating foot landing status based on preset edge masks and support area coverage. Instead, when the humanoid robot's supporting foot is in the support phase, it performs ray mesh sampling on a local area of ​​the supporting foot's sole. The median of the terrain height values ​​from N sampling points is used as the local reference height value. The proportion of defective points, R, is determined based on the absolute deviation of each sampling point relative to the local reference height value. This allows the local terrain height distribution at the actual foot landing location to serve as the evaluation benchmark, reducing the impact of variations in reference height across different terrains and a few abnormal height values ​​on the landing evaluation results. Furthermore, it transforms the actual flatness of the local area of ​​the sole into a continuous quantitative indicator, the proportion of defective points, R. This improves the ability to distinguish between different landing states, such as stable full-foot landing, partial edge stepping, foot suspension, or stepping on a local protrusion, thus more accurately reflecting the actual flatness of the local area of ​​the sole.

[0030] By determining reward and penalty terms based on the defect rate R, and making the penalty level of the reward and penalty terms increase or remain unchanged as the defect rate R increases, the motion control policy network can learn the correspondence between the local flatness of the sole and the landing quality from the reward feedback. During training, it tends to choose landing states with a smaller defect rate R and a more uniform height distribution between the various positions of the sole and the local support surface. This helps to reduce landing actions with insufficient effective support areas, such as the foot being partially suspended, stepping on the edge of a step, or stepping on a local protrusion. This reduces the possibility of foot slippage, body swaying, or walking instability when the humanoid robot walks on stair terrain, and improves the walking stability of the humanoid robot on stair terrain. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the specific embodiments are briefly described below. It should be understood that the drawings described below only show some embodiments of the present invention.

[0032] Figure 1 This illustration shows a scene of a humanoid robot walking on a staircase in an embodiment of the present invention.

[0033] Figure 2 The diagram illustrates a training method flowchart for a humanoid robot motion control policy network based on reinforcement learning, according to an embodiment of the present invention.

[0034] Figure 3 This diagram illustrates the principle of ray mesh sampling for supporting a local area of ​​the sole of the foot in an embodiment of the present invention.

[0035] Figure 4 A schematic diagram of the planar layout of the ray grid in a local area of ​​the sole of the foot is shown in an embodiment of the present invention.

[0036] Figure 5 This diagram illustrates how a local reference height value is determined based on the median and a bad pixel determination is performed in an embodiment of the present invention.

[0037] Figure 6 This diagram illustrates the mapping relationship between the percentage of bad pixels R and the reward / penalty items in an embodiment of the present invention.

[0038] Figure 7 A schematic diagram of the construction of the contact state mask and reward signal in an embodiment of the present invention is shown.

[0039] Figure 8 A flowchart illustrating a humanoid robot motion control method based on reinforcement learning in an embodiment of the present invention is shown.

[0040] Figure 9 A schematic diagram of the first motion state of the humanoid robot during continuous motion in an embodiment of the present invention is shown.

[0041] Figure 10 A schematic diagram of a humanoid robot in an embodiment of the present invention is shown, showing one leg supporting the robot while the other leg is in a swinging state.

[0042] Figure 11 This diagram illustrates the movement of a humanoid robot's swinging foot toward the target landing position in an embodiment of the present invention.

[0043] Figure 12 A schematic diagram of a humanoid robot swinging its foot toward a target support surface is shown in an embodiment of the present invention.

[0044] Figure 13This diagram illustrates the contact between the swinging foot of the humanoid robot and the target support surface in an embodiment of the present invention.

[0045] Figure 14 This diagram illustrates the state of the humanoid robot after the foot support relationship has been transformed in an embodiment of the present invention.

[0046] Figure 15 The diagram shows a structural block diagram of a training device for a humanoid robot motion control policy network based on reinforcement learning, according to an embodiment of the present invention.

[0047] Figure 16 A block diagram of a humanoid robot motion control device based on reinforcement learning is shown in an embodiment of the present invention.

[0048] Figure 17 A schematic diagram of the structure of an electronic device according to an embodiment of the present invention is shown.

[0049] Figure 18 A first external schematic diagram of a humanoid robot according to an embodiment of the present invention is shown.

[0050] Figure 19 A second appearance schematic diagram of a humanoid robot according to an embodiment of the present invention is shown.

[0051] Figure 20 A schematic diagram of a computer-readable storage medium according to an embodiment of the present invention is shown. Detailed Implementation

[0052] In the description of this invention, the terms "first" and "second" are used for description only and do not indicate relative importance or imply the number of technical features. Therefore, the features referred to as "first" or "second" may explicitly or implicitly include at least one of those features. "A plurality of" means at least two, unless otherwise expressly defined.

[0053] The following describes the main terms used in the specific embodiments. Unless otherwise specified in the table, the terms should be understood in accordance with the specific descriptions in the subsequent embodiments of this specification.

[0054] The specific implementation of this embodiment will be described below with reference to the accompanying drawings. (See also...) Figure 1 The humanoid robot 110 walks on the stair steps 120. In the picture, one foot acts as a supporting foot 130 and contacts the step, while the other foot acts as a swinging foot 140 and moves to the next landing position.

[0055] The robot body can be equipped with joint angle encoders, joint velocity detection components, and inertial measurement units to obtain the joint angles, joint velocities, and body posture. During actual operation, the foot contact status can also be obtained by components such as foot force sensors and contact switches. In the physical simulation environment, the above states, as well as the intersection points of rays and the ground, can be provided by the simulation engine.

[0056] The motion control policy network receives robot state data and outputs control variables for each joint. During training, the robot interacts with the environment to generate state, action, and reward data, which the training program uses to update the network parameters. The network structure and the reinforcement learning algorithm employed can be determined based on the specific robot platform.

[0057] The following explanation uses physical simulation training as an example. A rigid body model, joint actuation model, and staircase terrain model of the robot are established in the simulation environment. At the start of a control cycle, the robot's current state is read and input into the motion control strategy network. The robot's movement is propelled according to the network output, and then the motion task reward and supporting foot contact quality reward for that cycle are calculated.

[0058] Local contact quality is calculated only for the area near the current supporting foot, without requiring the creation of a complete global terrain map for each control cycle. Local ray queries can be performed at the same or similar cycles as the underlying whole-body control (WBC) or reinforcement learning reward calculations. For example, updates can be performed at 50Hz to 100Hz, with the actual frequency set according to the simulation step size and control cycle.

[0059] refer to Figure 2 The training method includes steps S201 to S203.

[0060] Step S201: When the humanoid robot's supporting foot is in the supporting phase, obtain the terrain height values ​​of N sampling points in the local area of ​​the foot's sole obtained by ray mesh sampling. The terrain height values ​​represent the measured values ​​of the ground height at the sampling points.

[0061] Step S202: Using the median of the terrain height values ​​of N sampling points as the local reference height value, calculate the absolute deviation of the terrain height value of each sampling point relative to the local reference height value, and count the number of sampling points whose absolute deviation exceeds the preset flatness threshold to obtain the bad point ratio R.

[0062] Step S203: Determine the reward / penalty item based on the bad pixel ratio R, write the reward / penalty item into the reward signal of the motion control policy network, and update the parameters of the motion control policy network based on the reward signal. The penalty degree of the reward / penalty item increases or remains unchanged as the bad pixel ratio R increases.

[0063] Figure 2The steps shown are processed according to the local terrain data of the current supporting foot. After obtaining the terrain height values ​​of N sampling points in step S201, step S202 uses the median of the N terrain height values ​​as the local reference height value, and calculates the bad point ratio R based on the absolute deviation of each sampling point relative to the local reference height value; step S203 determines the reward and penalty items based on the bad point ratio R and writes them into the reward signal. When going upstairs and downstairs, the corresponding bad point ratio constraint range can be used respectively; when the foot is in the supporting phase but has not yet formed actual contact, the corresponding penalty item can be shielded according to the contact state information.

[0064] refer to Figure 3 and Figure 4 As shown, in step S201, the current supporting foot is determined based on the current gait phase or foot movement state. For the current supporting foot, a local area 330 of the sole is determined near the sole, and multiple sampling points 340 are set within the local area 330 of the sole. Figure 4 The planar arrangement of sampling points in a local area of ​​the sole is further shown, with N sampling points 340 arranged in a preset grid within the local area 330 of the sole. Figure 3 Sampling point 340 and Figure 4 Sampling point 340 in the image represents a sampling location of the same type. Figure 3 It is mainly used to illustrate the relationship between the sampling point and the staircase terrain during the ray sampling process. Figure 4 It is mainly used to illustrate the arrangement of each sampling point in a local area of ​​the sole of the foot.

[0065] Specifically, the positions of N sampling points can be pre-set in the local coordinate system of the foot supporting the foot. The N sampling points can be arranged according to a rectangular grid, or the positions of the sampling points near the edge of the foot can be adjusted according to the contour of the foot. Figure 4 The local area 330 of the sole shown can cover the main support area of ​​the sole and can be appropriately extended to the toes, heels, and left and right sides of the foot as needed, so that the sampling range can cover the areas where the foot may be partially suspended or stepped on when actually landing. The spacing between adjacent sampling points 340 can be set according to the size of the foot and the dimensions of the stair steps, and it is not required that the sampling spacing in all directions be exactly the same.

[0066] In some implementations, different sampling point spacings can be set according to the contact quality evaluation requirements of different areas of the sole. For example, a smaller sampling point spacing can be set in the toe, heel, or edge areas of the foot, while a larger sampling point spacing can be set in the middle area of ​​the sole. The specific arrangement of the sampling points is not limited to a regular rectangular grid. For example, the position of each sampling point can be adjusted according to the robot's foot contour and the contact situation of different areas of the sole during walking, so that the sampling points can cover the main support area of ​​the sole and the areas prone to stepping on edges.

[0067] In some implementations, different sized local areas on the sole of the foot can be set. For example, a first local area can be set near the center of the sole, and a second local area can be set around the first local area. The first local area can be used to evaluate the flatness of the terrain in the robot's current main support area, and the second local area can be used to obtain terrain height information near the edge of the sole. When evaluating foot contact quality, the terrain height value corresponding to one of the local areas can be selected for calculation according to actual needs, or evaluation results corresponding to different local areas can be obtained separately.

[0068] Within the current control cycle, based on the current pose of the supporting foot, each sampling point in the local coordinate system of the foot is transformed to the reference coordinate system used in the training environment, thereby obtaining the actual position of each sampling point 340. When the supporting foot translates or rotates in a subsequent control cycle, the local area 330 of the foot and the sampling points 340 therein are redefined according to the current pose of the supporting foot, without using the absolute sampling positions from the previous control cycle. Therefore, the local ray mesh always corresponds to the terrain region near the current supporting foot.

[0069] For each sampling point 340, a ray 350 is projected in a direction parallel to the direction of gravity and toward the ground. Figure 3 In the diagram, direction g represents the direction of gravity. Each ray 350 extends along this direction from the corresponding sampling point 340 towards the ground and intersects with the corresponding step surface, forming a corresponding intersection point 360. For N sampling points 340, N rays 350 and corresponding N intersection points 360 can be obtained respectively.

[0070] Figure 3 The diagram also shows the distance between each sampling point 340 and the corresponding intersection point 360. Each distance represents the distance from the ray origin at the corresponding sampling location to the terrain surface. For example, the ray distance corresponding to the i-th sampling point can be denoted as d. i The terrain height value corresponding to the sampling point can be obtained directly from the height coordinates of the intersection point 360 in the reference coordinate system, or it can be calculated based on the ray starting point height of the sampling point and the corresponding ray distance. If the ray starting point height of the i-th sampling point is z... i And since the ray is projected vertically downwards, it can be calculated according to h. i =z i -d i Determine the terrain height value h of the i-th sampling point i Each sampling point uses the same reference coordinate system and projection direction, so that the obtained N terrain height values ​​can be used for subsequent comparisons.

[0071] Figure 3In this design, the first step surface 310 and the second step surface 320 are located at different heights. The first step surface 310 represents the surface of the step on which the supporting foot is currently located, and the second step surface 320 represents the adjacent, lower step surface. When most of the sole of the supporting foot is on the first step surface 310, and part of the toe, heel, or side of the foot approaches or crosses the edge of the step, a portion of the ray 350 within the current step area intersects with the first step surface 310, with the corresponding intersection point 360 located on the first step surface 310. The portion of the ray 350 that crosses the edge of the step continues downward and intersects with the second step surface 320, with the corresponding intersection point 360 located on the second step surface 320.

[0072] Therefore, when the support foot rests completely within the surface 310 of the first step, the terrain height values ​​corresponding to most or all sampling points are usually within a similar height range. When part of the support foot crosses the edge of the step, the terrain height values ​​corresponding to some sampling points will have a more significant height difference from the other sampling points. For cases where there are local protrusions or depressions under the foot, the terrain height values ​​at the corresponding locations can also be obtained using the intersection points 360 of each ray 350 and the terrain surface in the same way.

[0073] It should be noted that steps S201 and... Figure 3 , Figure 4 The process shown is mainly used to obtain the terrain height values ​​of N sampling points within a local area of ​​the current supporting foot. In this step, the height of a single sampling point is not directly used to determine whether it is a bad point. After completing the local ray sampling, the median of the N terrain height values ​​is used as the local reference height value in subsequent steps, and bad points are determined based on the absolute deviation of each sampling point relative to this local reference height value.

[0074] In the single-support phase, the aforementioned local ray sampling can be performed only on the current supporting foot. When the left foot is in the support phase, the corresponding local area of ​​the sole and the sampling point position are determined based on the current pose of the left foot; when the right foot is in the support phase, the same processing is performed based on the current pose of the right foot. In the dual-support phase, separate local ray meshes can be created for the left and right feet, and N terrain height values ​​are obtained for each foot. Subsequently, their respective local reference height values ​​and bad pixel ratios are calculated separately, without mixing the sampling data from the left and right feet in the calculation.

[0075] refer to Figure 5 Explain step S202. Obtain h1, h2, ..., h N Then, the height values ​​are sorted, and the median is used as the local reference height value h. ref It can be written as: h ref =median(h1, h2, ..., h N).

[0076] Figure 5 Local reference height h in China and Israel ref As the intermediate baseline, and with h ref+ε and h ref-ε This indicates the allowable height deviation range. Sampling points within this range are considered normal points, while those outside the range are considered defective points. The normal and defective points in the figure are only used to illustrate the classification relationship based on height deviation; the actual number of defective points is still based on the actual sampling results of the current control cycle.

[0077] When the support feet are mostly located on the same step surface, and only a few sampling points extend beyond the step edge, most sampled values ​​remain concentrated around the height corresponding to the current step. The median can be used to represent the main support height in this local area. Compared to the mean, a few higher or lower sampled values ​​have a smaller impact on the median.

[0078] Since the defect determination is based on the absolute deviation of each sampling point relative to the current local reference height value, the judgment is mainly based on the relative height relationship between each sampling point in the current local area of ​​the foot, and it is not required to obtain the absolute height of the current step in advance.

[0079] In some implementations, the local reference height value can also be determined based on a subset of the N terrain height values. For example, the N terrain height values ​​can be sorted according to their magnitude, and the terrain height value corresponding to a preset quantile position can be selected as the local reference height value. Alternatively, after sorting, some of the larger and smaller terrain height values ​​can be removed, and the local reference height value can be determined based on the remaining terrain height values. Using the above methods can reduce the influence of a few high or low sampling points on the local reference height determination result.

[0080] In other embodiments, a local terrain plane corresponding to a local area of ​​the sole can be fitted based on the positions of the N sampling points and the terrain height values, and a reference height corresponding to each sampling point can be determined based on the local terrain plane. In this case, the deviation between the terrain height value of each sampling point and the corresponding reference height can be used to determine whether the sampling point is a bad point. This method can be used when there is a certain degree of inclination in the local area of ​​the sole.

[0081] Determine h ref Then, the absolute deviation δ of each sampling point is calculated. i : δ i =|h i -h ref |

[0082] δ i Compare with the preset flatness threshold ε. When δi When δ > ε, the i-th sampling point is recorded as a bad point; when δ > ε, the i-th sampling point is recorded as a bad point. i When N ≤ ε, it is not included in the number of defective pixels. Number of defective pixels N bad The ratio of dead pixels, R, can be expressed as: ; R = N bad / N.

[0083] Where I(·) is an indicator function, taking the value 1 when the condition is true and 0 otherwise. The value of R ranges from 0 to 1. Once a sampling point exceeds ε, further increases in its deviation will not increase the number of bad pixels; only when more sampling points exceed ε will R continue to increase.

[0084] For example, when the support foot rests entirely on a relatively flat step surface, the sampling heights are usually quite close, and R is small. When part of the support foot crosses the edge of the step, the ray that crosses the edge may fall onto the lower ground or the next step, and the corresponding sampling point is easily recorded as a bad point. Local protrusions may also cause the absolute deviation of the corresponding sampling point to exceed ε.

[0085] The preset flatness threshold ε can be set according to foot size, step size, and training task; for example, it can be set to around 4 cm. The "dead points" here are determined based on local terrain height deviations and are not equivalent to the non-contact points detected by the contact sensor. Contact detection is mainly used to confirm whether the foot has made actual contact.

[0086] refer to Figure 6 and Figure 7 Explain step S203. Figure 6 This indicates the relationship between the percentage of bad pixels (R) and the reward / penalty terms. Figure 7 This indicates the combination of the reward / penalty item with the motion reward item after processing by the contact state mask and penalty weight.

[0087] Figure 6 The horizontal axis represents the percentage of bad pixels, R, and the vertical axis represents the reward / penalty item, P(R). The solid and dashed lines represent the changes when different threshold ranges are used in the upstairs and downstairs directions, respectively. Figure 7 The data flow for reward calculation is as follows: first, P(R) is obtained from R, and then the contact state coefficient M is used. contact Obtain the mask penalty value P mask After the preset penalty weight w p The weighted penalty value r is obtained after processing. flat Finally, with the sports reward item r motion The combination yields a reward signal r.

[0088] During actual training, a lower threshold R can be preset. L and upper limit threshold R U And R L <RU R≤R L When the reward / penalty term P(R) is at its minimum penalty value, R... L <R<R U When R is greater than or equal to R, calculate the transition penalty value; U When P(R) reaches its maximum penalty value.

[0089] The reward / penalty term P(R) can be normalized to 0 to 1, for example, the minimum penalty value is 0 and the maximum penalty value is 1. When R is between two thresholds, the transition penalty value is between 0 and 1 and is positively correlated with R or remains unchanged.

[0090] The transition region can employ an exponential function. Continue using the aforementioned lower threshold R. L and upper limit threshold R U One calculation method is as follows: P(R) = 0, R ≤ R L ; P(R) = [1 - exp(-λ(R - R)] L ))] / [1-exp(-λ(R U -R L ))],R L <R<R U ; P(R) = 1, R ≥ R U Where λ>0.

[0091] The parameter λ is used to adjust R L With R U The rate of change of the penalty value between [the two parameters]. In actual training, other monotonically increasing functions can also be used, as long as [the value is within R]. L With R U Simply maintain a proper transitional relationship between them.

[0092] The exponential form described above is only one implementation of the reward and penalty terms. In some implementations, other functions that increase or remain unchanged in penalty intensity as the proportion of defective points increases can also be used to determine the reward and penalty terms. For example, piecewise linear functions or other continuous nonlinear functions can be used to determine the reward and penalty values ​​corresponding to different proportions of defective points. The specific function form can be set according to the constraint strength of foot contact quality during training. When using different function forms, a smaller proportion of defective points can correspond to a smaller penalty, and the penalty intensity for the corresponding foot landing state can increase as the proportion of defective points increases.

[0093] When using the above function, in R L <R<R U Within the range, P(R) is positively correlated with the proportion of bad pixels R or remains unchanged.

[0094] Different dead pixel ratio constraint ranges can be applied for going upstairs and downstairs. The upstairs direction uses the first dead pixel ratio constraint range [R]. L1 R U1 The second bad point ratio constraint range is used in the direction of going downstairs [R]. L2 R U2 ], where R L2 <R L1 R U2 <R U1 This means that the penalty begins to incur when going downstairs at a relatively low percentage of defective pixels, and reaches its maximum penalty at a relatively low percentage of defective pixels. The calculation method for the defective pixel percentage R itself does not change whether going upstairs or downstairs.

[0095] Specifically, the defective pixel ratio ranges for the uphill and downhill directions can be set separately based on the actual contact characteristics between the robot's feet and the stair steps. For example, during the uphill process, a certain degree of local suspension near the robot's toes is permissible, so the corresponding defective pixel ratio range can be appropriately relaxed; during the downhill process, to reduce the possibility of the foot stepping on the edge of the step or insufficient support area, a relatively strict defective pixel ratio range can be set. The specific thresholds can be set based on the robot's foot size, step size, and the motion state obtained during training.

[0096] When going upstairs and downstairs, corresponding defect percentage constraint ranges can be set separately. During the process of going upstairs, when the swinging foot moves towards the next higher step, the toes may be close to the edge of the step, so a certain range of partial toe suspension is allowed; during the process of going downstairs, in order to reduce the situation of the heel missing or insufficient local support, the preset lower limit threshold and preset upper limit threshold used in the direction of going downstairs can be lower than the corresponding thresholds used in the direction of going upstairs.

[0097] The direction of the staircase can be determined based on the height of the machine's base or center of gravity over a period of time. Let the sampling time within the preset time window be from t1 to t2. K The corresponding altitude information is H1 to H K H K Let represent the height of the body base or the height of the center of mass at the k-th sampling time. The average height change rate can be calculated using the following formula: v H =(H K -H1) / (t K -t1).

[0098] Alternatively, the rate of change of height between adjacent sampling times can be calculated first, and then averaged. If v H Greater than the preset rate of change threshold θ up , and θ up If v > 0, then it is determined to be the direction of going upstairs; if v H Less than the preset rate of change threshold θdown , and θ down If the value is less than 0, then the direction is determined as the direction to go downstairs.

[0099] When the average height change rate is at θ down With θ up During this process, the previously determined direction of stair travel can be maintained, or it can be determined by combining historical trajectory data to avoid frequent changes in direction due to height fluctuations within a single gait cycle. Preset time windows can be set according to gait cycles and control cycles. Stair travel direction determination can be performed in parallel with local foot scanning.

[0100] The preset time window can be set according to the gait cycle and control cycle. This preset time window is used to determine the direction of stair travel and is not involved in the calculation of the defect ratio R.

[0101] In some implementations, when the average height change rate is between a preset decreasing change rate threshold and a preset increasing change rate threshold, the stair travel direction determined in the previous control cycle can be maintained.

[0102] In other implementations, when the average height change rate is between a preset decreasing change rate threshold and a preset increasing change rate threshold, the direction of stair travel can be determined based on the historical height change trend within a preset time range; when the historical height shows an overall upward trend, it is determined as the direction of going upstairs, and when the historical height shows an overall downward trend, it is determined as the direction of going downstairs.

[0103] In other implementations, other methods may be used to determine the direction of stair travel, which are not limited here.

[0104] When writing reward and penalty items into the reward signal, the contact state information of the supporting foot is first obtained, and the contact state coefficient M is determined. contact When the support meets the preset contact conditions, M contact Option 1 is acceptable; before actual contact is established, M... contact 0 is acceptable. Figure 7 The mask penalty value P in mask It can be written as P mask =M contact ×P(R). The contact state information of the supporting foot can also be obtained in parallel with ray mesh sampling.

[0105] The contact status can be determined based on the contact force of the foot, or the foot-terrain contact markers provided by the simulation environment can be used directly. Figure 7 In the implementation shown, step S201 has already screened the foot that needs to be evaluated based on the support phase, therefore the contact state coefficient is M. contact That's fine. If the system also has a support phase coefficient M... step Alternatively, you can first use M step Complete the support foot screening, and then use Mcontact Determine whether actual contact has been made with the foot.

[0106] Understandably, the purpose of setting a contact state mask is to ensure that the evaluation results of the local topography of the foot primarily affect the foot that is currently providing support. For example, when a foot is in the swing phase or has not yet made effective contact with the terrain, even if there are many bad spots in the corresponding local area of ​​the foot, the current movement state can be treated without applying the same level of reward or penalty based on the proportion of bad spots. Once the foot enters the support phase and makes effective contact, the corresponding reward or penalty is determined based on the proportion of bad spots.

[0107] When the supporting feet have made actual contact, M contact Take 1; when the supporting feet have not yet formed actual contact, M contact The value is set to 0. Therefore, feet that have not yet actually made contact with the ground are not included in the contact quality penalty calculation for the current control cycle.

[0108] According to P(R) and M contact Get P mask Then, it is combined with a preset penalty weight w that is less than 0. p Obtain the weighted penalty value r flat ,For example: r flat =w p ×P mask w p <0.

[0109] Then, with sports reward items r motion The combination yields the reward signal r for the current control cycle: r = r motion +r flat .

[0110] Sports reward items r motion The specific configuration can be set according to the reinforcement learning training framework adopted. When the left and right feet are supporting the weight simultaneously, the R, P(R), and M values ​​of each foot are calculated separately. contact Then, the weighted penalty values ​​for both feet are combined with the exercise reward. Each foot still uses the median of its respective local plantar region.

[0111] In implementation, M contact This can be obtained before reward calculation and corresponds one-to-one with the P(R) of the current supporting foot. For dual-support situations, the M values ​​for the left and right feet are saved separately. contact This avoids using the contact state of one foot for the penalty calculation of the other foot.

[0112] The training program optimizes parameters using state data, joint control variables output by the policy network, state changes after robot execution, and reward signals. Network parameters are not necessarily updated in every control cycle; for example, a segment of interaction data can be collected first, and then updated according to training batches or preset trajectory lengths.

[0113] In one implementation, the training program can also record training logs, which include at least one of the following: training batches, state data, joint control quantities output by the motion control policy network, reward signals, and corresponding network parameter versions. When saving the network parameters of the motion control policy network, a corresponding hash value can be generated based on the saved network parameters, and the hash value can be associated with and saved in the corresponding training log to distinguish network parameters obtained at different training stages.

[0114] Training rounds can end upon reaching the maximum number of control cycles, completing the current stair-walking task, or the robot falling. Additional penalties can be set according to the adopted training framework for abnormal termination such as a fall.

[0115] refer to Figure 8 After training, the motion control strategy network can be used to control the movement of the humanoid robot. The motion control method includes steps S801 to S803.

[0116] Step S801: Obtain the state data of the humanoid robot, which includes the current angle of each joint, the current speed of each joint, and the body posture.

[0117] Step S802: Input the state data into the trained motion control strategy network, and output the control quantities of each joint from the trained motion control strategy network.

[0118] Step S803: Control the joint movement of the humanoid robot according to the control values ​​of each joint so that the humanoid robot can walk on the staircase terrain.

[0119] The bad pixel ratio R and the reward / penalty term are used in the aforementioned training phase. After training is completed, the trained motion control policy network is directly invoked during the deployment phase to obtain the control quantities of each joint based on the current state data, without updating the parameters of the motion control policy network based on the reward signal.

[0120] During the deployment phase, the angles and velocities of each joint can be obtained by the joint status detection component, and the body posture can be obtained by the posture measurement component. The definition and arrangement of the status data remain consistent with those during the training phase. If target motion commands are used during training, the corresponding desired travel speed, desired travel direction, and other commands are also input during deployment.

[0121] When using target motion commands, these commands can be provided by the upper-level task module. The desired travel speed, desired travel direction, current joint angles, joint velocities, and body posture can be combined into state data according to the input order during training, and then fed into the trained policy network. If the target command remains unchanged for several control cycles, the current command can continue to be used; after the upper-level task module updates the target command, subsequent control cycles will use the updated target command.

[0122] In step S802, the current state data is input into the trained policy network to obtain the joint control variables for the current control cycle. The form of each joint control variable is consistent with the network output definition during the training phase. For example, the target joint position is output during training, and the same output definition is used during deployment.

[0123] In step S803, the corresponding joint movement is driven according to the joint control quantities. The joint control quantities can be sent directly to the joint control interface, or they can be converted into corresponding drive commands by the robot's underlying controller.

[0124] Motion control can be executed in a closed loop according to the control cycle. After the joint control quantity is executed in the t-th control cycle, the joint angle, joint velocity, and body posture are re-acquired as the state data for the (t+1)-th control cycle, and then input into the strategy network again.

[0125] For example, at the start of a control cycle, the current state data is acquired, and the joint control input is obtained by the policy network. After the robot executes the joint control input, the new joint angle, joint velocity, and body posture are read as inputs for the next control cycle. In this way, each control cycle uses the actual state of the robot after executing the control of the previous cycle.

[0126] refer to Figures 9 to 14 The figure shows several foot states of the robot during continuous movement. Figure 9 The robot's two legs are positioned near the support surface, and the body is in one state during continuous motion; upon entering the next state, such as... Figure 10 As shown, one foot remains supporting the weight, while the other foot has been lifted and entered the swing phase.

[0127] like Figure 11 and Figure 12 As shown, the swinging foot continues to move forward and downward while the supporting foot maintains the body's support, and the distance between the sole of the foot and the target support surface gradually decreases. This stage is still part of the landing process, and whether actual contact has been formed can be determined by contact state information, rather than solely by the foot position shown in the diagram.

[0128] like Figure 13 As shown, the swinging foot moves to a new landing position and makes contact with the supporting surface; subsequently, the load-bearing relationship between the two feet changes, as... Figure 14 As shown, the original swinging foot gradually assumes the supporting role, while the other foot enters the subsequent gait. During continuous walking, the above-mentioned processes of support, swinging, landing, and support transition can alternate between the left and right feet.

[0129] Figures 9 to 14 The diagram illustrates the robot's support, swing, footing, and support relationship transitions during continuous movement. During the training phase, the proportion of defective nodes (R) is determined based on the sampling results of the aforementioned local ray mesh on the foot. R is also calculated and rewards are constructed during the training phase using the aforementioned local ray mesh. During the deployment phase, the joint control quantities are output from the trained policy network based on the state data.

[0130] The aforementioned contact quality evaluation method can also be applied to uneven terrain such as gravel roads. When there are bumps or depressions in a local area of ​​the sole, if the absolute deviation of the terrain height value of the corresponding sampling point from the local reference height value exceeds a preset flatness threshold, the sampling point is identified as a bad point. Training can be conducted in a physical simulation environment or in a real environment where the local terrain height value of the sole and the robot's state data can be obtained.

[0131] During the dual-support phase, the local reference height value and defect rate R of the left and right feet are calculated separately, without merging the sampling points corresponding to the two feet. The number of sampling points N, the coverage of the sampling grid, and the sampling interval can be set according to the foot size, stair geometry, and training computation load.

[0132] The following is an explanation of a specific training process. The robot has a left and a right foot and undergoes reinforcement learning training in a simulated staircase environment. For ease of description, the foot currently in the support phase of the control cycle is referred to as the target support foot. For single support, only the contact quality of that foot is calculated; for double support, the contact quality of the left and right feet is calculated separately.

[0133] After the training round begins, the robot is positioned at a preset location in front of or within the stairs, with the desired travel speed and direction specified. The training environment reads state data such as joint angles, joint velocities, and body posture, inputs this data into the motion control strategy network to obtain control variables for each joint, and then advances the simulated dynamic state.

[0134] After completing the state progression of the current control cycle, read the pose of the target support foot, update the position of N sampling points in RayCast Grid according to the pose, project rays from each sampling point along the direction of gravity to the ground, obtain the corresponding intersection points, and obtain N terrain height values.

[0135] Taking the left foot as an example, after the left foot enters the support phase, the median of the above N terrain height values ​​is taken to obtain the local reference height h. ref ; then calculate δ i =|h i -href | and statistically analyze N using ε as a preset flatness threshold. bad Thus, we get R = N bad / N. The calculation is performed independently using the same method when the right foot enters the support phase. The above calculation uses the current local scan height data and does not require querying the current step number or absolute height.

[0136] Simultaneously, the machine's base height or center of gravity height within the most recent preset time window is read and the average height change rate is calculated. Based on this, the corresponding bad point ratio constraint range for the uphill or downhill direction is selected. The stair travel direction is only used to select the constraint range; the calculation method for R remains unchanged.

[0137] P(R) is calculated based on the position of R relative to the preset lower and upper thresholds. P(R) = 0 when R is not greater than the lower threshold; a transition penalty value between 0 and 1 is calculated using a preset function when R is between the two thresholds; and P(R) = 1 when R is not less than the upper threshold. Then, a mask is applied based on the foot contact state, multiplied by a penalty weight less than 0, and this, along with the motion reward, constitutes the reward signal for the current control cycle.

[0138] When the target support foot has not yet made actual contact, through M contact The contact quality bonus of the foot is shielded; feet not in the support phase pass the support foot screening or M. step Not participating in the current support quality evaluation.

[0139] The training program optimizes parameters using the state, joint control variables, reward signals, and updated state of each control cycle. After each control cycle ends, the updated robot state enters the next control cycle. When a foot changes position, the local ray mesh is also updated accordingly.

[0140] The training process is repeated for multiple rounds until the training termination condition is met, and the parameters of the trained policy network are saved. After deployment to the robot controller, the robot state is read in each control cycle, the trained policy network is input to obtain the control variables for each joint, and the robot's movement is controlled. During the deployment phase, reinforcement learning rewards are no longer calculated based on P(R), nor are these rewards used to update network parameters.

[0141] refer to Figure 15 The training device 1500 includes a terrain sampling module 1510, a contact evaluation module 1520, and a parameter update module 1530. The terrain sampling module 1510 is used to obtain the terrain height values ​​of N sampling points in a local area of ​​the support foot sole; the contact evaluation module 1520 is used to calculate the local reference height value, absolute deviation, and bad point ratio R; the parameter update module 1530 is used to determine the reward and penalty terms based on R and update the policy network parameters based on the reward signal.

[0142] The terrain sampling module 1510, contact evaluation module 1520, and parameter update module 1530 can be implemented in software or a combination of software and hardware. For example, the terrain sampling module 1510 can call the ray detection interface based on the current pose of the supporting foot to obtain the terrain height values ​​corresponding to N sampling points; the contact evaluation module 1520 can calculate the local reference height value, the absolute deviation of each sampling point, and the bad point ratio R based on the N terrain height values; the parameter update module 1530 can determine the reward signal based on the bad point ratio R and the contact state information, and update the parameters of the motion control strategy network based on the reward signal. The remaining processing of each module can refer to the aforementioned training method embodiment.

[0143] refer to Figure 16 The motion control device 1600 includes a state acquisition module 1610, a strategy output module 1620, and a drive control module 1630. The state acquisition module 1610 is used to acquire robot state data; the strategy output module 1620 inputs the state data into the trained motion control strategy network and obtains the control quantities of each joint; the drive control module 1630 controls the movement of the robot joints according to the control quantities of each joint.

[0144] Figure 16 The state acquisition module 1610, strategy output module 1620, and drive control module 1630 correspond to the three main processing stages during deployment. The state acquisition module 1610 provides the current state data to the strategy output module 1620, and the output of the strategy output module 1620 is then sent to the drive control module 1630, which acts on the robot joints.

[0145] The motion control device 1600 can repeatedly execute the above processing according to the control cycle. After the current cycle ends, the state acquisition module 1610 obtains the updated state, and the strategy output module 1620 generates the joint control quantities for the next control cycle accordingly. Each module can be implemented by different program segments in the same robot controller, or by different functional units.

[0146] When implemented on the robot body, the state acquisition module can read data from the joint state interface and the posture measurement interface, the policy output module calls the trained motion control policy network, and the drive control module then sends the network output to the corresponding joint control interface. If the underlying joint controller requires other instruction formats, the corresponding conversion can be performed in the drive control module.

[0147] By separating state acquisition, policy reasoning, and drive control, the connection between the policy network and the underlying joint interface becomes clearer. Different robot platforms can use the same deployment process as long as they can provide the corresponding state data and convert the policy output into the joint control commands required by their platform.

[0148] refer to Figure 17 The electronic device 1700 includes a processor 1701 and a memory 1702. The memory 1702 stores computer-readable instructions. When the processor 1701 executes these computer-readable instructions, it can implement the aforementioned training method or motion control method for the motion control strategy network. The processor 1701 can be a central processing unit, a graphics processing unit, a digital signal processor, an application-specific integrated circuit, a field-programmable gate array, or other processing unit capable of executing corresponding instructions; the memory 1702 can be a random access memory, a read-only memory, flash memory, or other storage devices.

[0149] Figure 17 This diagram illustrates the basic computational and storage relationships required to implement the aforementioned method. The processor 1701 and memory 1702 read and write program instructions, network parameters, and runtime data. The diagram does not specify the exact models of the processor and memory, nor does it require them to be located in the same physical location on the robot body.

[0150] The training and deployment phases can be performed using the same electronic device 1700, or they can be executed on different electronic devices. The specific deployment method used depends on the computing resources and control architecture of the robot system.

[0151] When the processor 1701 executes the training program, it can read robot state data, training parameters, and policy network parameters from the memory 1702, and complete terrain sampling result processing, reward calculation, and network parameter updates. When executing the deployment program, it reads the trained policy network parameters and calculates the control quantities of each joint based on the current state data. The two programs can be stored in the same memory, or they can be configured separately for the training device and the robot controller.

[0152] Using the above-described electronic device structure, the processor 1701 can call the program and parameters in the memory 1702 to complete the training or deployment process. Training and deployment can be completed on the same device or configured separately on the training server and the robot controller, which facilitates arranging the program execution location according to the actual computing power conditions.

[0153] refer to Figure 18 and Figure 19 The image shows two humanoid robots with different shapes. The humanoid robot consists of a robot body and electronic equipment. The robot body has a torso, left lower limb, right lower limb, left foot, and right foot. Each lower limb is connected to the torso through multiple joints, which are driven by joint drive mechanisms. The left and right feet can alternately serve as supporting feet and swinging feet during walking.

[0154] Figure 18 and Figure 19This is mainly used to illustrate that the method can be applied to different humanoid robot bodies. Both robots have two feet capable of ground support and driveable lower limb joints. The specific torso shape, joint installation method, and foot shape may differ. This embodiment does not change the aforementioned local evaluation and strategy control process of the foot based on these shape differences.

[0155] The electronic devices can be integrated into the robot body or housed in a controller that communicates with the robot body. The electronic devices are connected to the joint drive mechanisms and can deploy the aforementioned trained motion control strategy network. During operation, the electronic devices acquire robot state data according to the control cycle, output control quantities for each joint, and control the corresponding joint drive mechanisms.

[0156] Taking a bipedal robot as an example, the left and right feet are connected to the torso via corresponding lower limb joints. After the electronic device outputs the joint control quantity for a certain control cycle, it is executed by the corresponding joint drive mechanism; after the robot moves, the state is re-acquired before entering the next control cycle. Figure 18 and Figure 19 The robots shown have different shapes, but the control relationships described above are the same.

[0157] Figure 18 and Figure 19 The shape shown is only an example of the robot body. The torso, lower limbs, feet, and joint structures may differ between different robot platforms, as long as the aforementioned state data can be obtained and the corresponding joint control can be executed.

[0158] After the electronic equipment establishes a control connection with the joint drive mechanism, the output of the trained motion control strategy network can directly affect the joint movements of the robot body. Therefore, even Figure 18 and Figure 19 Although the robots shown have different shapes and mechanical layouts, as long as the state data and joint control interfaces meet the aforementioned relationship, motion control can still be completed according to the same control cycle.

[0159] refer to Figure 20 The computer-readable storage medium 2000 stores a computer program. When executed by a processor, the computer program can implement the training method or motion control method of the aforementioned motion control strategy network. The computer-readable storage medium can be a read-only memory, random access memory, flash memory, solid-state drive, hard disk, optical disk, USB flash drive, or other media capable of storing programs.

[0160] Figure 20 The storage medium 2000 shown is used to store program content that can be read by the processor. For training purposes, it can store programs related to local terrain sampling, bad point ratio calculation, reward construction, and parameter updates; for deployment purposes, it can also store the trained policy network parameters, as well as programs related to state reading, policy inference, and joint control output.

[0161] The computer program can be divided into several program segments according to its function. For example, the training program may include program segments for local terrain sampling, bad spot ratio calculation, reward signal calculation, and policy parameter updating; the deployment program may include program segments for state reading, policy network inference, and joint control output. The specific division of program segments does not affect the execution relationship of the aforementioned method steps.

[0162] After the above program is stored in a computer-readable storage medium, it can be loaded and executed by a corresponding processor without changing the aforementioned method steps themselves. In this way, the same training program or deployment program can be installed on different training devices or robot controllers as needed.

[0163] The foregoing describes several specific implementations of the present invention. In practical use, the sampling grid form, threshold value, exponential function parameters, and implementation method of each functional module can be determined in conjunction with the robot platform and training environment.

Claims

1. A training method for a humanoid robot motion control policy network based on reinforcement learning, characterized in that, include: When the humanoid robot's supporting foot is in the supporting phase, the terrain height values ​​of N sampling points obtained by ray mesh sampling in a local area of ​​the sole of the supporting foot are acquired. The terrain height values ​​represent the measured values ​​of the ground height at the sampling points. Using the median of the terrain height values ​​of the N sampling points as the local reference height value, the absolute deviation of the terrain height value of each sampling point relative to the local reference height value is calculated, and the number of sampling points whose absolute deviation exceeds a preset flatness threshold is counted to obtain the bad point ratio R. The reward and penalty items are determined based on the bad pixel ratio R, and the reward and penalty items are written into the reward signal of the motion control strategy network. The parameters of the motion control strategy network are updated based on the reward signal. The penalty degree of the reward and penalty items increases or remains unchanged as the bad pixel ratio R increases.

2. The training method for the humanoid robot motion control policy network based on reinforcement learning according to claim 1, characterized in that, The step of determining the reward and penalty items based on the bad pixel ratio R includes: Obtain a preset defective pixel ratio constraint range, wherein the defective pixel ratio constraint range includes a preset lower threshold and a preset upper threshold, and the preset lower threshold is less than the preset upper threshold; If the percentage of bad pixels R is less than or equal to the preset lower threshold, then the value of the reward / penalty item is determined as the minimum penalty value. If the bad pixel ratio R is greater than the preset lower threshold and less than the preset upper threshold, then a transition penalty value is determined based on the bad pixel ratio R, and the value of the reward penalty item is determined as the transition penalty value. The transition penalty value is positively correlated with the bad pixel ratio R. If the percentage of bad pixels R is greater than or equal to the preset upper limit threshold, then the value of the reward / penalty item is determined as the maximum penalty value.

3. The training method for the humanoid robot motion control policy network based on reinforcement learning according to claim 2, characterized in that, The process of obtaining the preset defective pixel ratio constraint range includes: Obtain the staircase travel direction of the humanoid robot; When the direction of travel of the stairs is the upward direction, the first preset lower limit threshold value for the upward direction is determined as the preset lower limit threshold value, and the first preset upper limit threshold value for the upward direction is determined as the preset upper limit threshold value, so as to obtain the preset bad point ratio constraint range for the upward direction. When the direction of travel of the stairs is downstairs, the second preset lower limit threshold set for the downstairs direction is determined as the preset lower limit threshold, and the second preset upper limit threshold set for the downstairs direction is determined as the preset upper limit threshold, so as to obtain the preset bad point ratio constraint range for the downstairs direction; Wherein, the second preset upper limit threshold is less than the first preset upper limit threshold, and the second preset lower limit threshold is less than the first preset lower limit threshold.

4. The training method for the humanoid robot motion control policy network based on reinforcement learning according to claim 3, characterized in that, The step of obtaining the staircase travel direction of the humanoid robot includes: During the movement of the humanoid robot, the height information sequence of the humanoid robot within a preset time window is acquired, and the height information sequence includes the height sequence of the robot's base or the height sequence of its center of mass. Based on the height information sequence, determine the average height change rate of the humanoid robot within the preset time window; The direction of the humanoid robot's movement on the stairs is determined based on the average height change rate.

5. The training method for the humanoid robot motion control policy network based on reinforcement learning according to claim 4, characterized in that, Determining the stairwell travel direction of the humanoid robot based on the average height change rate includes: If the average height change rate is greater than a preset upward change rate threshold, the staircase travel direction is determined to be the upward direction, and the preset upward change rate threshold is greater than 0. If the average height change rate is less than a preset descent change rate threshold, the direction of travel on the stairs is determined to be the direction of going downstairs, and the preset descent change rate threshold is less than 0.

6. The training method for the humanoid robot motion control policy network based on reinforcement learning according to claim 2, characterized in that, If the bad pixel ratio R is greater than the preset lower threshold and less than the preset upper threshold, then a transition penalty value is determined based on the bad pixel ratio R, including: If the bad pixel ratio R is greater than the preset lower threshold and less than the preset upper threshold, then a transition penalty value is calculated based on the bad pixel ratio R and a preset exponential function, wherein the preset exponential function is a monotonically increasing function.

7. The training method for the humanoid robot motion control policy network based on reinforcement learning according to claim 2, characterized in that, The reward / penalty item is a normalized penalty value, the minimum penalty value is 0, the maximum penalty value is 1, and the transition penalty value is greater than 0 and less than 1.

8. The training method for the humanoid robot motion control policy network based on reinforcement learning according to claim 1, characterized in that, The step of counting the number of sampling points whose absolute deviation exceeds a preset flatness threshold to obtain the bad pixel ratio R includes: Sampling points whose absolute deviation exceeds the preset flatness threshold are identified as bad points, and the number of bad points is counted. The ratio between the number of bad pixels and N is defined as the bad pixel ratio R, where N is an integer greater than 1.

9. The training method for a humanoid robot motion control policy network based on reinforcement learning according to claim 1, characterized in that, When the humanoid robot's supporting foot is in the supporting phase, the terrain height values ​​of N sampling points obtained by ray mesh sampling in a local area of ​​the sole of the supporting foot are acquired, including: In a local area of ​​the sole of the supporting foot of the humanoid robot, N sampling points are determined according to a preset grid arrangement; Rays are projected from each of the sampling points along a predetermined projection direction toward the ground to determine the intersection point of the ray corresponding to each sampling point with the ground. The predetermined projection direction is parallel to the direction of gravity. Obtain the distance between each sampling point and its corresponding intersection point, and determine the terrain height value of the corresponding sampling point based on each distance.

10. The training method for a humanoid robot motion control policy network based on reinforcement learning according to claim 1, characterized in that, The step of writing the reward / penalty term into the reward signal of the motion control policy network includes: Obtain the contact state information of the supporting foot, and determine the contact state coefficient based on the contact state information; Based on the value of the reward / penalty item and the contact state coefficient, a mask penalty value is determined, and a weighted penalty value is determined based on the mask penalty value and a preset penalty weight, wherein the preset penalty weight is less than 0. The reward signal is obtained based on the weighted penalty value and the preset motion reward item, and the motion reward item is used to characterize the degree of completion of the humanoid robot's motion task.

11. A motion control method for humanoid robots based on reinforcement learning, characterized in that, The motion control policy network trained using the training method described in any one of claims 1 to 10, wherein the motion control method comprises: Acquire the state data of the humanoid robot, the state data including the current angle of each joint, the current speed of each joint, and the body posture; The state data is input into the trained motion control strategy network, and the trained motion control strategy network outputs the control quantities of each joint. The joint movements of the humanoid robot are controlled according to the joint control values, so that the humanoid robot can walk on the staircase terrain.

12. The humanoid robot motion control method based on reinforcement learning according to claim 11, characterized in that, The status data also includes the target motion command of the humanoid robot, the target motion command including at least one of the desired speed and desired direction of travel; The step of inputting the state data into the trained motion control strategy network, and having the trained motion control strategy network output the control quantities for each joint, includes: The state data, including the target motion command, the current angles of each joint, the current speeds of each joint, and the body posture, is input into the trained motion control strategy network, and the trained motion control strategy network outputs the control quantities of each joint corresponding to the target motion command.

13. The humanoid robot motion control method based on reinforcement learning according to claim 11, characterized in that, The method further includes: Within each control cycle, the joint movement of the humanoid robot is controlled according to the joint control quantities of the current control cycle output by the motion control strategy network. Obtain updated state data of the humanoid robot after joint movement; The updated state data is used as the state data for the next control cycle and input into the trained motion control strategy network to obtain the joint control quantities for the next control cycle.

14. A training device for a humanoid robot motion control policy network based on reinforcement learning, characterized in that, include: The terrain sampling module is used to acquire the terrain height values ​​of N sampling points obtained by ray mesh sampling in a local area of ​​the sole of the humanoid robot when the supporting foot is in the supporting phase. The terrain height values ​​represent the measured values ​​of the ground height at the sampling points. The contact assessment module is used to calculate the absolute deviation of the terrain height value of each sampling point relative to the local reference height value, using the median of the terrain height values ​​of the N sampling points as the local reference height value, and to count the number of sampling points whose absolute deviation exceeds a preset flatness threshold, thereby obtaining the bad point ratio R. The parameter update module is used to determine a reward / penalty item based on the bad pixel ratio R, write the reward / penalty item into the reward signal of the motion control strategy network, and update the parameters of the motion control strategy network based on the reward signal, wherein the penalty degree of the reward / penalty item increases or remains unchanged as the bad pixel ratio R increases.

15. A motion control device for a humanoid robot based on reinforcement learning, characterized in that, The motion control device comprises a motion control strategy network trained using the training method described in any one of claims 1 to 10, and the network includes: The status acquisition module is used to acquire the status data of the humanoid robot, including the current angle of each joint, the current speed of each joint, and the body posture. The strategy output module is used to input the state data into the trained motion control strategy network, and the trained motion control strategy network outputs the control quantities of each joint. The drive control module is used to control the joint movement of the humanoid robot according to the joint control values, so that the humanoid robot can walk on the staircase terrain.

16. An electronic device, characterized in that, include: processor; as well as The memory stores computer-readable instructions, which, when executed by the processor, implement the training method for a humanoid robot motion control policy network based on reinforcement learning as described in any one of claims 1 to 10, or implement the humanoid robot motion control method based on reinforcement learning as described in any one of claims 11 to 13.

17. A humanoid robot, characterized in that, include: A robot body having supporting legs and a joint drive mechanism for driving the joint movement of the robot body; as well as The electronic device of claim 16, wherein the electronic device is controlled to the joint drive mechanism for performing the humanoid robot motion control method based on reinforcement learning as described in any one of claims 11 to 13.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the training method for a humanoid robot motion control policy network based on reinforcement learning as described in any one of claims 1 to 10, or implements the humanoid robot motion control method based on reinforcement learning as described in any one of claims 11 to 13.

Images