Foot lifting movement training method and device for humanoid robot, electronic equipment and medium

Abstract

The embodiment of the invention discloses a humanoid robot foot lifting movement training method and device, electronic equipment and a medium. A specific embodiment of the method comprises the following steps: acquiring current observation sample information; generating simulated elevation terrain information; inputting the current observation sample information into an action network to obtain an action probability distribution value set; inputting the current observation sample information into a strategy network to obtain an action evaluation value set; generating a target joint position set according to the action probability distribution numerical value set and the action evaluation value set; inputting the target joint position set into a joint response controller to generate a joint torque set; controlling the humanoid robot to carry out foot lifting and moving training; and ending the training of the motor position control network. According to the embodiment, the humanoid robot is controlled to carry out foot lifting movement training on the rough ground through the motor position control network and the joint response controller, and the movement stability of the humanoid robot and the performance and pertinence of the motor position control network can be improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a method, device, electronic device, and medium for training a humanoid robot to lift its feet and move. Background Art

[0002] A humanoid robot is a robot that imitates human structure to perform similar movement behaviors as humans. Humanoid robots have complex leg joint structures, which can achieve higher flexibility and environmental adaptability, and achieve smooth walking in various complex environments. With the rapid development of computer technology, foot-lifting movement training based on humanoid robots has received more and more attention. For the training of robot foot movement off the ground, the usual method is to build a motion reference trajectory and gait library for the humanoid robot. Then, the motion reference trajectory and gait library are used as gait priors to guide the strategy network to imitate and learn similar gaits to complete the foot-lifting movement training of the humanoid robot.

[0003] However, it is found in practice that when the above method is used to train the foot lift-off movement of a humanoid robot, the following technical problems often occur: the need to collect motion reference trajectories and gait libraries increases the cost of additional data collection and wastes storage and computing resources, and the strategy model using motion reference trajectories and gait libraries learns specific gaits for specific environments, which has poor environmental adaptability to humanoid robots, resulting in poor stability in controlling the movement of the humanoid robot and a high damage rate of the humanoid robot.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0005] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] Some embodiments of the present disclosure propose a method, device, electronic device and medium for training a humanoid robot to lift its feet and move, so as to solve one or more of the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide a method for training a humanoid robot to lift its feet and move, comprising: obtaining current observation sample information of the humanoid robot, wherein the humanoid robot is a bipedal humanoid robot with clog-type soles, and the current observation sample information includes at least one of the following: joint positions, joint velocities, joint angles, and ground coupling pressure information of the two feet of the humanoid robot; generating simulated elevation terrain information for the humanoid robot; inputting the current observation sample information into an action network included in a motor position control network to obtain a numerical set of action probability distributions; and inputting the current observation sample information into an action network included in a motor position control network to obtain a numerical set of action probability distributions. The strategy network includes a strategy network, obtains the action evaluation value set corresponding to the above-mentioned action probability distribution value set; generates the target joint position set of the above-mentioned humanoid robot according to the above-mentioned action probability distribution value set and the above-mentioned action evaluation value set; inputs the above-mentioned target joint position set into a joint response controller to generate a joint torque set for the above-mentioned humanoid robot; controls the above-mentioned humanoid robot to perform foot lift-off movement training on the simulated elevation terrain corresponding to the above-mentioned simulated elevation terrain information according to the above-mentioned joint torque set; in response to determining that the above-mentioned action probability distribution value set and the above-mentioned action evaluation value set meet the preset loss condition, ends the training of the above-mentioned motor position control network.

[0008] In a second aspect, some embodiments of the present disclosure provide a humanoid robot foot-lifting movement training device, comprising: an acquisition unit, configured to acquire current observation sample information of the humanoid robot, wherein the humanoid robot is a bipedal humanoid robot with clog-type soles, and the current observation sample information includes at least one of the following: joint position, joint speed, joint angle, and ground coupling pressure information of the two feet of the humanoid robot; a first generation unit, configured to generate simulated elevation terrain information for the humanoid robot; a first input unit, configured to input the current observation sample information into an action network included in a motor position control network to obtain a numerical set of action probability distribution; a second input unit, configured to input the current observation sample information into a motor position control network. The network includes a strategy network, which obtains an action evaluation value set corresponding to the above-mentioned action probability distribution value set; a second generation unit, which is configured to generate a target joint position set of the above-mentioned humanoid robot according to the above-mentioned action probability distribution value set and the above-mentioned action evaluation value set; a third input unit, which is configured to input the above-mentioned target joint position set into a joint response controller to generate a joint torque set for the above-mentioned humanoid robot; a control unit, which is configured to control the above-mentioned humanoid robot to perform foot lift-off movement training on the simulated elevation terrain corresponding to the above-mentioned simulated elevation terrain information according to the above-mentioned joint torque set; and an ending unit, which is configured to end the training of the above-mentioned motor position control network in response to determining that the above-mentioned action probability distribution value set and the above-mentioned action evaluation value set meet a preset loss condition.

[0009] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0010] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method described in any implementation manner in the first aspect is implemented.

[0011] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: the humanoid robot foot lifting movement training method of some embodiments of the present disclosure controls the humanoid robot to perform foot lifting training on rough ground through a motor position control network and a joint response controller, which can improve the movement stability of the humanoid robot, the performance and generalization ability of the motor position control network. Specifically, the reasons for the poor movement stability of the related controlled humanoid robot and the high damage rate of the humanoid robot are: the need to collect motion reference trajectories and gait libraries increases the waste of additional data collection costs and storage computing resources, and the strategy model using motion reference trajectories and gait libraries is to learn specific gaits for specific environments, which has poor environmental adaptability to humanoid robots, resulting in poor movement stability of the controlled humanoid robot and a high damage rate of the humanoid robot. Based on this, the humanoid robot foot lifting movement training method of some embodiments of the present disclosure can first obtain the current observation sample information of the humanoid robot, wherein the above-mentioned humanoid robot is a bipedal humanoid robot with a clog-type sole, and the above-mentioned current observation sample information includes at least one of the following: the joint position, joint speed, joint angle, and ground coupling pressure information of the humanoid robot. Here, the current observation sample information is used for the generation of subsequent model elevation terrain information and the training of the movement control of the humanoid robot's feet off the ground. Secondly, the simulated elevation terrain information for the above-mentioned humanoid robot is generated. Here, the simulated elevation terrain information can represent any environmental information used for the movement of the humanoid robot, and can improve the adaptability of the humanoid robot to various environments. Subsequently, the above-mentioned current observation sample information is input into the action network included in the motor position control network to obtain a numerical set of action probability distribution. Here, the action of the humanoid robot is strategically selected through the action network included in the motor position control network, which can remove unnecessary motion trajectory references and improve the decision-making ability of the humanoid robot's action selection. Then, the above-mentioned current observation sample information is input into the strategy network included in the motor position control network to obtain the action evaluation value set corresponding to the above-mentioned action probability distribution numerical set. Here, the strategy network included in the motor position control network evaluates the cumulative return of the action selected by the humanoid robot, which can improve the accuracy of the action selection of the humanoid robot, realize the alternating support of the two feet of the humanoid robot during walking and the movement training of the foot end off the ground, and improve the steady movement of the humanoid robot. Subsequently, the target joint position set of the humanoid robot is generated according to the action probability distribution value set and the action evaluation value set. Here, the action with the highest cumulative reward value is determined by the action probability distribution value set and the action evaluation value set, which can improve the accuracy of the motor position control network and the stability of the humanoid robot in executing the action. Then, the target joint position set is input into the joint response controller to generate a joint torque set for the humanoid robot.Here, through the joint response controller, the humanoid robot can be accurately tracked, high-precision positioned and real-time dynamic response can be achieved, thereby improving the control stability of the humanoid robot. Afterwards, according to the joint torque set, the humanoid robot is controlled to perform foot lift-off movement training on the simulated elevation terrain corresponding to the simulated elevation terrain information. Here, the humanoid robot can be prevented from performing movement training through sliding friction, so that the robot can produce a foot lift action. Finally, in response to determining that the action probability distribution numerical set and the action evaluation value set meet the preset loss condition, the training of the motor position control network is terminated. Here, the motor position control network is iteratively cycled trained through the action probability distribution numerical set and the action evaluation value set, which can improve the accuracy of the motor position control network in the foot lift-off training scenario for the humanoid robot. It can be obtained that the humanoid robot foot lift-off movement training method controls the humanoid robot to perform foot lift-off movement training on rough ground through the motor position control network and the joint response controller, which can improve the movement stability of the humanoid robot, the performance and generalization ability of the motor position control network. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0013] Figure 1 is a flow chart of some embodiments of the humanoid robot foot lifting and movement training method according to the present disclosure;

[0014] Figure 2 is a schematic diagram of a clog-type foot of a humanoid robot with 1 degree of freedom in a simulated environment according to some embodiments of the humanoid robot foot lifting and movement training device disclosed herein;

[0015] Figure 3 is a schematic diagram of a straight-plate-shaped sole of a humanoid robot with 1 degree of freedom in a real environment according to some embodiments of the humanoid robot foot-lifting movement training device disclosed herein;

[0016] Figure 4 is a schematic diagram of a clog-type foot of a humanoid robot with 2 degrees of freedom in a simulated environment according to some embodiments of the humanoid robot foot lifting and movement training device disclosed herein;

[0017] Figure 5 is a schematic diagram of a straight-plate-shaped foot of a humanoid robot with 2 degrees of freedom in a real environment according to some embodiments of the humanoid robot foot-lifting movement training device disclosed herein;

[0018] Figure 6 is a schematic diagram of a humanoid robot with one degree of freedom in a simulated environment, wherein a clog-type sole forms collision coupling with a rough ground surface simulating elevation terrain according to some embodiments of the humanoid robot foot-lifting movement training device disclosed herein;

[0019] Figure 7 is a schematic diagram of a humanoid robot with two degrees of freedom in a simulated environment, wherein a clog-type sole forms collision coupling with a rough ground surface simulating elevation terrain according to some embodiments of the humanoid robot foot-lifting movement training device disclosed herein;

[0020] Figure 8 is a schematic structural diagram of some embodiments of a humanoid robot foot-lifting movement training device according to the present disclosure;

[0021] Fig. 9 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0022] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0023] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0024] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0025] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0026] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0027] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0028] Figure 1 The process 100 of some embodiments of the humanoid robot foot lifting and movement training method according to the present disclosure is shown. The humanoid robot foot lifting and movement training method comprises the following steps:

[0029] Step 101, obtaining current observation sample information.

[0030] In some embodiments, the execution subject (e.g., electronic device) of the above-mentioned humanoid robot foot lifting and movement training method can obtain the current observation sample information through a wired connection method or a wireless connection method, wherein the above-mentioned humanoid robot is a bipedal humanoid robot with a clog-shaped sole, and the above-mentioned current observation sample information includes at least one of the following: the joint position, joint speed, joint angle, and ground coupling pressure information of the bipedal robot. Among them, the above-mentioned humanoid robot can be a humanoid bipedal robot model constructed in a physical simulation environment including a thigh, a calf, and a clog-shaped foot. The above-mentioned humanoid robot can be used as an intelligent body and can perform corresponding actions. For example, the above-mentioned corresponding action can be but is not limited to at least one of the following: walking, bending, running, and wrestling. The above-mentioned current observation sample information can be obtained through the body sensor sample of the humanoid bipedal robot sample, and the body sensor sample can be obtained through simulation, and can also include a joint encoder and an inertial measurement unit (InertialMeasurement Unit, IMU). The above-mentioned joint speed can be the angular velocity and linear velocity of each joint obtained after applying torque to the motor on each joint of the humanoid robot. The angular velocity of each of the above-mentioned joints may include: knee joint angular velocity, ankle joint angular velocity. The linear velocity may be the linear velocity of each coordinate axis in the body coordinate system of the humanoid robot. The angular velocity may be the turning speed of the humanoid robot relative to each coordinate axis in the body coordinate system. The above-mentioned ground coupling pressure information on the two feet may be information on the pressure of the terrain coupled with the terrain obtained by the sensor of the foot of the humanoid robot. The above-mentioned ground coupling pressure information on the two feet may include but is not limited to at least one of the following: the magnitude of the ground reaction force and the height of the terrain.

[0031] The clog-shaped sole of the humanoid robot with one degree of freedom is as follows: Figure 2 As shown in FIG. 1 , two capsule-shaped protrusions are added to the bottom of the humanoid robot's sole. The sole structure of a real robot's sole is not necessarily manufactured in the clog-like shape. The sole structure of a real robot's sole can be a straight sole structure, such as Figure 3 The clog-shaped sole of the humanoid robot with two degrees of freedom is shown in FIG. Figure 4 As shown in FIG. 4 , four circular collision shapes are added to the bottom of the humanoid robot's foot. The foot structure of a real robot is not necessarily manufactured according to the clog type. The foot structure of a real robot can be a straight plate foot structure, such as Figure 5 shown.

[0032] In some optional implementations of some embodiments, the humanoid robot comprises: a thigh, a calf and a foot, wherein:

[0033] The thigh and the calf are connected by a locking structure of the knee joint. The knee joint is composed of a ratchet pawl assembly and a four-bar mechanism assembly. The locking structure of the knee joint is used to control the humanoid robot to perform knee extension and bending postures. The ratchet pawl assembly includes: a pawl, a first spring, a ratchet, a reduction motor, a first bevel gear and a second bevel gear. The four-bar mechanism assembly includes: a drive motor, a fixed joint, a long connecting rod, a short connecting rod, a block, a synchronous pulley, a harmonic reducer, a thigh and a calf. Among them, the locking structure of the knee joint can limit the state of knee extension when the joint is extended and flexed, so as to smoothly complete the structure of the leg support and swing process. One end of the short connecting rod in the four-bar mechanism assembly extends a cantilever, and a block is arranged on the cantilever. When the rotation of the calf reaches the required maximum rotation angle, the calf and the thigh are in a colinear state. Driven by the calf, the long connecting rod drives the short connecting rod to swing, and the short connecting rod finally reaches a vertical state. The block on the cantilever of the short connecting rod just hits the thigh surface, forming a limit of the straight state of the knee joint. When the thigh and the calf are in a colinear state, the short connecting rod that has reached a vertical state is also in a straight line. The three are in a straight line, reaching a state close to a singular point. Only by applying a greater force or torque can they make a transition and continue to rotate, forming a knee extension limit. The reduction motor in the ratchet pawl assembly drives the first bevel gear to rotate with the torque after deceleration, and the second bevel gear drives the shaft to rotate, and the shaft is equipped with a ratchet. The rotation of the shaft drives the rotation of the ratchet. The pawl is installed at the top of the cantilever. After the ratchet and pawl are matched, the ratchet will hinder the movement of the pawl, forming a bending limit of the knee joint. The knee extension limit composed of the four-bar mechanism and the knee joint bending limit composed of the ratchet pawl part are used together to form a locking mechanism of the knee joint.

[0034] The foot and the calf are connected through an ankle joint, and the foot is a foot with a target number of circular collision shapes added to the bottom of the sole of the foot, and the circular collision shape and the simulated elevation map corresponding to the simulated elevation terrain information generate a collision shape of coupled collision. The target number can be a preset value. For example, the target number can be 4. The coupled collision can be controlled by the pressure on the foot of the humanoid robot and the reaction force of the elevation map corresponding to the simulated elevation terrain information on the foot, so as to control the humanoid robot to lift the foot a certain distance before moving.

[0035] Step 102: Generate simulated elevation terrain information for the humanoid robot.

[0036] In some embodiments, the execution entity may generate simulated elevation terrain information for the humanoid robot. The simulated elevation terrain information may be information related to the continuous rough ground of the simulated environment in which the humanoid robot is located. The simulated elevation terrain information may include but is not limited to at least one of the following: the height of the ground, the roughness of the ground. In practice, the execution entity may generate simulated elevation terrain information for the humanoid robot through a digital elevation model.

[0037] In some optional implementations of some embodiments, the generating of simulated elevation terrain information for the humanoid robot may include the following steps:

[0038] The first step is to obtain a terrain plane grid of a preset size, wherein each grid vertex in the above-mentioned terrain plane grid includes a pseudo-random gradient vector. The above-mentioned terrain plane grid of a preset size can be a two-dimensional network with a preset terrain size. The above-mentioned preset terrain size can be specifically set according to actual conditions and is not limited here. The above-mentioned pseudo-random gradient vector can characterize the pseudo-random positive or negative influence of any vertex in the terrain plane grid relative to any point in the unit grid. Pseudo-randomness can represent that for any identical input value, the output result is the same. The above-mentioned positive influence can be that the direction indicated by the gradient vector is the direction of the force exerted on any point. The above-mentioned reverse influence can be that the direction indicated by the gradient vector is opposite to the direction of the force exerted on any point.

[0039] The second step is to determine the adjacent mesh vertex group of each input random coordinate point in the above terrain plane network, and obtain the adjacent mesh vertex group set, wherein the input random coordinate point is an arbitrary two-dimensional coordinate point randomly generated and located in the above terrain plane network. Among them, the adjacent mesh vertex group can be four mesh vertices located near the input random coordinate point. For example, if the input random coordinate point is a coordinate point within a unit grid, the adjacent mesh vertex group is the four vertices within the unit grid. If the input random coordinate point is located at a grid vertex, the adjacent mesh order group is four grid vertices with a distance of the unit cell distance.

[0040] In the third step, for each adjacent mesh vertex group in the adjacent mesh vertex group set, perform the following linear interpolation steps:

[0041] Sub-step 1, determining the coordinate distance vector between each adjacent network vertex in the adjacent network vertex group and the corresponding input random coordinate point, and obtaining a coordinate distance vector group. The coordinate distance vector may be a coordinate vector difference from the adjacent mesh vertex to the input random coordinate point.

[0042] Sub-step 2: performing vector dot multiplication on each coordinate distance vector in the above coordinate distance vector group and the corresponding pseudo-random gradient vector to obtain a coordinate dot product vector group.

[0043] Sub-step 3, performing nonlinear interpolation on the above-mentioned coordinate point multiplication vector group to obtain a random noise value group for the input random coordinate point. The random noise value in the above-mentioned random noise value group may be the coordinate data newly inserted into the plane grid.

[0044] The third step is to generate initial noise fractal terrain information according to the preset relaxation curve and the obtained random noise value groups. The preset relaxation curve can be a curve function of a coordinate data set in a preset smooth plane grid. The initial noise fractal terrain information can be the ground information of the environment where the humanoid robot is located.

[0045] The fourth step is to perform terrain rendering on the above initial noise fractal terrain information to obtain the rendered initial noise fractal terrain information as simulated elevation terrain information.

[0046] In the process of adopting technical solutions to solve the above-mentioned technical problem one, the following technical problem two is often accompanied: how to generate simulated elevation terrain information that is close to the real terrain and accurate, so as to restore the foot-lifting movement training of the humanoid machine on different terrains as much as possible. For the above-mentioned technical problem two, the conventional solution is generally: generate simulated elevation terrain information through a two-dimensional Perlin noise algorithm. However, the above-mentioned conventional solution still has the following problems: due to the lack of terrain details generated by the two-dimensional Perlin noise algorithm and the problem of terrain distortion, it is impossible to truly simulate the real terrain information, resulting in a large difference between the humanoid robot's foot lift-off movement training and the training in the real environment, and it is impossible to accurately grasp the training problems that may arise in the humanoid robot in the real environment, increase the damage rate of the humanoid robot, and reduce the safety of the humanoid robot. The inventor, taking into account the shortcomings of the conventional solution and combining the technical status of the mobile training and deep learning technology of the humanoid robot owned by the inventor's scientific research institute, can decide to adopt the following solution.

[0047] In some optional implementations of some embodiments, the generating of simulated elevation terrain information for the humanoid robot may include the following steps:

[0048] The first step is to obtain a set of terrain style images, wherein the terrain style images in the set of terrain style images may be images of a terrain situation collected from different shooting angles.

[0049] The second step is to determine the terrain position information of each terrain style image in the terrain style image set to obtain a style two-dimensional terrain information set, wherein the style two-dimensional terrain information in the style two-dimensional terrain information set corresponds to each two-dimensional position coordinate information in the terrain style image. The style two-dimensional terrain information in the style two-dimensional terrain information set may be two-dimensional information of each point of the terrain corresponding to the terrain style image set.

[0050] The third step is to determine the average value of the above style two-dimensional terrain information set to obtain the style two-dimensional average terrain information.

[0051] The fourth step is to construct a noise fractal map for the above-mentioned two-dimensional average terrain information to generate initial noise fractal terrain information. The above-mentioned initial fractal terrain information can be terrain information with randomness.

[0052] In the fifth step, each terrain style image in the terrain style image set is partitioned to generate a terrain style image block group, thereby obtaining a terrain style image block group set. The partitioning process may be performed based on different topography of the terrain.

[0053] In the sixth step, the terrain style image block set is input into the terrain style transfer model to obtain terrain style information. The terrain style information can represent the terrain detail information of the terrain corresponding to the terrain style image set. The terrain style transfer model can be a model including a transformation network and a loss network. The transformation network can be a deep residual network including 5 residual blocks. The loss network can be a loss network including terrain content loss and style loss. The terrain content loss can represent the topographic information of the terrain. The style loss can represent the degree of similarity between the processed terrain and the terrain style image.

[0054] In the seventh step, based on the terrain style information, terrain style conversion is performed on the initial noise fractal terrain information to obtain style fractal terrain information.

[0055] In the eighth step, the style fractal terrain information is partitioned and smoothed to obtain smoothed style fractal terrain information as simulated elevation terrain information.

[0056] The above technical solution and its related contents, combined with step "step 107", as an inventive point of the embodiment of the present disclosure, solves the second technical problem mentioned in the background technology: "Due to the fact that the terrain details generated by the two-dimensional Perlin noise algorithm are less and the terrain is distorted, it is impossible to truly simulate the real terrain information, resulting in a large difference between the training of the humanoid robot in the foot lift-off movement training and the training in the real environment, and it is impossible to accurately grasp the training problems that may arise in the humanoid robot in the real environment, increase the damage rate of the humanoid robot, and reduce the safety of the humanoid robot." The factors that cause the humanoid robot to have a large difference between the training of the foot lift-off movement training and the real environment, and it is impossible to accurately grasp the training problems that may arise in the humanoid robot in the real environment, increase the damage rate of the humanoid robot, and reduce the safety of the humanoid robot are often as follows: Due to the fact that the terrain details generated by the two-dimensional Perlin noise algorithm are less and the terrain is distorted, it is impossible to truly simulate the real terrain information. If the above factors are solved, the effect of reducing the training difference between the humanoid robot in the foot lift-off movement training and the real environment, accurately grasping the training problems that may arise in the humanoid robot in the real environment, reducing the damage rate of the humanoid robot, and increasing the safety of the humanoid robot can be achieved. In order to achieve this effect, the present invention adopts a generation method that combines fast style transfer with a two-dimensional noise terrain generation method to perform fast style transfer on the basic terrain partitions generated by noise, and then ensures the smooth transition of the blocks through post-processing, and adds post-effects such as denoising. It can generate terrain with better details and specific style. While enriching the terrain details and improving the authenticity of the terrain, it ensures that the generation is fast and efficient, and has good use value.

[0057] Step 103: input the current observation sample information into the action network included in the motor position control network to obtain an action probability distribution value set.

[0058] In some embodiments, the execution subject may input the current observation sample information into the action network included in the motor position control network to obtain an action probability distribution value set. The motor position control network may be a deep reinforcement learning model for training the input current observation sample information to lift the foot and move. The motor position control network may be a model using an actor-critic neural network structure. The excitation function of the motor position control network may be a reward function of a target instruction for movement and a penalty item related to power and safety. The target instruction may be the speed of target movement. The difference between the actual movement speed and the target movement speed is used as a reward item. Power is the output product of the joint angular velocity and the motor torque. The penalty item related to safety is the value exceeding a certain torque limit. The action probability distribution value in the action probability distribution value set may be the probability value of each action selected by the humanoid robot at the next moment under the influence of the environmental information of the simulated elevation terrain information, which conforms to the normal distribution. The action network is obtained by training in the following steps: the robot action posture set in the experience pool of the motor position control network is input into the action network under the new and old strategies, and the first normal distribution and the second normal distribution of the action probability of the humanoid robot under different strategies are obtained. Secondly, the stored robot motion posture set is input into the first normal distribution and the second normal distribution to obtain the first motion probability value and the second motion probability value corresponding to each robot motion posture. Then, the value of the first motion probability value divided by the second motion probability value is determined as the importance weight. Finally, the difference between the two motion distributions under the new and old strategies is corrected by importance sampling to obtain the action loss function value. In response to determining that the action loss function value is less than or equal to the preset action loss threshold, the training is completed. In response to determining that the action loss function value is greater than the preset action loss threshold, the action network is retrained. Among them, the above-mentioned action loss function can be a minimized advantage function. The advantage function can be the deviation between the predicted value function and the expected true cumulative return. The above-mentioned preset action loss threshold can be a pre-set loss threshold.

[0059] The multi-round random sampling actions output by the above action network are regarded as a motion trajectory, starting from the initial action posture of the robot standing upright and interacting with the simulated elevation terrain to the terminal action posture of standing upright again after alternating bipedal walking. In an action trajectory, the robot determines the current action state, takes an action posture according to the strategy, obtains a reward and obtains the next state. The goal of strategy optimization is to end the round when the expected cumulative reward value is maximized after taking an action posture in the current action state.

[0060] The above-mentioned motion trajectory may include: 2 double support (DS) stages and 2 single support (SS) stages. The movement cycle may include the left foot support stage (i.e., DS1 stage), the first double support stage (i.e., DD1 stage), the right foot support stage (i.e., DS2 stage), and the second double support stage (i.e., DD2 stage) in chronological order. The DS1 stage and the DD1 stage may constitute the left foot landing stage, the DS2 stage and the DD2 stage may constitute the right foot landing stage, and the left foot landing stage and the right foot landing stage both account for 50% of the movement cycle T. The two double support stages account for 20% of the movement cycle, and the two single support stages account for 80% of the movement cycle. For a single leg, there is only one swing cycle in the entire movement cycle, which accounts for 40% of the movement cycle. The periodic movement reward can penalize the speed of both feet and encourage the ground reaction force of both feet in the double-foot support stage, encourage the ground reaction force of the right foot and the speed of the left foot and penalize the ground reaction force of the left foot and the speed of the right foot in the right-foot support stage, and encourage the ground reaction force of the left foot and the speed of the right foot in the left-foot support stage, and penalize the ground reaction force of the right foot and the speed of the left foot, so as to help the motor position control network learn the reciprocating center of gravity and alternating support movements of both feet during walking of humanoid bipedal robot samples.

[0061] In some optional implementations of some embodiments, the action network includes: an action input layer, a first action hidden layer, a second action hidden layer, a third action hidden layer, an attention mechanism layer, and an action output layer. The first action hidden layer, the second action hidden layer, and the third action hidden layer may be fully connected layers whose activation functions may all be exponential linear activation functions (Rectified Linear Unit, ReLU). The action input layer may be a network layer that receives the current observation sample information and transmits it to the first action hidden layer. The action output layer may be a network layer that outputs a numerical set of action probability distributions.

[0062] Optionally, the step of inputting the current observation sample information into the action network included in the motor position control network to obtain the action probability distribution value set may include the following steps:

[0063] The first step is to input the current observation sample information into the action input layer to obtain a first humanoid robot action feature vector, wherein the first humanoid robot action feature vector can represent the walking posture information of the humanoid robot on the simulated elevation terrain corresponding to the simulated elevation terrain information.

[0064] The second step is to input the first humanoid robot action feature vector into the first action hidden layer to obtain a second humanoid robot action feature vector. The second humanoid robot action feature vector may be a feature vector obtained by linearly processing the current observation sample weight feature vector using an exponential linear activation function.

[0065] The third step is to input the second humanoid robot action feature vector into the second action hidden layer to obtain a third humanoid robot action feature vector.

[0066] The fourth step is to input the third humanoid robot action feature vector into the third action hidden layer to obtain a fourth humanoid robot action feature vector.

[0067] The fifth step is to input the fourth humanoid robot action feature vector into the attention mechanism layer to obtain a humanoid robot action weight feature vector. The humanoid robot action weight feature vector can represent feature vectors of different weights of different dimensions of the fourth humanoid robot action feature vector.

[0068] In the sixth step, the weighted feature vector of the humanoid robot action is input into the action output layer to obtain the action mean and action variance. The action mean can represent the average expected value of the action selected by the humanoid robot in the current state under the simulated elevation terrain corresponding to the simulated elevation terrain information. The action variance can represent the discrete degree and uncertainty of different actions taken by the humanoid robot under the simulated elevation terrain.

[0069] The seventh step is to construct a numerical set of action probability distribution for the above action mean and the above action variance. In practice, the above execution subject can generate a normal probability distribution of action through the above action mean and action variance. Then, the above action normal probability distribution is randomly sampled to obtain a numerical set of action probability distribution. It should be noted that the action network can accelerate the learning of stable action strategies by humanoid robots.

[0070] Step 104 , input the current observation sample information into the strategy network included in the motor position control network to obtain an action evaluation value set corresponding to the action probability distribution value set.

[0071] In some embodiments, the above-mentioned execution subject can input the above-mentioned current observation sample information into the strategy network included in the motor position control network to obtain the action evaluation value set corresponding to the above-mentioned action probability distribution value set. Among them, the above-mentioned strategy network can be a network that evaluates the value of the action corresponding to the action probability distribution value set of the input current observation sample information under the above-mentioned simulated elevation terrain information. The action evaluation value in the above-mentioned action evaluation value set can be the reward value obtained after the humanoid robot takes any action posture. The above-mentioned strategy network can be trained by the following steps: First, the action trajectory is input into the initial strategy network to obtain the state value corresponding to all states of the humanoid robot in an action trajectory. Among them, the above-mentioned state value can be expressed as:

[0072]

[0073] Among them, V(s t ) indicates state s t The state value under a. a represents an action trajectory. a t Represents the action performed by the humanoid robot at time t. t Indicates the state of the humanoid robot at time t. t |s t ) represents the policy function corresponding to the policy network. t+1 Represents the state-action transition probability function at time t+. t It represents the reward function after the humanoid robot takes the next action at the time t. t+1 represents the state of the humanoid robot at time t+1. γ represents the discount factor. V(s t+1 ) means in state s t+1 The state value below.

[0074] Secondly, the expected cumulative reward value is regarded as the average value of the expected cumulative rewards obtained by taking different action postures after the humanoid robot performs an action and reaches a state, and the expected cumulative reward value mean is obtained. And according to the expected cumulative reward value mean, the advantage function is generated. Among them, the above expected cumulative reward value mean can be expressed as:

[0075] G t =r t +γV(s t+1 ).

[0076] Among them, G t It represents the expected cumulative return value of the humanoid robot at time t.

[0077] The above advantage function can be expressed as:

[0078] Aπ (a t ,s t )=G t -V(s t ).

[0079] Among them, A π (a t ,s t ) represents the advantage function.

[0080] Then, the advantage function is estimated by first-order time difference to obtain the differenced advantage function.

[0081] Among them, the difference advantage function can be expressed as:

[0082]

[0083] in, Denotes the advantage function after difference processing. δ t represents the time difference deviation, δ t =r t +γV(s t+1 )-V(s t )"λ represents the factor that speeds up the learning of the policy network. L represents the action value loss function. T represents the number of steps.

[0084] Finally, based on the differential advantage function, the action value loss function of the policy network is generated.

[0085] Among them, the action value loss function can be expressed as:

[0086]

[0087] Wherein, N represents the number of samples included in the above-mentioned currently observed sample information.

[0088] In some optional implementations of some embodiments, the above-mentioned strategy network includes: a value input layer, a first value hidden layer, a second value hidden layer, a third value hidden layer, a long short-term memory neural network and a value output layer. Among them, the above-mentioned first value hidden layer, the above-mentioned second value hidden layer and the above-mentioned third value hidden layer can be fully connected layers whose activation function is an exponential linear function (ELU). The above-mentioned value input layer can be a network layer that receives the current observation sample information and transmits it to the first value hidden layer. The above-mentioned value output layer can be a network layer that outputs the action evaluation value set.

[0089] Optionally, the step of inputting the current observation sample information into the strategy network included in the motor position control network to obtain the action evaluation value set corresponding to the action probability distribution value set may include the following steps:

[0090] In the first step, the current observation sample information is input into the value input layer to obtain the first humanoid robot action value feature vector. The first humanoid robot action value feature vector can represent the reward value obtained after the humanoid robot selects any action.

[0091] The second step is to input the first humanoid robot action value feature vector into the first value hidden layer to obtain the second humanoid robot action value feature vector.

[0092] The third step is to input the second humanoid robot action value feature vector into the second value hidden layer to obtain the third humanoid robot action value feature vector.

[0093] The fourth step is to input the third humanoid robot action value feature vector into the third value hidden layer to obtain the fourth humanoid robot action value feature vector.

[0094] The fifth step is to input the action value feature vector of the fourth humanoid robot into the long short-term memory neural network to obtain a walking posture temporal feature vector set. The posture temporal feature vector in the posture temporal feature vector set can represent the temporal information of the action trajectory.

[0095] The sixth step is to input the walking posture temporal feature vector set into the value output layer to obtain the action evaluation value set corresponding to the action probability distribution.

[0096] Step 105, generating a target joint position set of the humanoid robot according to the action probability distribution value set and the action evaluation value set.

[0097] In some embodiments, the execution subject may generate a target joint position set for the humanoid robot based on the action probability distribution value set and the action evaluation value set. The target joint positions in the target joint position set may be the positions of the joints corresponding to the desired actions of the humanoid robot. A humanoid robot with one degree of freedom performs foot-lifting movement training on a simulated elevation terrain corresponding to simulated elevation terrain information to produce a certain coupling with the rough ground of the simulated elevation terrain, thereby preventing the humanoid robot from moving horizontally through friction with the ground. Figure 6 A humanoid robot with two degrees of freedom performs foot-lifting movement training on a simulated elevation terrain corresponding to simulated elevation terrain information, so as to produce a certain coupling with the rough ground of the simulated elevation terrain, thereby preventing the humanoid robot from moving horizontally through friction with the ground. Figure 7 shown.

[0098] As an example, the execution subject may select the action evaluation value with the largest value from the action evaluation value set to obtain the target action evaluation value. Then, the action probability distribution value corresponding to the target action evaluation value is selected from the action probability distribution value set to obtain the target action probability distribution value. Finally, the position set of each joint of the humanoid robot corresponding to the target action probability distribution value is determined as the target joint position set.

[0099] Step 106 : input the target joint position set to the joint response controller to generate a joint torque set for the humanoid robot.

[0100] In some embodiments, the execution subject may input the target joint position set into a joint response controller to generate a joint torque set for the humanoid robot. The joint torque may be a torque that notifies the motor to generate a change in the motion posture of the humanoid robot. The joint response controller may be a controller that integrates a PD controller and a PID controller.

[0101] In the process of adopting technical solutions to solve the above-mentioned technical problem one, the following technical problem three is often accompanied: the joint torque output by the joint response controller is not accurate and flexible enough, resulting in the humanoid robot being unable to respond and control to the expected joint position in time. For the above-mentioned technical problem three, the conventional solution is generally: using the traditional parameter setting algorithm to dynamically improve the joint response controller to output accurate joint torque. However, the above-mentioned conventional solution still has the following problems: because the traditional parameter setting algorithm relies on certain expert experience and simple calculation steps, the accuracy and limitations of the adjustment of the joint response controller in complex scenes are low, and there is a certain subjectivity, resulting in the low accuracy of the output torque of the joint response controller, which cannot well control the humanoid robot's foot lifting and moving training, and increases the damage rate and instability of the humanoid robot. The inventor, taking into account the shortcomings of the conventional solution, and combining the technical status of the mobile training and deep learning technology of the humanoid robot owned by the inventor's scientific research institute, can decide to adopt the following solution.

[0102] In some optional implementations of some embodiments, inputting the target joint position set into a joint response controller to generate a joint torque set for the humanoid robot may include the following steps:

[0103] The first step is to generate a joint control fitness function for a joint response controller, wherein the joint response controller is a function including a joint control proportional parameter, a joint control differential parameter and a joint control differential order parameter. The joint control proportional parameter may be a parameter for rapid response to the error between the desired joint state and the actual joint state. The joint control proportional parameter generates a control output in proportion to the error, which may characterize the sensitivity of the joint response controller to the error. The joint control differential parameter may characterize the adjustment of the control output based on the error change rate. The joint control differential parameter may improve the dynamic performance of the joint response controller. The joint control differential order parameter may characterize the adjustment of the differential term. The higher the order of the joint control differential order parameter, the better the effect of the control output. The joint control fitness function may be the weighted sum of the angle error, velocity error, control input and overshoot of the step response.

[0104] In the second step, a chaos algorithm is used to generate an initial population for the target joint position set and the joint torque set according to the target joint position set, wherein each initial individual included in the initial population is a parameter determination scheme for determining the parameter values ​​corresponding to the joint control proportional parameters, joint control differential parameters and joint control differential order parameters.

[0105] The third step is to use the reverse learning algorithm to determine the initial reverse population of the initial population. The initial reverse individuals in the initial direction population can be the sum of the upper and lower bound values ​​of the initial individuals and the difference between the values ​​corresponding to the initial individuals.

[0106] The fourth step is to input the above initial population and the above initial reverse population into the above joint control fitness function to obtain an initial fitness value set and an initial reverse fitness set.

[0107] The fifth step is to screen the above initial fitness value set and the above initial reverse fitness value set to obtain a screened fitness value set, wherein the screened fitness value in the above screened fitness value set can be the largest value among the corresponding initial fitness value and the initial reverse fitness value.

[0108] Step 6: Select individuals corresponding to the fitness value set after the selection from the initial population and the initial reverse population as the target initial population. The number of the target initial population, the initial population and the initial reverse population is the same.

[0109] Step 7: Based on the target initial population, perform the following input steps:

[0110] Sub-step 1, performing a cosine position update process on each target initial individual included in the target initial population to generate an updated target initial individual and obtain an updated target population. The updated target individual in the updated target population can be updated by an update function to obtain an updated target initial individual. The updated target initial individual is obtained by the following steps: the first step, in response to determining that the target tracked initial individual is found to be tracked by the target initial individual, first, determining whether the random value is greater than a preset cosine-sine conversion threshold. The preset cosine-sine conversion threshold can be a preset critical value for the conversion of the cosine function and the sine function. For example, the preset cosine-sine conversion threshold can be 0.5. The target tracked initial individual can be any target initial individual excluding the target initial individual. Secondly, determining the position difference between the target tracked individual and the target initial individual as the individual position difference. Thirdly, in response to determining that the random value is less than the preset cosine-sine conversion threshold, determining the product of the search range conversion parameter, the cosine function, the flight step length and the individual position difference as the target individual position. Among them, the above-mentioned search range conversion parameters can characterize the conversion between global search and local search. Then, in response to determining that the random value is greater than or equal to the preset cosine-sine conversion threshold, determine the product of the search range conversion parameters, the sine function, the flight step value and the individual position difference as the target individual position. Finally, the sum of the above-mentioned target individual position and the above-mentioned target initial individual is determined as the updated individual. Among them, the above-mentioned search range conversion parameters can characterize the conversion parameters of global search and local search. In the second step, in response to determining that the target tracked initial individual is not found to be tracked by the target initial individual, randomly determine the target initial individual as the updated target individual.

[0111] Sub-step 2: performing mutation processing on the updated target population to obtain a mutated target population. The mutated target individuals in the mutated target population may be individuals obtained by multiplying the updated target individuals by a mutation probability value.

[0112] Sub-step 3, crossover processing is performed on the above-mentioned mutated target population to obtain a crossover target population. The crossover individuals in the above-mentioned crossover target population can be individuals obtained by crossovering the preset number of dimension values ​​of the above-mentioned mutated target individual and the corresponding updated target individual in response to determining that the crossover probability value corresponding to the mutated target individual is greater than or equal to the preset crossover probability threshold. The above-mentioned crossover probability value can be a random value. The above-mentioned preset crossover probability threshold can be the maximum value of the preset crossover probability value. The above-mentioned preset number of dimensions can be 3.

[0113] Sub-step 4, determining the number of times the above input step has been executed.

[0114] Sub-step 5, in response to determining that the number of executions is greater than or equal to a preset execution threshold, inputting the updated target population into the joint control fitness function to obtain an updated fitness value set. The preset execution threshold may be a preset maximum number of executions.

[0115] In step 8, in response to determining that the number of executions is less than the preset execution threshold, the target population after the crossover is determined as the target initial population, and the sum of the number of executions and the preset value is determined as the number of executions, so as to perform the input step again. The preset value may be a pre-set value. For example, the preset value may be 1.

[0116] In the ninth step, the updated fitness value with the largest value is selected from the above updated fitness value set as the target updated fitness value.

[0117] In the tenth step, the parameter value set of the crossover target individual corresponding to the above target updated fitness value is determined as the target joint control ratio parameter value, the target joint control differential parameter value and the target joint control differential order parameter value.

[0118] In the eleventh step, the above target joint control proportional parameter value, target joint control differential parameter value and target joint control differential order parameter value are substituted into the joint response controller to obtain the target joint response controller.

[0119] In the twelfth step, the target joint position set is input into the target joint response controller to obtain the joint torque set of the humanoid robot.

[0120] The above technical scheme and its related contents, combined with step "step 107" as an inventive point of the embodiment of the present disclosure, solve the technical problem three mentioned in the background technology "Because the traditional parameter setting algorithm relies on certain expert experience and the calculation steps are simple, the adjustment accuracy of the joint response controller in complex scenes is low and limited, and there is a certain subjectivity, resulting in low accuracy of the output torque of the joint response controller, which cannot well control the foot lifting and movement training of the humanoid robot, and increase the damage rate and instability of the humanoid robot". The factors that lead to the low accuracy of the output torque of the joint response controller, the inability to well control the foot lifting and movement training of the humanoid robot, and the increase of the damage rate and instability of the humanoid robot are often as follows: Because the traditional parameter setting algorithm relies on certain expert experience and the calculation steps are simple, the adjustment accuracy of the joint response controller in complex scenes is low and limited, and there is a certain subjectivity. If the above factors are solved, the accuracy of the output torque of the joint response controller can be improved, the foot lifting and movement training of the humanoid robot can be accurately controlled, the damage rate of the humanoid robot can be reduced, and the stability can be improved. In order to achieve this effect, the present invention first generates a joint control fitness function so that the optimal parameter set can be obtained under the constraint condition of satisfying the joint control fitness function. Secondly, the initial population is adjusted by the chaos algorithm and the reverse learning algorithm, which can improve the uniform distribution of the initial population in the search space and the diversity of the initial population, and generate a feasible solution space with good distribution. Then, the initial population is iteratively updated by the cosine function and the sine function, which can increase the search range in the early global search process and fully search the neighborhood of the current solution in the later local search process. After that, the updated target population is subjected to iterative cycle mutation processing and cross processing, which can increase the diversity of the updated target population, enhance the search ability of the parameter set, reduce the probability of falling into the local optimal solution, and accelerate the convergence speed. Finally, a target joint response controller is generated, and the target joint position set is input into the target joint response controller to obtain the joint torque, and the humanoid robot is controlled to perform foot lift-off movement training, which can improve the accuracy of the joint torque, accurately control the humanoid robot to reach the expected joint position, and improve the stability of the humanoid robot.

[0121] Step 107 , according to the joint torque set, controlling the humanoid robot to perform foot lift-off movement training on the simulated elevation terrain corresponding to the simulated elevation terrain information.

[0122] In some embodiments, the execution subject can control the humanoid robot to perform foot lift-off movement training on the simulated elevation terrain corresponding to the simulated elevation terrain information according to the joint torque set. The foot lift-off ground control training can prevent the sole of the humanoid robot from moving by friction with the ground.

[0123] Step 108 , in response to determining that the action probability distribution value set and the action evaluation value set satisfy a preset loss condition, the training of the motor position control network is terminated.

[0124] In some embodiments, the execution subject may terminate the training of the motor position control network in response to determining that the action probability distribution value set and the action evaluation value set satisfy a preset loss condition. The preset loss condition may be that the sum of the action loss value and the action value loss function value is less than or equal to a preset loss threshold. The preset loss threshold may be a pre-set minimum loss value for training the motor position control network. The preset loss threshold may be determined based on the specific circumstances and is not limited here.

[0125] As an example, the execution subject may first determine the difference between the action probability distribution value set and the sample action probability distribution value set as the action loss value. The sample action probability distribution value set may be the action probability value set corresponding to the expected action selected by the actual expected humanoid robot. Then, determine the difference between the action evaluation value set and the sample action evaluation value set as the action value loss function value. The sample action evaluation value set may be the real reward value set corresponding to the action selected by the real humanoid robot. Finally, in response to determining that the sum of the action loss value and the action value loss function value is less than or equal to the preset loss threshold, the training of the motor position control network is terminated.

[0126] In some optional implementations of some embodiments, in response to determining that the action loss function value and the action reward value loss function value do not satisfy the preset loss condition, the observation sample information of the humanoid robot is reacquired as the current observation sample information, the relevant parameters of the motor position control network are readjusted to obtain the adjusted motor position control network, and the adjusted motor position control network is determined as the motor position control network to perform network training on the motor position control network again.

[0127] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: the humanoid robot foot lifting movement training method of some embodiments of the present disclosure controls the humanoid robot to perform foot lifting training on rough ground through a motor position control network and a joint response controller, which can improve the movement stability of the humanoid robot and the performance and generalization of the motor position control network. Specifically, the reasons for the poor movement stability of the related controlled humanoid robot and the high damage rate of the humanoid robot are: the need to collect motion reference trajectories and gait libraries increases the waste of additional data collection costs and storage computing resources, and the strategy model using motion reference trajectories and gait libraries is to learn specific gaits for specific environments, which has poor environmental adaptability to humanoid robots, resulting in poor movement stability of the controlled humanoid robot and a high damage rate of the humanoid robot. Based on this, the humanoid robot foot lifting movement training method of some embodiments of the present disclosure can first obtain the current observation sample information of the humanoid robot, wherein the above-mentioned humanoid robot is a bipedal humanoid robot with a clog-type sole, and the above-mentioned current observation sample information includes at least one of the following: the joint position, joint speed, joint angle, and ground coupling pressure information of the humanoid robot. Here, the current observation sample information is used for the generation of subsequent model elevation terrain information and the training of the movement control of the humanoid robot's feet off the ground. Secondly, the simulated elevation terrain information for the above-mentioned humanoid robot is generated. Here, the simulated elevation terrain information can represent any environmental information used for the movement of the humanoid robot, and can improve the adaptability of the humanoid robot to various environments. Subsequently, the above-mentioned current observation sample information is input into the action network included in the motor position control network to obtain a numerical set of action probability distribution. Here, the action of the humanoid robot is strategically selected through the action network included in the motor position control network, which can remove unnecessary motion trajectory references and improve the decision-making ability of the humanoid robot's action selection. Then, the above-mentioned current observation sample information is input into the strategy network included in the motor position control network to obtain the action evaluation value set corresponding to the above-mentioned action probability distribution numerical set. Here, the strategy network included in the motor position control network evaluates the cumulative return of the action selected by the humanoid robot, which can improve the accuracy of the action selection of the humanoid robot, realize the alternating support of the two feet of the humanoid robot during walking and the lifting movement control of the foot end off the ground, and improve the steady movement of the humanoid robot. Subsequently, the target joint position set of the humanoid robot is generated according to the action probability distribution value set and the action evaluation value set. Here, the action with the highest cumulative reward value is determined by the action probability distribution value set and the action evaluation value set, which can improve the accuracy of the motor position control network and the stability of the humanoid robot in executing the action. Then, the target joint position set is input into the joint response controller to generate a joint torque set for the humanoid robot.Here, through the joint response controller, the humanoid robot can be accurately tracked, high-precision positioned and real-time dynamic response can be achieved, thereby improving the control stability of the humanoid robot. Afterwards, according to the joint torque set, the humanoid robot is controlled to perform foot lift-off movement training on the simulated elevation terrain corresponding to the simulated elevation terrain information. Here, the humanoid robot can be prevented from performing movement training through sliding friction, so that the robot can produce a foot lift action. Finally, in response to determining that the action probability distribution numerical set and the action evaluation value set meet the preset loss condition, the training of the motor position control network is terminated. Here, the motor position control network is iteratively cycled trained through the action probability distribution numerical set and the action evaluation value set, which can improve the accuracy of the motor position control network in the foot lift-off training scenario for the humanoid robot. It can be obtained that the humanoid robot foot lift-off movement training method controls the humanoid robot to perform foot lift-off movement training on rough ground through the motor position control network and the joint response controller, which can improve the movement stability of the humanoid robot and the performance and generalization of the motor position control network.

[0128] Further references Figure 8 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a humanoid robot foot lifting and movement training device. These device embodiments are similar to Figure 1 Corresponding to the method embodiments shown, the humanoid robot foot-lifting and movement training device can be specifically applied to various electronic devices.

[0129] like Figure 8As shown, a humanoid robot foot lifting and moving training device 800 includes: an acquisition unit 801, a first generation unit 802, a first input unit 803, a second input unit 804, a second generation unit 805, a third input unit 806, a control unit 807 and an end unit 808. The acquisition unit 801 is configured to: acquire the current observation sample information of the humanoid robot, wherein the humanoid robot is a bipedal humanoid robot with a clog-type sole, and the current observation sample information includes at least one of the following: the joint position, joint speed, joint angle, and ground coupling pressure information of the humanoid robot. The first generation unit 802 is configured to: generate simulated elevation terrain information for the humanoid robot. The first input unit 803 is configured to: input the current observation sample information into the action network included in the motor position control network to obtain an action probability distribution value set. The second input unit 804 is configured to: input the current observation sample information into the strategy network included in the motor position control network to obtain an action evaluation value set corresponding to the action probability distribution value set. The second generation unit 805 is configured to: generate a target joint position set of the above-mentioned humanoid robot according to the above-mentioned action probability distribution numerical set and the above-mentioned action evaluation value set. The third input unit 806 is configured to: input the above-mentioned target joint position set into the joint response controller to generate a joint torque set for the above-mentioned humanoid robot. The control unit 807 is configured to: control the above-mentioned humanoid robot to perform foot lift-off movement training on the simulated elevation terrain corresponding to the above-mentioned simulated elevation terrain information according to the above-mentioned joint torque set. The ending unit 808 is configured to: in response to determining that the above-mentioned action probability distribution numerical set and the above-mentioned action evaluation value set meet the preset loss condition, end the training of the above-mentioned motor position control network.

[0130] It is understandable that the units described in the humanoid robot foot lifting and moving training device 800 are similar to those described in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the humanoid robot foot lifting and movement training device 800 and the units contained therein, and will not be described in detail here.

[0131] Reference below Fig. 9 , which shows a structural schematic diagram of an electronic device (eg, an electronic device) 900 suitable for implementing some embodiments of the present disclosure. Fig. 9 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0132] like Fig. 9As shown, the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0133] Typically, the following devices may be connected to the I / O interface 905: an input device 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 908 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 909. The communication device 909 may allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although Fig. 9 The electronic device 900 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Fig. 9 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0134] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 909, or installed from the storage device 908, or installed from the ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0135] It should be noted that the computer-readable medium in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0136] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0137] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains the current observation sample information of the humanoid robot, wherein the humanoid robot is a bipedal humanoid robot with clog-type soles, and the current observation sample information includes at least one of the following: the joint position, joint speed, joint angle, and ground coupling pressure information of the two feet of the humanoid robot; generates simulated elevation terrain information for the humanoid robot; inputs the current observation sample information into the action network included in the motor position control network to obtain a numerical set of action probability distribution; inputs the current observation sample information into The motor position control network includes a strategy network, which obtains the action evaluation value set corresponding to the above-mentioned action probability distribution value set; generates the target joint position set of the above-mentioned humanoid robot based on the above-mentioned action probability distribution value set and the above-mentioned action evaluation value set; inputs the above-mentioned target joint position set into the joint response controller to generate a joint torque set for the above-mentioned humanoid robot; controls the above-mentioned humanoid robot to perform foot lift-off movement training on the simulated elevation terrain corresponding to the above-mentioned simulated elevation terrain information based on the above-mentioned joint torque set; in response to determining that the above-mentioned action probability distribution value set and the above-mentioned action evaluation value set meet the preset loss condition, ends the training of the above-mentioned motor position control network.

[0138] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0139] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0140] The units described in some embodiments of the present disclosure may be implemented by software or by hardware. The described units may also be provided in a processor, for example, may be described as: a processor including an acquisition unit, a generation unit, a first input unit, a second input unit, a generation unit, a third input unit, a control unit, and an end unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the acquisition unit may also be described as a "unit for acquiring information of the currently observed sample".

[0141] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0142] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A humanoid robot foot-lifting movement training method, include: Acquire current observation sample information of the humanoid robot, wherein the humanoid robot is a bipedal humanoid robot with clog-type soles, and the current observation sample information includes at least one of the following: joint positions, joint velocities, joint angles, and ground coupling pressure information of the two feet of the humanoid robot; generating simulated elevation terrain information for the humanoid robot; Inputting the current observation sample information into the action network included in the motor position control network to obtain an action probability distribution value set; Inputting the current observation sample information into the strategy network included in the motor position control network to obtain an action evaluation value set corresponding to the action probability distribution value set; Generating a target joint position set of the humanoid robot according to the action probability distribution value set and the action evaluation value set; inputting the target joint position set into a joint response controller to generate a joint torque set for the humanoid robot; According to the joint torque set, controlling the humanoid robot to perform foot lift-off movement training on the simulated elevation terrain corresponding to the simulated elevation terrain information; In response to determining that the action probability distribution value set and the action evaluation value set satisfy a preset loss condition, the training of the motor position control network is terminated.

2. The method according to claim 1, in, The method further comprises: In response to determining that the action probability distribution value set and the action evaluation value set do not satisfy the preset loss condition, the observation sample information of the humanoid robot is reacquired as the current observation sample information, the relevant parameters of the motor position control network are readjusted to obtain the adjusted motor position control network, and the adjusted motor position control network is determined as the motor position control network to perform network training on the motor position control network again.

3. The method according to claim 1, in, The action network includes: an action input layer, an attention mechanism layer, a first action hidden layer, a second action hidden layer, a third action hidden layer and an action output layer; and The step of inputting the current observation sample information into the action network included in the motor position control network to obtain an action probability distribution value set includes: Inputting the current observation sample information and the simulated elevation terrain information into the action input layer to obtain a first humanoid robot action feature vector; Inputting the first humanoid robot action feature vector into the first action hidden layer to obtain a second humanoid robot action feature vector; Inputting the second humanoid robot action feature vector into the second action hidden layer to obtain a third humanoid robot action feature vector; Inputting the third humanoid robot action feature vector into the third action hidden layer to obtain a fourth humanoid robot action feature vector; Inputting the fourth humanoid robot action feature vector into the attention mechanism layer to obtain a humanoid robot action weight feature vector; Inputting the action weight feature vector of the humanoid robot into the action output layer to obtain the action mean and action variance; Constructing a set of action probability distribution values ​​for the action mean and the action variance.

4. The method according to claim 1, in, The strategy network includes: a value input layer, a long short-term memory neural network, a first value hidden layer, a second value hidden layer, a third value hidden layer and a value output layer; and The step of inputting the current observation sample information into the strategy network included in the motor position control network to obtain the action evaluation value set corresponding to the action probability distribution value set includes: Inputting the current observation sample information and the simulated elevation terrain information into the value input layer to obtain a first humanoid robot action value feature vector; Inputting the first humanoid robot action value feature vector into the first value hidden layer to obtain a second humanoid robot action value feature vector; Inputting the second humanoid robot action value feature vector into the second value hidden layer to obtain a third humanoid robot action value feature vector; Inputting the third humanoid robot action value feature vector into the third value hidden layer to obtain a fourth humanoid robot action value feature vector; Inputting the action value feature vector of the fourth humanoid robot into the long short-term memory neural network to obtain a walking posture time series feature vector set; The walking posture temporal feature vector set is input into the value output layer to obtain the action evaluation value set corresponding to the action probability distribution.

5. The method according to claim 1, in, The humanoid robot comprises: a thigh, a calf and a foot, wherein: The thigh and the calf are connected via a locking structure of a knee joint, the knee joint is composed of a ratchet pawl assembly and a four-bar mechanism assembly, and the locking structure of the knee joint is used to control the humanoid robot to perform knee extension and bending movements; The foot and the calf are connected through an ankle joint. The foot is a foot with a target number of circular collision shapes added to the bottom of the sole of the foot. The circular collision shape and the simulated elevation terrain corresponding to the simulated elevation terrain information generate a collision shape that couples the collision.

6. The method according to claim 1, in, The generating of simulated elevation terrain information for the humanoid robot comprises: Acquire a terrain plane grid of a preset size, wherein each grid vertex in the terrain plane grid includes a pseudo-random gradient vector; Determine an adjacent mesh vertex group of each input random coordinate point in the input random coordinate point set in the terrain plane network to obtain an adjacent mesh vertex group set, wherein the input random coordinate point is an arbitrary two-dimensional coordinate point randomly generated and located in the terrain plane network; For each adjacent mesh vertex group in the set of adjacent mesh vertex groups, perform the following linear interpolation steps: Determine a coordinate distance vector between each adjacent network vertex in the adjacent network vertex group and the corresponding input random coordinate point to obtain a coordinate distance vector group; Performing vector dot multiplication processing on each coordinate distance vector in the coordinate distance vector group and the corresponding pseudo-random gradient vector to obtain a coordinate dot product vector group; Performing linear interpolation on the coordinate point multiplication vector group to obtain a random noise value group for the input random coordinate point; According to the preset relaxation curve, initial noise fractal terrain information is generated according to each obtained random noise value group; The initial noise fractal terrain information is subjected to terrain rendering to obtain the rendered initial noise fractal terrain information as simulated elevation terrain information.

7. A humanoid robot foot lifting and moving training device, include: an acquisition unit, configured to acquire current observation sample information of a humanoid robot, wherein the humanoid robot is a bipedal humanoid robot with clog-type soles, and the current observation sample information includes at least one of the following: joint positions, joint velocities, joint angles, and ground coupling pressure information of the two feet of the humanoid robot; A first generating unit, configured to generate simulated elevation terrain information for the humanoid robot; A first input unit is configured to input the current observation sample information into the action network included in the motor position control network to obtain an action probability distribution value set; A second input unit is configured to input the current observation sample information into the strategy network included in the motor position control network to obtain an action evaluation value set corresponding to the action probability distribution value set; a second generating unit, configured to generate a target joint position set of the humanoid robot according to the action probability distribution value set and the action evaluation value set; a third input unit configured to input the target joint position set to a joint response controller to generate a joint torque set for the humanoid robot; a control unit configured to control the humanoid robot to perform foot lift-off movement training on the simulated elevation terrain corresponding to the simulated elevation terrain information according to the joint torque set; The ending unit is configured to end the training of the motor position control network in response to determining that the action probability distribution value set and the action evaluation value set meet a preset loss condition.

8. An electronic device, include: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, in, When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images