Quadruped robot path planning method based on deep reinforcement learning, computer equipment and storage medium
Through the deep reinforcement learning method combined with radar and image information, the motion fusion framework and multiple control algorithms are designed, which solves the problem of complex multi-joint control of four-legged robots and improves the stability and motion flexibility of path planning.
Patent Information
- Application Number
- CN202411187545.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-28
- Publication Date
- 2024-12-03
Smart Images

Figure CN119063733A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of robot control technology, and specifically to a quadruped robot path planning method based on deep reinforcement learning, a computer device and a storage medium. Background Art
[0002] As a robot with a unique structure, quadruped robots can adapt to most terrains and perform more complex tasks; this robot is equipped with a biological-like multi-legged structure, which enables it to move stably on various uneven or rugged terrains; they can cope with various challenges, such as crossing obstacles, walking on slippery surfaces, and maneuvering flexibly in narrow spaces.
[0003] Due to the flexibility and adaptability of quadruped robots, they have shown great potential in many fields; for example, in rescue missions, they can enter ruins that are difficult for traditional vehicles to reach for search and rescue; in the agricultural field, quadruped robots can perform operations such as planting and harvesting on uneven fields; in the military and security fields, they can also be used for patrolling and surveillance to improve the efficiency and safety of mission execution.
[0004] However, quadruped robots are more difficult to control than wheeled robots and tracked robots because they have more joints. Quadruped robots need to accurately coordinate the movements of multiple joints to ensure a stable gait and flexible movement. This complex control requirement makes quadruped robots face greater challenges in algorithm and hardware design. The movement of each joint must not only consider independent control, but also comprehensive consideration of the overall balance and dynamic adjustment to cope with changes in different terrains and environments. Currently available control algorithms mostly use a single control strategy, which often leads to failure in path planning or body posture maintenance when the output value is disturbed. Therefore, the difficulty in controlling quadruped robots due to the large number of joints is an urgent problem that needs to be solved. Summary of the invention
[0005] The purpose of the present invention is to provide a quadruped robot path planning method, computer equipment and storage medium based on deep reinforcement learning.
[0006] Based on the above purpose, the present invention adopts the following technical solution:
[0007] A quadruped robot path planning method based on deep reinforcement learning comprises the following steps:
[0008] S1. Create a ground obstacle simulation environment: Use the simulation platform to establish a ground obstacle simulation environment with complex terrain, and set static and dynamic obstacles that cannot be climbed over;
[0009] S2. Parameter initialization: Randomly initialize the target position and the quadruped robot position so that the initial distance between them is not less than the preset value; initialize the success threshold. When the distance between the target position and the quadruped robot position is less than the success threshold, it is considered that the obstacle is successfully crossed; the target position is expressed as:
[0010] G=(x g ,y g )
[0011] The position of the quadruped robot is expressed as:
[0012]
[0013] S3, deploy control algorithm: deploy the motion fusion framework to the industrial computer of the quadruped robot;
[0014] S4. Environmental perception: The quadruped robot perceives the environment through radar and industrial camera. The quadruped robot is equipped with radar and industrial camera. The radar is used to obtain the environment's corona information, and the industrial camera is used to obtain the environment's image information.
[0015] S5, generating fused action: inputting the state sensed in step S4 into the action fusion framework to generate a fused action;
[0016] S6, executing action: the quadruped robot executes the fusion action generated in step S5;
[0017] S7, loop action: repeat the above steps S4-S6 until the distance between the quadruped robot and the target is less than the success threshold set in step S2.
[0018] Preferably, the input channel of the action fusion framework in step S3 includes a point cloud information input channel and an image information input channel; the point cloud information is de-redundantly processed before being input into the action fusion framework, and the image information is enhanced before being input into the action fusion framework; the point cloud information after de-redundantly processing and removing redundant information is represented as Radar t , the enhanced image information is represented as Image t .
[0019] Preferably, the action fusion framework in step S3 includes multiple deep reinforcement learning control algorithm modules, and the input of each deep reinforcement learning control algorithm module comes from one of the point cloud information input channel or the image information input channel. The action space of the nth deep reinforcement learning control algorithm module is expressed as:
[0020]
[0021] In the formula, represents the pitch and roll hip joint angles of each leg, Indicates the pitch and elevation knee joint angles of each leg.
[0022] Preferably, different deep reinforcement learning control algorithms respectively maintain corresponding experience replay pools, and samples in different experience replay pools are isolated from each other.
[0023] Preferably, the deep reinforcement learning control algorithm includes a reward function, and the reward function is expressed as:
[0024]
[0025] In the formula, r d represents the distance reward, r θ represents the stability reward, r q represents the posture stability reward, r e Represents energy reward. Reward function r d 、r θ 、r q 、r e Respectively expressed as:
[0026] r d =tanh(d t -d t-1 )
[0027]
[0028] Where, d t represents the distance from the quadruped robot to the target at time t, Represents the change in the roll angle within a time step, represents the change in pitch angle within a time step, represents the swing amplitude of the hip joint of the i-th leg, q ref Indicates the swing zero value of the leg hip joint. represents the swing amplitude of the knee joint of the i-th leg, g ref Indicates the swing zero value of the knee joint of the leg. represents the torque of the i-th joint motor, represents the angular velocity of the i-th joint motor;
[0029] where d t It is expressed as:
[0030]
[0031] In the formula, is the position coordinate of the quadruped robot, (x g ,y g ) are the coordinates of the target position.
[0032] Preferably, the point cloud information input channel is fused with the point cloud information Radart , the distance d between the quadruped robot and the target point t , the position of the target point on the quadruped robot γ t 、Flag roll angle Fuselage pitch angle As a state, it is represented as:
[0033]
[0034] Image information input channel fusion image information Image t , the distance d between the quadruped robot and the target point t , the position of the target point on the quadruped robot γ t 、Flag roll angle Fuselage pitch angle As a state, it is represented as:
[0035]
[0036] Preferably, the action fusion module of the action fusion framework in step S3 is expressed as:
[0037]
[0038] β1+β2+…+β n =1
[0039] In the formula, is the output action of the nth deep reinforcement learning control algorithm module, β n The weight of the output action of the nth deep reinforcement learning control algorithm module in the fusion action; the action space of the deep reinforcement learning control algorithm module should all be continuous or discrete types.
[0040] Preferably, the output action of the nth deep reinforcement learning control algorithm module is It is expressed as:
[0041]
[0042] In the formula, represents the pitch and roll hip joint angles of each leg, Indicates the pitch and elevation knee joint angles of each leg.
[0043] A computer device includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, any step in the above-mentioned quadruped robot path planning method based on deep reinforcement learning is implemented.
[0044] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a computer processor, implements any step in the above-mentioned method for path planning of a quadruped robot based on deep reinforcement learning.
[0045] The beneficial effects of the present invention include:
[0046] The present invention sets up two types of input channels, taking radar point cloud information and image information as part of the state, and considering the representation of the environment by depth information and texture information. The complementarity of the two can be more conducive to the mapping of actions by the controller.
[0047] The present invention proposes an action fusion framework, which cooperates with the point cloud information and image information included in the input state of the algorithm, thereby effectively improving the overall generalization ability of the control algorithm; for a quadruped robot with multiple action value outputs, it is easy to cause large errors in the output of a certain part of the action value due to network parameter oscillation and dynamic changes in model parameters; by embedding multiple deep reinforcement learning algorithms in the action fusion framework, the final fused action can be generated by the fusion of different actions, avoiding the problem of inaccurate action due to the oscillation of a single network parameter, and effectively improving the stability of the action.
[0048] The present invention designs a corresponding reward function for the specific carrier of a quadruped robot. The reward function consists of a distance reward, a stability reward, a posture reward, and an energy reward. The distance reward is used to drive the robot to approach the target position, the stability reward is used to maintain the stability of the body of the quadruped robot during movement, and the posture reward is used to coordinate the regular movement between the legs of the quadruped robot to produce gait characteristics that conform to the movement pattern of quadrupeds in nature and produce regular movement; the energy reward is used to punish the quadruped robot for energy consumption, which encourages the quadruped robot to achieve movement with as little energy consumption as possible. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 It is a schematic diagram of the process of the present invention;
[0050] Figure 2 It is a schematic diagram of the action fusion framework of the present invention. DETAILED DESCRIPTION
[0051] Example 1
[0052] The following is a further explanation of the present invention in conjunction with specific embodiments. Figure 1 As shown, this embodiment is a quadruped robot path planning method based on deep reinforcement learning, comprising the following steps:
[0053] S1. Create a ground obstacle simulation environment: In this embodiment, a ground obstacle simulation environment is developed on the Gazebo platform, and the environment size is set to 60 meters × 60 meters; complex terrain is set in the environment, including gravel areas, stairs, surmountable obstacles, etc.; static non-surmountable obstacles and dynamic non-surmountable obstacles are set, and the motion trajectory of dynamic obstacles is set to a straight line or a circle.
[0054] S2, parameter initialization: randomly initialize the target position and the quadruped robot position so that the initial distance between them is not less than 30 meters; the target position is expressed as:
[0055] G=(x g ,y g )
[0056] The position of the quadruped robot is expressed as:
[0057]
[0058] S3. Deploy control algorithm: deploy the motion fusion framework to the industrial computer of the quadruped robot, where the input channel of the motion fusion framework includes the environmental point cloud information obtained by radar and the environmental image information obtained by industrial camera.
[0059] S4. Environmental perception: The environment is perceived through the radar and industrial camera installed on the quadruped robot. The radar is used to obtain point cloud information of the environment, and the industrial camera is used to obtain image information of the environment.
[0060] The point cloud information needs to be de-redundant before being input into the action fusion framework to delete the redundant point cloud information. The de-redundant processing operation in this embodiment is: setting the angle interval, which is expressed as:
[0061] Gap={[0,10°],[10°,20°],…,[170°,180°]}
[0062] Calculate the distance and angle of the point cloud, expressed as:
[0063]
[0064] In the formula, represents the geometric distance from the i-th point cloud information to the radar at time t, Represents the horizontal angle between the i-th point cloud information and the x-axis at time t.
[0065] By comparing the angle gap and the included angle Select each interval The smallest three point clouds are stored in Radar t .
[0066] Image information needs to be enhanced before entering the action fusion framework. In this embodiment, the Retinex algorithm is used to enhance the image. The processed image is represented as Image t .
[0067] S5, generating fused action: inputting the state perceived in step S4 into the action fusion framework to generate a fused action.
[0068] The input channels of the action fusion framework include point cloud information input channel and image information input channel, where the point cloud information input channel is fused with point cloud information Radar t , the distance d between the quadruped robot and the target point t , the position of the target point on the quadruped robot γ t 、Flag roll angle Fuselage pitch angle As a state, it is represented as:
[0069]
[0070] Image information input channel fusion image information Image t , the distance d between the quadruped robot and the target point t , the position of the target point on the quadruped robot γ t 、Flag roll angle Fuselage pitch angle θ pitch As a state, it is represented as:
[0071]
[0072] In this embodiment, three deep reinforcement learning control algorithms, namely, PPO algorithm, DDPG algorithm, and TD3 algorithm, are deployed in the action fusion framework; the outputs of the three control algorithms are all continuous values, which is convenient for the action fusion module to fuse actions; the input channel of the PPO algorithm is the image information input channel, and the input channels of the DDPG algorithm and the TD3 algorithm are the point cloud information input channel.
[0073] The action space of the deep reinforcement learning control algorithm module in this embodiment is expressed as:
[0074]
[0075] In the formula, represents the pitch and roll hip joint angles of each leg, Indicates the pitch and elevation knee joint angles of each leg.
[0076] The deep reinforcement learning control algorithm is set with a reward function, which is expressed as:
[0077]
[0078] In the formula,
[0079] r d =tanh(d t -d t-1 )
[0080]
[0081] In the formula, r d represents the distance reward, r θ represents the stability reward, r q represents the posture stability reward, r e Indicates energy reward; d t represents the distance from the quadruped robot to the target at time t, and the position coordinates of the quadruped robot are expressed as The coordinates of the target position are expressed as (x g ,y g ), Represents the change in the roll angle within a time step, represents the change in pitch angle within a time step, represents the swing amplitude of the hip joint of the i-th leg, q ref Indicates the swing zero value of the leg hip joint. represents the swing amplitude of the knee joint of the i-th leg, g ref Indicates the swing zero value of the knee joint of the leg. represents the torque of the i-th joint motor, Represents the angular velocity of the i-th joint motor.
[0082] In this embodiment, the PPO algorithm, the DDPG algorithm, and the TD3 algorithm respectively maintain their own experience replay pools, and samples in different experience replay pools are isolated from each other.
[0083] The action fusion module in the action fusion framework in this embodiment is represented as:
[0084]
[0085] β1+β2+β3=1
[0086] In the formula, is the output action of the PPO algorithm, is the output action of the DDPG algorithm, It is the output action of TD3 algorithm.
[0087] β n is the weight of the output action of the nth deep reinforcement learning control algorithm module in the fusion action. In this embodiment, radar perception information is mainly used, and image information is supplemented. In this embodiment, β1=0.2, β2=β3=0.4.
[0088] S6. Execute action: The quadruped robot controls each joint to execute action according to the fusion action generated in step S5.
[0089] S7, loop action: repeat the above steps S4-S6 until the distance d between the quadruped robot and the target is t is less than the success threshold set in step S2; when the distance d between the quadruped robot and the target t When the value is less than the success threshold set in step S2, it is considered that the obstacle is successfully overcome and the loop stops.
[0090] Example 2
[0091] The difference between this embodiment and embodiment 1 is that this embodiment is a computer device, including a memory and a processor, the memory stores a computer program, the computer program includes the quadruped robot path planning method based on deep reinforcement learning in the above-mentioned embodiment 1; or / and the processor implements any step in the quadruped robot path planning method based on deep reinforcement learning in the above-mentioned embodiment 1 when executing the computer program.
[0092] Example 3
[0093] The difference between this embodiment and embodiment 2 is that this embodiment is a computer-readable storage medium on which a computer program is stored, and the computer program includes the deep reinforcement learning quadruped robot path planning method of the above-mentioned embodiment 1; or / and when the computer program is executed by the processor, any step in the deep reinforcement learning quadruped robot path planning method of the above-mentioned embodiment 1 is implemented.
[0094] The above description is only a further explanation of the present invention in combination with specific embodiments. All descriptions do not limit the scope of protection of the present invention. Any changes or replacement schemes that can be easily thought of by any technician in this field within the technical scope disclosed by the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.
Claims
1. A quadruped robot path planning method based on deep reinforcement learning, characterized in that: The following steps are involved: S1. Create a ground obstacle simulation environment: Use the simulation platform to establish a ground obstacle simulation environment with complex terrain, and set static and dynamic obstacles that cannot be climbed over; S2, parameter initialization: randomly initialize the target position and the quadruped robot position so that the initial distance between them is not less than the preset value; initialize the success threshold, and when the distance between the target position and the quadruped robot position is less than the success threshold, it is considered that the obstacle is successfully crossed; S3, deploy control algorithm: deploy the motion fusion framework to the industrial computer of the quadruped robot; S4, Environmental Perception: The quadruped robot perceives the environment through radar and industrial cameras; S5, generating fused action: inputting the state sensed in step S4 into the action fusion framework to generate a fused action; S6, executing action: the quadruped robot executes the fusion action generated in step S5; S7, loop action: repeat the above steps S4-S6 until the distance between the quadruped robot and the target is less than the success threshold set in step S2.
2. The quadruped robot path planning method based on deep reinforcement learning according to claim 1, characterized in that: The input channels of the action fusion framework in step S3 include a point cloud information input channel and an image information input channel; the point cloud information is de-redundantly processed before being input into the action fusion framework, and the image information is enhanced before being input into the action fusion framework.
3. The quadruped robot path planning method based on deep reinforcement learning according to claim 2, characterized in that: The action fusion framework in step S3 includes a plurality of different deep reinforcement learning control algorithm modules, and the input of each deep reinforcement learning control algorithm module comes from one of the point cloud information input channels or the image information input channels.
4. The quadruped robot path planning method based on deep reinforcement learning according to claim 3, characterized in that: Different deep reinforcement learning control algorithms respectively maintain corresponding experience replay pools, and samples in different experience replay pools are isolated from each other.
5. The quadruped robot path planning method based on deep reinforcement learning according to claim 3, characterized in that: The deep reinforcement learning control algorithm includes a reward function, which is expressed as: In the formula, r d represents the distance reward, r θ represents the stability reward, r q represents the posture stability reward, r e Indicates energy reward; The reward function r d 、r θ 、r q 、r e Respectively expressed as: r d =tanh(d t -d t-1 ) Where, d t represents the distance from the quadruped robot to the target at time t, Represents the change in the roll angle within a time step, represents the change in pitch angle within a time step, represents the swing amplitude of the hip joint of the i-th leg, q ref Indicates the swing zero value of the leg hip joint. represents the swing amplitude of the knee joint of the i-th leg, g ref Indicates the swing zero value of the knee joint of the leg. represents the torque of the i-th joint motor, represents the angular velocity of the i-th joint motor; where d t It is expressed as: In the formula, is the position coordinate of the quadruped robot, (x g ,y g ) are the coordinates of the target position.
6. The quadruped robot path planning method based on deep reinforcement learning according to claim 3, characterized in that: The point cloud information input channel is fused with the point cloud information Radar t , the distance d between the quadruped robot and the target point t , the position of the target point on the quadruped robot γ t 、Flag roll angle Fuselage pitch angle As a state; the image information input channel fuses the image information Image t , the distance d between the quadruped robot and the target point t , the position of the target point on the quadruped robot γ t 、Flag roll angle Fuselage pitch angle As a state.
7. The quadruped robot path planning method based on deep reinforcement learning according to claim 1, characterized in that: The action fusion module of the action fusion framework in step S3 is expressed as: β1+β2+…+β n =1 In the formula, is the output action of the nth deep reinforcement learning control algorithm module, β n The weight of the output action of the nth deep reinforcement learning control algorithm module in the fusion action.
8. The quadruped robot path planning method based on deep reinforcement learning according to claim 7, characterized in that: The output action of the nth deep reinforcement learning control algorithm It is expressed as: In the formula, represents the pitch and roll hip joint angles of each leg, Indicates the pitch and elevation knee joint angles of each leg.
9. A computer device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, it implements any step in the quadruped robot path planning method based on deep reinforcement learning as described in any one of claims 1-8.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a computer processor, it implements any step in the quadruped robot path planning method based on deep reinforcement learning as described in any one of claims 1-8.
Citation Information
Cited By
Continuous trajectory planning method for excavator working device
CN119692582A
Aquaculture inspection robot path planning method
CN119717831A
Path planning method for aquaculture inspection robot
CN119717831B
