A humanoid robot inclined ladder climbing method, system and storage medium based on deep reinforcement learning
Through deep reinforcement learning technology and the method of decomposing ladder climbing tasks, an end-to-end decision-making model is built, which solves the problem of poor adaptability to environmental changes in traditional robots in climbing tasks, and improves the robustness of humanoid robots and task execution effect.
Patent Information
- Application Number
- CN202410615216.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-17
- Publication Date
- 2024-12-17
- Estimated Expiration
- 2044-05-17
AI Technical Summary
Traditional robots find it difficult to build a decision model that adapts to changes in the external environment, resulting in excessive dependence on point cloud data quality in climbing slanted ladder tasks, affecting the robustness of action decisions and task execution.
Using deep reinforcement learning technology, by designing deep neural networks and reward functions, humanoid robots are trained in a simulation environment, using camera images and servo parameters to directly output the servo angle, decomposing the ladder task into three stages, and building end-to-end decision-making Model.
A highly robust action decision model is realized, which can make optimal action choices based on different environments, reduce dependence on point cloud data, and improve the execution ability of humanoid robots in climbing tasks.
Smart Images
Figure CN118578375B_ABST
Abstract
Description
Technical Field
[0001] The present invention provides a humanoid robot inclined ladder climbing method based on deep reinforcement learning, which relates to the field of intelligent execution of tasks by humanoid robots. Background Art
[0002] With the development of science and technology, robots have entered every corner of life, assisting or replacing humans in various complex or repetitive tasks, such as industrial robotic arms. The development of robotics technology has improved the efficiency of human production activities, reduced human repetitive work, and reduced unnecessary labor costs. However, traditional robots can only follow their pre-set programs to make decisions, so they are destined to be used to complete unchanging work, and they cannot make the current optimal solution according to changes in the external environment. The development of artificial intelligence technology has injected new vitality into the field of traditional robots. Improved intelligent robots can have better adaptability to random changes in the external environment. They use intelligent algorithms or decision-making models to reduce their dependence on human expert knowledge and make the most appropriate action choices based on the specific conditions of the real environment. The intelligent reform of traditional robots has become a future trend in the field of robotics.
[0003] Deep reinforcement learning technology has been developing rapidly in recent years and has achieved good results in autonomous driving, chess AI, simulation, etc. It combines the feature extraction ability of deep learning and the autonomous learning ability of reinforcement learning, which can solve the training difficulties of decision-making models in robot control and provide new ideas for the intelligence of traditional robots.
[0004] Humanoid robots, as humanoid robots, are being studied in the direction of intelligence in full swing. There are related papers that use humanoid robots to study ladder climbing problems and explore methods for humanoid robots to autonomously perform tasks. It uses laser point cloud technology to collect data coordinates of ladders in real environments, and then uses intelligent algorithms to analyze the acquired point cloud data and sensor parameters to plan specified angle parameters for the humanoid robot servo, thereby achieving the execution of climbing tasks. However, this method needs to ensure the quality of the collected point cloud data. If the gap between the collected data information and the actual parameters is too large, it will have a great impact on the action decision of the humanoid robot, resulting in the failure of the ladder climbing task. In addition, the decision algorithm needs to be designed by constructing a mathematical model, and the quality of the decision algorithm design also directly affects the effect of the humanoid robot in performing tasks. Summary of the invention
[0005] In view of the defects or shortcomings in the prior art, the present invention provides a humanoid robot inclined ladder climbing method based on deep reinforcement learning technology, thereby solving technical problems such as the difficulty in constructing a traditional robot control decision model and the poor adaptability to changes in the external environment.
[0006] To achieve the above object, the technical solution of the present invention comprises the following steps:
[0007] S1: The humanoid robot's inclined ladder climbing task is divided into stages;
[0008] S2: Design deep reinforcement learning neural networks;
[0009] S3: Design reward functions for training deep reinforcement learning neural networks;
[0010] S4: Building training scenarios in a simulation environment;
[0011] S5: Based on the inclined ladder climbing scenario, a humanoid robot is used to train the deep reinforcement learning network model. The image obtained by the humanoid robot's camera and its own servo parameters are used as input to directly obtain the next angle value of its own servo, thereby realizing the execution of the ladder climbing task.
[0012] Furthermore, in step S1, the process of the humanoid robot climbing the inclined ladder is divided into three stages: the stage of grasping the target step, the stage of stepping on the target step, and the stage of adjusting the angle of the whole body servo. The humanoid robot uses the cycle of these three stages to simplify the complex ladder climbing task, thereby ensuring the robustness of the climbing task execution.
[0013] Further, the specific design process of step S2 is as follows:
[0014] S21: Constructing a decision neural network,
[0015] The convolutional neural network in the decision neural network consists of three convolutional layers. The images collected by the humanoid robot camera are sent to the convolutional neural network after preprocessing. Subsequently, the extracted feature map is expanded and the humanoid robot's own servo angle parameters are processed by two fully connected layers respectively, and then connected and sent to the fully connected layer to output the expected reward value of the corresponding action group;
[0016] S22: Calculate the gradient of the decision neural network,
[0017] The loss function of the decision neural network adopts the mean square error loss function:
[0018]
[0019] Among them, n represents the number of randomly sampled samples, i represents the sample number, and y i The reward for the original network is:
[0020] y i =Q w (s i ,a i )
[0021] Rewards for target network:
[0022]
[0023] S23: Update the target network by updating parameters.
[0024] Furthermore, the reward function reward in step S3 is designed as follows:
[0025]
[0026] Among them, x represents the distance between the reference point on the gripper of the humanoid robot and the reference point on the target step on the x-axis, y represents the distance between the reference point on the gripper of the humanoid robot and the reference point on the target step on the y-axis, and get represents whether the humanoid robot successfully clamps the target step.
[0027] Further, the simulation environment in step S4 is designed as follows:
[0028] S41: Build a scene with inclined ladders and humanoid robots required for the experiment in a virtual simulation environment;
[0029] S42: Design a hand gripper for a humanoid robot based on the size of the steps of an inclined ladder, so that it can clamp the target step by closing the hand servo, thereby performing the inclined ladder climbing task;
[0030] Furthermore, the steps of training the deep reinforcement learning network model in step S5 are as follows:
[0031] S51: The virtual simulation software collects the camera images and servo angle parameters of the humanoid robot during the experiment and sends them to the deep reinforcement learning framework;
[0032] S52: The simulation software determines whether the target ladder grasping task is completed based on the sensor information, gives a reward value for each action through a reward function, and sends the reward value to the deep reinforcement learning framework;
[0033] S53: Deep reinforcement learning decides actions based on existing information, calculates rewards for the decided actions, and performs backpropagation on the neural network by calculating the rewards, thereby tuning the parameters of the neural network, and continuously repeats the above steps until the decision model training is completed.
[0034] The second aspect of the present invention provides a humanoid robot inclined ladder climbing system based on deep reinforcement learning, comprising: a processor, and a memory and a network interface connected to the processor; the network interface is connected to a non-volatile memory in a server; the processor retrieves a computer program from the non-volatile memory through the network interface during operation, and runs the computer program through the memory to execute any one of the methods in the claims.
[0035] A third aspect of the present invention provides a humanoid robot ladder climbing storage medium based on deep reinforcement learning, wherein the storage medium is burned with a computer program, and when the computer program is run in the memory of a server, it implements any one of the methods in the claims.
[0036] A fourth aspect of the present invention provides a ladder-climbing humanoid robot, comprising a computer program / instruction, which implements any method in the claims when executed by a processor.
[0037] Beneficial effects of the present invention:
[0038] 1. The method of dividing the stages according to the inclined ladder climbing process proposed in the present invention can effectively modularize the complex ladder climbing task and avoid the problem of difficulty in training the decision model due to directly starting from the whole.
[0039] 2. The present invention proposes a method for training a decision model based on deep reinforcement learning technology. By relying only on the information provided by the humanoid robot camera and its internal servo sensor, an end-to-end neural network capable of outputting the servo angle action is constructed. By continuously changing the initial position of the humanoid robot, a highly robust action decision model can be obtained, which can make the best action selection that best suits the current environment according to the different external environments. This solves the problem that traditional robots are overly dependent on point cloud data and do not have good robustness when performing ladder climbing tasks, and realizes the intelligent ladder climbing of humanoid robots. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 It is a flow chart of a humanoid robot inclined ladder climbing method based on deep reinforcement learning of the present invention.
[0041] Figure 2 It is a diagram for dividing the stages of the inclined ladder climbing task of the present invention.
[0042] Figure 3 It is a decision neural network model diagram of the present invention.
[0043] Figure 4 This is a scene diagram of the simulation platform experiment of the present invention.
[0044] Figure 5 The present invention uses modeling software to model the gripper of a humanoid robot hand.
[0045] Figure 6 The present invention uses modeling software to model the gripper of a humanoid robot hand.
[0046] Figure 7 The present invention adds a gripper to the hand of a humanoid robot.
[0047] Figure 8 This is a framework diagram of the DQN algorithm used in the present invention.
[0048] Fig. 9 It is the action group diagram finally adopted by the present invention. DETAILED DESCRIPTION
[0049] With the following combination Figures 1 to 9 The description and specific implementation methods further illustrate the present invention.
[0050] like Figure 1 As shown, the present invention comprises the following steps:
[0051] S1: The humanoid robot inclined ladder climbing task is divided into stages. The specific design process is as follows:
[0052] like Figure 2 As shown, the inclined ladder climbing task is divided into the stage of grabbing the target step, the stage of stepping on the target step, and the stage of adjusting the whole body servo angle. In this embodiment, the stage of grabbing the target step is that the humanoid robot collects data information through the head camera and the internal servo angle sensor, sends it to the action decision model obtained by training, and controls the gripper to approach the target step according to the next step servo angle data obtained, thereby completing the target step grab; the stage of stepping on the target step is that the humanoid robot adjusts the leg servo angle by using a fixed action group so that the foot pedal can step on the target step smoothly; the stage of adjusting the whole body servo angle is that after the humanoid robot grabs or steps on the target step, it uses a fixed action group to adjust the whole body servo angle to a preset angle, ensuring that the robot's whole body servo angle remains basically unchanged when the next servo movement is controlled, thereby reducing the training difficulty of the decision model.
[0053] S2: Design a deep reinforcement learning neural network. The specific design process is as follows:
[0054] S21: Constructing a decision neural network:
[0055] like Figure 3As shown, the convolutional neural network model in the decision neural network consists of three convolutional layers. In this embodiment, a camera image of size 128×128×1 is first used as input, and after three convolutional layers, the convolution kernel size is 5×5, the number of convolution kernels is 32, and the horizontal and vertical step lengths are 2. Then the obtained feature map is expanded into a 1-dimensional vector and subjected to dimensionality reduction calculations through two layers of fully connected layers. At the same time, the servo parameters of the humanoid robot of size 20×1 are subjected to dimensionality increase through two layers of fully connected layers. Then the two are connected to each other and input into the fully connected layer together. The sizes of fully connected layers 1, 2, 3, 4 and 5 are 100, 100, 6000, 100 and 200 respectively. Finally, the target action group with the largest predicted Q value in the output layer is selected for execution.
[0056] S22: Computing the gradient of the decision neural network
[0057] The loss function of the decision neural network adopts the mean square error loss function:
[0058]
[0059] Among them, n represents the number of randomly sampled samples, i represents the sample number, and y i The reward for the original network is:
[0060] y i =Q w (s i ,a i )
[0061] Rewards for target network:
[0062]
[0063] S23: Update the target network by updating parameters;
[0064] S3: Design a reward function for the humanoid robot's inclined ladder step grasping task:
[0065]
[0066] Among them, xx represents the distance between the reference point on the gripper of the humanoid robot and the reference point on the target step on the x-axis, yy represents the distance between the reference point on the gripper of the humanoid robot and the reference point on the target step on the y-axis, and get represents whether the humanoid robot successfully clamps the target step.
[0067] S4: Build training scenarios in a simulation environment:
[0068] S41: A scene with an inclined ladder and a humanoid robot required for the experiment is built in a virtual simulation environment. In this embodiment, the simulation platform Webots is used to build the environment, and the humanoid robot uses the Robotis-op2 robot model. Figure 4 As shown, an inclined ladder of the required size is built on the simulation platform Webots, with a length and height of 1m, a width of 0.5m, an inclination angle of 45°, and a step cross section of 3cm×3cm;
[0069] S42: According to the size of the steps of the inclined ladder, a hand gripper is designed for the humanoid robot so that the robot can clamp the target step by closing the hand servo, thereby performing the inclined ladder climbing task. Figure 5-7 As shown, the hand gripper of the humanoid robot is modeled using Solidwork modeling software, and is installed on the hand of the humanoid robot so that it can obtain the ability to grip the steps of the inclined ladder;
[0070] S5: Using a humanoid robot to train a deep reinforcement learning network model based on an inclined ladder climbing scenario:
[0071] like Figure 8 As shown, by using the DQN deep reinforcement learning algorithm, the agent interacts with the environment to obtain rewards to train the action decision model. In this embodiment, according to the humanoid robot inclined ladder climbing task, the state space and action space of the target ladder grasping are designed. Among them, the state space S is as follows:
[0072] S = [observe, angle]
[0073] observe represents the image observed by the head camera of the humanoid robot Robotis-op2, and angle represents the current angle parameter of the whole body servo of the humanoid robot Robotis-op2.
[0074] The action space A is as follows:
[0075] A=[index]
[0076] Index indicates the target action group number selected. There are 2 different action groups in total. The humanoid robot controls the movement of the arm servo by selecting the action group. Its design is as follows:
[0077] The Robotis-op2 arm has two servos, Shoulder and Armlower. Each servo is designed with two actions of +0.1 radian and 0 radian. By permuting and combining the actions of the two servos, four different action groups can be obtained. Excluding the action group where both servos are 0 and +0.1 radian, such as Fig. 9As shown, this embodiment finally has 2 action groups.
Claims
1. A humanoid robot inclined ladder climbing method based on deep reinforcement learning, characterized in that: The following steps are involved: S1: The humanoid robot's inclined ladder climbing task is divided into stages; S2: Design deep reinforcement learning neural networks; S3: Design reward functions for training deep reinforcement learning neural networks; S4: Building training scenarios in a simulation environment; S5: Based on the inclined ladder climbing scenario, a humanoid robot is used to train the deep reinforcement learning network model. The image obtained by the humanoid robot's camera and its own servo parameters are used as input to directly obtain the next angle value of its own servo, thereby realizing the execution of the ladder climbing task.
2. The method for climbing an inclined ladder of a humanoid robot based on deep reinforcement learning according to claim 1, characterized in that: The specific process of dividing the stages of the humanoid robot's inclined ladder climbing task includes: The inclined ladder climbing task of the humanoid robot is divided into the stage of grasping the target step, the stage of stepping on the target step and the stage of adjusting the angle of the whole body servo. The humanoid robot uses the cycle of the above three stages to simplify the complex ladder climbing task, thereby ensuring the robustness of the climbing task execution.
3. The method for climbing an inclined ladder of a humanoid robot based on deep reinforcement learning according to claim 2, characterized in that: The process of designing a deep reinforcement learning neural network includes: S21: Constructing a decision neural network, The convolutional neural network in the decision neural network consists of three convolutional layers. The images collected by the humanoid robot camera are sent to the convolutional neural network after preprocessing. Then, the extracted feature map is expanded and the angle parameters of the humanoid robot's own servo are processed by two fully connected layers respectively, and then connected and sent to the fully connected layer to output the expected reward value of the corresponding action group. S22: Calculate the gradient of the decision neural network, The loss function of the decision neural network adopts the mean square error loss function: Among them, n represents the number of randomly sampled samples, i represents the sample number, and y i The reward for the original network is: and i =Q w (s i ,to i ) Rewards for target network: S23: Update the target network by updating parameters.
4. The method for climbing an inclined ladder of a humanoid robot based on deep reinforcement learning according to claim 1, characterized in that: The reward function for training the deep reinforcement learning neural network is designed as follows: Among them, x represents the distance between the reference point on the gripper of the humanoid robot and the reference point on the target step on the x-axis, y represents the distance between the reference point on the gripper of the humanoid robot and the reference point on the target step on the y-axis, and get represents whether the humanoid robot successfully clamps the target step.
5. The method for climbing an inclined ladder of a humanoid robot based on deep reinforcement learning according to claim 1, characterized in that: The process of building a training scenario in a simulation environment includes: S41: Build a scene with inclined ladders and humanoid robots required for the experiment in a virtual simulation environment; S42: Design a hand gripper for a humanoid robot based on the size of the inclined ladder steps so that it can clamp the target step by closing the hand servo, thereby performing the inclined ladder climbing task.
6. The method for climbing an inclined ladder of a humanoid robot based on deep reinforcement learning according to claim 1, characterized in that: Based on the inclined ladder climbing scenario, a humanoid robot is used to train the deep reinforcement learning network model. The image obtained by the humanoid robot camera and its own servo parameters are used as input to directly obtain the next angle value of its own servo, thereby realizing the execution process of the ladder climbing task, including: S51: The virtual simulation software collects the camera images and servo angle parameters of the humanoid robot during the experiment and sends them to the deep reinforcement learning framework; S52: The simulation software determines whether the target ladder grasping task is completed based on the sensor information, gives a reward value for each action through a reward function, and sends the reward value to the deep reinforcement learning framework; S53: Deep reinforcement learning decides actions based on existing information, calculates rewards for the decided actions, and performs backpropagation on the neural network by calculating the rewards, thereby tuning the parameters of the neural network, and continuously repeats the above steps until the decision model training is completed.
7. A humanoid robot ladder climbing system based on deep reinforcement learning, characterized by: include: A processor, and memory and network interfaces connected to the processor; The network interface is connected to a non-volatile memory in the server; When running, the processor retrieves a computer program from the non-volatile memory through the network interface, and runs the computer program through the memory to execute the method according to any one of claims 1 to 6.
8. A humanoid robot ladder climbing storage medium based on deep reinforcement learning, characterized in that: The storage medium is burned with a computer program, and the computer program implements the method according to any one of claims 1 to 6 when running in the memory of the server.
9. An inclined ladder climbing humanoid robot, comprising a computer program / instructions, characterized in that: When the computer program / instructions are executed by a processor, the method described in any one of claims 1 to 6 is implemented.