Mechanical arm path planning system based on reinforcement learning

Abstract

The invention discloses a mechanical arm path planning system based on reinforcement learning, and relates to the field of robots, and the mechanical arm path planning system comprises the following steps: constructing a mechanical arm motion path model based on reinforcement learning in an upper computer by applying a DQN model; the mechanical arm is trained through the generated model to achieve accurate point-to-point movement; model information is transmitted to the mechanical arm; and obstacle space information of the motion environment of the mechanical arm is collected and input into the trained mechanical arm motion path model, real-time path adjustment is conducted, and work is completed. According to the method, based on the neural network model of the DQN model reinforcement learning algorithm, the relatively optimal path in the specific obstacle space is searched through multiple times of training, the working efficiency of the mechanical arm is improved, and the mechanical arm can better perform path planning during working.

Description

Technical Field

[0001] The present invention belongs to the field of automation and intelligent manufacturing, and specifically relates to a robot arm path planning system based on reinforcement learning Background Art

[0002] In the field of automation and intelligent manufacturing, accurate path planning of robotic arms is crucial to improving production efficiency and ensuring operational safety. Traditional path planning methods often rely on pre-set rules and algorithms, which are difficult to adapt to dynamically changing environments and complex tasks. In recent years, reinforcement learning, as an effective deep learning method, has been widely used in the autonomous learning and decision-making process of robotic arms due to its advantages in dealing with uncertainty and continuous action space. This reinforcement learning-based robotic arm path planning system can learn the optimal path through interaction with the environment, showing adaptability and flexibility in complex and dynamic environments, thereby promoting the development of intelligent robotics technology. Summary of the invention

[0003] Technical problems solved: In the field of automation and intelligent manufacturing, reinforcement learning technology solves the problem that traditional robot arm path planning methods are not adaptable enough in dynamic and complex environments. Through autonomous learning and decision-making processes, the robot arm can optimize the path in real time, improve production efficiency and ensure operational safety.

[0004] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0005] In a first aspect, a robot arm path planning system based on reinforcement learning comprises:

[0006] The DQN model is applied in the upper computer to build a robot arm motion path model based on reinforcement learning; the generated model is used to train the robot arm to achieve precise point-to-point movement; the model information is transmitted to the lower computer, i.e. the robot arm; the obstacle space information of the robot arm's motion environment is collected and input into the trained robot arm motion path model to perform real-time path adjustment to complete the work.

[0007] The robot arm path planning system based on reinforcement learning is characterized by applying the deep Q network (DQN) model to train the robot arm to achieve accurate point-to-point movement in an environment with random obstacles. The specific steps include:

[0008] S1: Randomly generate spatial relative position information of the robot arm sensor and the robot arm end effector and random obstacles

[0009] S2: Input information into the DQN model

[0010] S3: Output each path that can achieve collision-free point-to-point precise movement through the DQN model

[0011] S4: Select the relatively optimal path

[0012] The robot arm path planning system based on reinforcement learning is characterized in that the reward function of the reinforcement learning DQN model includes:

[0013] R(s,a,s′)=w1·R distance (s,s′)+w2·R speed (a)+w3·R collision (s′)

[0014] Where: R(s,a,s′) is the total reward function, s is the current state of the robot, a is the action performed, and s′ is the next state of the robot.

[0015] The robot arm path planning system based on reinforcement learning is characterized in that the reward function of the reinforcement learning DQN model includes:

[0016] R(s,a,s′)=w1·R distance (s,s′)+w2·R speed (a)+w3·R collision (s′)

[0017] Where: R(s,a,s′) is the total reward function, s is the current state of the robot, a is the action performed, and s′ is the next state of the robot.

[0018] Distance reward function: R distance (s,s′)=d current -d next , d current is the distance from the current point to the target point, d next is the distance from the next state to the target point.

[0019] Speed ​​reward function: R speed (a)=-‖a‖ 2 , ‖a‖ is the amplitude of the action, and the negative sign minimizes the amplitude of the action, that is, encourages smaller joint speeds to avoid excessively fast movements that may cause instability or collision of the robot arm.

[0020] Collision penalty function: Among them, C is a large negative number, which indicates the penalty for collision. ω1, ω2, and ω3 represent the weights of each function according to the amount of motion.

[0021] The reinforcement learning-based robotic arm path planning system is characterized by constructing a state-action Q table through a Q-learning algorithm, and the Q table maps the robotic arm action and state to Q values.

[0022] The robot arm path planning system based on reinforcement learning is characterized by constructing a Q network and a target network, wherein the Q network is identical to the target network initially, and after training, the Q network is the agent with the best state-action value, and the target network remains stable.

[0023] The robot path planning system based on reinforcement learning is characterized by minimizing the loss between the Q value predicted by the DQN model and the target Q value:

[0024] Loss function:

[0025] Where: L(θ) represents the loss function, θ is the network parameter. N is the size of the batch sample; is the current network state s i and action a i Q value prediction of y i is the Q value of the i-th target sample, and its calculation formula is as follows:

[0026] y i =r i +γmax a′ Q(s i+1 ,a′;θ - )γ is the discount factor, Q(s i+1 ,a′;θ - ) The target network has the next state s i+1 The maximum Q value of all possible actions a′.

[0027] The robot arm path planning system based on reinforcement learning is characterized by making the robot arm move to the target position according to the optimized path:

[0028] S1: Define DH parameters. For each joint, define four DH parameters: link length (a), link torsion angle (α), link offset (d), and joint angle (θ).

[0029] S2: Construct the homogeneous transformation matrix T from the base to the end effector using the DH parameters.

[0030]

[0031] T=A1·A2·...·A n

[0032] S3: Given the target position and posture, the joint angles are obtained by inverse kinematics.

[0033] The robot arm path planning system based on reinforcement learning is characterized by transmitting model information to the lower computer, i.e., the robot arm; collecting obstacle space information of the robot arm's motion environment, inputting it into the trained robot arm motion path model, performing real-time path adjustment, and completing the work. The specific steps include:

[0034] S1: After training, save the above deep reinforcement learning model in ONNX format, use the TensorRT inference engine to infer the ONNX format model and convert it into a model that can be deployed on an ARM architecture embedded processor;

[0035] S2: Import the processed model into the robot arm through serial communication;

[0036] S3: Using the ARM architecture multi-core processor, the reasoning task is split into multiple parallel executable tasks and assigned to different cores;

[0037] S4: Collect obstacle space information of the robot's motion environment and input it into the trained robot's motion path model;

[0038] S5: According to the environmental information, select the relatively optimal path in the current state in the trained model;

[0039] S6: Update environmental information in real time and adjust the path until the work is completed.

[0040] The path is recorded in the memory so that it can be transmitted back to the host computer for further training.

[0041] The robot arm path planning system based on reinforcement learning is characterized by the multi-core architecture of the ARM main controller. This design enables the main controller to handle multiple tasks at the same time, improving computing efficiency and response speed. Different cores can be specialized to handle specific workloads, optimize performance, and reduce power consumption. This feature enables the ARM main controller to provide flexible and powerful computing capabilities in a wide range of applications from mobile devices to high-performance computing.

[0042] The robotic arm path planning system based on reinforcement learning is characterized by significantly improving computing efficiency and processing power by decomposing large-scale computing tasks into multiple small tasks, organizing data into vector form, and allocating them to multiple ARM processor cores for parallel processing.

[0043] The robot path planning system based on reinforcement learning is characterized by: first, the model needs to be optimized to ensure that its size and computing requirements match the resources of the controller. Then, a cross-compilation tool chain is used to convert the model into a format that the controller can understand, and ensure that all dependencies and libraries are installed on the controller. Next, the model and related code are uploaded to the controller and configured to ensure that the model can receive data input from sensors and output control commands. Finally, actual testing and fine-tuning are carried out to verify the performance of the model in a real environment, and parameters are adjusted as needed to achieve the best control effect. This process ensures that the reinforcement learning model can run efficiently on the controller to achieve autonomous decision-making and intelligent control.

[0044] The robot arm path planning system based on reinforcement learning is characterized by: the stored path data provides new training samples for reinforcement learning, and the data can be transmitted to the host computer for further analysis and training. By continuously optimizing the model, the robot arm gradually improves its movement ability. At the same time, through the analysis of historical paths, the system can better adapt to changing environments and task requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to make the embodiments of the present invention or the prior art solutions easier to understand, the following briefly introduces the drawings used in the embodiments or the prior art descriptions. Obviously, these drawings only show several examples of the present invention, and for those skilled in the art, they can easily generate other drawings based on these drawings without performing creative work.

[0046] Figure 1 A robot arm path planning system based on reinforcement learning according to an embodiment of the present invention

[0047] Figure 2 This is a specific step diagram of a robotic arm path planning system based on reinforcement learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention discloses a robot arm path planning system based on reinforcement learning, such as Figure 1 As shown, it includes: applying the DQN model in the host computer to build a robot arm motion path model based on reinforcement learning; using the generated model to train the robot arm to achieve precise point-to-point movement; transmitting the model information to the robot arm; collecting the obstacle space information of the robot arm's motion environment, inputting it into the trained robot arm motion path model, performing real-time path adjustment, and completing the work.

[0049] In a specific implementation, a robot arm path planning system based on reinforcement learning, in a host computer, trains the robot arm to achieve accurate point-to-point movement in an uncertain environment with random obstacles by applying a deep Q network (DQN) model, and generates a robot arm motion path model based on reinforcement learning. The specific steps are:

[0050] First, build the kinematic solution model and motion environment of the robot arm, including:

[0051] Define DH parameters. For each joint, define four DH parameters: link length (a), link torsion angle (α), link offset (d), and joint angle (θ).

[0052] Plan the robot's motion path, including:

[0053] Build a deep reinforcement learning framework, establish a DQN model, and construct a reward function; the reward function consists of distance reward, speed reward, and collision penalty and their respective weights; the distance reward is fed back to the robot to quickly reach the target, the collision penalty is fed back to the robot to avoid collisions with random obstacles, and the speed reward is fed back to the smooth movement of the robot. The three functions have different weights. The weight ratio between the three is adjusted according to the actual needs of the robot to maximize the robot's obstacle avoidance and precise point-to-point movement effects; use the Q-learning algorithm to build a state-action Q table, which maps the robot's action and state to the Q value. Construct a Q network and a target network. Initially, the Q network is exactly the same as the target network. After training, the Q network is the agent with the best state-action value, and the target network remains stable. Construct a loss function and minimize the loss between the Q value predicted by the DQN model and the target Q value.

[0054] The Q value is the current Q network's prediction of the robot's state and action. It is the expected cumulative reward after the robot takes action after receiving instructions from the host computer. The target value Q is the maximum Q value of the robot's next possible action. After the robot executes the instruction, it will receive an immediate reward from the environment to judge the quality of the result of the action, such as whether it will collide with an obstacle or pose a safety threat to the operator.

[0055] The specific steps of making the robot arm move to the target position according to the optimized path, reach the target position and achieve the target posture include:

[0056] Build a robot model, randomly generate a working environment with random obstacles for the robot, and then plan the path to find the best one. Then check whether there is a collision. If there is a collision, re-plan it. If there is no collision, move to the target position and achieve the target posture.

[0057] To avoid collision between the robot arm and obstacles, build a collision detection mechanism:

[0058] The spatial occupation relationship between the robot arm link and the obstacle is simplified into the relative position relationship between a straight line segment and an entity. Based on the idea of ​​spherical bounding box and spatial superposition, the radial maximum spherical bounding box radius of the robot arm link is superimposed on the thickness of the obstacle to simplify the judgment and detection of collision.

[0059] The theoretical current of each joint is calculated using a dynamic model, and the actual current of each joint is monitored and compared with the theoretical value. If the difference between the two exceeds the preset threshold, the joint may encounter external resistance or collision and trigger collision detection.

[0060] The collision detection algorithm is used to calculate the distance from the connecting rod to the center of the obstacle and compare it with the radius of the enclosing sphere to determine whether the robotic arm collides with the obstacle.

[0061] If a collision occurs, the information is fed back to the DQN model, and the DQN model replans the path

[0062] Goal achievement postures include:

[0063] The homogeneous transformation matrix T from the base to the end effector is constructed using the DH parameters.

[0064] After solving, the four parameters of the current link length (a), link torsion angle (α), link offset (d) and joint angle (θ) of the robot arm and the four expected parameters corresponding to the target posture are obtained. The controller controls the rotation of each joint of the robot arm to achieve the target posture.

[0065] Convert the above trained deep reinforcement learning model into ONNX format suitable for export and deployment.

[0066] Use the TensorRT inference engine to infer the ONNX model and convert it into a model that can be deployed on an ARM architecture embedded processor. Inference specifically includes splitting the inference task into multiple parallel executable tasks so that multi-core processors can handle parallel executable tasks; using quantization technology to convert model parameters from 32-bit floating point numbers to 16-bit or 8-bit integers; removing unimportant neurons or connections based on absolute values ​​and weight importance; dynamically adjusting the network structure based on the input, and selectively enabling or disabling certain paths during the inference process. This not only reduces the memory footprint of the model, but also improves the computing speed on ARM processors.

[0067] The converted model is deployed to the embedded processor of ARM architecture, and the memory pool and buffer technology are used to optimize the memory management of the embedded system to ensure that the model and its related data can be accommodated; the multi-core processor of ARM architecture is used to split the reasoning task into multiple parallel executable tasks and assign them to different cores; the SIMD instruction set NEON is used to process multiple data elements in parallel in each clock cycle, and multiple samples are merged into batches. NEON can operate on multiple samples at the same time to improve throughput; through vectorized operations, the matrix operations in the reasoning process are converted into parallel execution, thereby accelerating the calculation.

[0068] First, the model needs to be optimized to ensure that its size and computational requirements match the resources of the controller. Then, use a cross-compilation toolchain to convert the model into a format that the controller can understand and ensure that all dependencies and libraries are installed on the controller. Next, upload the model and related code to the controller and configure it to ensure that the model can receive data input from sensors and output control commands. Finally, perform actual testing and fine-tuning to verify the performance of the model in a real environment and adjust parameters as needed to achieve the best control effect.

[0069] After achieving collision-free point-to-point motion of the robotic arm, the motion path is stored in the system memory, and the data can be transmitted to the host computer for real-time analysis and training. The stored path data provides new training samples for reinforcement learning. By continuously optimizing the model, the robotic arm can gradually improve its motion capabilities. At the same time, through the analysis of historical paths, the system can better adapt to changing environments and task requirements. When performing multiple tasks, the stored path can provide a reference for path planning, reduce the computational burden, and optimize resource usage.

[0070] The beneficial effects of the present invention are:

[0071] Improve work efficiency: Using reinforcement learning, the system can intelligently optimize the robot's moving path, effectively shorten the path length and reduce the movement time, significantly improving the robot's work efficiency. This advantage is particularly evident in environments that require fast response and efficient operation, such as production lines and automated warehouses.

[0072] Enhanced environmental adaptability: The reinforcement learning model gives the robot arm the ability to adapt in complex and dynamic environments, enabling it to identify and avoid obstacles in real time and adjust its path autonomously, thereby enhancing the flexibility and environmental adaptability of the robot arm.

[0073] Easy to continuously improve: The reinforcement learning algorithm enables the robot arm to have the ability to continuously learn and improve, and constantly update the decision-making strategy through interaction with the environment. This self-improvement mechanism enables the system to adapt to new tasks and changing needs. Over time, the intelligence level and operation performance of the robot arm will continue to improve. The specific implementation plan is as follows: The embedded host computer includes AMD or ARM main control chip, two M.2Key interfaces, four USB 3.0 interfaces, RJ45 Ethernet interface, eight-way CAN communication bus, power module, two CSI camera interfaces, DL acceleration module, GPU, HDMI interface, four-way serial port 485 communication bus.

[0074] The detailed description of the present invention is intended to enable those skilled in the relevant art to understand and implement the present invention. Through the description of these embodiments, professional and technical personnel will be able to operate or apply the present invention. Obviously, these embodiments are not exhaustive of all possible variations, and it is obvious to those skilled in the art that these embodiments can be modified in many ways without departing from the core principles and innovative features of the present invention. Therefore, the scope of protection of the present invention is not limited to the specific embodiments described herein, but extends to all implementation methods consistent with the principles and novel features disclosed herein to cover the widest scope of protection.

Claims

1. A robotic arm path planning system based on reinforcement learning, characterized in that: include: The DQN model is applied in the host computer to build a robot arm motion path model based on reinforcement learning; the generated model is used to train the robot arm to achieve precise point-to-point movement; The model information is transmitted to the lower computer, i.e. the robotic arm; the obstacle space information of the robotic arm's motion environment is collected and input into the trained robotic arm motion path model to perform real-time path adjustment to complete the work.

2. The robot arm path planning system based on reinforcement learning according to claim 1 is characterized in that Apply the Deep Q Network (DQN) model to train the robot arm to achieve accurate point-to-point movement in an environment with random obstacles. The specific steps include: S1: Randomly generate spatial relative position information of the robot arm sensor and the robot arm end effector and random obstacles S2: Input information into the DQN model S3: Output each path that can achieve collision-free point-to-point precise movement through the DQN model S4: Select the relatively optimal path.

3. The robot arm path planning system based on reinforcement learning according to claim 1, characterized in that: The reward function of the reinforcement learning DQN model includes: R(s,a,s′)=w1·R distance (s,s′)+w2·R speed (a)+w3·R collision (s′) in: R(s,a,s′) is the total reward function, s is the current state of the robot, a is the action to be performed, and s′ is the next state of the robot. Distance reward function: R distance (s,s′)=d current -d next , d current is the distance from the current point to the target point, d next is the distance from the next state to the target point; Speed ​​reward function: R speed (a)=-||a|| 2 , ||a|| is the amplitude of the action, and the negative sign minimizes the amplitude of the action, that is, encourages smaller joint speeds to avoid excessive movements that cause the robot to be unstable or collide; collision penalty function: Among them, C is a large negative number, which indicates the penalty for collision. ω1, ω2, and ω3 represent the weights of each function according to the amount of motion.

4. The robot arm path planning system based on reinforcement learning according to claim 1, characterized in that The state-action Q table is constructed through the Q-learning algorithm, and the Q table maps the robot arm action and state to Q value.

5. The robot arm path planning system based on reinforcement learning according to claim 1, characterized in that Construct the Q network and the target network, where the Q network is exactly the same as the target network at the beginning. After training, the Q network is the agent with the best state-action value, and the target network remains stable.

6. The robot arm path planning system based on reinforcement learning according to claim 1, characterized in that Minimize the loss between the Q value predicted by the DQN model and the target Q value: Loss function: in: L(θ) represents the loss function, θ is the network parameter. N is the size of the batch sample; is the current network state s i and action a i Q value prediction of y i is the Q value of the i-th target sample, and its calculation formula is as follows: y i =r i +γmax a′ Q(s i+1 ,a′;θ - ) γ is the discount factor, Q(s i+1 ,a′;θ - ) The target network has the next state s i+1 The maximum Q value of all possible actions a′.

7. According to claim 2, it is characterized in that Make the robot arm move to the target position according to the optimized path: S1: Define DH parameters. For each joint, define four DH parameters: link length (a), link torsion angle (α), link offset (d), and joint angle (θ). S2: Use DH parameters to construct the homogeneous transformation matrix T from the base to the end effector; T=A1·A2·...·A n S3: Given the target position and posture, the joint angles are obtained by inverse kinematics.

8. The robot arm path planning system based on reinforcement learning according to claim 1, characterized in that The model information is transmitted to the lower computer, i.e. the robotic arm; the obstacle space information of the robotic arm's motion environment is collected and input into the trained robotic arm motion path model to make real-time path adjustments to complete the work. The specific steps include: S1: After training, save the above deep reinforcement learning model in ONNX format, use the TensorRT inference engine to infer the ONNX format model and convert it into a model that can be deployed on an ARM architecture embedded processor; S2: Import the processed model into the robot arm through serial communication; S3: Using the ARM architecture multi-core processor, the reasoning task is split into multiple parallel executable tasks and assigned to different cores; S4: Collect obstacle space information of the robot's motion environment and input it into the trained robot's motion path model; S5: According to the environmental information, select the relatively optimal path in the current state in the trained model; S6: Update environmental information in real time and adjust the path until the work is completed; The path is recorded in the memory so that it can be transmitted back to the host computer for further training.

9. The robot arm path planning system based on reinforcement learning according to claim 9 is characterized in that The multi-core architecture of the ARM main controller enables the main controller to process multiple tasks simultaneously, improve computing efficiency and response speed, and different cores can be specialized to process specific workloads, optimize performance, and reduce power consumption. This feature enables the ARM main controller to provide flexible and powerful computing capabilities in a wide range of applications from mobile devices to high-performance computing.

10. The robot arm path planning system based on reinforcement learning according to claim 9, characterized in that By breaking down large-scale computing tasks into multiple small tasks and organizing data into vector form, and assigning them to multiple ARM processor cores for parallel processing, computing efficiency and processing power are significantly improved.

11. The robot arm path planning system based on reinforcement learning according to claim 9, characterized in that: First, the model needs to be optimized to ensure that its size and computing requirements match the resources of the controller. Then, a cross-compilation toolchain is used to convert the model into a format that the controller can understand and ensure that all dependencies and libraries are installed on the controller. Next, the model and related code are uploaded to the controller and configured to ensure that the model can receive data input from sensors and output control commands. Finally, actual testing and fine-tuning are performed to verify the performance of the model in a real environment and adjust parameters as needed to achieve the best control effect. This process ensures that the reinforcement learning model can run efficiently on the controller to achieve autonomous decision-making and intelligent control.

12. The robot arm path planning system based on reinforcement learning according to claim 9, characterized in that: The stored path data provides new training samples for reinforcement learning. The data can be transmitted to the host computer for further analysis and training. By continuously optimizing the model, the robot arm can gradually improve its movement ability. At the same time, through the analysis of historical paths, the system can better adapt to changing environments and task requirements.

Images