Collision-free path planning method and device for picking mechanical arm, terminal and medium

Abstract

The invention relates to the technical field of mechanical arm path planning, and particularly discloses a picking mechanical arm collision-free path planning method and device, a terminal and a readable storage medium. The method comprises the steps that a three-dimensional scene model of the mechanical arm and the working environment is established; constructing a collision detection model containing obstacle boundary description, motion constraint and a collision judgment rule; modeling action control into a decision model consisting of a state, an action, a state transition and a reward and punishment mechanism; and training is carried out by adopting an Informed-Q algorithm, and a path planning strategy is generated. According to the method, dynamic obstacle avoidance and efficient path planning of the mechanical arm in a complex orchard environment can be achieved by combining environment modeling and reinforcement learning, the intelligence, robustness and real-time performance of path planning are remarkably improved, the problems that a traditional algorithm is complex in calculation, slow in convergence and poor in adaptability in high-degree-of-freedom mechanical arm path planning are solved, and the method is suitable for large-scale popularization and application. The intelligent picking robot system is suitable for changeable orchard scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of robotic arm path planning, and particularly relates to a method, device, terminal and medium for collision-free path planning of a picking robotic arm. Background Art

[0002] With the development of modern agriculture, automated and intelligent picking technologies have gradually become the key means to improve fruit picking efficiency and reduce labor costs. As a major fruit variety in China, apples have a concentrated picking period and a high labor intensity, and their dependence on human resources in the picking process is particularly significant. In recent years, picking robots, as intelligent equipment integrating visual recognition, path planning and servo control, have become a hot research direction in agricultural engineering.

[0003] A picking robot usually consists of a robotic arm mounted on a mobile chassis, with flexible operation capabilities and a wide working range, and can adapt to multi-target picking tasks in complex orchard environments. Among them, path planning, as the core link to achieve automatic picking, directly affects the efficiency and safety of the picking operation. In a complex orchard environment, fruits are mostly distributed inside the lush canopy of branches and leaves, with a large number of irregular obstacles (such as branches, fruit supports, etc.) around them. At the same time, the distribution height and density of fruits in space also show significant differences. Therefore, to avoid collisions between the robotic arm and the environment during picking, it is necessary to perform efficient and robust collision-free path planning for it.

[0004] Existing path planning methods can be mainly divided into two categories: traditional heuristic algorithms and intelligent algorithms based on machine learning. Traditional algorithms such as A*, artificial potential field method (APF), and rapidly exploring random tree (RRT), etc., although showing good planning capabilities in two-dimensional environments, often have limitations such as large computational amounts, slow path convergence, and serious local optimum problems when dealing with the motion planning of high-degree-of-freedom robotic arms. For example, the QH-A* algorithm improves the search efficiency by constructing a hierarchical map, but it relies on offline preprocessing and still faces the problem of increasing search complexity in high-dimensional spaces; the artificial potential field method has real-time performance, but often leads to path failure due to ignoring link collisions when avoiding obstacles at the end; the RRT algorithm is more general in high-degree-of-freedom environments, but has a long planning time and poor path smoothness, making it difficult to meet the real-time requirements.

[0005] In recent years, due to its advantages in dynamic obstacle avoidance and policy self-learning, deep reinforcement learning (DRL) has been widely applied to the field of robotic arm path planning. Path planning models based on algorithms such as Q-learning and DDPG can learn action strategies in complex environments through continuous interaction. However, existing methods mostly have the following problems: First, the state modeling is incomplete, and the dynamic information of obstacles in the environment fails to be effectively integrated, resulting in weak generalization ability; second, the training convergence speed is slow. Especially for the cyclic reinforcement learning network structure, although it can improve the policy memory ability, it significantly increases the training complexity and path planning delay; third, most reinforcement learning methods lack effective constraints in state space exploration, leading to low sample efficiency and unstable training. Summary of the Invention

[0006] The present invention addresses the problems in the prior art and provides a collision-free path planning method, device, terminal, and medium for a picking robotic arm, solving the problems in the prior art that the robotic arm path planning scheme has a slow training convergence speed and lacks effective constraints, resulting in low sample efficiency and unstable training.

[0007] The technical solution adopted by the present invention is as follows:

[0008] In a first aspect, the present application provides a collision-free path planning method for a picking robotic arm, including the following steps:

[0009] Step S1: Establish a three-dimensional scene model of the robotic arm and its working environment, and model the elements that can collide, non-picking targets, and picking targets in the scene;

[0010] Step S2: Based on the established three-dimensional model, construct a collision detection model for the robotic arm during operation. The collision detection model includes the boundary description of obstacles, the motion constraints of the robotic arm, and the collision determination rules;

[0011] Step S3: Model the action control process of the robotic arm as a decision-making model based on state, action, state transition, and reward and punishment mechanisms;

[0012] Step S4: Use the reinforcement learning Informed-Q algorithm to train the decision-making model of the robotic arm to obtain a path planning model.

[0013] Preferably, in step S1, the CBB envelope box method is used to construct the robotic arm link model and the element model that can collide; the SBB envelope box method is used to construct the non-picking target model.

[0014] Preferably, the motion constraints of the robotic arm in the collision detection model include the physical motion boundaries, motion speed limits, and acceleration limits within the degrees of freedom of each joint of the robotic arm;

[0015] The collision determination rules in the collision detection model include a combined strategy of geometric overlap detection, minimum distance threshold determination, and predicted collision within a time step.

[0016] Preferably, the physical motion boundary is defined by the structural parameters of the robotic arm and the joint angle limit parameters;

[0017] The motion speed limit and acceleration limit are set based on the control performance of each joint motor and the actual load conditions respectively.

[0018] Preferably, in step S3, the state includes the positions, postures, target positions of each joint of the robotic arm, and environmental obstacle information;

[0019] The actions include the driving modes and displacement combinations of each joint;

[0020] The state transition relationship is used to describe the possible next state after executing a specific action in the current state, and a corresponding state transition probability model is constructed;

[0021] The reward and punishment mechanism is used to give a numerical feedback on the action result, and is used to train the policy model to realize the learning of the path planning control strategy.

[0022] Preferably, in step S3, the decision-making model adopts a Markov decision process, and the specific state transition probability function and reward function are:

[0023]

[0024] In the formula, P(s'|s,a) represents the probability that the agent transfers from state s to state s' by executing action a, and r(s,a) represents the expected reward that the agent can obtain when transferring from state s to state s' by executing action a. S t 、S t+1 represent the sets of the current state and the next state respectively, A t represents the set of actions, and R t+1 represents the set of rewards;

[0025] The agent interacts with the environment according to the policy π. The policy π represents a mapping relationship between states and actions. To evaluate the advantages and disadvantages of different policies π, reinforcement learning defines a value function and an action value function:

[0026]

[0027] In the formula, G t is the return at time step t, which is the income after discounting the rewards in the future;

[0028] The update formula of the Q-value function is:

[0029] Q(s,a) = Q(s,a) + α(R t + γ max Q(s',a') - Q(s,a))

[0030] Where α is the learning factor, γ is the discount factor, and max Q(s',a') represents the maximum expected return for selecting an action in the next state;

[0031] The Q-learning algorithm adopts the ε-greedy strategy, and the expression is:

[0032]

[0033] When the agent reaches the target point, the reward value is set to a positive number. When the agent collides with an obstacle, the reward value is set to a negative number. In other cases, the reward value is set to zero; the expression is:

[0034]

[0035] Where s goal , s obs represent the target state and the obstacle state respectively, and R0 represents a positive value.

[0036] Preferably, in step S4, Informed-Q restricts the state space of each episode within an ellipse according to the path length of the previous episode;

[0037] To obtain the state s' ellipes in the ellipse, first obtain the state s ellipes of the standard ellipse. The standard ellipse equation is: ellipes The standard ellipse equation is:

[0038]

[0039] s ellipes = (x, y) is the state coordinate in the standard ellipse, and a and b are respectively half of the major axis length and the minor axis length of the ellipse, that is After that, perform coordinate transformation on s ellipes to obtain s' ellipes . The coordinate transformation equation is: ellipes The coordinate transformation equation is:

[0040] s' ellipes = Rs ellipes + T

[0041] Where R is the rotation matrix and T is the translation matrix, and their expressions are:

[0042]

[0043] Where θ represents the angle between the line connecting the starting state and the target state and the x-axis, (x center , y center) is the midpoint coordinate of the major axis of the ellipse;

[0044] The reward function for sparse rewards is:

[0045]

[0046] The expression of the dynamic ε-greedy strategy is:

[0047]

[0048] In the formula, ε start and ε end respectively represent the greedy coefficients at the beginning and end. Episode represents the current round number, and e total represents the total number of rounds.

[0049] In a second aspect, the present application provides a collision-free path planning system for a picking robotic arm, including:

[0050] A three-dimensional modeling module for establishing a three-dimensional scene model of the robotic arm and its working environment, and modeling the elements that can collide, non-picking targets, and picking targets in the scene;

[0051] A collision detection module for constructing a collision detection model based on the established three-dimensional model. The collision detection model includes the boundary description of obstacles, the motion constraints of the robotic arm, and the collision determination rules. The boundary description simplifies the spatial boundary through a bounding box. The motion constraints include the degrees of freedom of the robotic arm joints, speed limits, and acceleration limits. The collision determination rules include geometric overlap detection, minimum distance determination, and time step prediction mechanism;

[0052] A decision-making modeling module for modeling the robotic arm motion control process as a Markov decision model based on state, action, state transition, and reward and punishment mechanisms, and constructing a state transition probability model and a reward function by the module;

[0053] A path training module for training the decision model based on the Informed-Q algorithm to obtain a path planning model and realizing the update of the greedy coefficient of the dynamic -greedy strategy;

[0054] A control instruction output module for converting the action sequence generated by the trained path planning model into a robotic arm drive signal and sending it to the actuator to realize path tracking control.

[0055] In a third aspect, the present application provides a terminal, including:

[0056] A memory for storing a collision-free path planning program for a picking robotic arm;

[0057] A processor, configured to execute the steps of the collision-free path planning method for the picking robotic arm as described in the first aspect when implementing the collision-free path planning system for the picking robotic arm.

[0058] In a fourth aspect, the present application provides a computer-readable storage medium storing computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes a collision-free path planning method for a picking robotic arm as described in the first aspect.

[0059] As can be seen from the above technical solutions, the present application has the following advantages:

[0060] 1. By constructing a three-dimensional model of the working environment of the picking robotic arm, clarifying the picking targets, non-picking targets, and environmental elements that may cause collisions, and combining the Informed-Q algorithm of reinforcement learning to establish a path planning model, autonomous obstacle avoidance and efficient path planning of the robotic arm in a complex orchard environment can be achieved, significantly improving the robustness and real-time performance of path generation. This method can effectively overcome the problems of heavy computational burden and discontinuous planned paths of traditional A* and artificial potential field methods under high degrees of freedom, and also solve the technical problems of incomplete state representation, weak dynamic obstacle avoidance ability, and low training efficiency in existing deep reinforcement learning path planning, providing a practical solution for the picking robotic arm to achieve efficient and accurate collision-free operation.

[0061] 2. The CBB method is used to model the robotic arm linkages and collidable objects, and the SBB method is combined to model the non-picking targets, reducing the computational complexity of collision detection while ensuring the collision detection accuracy. By selecting different types of bounding boxes according to the geometric characteristics of different structural elements, the robustness and real-time performance of the collision detection model are further improved, which helps to improve the modeling efficiency and obstacle avoidance accuracy in the robotic arm path planning.

[0062] 3. By systematically modeling the physical constraint conditions (such as joint degree-of-freedom boundaries, speed, and acceleration) of the robotic arm in the collision detection model, it can be ensured that the path planning results meet the dynamic performance requirements of the robotic arm control system, avoiding failures caused by violating the hardware limits during the path execution stage. At the same time, by integrating multi-strategies such as geometric overlap detection, minimum distance threshold, and collision prediction within a time step, the accuracy and forward-looking of collision determination are significantly enhanced, providing a more reliable obstacle avoidance judgment basis for path planning.

[0063] 4. By setting physical boundary conditions based on the structural parameters of the robotic arm and the joint angle limits, the path planning is more in line with the actual operation constraints; by combining the motor performance and load conditions to set the speed and acceleration boundaries, the executability and control stability of the path are guaranteed, preventing action failures caused by the path exceeding the capabilities of the actuator, which helps to improve the consistency and safety between path planning and actual picking operations.

[0064] 5. By defining the state as a combination of the robotic arm joint state, target position, and obstacle information, and the action as the driving mode and displacement of each joint, a complete state-action-state transition model is constructed, which helps to improve the perception ability of the learning model for the environment and the pertinence of strategy selection. With the help of the reward and punishment mechanism for strategy training, the path planning model can gradually approach the optimal strategy in the high-dimensional state space, improve the path generation efficiency, and reduce the collision rate.

[0065] 6. By modeling the robotic arm path planning as a Markov decision process and clarifying the state transition function and reward function, it helps the reinforcement learning model to converge efficiently during training and have good policy generalization ability. Combining the value function, action value function, and the ε-greedy policy of the Q-learning algorithm endows the model with the ability to balance dynamic exploration and exploitation, improves the ability to learn effective obstacle avoidance paths in a random obstacle environment, and effectively improves the success rate of the planned path and the adaptability of the strategy.

[0066] 7. By introducing an elliptical state space sampling strategy with path length as a constraint in path planning through the Informed-Q algorithm, the search space is significantly reduced, and the sample acquisition efficiency and training speed are improved. Combining the dynamic greedy coefficient adjustment mechanism realizes the adaptive balance between exploration and exploitation during the training process of the model, and improves the quality and convergence stability of path planning. The introduction of the sparse reward strategy makes the training focus on key state changes, improves the discrimination of rewards, and the directionality of strategy learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] To more clearly illustrate the technical solutions of the present application, the drawings required for description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0068] Figure 1 Schematic diagram of the process of the collision-free path planning method for the picking robotic arm shown in some embodiments;

[0069] Figure 2 Schematic diagram of the simplified model of the robotic arm shown in some embodiments;

[0070] Figure 3 Schematic diagram of the simplified model of the fruit tree branches shown in some embodiments;

[0071] Figure 4 Schematic diagram of the simplified model of the non-target fruits shown in some embodiments;

[0072] Figure 5Schematic diagram of a collision detection model shown in some embodiments;

[0073] Figure 6 Schematic diagram of the free configuration space of a robotic arm shown in some embodiments;

[0074] Figure 7 Process diagram of reinforcement learning shown in some embodiments;

[0075] Figure 8 Process diagram of the classic Q - learning episode training shown in some embodiments;

[0076] Figure 9 Schematic diagram of the exploration space of the Informed - Q algorithm shown in some embodiments;

[0077] Figure 10 Process diagram of the improved Q - learning algorithm training shown in some embodiments;

[0078] Figure 11 Process diagram of the inverse curriculum learning shown in some embodiments. Detailed implementation manners

[0079] In order to make the application purpose, features, and advantages of the present application more obvious and understandable, the technical solutions protected by the present application will be clearly and completely described below by using specific embodiments and the accompanying drawings. Obviously, the embodiments described below are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in this patent, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection of this patent.

[0080] With the development of modern agriculture, automated and intelligent picking technologies have gradually become the key means to improve fruit picking efficiency and reduce labor costs. As a major fruit variety in China, apples have a concentrated picking period and a high labor intensity, and the dependence on human resources in the picking link is particularly significant. In recent years, picking robots, as intelligent equipment integrating visual recognition, path planning, and servo control, have become a hot research direction in agricultural engineering.

[0081] Harvesting robots usually consist of robotic arms mounted on mobile chassis, with flexible operation capabilities and a wide range of working areas, capable of adapting to multi-target harvesting tasks in complex orchard environments. Among them, path planning, as the core link to achieve automatic harvesting, directly affects the efficiency and safety of the harvesting operation. In a complex orchard environment, fruits are mostly distributed inside the lush tree canopies, with a large number of irregular obstacles (such as branches, fruit supports, etc.) around. At the same time, the distribution height and density of fruits in space also show significant differences. Therefore, to avoid collisions between the robotic arm and the environment during harvesting, it is necessary to perform efficient and robust collision-free path planning for it.

[0082] Existing path planning methods can be mainly divided into two categories: traditional heuristic algorithms and intelligent algorithms based on machine learning. Traditional algorithms such as A*, artificial potential field method (APF), and rapidly-exploring random tree (RRT), etc., although showing good planning capabilities in two-dimensional environments, often have limitations such as large computational amounts, slow path convergence, and serious local optimum problems when dealing with the motion planning of robotic arms with high degrees of freedom. For example, the QH-A* algorithm improves the search efficiency by constructing a hierarchical map, but it relies on offline preprocessing and still faces the problem of increasing search complexity in high-dimensional spaces; the artificial potential field method has real-time performance, but often leads to path failure due to ignoring link collisions when avoiding obstacles at the end; the RRT algorithm is more general in high-degree-of-freedom environments, but has a long planning time and poor path smoothness, making it difficult to meet real-time requirements.

[0083] In recent years, deep reinforcement learning (DRL) has been widely applied to the field of robotic arm path planning due to its advantages in dynamic obstacle avoidance and policy self-learning. Path planning models based on algorithms such as Q-learning and DDPG can learn action strategies in complex environments through continuous interaction. However, existing methods mostly have the following problems: First, the state modeling is incomplete, and the dynamic information of obstacles in the environment is not effectively integrated, resulting in weak generalization ability; second, the training convergence speed is slow. Especially for recurrent reinforcement learning network structures, although they can improve the policy memory ability, they significantly increase the training complexity and path planning delay; third, most reinforcement learning methods lack effective constraints in state space exploration, resulting in low sample efficiency and unstable training.

[0084] The present invention addresses the problems in the prior art and provides a collision-free path planning method, device, terminal, and medium for a harvesting robotic arm, solving the problems in the prior art that the robotic arm path planning scheme has a slow training convergence speed, lacks effective constraints, resulting in low sample efficiency and unstable training.

[0085] This application provides a collision-free path planning method for a harvesting robotic arm, including the following steps:

[0086] Step S1: Establish a three-dimensional scene model of the robotic arm and its working environment, and model the elements that can collide, non-picking targets, and picking targets in the scene;

[0087] Step S2: Based on the established three-dimensional model, construct a collision detection model for the robotic arm during the operation process. The collision detection model includes the boundary description of the obstacle, the motion constraints of the robotic arm, and the collision determination rules;

[0088] Step S3: Model the motion control process of the robotic arm as a decision-making model based on state, action, state transition, and reward and punishment mechanism;

[0089] Step S4: Use the reinforcement learning Informed-Q algorithm to train the decision-making model of the robotic arm to obtain a path planning model.

[0090] In some embodiments, the geometric body models of the robotic arm and fruit tree branches are constructed using the CBB envelope box method. As Figure 2 shown, according to the actual situation of the robotic arm in the picking scene, link 1 will not collide with obstacles, so only the models of links 2-6 need to be constructed. In addition, the relative positions of links 4-6 change little during the movement of the robotic arm, and a cylinder is used to replace links 4-6, thereby further simplifying the model;

[0091] For facilitating mechanized fruit picking, the fruit tree type adopts the hedge cultivation mode. The simplified fruit tree model is as Figure 3 shown. The geometric body model of non-target fruits is constructed using the SBB envelope box method, as Figure 4 shown.

[0092] In some embodiments, according to the models of the robotic arm and obstacles constructed above, the collision detection situations are divided into two types, namely the collision detection between the robotic arm and fruit tree branches and the collision detection between the robotic arm and non-target fruits. As Figure 5 shown, it can be equivalent to the collision detection between cylinders and the collision detection between cylinders and spheres;

[0093] r1, r2, and r3 respectively represent the radii of the robotic arm link model, fruit tree branch model, and non-target fruit model. AB represents the axis of the link model, CD represents the axis of the fruit tree branch model, and O represents the center of the sphere of the non-target fruit model. D 12 represents the distance between AB and CD, and D 13 represents the distance between AB and O. When D 12 > r1 + r2, there is no collision between the robotic arm link and the fruit tree branch; when D 13 > r1 + r3, there is no collision between the robotic arm link and the non-target fruit.

[0094] In some embodiments, a robotic arm collision detection algorithm is used to obtain the Since the 4-6 joints of the robotic arm have little influence on the position of the robotic arm link model, collision detection is mainly performed on the states of the first three joints of the robotic arm, as shown in As Figure 6 shown.

[0095] Reinforcement learning is a machine learning method, and its overall architecture is that an agent starts from the current state s t and selects an action according to a policy to act in an environment with Markov properties. At the same time, the environment feedbacks a reward to the agent and enters the next state s t+1 , as Figure 7 shown. The definition of the Markov property is: when the conditional probability distribution of the future state of a stochastic process depends only on the current state given the current state and all past states, then this stochastic process is said to have the Markov property. The basis of reinforcement learning is the Markov decision process, and the specific state transition probability function and reward function are:

[0096]

[0097] In the formula, P(s'|s,a) represents the probability that the agent transfers from state s to state s' by executing action a, and r(s,a) represents the expected reward value that the agent can obtain when transferring from state s to state s' by executing action a. S t , S t+1 respectively represent the sets of the current state and the next state, A t represents the set of actions, and R t+1 represents the set of rewards.

[0098] The agent interacts with the environment according to the policy π, and the policy π represents a mapping relationship between the state and the action. To evaluate the advantages and disadvantages of different policies π, reinforcement learning defines the value function and the action value function:

[0099]

[0100] In the formula, G t is the return at time step t, which is the income after discounting the rewards in the future time. The purpose of reinforcement learning is to continuously improve the policy π during the learning process and find an optimal policy to maximize the cumulative return.

[0101] The path planning process can be regarded as a standard Markov decision process and solved using the Q-learning method in reinforcement learning. The Q-learning method belongs to the value function method and evaluates the value of an agent executing an action a in state s by learning a Q-value function, that is, the action-value function in reinforcement learning. The core of Q-learning is to learn an optimal Q-value function so that the agent selects the action that can maximize the reward in each state. The update formula of the Q-value function is:

[0102] Q(s,a) = Q(s,a) + α(R t + γmaxQ(s',a') - Q(s,a))

[0103] In the formula, α is the learning factor, γ is the discount factor, and maxQ(s',a') represents the maximum expected reward for selecting an action in the next state.

[0104] There are generally two methods for action selection in the classical Q-learning algorithm, exploration and exploitation. Exploration means randomly selecting an action a from the action set A t . Exploitation means selecting the action with the largest Q-value in the current state, and this method is also called the greedy policy. The classical Q-learning algorithm combines the two methods for action selection, that is, the ε-greedy policy, and the expression is:

[0105]

[0106] After executing the action, the environment will feedback a reward to the agent, and the setting of the reward function becomes crucial, which reflects the quality of the executed action. In path planning, the setting of the reward function in the classical Q-learning algorithm mainly considers whether the agent reaches the target point and whether it collides with obstacles. When the agent reaches the target point, the reward value is set to a positive number, and when the agent collides with an obstacle, the reward value is set to a negative number, and in other cases, the reward value is set to zero. The expression is:

[0107]

[0108] In the formula, s goal 、s obs represent the target state and the obstacle state respectively, and R0 represents a positive value.

[0109] Finally, update the Q value, and the agent has completed a Markov decision process. Repeat the Markov decision process until the termination condition is met (such as reaching the target state or a certain number of steps), and this process is called an episode, such asFigure 8 As shown, the Q-value table is continuously updated through a certain number of training rounds, making the Q-value gradually increase until convergence, and these training rounds constitute the entire training process of the agent.

[0110] There are also some problems in the actual application process of the classical Q-learning method, such as slow convergence speed, sparse rewards, local optimality, and non-smooth paths. This paper improves the classical Q-learning algorithm for the above problems and obtains the Informed-Q algorithm.

[0111] For the classical Q-learning algorithm, the exploration time is long and the convergence speed is slow. The key lies in the fact that the exploration space is too large, resulting in many useless explorations by the agent. And when each round ends, the agent needs to re-select actions from the entire state space, which leads to too many training steps for the agent and reduces the learning efficiency.

[0112] The Informed-Q algorithm improves the exploration space on the basis of the classical Q-learning algorithm. The state space of each round is restricted within an ellipse according to the path length of the previous round, and the agent is not allowed to exceed the ellipse when selecting actions. As Figure 9 shown, the starting state s start and the target state s goal are used as the two foci of the ellipse; c best is the length of the major axis of the ellipse and is also the path length obtained in the previous iteration; is the length of the minor axis of the ellipse; c min is the Euclidean distance between the starting state and the target state.

[0113] Based on the property of the ellipse: the sum of the distances from any point on the ellipse to the two foci of the ellipse is equal to the length of the major axis c best , and the sum of the distances from any point inside the ellipse to the two foci is less than the sum of the distances from the points on the ellipse to the two foci of the ellipse. Therefore, when exploring inside the ellipse, the obtained path length will be better than the path length c best of the previous round.

[0114] According to the definition of the ellipse parameters, when the starting state and the target state are determined, the Euclidean distance c min between the starting state and the target state remains unchanged. As the number of iterations increases, the path length c best gradually decreases, and the ellipse also becomes smaller. Therefore, the sampling space in the training process of the Informed-Q algorithm continuously shrinks. Based on the action selection strategy of the Informed-Q algorithm, it avoids too many useless explorations, improves the learning efficiency of the algorithm, and the effect of improving the path planning efficiency in the high-dimensional space is particularly obvious.

[0115] To obtain the state s' in the ellipse ellipes , the state s of the standard ellipse can be obtained first ellipes . The standard ellipse equation is:

[0116]

[0117] where s ellipes =(x,y) is the state coordinate in the standard ellipse, and a and b are respectively half of the lengths of the major axis and minor axis of the ellipse, that is After that, a series of coordinate transformations are performed on s ellipes to obtain s' ellipes . The coordinate transformation equation is:

[0118] s' ellipes =Rs ellipes +T

[0119] where R is the rotation matrix and T is the translation matrix, and their expressions are:

[0120]

[0121] where θ represents the angle between the line connecting the starting state and the target state and the x-axis, and (x center ,y center ) is the midpoint coordinate of the major axis of the ellipse. The training process of the Informed-Q algorithm using the ellipse to limit the exploration space is as Figure 10 shown

[0122] Sparse reward means that the agent only obtains non-zero rewards under specific actions during the training process, while the rewards obtained under most actions are zero, which leads to low learning efficiency of the agent and even inability to converge to the optimal Q-value function. In this paper, the sparse reward problem is effectively solved by improving the reward function and the reverse curriculum learning method. The improved reward function is:

[0123]

[0124] Reverse curriculum learning is a reinforcement learning method for sparse rewards and complex tasks. Its core idea is to start from near the target state and gradually expand towards the initial state, thereby helping the agent learn more efficiently. This method is particularly suitable for scenarios where the task goal is clear but the exploration space is large and the rewards are sparse. As Figure 11 shown Figure 11For the reverse curriculum learning process, the agent starts from the target state s9, selects an action to reach the next state s8, and then obtains the reward in the way of reaching state s9 from state s8 in the classical Q-learning method, and calculates the Q value. This is a Markov decision process of the Q-learning method using reverse curriculum learning. Repeat the above reverse curriculum learning process until reaching the initial state s1.

[0125] The ε-greedy policy action selection method has problems of over-exploitation and over-exploration. When the value of ε is too small, the agent tends to explore, that is, randomly selects actions to explore the new environment to obtain greater rewards, but it will lead to a longer learning time for the agent and a slower convergence rate of the Q value. When the value of ε is too large, the agent is more inclined to exploit, directly selects the action with the currently known larger Q value to reduce the learning time of the agent and improve the convergence rate of the Q value, but it will cause the Q value to fall into a local optimum. In this paper, to balance the problems of over-exploitation and over-exploration, the ε-greedy policy is improved to a dynamic ε-greedy policy, and the expression is:

[0126]

[0127] In the formula, ε start and ε end respectively represent the greedy coefficients at the beginning and end, episode represents the current round number, and e total represents the total number of rounds. Since the initial Q value has no reference significance for the agent's action selection, ε start can be set to a value close to zero, so that the agent fully explores the new environment in the early stage and optimizes the Q value. As the number of training rounds increases, the Q value gradually converges, and its guiding role for the agent to select actions becomes greater and greater. Therefore, ε end is set to a value close to one, and ε will increase with the increase of epsode. While retaining a certain exploration ability, the agent gradually tends to exploit, so that the Q value converges stably in the later stage of training.

[0128] In some embodiments, the present application provides a collision-free path planning system for a picking manipulator, including:

[0129] A three-dimensional modeling module for establishing a three-dimensional scene model of the manipulator and its working environment, and modeling the elements that can collide, non-picking targets, and picking targets in the scene;

[0130] A collision detection module, configured to construct a collision detection model based on the established three-dimensional model. The collision detection model includes the boundary description of obstacles, the motion constraints of the robotic arm, and the collision determination rules. The boundary description simplifies the spatial boundary through a bounding box. The motion constraints include the degrees of freedom of the robotic arm joints, speed limits, and acceleration limits. The collision determination rules include geometric overlap detection, minimum distance determination, and a time step prediction mechanism;

[0131] A decision-making modeling module, configured to model the robotic arm motion control process as a Markov decision model based on states, actions, state transitions, and a reward and punishment mechanism. The module constructs a state transition probability model and a reward function;

[0132] A path training module, configured to train the decision model based on the Informed-Q algorithm to obtain a path planning model and update the greedy coefficient of the dynamic-greedy strategy;

[0133] A control instruction output module, configured to convert the action sequence generated by the trained path planning model into a robotic arm drive signal and send it to the actuator to achieve path tracking control.

[0134] In some embodiments, the present application provides a terminal, including:

[0135] A memory, configured to store a program for collision-free path planning of the picking robotic arm;

[0136] A processor, configured to execute the steps of the collision-free path planning method for the picking robotic arm as described in the first aspect when executing the collision-free path planning system for the picking robotic arm.

[0137] In some embodiments, the present application provides a computer-readable storage medium. The storage medium stores computer instructions. When a computer reads the computer instructions in the storage medium, the computer executes the collision-free path planning method for the picking robotic arm.

[0138] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only an example and does not constitute a limitation to this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements, and corrections to this specification. Such modifications, improvements, and corrections are proposed in this specification, so such modifications, improvements, and corrections still fall within the spirit and scope of the exemplary embodiments of this specification.

[0139] In addition, unless clearly stated in the claims, the order of the processing elements and sequences, the use of numerical and alphabetical characters, or the use of other names described in this specification are not used to limit the order of the processes and methods of this specification. Although some currently useful embodiments of the invention have been discussed through various examples in the above disclosure, it should be understood that such details are only for illustrative purposes, and the appended claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that conform to the essence and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only through software solutions, such as installing the described system on existing servers or mobile devices.

[0140] Similarly, it should be noted that, in order to simplify the presentation of the disclosure in this specification and thus help the understanding of one or more embodiments of the invention, in the foregoing description of the embodiments of this specification, sometimes multiple features are grouped into one embodiment, drawing, or description thereof. However, this method of disclosure does not mean that the features required by the subject matter of this specification are more than those mentioned in the claims. In fact, the features of the embodiments are fewer than all the features of the individual embodiments disclosed above.

[0141] In some embodiments, numbers are used to describe components and the quantity of attributes. It should be understood that such numbers used in the description of the embodiments are, in some examples, modified by the modifiers "about", "approximate", or "substantially". Unless otherwise stated, "about", "approximate", or "substantially" indicate that the said numbers allow a variation of ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, and such approximate values may vary according to the characteristics required by individual embodiments. In some embodiments, the numerical parameters should consider the specified significant digits and adopt the method of retaining the general number of digits. Although the numerical ranges and parameters used in some embodiments of this specification to confirm the breadth of their scope are approximate values, in specific embodiments, the setting of such numerical values is as precise as possible within the feasible range.

[0142] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of this specification. Therefore, by way of example and not limitation, alternative configurations of the embodiments of this specification can be regarded as consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments clearly introduced and described in this specification.

Claims

1. A collision-free path planning method for a picking robotic arm, characterized in that, It includes the following steps: Step S1: Establish a three-dimensional scene model of the robotic arm and its working environment, and model the elements that can collide, non-picking targets, and picking targets in the scene; Step S2: Based on the established three-dimensional model, construct a collision detection model for the robotic arm during the operation process. The collision detection model includes the boundary description of obstacles, the motion constraints of the robotic arm, and the collision determination rules; Step S3: Model the motion control process of the robotic arm as a decision-making model based on state, action, state transition, and reward and punishment mechanism; Step S4: Use the reinforcement learning Informed-Q algorithm to train the decision-making model of the robotic arm to obtain a path planning model.

2. The collision-free path planning method for the picking robotic arm according to claim 1, wherein, In step S1, the CBB envelope box method is used to construct the robotic arm link model and the element model that can collide; the SBB envelope box method is used to construct the non-picking target model.

3. The collision-free path planning method for the picking robotic arm according to claim 1, wherein The motion constraints of the robotic arm in the collision detection model include the physical motion boundaries within the range of each joint degree of freedom of the robotic arm, the motion speed limit, and the acceleration limit; The collision determination rules in the collision detection model include a combined strategy based on geometric overlap detection, minimum distance threshold determination, and predicted collision within a time step; 4. The collision-free path planning method for the picking robotic arm according to claim 3, wherein The physical motion boundary is defined by the structural parameters of the robotic arm and the joint angle limit parameters; The motion speed limit and the acceleration limit are set based on the control performance of each joint motor and the actual load condition respectively.

5. The collision-free path planning method for the picking robotic arm according to claim 1, wherein In step S3, the state includes the positions, postures, target positions, and environmental obstacle information of each joint of the robotic arm; The action includes the driving mode and displacement combination of each joint; The state transition relationship is used to describe the next state that may be reached after executing a specific action in the current state, and a corresponding state transition probability model is constructed; The reward and punishment mechanism is used to numerically feedback the action result, and is used to train the policy model to realize the learning of the path planning control strategy.

6. The collision-free path planning method for the picking robotic arm according to claim 1, characterized in that In step S3, the decision-making model adopts a Markov decision process. The specific state transition probability function and reward function are: Wherein, P(s'|s,a) represents the probability that the agent transfers from state s to state s' by executing action a, and r(s,a) represents the expected value of the reward that the agent can obtain when transferring from state s to state s' by executing action a, S t and S t+1 respectively represent the sets of the current state and the next state, A t represents the set of actions, and R t+1 represents the set of rewards; The agent interacts with the environment according to the policy π. The policy π represents a mapping relationship between the state and the action. In order to evaluate the advantages and disadvantages of different policies π, reinforcement learning defines the value function and the action value function: where G t is the return at time step t, which is the discounted reward over future time; The update formula of the Q-value function is: Q(s,a) = Q(s,a) + α(R t + γ max Q(s',a') - Q(s,a)) In the formula, α is the learning factor, γ is the discount factor, and maxQ(s',a') represents the maximum expected return of selecting an action in the next state; The Q-learning algorithm adopts the ε-greedy strategy, and the expression is: When the agent reaches the target point, the reward value is set to a positive number. When the agent collides with an obstacle, the reward value is set to a negative number. In other cases, the reward value is set to zero; the expression is: where s goal and s obs represent the target state and the obstacle state respectively, and R0 represents a positive value.

7. The method for collision-free path planning of a picking robotic arm according to claim 1, wherein In step S4, Informed-Q limits the state space of each round within an ellipse according to the path length of the previous round; To obtain the state s' in the ellipse ellipes , first obtain the state s of the standard ellipse ellipes . The standard ellipse equation is: s ellipes =(x, y) is the state coordinate in the standard ellipse, and a and b are respectively half of the lengths of the major axis and the minor axis of the ellipse, that is After that, s ellipes is subjected to coordinate transformation to obtain s' ellipes , and the coordinate transformation equation is: s' ellipes = Rs ellipes + T In the formula, R is the rotation matrix, T is the translation matrix, and their expressions are: where θ represents the angle between the line connecting the starting state and the target state and the x-axis, and (x center , y center ) is the coordinate of the midpoint of the major axis of the ellipse; The reward function of the sparse reward is: The expression of the dynamic ε-greedy strategy is: where ε start and ε end represent the greedy coefficients at the start and end respectively, episode represents the current episode number, and e total represents the total number of episodes.

8. A collision-free path planning system for a picking robotic arm, characterized in that, It includes: A three-dimensional modeling module, which is used to establish a three-dimensional scene model of the robotic arm and its working environment, and model the elements that can collide, non-picking targets, and picking targets in the scene; A collision detection module, which is used to construct a collision detection model based on the established three-dimensional model. The collision detection model includes the boundary description of obstacles, the motion constraints of the robotic arm, and the collision determination rules. The boundary description simplifies the spatial boundary through a bounding box. The motion constraints include the degrees of freedom of the robotic arm joints, speed limits, and acceleration limits. The collision determination rules include geometric overlap detection, minimum distance determination, and time step prediction mechanism; A decision-making modeling module, which is used to model the robotic arm motion control process as a Markov decision model based on states, actions, state transitions, and reward mechanisms. The module constructs a state transition probability model and a reward function; A path training module, which is used to train the decision model based on the Informed-Q algorithm to obtain a path planning model and realize the update of the greedy coefficient of the dynamic-greedy strategy; A control instruction output module, which is used to convert the action sequence generated by the trained path planning model into a robotic arm drive signal and send it to the execution mechanism to realize path tracking control.

9. A terminal, characterized in that, Including: A memory, which is used to store the collision-free path planning program for the picking robotic arm; A processor, which is used to execute the steps of the collision-free path planning method for the picking robotic arm as described in any one of claims 1-7 when executing the collision-free path planning system for the picking robotic arm.

10. A computer-readable storage medium, characterized in that, The storage medium stores computer instructions. When the computer reads the computer instructions in the storage medium, the computer executes a collision-free path planning method for a picking robotic arm as described in any one of claims 1 to 7.

Images