Picking robot moving chassis and mechanical arm collaborative path planning method and device
Through a reinforcement learning-based method, combined with Markov decision-making process model and PPO algorithm, the coordinated path planning of the robot's robot's robot and mobile chassis is optimized, which solves the problems of low picking flexibility and low efficiency in traditional methods, and realizes the robot's robot Automatic and efficient arrival and picking path optimization at the end.
Patent Information
- Application Number
- CN202410898209.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-05
- Publication Date
- 2024-11-08
AI Technical Summary
When planning the picking path, traditional picking robots usually regard the mobile chassis and robotic arms as independent systems, resulting in low picking flexibility and small range, and requires manual adjustment of the chassis position, which is inefficient and difficult to solve the problem of collaborative path planning.
Using reinforcement learning-based method, the Markov decision-making process model of the picking robot is constructed, the distance-guided reward, step-size punishment and collision punishment functions are designed, and the deep neural network is trained using the PPO algorithm to optimize the coordinated path planning of the robot arm and the mobile chassis. , generates the shortest path without collision.
It realizes that the end of the robot arm can automatically and efficiently reach the designated position, improves the flexibility and efficiency of picking, reduces manual intervention, and improves the automation and intelligence level of agricultural picking robots.
Smart Images

Figure CN118915719A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of path planning, and in particular to a method and device for collaborative path planning between a mobile chassis and a mechanical arm of a picking robot. Background Art
[0002] A harvesting robot is a system in which a robotic arm is installed on a mobile chassis. It is mainly used for picking fruits. Different from a separate mobile chassis and a separate robotic arm, a harvesting robot has a wider operating space and more flexible operability. Therefore, a harvesting robot has a very broad application prospect in the field of agricultural picking.
[0003] In order for the picking robot to complete the picking task efficiently, it is usually necessary to perform path planning for it, that is, to generate a collision-free path from the starting position to the target position for the picking robot. Many scholars have conducted in-depth research on the path planning problem in picking robots. For example, the first prior art solution discloses a collision-free path planning method for a picking robot based on deep reinforcement learning. The trained path planning model can plan a collision-free picking path for the robotic arm. However, current research does not combine the mobile chassis with the robotic arm. Only the robotic arm is considered when planning the picking path. In many cases, the two subsystems are even considered to be independent, but in fact the two are closely connected. The final positioning of the mobile chassis has a great impact on the subsequent picking path planning of the robotic arm.
[0004] The traditional picking process often involves adjusting the initial position of the mobile chassis first, and then planning the picking path of the robotic arm. However, the initial position of the mobile chassis is usually set manually. Since it is impossible for humans to determine whether its position is optimal, the robotic arm may not be able to reach the specified position or the target may exceed the working range of the robotic arm, resulting in picking failure. Humans need to adjust the position of the mobile chassis again until the robotic arm can reach the target and complete the picking task. The traditional method requires manual participation, has complicated steps and is inefficient. Therefore, it is necessary to consider a collaborative path planning method between the mobile chassis and the robotic arm of the picking robot to solve the collaborative path planning problem of the picking robot and improve the picking efficiency.
[0005] Picking robots often need to effectively plan picking paths in an environment with obstacles so that the end of the robotic arm can reach the specified position for subsequent picking operations. However, picking robots have a high degree of freedom and redundancy, and their collaborative path planning problem is more complicated than the separate planning problems of the mobile chassis and robotic arm. Traditional path planning methods such as the rapidly expanding random tree algorithm and the A* algorithm face significant challenges and limitations when solving high-dimensional problems, and it is difficult to solve the collaborative path planning problem of picking robots. Summary of the invention
[0006] In order to at least solve one of the technical problems existing in the prior art to a certain extent, the purpose of the present invention is to provide a method and device for collaborative path planning between a mobile chassis and a robotic arm of a picking robot based on reinforcement learning, so as to realize collaborative path planning of the picking robot, plan a collision-free shortest path for the picking robot, and enable the end of the robotic arm to reach a specified position.
[0007] The technical solution adopted by the present invention is:
[0008] A method for collaborative path planning between a mobile chassis and a mechanical arm of a picking robot comprises the following steps:
[0009] Building a picking robot picking simulation environment, the simulation environment including a picking robot model, obstacles and picking target points;
[0010] The whole picking robot is taken as an intelligent agent, and a Markov decision process model of the picking robot is constructed; wherein the Markov decision process model includes a state space, an action space, and a reward function;
[0011] Interact with the simulation environment through reinforcement learning to adjust the optimal strategy and obtain the highest reward value;
[0012] Use the reinforcement learning PPO algorithm to build a deep neural network to achieve strategy optimization and value estimation;
[0013] Based on the reinforcement learning PPO algorithm, a collision-free shortest path is planned for the picking robot, so that the end of the robotic arm reaches the target position.
[0014] Furthermore, the construction of a simulation environment for picking by a picking robot includes:
[0015] A picking robot simulation environment is built based on robot simulation software, where the picking robot model is composed of a mobile chassis and a robotic arm;
[0016] Set the initial position of the picking robot and the position variation range of obstacles and targets.
[0017] Furthermore, the Markov decision process model of the picking robot is constructed, including:
[0018] The Markov decision process model of the picking robot is expressed as a five-tuple (S, A, P, R, γ); the state set S is the joint angle q of each rotating joint axis of the robot arm, the relative position p1 between the end of the robot arm and the target point, the distance d1 from the end of the robot arm to the target point, the relative position p2 between the key points of each rotating joint axis of the robot arm and the obstacle, the distance d2 between the key points of each rotating joint axis of the robot arm and the obstacle, the relative position p3 and the rotation angle h1 between the mobile chassis and the target point, which constitute the initial state set S; the action set A is the angle increment △θ of each joint of the robot arm and the linear displacement increment △x and angular displacement increment of the mobile chassis P is the state transition probability, P: S×A×S→(0,1); R is the reward function; γ is the discount factor used to calculate the cumulative reward, γ∈(0,1).
[0019] Furthermore, the reward function includes a distance-guided reward function R goal , step size penalty function R step and collision penalty function R col ;
[0020] Where the distance-guided reward function R goal include:
[0021]
[0022]
[0023]
[0024]
[0025] In the formula, (P xi , P yi , P zi ) is the three-dimensional coordinate of the end of the robot at time i in the coordinate system with the target point as the origin, D i is the Euclidean distance between the end of the robot and the target point at the current moment i, D min is the minimum distance between the end of the robot and the target point in a training round, τ1 is the distance threshold to the target position, k1, k2, k3 represent adjustable weight coefficients;
[0026] The step size penalty function R step include:
[0027]
[0028] Where D0 is the Euclidean distance between the end of the robot arm and the target point at the initial moment in a training round, and k4 represents the adjustable weight coefficient;
[0029] The collision penalty function R colinclude:
[0030]
[0031] In the formula, colli=1 indicates a collision occurs, colli=0 indicates no collision occurs, and k5 indicates an adjustable weight coefficient.
[0032] The single-step reward of the picking robot is:
[0033] R=R goal +R step +R col .
[0035] Furthermore, the interaction with the simulation environment through reinforcement learning to adjust the optimal strategy and obtain the highest reward value includes:
[0036] Select different actions from the action set A to control the picking robot so that the state changes from S t Transition to S t+1 ; Evaluate the decision-making behavior at time t according to the reward function, and continuously interact with the simulation environment through reinforcement learning to adjust the optimal strategy and obtain the highest reward value.
[0037] Furthermore, the calculation formula of the highest reward value is:
[0038]
[0039] Where π is the probability distribution function, γ is the discount (attenuation) factor, ranging from 0 to 1; R i is the reward value obtained by the agent each time it interacts with the simulation environment; P is the collection of probability distribution functions, T is the maximum number of time steps, Represents the expected value of the cumulative discounted reward.
[0040] Furthermore, the use of the reinforcement learning PPO algorithm to construct a deep neural network to achieve strategy optimization and value estimation includes:
[0041] Construct the policy network and value network in the PPO algorithm, where the policy network and value network include input layer, hidden layer and output layer. The hidden layer activation function uses the relu function, and the network parameters are orthogonally initialized. The input of the policy network is the state quantity of the picking robot, and the output is the action quantity of the picking robot. The input of the value network is the state quantity of the picking robot, and the output is the estimated value corresponding to the current state quantity.
[0042] Furthermore, the training based on the reinforcement learning PPO algorithm to plan a collision-free shortest path for the picking robot includes:
[0043] Set the number of rounds and perform the training steps for each round based on the reinforcement learning PPO algorithm;
[0044] After all rounds are completed, the training of the picking robot collaborative path planning strategy is completed;
[0045] Save the strategy network parameters and value network parameters. The strategy network parameter output is the shortest collision-free path of the picking robot, so that the end of the robotic arm can reach the specified target position.
[0046] Furthermore, the training steps of executing each round based on the reinforcement learning PPO algorithm include:
[0047] Set the maximum number of steps of the picking robot in each round;
[0048] Randomly initialize the positions of obstacles and targets;
[0049] The policy network inputs the current observation state s t , output action a under this observation state t , the value network inputs the current observation state s t , output the state estimate of the current observation state; execute action a t , collect new observation states s t+1 And the corresponding reward r t and the variable d that determines whether the terminal state has been reached, the sequence (s t , a t , r t ,s t+1 , d) transfer to the experience buffer B;
[0050] When the amount of data accumulated in the experience buffer B reaches the threshold, small batch sampling is performed to calculate the advantage function A t Valuation of
[0051] Using the sampled data, calculate the policy loss function, update the gradient of the policy network, and use the output of the value network to calculate the loss and update the value network to maximize the overall objective function.
[0052] A round ends when the number of steps reaches the maximum limit.
[0053] Another technical solution adopted by the present invention is:
[0054] A picking robot mobile chassis and mechanical arm collaborative path planning device, comprising:
[0055] at least one processor;
[0056] at least one memory for storing at least one program;
[0057] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.
[0058] Another technical solution adopted by the present invention is:
[0059] A computer-readable storage medium stores a program executable by a processor, wherein the program executable by the processor is used to execute the method described above when executed by the processor.
[0060] Compared with the existing technology, the method of the present invention has the following advantages and beneficial effects:
[0061] (1) In view of the problems that traditional picking robots regard the mobile chassis and the robotic arm as independent systems during the picking process and only consider the robotic arm when planning the picking path, resulting in low picking flexibility and small picking range, as well as the cumbersome operation of manually adjusting the position of the mobile chassis in the traditional picking process, the present invention considers the mobile chassis and the robotic arm of the picking robot in a coordinated manner to improve the picking efficiency and picking flexibility.
[0062] (2) The present invention proposes a method for collaborative path planning between the mobile chassis and the robotic arm of a harvesting robot based on reinforcement learning. A harvesting training environment for the harvesting robot is built based on robot simulation software. The Markov decision process (MDP) model is performed on the entire harvesting robot, and a related reward function is designed. Based on the PPO reinforcement learning algorithm, after a certain period of training, a collision-free shortest path can be planned for the harvesting robot so that the end of the robotic arm can reach the specified position. This method takes into account the collaborative path planning problem between the mobile chassis and the robotic arm, has higher versatility, can realize fully automatic harvesting of the robot, and can improve the automation and intelligence level of agricultural harvesting robots. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0064] Figure 1 This is a flowchart of the steps of a method for collaborative path planning between a mobile chassis and a mechanical arm of a picking robot in an embodiment of the present invention;
[0065] Figure 2 is a scene diagram of a simulation training environment of a harvesting robot in an embodiment of the present invention;
[0066] Figure 3 It is a framework diagram of the PPO reinforcement learning algorithm in an embodiment of the present invention;
[0067] Figure 4 is a training flow chart of collaborative path planning of a picking robot in an embodiment of the present invention;
[0068] Figure 5 It is a training reward curve diagram of the PPO reinforcement learning algorithm in an embodiment of the present invention;
[0069] Figure 6 It is a diagram of the path planning results of the PPO reinforcement learning algorithm training the picking robot in the case of no mobile chassis movement and in the case of collaborative path planning in an embodiment of the present invention. DETAILED DESCRIPTION
[0070] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.
[0071] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., and orientations or positional relationships indicated are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present invention.
[0072] In the description of the present invention, the meaning of "several" is one or more, the meaning of "more" is two or more, and the meanings of "greater than", "less than", "exceed" and the like are understood as not including the number itself, and the meanings of "above", "below", "within" and the like are understood as including the number itself. If there is a description of the first and the second, it is only used for the purpose of distinguishing the technical features, and cannot be understood as indicating or implying the relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features. In addition, "and / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects before and after are in an "or" relationship.
[0073] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.
[0074] Terminology explanation:
[0075] Reinforcement Learning PPO Algorithm: PPO (Proximal Policy Optimization) is a policy gradient-based reinforcement learning algorithm that optimizes the strategy to allow the agent to select the best action in different states, thereby maximizing the expected reward. The PPO algorithm limits the magnitude of the policy update, which not only maintains the policy's exploration ability but also prevents instability caused by excessive policy updates, and has high sample efficiency and stability.
[0076] like Figure 1 As shown, this embodiment provides a method for collaborative path planning of a mobile chassis and a mechanical arm of a picking robot, which is used for collaborative path planning of a picking robot. In this embodiment, the picking robot is composed of a four-wheel differential mobile chassis and a six-axis mechanical arm. Considering that physical training is expensive, training is performed in simulation software, which specifically includes the following steps:
[0077] S1. Building a simulation environment for picking by a picking robot, wherein the simulation environment includes a picking robot model, obstacles, and picking target points.
[0078] Specifically, a picking robot picking training environment is built based on robot simulation software, including establishing a picking robot model, setting obstacles and picking target points. In some embodiments, Figure 2 As shown, a simulation training environment is built based on the CoppeliaSim robot simulation software, the initial position of the picking robot is set, the blue ball is used as an obstacle, the yellow ball is used as a target, and the position change range of the obstacle and the target is set.
[0079] S2. Taking the entire harvesting robot as an intelligent agent, a Markov decision process model of the harvesting robot is constructed; wherein the Markov decision process model includes a state space, an action space, and a reward function.
[0080] The picking robot as a whole is taken as an intelligent agent, and a Markov decision (MDP) process model of the picking robot is constructed based on the Markov decision (MDP) process, including the construction of state space, action space and reward function.
[0081] As an optional implementation, step S2 includes the following steps:
[0082] S21. Construct the state space and action space of the picking robot based on the Markov decision process (MDP).
[0083] Specifically, the Markov decision process model of the picking robot is represented as a quintuple (S, A, P, R, γ).
[0084] The state set S is the joint angle q of each rotating joint axis of the robot, the relative position p1 of the end of the robot and the target point, the distance d1 from the end of the robot to the target point, the relative position p2 of the key points of each rotating joint axis of the robot and the obstacle, the distance d2 of the key points of each rotating joint axis of the robot and the obstacle, the relative position p3 and the rotation angle h1 of the mobile chassis and the target point, which constitute the initial state set S; the action set A is the angle increment △θ of each joint of the robot, the linear displacement increment △x of the mobile chassis and the angular displacement increment P is the state transition probability, P: S×A×S→(0,1); R is the reward function; γ is the discount factor used to calculate the cumulative reward, γ∈(0,1).
[0085] S22. Construct a reward function for the picking robot.
[0086] Specifically, the reward function includes a distance-guided reward function R goal , step size penalty function R step and collision penalty function R col :
[0087] The distance-guided reward function R goal include:
[0088]
[0089]
[0090]
[0091]
[0092] Among them (P xi , P yi , P zi ) is the three-dimensional coordinate of the end of the robot at time i in the coordinate system with the target point as the origin, D i is the Euclidean distance between the end of the robot and the target point at the current moment i, D min is the minimum distance between the end of the robot arm and the target point in a training round, τ1 is the distance threshold to reach the target position, and k1, k2, and k3 represent adjustable weight coefficients.
[0093] The step size penalty function R step include:
[0094]
[0095] Where D0 is the Euclidean distance between the end of the robot arm and the target point at the initial moment in a training round, and k4 represents the adjustable weight coefficient.
[0096] The collision penalty function R col include:
[0097]
[0098] Where colli is 1, indicating a collision has occurred, colli is 0, indicating no collision has occurred, and k5 indicates an adjustable weight coefficient.
[0099] The single-step reward of the picking robot is:
[0100] R=R goal +R step +R col
[0101] S3. Interact with the simulation environment through reinforcement learning to adjust the optimal strategy and obtain the highest reward value.
[0102] Through reinforcement learning and interaction with the simulation environment, the appropriate decision value is selected from the action space to control the picking robot. At the same time, the decision-making behavior at time t is evaluated based on the reward function to adjust the optimal strategy and obtain the highest reward value.
[0103] Specifically, different actions are selected from the action set A to control the picking robot so that the state changes from S t Transition to S t+1 , evaluate the decision-making behavior at time t according to the reward function, continuously interact with the simulation environment through reinforcement learning, adjust the optimal strategy and obtain the highest reward value; the calculation formula of the highest reward value is:
[0104]
[0105] Where π is the probability distribution function, γ is the discount (attenuation) factor, which ranges from 0 to 1; R i is the reward value obtained by the agent each time it interacts with the simulation environment; P is the collection of probability distribution functions, T is the maximum number of time steps, Represents the expected value of the cumulative discounted reward.
[0106] S4. Use the reinforcement learning PPO algorithm to build a deep neural network to achieve strategy optimization and value estimation.
[0107] As an optional implementation, step S4 specifically includes the following steps:
[0108] S41. Construct the strategy network and value network in the PPO algorithm.
[0109] The structure of the policy network and the value network includes an input layer, a hidden layer, and an output layer. The hidden layer activation function uses the relu function, and the network parameters are orthogonally initialized. The input of the policy network is the state of the picking robot, and the output is the action of the picking robot. The input of the value network is the state of the picking robot, and the output is the estimated value corresponding to the current state.
[0110] S42. Set PPO algorithm related hyperparameters
[0111] Set the learning rate of the policy network and the value network to 0.0003, the discount factor γ to 0.99, and the size of the experience buffer B to 2048.
[0112] S5. Based on the reinforcement learning PPO algorithm, a collision-free shortest path is planned for the picking robot, so that the end of the robotic arm reaches the target position.
[0113] See also Figure 4 As an optional implementation, step S5 specifically includes the following steps:
[0114] S51, the preset number of rounds is 5000, see Figure 3 , perform the following steps A1-A6 in each round:
[0115] A1. It is preset that the maximum number of steps of the picking robot in each round is 200.
[0116] A2. Randomly initialize the positions of obstacles and targets.
[0117] A3. The policy network inputs the current observation state s t , output action a under this observation state t , the value network inputs the current observation state s t , output the state estimate of the current observation state; execute action a t , collect new observation states s t+1 And the corresponding reward r t and the variable d that determines whether the terminal state has been reached, the sequence (s t , a t , r t ,s t+1 , d) Transfer to the experience buffer B.
[0118] A4. When the amount of data accumulated in the experience buffer B reaches the threshold, small batch sampling is performed and the advantage function A is calculated. t The valuation of the agent at time t; the advantage function A of the agent at time tt Defined as:
[0119] A t =Q(s t ,a t )-V(s t )
[0120] Where Q(s t ,a t ) is the observed state s t Take action a t The cumulative reward value of V(s t ) is the observed state s t The estimated value of the state;
[0121] In order to balance the variance and bias, the generalization advantage estimation (GAE) is used to calculate the advantage function, and its formula is:
[0122]
[0123] Where λ is the parameter for balancing variance and bias in GAE; T is the maximum number of time steps, δ t Defined as δ t =r t +γV(s t+1 )-V(s t ), rt is the reward value at time t.
[0124] A5. Use the sampled data to calculate the policy loss function, use the Adam optimizer to update the gradient of the policy network, and use the output of the value network to calculate The loss updates the value network to maximize the overall objective function, where R t For discount rewards.
[0125] The policy loss function is defined as:
[0126]
[0127] In the formula represents the mean value of the interval 0 to t, ε is the cutoff factor, and the action probability ratio π θ represents the current strategy, π θold represents the old strategy, a t is the action at time t, s t is the observed state at time t.
[0128] A6. When the number of steps reaches the maximum limit, a round ends.
[0129] S52: After all rounds are completed, the training of the picking robot collaborative path planning strategy is completed.
[0130] S53, saving the strategy network parameters and the value network parameters, the strategy network parameter output is the shortest collision-free path of the picking robot, so that the end of the robot arm can reach the specified target position.
[0131] Table 1 shows Figure 2 The task success rate of the picking robot in the simulation environment under the collaborative path planning and the task success rate under the non-mobile chassis movement, wherein the collaborative path planning is an embodiment of the present invention:
[0132] Table 1
[0133]
[0134] As can be seen from Table 1, collaborative path planning for the picking robot can greatly improve the success rate of picking. For some target points outside the range of the robotic arm, the picking robot can reach the target by coordinating the movement of the mobile chassis. Compared with traditional picking methods, it is more efficient and more intelligent. Figure 5 This is the training reward curve for the picking robot considering collaborative path planning. The PPO reinforcement learning algorithm basically reaches convergence after 3000 rounds of training. The training speed is fast and stable, and the path planning success rate of the test reaches 88%; Figure 6 (a) is the picking path output by the PPO algorithm without the movement of the mobile chassis. The purple curve in the figure is its path. It can be seen that the end fails to reach the target point smoothly; Figure 6 (b) is the collision-free shortest path of the picking robot output by the PPO algorithm considering collaborative path planning, that is, the method of the present invention. The purple curve in the figure is its path, and the end of the robotic arm can accurately reach the target position. The above results show that collaborative path planning can effectively improve the picking success rate of the picking robot, and the reinforcement learning method can effectively solve the collaborative path planning problem of the picking robot.
[0135] The present invention aims at the problem that the traditional picking robot regards the mobile chassis and the mechanical arm as independent systems during the picking process, and only considers the mechanical arm when planning the picking path, resulting in low picking flexibility and small range, as well as the cumbersome operation of manually adjusting the position of the mobile chassis in the traditional picking process. The mobile chassis and the mechanical arm of the picking robot are considered in a coordinated manner to improve the picking efficiency and picking flexibility. A picking training environment for the picking robot is built based on robot simulation software, the Markov decision process (MDP) model is performed on the entire picking robot, and a related reward function is designed. Based on the PPO reinforcement learning algorithm, after a certain period of training, the shortest collision-free path can be planned for the picking robot, so that the end of the mechanical arm can reach the specified position. This method solves the problem of decoupling the mobile chassis and the mechanical arm for path planning in the traditional method, has higher versatility, can realize fully automatic picking of the robot, and can improve the automation and intelligence level of the agricultural picking robot.
[0136] This embodiment also provides a picking robot mobile chassis and mechanical arm collaborative path planning device, including:
[0137] at least one processor;
[0138] at least one memory for storing at least one program;
[0139] When the at least one program is executed by the at least one processor, the at least one processor implements Figure 1 The method shown.
[0140] A picking robot mobile chassis and robotic arm collaborative path planning device in this embodiment can execute a picking robot mobile chassis and robotic arm collaborative path planning method provided by the method embodiment of the present invention, can execute any combination of implementation steps of the method embodiment, and has the corresponding functions and beneficial effects of the method.
[0141] The present application also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes Figure 1 The method shown.
[0142] This embodiment also provides a storage medium, which stores instructions or programs that can execute a method for collaborative path planning between a mobile chassis and a robotic arm of a harvesting robot provided by an embodiment of the method of the present invention. When the instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the corresponding functions and beneficial effects of the method can be obtained.
[0143] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.
[0144] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.
[0145] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.
[0146] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.
[0147] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.
[0148] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0149] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0150] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.
[0151] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.
Claims
1. A method for collaborative path planning between a mobile chassis and a mechanical arm of a picking robot, characterized in that: The following steps are involved: Building a picking robot picking simulation environment, the simulation environment including a picking robot model, obstacles and picking target points; The whole picking robot is taken as an intelligent agent, and a Markov decision process model of the picking robot is constructed; wherein the Markov decision process model includes a state space, an action space, and a reward function; Interact with the simulation environment through reinforcement learning to adjust the optimal strategy and obtain the highest reward value; Use the reinforcement learning PPO algorithm to build a deep neural network to achieve strategy optimization and value estimation; Based on the reinforcement learning PPO algorithm, a collision-free shortest path is planned for the picking robot.
2. The method for collaborative path planning of a mobile chassis and a mechanical arm of a picking robot according to claim 1, characterized in that: The construction of a picking robot picking simulation environment includes: A picking robot simulation environment is built based on robot simulation software, where the picking robot model is composed of a mobile chassis and a robotic arm; Set the initial position of the picking robot and the position variation range of obstacles and targets.
3. The method for collaborative path planning of a mobile chassis and a mechanical arm of a picking robot according to claim 1, characterized in that: The Markov decision process model for constructing the picking robot includes: The Markov decision process model of the picking robot is expressed as a five-tuple (S, A, P, R, γ); the state set S is the joint angle q of each rotating joint axis of the robot arm, the relative position p1 between the end of the robot arm and the target point, the distance d1 from the end of the robot arm to the target point, the relative position p2 between the key points of each rotating joint axis of the robot arm and the obstacle, the distance d2 between the key points of each rotating joint axis of the robot arm and the obstacle, the relative position p3 and the rotation angle h1 between the mobile chassis and the target point, which constitute the initial state set S; the action set A is the angle increment △θ of each joint of the robot arm and the linear displacement increment △x and angular displacement increment of the mobile chassis P is the state transition probability, P: S×A×S→(0,1); R is the reward function; γ is the discount factor used to calculate the cumulative reward, γ∈(0,1).
4. The method for collaborative path planning of a mobile chassis and a mechanical arm of a picking robot according to claim 3, characterized in that: The reward function includes a distance-guided reward function R goal , step size penalty function R step and collision penalty function R col ; in The distance-guided reward function R goal include: In the formula, (P xi , P yi , P zi ) is the three-dimensional coordinate of the end of the robot at time i in the coordinate system with the target point as the origin, D i is the Euclidean distance between the end of the robot and the target point at the current moment i, D min is the minimum distance between the end of the robot and the target point in a training round, τ1 is the distance threshold to the target position, k1, k2, k3 represent adjustable weight coefficients; The step size penalty function R step include: Where D0 is the Euclidean distance between the end of the robot arm and the target point at the initial moment in a training round, and k4 represents the adjustable weight coefficient; The collision penalty function R col include: In the formula, colli=1 indicates a collision occurs, colli=0 indicates no collision occurs, and k5 indicates an adjustable weight coefficient.
5. The method for collaborative path planning of a mobile chassis and a mechanical arm of a harvesting robot according to claim 1, characterized in that: The interaction between reinforcement learning and the simulation environment to adjust the optimal strategy and obtain the highest reward value includes: Select different actions from the action set A to control the picking robot so that the state changes from S t Transition to S t+1 ; Evaluate the decision-making behavior at time t according to the reward function, and continuously interact with the simulation environment through reinforcement learning to adjust the optimal strategy and obtain the highest reward value.
6. The method for collaborative path planning of a mobile chassis and a mechanical arm of a harvesting robot according to claim 5, characterized in that: The calculation formula for the maximum reward value is: In the formula, π is the probability distribution function, γ is the discount factor; R i is the reward value obtained by the agent each time it interacts with the simulation environment; P is the collection of probability distribution functions, T is the maximum number of time steps, Represents the expected value of the cumulative discounted reward.
7. The method for collaborative path planning of a mobile chassis and a mechanical arm of a harvesting robot according to claim 1, characterized in that: The reinforcement learning PPO algorithm is used to build a deep neural network to achieve strategy optimization and value estimation, including: Construct the policy network and value network in the PPO algorithm, where the policy network and value network include input layer, hidden layer and output layer, and the network parameters are orthogonally initialized; the input of the policy network is the state quantity of the picking robot, and the output is the action quantity of the picking robot; the input of the value network is the state quantity of the picking robot, and the output is the estimated value corresponding to the current state quantity.
8. The method for collaborative path planning of a mobile chassis and a mechanical arm of a picking robot according to claim 1, characterized in that: The training based on the reinforcement learning PPO algorithm is used to plan a collision-free shortest path for the picking robot, including: Set the number of rounds and perform the training steps for each round based on the reinforcement learning PPO algorithm; After all rounds are completed, the training of the picking robot collaborative path planning strategy is completed; Save the strategy network parameters and value network parameters. The strategy network parameter output is the shortest collision-free path of the picking robot, so that the end of the robotic arm can reach the specified target position.
9. The method for collaborative path planning of a mobile chassis and a mechanical arm of a harvesting robot according to claim 8, characterized in that: The training steps for each round of the reinforcement learning PPO algorithm include: Set the maximum number of steps of the picking robot in each round; Randomly initialize the positions of obstacles and targets; The policy network inputs the current observation state s t , output action a under this observation state t , the value network inputs the current observation state s t , output the state estimate of the current observation state; execute action a t , collect new observation states s t+1 And the corresponding reward r t and the variable d that determines whether the terminal state has been reached, the sequence (s t , a t , r t ,s t+1 , d) transfer to the experience buffer B; When the amount of data accumulated in the experience buffer B reaches the threshold, small batch sampling is performed to calculate the advantage function A t ; Using the sampled data, calculate the policy loss function, update the gradient of the policy network, and use the output of the value network to calculate the loss and update the value network to maximize the overall objective function; A round ends when the number of steps reaches the maximum limit.
10. A picking robot mobile chassis and mechanical arm collaborative path planning device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 9.
Citation Information
Cited By
Fruit picking method and device for multi-arm robot
CN119817333A
Collision-free path planning method and device for picking mechanical arm, terminal and medium
CN120395879A
Robot control method and device and robot control system
CN121560031A
