Fruit picking method and device for multi-arm robot

Abstract

The invention provides a multi-arm robot fruit picking method and device, and relates to the technical field of artificial intelligence, and the method comprises the steps: building a simulation environment of discrete time steps, simulating the fruit picking process of a multi-arm robot, obtaining the current state of the multi-arm robot in the simulation environment, inputting the current state to a trained LSTM-PPO model, and obtaining a fruit picking result of the multi-arm robot. And the target action of the multi-arm robot in the simulation environment is obtained, so that the movement path of the multi-arm robot for picking fruits in the simulation environment is determined. According to the multi-arm robot fruit picking method provided by the invention, balanced and reasonable task distribution of the multiple mechanical arms is realized, interactive interference among the mechanical arms is avoided, and the total success rate and completion rate of picking operation are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multi-arm robot fruit picking method and device. Background Art

[0002] As a new type of agricultural robot, the fruit picking robot mainly uses a variety of technical means such as visual recognition, robot positioning, grasping control, etc. to independently complete the tasks of identifying, picking and sorting fruits (such as apples, oranges, kiwis and other fruits), thereby realizing full automation of fruit picking.

[0003] In existing fruit picking robots, numerical solutions or meta-heuristic algorithms are usually used for motion path planning during the picking process, but these methods are difficult to cope with dynamic working conditions. In the actual working scenarios of the picking robots, picking failures and accidental drops of picking targets often occur. In addition, for rectangular coordinate multi-arm picking robots with overlapping working spaces, the picking behavior of one robot arm will affect the decision of another robot arm during the picking process. The existing technology cannot balance and reasonably allocate tasks to each arm, resulting in a low overall success rate and completion rate of the picking operation. Summary of the invention

[0004] The present invention provides a multi-arm robot fruit picking method and device, which are used to solve the technical problem of low overall success rate and completion rate of multi-arm robot picking operations in the prior art.

[0005] The present invention provides a multi-arm robot fruit picking method, comprising the following steps: Constructing a simulation environment of discrete time steps; the simulation environment is used to simulate the process of fruit picking by a multi-arm robot; each time step in the discrete time steps corresponds to the time length in the real world; Acquire the current state of the multi-arm robot in the simulation environment; Based on the current state, the target action of the multi-arm robot in the simulation environment is obtained through the trained LSTM-PPO model; the LSTM-PPO model is constructed based on the long short-term memory network, the proximal strategy optimization algorithm and the Actor-Critic model, and is obtained through a predetermined number of iterative training; Based on the target action of the multi-arm robot in the simulation environment, a motion path of the multi-arm robot for picking fruits in the simulation environment is determined.

[0006] According to a multi-arm robot fruit picking method provided by the present invention, the training step of the LSTM-PPO model includes: Acquire a first state of the multi-arm robot in the simulation environment; Based on the first state, obtaining a first action of the multi-arm robot in the simulation environment through an LSTM-PPO model; Interacting the first action with the simulation environment to obtain a second state of the multi-arm robot in the simulation environment and an instant reward corresponding to the second state; Based on the second state and the instant reward corresponding to the second state, updating the parameters of the LSTM-PPO model through the PPO-Clip optimization strategy; Repeat the iteration for a predetermined number of times to obtain the trained LSTM-PPO model.

[0007] According to a multi-arm robot fruit picking method provided by the present invention, the instant reward includes a movement time cost penalty, an idle state penalty, an action execution penalty, a collision penalty, a successful picking reward, and a task completion reward; The movement time cost penalty is used to reduce the movement distance and time of the multi-arm robot picking fruits; The idle state penalty is used to reduce the idle time of the mechanical arms of the multi-arm robot; The action execution penalty is used to characterize the time it takes for the multi-arm robot to execute the target action; The collision penalty is used to avoid spatial conflicts between the mechanical arms of the multi-arm robot by imposing a high penalty on the collision behavior; The successful picking reward is used to apply a positive reward to the mechanical arm of the multi-arm robot that successfully picks the target fruit; The task completion reward is used to provide an additional reward to the multi-arm robot when all target fruits are picked and placed and all the robot arms of the multi-arm robot remain idle.

[0008] According to a multi-arm robot fruit picking method provided by the present invention, based on the current state, obtaining the target action of the multi-arm robot in the simulation environment through a trained LSTM-PPO model includes: Based on the current state of the multi-arm robot in the simulation environment, obtaining the action probability distribution of the multi-arm robot; the action probability distribution is used to characterize the probability of the multi-arm robot performing any action; Based on the action probability distribution, the target action of the multi-arm robot in the simulation environment is obtained through an importance sampling algorithm.

[0009] According to a multi-arm robot fruit picking method provided by the present invention, the current state includes the position of the fruit, the picking state, the position of the multi-arm robot's robotic arms, the action state of the multi-arm robot's robotic arms, and the state of the multi-arm robot's robotic arms holding the fruit.

[0010] According to a multi-arm robot fruit picking method provided by the present invention, the target action of the multi-arm robot in the simulation environment includes any one of approaching target fruit, picking target fruit, placing target fruit and remaining idle.

[0011] The present invention also provides a multi-arm robot fruit picking device, comprising the following modules: A simulation module is used to construct a simulation environment of discrete time steps; the simulation environment is used to simulate the process of fruit picking by a multi-arm robot; each time step in the discrete time steps corresponds to the time length in the real world; An acquisition module, used for acquiring the current state of the multi-arm robot in the simulation environment; An action module, used to obtain the target action of the multi-arm robot in the simulation environment based on the current state through a trained LSTM-PPO model; the LSTM-PPO model is constructed based on a long short-term memory network, a proximal policy optimization algorithm and an Actor-Critic model, and is obtained through a predetermined number of iterative training; A path module is used to determine a motion path of the multi-arm robot for picking fruits in the simulation environment based on a target action of the multi-arm robot in the simulation environment.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, it implements any of the multi-arm robot fruit picking methods described above.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the multi-arm robot fruit picking methods described above.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the multi-arm robot fruit picking method as described above is implemented.

[0015] The present invention provides a multi-arm robot fruit picking method, which constructs a simulation environment with discrete time steps to simulate the process of fruit picking by the multi-arm robot, obtains the current state of the multi-arm robot in the simulation environment, and inputs the current state into a trained LSTM-PPO model, thereby modeling the task allocation of multiple robotic arms as a Markov decision process, and combines with a reinforcement learning algorithm to achieve balanced and reasonable task allocation for multiple robotic arms, obtain the target action of the multi-arm robot in the simulation environment, thereby determining the motion path of the multi-arm robot in picking fruits in the simulation environment, avoiding mutual interference between robotic arms, and improving the overall success rate and completion rate of the picking operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 The present invention provides a flowchart of a multi-arm robot fruit picking method.

[0018] Figure 2 This is a schematic diagram of a fruit picking work area of ​​a dual-arm robot provided by the present invention.

[0019] Figure 3 It is a structural schematic diagram of an LSTM-PPO model in a multi-arm robot fruit picking method provided by the present invention.

[0020] Figure 4 It is a schematic diagram of the training process of the LSTM-PPO model in the multi-arm robot fruit picking method provided by the present invention.

[0021] Figure 5 It is a structural schematic diagram of a recurrent strategy network in an LSTM-PPO model provided by the present invention.

[0022] Figure 6 It is a structural schematic diagram of a multi-arm robot fruit picking device provided by the present invention.

[0023] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] With the continuous development of robotics technology, its application in the agricultural field is becoming more and more extensive, for example, the application of robots in agricultural picking.

[0025] Compared with the traditional manual picking method, fruit (such as apples, oranges, kiwis and other fruits) picking robots have the advantages of high picking efficiency, stable picking quality, and reduced labor costs. Therefore, the application of fruit picking robots can not only effectively solve the problem of insufficient agricultural labor, but also has important significance for promoting the modernization and intelligence of agricultural production.

[0026] Traditional task planning / decision-making methods usually use numerical solutions or meta-heuristic algorithms, but numerical solutions are only applicable to situations with fewer target tasks. As the number of task targets increases, the planning time and difficulty increase exponentially, making it difficult for picking robots to meet actual needs. For example, meta-heuristic algorithms are difficult to cope with dynamic working conditions, and picking failures and accidental drops of picking targets often occur in the actual working scenarios of picking robots.

[0027] In addition, for rectangular coordinate multi-arm picking robots with overlapping working spaces, the picking behavior of one robot arm will affect the decision of another robot arm during the picking process, and the task allocation of each robot arm has an interactive relationship. Robot planning and control needs to consider both the changes in target distribution information and the balance and rationality of task allocation to each arm to ensure the overall success rate and completion rate of the picking operation. In view of the above problems faced in the prior art multi-arm collaborative planning and control, the present invention proposes a multi-arm robot fruit picking method.

[0028] The motion planning scheme of the traditional picking robot regards it as task planning. The multi-arm robot fruit picking method proposed in the present invention models the problem of two-arm collaborative picking in a three-dimensional constrained space as a path planning problem under multiple constraints in a two-dimensional space, so that it can be solved more efficiently when motion interference occurs, but this also means that the intelligent agent needs to face more complex action spaces and state spaces. Therefore, the present invention introduces the LSTM-PPO state perception network, integrates the picking environment, state perception network and reinforcement learning, enhances the robot's ability to understand complex features, realizes real-time perception and analysis of the picking environment, reduces the mutual waiting, unsatisfactory picking order, redundant and cumbersome path traversal of the robot arm actuators, etc. in the collaborative operation of each robot arm, optimizes the operation planning scheme, and thus improves the efficiency and success rate of fruit picking.

[0029] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] Combine the following Figures 1 to 7 The invention describes a multi-arm robot fruit picking method and device.

[0031] Figure 1 FIG. 1 is a flow chart of a multi-arm robot fruit picking method provided by the present invention, such as Figure 1 As shown, the method comprises the following steps: Step 101, construct a simulation environment of discrete time steps; the simulation environment is used to simulate the process of fruit picking by a multi-arm robot; each time step in the discrete time steps corresponds to the time length in the real world.

[0032] Specifically, the embodiment of the present invention takes a dual-arm robot for picking apples as an example to illustrate the scenario of a multi-arm robot in the process of fruit picking. In the embodiment of the present invention, the fruits picked are not limited to apples, but can also be a variety of fruits such as oranges and kiwis. The multi-arm robot can use multiple mechanical arms such as dual arms, triple arms, and quad arms according to actual conditions.

[0033] Figure 2 Schematic diagram of the fruit picking work area of ​​a dual-arm robot provided by the present invention. Figure 2 As shown, the horizontal working width of the dual-arm robot is 1395mm, the vertical working height is 2074mm, and there is an overlapping working area in the middle part. The embodiment of the present invention models and solves the motion path planning problem of a dual-arm robot that collaborates to complete the apple picking task in a confined workspace. Take the scenario consisting of two robotic arms with partial overlapping working spaces and several apples distributed on a fruit wall on one side as an example: the two robotic arms need to work together with high efficiency, and need to pick apples distributed in different positions and at different heights in the shortest possible time, and place them at the designated collection point as required. This scenario is subject to multiple constraints at the same time, including at least: (1) Robotic arm motion constraints: the robot’s moving speed, vertical range of motion, and the minimum safe distance between the two robots.

[0034] (2) Action timing constraints: The robot arm needs to perform actions such as approach, reach out, pick, retract, and place in a prescribed order. Each action has a fixed time cost and success condition.

[0035] (3) Fruit distribution constraint: Different robotic arms cannot select the same fruit target at the same time.

[0036] This scenario faces many problems in actual robot operation, such as mutual interference between robotic arms, coordination of action timing, and satisfaction of physical constraints. In order to solve these problems, the embodiment of the present invention designs a discrete time step simulation environment to simulate the apple picking task of dual robotic arms, so that the entire picking process is carried out on a discrete time scale.

[0037] In the simulation environment, the action execution and state changes of the robotic arm are simulated through discrete time steps, and each time step in the discrete time step is mapped to a fixed time length in the real world, so that the number of action execution steps in the simulation environment corresponds to the time length in the real world. This makes the fruit picking task performed by the robot in the simulation environment correspond to the real world, and thus makes the simulation results more reliable and accurate.

[0038] Based on the above simulation environment, many problems faced by multi-arm robots in actual operations can be modeled as a Markov decision process (MDP) and solved using reinforcement learning methods. Through agent strategy learning, various constraints and optimization goals can be met, thereby solving the motion path planning problem of multi-arm robots when picking fruit in the real world.

[0039] Step 102: Obtain the current state of the multi-arm robot in the simulation environment.

[0040] Specifically, in the training process of reinforcement learning, the state, action, and reward are called<S,A,R> . Due to the limitations of the operating environment (such as smoke, dust, and occlusion by branches and leaves of fruit trees), it is difficult to directly capture the working view of a two-arm apple picking robot (taking a two-arm robot picking apples as an example) in a confined space and its corresponding encoding. Therefore, an embodiment of the present invention adopts a feature-based state vector representation method to characterize the current state of the robot in a simulation environment. Specifically, the entire orchard space is divided into several grids, each of which may be distributed with apples, and the current position and current action state of each robotic arm are also included in the state representation.

[0041] The status includes the location of the apple , picking status (0 means not picked, 1 means picked), the position of the robot arm , the action status of the robot arm (0: idle, 1: approaching the target, 2: picking, 3: placing) and the state of the robot arm holding the apple (0: no apples, 1: apples). Therefore, the current state can be represented as a vector:

[0042] in, The number of apples that need to be picked.

[0043] The embodiment of the present invention adopts a feature-based state vector representation method to characterize the current state of the robot in a simulation environment, accurately and comprehensively describing the state of the multi-arm robot at any moment in the orchard picking task, thereby providing a basis for subsequently accurately deciding the robot's target action through the model.

[0044] Step 103: Based on the current state, the target action of the multi-arm robot in the simulation environment is obtained through the trained LSTM-PPO model; the LSTM-PPO model is constructed based on a long short-term memory network, a proximal strategy optimization algorithm and an Actor-Critic model, and is obtained through a predetermined number of iterative training.

[0045] Specifically, Figure 3 Schematic diagram of the structure of the LSTM-PPO model in the multi-arm robot fruit picking method provided by the present invention. Figure 3 shown.

[0046] The Actor-Critic model architecture is applied in the LSTM-PPO model, combining the policy (Actor) network (including the new policy network and the old policy network) and the value (Critic) network, and then constructed in combination with the long short-term memory network and the proximal policy optimization algorithm.

[0047] Among them, in the LSTM-PPO model, the policy network (Actor) is used to generate the probability distribution of actions based on the current state, and the value network (Critic) is used to evaluate the effects of these actions, measure the quality of each action, and provide feedback to the policy network to optimize the strategy.

[0048] Proximal Policy Optimization (PPO) is an algorithm based on the Trust Region Policy Gradient (TRPO) algorithm. The PPO algorithm improves stability and performance by limiting the policy update range. It has the characteristics of high stability and reliability, high sample efficiency, and wide applicability.

[0049] In order to avoid excessive updates to the policy parameters, the embodiment of the present invention introduces a variant of the PPO algorithm in the LSTM-PPO model: the PPO-Clip strategy. The PPO-Clip strategy limits the difference between the new policy network and the old policy network by introducing a clipping term in the objective function. For example, if the output of the new policy network exceeds a certain range compared to the output of the old policy network, it is clipped to ensure that the amplitude of the policy update is not too large, thereby effectively limiting the amplitude of the policy update and ensuring that the amplitude of the policy update is not too large, thereby improving the stability of the training.

[0050] The embodiment of the present invention introduces the concept of truncated ratio in the PPO algorithm to represent the trimming term to measure the relative change between the new and old strategies and defines the ratio function The expression is as follows:

[0051] in, Indicates the new strategy, represents the old policy, represents the new policy parameters, represents the old policy parameters, Indicates the action status. Indicates the current status.

[0052] Objective Function The expression is as follows:

[0053] in, is the advantage function, representing the action state The larger the function value, the higher the reward value. is the cropping parameter, limit The range of variation is Within the interval, It is the average of all the results.

[0054] The Long Short-Term Memory (LSTM) network is a special recurrent neural network (RNN) that can solve long-term dependency problems. In order to enhance the robot's memory of past states and help the robot make more reasonable decisions in a dynamic environment, the embodiment of the present invention introduces the LSTM network into the PPO algorithm. In the LSTM network, each LSTM unit consists of a forget gate, an input gate, and an output gate, which control the retention and update of information through different mechanisms. The internal structure and calculation formula of the LSTM network are as follows:

[0055]

[0056]

[0057]

[0058] in, represents the activation function, represents the output of the forget gate, represents the forget gate weight matrix, Represents the combined vector of past information and current new information, represents the hidden state at the previous time step, represents the input of the current time step, represents the forget gate bias vector, represents the input gate output at time step t, represents the input gate weight matrix, represents the input gate bias vector, represents the weight matrix for calculating the candidate unit state, represents the bias vector for calculating the candidate unit state, represents the output gate weight matrix, represents the output gate bias vector, represents the hidden state at the current time step, Represents the LSTM cell state at the current time step.

[0059] After building the LSTM-PPO model based on the long short-term memory network, the proximal policy optimization algorithm, and the Actor-Critic model, the LSTM-PPO model needs to be iteratively trained for a predetermined number of times.

[0060] Specifically, in the picking operation scenario of the dual-arm robot, the new strategy network in the LSTM-PPO model will first receive the current state information, and then process the state features in the input current state information through the LSTM network layer, and mine the time series information and dependencies, that is, according to the current apple position, picking state, robotic arm position, robotic arm action state, and robotic arm holding apple state information, calculate the probability of each possible action (such as approaching a specific apple, picking, placing, staying idle, etc.), generate an action probability distribution, and then sample specific actions from the action probability distribution, so that the robot interacts with the environment and generates corresponding new strategies.

[0061] At the same time, the old policy network in the LSTM-PPO model will perform comparative calculations for policy updates. In each iteration, the new policy generated by the new policy network is compared with the old policy represented by the old policy network, and the relative changes between the new and old policies are measured by calculating indicators such as the proportional function. Based on these calculation results, the parameters of the new policy network are adjusted and the weights are updated to achieve policy optimization.

[0062] After a predetermined number of iterations, the updated weights of the new policy network are passed to the old policy network so that the old policy network can keep up with the optimization process of the policy, thereby obtaining a trained LSTM-PPO model.

[0063] By inputting the current state of the robot (including the position of the apple, the picking state, the position of the robotic arm, the action state of the robotic arm, and the state of the robotic arm holding the apple) into the trained LSTM-PPO model, we can obtain the target action of the robot in the simulation environment output by the LSTM-PPO model, such as approaching a specific apple, picking apples, placing apples, and staying idle.

[0064] In an embodiment of the present invention, an LSTM-PPO model is constructed based on a long short-term memory network, a proximal strategy optimization algorithm and an Actor-Critic model, and the LSTM-PPO model is iteratively trained for a predetermined number of times to obtain a trained LSTM-PPO model. Then, the trained LSTM-PPO model outputs a target action according to the current state of the robot, and the task allocation of multiple robotic arms is modeled as a Markov decision process. In combination with a reinforcement learning algorithm, the picking order is determined by fully considering the distance between the robotic arm and the fruit, the number of fruits, and the interference between the operations of the robotic arms, and multiple robotic arms are coordinated to harvest the fruits in the same area. The robotic arm movements are reasonably scheduled, thereby solving the problem of overlapping operating areas of the robotic arms and improving operating efficiency.

[0065] At the same time, the value assessment network in the LSTM-PPO model will also perform a value assessment on the effect of each action, helping the policy network in the LSTM-PPO model to quickly adjust the strategy and optimize the action selection, so that the robot can continuously improve its decision-making ability in a constantly changing environment and further improve its operating efficiency.

[0066] Step 104: Based on the target action of the multi-arm robot in the simulation environment, determine a motion path of the multi-arm robot for picking fruits in the simulation environment.

[0067] Specifically, in an embodiment of the present invention, for a dual-arm robot scenario that collaborates to complete an apple picking task in a confined workspace, after obtaining the target action of the multi-arm robot in a simulation environment, the robotic arms are intelligently scheduled through a reinforcement learning algorithm to allocate the optimal picking sequence, so that the two robotic arms can work together. While one robotic arm completes a simulated grasping and placement action and returns to the starting point, the other robotic arm can use this time to pick the fruit in the overlapping area, avoiding the idle waiting time of the robotic arms and achieving efficient picking in complex working environments.

[0068] At the same time, by calling the reinforcement learning algorithm to take the target distribution and picking status as environmental information, and setting constraints such as the operating range and interference range, the vertical dimension of the space is fully utilized, so that the picking area can be covered more efficiently within a limited space. Especially for areas with densely distributed fruits, the two robotic arms can better play their roles, avoiding operational interference between the robotic arms and achieving optimal use of space.

[0069] After obtaining the target action of the multi-arm robot in a simulation environment, the embodiment of the present invention adopts path planning instead of task planning to solve the motion planning problem, and shifts the focus of solving the motion planning problem from the macroscopic decomposition and sorting of tasks to the construction of specific motion paths, thereby determining a suitable motion path to complete the task of the robotic arm to grasp the target fruit, avoiding mutual interference between the robotic arms, and improving the overall success rate and completion rate of the picking operation.

[0070] The present invention provides a multi-arm robot fruit picking method, which constructs a simulation environment with discrete time steps to simulate the process of fruit picking by the multi-arm robot, obtains the current state of the multi-arm robot in the simulation environment, and inputs the current state into a trained LSTM-PPO model, thereby modeling the task allocation of multiple robotic arms as a Markov decision process, and combines with a reinforcement learning algorithm to achieve balanced and reasonable task allocation for multiple robotic arms, obtain the target action of the multi-arm robot in the simulation environment, thereby determining the motion path of the multi-arm robot in picking fruits in the simulation environment, avoiding mutual interference between robotic arms, and improving the overall success rate and completion rate of the picking operation.

[0071] Optionally, the training step of the LSTM-PPO model includes: Acquire a first state of the multi-arm robot in the simulation environment; Based on the first state, obtaining a first action of the multi-arm robot in the simulation environment through an LSTM-PPO model; Interacting the first action with the simulation environment to obtain a second state of the multi-arm robot in the simulation environment and an instant reward corresponding to the second state; Based on the second state and the instant reward corresponding to the second state, updating the parameters of the LSTM-PPO model through the PPO-Clip optimization strategy; Repeat the iteration for a predetermined number of times to obtain the trained LSTM-PPO model.

[0072] Specifically, Figure 4 Schematic diagram of the training process of the LSTM-PPO model in the multi-arm robot fruit picking method provided by the present invention. Figure 4 shown.

[0073] In the process of iterative training of the LSTM-PPO model, the specific steps are as follows: (1) Initialize the environment and LSTM-PPO model, and obtain the first state of the multi-arm robot in the simulation environment. The first state includes the position of the apple, the picking state, the position of the robotic arm, the action state of the robotic arm, and the state of the robotic arm holding the apple. (2) The first state is input into the new policy network in the LSTM-PPO model, the probability of each possible action (such as approaching a specific apple, picking, placing, remaining idle, etc.) is calculated, the action probability distribution and the new policy are generated, and then a specific action is sampled from the action probability distribution as the first action of the multi-arm robot in the simulation environment; (3) interacting the first action with the simulation environment to obtain a second state (i.e., a new state) of the multi-arm robot in the simulation environment and an immediate reward corresponding to the second state; Figure 5 It is a schematic diagram of the structure of a cyclic strategy network in a LSTM-PPO model provided by the present invention, such as Figure 5 As shown in the figure, the recurrent policy network is a deep neural network built on the basis of the LSTM network.

[0074] In the embodiment of the present invention, a dual-arm picking robot environment is taken as an example for explanation. In the robot's picking operation scenario, the recurrent strategy network first receives the current state (i.e., the first state) information of the dual-arm robot in the simulation environment through the input layer, and then processes the state features in the input current state information through the LSTM layer and the fully connected layer, and mines the time series information and dependency relationships, that is, according to the current apple position, picking state, position of the robotic arm, action state of the robotic arm, and state of the robotic arm holding the apple, etc., calculate the probability of occurrence of each possible action (such as approaching a specific apple, picking, placing, staying idle, etc.), generate an action probability distribution, and then sample a specific action (i.e., the first action) from the action probability distribution, interact the first action with the simulation environment, and generate a corresponding strategy, thereby obtaining a new state (i.e., the second state) of the dual-arm robot in the simulation environment and an immediate reward corresponding to the second state.

[0075] (4) The first action, the second state, and the immediate reward corresponding to the second state are output to the experience pool through the output layer of the recurrent strategy network (see Figure 3 ) and store it as historical data (including historical state data and historical action data). Then, based on the historical data in the experience pool, the old policy network in the LSTM-PPO model performs comparative calculations for policy updates. Combined with the PPO-Clip optimization strategy, the new policy generated by the new policy network is compared with the old policy represented by the old policy network. The relative changes of the new and old policies are measured by calculating indicators such as the proportional function. Based on these calculation results, the parameters of the new policy network are adjusted and the weights are updated. (5) Repeat (1) to (4) above. After a predetermined number of iterations, the updated weights of the new policy network are transferred to the old policy network so that the old policy network can keep up with the optimization process of the policy, thereby continuously improving the model performance and obtaining a trained LSTM-PPO model.

[0076] The embodiment of the present invention adopts a reinforcement learning algorithm for training, so that the LSTM-PPO model can adaptively determine the picking order and actions according to factors such as the distribution of fruits and the difficulty of picking during the training process. Compared with the traditional fixed-mode robotic arm picking method, the LSTM-PPO model trained in the embodiment of the present invention is used to control the robot to pick fruits. It has stronger operational flexibility and can better cope with different picking scenarios and fruit distribution conditions, thereby improving the adaptability and reliability of the robotic arm in actual picking operations.

[0077] Optionally, the instant reward includes a movement time cost penalty, an idle state penalty, an action execution penalty, a collision penalty, a successful picking reward, and a task completion reward; The movement time cost penalty is used to reduce the movement distance and time of the multi-arm robot picking fruits; The idle state penalty is used to reduce the idle time of the mechanical arms of the multi-arm robot; The action execution penalty is used to characterize the time it takes for the multi-arm robot to execute the target action; The collision penalty is used to avoid spatial conflicts between the mechanical arms of the multi-arm robot by imposing a high penalty on the collision behavior; The successful picking reward is used to apply a positive reward to the mechanical arm of the multi-arm robot that successfully picks the target fruit; The task completion reward is used to provide an additional reward to the multi-arm robot when all target fruits are picked and placed and all the robot arms of the multi-arm robot remain idle.

[0078] Specifically, in the training process of reinforcement learning, the state, action, and reward are called<S,A,R> , specifically including state representation, action space, and reward function.

[0079] Optionally, the current state includes the position of the fruit, the picking state, the position of the robotic arms of the multi-arm robot, the action state of the robotic arms of the multi-arm robot, and the state of the robotic arms of the multi-arm robot holding the fruit.

[0080] Specifically, the current state is characterized by state representation, and the state representation uses a feature vector to represent the state of each time step, including the position of the fruit, the picking state, the position of the multi-arm robot's robotic arms, the motion state of the multi-arm robot's robotic arms, and the state of the multi-arm robot's robotic arms holding the fruit.

[0081] In this embodiment of the present invention, the position of the fruit is defined , picking status (0 means not picked, 1 means picked), the position of the multi-arm robot's mechanical arm , the action state of the multi-arm robot's mechanical arm (0: idle, 1: approaching the target, 2: picking, 3: placing) and the state of the multi-arm robot's mechanical arm holding the apple (0: no apples, 1: apples). Therefore, the current state can be represented as a vector:

[0082] in, The number of apples that need to be picked.

[0083] The embodiment of the present invention adopts a feature-based state vector representation method to characterize the current state of the robot in a simulation environment, accurately and comprehensively describing the state of the multi-arm robot at any moment in the orchard picking task, thereby helping the robot perceive environmental information and make decisions.

[0084] Optionally, the target action of the multi-arm robot in the simulation environment includes any one of approaching target fruit, picking target fruit, placing target fruit and remaining idle.

[0085] Specifically, the target action is represented by the action space. The definition of involves the discrete actions that each robot arm can perform at each time step, including approaching the target fruit, picking the target fruit, placing the target fruit, and remaining idle. Therefore, the action space can be expressed as:

[0086] in, Indicates close to Target fruit, Indicates the action of picking the target fruit. Indicates the action of placing the target fruit. Indicates no action (i.e., remains idle). In the case of a robot using two manipulators, the joint action of the two manipulators constitutes the decision of the entire system at that time step:

[0087] The embodiment of the present invention represents the target action through the action space, defines the discrete actions that each robotic arm can perform at each time step, and constitutes the action selection range of the robot, thereby satisfying the constraint conditions of the robot in the fruit picking task.

[0088] Reward Function The design aims to guide the algorithm to optimize the efficiency and quality of task completion. Includes movement time penalty , Idle state penalty , Action Execution Penalty , Collision Penalty , Rewards for successful picking And task completion rewards , the expression of immediate reward is as follows:

[0089] in: (1) Movement time penalty, which encourages robots to minimize movement distance and time and improve picking efficiency. is the weight coefficient of the moving cost:

[0090] in, , represents the time cost of robot arm i, The expression is as follows:

[0091] in, It represents the Euclidean norm of the deviation vector between the end of robot i and the target position when approaching the target fruit state, represents the time step required to pick the target fruit state, Represents the time step required to place the target fruit state.

[0092] (2) Idle state penalty: encourage the robot to stay busy as much as possible and reduce the idle time of the robot arm. =0.5, representing the penalty coefficient for the idle state:

[0093] in, Represents the action of robot arm i at time t.

[0094] (3) Action execution penalty, which represents the time cost of the robot picking the target fruit and placing the target fruit:

[0095] in, Represents the time cost penalty value of the robot arm performing the actions of picking the target fruit and placing the target fruit.

[0096] (4) Collision penalty: By imposing a high penalty on collision behavior, the robot is prompted to learn to avoid spatial conflicts between robotic arms and ensure that the robot maintains a sufficient safety distance when performing tasks. When the robot uses two robotic arms, the distance between the two robotic arms is Triggered when:

[0097] in, represents the vertical coordinate value of the first robot arm at time t, represents the ordinate value of the second robot arm at time t, represents the distance threshold, Indicates the fixed penalty value for collision.

[0098] (5) Successful picking reward: When the robot arm successfully completes the picking of a target fruit, a positive reward is given:

[0099] in, Represents the fixed reward value for successfully picking a target fruit.

[0100] (6) Task completion reward: When all target fruits are picked and placed, and all robotic arms are idle, the robot will be given an additional reward. Indicates the fixed reward coefficient for completing the task, the second item of the reward Used to encourage robots to complete tasks in the shortest possible time, task completion rewards The expression is:

[0101] in, represents the reward factor, Indicates the maximum time step of training. Indicates the actual time step to complete the task.

[0102] The embodiment of the present invention sets an instant reward , guiding the robot to optimize the efficiency and quality of task execution, improving the overall success rate and completion rate of the robot in fruit picking.

[0103] Optionally, obtaining a target action of the multi-arm robot in the simulation environment based on the current state through a trained LSTM-PPO model includes: Based on the current state of the multi-arm robot in the simulation environment, obtaining the action probability distribution of the multi-arm robot; the action probability distribution is used to characterize the probability of the multi-arm robot performing any action; Based on the action probability distribution, the target action of the multi-arm robot in the simulation environment is obtained through an importance sampling algorithm.

[0104] Specifically, in the picking operation scenario of the dual-arm robot, the new strategy network in the trained LSTM-PPO model will first receive the current state information of the dual-arm robot in the simulation environment, and then process the state features in the current state information to mine the time series information and dependencies, that is, according to the current apple position, picking state, robotic arm position, robotic arm motion state, and robotic arm holding apple state information, calculate the probability of each possible action (such as approaching a specific apple, picking, placing, staying idle, etc.), generate an action probability distribution, and finally sample specific actions (i.e., target actions) from the action probability distribution, such as approaching a specific apple, picking apples, placing apples, staying idle, etc., so as to coordinate multiple robotic arms to harvest fruits in the same area, reasonably schedule the robotic arm actions, solve the problem of overlapping robotic arm operating areas, and improve operating efficiency.

[0105] In order to improve sample efficiency, the embodiment of the present invention adopts an importance sampling algorithm to avoid resetting the empirical data and discarding previous samples after each update. The expression of the importance sampling algorithm is as follows:

[0106] Among them, the variable Follow the distribution , recorded as , represents the expected value, Corresponding to the old strategy , Corresponding to new strategies .

[0107] In the embodiment of the present invention, in order to solve , which introduces a more computationally efficient distribution into the importance sampling algorithm , so that we can distribute The sample is used to calculate expected value.

[0108] A multi-arm robot fruit picking device provided by the present invention is described below. The multi-arm robot fruit picking device described below and the multi-arm robot fruit picking method described above can be referenced to each other.

[0109] Based on any of the above embodiments, Figure 6 Schematic diagram of the structure of a multi-arm robot fruit picking device provided by the present invention. Figure 6 The embodiment of the present invention provides a multi-arm robot fruit picking device, including a simulation module 601, an acquisition module 602, an action module 603 and a path module 604, wherein: The simulation module 601 is used to construct a simulation environment of discrete time steps; the simulation environment is used to simulate the process of a multi-arm robot picking fruits; each time step in the discrete time steps corresponds to the time length in the real world; the acquisition module 602 is used to obtain the current state of the multi-arm robot in the simulation environment; the action module 603 is used to obtain the target action of the multi-arm robot in the simulation environment based on the current state through the trained LSTM-PPO model; the LSTM-PPO model is constructed based on a long short-term memory network, a proximal strategy optimization algorithm and an Actor-Critic model, and is obtained after a predetermined number of iterative training; the path module 604 is used to determine the motion path of the multi-arm robot in picking fruits in the simulation environment based on the target action of the multi-arm robot in the simulation environment.

[0110] A multi-arm robot fruit picking device provided by the present invention constructs a simulation environment with discrete time steps to simulate the process of fruit picking by the multi-arm robot, obtains the current state of the multi-arm robot in the simulation environment, and inputs the current state into a trained LSTM-PPO model, thereby modeling the task allocation of multiple robotic arms as a Markov decision process, and combines with a reinforcement learning algorithm to achieve balanced and reasonable task allocation for multiple robotic arms, obtain the target action of the multi-arm robot in the simulation environment, thereby determining the motion path of the multi-arm robot in picking fruits in the simulation environment, avoiding mutual interference between robotic arms, and improving the overall success rate and completion rate of the picking operation.

[0111] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communications interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the multi-arm robot fruit picking method, which includes: Constructing a simulation environment of discrete time steps; the simulation environment is used to simulate the process of fruit picking by a multi-arm robot; each time step in the discrete time steps corresponds to the time length in the real world; Acquire the current state of the multi-arm robot in the simulation environment; Based on the current state, the target action of the multi-arm robot in the simulation environment is obtained through the trained LSTM-PPO model; the LSTM-PPO model is constructed based on the long short-term memory network, the proximal strategy optimization algorithm and the Actor-Critic model, and is obtained through a predetermined number of iterative training; Based on the target action of the multi-arm robot in the simulation environment, a motion path of the multi-arm robot for picking fruits in the simulation environment is determined.

[0112] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0113] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the multi-arm robot fruit picking method provided by the above methods, the method comprising: Constructing a simulation environment of discrete time steps; the simulation environment is used to simulate the process of fruit picking by a multi-arm robot; each time step in the discrete time steps corresponds to the time length in the real world; Acquire the current state of the multi-arm robot in the simulation environment; Based on the current state, the target action of the multi-arm robot in the simulation environment is obtained through the trained LSTM-PPO model; the LSTM-PPO model is constructed based on the long short-term memory network, the proximal strategy optimization algorithm and the Actor-Critic model, and is obtained through a predetermined number of iterative training; Based on the target action of the multi-arm robot in the simulation environment, a motion path of the multi-arm robot for picking fruits in the simulation environment is determined.

[0114] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the multi-arm robot fruit picking method provided by the above methods, the method comprising: Constructing a simulation environment of discrete time steps; the simulation environment is used to simulate the process of fruit picking by a multi-arm robot; each time step in the discrete time steps corresponds to the time length in the real world; Acquire the current state of the multi-arm robot in the simulation environment; Based on the current state, the target action of the multi-arm robot in the simulation environment is obtained through the trained LSTM-PPO model; the LSTM-PPO model is constructed based on the long short-term memory network, the proximal strategy optimization algorithm and the Actor-Critic model, and is obtained through a predetermined number of iterative training; Based on the target action of the multi-arm robot in the simulation environment, a motion path of the multi-arm robot for picking fruits in the simulation environment is determined.

[0115] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0116] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0117] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0118] It should also be noted that the terms "target", "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0119] "Determine B based on A" in the embodiments of the present application means that the factor A should be considered when determining B. It is not limited to "B can be determined based on A alone", but should also include: "Determine B based on A and C", "Determine B based on A, C and E", "Determine C based on A, and further determine B based on C", etc. In addition, it can also include taking A as a condition for determining B, for example, "When A meets the first condition, use the first method to determine B"; for another example, "When A meets the second condition, determine B", etc.; for another example, "When A meets the third condition, determine B based on the first parameter", etc. Of course, it can also be a condition that takes A as a factor for determining B, for example, "When A meets the first condition, use the first method to determine C, and further determine B based on C", etc.

[0120] In the present invention, the term "plurality" refers to two or more than two, and other quantifiers are similar to them.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-arm robot fruit picking method, characterized in that: include: Constructing a discrete time step simulation environment; the simulation environment is used to simulate the process of fruit picking by a multi-arm robot; Each of the discrete time steps corresponds to a length of time in the real world; Acquire the current state of the multi-arm robot in the simulation environment; Based on the current state, the target action of the multi-arm robot in the simulation environment is obtained through the trained LSTM-PPO model; the LSTM-PPO model is constructed based on the long short-term memory network, the proximal strategy optimization algorithm and the Actor-Critic model, and is obtained through a predetermined number of iterative training; Based on the target action of the multi-arm robot in the simulation environment, a motion path of the multi-arm robot for picking fruits in the simulation environment is determined.

2. The multi-arm robot fruit picking method according to claim 1, characterized in that: The training steps of the LSTM-PPO model include: Acquire a first state of the multi-arm robot in the simulation environment; Based on the first state, obtaining a first action of the multi-arm robot in the simulation environment through an LSTM-PPO model; Interacting the first action with the simulation environment to obtain a second state of the multi-arm robot in the simulation environment and an instant reward corresponding to the second state; Based on the second state and the instant reward corresponding to the second state, updating the parameters of the LSTM-PPO model through the PPO-Clip optimization strategy; Repeat the iteration for a predetermined number of times to obtain the trained LSTM-PPO model.

3. The multi-arm robot fruit picking method according to claim 2, characterized in that: The instant rewards include movement time penalty, idle state penalty, action execution penalty, collision penalty, successful picking reward and task completion reward; The movement time cost penalty is used to reduce the movement distance and time of the multi-arm robot picking fruits; The idle state penalty is used to reduce the idle time of the mechanical arms of the multi-arm robot; The action execution penalty is used to characterize the time it takes for the multi-arm robot to execute the target action; The collision penalty is used to avoid spatial conflicts between the mechanical arms of the multi-arm robot by imposing a high penalty on the collision behavior; The successful picking reward is used to apply a positive reward to the mechanical arm of the multi-arm robot that successfully picks the target fruit; The task completion reward is used to provide an additional reward to the multi-arm robot when all target fruits are picked and placed and all the robot arms of the multi-arm robot remain idle.

4. The multi-arm robot fruit picking method according to claim 1, characterized in that: The obtaining of the target action of the multi-arm robot in the simulation environment based on the current state through the trained LSTM-PPO model includes: Based on the current state of the multi-arm robot in the simulation environment, obtaining the action probability distribution of the multi-arm robot; the action probability distribution is used to characterize the probability of the multi-arm robot performing any action; Based on the action probability distribution, the target action of the multi-arm robot in the simulation environment is obtained through an importance sampling algorithm.

5. The multi-arm robot fruit picking method according to claim 1, characterized in that: The current state includes the position of the fruit, the picking state, the position of the robotic arms of the multi-arm robot, the action state of the robotic arms of the multi-arm robot, and the state of the robotic arms of the multi-arm robot holding the fruit.

6. The multi-arm robot fruit picking method according to claim 1, characterized in that: The target action of the multi-arm robot in the simulation environment includes any one of approaching the target fruit, picking the target fruit, placing the target fruit and remaining idle.

7. A multi-arm robot fruit picking device, characterized in that: include: A simulation module is used to construct a discrete time step simulation environment; the simulation environment is used to simulate the process of fruit picking by a multi-arm robot; Each of the discrete time steps corresponds to a length of time in the real world; An acquisition module, used for acquiring the current state of the multi-arm robot in the simulation environment; An action module, used to obtain the target action of the multi-arm robot in the simulation environment based on the current state through a trained LSTM-PPO model; the LSTM-PPO model is constructed based on a long short-term memory network, a proximal policy optimization algorithm and an Actor-Critic model, and is obtained through a predetermined number of iterative training; A path module is used to determine a motion path of the multi-arm robot for picking fruits in the simulation environment based on a target action of the multi-arm robot in the simulation environment.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the multi-arm robot fruit picking method as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-arm robot fruit picking method as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the multi-arm robot fruit picking method as described in any one of claims 1 to 6 is implemented.

Images