Picking mechanical arm control method based on density clustering and agent integration

Abstract

The invention belongs to the technical field of mechanical arm motion planning control, and discloses a picking mechanical arm control method based on density clustering and agent integration, which comprises the following steps: firstly, simplifying a picking scene by using a DBSCAN algorithm, and determining an objective function, a constraint condition, an input state space, an action space, a reward function and a termination condition of an agent in a picking scene model; the method comprises the following steps: selecting a DBSCAN algorithm clustering result, preparing a training agent based on the DBSCAN algorithm clustering result, then training the selected agent algorithm, and finally realizing motion control decision integration of each trained agent algorithm for the same mechanical arm by using an error reciprocal weighted combination method based on agent rewards, and accumulating a total reward value after integration processing. The method is suitable for control over the picking mechanical arm, multi-angle decision making is achieved, the defect of a single algorithm is effectively overcome, the precision of movement of the mechanical arm is improved, and therefore reliable support is provided for the modern agricultural automation technology.

Description

Technical Field

[0001] The invention belongs to the technical field of robot arm motion planning and control, and relates to a picking robot arm control method based on density clustering and intelligent body integration. Background Art

[0002] In agricultural production, picking is usually done manually, but with the increase in labor costs and the increase in agricultural product output, manual picking is time-consuming and labor-intensive, and it is difficult to improve efficiency. Therefore, in order to improve picking efficiency and reduce labor costs, picking robotic arms are gradually being used to replace manual picking.

[0003] With the continuous improvement of social science and technology, the research and application of robotic arm picking technology has increased, and it has gradually developed in the direction of mechanization, automation, and intelligence, playing an increasingly important role in modern agriculture. However, faced with complex and changing agricultural environments, such as differences in crop growth cycles, uneven fruit distribution, and other unfavorable factors, the accuracy and efficiency of the motion control of the harvesting robot arm have become key challenges. Traditional methods of motion control of harvesting robots mostly rely on a single control algorithm or strategy, which often seems inadequate when dealing with complex and changing agricultural scenes, and is prone to invalid actions or even accidental damage to crops, which in turn affects the economic benefits and sustainability of agricultural production.

[0004] Density-Based Spatial Clustering of Applications with Noise (DBSCAN) is a density-based spatial clustering algorithm that defines the density of the sample space according to the number of neighbors in a certain neighborhood of the sample point. This type of algorithm can find irregular clusters in space without specifying the number of clusters. The DBSCAN algorithm can be used in the motion control of the picking robot arm to cluster the spatial density of the picking target points, thereby effectively simplifying the environmental model and reducing data complexity. On this basis, multiple control algorithms are integrated to achieve multi-dimensional control decisions to achieve efficient fruit picking. Therefore, providing a picking robot arm control method based on density clustering and agent integration has important practical significance. Summary of the invention

[0005] The purpose of the present invention is to provide a harvesting robot arm control method based on density clustering and agent integration. The complex agricultural picking scene is simplified by adopting the DBSCAN algorithm, and the decision results of each agent algorithm are dynamically weighted and integrated by using the inverse error weighted combination method based on agent reward, so as to achieve complementary advantages between algorithms and realize accurate and efficient control of the robot arm movement.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A harvesting robot arm control method based on density clustering and agent integration includes the following steps:

[0008] S1. Apply the DBSCAN algorithm to cluster the spatial density of the coordinate points of the picked targets, obtain the clustering results of the DBSCAN algorithm, and simplify the picking scene;

[0009] S2, determine the objective function, constraints, input state space, action space, reward function and termination condition of the agent in the picking scene model, and start preparing to train the agent based on the clustering results of the DBSCAN algorithm;

[0010] The intelligent agent is a robotic arm integrated with a reinforcement learning algorithm;

[0011] S3, selecting at least two intelligent agent algorithms, and training the selected intelligent agent algorithms according to the objective function, constraint conditions, input state space, action space, reward function and termination condition of the intelligent agent obtained in step S2 to obtain each trained intelligent agent algorithm;

[0012] S4. Use the inverse error weighted combination method based on agent rewards to obtain the decision weights of each trained agent algorithm, and implement the weight distribution of each trained agent algorithm, weightedly integrate the decision output results of each trained agent algorithm, implement the motion control decision integration of each trained agent algorithm for the same robotic arm and accumulate the total reward value after integration processing.

[0013] As a limitation, in step S2, the objective function of the agent is:

[0014]

[0015] Among them, c is the number of missed targets. is the total time for the robot to pick all the picked objects, and θ is the damage rate of the picked objects;

[0016] The constraints of the agent are the movement angle, movement speed, and movement range of each joint of the robot arm;

[0017] The input state space of the agent is:

[0018] [dst1,dst2,…,dst i ,x t ,y t ,z t ](2)

[0019] Among them, dst i is the distance between the end of the robot arm and the picked target i, i ≥ 1, x t ,y t、z t is the spatial coordinate of the end of the robotic arm at the current moment;

[0020] The action space of the agent is:

[0021] Action=[angle1,angle2,…,angle m ] (3)

[0022] Among them, angle m is the rotation angle of the mth robot joint, m is the total number of robot joints, m≥1;

[0023] The agent's reward function is obtained based on dense rewards and sparse rewards. The agent's dense reward is:

[0024]

[0025] Among them, x i ,y i 、z i is the spatial coordinate of the i-th picked target; β l is the guidance weight of the density region l where the picked target i is located, n is the average spatial height ranking of the density region l, q is the number of density regions after clustering, and r t is the dense reward for the agent;

[0026] The sparse reward of the agent is:

[0027]

[0028] Among them, r s is a one-time reward for successfully picking the picked target, a is a positive integer, r p is a one-time penalty for destroying the picked target, and b is a negative integer;

[0029] The reward function of the agent is obtained as:

[0030] R t =r t +r s +r p (8)

[0031] The termination condition of the agent is:

[0032]

[0033] Among them, t2 is the time spent on picking operation, t 2max The maximum time allowed for picking operations, The number of remaining picked objects in the picking scene.

[0034] As a second limitation, the specific process in step S3 is:

[0035] S31, algorithm selection: select at least two intelligent agent algorithms;

[0036] S32, independent training: for the selected agent algorithm, according to the picking scene model, using the agent's objective function, constraints, input state space, action space, reward function and termination condition, independent training is performed to enable each agent algorithm to learn and adapt to the given picking scene;

[0037] S33, parameter optimization: adjust the parameters of each agent algorithm;

[0038] S34, initializing decision weights: After the intelligent agent algorithm is trained, an initial decision weight is set for each intelligent agent algorithm. The initial decision weight is set according to the reward size distribution during the picking scene model verification to obtain each trained intelligent agent algorithm.

[0039] As a limitation, in step S31, the agent algorithm is the PPO algorithm, the TD3 algorithm, the DDPG algorithm, the SAC algorithm, the A2C algorithm and the A3C algorithm.

[0040] As a further limitation, in step S33, the parameters of the agent algorithm include learning rate, decay factor, network architecture parameters, training rounds and reward discount factor.

[0041] As a limitation, the specific process of step S4 is:

[0042] S41, arranging the historical rewards obtained by each trained intelligent agent algorithm in descending order to obtain a performance ranking of each trained intelligent agent algorithm;

[0043] S42, accumulating the historical rewards obtained by each trained agent algorithm to obtain the total reward value of all trained agent algorithms as the denominator of the weight;

[0044] S43, obtaining the decision weights of the trained agent algorithms, and implementing weight allocation of the trained agent algorithms;

[0045] S44, weighted integration of the decision output results of each trained intelligent agent algorithm, to achieve the integration of the motion control decisions of each trained intelligent agent algorithm for the same robotic arm and accumulate the total reward value after the integration processing;

[0046] As a further limitation, in step S42, the total reward value of all trained agent algorithms is:

[0047]

[0048] Among them, Sum(r) is the total reward value of all trained agent algorithms, u is the total number of agent algorithms, and r k is the historical reward value obtained by the k-th agent algorithm, k∈u, and T is the set historical time scale.

[0049] As a further limitation, in step S43, the decision weight of each trained body algorithm is:

[0050]

[0051] Among them, ω d is the weight of the trained agent algorithm ranked d after the historical rewards are sorted in descending order, r u-d+1 It is the historical rewards obtained by the algorithm ranked u-d+1 after sorting the historical rewards in descending order.

[0052] As a further limitation, in step S44, the decision output result of the weighted integration of the trained agent algorithms is:

[0053]

[0054] in, Output results for actions after integrating multiple agent algorithms;

[0055] The motion control decision set for the same robotic arm by each trained agent algorithm is:

[0056]

[0057] The total reward value after cumulative integration is:

[0058]

[0059] in, is the total reward value after the integrated processing of each trained agent algorithm, is the dense reward obtained by the agent algorithm at the current moment, is the sparse reward obtained by the agent algorithm at the current moment, is the penalty that the agent algorithm receives for exceeding the constraint at the current moment.

[0060] Due to the adoption of the above technical solution, the present invention has the following technical advances compared with the prior art:

[0061] (1) Aiming at the operational characteristics of complex agricultural picking scenes, the present invention adopts the DBSCAN algorithm to perform spatial density clustering of picked objects, thereby effectively simplifying the picking environment model and improving the training and operation efficiency of the intelligent agent;

[0062] (2) The present invention designs the objective function, constraint conditions, input state space, action space, reward function and termination condition of the intelligent agent for the simplified picking scene model, and trains the intelligent agent based on the clustering results of the DBSCAN algorithm, which effectively improves the work efficiency;

[0063] (3) In order to reasonably allocate the decision weights of each agent algorithm, the present invention proposes an error inverse weighted combination method based on agent rewards. This method dynamically adjusts the weights of each agent according to its historical performance (i.e., historical rewards) in actual operations. This overcomes the problem that traditional methods that rely on error weighting are difficult to generalize and apply to the field of reinforcement learning. This ensures that the decisions of excellent agent algorithms are given greater attention, while retaining the decision reference value of other agent algorithms, thereby improving the practicality and effectiveness of the overall integrated strategy.

[0064] (4) The present invention takes into account the complementarity of intelligent agents, selects multiple intelligent agents with large algorithm differences, and uses the reward-based inverse error combination method to weightedly integrate them together, thereby achieving the effect of complementary advantages, alleviating the shortcomings of the existing control that relies on a single method, such as instability and easy to fall into local optimal solutions, and improving the control effect.

[0065] In summary, the present invention is suitable for the control of harvesting robot arms, realizes multi-angle decision-making, effectively makes up for the defects of a single algorithm, improves the accuracy and operating efficiency of the robot arm movement, and thus provides reliable support for modern agricultural automation technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 Shown is a flow chart of the overall method of an embodiment of the present invention;

[0067] Figure 2 The method flow chart of step S4 of the embodiment of the present invention is shown;

[0068] Figure 3 The figure shows a Pybullet simulation test environment diagram of a harvesting robot arm according to an embodiment of the present invention;

[0069] Figure 4 Shown is a clustering result diagram of the picking target coordinate points using the DBSCAN algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0070] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below through specific implementation modes in conjunction with the accompanying drawings.

[0071] Embodiment A harvesting robot arm control method based on density clustering and agent integration

[0072] like Figure 1As shown, this embodiment is a harvesting robot arm control method based on density clustering and agent integration, comprising the following steps:

[0073] S1. Apply the DBSCAN algorithm to cluster the spatial density of the coordinate points of the picked targets, obtain the clustering results of the DBSCAN algorithm, and simplify the picking scene;

[0074] In complex agricultural picking scenarios, since the size, height, location, and distribution density of crop fruits are inconsistent, the training cost and instability probability of the intelligent agent are greatly increased. Therefore, spatial density clustering analysis of the three-dimensional spatial coordinate points of the picked targets can be performed to obtain several high-density to low-density operation areas. The clustered density areas directly reflect the concentration and sparseness of the picked targets in spatial distribution, thereby simplifying the application scenario, reducing data complexity, and providing a data basis for the subsequent picking control of the robotic arm;

[0075] In this step, the DBSCAN algorithm is based on two main parameters: scanning radius (eps) and minimum number of included points (minPts). These parameters need to be adjusted according to the specific operation scenario. The specific steps of using the DBSCAN algorithm are as follows:

[0076] 1. Initialization: Select a point from the unvisited target coordinate points as the starting point;

[0077] 2. Neighborhood query and cluster formation:

[0078] Calculate the distance between the starting point and all other points, and find all points whose distance to the starting point is within (including) eps; if the number of these points is greater than or equal to minPts, then consider the starting point and its neighboring points as preliminary members of a cluster and mark these points as visited; repeat the above process for each unvisited point in the cluster to expand the range of the cluster;

[0079] 3. Noise point processing:

[0080] If the number of neighboring points of a point is less than minPts, the point is considered a noise point and no further processing is performed;

[0081] To reduce computing power and improve efficiency, set minPts to 5 and eps to 0.5 according to the scenario in this step;

[0082] 4. Continue clustering:

[0083] When a cluster is fully expanded (i.e., all points in the cluster are marked as visited), select the next unvisited point as the new starting point, and repeat steps 2 to 4 until all target point coordinates are processed;

[0084] 5. Output results:

[0085] After the algorithm is finished, all the formed clusters and the points identified as noise will be output. These results reflect the cluster distribution of the picked target coordinate points in space;

[0086] S2. Determine the objective function, constraints, input state space, action space, reward function and termination condition of the intelligent agent in the picking scene model, and start preparing to train the intelligent agent based on the clustering results of the DBSCAN algorithm; wherein the intelligent agent is a robotic arm integrated with a reinforcement learning algorithm;

[0087] In the reinforcement learning environment, the effectiveness of the agent is affected by many factors. The design of its objective function, constraints, input state space, action space, reward function and termination conditions is particularly critical. In order to determine the above conditions, this step combines the simplified picking scene and the inherent parameters of the robot arm with the DBSCAN algorithm clustering to analyze the specific design as follows:

[0088] 1. In the agricultural picking scenario, the picking completion rate, operation efficiency and the degree of damage to the picked target are important indicators. Therefore, the objective function of the designed agent is to minimize the time taken for picking and the loss rate of the picked target. The objective function of the agent is:

[0089]

[0090] Among them, c is the number of missed targets. is the total time for the robot to pick all the picked objects, and θ is the damage rate of the picked objects;

[0091] 2. Constraints of the Agent:

[0092] According to the robot model used, its movement must meet the physical parameter settings of the relevant structure, such as movement angle, movement speed, etc., and limit the movement range of the robot in combination with the specific environment; among them, the robot model selects the KUKA robot with 7 degrees of freedom;

[0093] 3. The input state space of the agent is the data support for the agent to perceive the environment. The input state space of the agent is:

[0094] [dst1,dst2,…,dst i ,x t ,y t ,z t ](2)

[0095] Among them, dst i is the distance between the end of the robot arm and the picked target i, i ≥ 1, x t ,y t 、zt is the spatial coordinate of the end of the robotic arm at the current moment;

[0096] 4. The action of the robot agent can be defined as the rotation angle of each robot joint, and the value range of each robot joint rotation angle can be mapped to the [-1,1] interval. The action space of the agent is:

[0097] Action=[angle1,angle2,…,angle m ] (3)

[0098] Among them, angle m is the rotation angle of the mth robot joint, m is the total number of robot joints, m≥1;

[0099] 5. The reward function of the agent is an important reference for the agent to evaluate and optimize the effectiveness of its own strategy. The traditional reward function design of the harvesting robot usually ignores the density of the target area, which makes the agent pick crops randomly, which not only increases the training burden of the agent, but also affects the work efficiency. Therefore, the reward function is improved;

[0100] In this step, the agent's reward function is obtained based on the dense reward and sparse reward. The agent's dense reward is:

[0101]

[0102] Among them, x i ,y i 、z i is the spatial coordinate of the i-th picked target; β l is the guidance weight of the density region l where the picked target i is located, n is the average spatial height ranking of the density region l, q is the number of density regions after clustering, and r t is the dense reward for the agent;

[0103] The sparse reward of the agent is:

[0104]

[0105] Among them, r s is a one-time reward for successfully picking the picked target, a is a large positive integer, the range of a is [+1, +10], r p is a one-time penalty for destroying the picked target, b is a large negative integer, the range of b is [-10, -1], a and b have fixed values ​​within the range and are opposite to each other;

[0106] The reward function of the agent is obtained as:

[0107] R t=r t +r s +r p (8)

[0108] 6. If the time taken for the picking operation exceeds the maximum time allowed for the picking operation or the picking task is completed, the process terminates. The termination condition of the agent is:

[0109]

[0110] Among them, t2 is the time spent on picking operation, t 2max The maximum time allowed for picking operations, is the number of remaining picked objects in the picking scene;

[0111] S3, selecting at least two intelligent agent algorithms, and training the selected intelligent agent algorithms according to the objective function, constraint conditions, input state space, action space, reward function and termination condition of the intelligent agent obtained in step S2 to obtain each trained intelligent agent algorithm;

[0112] The specific process of this step is:

[0113] S31. Algorithm selection: Select at least two intelligent agent algorithms, where the selected intelligent agent algorithms need to be significantly different in principle and perform well in the current application scenario. In practical applications, the intelligent agent algorithms use at least two of the PPO algorithm, TD3 algorithm, DDPG algorithm, SAC algorithm, A2C algorithm and A3C algorithm;

[0114] In this step, according to the computing power level of the computer and the performance of the agent, the agent algorithm selects the PPO algorithm, DDPG algorithm, and SAC algorithm, which can complement each other when integrated to improve the overall performance;

[0115] S32, independent training: for the selected agent algorithm, according to the picking scene model, using the agent's objective function, constraints, input state space, action space, reward function and termination condition, independent training is performed to enable each agent algorithm to learn and adapt to the given picking scene;

[0116] In this step, based on the selected intelligent agent algorithms PPO algorithm, DDPG algorithm and SAC algorithm, and the robot arm is a 7-DOF robot arm of KUKA, the constraints are set according to its specific physical parameters and operation scenarios;

[0117] Among them, the objective function, input state space, and action space settings of the agent do not need to be changed, and the constraints, reward function, and termination conditions of the agent need to be adjusted specifically in combination with the scenario of this embodiment;

[0118] Combined with the specific operation scenario, the intelligent body constraint conditions are specifically set as follows: the minimum motion angle values ​​that the 7 joints of the robot arm can reach are: -.967°, -2°, -2.96°, 0.19°, -2.96°, -2.09°, -3.05°; the maximum motion angle values ​​that the 7 joints of the robot arm can reach are 967°, 2°, 2.96°, 2.29°, 2.96°, 2.09°, 3.05°; the robot arm movement speed is 0.1m / s. The coordinate movement range that the end of the robot arm or its joints can reach is the x-coordinate [-1,1], y-coordinate [-1,1], and z-coordinate [0.1,1]. In this embodiment, the height of the desktop (simulating the ground) is 0.1m, so the minimum z-coordinate reached by the robot arm is set to 0.1;

[0119] The agent's reward function: After optimization, the agent's sparse reward is specifically set to:

[0120]

[0121] Termination condition of the agent: it will terminate immediately after all crops are picked or the robot arm exceeds the set constraints;

[0122] S33, parameter optimization: adjust the parameters of each agent algorithm, including learning rate, decay factor, network architecture parameters, training rounds and reward discount factor, as shown in Table 1, to further improve the stability and generalization ability of the algorithm;

[0123] Table 1 Parameters of each agent algorithm

[0124]

[0125]

[0126] S34, initializing decision weights: after completing the training of the intelligent agent algorithm, initial decision weights are set for the PPO algorithm, the DDPG algorithm, and the SAC algorithm respectively. The initial decision weights are set according to the reward size distribution during the verification of the picking scene model to obtain each trained intelligent agent algorithm;

[0127] S4. Use the inverse error weighted combination method based on agent rewards to obtain the decision weights of each trained agent algorithm, and implement the weight distribution of each trained agent algorithm, weightedly integrate the decision output results of each trained agent algorithm, implement the motion control decision integration of each trained agent algorithm for the same robot arm and accumulate the total reward value after integration processing, such as Figure 2 As shown;

[0128] In order to control the same robotic arm according to the trained PPO algorithm, DDPG algorithm, and SAC algorithm, the specific process of this step is as follows:

[0129] S41, arranging the historical rewards obtained by each trained intelligent agent algorithm in descending order to obtain a performance ranking of each trained intelligent agent algorithm;

[0130] S42, accumulating the historical rewards obtained by each trained agent algorithm to obtain the total reward value of all trained agent algorithms as the denominator of the weight;

[0131] The total reward value of all trained agent algorithms is:

[0132]

[0133] Among them, Sum(r) is the total reward value of all trained agent algorithms, u is the total number of agent algorithms, and r k is the historical reward value obtained by the k-th agent algorithm, k∈u, T is the set historical time scale;

[0134] S43, obtaining the decision weights of the trained agent algorithms, and implementing weight allocation of the trained agent algorithms;

[0135] The decision weights of each trained algorithm are:

[0136]

[0137] Among them, ω d is the weight of the trained agent algorithm ranked d after the historical rewards are sorted in descending order, r u-d+1 The historical rewards obtained by the algorithm ranked u-d+1 after sorting the historical rewards in descending order;

[0138] From this, we can derive the decision weights of each trained agent algorithm, and the more historical rewards each trained agent algorithm obtains, the higher the weight assigned to it will be, and the sum of the weights of each trained agent algorithm is 1;

[0139] S44, weighted integration of the decision output results of each trained intelligent agent algorithm, to achieve the integration of the motion control decisions of each trained intelligent agent algorithm for the same robotic arm and accumulate the total reward value after the integration processing;

[0140] The decision output of the weighted integration of the trained agent algorithms is:

[0141]

[0142] in, Output results for actions after integrating multiple agent algorithms;

[0143] The motion control decision set for the same robotic arm by each trained agent algorithm is:

[0144]

[0145] Finally, the total reward value after cumulative integration is:

[0146]

[0147] in, is the total reward value after the integrated processing of each trained agent algorithm, is the dense reward obtained by the agent algorithm at the current moment, is the sparse reward obtained by the agent algorithm at the current moment, is the penalty that the agent algorithm receives for exceeding the constraint at the current moment.

[0148] In order to verify this embodiment, the PPO-DDPG-SAC integrated control method without the DBSCAN clustering algorithm is compared with the PPO algorithm, DDPG algorithm, and SAC algorithm without DBSCAN in terms of evaluation indicators, and the PPO-DDPG-SAC integrated control method using the DBSCAN clustering algorithm in this embodiment is compared with the PPO algorithm, DDPG algorithm, and SAC algorithm using DBSCAN in terms of evaluation indicators.

[0149] 1. The evaluation indicators of the PPO-DDPG-SAC integrated control method without the DBSCAN clustering algorithm are compared with those of the PPO algorithm, DDPG algorithm, and SAC algorithm without DBSCAN.

[0150] Taking marigold picking as an application scenario, Figure 3 As shown in the figure, this example uses the Pybullet simulation environment to simulate the picking process of 18 marigold flowers of different heights, sizes and positions for 100 rounds. In each round, except for the number of marigolds, their height, size and position change with the round. In this simulation, the cylinder is used as a substitute model for marigolds. When the robot arm touches the top of the cylinder, it is considered to be picked successfully; if it touches other parts of the cylinder, it is considered to be crop damage. When the DBSCAN clustering algorithm is not used to simplify the picking scene, the final evaluation index results of each picking method are shown in Table 2. Observing the data in Table 2, it can be found that when the PPO-DDPG-SAC integrated control method is used, the crop damage rate is significantly reduced and the picking time is significantly shortened.

[0151] Table 2 Comparison of evaluation indicators of methods without using DBSCAN in the embodiments of the present invention

[0152] method Average crop damage rate / % Average picking completion time / s PPO 19.16 52.05 DDPG 15.84 42.07 SAC 17.97 40.02 PPO-DDPG-SAC integrated control 11.58 38.75

[0153] 2. This embodiment uses the PPO-DDPG-SAC integrated control method using the DBSCAN clustering algorithm to compare evaluation indicators with the PPO algorithm, DDPG algorithm, and SAC algorithm using DBSCAN.

[0154] Based on the original marigold picking example, this example further combines the DBSCAN clustering algorithm to simplify the picking scene. The clustering results are shown in the figure below. Figure 4 As shown in the figure, the clustering is divided into three categories: black, gray, and white-gray. On this basis, the intelligent agent completes the example picking task, and the final evaluation index results of each picking method are shown in Table 3.

[0155] Table 3 Comparison of evaluation indicators of the methods in the embodiments of the present invention

[0156] method Average crop damage rate / % Average picking completion time / s DBSCAN-PPO 15.09 49.66 DBSCAN-DDPG 12.73 38.57 DBSCAN-SAC 13.58 35.02 Method of this embodiment (DBSCAN-PPO-DDPG-SAC) 8.14 32.75

[0157] By analyzing the data in Tables 2 and 3, it can be clearly observed that after adopting the DBSCAN clustering algorithm, both the crop damage rate and the picking time, two key evaluation indicators, have achieved significant improvements. This result shows that the effective application of the DBSCAN clustering algorithm can optimize the picking scene and improve the overall performance of the picking operation.

[0158] Given that the effectiveness of the integrated strategy often depends on the complementarity between the algorithms, if in the latest environment, the total reward after integration fails to exceed the reward achieved by any single agent algorithm, it means that the current complementary integration is not possible. Based on the principle of historical best performance, the best performing single agent algorithm is intelligently selected and switched to perform the control task. Otherwise, the current integrated strategy is maintained and the advantages of multi-agent integration are continued to be used for decision-making.

[0159] It should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention is described in detail with reference to the above embodiments, those skilled in the art can still modify the technical solutions described in the above embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A harvesting robot arm control method based on density clustering and agent integration, characterized in that: The following steps are involved: S1. Apply the DBSCAN algorithm to cluster the spatial density of the coordinate points of the picked targets, obtain the clustering results of the DBSCAN algorithm, and simplify the picking scene; S2, determine the objective function, constraints, input state space, action space, reward function and termination condition of the agent in the picking scene model, and start preparing to train the agent based on the clustering results of the DBSCAN algorithm; The intelligent agent is a robotic arm integrated with a reinforcement learning algorithm; S3, selecting at least two intelligent agent algorithms, and training the selected intelligent agent algorithms according to the objective function, constraint conditions, input state space, action space, reward function and termination condition of the intelligent agent obtained in step S2 to obtain each trained intelligent agent algorithm; S4. Use the inverse error weighted combination method based on agent rewards to obtain the decision weights of each trained agent algorithm, and implement the weight distribution of each trained agent algorithm, weightedly integrate the decision output results of each trained agent algorithm, implement the motion control decision integration of each trained agent algorithm for the same robotic arm and accumulate the total reward value after integration processing.

2. The harvesting robot arm control method based on density clustering and agent integration according to claim 1 is characterized in that: In step S2, the objective function of the agent is: Among them, c is the number of missed targets. is the total time for the robot to pick all the picked objects, and θ is the damage rate of the picked objects; The constraints of the agent are the movement angle, movement speed, and movement range of each joint of the robot arm; The input state space of the agent is: [dst1,dst2,…,dst i ,x t ,y t ,z t ](2) Among them, dst i is the distance between the end of the robot arm and the picked target i, i ≥ 1, x t ,y t 、z t is the spatial coordinate of the end of the robotic arm at the current moment; The action space of the agent is: Action=[angle1,angle2,…,angle m ] (3) Among them, angle m is the rotation angle of the mth robot joint, m is the total number of robot joints, m≥1; The agent's reward function is obtained based on dense rewards and sparse rewards. The agent's dense reward is: Among them, x i ,y i 、z i is the spatial coordinate of the i-th picked target; β l is the guidance weight of the density region l where the picked target i is located, n is the average spatial height ranking of the density region l, q is the number of density regions after clustering, and r t is the dense reward for the agent; The sparse reward of the agent is: Among them, r s is a one-time reward for successfully picking the picked target, a is a positive integer, r p is a one-time penalty for destroying the picked target, and b is a negative integer; The reward function of the agent is obtained as: R t ＝r t +r s +r p (8) The termination condition of the agent is: Among them, t2 is the time spent on picking operation, t 2max The maximum time allowed for picking operations, The number of remaining picked objects in the picking scene.

3. The harvesting robot arm control method based on density clustering and agent integration according to claim 1 is characterized in that: The specific process in step S3 is: S31, algorithm selection: select at least two intelligent agent algorithms; S32, independent training: for the selected agent algorithm, according to the picking scene model, using the agent's objective function, constraints, input state space, action space, reward function and termination condition, independent training is performed to enable each agent algorithm to learn and adapt to the given picking scene; S33, parameter optimization: adjust the parameters of each agent algorithm; S34, initializing decision weights: After the intelligent agent algorithm is trained, an initial decision weight is set for each intelligent agent algorithm. The initial decision weight is set according to the reward size distribution during the picking scene model verification to obtain each trained intelligent agent algorithm.

4. The harvesting robot arm control method based on density clustering and agent integration according to claim 3 is characterized in that: In step S31, the agent algorithm is PPO algorithm, TD3 algorithm, DDPG algorithm, SAC algorithm, A2C algorithm and A3C algorithm.

5. The harvesting robot arm control method based on density clustering and agent integration according to claim 3 is characterized in that: In step S33, the parameters of the agent algorithm include learning rate, decay factor, network architecture parameters, training rounds and reward discount factor.

6. The harvesting robot arm control method based on density clustering and agent integration according to claim 2 is characterized in that: The specific process of step S4 is: S41, arranging the historical rewards obtained by each trained intelligent agent algorithm in descending order to obtain a performance ranking of each trained intelligent agent algorithm; S42, accumulating the historical rewards obtained by each trained agent algorithm to obtain the total reward value of all trained agent algorithms as the denominator of the weight; S43, obtaining the decision weights of the trained agent algorithms, and implementing weight allocation of the trained agent algorithms; S44, weighted integration of the decision output results of each trained intelligent agent algorithm, to achieve the integration of the motion control decisions of each trained intelligent agent algorithm for the same robotic arm and accumulate the total reward value after the integration processing.

7. The harvesting robot arm control method based on density clustering and agent integration according to claim 6 is characterized in that: In step S42, the total reward value of all trained agent algorithms is: Among them, Sum(r) is the total reward value of all trained agent algorithms, u is the total number of agent algorithms, and r k is the historical reward value obtained by the k-th agent algorithm, k∈u, and T is the set historical time scale.

8. The harvesting robot arm control method based on density clustering and agent integration according to claim 6 is characterized in that: In step S43, the decision weights of each trained algorithm are: Among them, ω d is the weight of the trained agent algorithm ranked d after the historical rewards are sorted in descending order, r u-d+1 It is the historical rewards obtained by the algorithm ranked u-d+1 after sorting the historical rewards in descending order.

9. The harvesting robot arm control method based on density clustering and agent integration according to claim 6 is characterized in that: In step S44, the decision output result of the weighted integration of the trained agent algorithms is: in, Output the action results after integrating multiple agent algorithms; The motion control decision set for the same robotic arm by each trained agent algorithm is: The total reward value after cumulative integration is: in, is the total reward value after the integrated processing of each trained agent algorithm, is the dense reward obtained by the agent algorithm at the current moment, is the sparse reward obtained by the agent algorithm at the current moment, is the penalty that the agent algorithm receives for exceeding the constraint at the current moment.

Images