Mechanical arm continuous picking sequence optimization method under visual guidance

Abstract

The invention provides a mechanical arm continuous picking sequence optimization method under visual guidance, and relates to the field of intelligent picking equipment, and the method comprises the steps: S1, firstly, constructing a spatial heterogeneous visual positioning system, and carrying out the target recognition and positioning of tomato fruits; then, three-dimensional coordinates of tomato fruit centroids are obtained, and a coordinate set of the tomatoes capable of being picked is obtained; s2, determining an objective function of an optimal path for the space traveling salesman problem, and performing optimization through an improved sparrow algorithm; and S3, starting a picking mechanical arm to realize sequential harvesting. According to the method, the optimal picking sequence of the tomato fruits can be effectively obtained, the purposes of reducing the length of the motion path of the mechanical arm and prolonging the response time are achieved, and therefore continuous picking of dwarf and close planting type fruit and vegetable crops with high efficiency, high precision, high success rate, low cost and low time consumption is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent picking equipment, and in particular to a method for optimizing a continuous picking sequence of a robotic arm under vision guidance. Background Art

[0002] Harvesting is an important part of the fruit and vegetable planting process, which is characterized by strong seasonality and labor-intensiveness. In recent years, with the gradual expansion of tomato planting in Xinjiang, the amount of labor required for tomato harvesting has also increased year by year. The traditional manual picking method can no longer meet the development needs of modern orchards, and its shortcomings of long time consumption, high labor costs and low harvesting efficiency are becoming more and more obvious.

[0003] At present, in order to cope with the problems of population aging and reduced labor productivity, intelligent picking methods that use picking robots to replace traditional manual picking have gradually been applied in various fruit and vegetable product planting fields; at the same time, in order to ensure the accuracy and efficiency of fruit harvesting, a large number of experts and scholars have conducted in-depth research on the picking problem in orchards. For example, "YOLO-C: An Efficient and Robust Detection Algorithm for Mature Long Staple Cotton Targets with High-Resolution RGB Images" (Liang Z, Cui G, Xiong M, et al. Agronomy, 2023, 13(8): 1988.) proposed a YOLO-C algorithm that combines deformable convolution and attention mechanism, which improves the accuracy of crop recognition in complex environments; "Visual localization and picking posture estimation method of tomato bunches based on instance segmentation" (Zhang Qin, Pang Yuesheng, Li Bin. Transactions of the Chinese Society of Agricultural Machinery, 2023, 54(10): 205-215.) The coordinates of the picking points are accurately located based on the YOLACT instance segmentation algorithm, thereby effectively improving the positioning accuracy of the coordinates of the picking point images. However, when the picking robot is used to pick multiple target tomatoes in a row, the existing picking robot cannot construct the optimal picking sequence, which results in low picking success rate and long planning path in the intelligent picking process. It is not only easy to cause problems such as wrong picking and missed picking, causing damage to fruits and fruit trees, but also easy to reduce picking efficiency, increase picking time, and increase harvesting costs. Summary of the invention

[0004] In view of the problems existing in the above-mentioned prior art, the object of the present invention is to provide a method for optimizing the continuous picking sequence of a robotic arm under vision guidance. The method obtains the spatial position of tomatoes by establishing a spatial heterogeneous vision system, and at the same time constructs a picking distance fitness function. The improved sparrow algorithm guided by vision is used to obtain the optimal picking sequence, so as to reduce the length of the robotic arm's motion path and improve the response time, thereby achieving continuous picking of dwarfed and densely planted fruit and vegetable crops with high efficiency, high precision, high success rate, low cost and low time consumption.

[0005] The purpose of the present invention is achieved through the following technical solutions: A vision-guided continuous picking sequence optimization method for a robotic arm includes: Step S1: First, based on the picking robot arm and the binocular stereo vision system, a spatial heterogeneous visual positioning system is constructed to perform target recognition and positioning of tomato fruits; then, the three-dimensional coordinates of the center of mass of the tomato fruit in the world coordinate system are obtained by the spatial heterogeneity method to obtain a coordinate set of the pickable tomatoes; Step S2, transforming the tomato continuous picking problem into a spatial traveling salesman problem, establishing an objective function with the shortest picking path as the goal, and completing the traversal of the coordinates of the pickable tomatoes; and optimizing the traveling salesman problem through the improved sparrow algorithm guided by vision to obtain the optimal picking sequence; Step S3, start the picking robot arm, and drive the end of the picking robot arm to pick the target tomato fruits in sequence according to the optimal picking sequence; after the picking is completed, the end of the picking robot arm returns to the starting point for the next identification and picking.

[0006] Based on further optimization of the above scheme, the spatial heterogeneous visual positioning system includes a mobile chassis, a column, a body camera, a picking robot arm, an end effector, a palm camera and a control system. The column is arranged on the end face of the mobile chassis and the body camera is arranged on the front side of the end face of the mobile chassis. The body camera is used to identify and locate the picking target area, thereby realizing the posture adjustment of the picking robot arm. The picking robot arm is arranged on the end face of the column and the end effector is arranged at the end of the picking robot arm. The palm camera is arranged at the palm position of the picking robot arm. The palm camera is used for secondary positioning of the picking area. Both the body camera and the palm camera adopt binocular stereo vision cameras; the control system includes an industrial computer and a robot arm controller. The robot arm controller controls the operation of the picking robot arm. The industrial computer is respectively connected to the robot arm controller, the mobile chassis, the end effector, the body camera and the palm camera.

[0007] Based on the further optimization of the above scheme, the specific method of "obtaining the three-dimensional coordinates of the centroid of the tomato fruit in the world coordinate system by the spatial isomerism method" in step S1 is: First, the relationship between the positions of the two cameras was calculated through calibration, and the two images taken by the left and right cameras were aligned in the same plane through binocular correction to obtain the projection coordinates of the tomato target fruit mass center on the left camera imaging plane ( u l , v l ) and the projection coordinates of the tomato target fruit mass center on the right camera imaging plane ( u r ,v r ); Then, according to the pinhole imaging principle, the left projection coordinates ( u l ,v l ) and the right projection coordinates ( u r ,v r ) to transform the coordinates and obtain the three-dimensional coordinates of the tomato's centroid by the least squares method P(x,y,z) :

[0008] Where: f xl 、f yl 、u ol 、v ol Represents the left channel internal reference of the binocular stereo vision camera; f xr 、f yr 、u or 、v or Represents the internal reference of the right channel of the binocular stereo vision camera; R 3x3 Represents the camera rotation matrix; T 3x1 Represents the camera translation matrix; Z cl 、Z cr Represents the Z-axis direction vector of the left and right cameras; Ml 3x4 、Mr 3x4 Represents the parameter matrix of the left and right cameras.

[0009] Based on the further optimization of the above scheme, the specific method of "converting the tomato continuous picking problem into a spatial traveling salesman problem and establishing the objective function with the shortest picking path as the goal" in step S2 is: The left camera aperture center of the palm camera is used as the origin coordinate of the world coordinate system O(0,0,0) And the starting point of the end of the picking robot arm; set the crown surface n tomato fruits, and the three-dimensional coordinates of the centroid of each tomato fruit are P i (x i ,y i ,z i ) , then the path set of the end of the picking robot arm from the starting point through all tomato fruits and back to the starting point is L (O,P 1 ,P 2 ,…,P n ,O) ; At this time, the optimization goal is to minimize the total distance of the straight line path, and the objective function is:

[0010] Where: D Indicates the total distance picked; d 1,O represents the distance from the starting point to the first tomato fruit; d n,O Indicates that from n The distance from the tomato fruit to the starting point; d i,i+1 Indicates that from i Tomato fruit to i+1 The distance between two tomato fruits.

[0011] Based on the further optimization of the above scheme, the specific method of "improving the sparrow algorithm to optimize the traveling salesman problem and obtain the optimal picking sequence" in step S2 is: First, in order to avoid the uneven distribution of populations caused by random generation during population initialization, which affects the subsequent iterative optimization, thereby improving the global search capability and avoiding the reduction of population diversity after iteration, cubic chaotic mapping is used to initialize the sparrow population: First formula:

[0012] Second formula:

[0013] Where: X i represents the position of individuals in the sparrow population, X l represents the lower boundary of the individual,X u represents the upper boundary of the individual; Indicates i Individual sparrows; exist d In the dimensional space, it is assumed that there is S sparrow individuals, then the steps to initialize the sparrow population are: Randomly generate an element of 1 or -1 d dimensional row vector Y As the first sparrow individual; then use the first formula to Y Iterate dimension by dimension to obtain the remaining S-1 individuals; finally, use the second formula to map the generated chaotic variables to the sparrow individuals; Then, the particle swarm algorithm is used to optimize the explorer position update in the sparrow algorithm, so as to avoid the situation where the algorithm search ability decreases due to the increase in the number of iterations and the algorithm falls into the local optimum in the later stage. In the particle swarm algorithm, particles are optimized through individual optimal solutions. p best and the global optimal solution g best To perform a location update:

[0014] Where: Indicates i The particle in d In the dimensional space k The speed of iterations; Indicates i The particle in d In the dimensional space k The position of the iteration; w represents the inertia weight; c 1 , c 2 represents the learning factor; r 1 , r 2 is a random number between [0,1]; Then the explorer position update in the sparrow algorithm after the particle swarm algorithm is introduced is:

[0015] Where: Represents individual sparrows i In the j dimension, t The location at the moment; Q is a random number that follows a normal distribution; L represents a unit row vector; R2 is the warning value and ; ST is a safe value; When the explorer finds better resources, the followers will move closer to the explorer's optimal position, causing the population density to increase rapidly in a short period of time, which in turn causes the algorithm to fall into a local optimum. In order to avoid the local optimum caused by the followers moving closer to the explorer and to enhance the traversal and randomness of the search, the search space is optimized through the flying strategy. Sparrow algorithm follower position update:

[0016] Where: Indicates the current worst position; E Indicates the position of the sparrow; Indicates the current optimal position;

[0017] Where: A A row vector whose elements are 1 or -1; F represents the random step size:

[0018] in:

[0019]

[0020] Where: represents the gamma function.

[0021] Based on the further optimization of the above table scheme, in order to ensure that the explorer expands the search range in the early stage and moves closer to the global optimum in the later stage, thereby reducing the time spent on optimization, the inertia weight in the explorer position update w and learning factor c 1 , c 2 To make adaptive adjustments:

[0022]

[0023] Where: w max , w min Respectively represent the maximum and minimum values ​​of the inertia weight; N max Indicates the maximum number of iterations; t Indicates time.

[0024] In the actual picking process of the picking robot arm, there are cases where tomatoes are blocked by tomatoes, leaves, branches, etc. Therefore, the improved sparrow algorithm is used as the theoretically optimal sequence. In the actual picking process, the picking fails due to the occlusion problem. Based on the further optimization of the above scheme, the improved sparrow algorithm in step S2 sets the constraint conditions through visual guidance: Condition 1: The picking robot arm picks the tomato fruit bunches in order. After completing the picking of a whole bunch of tomato fruit, it can pick the next bunch of tomato fruit. Condition 2: Unripe tomato fruits or tomato fruits that are blocked by branches are considered unharvestable; Condition 3: Ripe tomatoes that are not completely covered by leaves are considered ready for picking; Condition 4: Another ripe tomato fruit that is blocked by a ripe tomato fruit is ready for picking after the blocked ripe tomato fruit is picked, and the blocked ripe tomato fruit is picked first; According to the above constraints, visual depth information is introduced into the improved sparrow algorithm. The visual system partitions the tomatoes according to the whole bunch. The improved sparrow algorithm is used to optimize the traveling salesman problem to obtain the optimal path for each partition, and then all the areas are linked together to achieve picking sequence optimization.

[0025] The following are the technical effects of the solution of the present invention: The present invention acquires the three-dimensional coordinates of multi-target tomato fruits by establishing a spatial heterogeneous binocular stereo vision positioning system, which can not only accurately obtain the spatial positions of tomato fruits and provide support for subsequent path planning, but also judge the occlusion conditions of tomato fruits and effectively plan picking priorities, thereby avoiding problems such as wrong picking, missed picking or low picking efficiency. By establishing a tomato picking task space and set based on vision guidance in a non-closed space, the continuous picking problem is converted into a three-dimensional traveling salesman problem, and the picking sequence of the traveling salesman problem is optimized by constructing an improved sparrow algorithm, thereby avoiding the problems of the traveling salesman problem falling into local optimum and poor convergence accuracy in the late iteration. At the same time, when improving the sparrow algorithm, the population is initialized by cubic chaos mapping, which effectively solves the problems of uneven population distribution, lack of traversal and regularity in the process of randomly generating the initial population of the sparrow algorithm; the explorer position update in the sparrow algorithm is optimized by the particle swarm algorithm, which expands the explorer's search range in the early stage and prompts it to move closer to the global optimum in the later stage, thereby solving the problems of the sparrow algorithm with the increase of iterations, the decrease of computing power and the easy fall into the local optimum; the follower position update in the sparrow algorithm is optimized by the flight strategy, avoiding the local optimum that occurs in a short time when the follower approaches the explorer's optimal position and the population density increases, and enhancing the traversal of the follower. In addition, this method introduces visual information in the improved sparrow algorithm, sets the picking priority of tomato fruits, and ensures the continuous and efficient picking of tomato fruits.

[0026] Compared with traditional genetic algorithm, particle swarm algorithm and standard sparrow algorithm, the planning of tomato fruit picking sequence by this method reduces the response time by 19.8%, 32.9% and 42.4% respectively, and the picking path length by 25.8%, 24.0% and 16.24% respectively. The picking path is short and the response time is fast, thus achieving the purpose of fast, efficient and accurate harvesting of tomato fruits. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flowchart of the optimization of the continuous picking sequence of tomato fruits in an embodiment of the present invention.

[0028] Figure 2 Schematic diagram of a spatially heterogeneous visual positioning system in an embodiment of the present invention.

[0029] Figure 3 This is a flowchart of tomato positioning based on spatial heterogeneous binocular stereo vision in an embodiment of the present invention.

[0030] Figure 4 This is a schematic diagram of the principle of determining the three-dimensional coordinates of a tomato fruit using a binocular stereo vision camera in an embodiment of the present invention.

[0031] Figure 5This is a schematic diagram of the first group of tomato fruit picking path planning in an embodiment of the present invention; wherein: Figure 5 (a) is the first set of tomato coordinate extraction schematic diagram, Figure 5 (b) is the first set of tomato picking sequence planning diagram, Figure 5 (c) is the fitness change curve of the first group of tomatoes picked by various methods under visual guidance.

[0032] Figure 6 This is a schematic diagram of the second group of tomato fruit picking path planning in an embodiment of the present invention; wherein: Figure 6 (a) is the schematic diagram of the second set of tomato coordinate extraction. Figure 6 (b) is the second set of tomato picking sequence planning diagram. Figure 6 (c) is the fitness change curve of the second group of tomatoes picked by various methods under visual guidance.

[0033] Among them, 10, mobile chassis; 20, column; 30, body camera; 40, picking robot arm; 50, end effector; 60, palm camera. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0035] Embodiment 1: A vision-guided continuous picking sequence optimization method for a robotic arm includes: Step S1: First, based on the picking robot 40 and the binocular stereo vision system, a spatial heterogeneous visual positioning system is constructed. Figure 2 As shown: the spatial heterogeneous visual positioning system includes a mobile chassis 10, a column 20, a body camera 30, a picking robot arm 40, an end effector 50, a palm camera 60 and a control system. The column 20 is arranged on the end face of the mobile chassis 10 and the body camera 30 is arranged on the front side of the end face of the mobile chassis 10. The body camera 30 is used for identifying and positioning the picking target area, so as to realize the posture adjustment of the picking robot arm 40. The picking robot arm 40 is arranged on the end face of the column 20 and the end effector 50 is arranged at the end of the picking robot arm 40. A palm camera 60 is set at the palm position of the picking robot arm 40, and the palm camera 60 is used for secondary positioning of the picking area. The body camera 30 and the palm camera 60 both use binocular stereo vision cameras; the control system includes an industrial computer and a robot arm controller, the robot arm controller controls the operation of the picking robot arm 40, and the industrial computer is connected to the robot arm controller, the mobile chassis 10, the end effector 50, the body camera 30, and the palm camera 60 respectively (the communication, receiving and sending of instructions between the industrial computer and each component can be realized through Ethernet).

[0036] Reference Figure 3 As shown: the body camera 30 in the spatial heterogeneous visual positioning system first performs target recognition and positioning of tomato fruits, identifies the position of the tomato fruit cluster, that is, completes the recognition of the target area, and moves to the vicinity of the picking area through the mobile chassis 10, and then controls the picking robot arm 40 to move toward the positioning and area of ​​the body camera 30, and uses the palm camera 60 to zoom in and shoot the image of the target area (whether it is the body camera 30 or the palm camera 60, the recognition and processing of the target are all processed using the existing Yolo image recognition technology, such as Yolo v5).

[0037] Then, the three-dimensional coordinates of the tomato fruit mass center in the world coordinate system are obtained by spatial heterogeneity method, and the coordinate set of the tomato that can be picked is obtained. Figure 4 As shown, specifically: First, the relationship between the positions of the two cameras was calculated through calibration, and the two images taken by the left and right cameras were aligned in the same plane through binocular correction to obtain the projection coordinates of the tomato target fruit mass center on the left camera imaging plane ( u l , v l ) and the projection coordinates of the tomato target fruit mass center on the right camera imaging plane ( u r ,v r ); Then, according to the pinhole imaging principle, the left projection coordinates ( u l ,v l ) and the right projection coordinates ( u r ,v r ) to transform the coordinates and obtain the three-dimensional coordinates of the tomato's centroid by the least squares method P(x,y,z) :

[0038] Where: f xl 、f yl 、u ol 、v ol Represents the left channel internal reference of the binocular stereo vision camera; f xr 、f yr 、u or 、v orRepresents the internal reference of the right channel of the binocular stereo vision camera; R 3x3 Represents the camera rotation matrix; T 3x1 Represents the camera translation matrix; Z cl 、Z cr Represents the Z-axis direction vector of the left and right cameras; Ml 3x4 、Mr 3x4 Represents the parameter matrix of the left and right cameras.

[0039] Step S2: The continuous tomato picking problem is transformed into a spatial traveling salesman problem. The objective function is established with the shortest picking path as the goal to complete the traversal of the coordinates of the tomatoes that can be picked. Specifically, The left camera aperture center of the palm camera 60 is used as the origin coordinate of the world coordinate system O(0,0,0) and the starting point of the end of the picking robot arm 40; setting the existence of the crown surface n tomato fruits, and the three-dimensional coordinates of the centroid of each tomato fruit are P i (x i ,y i ,z i ) , then the path set of the end of the picking robot arm 40 from the starting point through all tomato fruits and back to the starting point is L(O,P 1 ,P 2 ,…,P n ,O) ; At this time, the optimization goal is to minimize the total distance of the straight line path, and the objective function is:

[0040] Where: D Indicates the total distance picked; d 1,O represents the distance from the starting point to the first tomato fruit; d n,O Indicates that from n The distance from the tomato fruit to the starting point; d i,i+1 Indicates that from i Tomato fruit to i+1 The distance between two tomato fruits.

[0041] And through the visually guided improved sparrow algorithm to optimize the traveling salesman problem, the optimal picking sequence is obtained: The traditional sparrow algorithm is divided into explorers and followers. The explorer has the best resources in the population and can find food and provide the follower with the direction of predation. The explorer position is updated as follows:

[0042] Where: a Represents a random number in [0,1]; Follower position updated to:

[0043] At the same time, in the population, there are 10% to 20% of sparrows that can detect risks and send out warning signals to notify other sparrows to evacuate to the safe area. The position of the alerter is updated as follows:

[0044] Where: Indicates the current global optimal position; represents the step size, which is a random number that follows a normal distribution with a mean of 0 and a variance of 1; K is a random number in [-1,1]; fit i Indicates the fitness value of the current sparrow individual; fit g Represents the global optimal fitness value; fit w Indicates the global worst fitness value; Represents a constant close to 0 and avoids denominator being 0; The improved sparrow algorithm is as follows: First, in order to avoid the uneven distribution of populations caused by random generation during population initialization, which affects the subsequent iterative optimization, thereby improving the global search capability and avoiding the reduction of population diversity after iteration, cubic chaotic mapping is used to initialize the sparrow population: First formula:

[0045] Second formula:

[0046] Where: X i represents the position of individuals in the sparrow population, X l represents the lower boundary of the individual, X u represents the upper boundary of the individual; Indicates i Individual sparrows; exist d In the dimensional space, it is assumed that there is Ssparrow individuals, then the steps to initialize the sparrow population are: Randomly generate an element of 1 or -1 d dimensional row vector Y As the first sparrow individual; then use the first formula to Y Iterate dimension by dimension to obtain the remaining S-1 individuals; finally, use the second formula to map the generated chaotic variables to the sparrow individuals; Then, the particle swarm algorithm is used to optimize the explorer position update in the sparrow algorithm, so as to avoid the situation where the algorithm search ability decreases due to the increase in the number of iterations and the algorithm falls into the local optimum in the later stage. In the particle swarm algorithm, particles are optimized through individual optimal solutions. p best and the global optimal solution g best To perform a location update:

[0047] Where: Indicates i The particle in d In the dimensional space k The speed of iterations; Indicates i The particle in d In the dimensional space k The position of the iteration; w represents the inertia weight; c 1 , c 2 represents the learning factor; r 1 , r 2 is a random number between [0,1]; Then the explorer position update in the sparrow algorithm after the particle swarm algorithm is introduced is:

[0048] Where: Represents individual sparrow i In the j dimension, t The location at the moment; Q is a random number that follows a normal distribution; L represents a unit row vector; R 2 is the warning value and ; ST is a safe value; In order to ensure that the explorer expands the search range in the early stage and moves closer to the global optimum in the later stage, thereby reducing the time spent on optimization, the inertia weight in the explorer position update wand learning factor c 1 , c 2 To make adaptive adjustments:

[0049]

[0050] Where: w max , w min Respectively represent the maximum and minimum values ​​of the inertia weight; N max Indicates the maximum number of iterations; t Indicates time.

[0051] When the explorer finds better resources, the followers will move closer to the explorer's optimal position, causing the population density to increase rapidly in a short period of time, which in turn causes the algorithm to fall into a local optimum. In order to avoid the local optimum caused by the followers moving closer to the explorer and to enhance the traversal and randomness of the search, the search space is optimized through the flying strategy. Sparrow algorithm follower position update:

[0052] Where: Indicates the current worst position; E Indicates the position of the sparrow; Indicates the current optimal position;

[0053] Where: A A row vector whose elements are 1 or -1; F represents the random step size:

[0054] in:

[0055]

[0056] Where: represents the gamma function; .

[0057] Step S3, start the picking robot arm 40, and drive the end of the picking robot arm 40 to pick the target tomato fruits in sequence according to the optimal picking sequence; after the picking is completed, the end of the picking robot arm 40 returns to the starting point to perform the next identification and picking.

[0058] In order to verify the effectiveness of the improved sparrow algorithm, the genetic algorithm (GA), particle swarm algorithm (PSO), standard sparrow algorithm (SSA) and the improved SSA algorithm in this embodiment are used to identify two groups of tomato sample coordinates (wherein the first group of tomato fruit samples has 7 pickable tomatoes and the second group of tomato fruit samples has 17 pickable tomatoes, see Figure 5 (a) with Figure 6 (a)), where the population size of each algorithm is 50, the maximum number of iterations is 1000, and the running results are shown in Table 1: Table 1:

[0059] It can be clearly seen from Table 1 that when picking 7 tomatoes, the average response times of GA, PSO, SSA and the improved SSA in this embodiment are 1.82s, 1.70s, 1.65s and 1.44s respectively. The improved SSA in this embodiment is 20.88%, 15.29% and 12.73% shorter in average response time than GA, PSO and SSA respectively. At the same time, the average picking distances of GA, PSO, SSA and the improved SSA in this embodiment are all 2775.14mm. Because the number of tomatoes is small at this time and there are only 5040 different picking paths, no matter which optimization algorithm is used, the optimal solution can be found. However, when picking 17 tomatoes, the average response times of GA, PSO, SSA, and the improved SSA in this embodiment are 5.68s, 5.49s, 6.38s, and 4.57s, respectively. Compared with GA, PSO, and SSA, the improved SSA in this embodiment shortens the average response time by 19.54%, 16.76%, and 28.37%, respectively. At the same time, the improved SSA in this embodiment shortens the average picking distance by 29.81%, 27.27%, and 19.80%, respectively, compared with GA, PSO, and SSA. It can be seen that the improved SSA algorithm in this embodiment can not only greatly shorten the picking sequence planning time, but also shorten the length of the picking path.

[0060] In addition, in order to further analyze the effect of the improved SSA algorithm in this embodiment, the fitness curves of GA, PSO, SSA, and the improved SSA in this embodiment are drawn respectively (such as Figure 5 (c) with Figure 6 (c) as well as the optimal picking path solved by SSA and the improved SSA in this embodiment (as shown in Figure 5 (b) with Figure 6(b)). It can be clearly seen from the figure that when solving the 7-tomato picking sequence, the improved SSA and SSA in this embodiment have the same picking order, which is: home-5-6-7-4-3-2-1-home, and each algorithm can find the optimal solution after several iterations; when solving the 17-tomato picking sequence, the former has a picking order of: home-9-11-10-8-7-5-6-3-4-12-13-15-17-16-14-1-2-home, and the latter has a picking order of: home- 9-10-11-14-16-17-15-13-12-4-6-8-7-5-3-1-2-home; at the same time, the number of iterations for GA, PSO, SSA, and the improved SSA in this embodiment to find the optimal solution are 146, 100, 252, and 77 times, respectively. That is, when the number of tomatoes is large, the improved SSA in this embodiment can find the picking sequence with the shortest path in a shorter time, thereby achieving the purpose of accurately identifying ripe tomatoes and quickly planning the optimal picking path.

[0061] Embodiment 2: As a preferred embodiment of the present invention, in the actual picking process of the picking robot arm 40, there is a situation where tomatoes are blocked by tomatoes, leaves, branches, etc. Therefore, the improved sparrow algorithm is used as the optimal sequence obtained in theory. In the actual picking process, the picking fails due to the blocking problem. The improved sparrow algorithm sets the constraint conditions through visual guidance: Condition 1: The picking robot arm 40 picks the tomato fruit bunches in order, and can only pick the next bunch of tomato fruit after completing the picking of a whole bunch of tomato fruit; Condition 2: Unripe tomato fruits or tomato fruits that are blocked by branches are considered unharvestable; Condition 3: Ripe tomatoes that are not completely covered by leaves are considered ready for picking; Condition 4: Another ripe tomato fruit that is blocked by a ripe tomato fruit is ready for picking after the blocked ripe tomato fruit is picked, and the blocked ripe tomato fruit is picked first; According to the above constraints, the visual depth information is introduced into the improved sparrow algorithm. The visual system divides the tomatoes into clusters. The improved sparrow algorithm is used to optimize the traveling salesman problem to obtain the optimal path for each partition. All the areas are then linked together to achieve picking sequence optimization. Since the occlusion situation is not considered in Example 1, the success rate of picking of each algorithm is low. Based on the solution in Example 1, visual information is introduced into the improved sparrow algorithm, and the picking priority is set for picking tomato fruits that are blocked by tomatoes, specifically: In the vision-guided improved sparrow algorithm, the picking priority is set for picking tomato fruits that are blocked by tomatoes, specifically:

[0062] Where: f i represents unobstructed tomato fruit; s i represents the obscured tomato fruit; j i Represents an obscured tomato fruit.

[0063] Since the occlusion situation is not considered, the success rate of picking of each algorithm is low. After introducing visual information into each algorithm, the running results are shown in Table 2: Table 2:

[0064] It can be clearly seen from Table 2 that after the introduction of visual information, although the average picking distance of each algorithm has increased, the picking success rate has been greatly improved; among them, the planning success rate of the scheme in this embodiment is 97.0%, which is 2.9% higher than that of VG-SSA planning success rate, 16.24% lower average picking distance, and 42.4% lower average response time; verification shows that the method adopted in this embodiment can not only greatly improve the picking success rate, but also shorten the picking sequence planning time and the length of the picking path.

Claims

1. A method for optimizing continuous picking sequence of a robotic arm under vision guidance, characterized in that: include: Step S1: First, based on the picking robot arm and binocular stereo vision system, a spatial heterogeneous visual positioning system is constructed to perform target recognition and positioning of tomato fruits; Then, the three-dimensional coordinates of the mass center of the tomato fruit in the world coordinate system are obtained by the spatial isomerism method, and the coordinate set of the tomato that can be picked is obtained; Step S2, transforming the tomato continuous picking problem into a spatial traveling salesman problem, establishing an objective function with the shortest picking path as the goal, and completing the traversal of the coordinates of the pickable tomatoes; and optimizing the traveling salesman problem through the improved sparrow algorithm guided by vision to obtain the optimal picking sequence; Step S3, start the picking robot arm, and drive the end of the picking robot arm to pick the target tomato fruits in sequence according to the optimal picking sequence; after the picking is completed, the end of the picking robot arm returns to the starting point for the next identification and picking.

2. The method for optimizing the continuous picking sequence of a robotic arm under vision guidance according to claim 1, characterized in that: The spatial heterogeneous visual positioning system includes a mobile chassis, a column, a body camera, a picking robot arm, an end effector, a palm camera and a control system. The column is arranged on the end face of the mobile chassis and the body camera is arranged on the front side of the end face of the mobile chassis. The body camera is used for identifying and positioning the picking target area, so as to realize the posture adjustment of the picking robot arm. The picking robot arm is arranged on the end face of the column and the end effector is arranged at the end of the picking robot arm. The palm camera is arranged at the palm position of the picking robot arm. The palm camera is used for secondary positioning of the picking area. Both the body camera and the palm camera adopt binocular stereo vision cameras. The control system includes an industrial computer and a robot arm controller. The robot arm controller controls the operation of the picking robot arm. The industrial computer is respectively connected with the robot arm controller, the mobile chassis, the end effector, the body camera and the palm camera.

3. The method for optimizing the continuous picking sequence of a robotic arm under vision guidance according to claim 1 or 2, characterized in that: The specific method of "obtaining the three-dimensional coordinates of the centroid of the tomato fruit in the world coordinate system by using the spatial isomerism method" in step S1 is: First, the relationship between the positions of the two cameras was calculated through calibration, and the two images taken by the left and right cameras were aligned in the same plane through binocular correction to obtain the projection coordinates of the tomato target fruit mass center on the left camera imaging plane ( u l ,v l ) and the projection coordinates of the tomato target fruit mass center on the right camera imaging plane ( u r ,v r ); Then, according to the pinhole imaging principle, the left projection coordinates ( u l ,v l ) and the right projection coordinates ( u r ,v r ) to transform the coordinates and obtain the three-dimensional coordinates of the tomato's centroid by the least squares method P(x,y,z) : Where: f xl 、f yl 、u ol 、v ol Represents the left channel internal reference of the binocular stereo vision camera; f xr 、f yr 、u or 、v or Represents the internal reference of the right channel of the binocular stereo vision camera; R 3x3 Represents the camera rotation matrix; T 3x1 Represents the camera translation matrix; Z cl 、Z cr Represents the Z-axis direction vector of the left and right cameras; Ml 3x4 、Mr 3x4 Represents the parameter matrix of the left and right cameras.

4. The method for optimizing the continuous picking sequence of a robotic arm under vision guidance according to claim 1 or 3, characterized in that: The specific method of "converting the tomato continuous picking problem into a spatial traveling salesman problem and establishing an objective function with the shortest picking path as the goal" in step S2 is: The left camera aperture center of the palm camera is used as the origin coordinate of the world coordinate system O(0,0,0) And the starting point of the end of the picking robot arm; set the crown surface n tomato fruits, and the three-dimensional coordinates of the centroid of each tomato fruit are P i (x i , y i ,z i ) , then the path set of the end of the picking robot arm from the starting point through all tomato fruits and back to the starting point is L(O, P 1 ,P 2 ,…,P n ,O) ; At this time, the optimization goal is to minimize the total distance of the straight line path, and the objective function is: Where: D Indicates the total distance picked; d 1,O represents the distance from the starting point to the first tomato fruit; d n,O Indicates that from n The distance from the tomato fruit to the starting point; d i,i+1 Indicates that from i Tomato fruit to i+1 The distance between two tomato fruits.

5. The method for optimizing the continuous picking sequence of a robotic arm under vision guidance according to claim 4, characterized in that: The specific method of "improving the sparrow algorithm to optimize the traveling salesman problem and obtain the optimal picking sequence" in step S2 is: First, cubic chaos mapping is used to initialize the sparrow population: First formula: Second formula: Where: X i represents the position of individuals in the sparrow population, X l represents the lower boundary of the individual, X u represents the upper boundary of the individual; Indicates i Individual sparrows; exist d In the dimensional space, it is assumed that there is S sparrow individuals, then the steps to initialize the sparrow population are: Randomly generate an element of 1 or -1 d dimensional row vector Y As the first sparrow individual; Then use the first formula to Y Iterate dimension by dimension to obtain the remaining S-1 individuals; finally, use the second formula to map the generated chaotic variables to the sparrow individuals; Then, the particle swarm algorithm is used to optimize the explorer position update in the sparrow algorithm. In the particle swarm algorithm, particles are selected through individual optimal solutions. p best and the global optimal solution g best To perform a location update: Where: Indicates i The particle in d In the dimensional space k The speed of iterations; Indicates i The particle in d In the dimensional space k The position of the iteration; w represents the inertia weight; c 1. c 2 represents the learning factor; r 1. r 2 is a random number between [0,1]; Then the explorer position update in the sparrow algorithm after the particle swarm algorithm is introduced is: Where: Represents individual sparrows i In the j dimension, t The location at the moment; Q is a random number that follows a normal distribution; L represents a unit row vector; R 2 is the warning value and ; ST is a safe value; Search space optimization through flying strategy Sparrow algorithm follower position update: Where: Indicates the current worst position; E Indicates the position of the sparrow; Indicates the current optimal position; Where: A A row vector whose elements are 1 or -1; F represents the random step size: in: Where: represents the gamma function.

6. The method for optimizing the continuous picking sequence of a robotic arm under vision guidance according to claim 5, characterized in that: Inertia weight in the explorer position update w and learning factor c 1. c 2 Make adaptive adjustments: Where: w max , w min Respectively represent the maximum and minimum values ​​of the inertia weight; N max Indicates the maximum number of iterations; t Indicates time.

Images