A working method and system of a multi-arm cooperative picking robot

Abstract

The present application relates to the technical field of agricultural picking robots, and discloses a working method and system of a multi-arm cooperative picking robot. The method acquires image data of a fruit crown area, identifies a fruit stem picking point and extracts a three-dimensional spatial coordinate and a main direction of the fruit stem, performs density clustering based on spatial distance and a main direction difference of the fruit stem, separates or merges the clustered units according to local density distribution and consistency of the fruit stem direction, and forms a cluster-shaped picking task unit; task distribution is completed according to the installation position of a picking mechanical arm, reachable space, base distance and the length of a picking operation queue; in multi-arm cooperative operation, online modulation of a motion trajectory is triggered according to the entering trend of an overlapping area or the arm spacing, the current planning position and the picking operation queue state are saved and restored, and the end effector is controlled to complete fruit stem shearing, fruit clamping, transfer and release. The present application enables the clustering result to be continuously connected to task distribution and picking execution, and keeps the queue connection of uncompleted tasks before and after avoidance.

Description

Technical Field

[0001] This invention relates to the field of agricultural harvesting robot technology, specifically to a method and system for operating a multi-arm collaborative harvesting robot. Background Technology

[0002] Mangoes and other fruits typically exhibit spatial clustering, foliage shading, and inconsistent stem orientation within unstructured fruit tree canopies. To reduce the labor intensity of manual harvesting, fruit-harvesting robots usually utilize visual recognition systems to acquire canopy images, determine fruit or stem harvesting points, and control robotic arms and end effectors to perform cutting, gripping, transferring, and releasing.

[0003] Existing multi-arm harvesting operation solutions typically perform spatial clustering or fixed longitudinal partitioning based on the three-dimensional location of the harvesting point, and then allocate harvesting tasks according to the working area, reachable range, or current load of each robotic arm. For overlapping areas reached by adjacent robotic arms, the harvesting order can be generated through reinforcement learning, path planning, or collision detection, and the robotic arm movement trajectory can be adjusted to avoid collisions when the arm spacing is too small.

[0004] However, when clustering based solely on the location of the picking points, spatially adjacent picking points with significant differences in stem direction are easily grouped into the same task, leading to instability in the subsequent robotic arm posture adjustments and task execution. Furthermore, when using fixed partitions or assigning tasks to individual picking points, issues such as mismatch between robotic arm accessibility, base distance, and queue load can easily arise in overlapping areas. Additionally, if the original picking queue is interrupted by a local avoidance maneuver, and the current planned location and queue status are not simultaneously saved, unfinished tasks cannot continue execution according to the original succession relationship.

[0005] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0006] This invention provides a method and system for operating a multi-arm collaborative harvesting robot, which solves the problems in existing multi-arm harvesting operations, such as incorrect merging of harvesting points with large differences in fruit stem direction due to clustering based solely on spatial location, mismatch between the accessibility of task units and robotic arms and queue load in overlapping work areas, and lack of continuity between the current planned position and the harvesting queue status after online avoidance, making it difficult to continue unfinished harvesting tasks.

[0007] The technical solution of this invention is as follows: On one hand, this invention provides a method for operating a multi-arm collaborative harvesting robot, including: acquiring image data of the fruit crown area, identifying fruit stalk harvesting points, and extracting the three-dimensional spatial coordinates and main direction of each fruit stalk harvesting point; constructing a composite similarity metric based on the spatial distance between any two fruit stalk harvesting points and the difference in the main direction of the fruit stalk, and performing density clustering; when the local density within the initial clustering unit is higher than its average density and the consistency of the fruit stalk direction does not meet the clustering criteria, separating the corresponding region into sub-clustering units; merging adjacent clustering units when the composite similarity metric at the boundary of adjacent clustering units meets the merging condition, to obtain clustered harvesting task units; dividing overlapping task areas according to the installation position of the harvesting robot arm, the reachable space, and the spatial range of the task units. The system performs accessibility screening on task units within overlapping areas, excluding unreachable picking robotic arms. Target picking robotic arms are determined based on the distance from the task unit to the base of each reachable picking robotic arm and the length of its picking queue. The task unit is then added to the picking queue of the target picking robotic arm. Each picking robotic arm is controlled to operate collaboratively according to its corresponding picking queue. If any picking robotic arm is predicted to enter an overlapping area or the arm spacing is less than a safety threshold, the system modulates the relevant motion trajectory online and saves the current planned position and picking queue status. Avoidance is performed according to the modulated joint control values. Once the spacing returns to above the safety threshold, the saved status is read, and the unfinished picking task continues. The end effector is controlled to complete fruit stem cutting, fruit clamping, transfer to the collection system, and release.

[0008] Optionally, the step of extracting the three-dimensional spatial coordinates and main direction of the fruit stalk at each fruit stalk picking point includes: extracting the candidate fruit region and fruit stalk picking point from the image data; transforming each fruit stalk picking point to the same spatial coordinate system according to the depth information of the image data; extracting fruit stalk features in the local area corresponding to each fruit stalk picking point, and fitting the main direction vector of the fruit stalk according to the fruit stalk features.

[0009] Optionally, the construction of the composite similarity metric and the density clustering include: calculating the three-dimensional Euclidean distance between any two fruit stalk picking points and the angle between the corresponding fruit stalk principal direction vectors; weighting and fusing the three-dimensional Euclidean distance and the angle to obtain a composite similarity metric value; and performing density clustering on the fruit stalk picking points based on the composite similarity metric value to form multiple initial clustering units.

[0010] Optionally, the determination of the clustering criteria and merging conditions includes: statistically analyzing the local density of fruit stalk picking points within each initial clustering unit to obtain the corresponding average density; determining the consistency of fruit stalk orientation based on the angle distribution between the main direction vectors of the fruit stalks; adjusting the neighborhood parameters of density clustering based on the average density and the consistency of fruit stalk orientation to form the clustering criteria; and calculating the composite similarity measure between fruit stalk picking points at the boundary of adjacent clustering units, comparing it with the merging threshold to form the merging conditions.

[0011] Optionally, the process of dividing adjacent overlapping work areas includes: determining the core work area of ​​each picking robot arm based on its installation position on the frame and the corresponding reachable space; expanding the boundaries of adjacent core work areas based on the spatial range of the clustered picking task units to form overlapping work areas that can be reached by adjacent picking robot arms; and assigning clustered picking task units located in non-overlapping work areas to the corresponding picking robot arms.

[0012] Optionally, determining the target harvesting robotic arm includes: acquiring distance data between the fruit stalk picking point in each cluster harvesting task unit and the base of each reachable harvesting robotic arm; acquiring the length data of the current harvesting queue of each reachable harvesting robotic arm; forming a comprehensive cost for each reachable harvesting robotic arm based on the distance data and length data; determining the one with the smallest comprehensive cost as the target harvesting robotic arm, and adding the entire cluster harvesting task unit to its harvesting queue.

[0013] Optionally, determining the target harvesting robotic arm includes: constructing a joint state by considering the joint states of each harvesting robotic arm, the three-dimensional spatial coordinates of the clustered harvesting task units, the end-effector positions of adjacent harvesting robotic arms, and the occupancy status of overlapping work areas; constructing a joint action by considering the selection of task units and the updating of the harvesting work queue; and constructing reward information based on the end-effector distance, task completion status, multi-arm parallel status, and work time. A multi-agent policy model is trained using the joint state, joint action, and reward information; the task association degree is determined through a self-attention mechanism; and the target harvesting robotic arm is determined by combining spatial accessibility, base distance, and harvesting work queue length.

[0014] Optionally, the online modulation includes: acquiring the motion state of adjacent harvesting robotic arms according to the control cycle, calculating the end-effector spacing and the motion trend of entering the overlapping work area; triggering online modulation when the end-effector spacing is less than the safety threshold or when adjacent harvesting robotic arms will enter the overlapping work area simultaneously; and in the inverse kinematics solution, constructing a weighted cost function with end-effector pose deviation and inter-arm collision constraints, and performing nonlinear optimization with joint angle as the variable to be determined to obtain the joint control quantity.

[0015] Optionally, the step of continuing to execute unfinished picking tasks includes: saving the current planned position of the relevant picking robotic arm and its picking operation queue status when online modulation is triggered; controlling at least one relevant picking robotic arm to perform avoidance according to the joint control amount, and continuously acquiring the end-effector distance of adjacent picking robotic arms; exiting online modulation when the end-effector distance recovers to above the safety threshold, reading the current planned position and picking operation queue status, and connecting the unfinished picking tasks to the corresponding picking operation queue.

[0016] On the other hand, the present invention also provides an operating system for a multi-arm collaborative harvesting robot, comprising: a perception clustering module for acquiring fruit crown images, identifying fruit stalk harvesting points and extracting their three-dimensional coordinates and main stalk orientation, and performing density clustering based on spatial distance and main stalk orientation difference; separating corresponding regions when local density is higher than the average unit density and fruit stalk orientation consistency does not meet the clustering criteria, merging adjacent units when the composite similarity metric of their boundaries meets the merging condition, and outputting clustered harvesting task units; and a task allocation module for dividing overlapping work areas according to the installation position of the harvesting robot arm, reachable space, and task units, and screening... The system selects the reachable robotic arm, determines the target harvesting robotic arm based on the distance from the task unit to the reachable robotic arm base and the length of the harvesting queue, and adds the entire task unit to its harvesting queue; the collaborative control module controls each harvesting robotic arm to operate according to the harvesting queue, and modulates the motion trajectory online and saves the current planned position and harvesting queue status when entering an overlapping area or when the arm spacing is less than the safety threshold, and reads the saved status and continues the unfinished tasks after the arm spacing is restored; the harvesting execution module drives the harvesting robotic arm according to the joint control values ​​and controls the end effector to complete the fruit stem cutting, fruit clamping, transfer and release.

[0017] The beneficial effects of this invention are as follows:

[0018] This invention extracts the three-dimensional spatial coordinates of the fruit stalk picking point and the main direction of the fruit stalk, and uses the spatial distance and the difference in the main direction of the fruit stalk for density clustering. Based on the local density distribution and the consistency of the fruit stalk direction, the clustered picking task units are separated or merged, so that the formed clustered picking task units simultaneously reflect the spatial aggregation relationship of the picking point and the orientation relationship of the fruit stalk. By performing accessibility screening on the task units in the overlapping operation area, and combining the distance from the task unit to the robot arm base and the length of the picking operation queue, the target picking robot arm is determined, so that the clustering results are directly carried over to the task allocation. By saving the current planned position and the state of the picking operation queue during trajectory online modulation, and reading the saved state after the arm spacing is restored, the unfinished tasks before and after avoidance are kept connected in the queue, thus forming a continuous operation process from crown width perception, task unit generation, overlapping area allocation, cooperative avoidance to fruit stalk shearing, fruit clamping, transfer and release. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the overall process of a multi-arm collaborative harvesting robot operation method provided in one embodiment of the present invention;

[0020] Figure 2 This is a schematic diagram of three-dimensional feature extraction, composite clustering, and cluster-shaped picking task unit generation of fruit stalk picking points provided in an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of online trajectory modulation, end-point spacing variation, and harvesting task succession timing provided in one embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram of the structure and information connection of a multi-arm collaborative harvesting robot operation system provided in one embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention; that is, the described embodiments are merely some embodiments of the invention, and not all embodiments. The components of the embodiments of the invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0024] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0025] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0026] It should also be noted that the term "arm spacing" in this manual is a higher-level description of the distance between adjacent harvesting robotic arms; in online modulation and recovery judgment, the minimum distance between the geometric envelopes of the end effectors of adjacent harvesting robotic arms is specifically used as the end effector spacing. When predicting that any related harvesting robotic arm will enter the overlapping work area, the collaborative control module simultaneously reads the occupancy status of the overlapping work area and the entry trend of adjacent harvesting robotic arms during the prediction period; the overlapping work area entry trigger condition is considered met only when adjacent harvesting robotic arms have already occupied or will simultaneously enter the same overlapping work area.

[0027] As mentioned earlier, fruits such as mangoes exhibit spatial clustering, foliage shading, and inconsistent pedicel orientation in unstructured canopies. When clustering based solely on fruit location or using fixed longitudinal partitioning, picking points with significant differences in posture may be assigned to the same task, leading to task competition among multiple robotic arms within overlapping work areas. When local avoidance interrupts the original picking queue, unfinished tasks may result in the loss of the original planned position and queue succession relationship.

[0028] To address this, the present invention provides a method and system for operating a multi-arm collaborative harvesting robot. By extracting the three-dimensional spatial coordinates of the fruit stalk harvesting point and the main direction of the fruit stalk, density clustering is formed with the spatial distance and the difference in fruit stalk direction participating together. After separation and merging, clustered harvesting task units are obtained. Subsequently, task allocation is completed according to the installation position of the harvesting robot arm, the reachable space, the base distance, and the length of the harvesting queue. During the collaborative operation of the robot arm, online trajectory modulation, state saving, and task continuation are performed. Finally, the end effector is controlled to complete the fruit stalk cutting, fruit clamping, transfer, and release.

[0029] The following is combined with Figures 1 to 4 This invention is described in detail.

[0030] Example 1:

[0031] like Figure 1 As shown in the embodiment of the present invention, a method for operating a multi-arm collaborative harvesting robot can be implemented by an onboard computer, a robotic arm controller, and an end effector controller of the multi-arm collaborative harvesting robot. The method mainly includes the following steps:

[0032] S100: Acquire image data of the fruit crown area, identify the fruit stalk picking point, and extract the three-dimensional spatial coordinates of each fruit stalk picking point and the main direction of the fruit stalk.

[0033] S200: Construct a composite similarity measure and perform density clustering based on the spatial distance between any two fruit stalk picking points and the difference in the main direction of the fruit stalk;

[0034] S300. When the local density within the initial cluster unit is higher than its average density and the consistency of the fruit stalk direction does not meet the clustering criterion, the corresponding region is separated into sub-cluster units. When the composite similarity measure at the boundary of adjacent cluster units meets the merging condition, they are merged to obtain clustered picking task units.

[0035] S400. Divide the overlapping work areas according to the installation position of the harvesting robot arm, the accessible space, and the spatial range of the task unit.

[0036] S500. Perform accessibility screening on the task units in the overlapping area, exclude unreachable picking robotic arms, determine the target picking robotic arm according to the distance from the task unit to the base of each reachable picking robotic arm and the length of its picking operation queue, and add the task unit to the picking operation queue of the target picking robotic arm.

[0037] S600 controls each harvesting robotic arm to operate collaboratively according to its corresponding harvesting queue;

[0038] S700: When it is predicted that any picking robot arm will enter the overlapping area or the arm spacing is less than the safety threshold, the relevant motion trajectory is modulated online and the current planned position and picking operation queue status are saved. The avoidance is performed according to the modulated joint control amount. After the spacing is restored to above the safety threshold, the saved status is read and the unfinished picking task is continued.

[0039] S800 controls the end effector to complete the fruit stem cutting, fruit clamping, transfer to the collection system and release.

[0040] Based on the above steps, this invention uses the spatial distance of the fruit stalk picking point and the main direction difference of the fruit stalk for density clustering, and then sequentially assigns the clustered picking task units formed after separation and merging to the overlapping operation area division, robotic arm accessibility screening, picking operation queue update, online trajectory modulation and end-point picking execution. This makes the clustering results have a clear robotic arm execution direction, and continues to process unfinished picking tasks after the avoidance is completed, thus forming a continuous operation process from visual perception, task allocation, collaborative control to fruit collection.

[0041] Example 2:

[0042] To provide a more detailed explanation of the technical solutions provided in the above embodiments, the present invention also provides Embodiment Two. In this embodiment, the operation method is still executed collaboratively by an onboard computer, a robotic arm controller, and an end effector controller. The multi-arm collaborative harvesting robot first moves to the location of the target mango tree according to a preset map, and adjusts the position of the moving chassis and the U-shaped frame relative to the canopy so that the field of view of the depth camera covers the canopy to be harvested; after harvesting the current canopy, the visual recognition system acquires the canopy image again. When no harvestable fruit is identified, the moving chassis moves to the next target mango tree according to the preset map.

[0043] like Figure 2 As shown, this embodiment extracts the three-dimensional spatial coordinates of the fruit stalk picking points and the main direction of the fruit stalks. Based on the spatial distance between picking points and the difference in the main direction of the fruit stalks, a composite similarity metric is constructed. The initial clustering units are then separated or merged according to local density and consistency of the fruit stalk direction to form clustered picking task units. Specifically, in this second embodiment, step S100 may further include:

[0044] S110. Extract the candidate fruit region and the fruit stalk picking point from the image data;

[0045] S120. Based on the depth information of the image data, convert each fruit stalk picking point to the same spatial coordinate system;

[0046] S130. Extract the fruit stalk features in the local area corresponding to each fruit stalk picking point, and fit the main direction vector of the fruit stalk based on the fruit stalk features.

[0047] Steps S110 to S130 may further include the following processes:

[0048] First, the visual recognition system acquires color and depth images using a depth camera mounted on a portal frame, and uses a target recognition network trained on the labeled images to form candidate fruit regions. The recognition confidence threshold is determined by the verification results of customer-labeled samples and the on-site initialization configuration (e.g., a confidence threshold of 0.80). Candidate regions below this threshold are not directly added to the picking point set. Subsequently, within each candidate fruit region, the stalk picking point where the stalk connects to the fruit is determined, and the corresponding depth is read.

[0049] Then, based on the intrinsic and extrinsic parameters in the depth camera calibration file, the fruit stalk picking point is transformed from the image coordinate system to the unified robot coordinate system. The calibration parameters are obtained during the device calibration process (e.g., both the lateral and longitudinal focal lengths are 600 pixels, the principal point coordinates are (640 pixels, 360 pixels), the rotation matrix from the camera coordinate system to the unified robot coordinate system is the identity matrix, and the translation vector is (0 mm, 0 mm, 0 mm)). Linear edges and 3D local points are extracted from the locally connected region corresponding to the fruit stalk picking point. The principal direction vector of the fruit stalk is obtained by line fitting or principal axis fitting and then normalized.

[0050] To eliminate the ambiguity of positive and negative directions caused by linear fitting or principal axis fitting, the direction along the fitting line pointing to the outside of the candidate area of ​​the fruit is uniformly determined as the positive direction of the principal direction vector of the fruit stalk, starting from the fruit stalk picking point. If the positive direction cannot be determined based on the local fruit stalk characteristics, it is handled according to the re-collection rules in the following text, and the direction vector is not used for the calculation of formula (1) and formula (2).

[0051] For example, let's say the current crown width task is denoted as T-001. The pixel coordinates of the fruit stalk picking point P1 are (640 pixels, 360 pixels), corresponding to a depth of 940 mm. After conversion according to the aforementioned calibration parameters, the three-dimensional spatial coordinates of P1 in the unified robot coordinate system are (0 mm, 0 mm, 940 mm). Using the same calibration relationship, the system obtains the coordinates of P2 as (30 mm, 20 mm, 945 mm) and P3 as (45 mm, 25 mm, 950 mm), and obtains the principal directions of the fruit stalks for P1, P2, and P3 respectively. These three-dimensional spatial coordinates and the principal directions of the fruit stalks are then passed to the subsequent composite clustering process.

[0052] When the color image and depth image are not synchronized, the picking point depth is invalid, or the local fruit stalk features are insufficient to form a stable direction, the visual recognition system retains the corresponding fruit candidate area and re-acquires the image of that area by adjusting the lateral movement screw sliding module or the position of the moving chassis; if the picking point still cannot form effective three-dimensional spatial coordinates and the main direction of the fruit stalk after re-acquisition, it will not use data without source to fill in the automatic clustering.

[0053] In this second embodiment, after obtaining the three-dimensional spatial coordinates of each fruit stalk picking point and the main direction of the fruit stalk, step S200 may further include:

[0054] S210. Calculate the three-dimensional Euclidean distance between any two fruit stalk picking points and the angle between the corresponding fruit stalk principal direction vectors.

[0055] S220. The three-dimensional Euclidean distance and the included angle are weighted and fused to obtain a composite similarity metric.

[0056] S230. Based on the composite similarity metric, density clustering is performed on the fruit stalk picking points to form multiple initial clustering units.

[0057] Steps S210 to S230 may further include the following processes:

[0058] Spatial distance and the difference in fruit stalk orientation are represented by the same composite similarity metric. The composite similarity metric can be calculated using the following formula:

[0059] (1)

[0060] in, For the first The fruit stalk picking point and the first The composite similarity measure between fruit stalk picking points is dimensionless. and These are three-dimensional spatial coordinates, in mm. and is the normalized principal direction vector of the fruit stalk, which is dimensionless; This is a distance reference scale, in mm; This is the reference scale for the orientation angle, in rad. and The weights are spatial distance and fruit stalk orientation, and are dimensionless. The smaller the composite similarity metric value, the closer the spatial distance between the two fruit stalk picking points and the smaller the difference in the main fruit stalk orientation, meaning the more similar they are. The distance reference scale is determined by the nearest neighbor distance statistics of the effective picking points within the current canopy width and the continuous adjustment range of the end effector (e.g., ...). =100mm); the reference scale for the orientation angle is half a circumference (e.g., =πrad); the two weights are determined by historical effective task statistics and on-site initialization configuration (e.g., πrad); =0.60, =0.40).

[0061] The neighborhood parameters for density clustering are determined by the distribution of the composite similarity metric within the current canopy and the in-situ initialization configuration (e.g., the neighborhood parameters are...). The minimum number of neighborhood points is determined by the statistical density of effective picking points in the current field of view (e.g., the minimum number of neighborhood points is 3, including the current picking point). Picking points whose composite similarity metric value is not greater than the neighborhood parameter are considered as neighboring points within the composite neighborhood, and the initial clustering unit is formed accordingly.

[0062] For example, using P1, P2, and P3, the three-dimensional Euclidean distance between P1 and P2 is approximately 36.401 mm, and the angle between the principal directions of the fruit stalk is 10°. Substituting these values ​​into formula (1) yields... The three-dimensional Euclidean distance between P1 and P3 is approximately 52.440 mm, and the included angle between the principal directions of the fruit stalk is 100°. The three-dimensional Euclidean distance between P2 and P3 is approximately 16.583 mm, and the angle between the principal directions of the fruit stalk is 90°. Therefore, P1 and P2, and P2 and P3 respectively satisfy the example neighborhood parameters. P1, P2 and P3 can form an initial clustering unit through neighborhood connection and enter the next clustering correction process.

[0063] In this second embodiment, after forming the initial clustering unit, step S300 may further include:

[0064] S310. Calculate the local density of fruit stalk picking points in each initial cluster unit to obtain the corresponding average density;

[0065] S320. Determine the consistency of the fruit stalk direction based on the distribution of the included angles between the main direction vectors of the fruit stalk.

[0066] S330. Adjust the neighborhood parameters of density clustering according to the average density and the consistency of the fruit stalk direction to form the clustering criterion;

[0067] S340. Calculate the composite similarity measure between fruit stalk picking points at the boundary of adjacent cluster units, compare it with the merging threshold, and form the merging condition.

[0068] Steps S310 to S340 may further include the following processes:

[0069] First, within each initial cluster unit, the number of other picking points whose composite similarity metric value is no greater than the neighborhood parameter is used as the local density of the current picking point, and the average value of all local densities within that cluster unit is calculated. Then, the consistency of the pedicel orientation is determined based on the length of the resultant vector of the pedicel principal direction vectors within the cluster unit. The pedicel principal direction vectors participating in the resultant vector calculation are all normalized vectors after the forward unification completed in step S130. The consistency of the pedicel orientation can be calculated using the following formula:

[0070] (2)

[0071] in, Initial clustering unit The direction of the fruit stalk is consistent, dimensionless, and the range of values ​​is [value missing]. ; For the first cluster unit Normalized principal direction vector of fruit stalk at each fruit stalk picking point, dimensionless; The number of effective stalk picking points within a cluster unit is represented by the number of points. When there is only one effective stalk picking point within a cluster unit, its stalk orientation consistency is recorded as 1. The stalk orientation consistency threshold is determined by the statistical analysis of stalk orientations of similar fruits and the end effector attitude adjustment range (e.g., the stalk orientation consistency threshold is 0.80). The merging threshold of adjacent cluster units is determined by the statistical analysis of the composite similarity metric of boundary picking points (e.g., the merging threshold is 0.25).

[0072] For each initial clustering unit C, if there are no picking points with a local density higher than the average density of the clustering unit, or if the consistency of the fruit stalk direction of the clustering unit is not lower than the aforementioned fruit stalk direction consistency threshold, the current neighborhood parameters and clustering results remain unchanged. If there are picking points with a local density higher than the average density and the consistency of the fruit stalk direction is lower than the aforementioned fruit stalk direction consistency threshold, the composite similarity metric values ​​within the clustering unit that are not greater than the current neighborhood parameters are used as candidate neighborhood parameters in descending order, and the neighborhood parameters are decreased sequentially. After each adjustment, adjacency relationships are established between picking points whose composite similarity metric values ​​are not greater than the corresponding candidate neighborhood parameters, and the interconnected picking points form candidate sub-clustering units.

[0073] When the consistency of the fruit stalk direction in each candidate sub-cluster unit is not lower than the fruit stalk direction consistency threshold, the largest candidate neighborhood parameter that satisfies this condition is selected as the adjusted neighborhood parameter, and the corresponding connected component is separated into sub-cluster units; the connected component containing the picking point with a local density higher than the average density is the corresponding region. If none of the candidate neighborhood parameters can form a connected component that satisfies the aforementioned condition, the previous stable clustering result is retained.

[0074] After two adjacent cluster units are separated, the composite similarity metric of all cross-unit picking point pairs between the two cluster units is calculated. The minimum value among these is taken as the boundary composite similarity metric, and the corresponding picking point pair is determined as the boundary picking point pair. When the minimum value is not greater than the aforementioned merging threshold, the two cluster units are merged. Each cluster-shaped picking task unit obtained after separation and merging saves the statistical results of member picking points, three-dimensional spatial range, and fruit stem direction, and transmits them to the work area division process.

[0075] For example, using P1, P2, and P3, under the condition of ε=0.45, the local densities of P1, P2, and P3 are 1, 2, and 1 respectively, and the average local density is 4 / 3. For ease of example calculation, it is assumed that the normalized pedicel principal direction vectors of the three are located in the same reference plane, and the direction angles are all determined relative to the same reference axis in the reference plane. The pedicel principal directions of the three are set to 0°, 10°, and 100° respectively, and substituted into formula (2) to obtain .

[0076] The local density of P2 is higher than the average local density, and the consistency of the stem orientation of the initial cluster units is less than 0.80. Therefore, the neighborhood parameters are adjusted in descending order of the composite similarity metric. Candidate neighborhood parameters are selected as follows: At this point, P1, P2, and P3 remain connected through neighborhood relations, and the consistency of the fruit stalk direction remains approximately 0.717; the neighborhood parameter is further adjusted to... At this point, P1 and P2 remain connected, while P3 is disconnected from both. The consistency of the stem direction of the connected components formed by P1 and P2 is approximately 0.996, while the consistency of the stem direction of P3 as a single-point connected component is 1, both not lower than 0.80. Therefore, 0.241 is used as the neighborhood parameter after this adjustment to separate the connected components formed by P1 and P2 from P3, resulting in clustered picking task units U1 and U2. For adjacent initial clustered units, after calculating the composite similarity metric of all cross-unit picking point pairs between the two clustered units, the minimum value corresponds to the boundary picking points P5 and P6, and this minimum value is 0.119, which is less than the merging threshold of 0.25. Therefore, the corresponding clustered units are merged into clustered picking task unit U3. U1, U2, and U3 continue into the work area division process.

[0077] When there is only one valid picking point in a cluster unit, the picking point is retained as a single-point cluster picking task unit; when there are multiple valid picking points in a cluster unit but the number of effective handle main directions is insufficient, the previous stable clustering result is retained, and separation is not performed based on invalid direction statistics; when two consecutive separations and merges yield opposite results, the previous stable clustering result is retained, and the repeated correction of the current crown width is stopped.

[0078] In this second embodiment, after obtaining the cluster-shaped harvesting task unit, step S400 may further include:

[0079] S410. Determine the core working area of ​​each harvesting robot arm based on its installation position on the frame and the corresponding accessible space.

[0080] S420. Expand the boundaries of adjacent core operation areas according to the spatial range of the clustered picking task unit to form an overlapping operation area that can be reached by adjacent picking robotic arms.

[0081] S430: Assign clustered picking task units located in non-overlapping work areas to the corresponding picking robotic arms.

[0082] Steps S410 to S430 may further include the following processes:

[0083] In this embodiment, the three harvesting robotic arms are respectively mounted on the upper, middle, and lower crossbeams of the portal frame. The lateral movement screw sliding module on each crossbeam is used to change the horizontal position of the corresponding robotic arm base. The core working area is determined based on the robotic arm base installation height, lateral movement range, joint limits, and workspace calibration results (for example, the core working areas of the lower, middle, and upper robotic arms are 700–920 mm, 980–1200 mm, and 1260–1480 mm respectively along the vertical direction of the unified robot coordinate system).

[0084] Subsequently, the spatial span of the current cluster-picking task unit within the canopy width along the normal direction of the adjacent core working area is calculated, and the boundary of the adjacent core working area is expanded in conjunction with the positioning margin of the robotic arm. The spatial span is derived from the member coordinates of the current cluster-picking task unit, and the positioning margin of the robotic arm is derived from the workspace calibration error and structural design parameters (for example, the vertical span of the largest task unit within the current canopy width is 60mm, the positioning margin is 20mm, and the adjacent boundaries are expanded outward by 50mm respectively). As a result, an overlapping working area of ​​930-970mm is formed between the lower robotic arm and the middle robotic arm, and an overlapping working area of ​​1210-1250mm is formed between the middle robotic arm and the upper robotic arm.

[0085] For example, in U1, the vertical coordinates of P1 and P2 are 940mm and 945mm respectively, located in the overlapping work area of ​​the lower and middle robotic arms. Therefore, U1 enters the reachability screening. The vertical coordinates of the members of U3 are between 1080 and 1100mm, located in the non-overlapping core work area of ​​the middle robotic arm. Therefore, U3 is directly assigned to the middle robotic arm. If a cluster-shaped picking task unit does not intersect with any expanded work area, the relative position of the robot and the crown is changed by moving the lead screw sliding module laterally or by moving the chassis, and then its work area assignment is recalculated.

[0086] In this second embodiment, for clustered picking task units entering overlapping work areas, S500 may further include:

[0087] S510. Obtain the distance data between the fruit stalk picking point and each reachable picking robot arm base in each cluster picking task unit.

[0088] S520: Obtain the length data of the current harvesting queue for each reachable harvesting robotic arm;

[0089] S530. Based on the distance data and length data, a comprehensive cost is calculated for each reachable picking robotic arm;

[0090] S540. The robot arm with the lowest overall cost is selected as the target picking robot arm, and the clustered picking task unit is added as a whole to its picking operation queue.

[0091] Steps S510 to S540 may further include the following processes:

[0092] First, inverse kinematics reachability verification is performed based on the three-dimensional spatial coordinates of all fruit stalk picking points within the cluster picking task unit and the workspace model of each picking robotic arm. If any member picking point exceeds the workspace or the corresponding joint solution exceeds the joint limit, the robotic arm is excluded from the set of reachable robotic arms in the current task unit. Subsequently, the representative distance of the task unit is calculated for the remaining reachable robotic arms, and the remaining task length of the current picking operation queue for each robotic arm is read.

[0093] The overall cost of assigning cluster-picking task units to the reachable picking robotic arm can be calculated using the following formula:

[0094] (3)

[0095] in, Cluster picking task unit Assigned to the picking robotic arm The overall cost is dimensionless; The average distance from all fruit stalk picking points within the task unit to the base of the picking robotic arm, in mm; The remaining task length in the current harvesting queue of the robotic arm, in units of tasks; This is a distance reference scale, in mm; This serves as a reference measure for queue length, with units of queues. and These are the distance weight and the queue length weight, and are dimensionless. For task units A collection of reachable robotic arms for harvesting; A robotic arm is used for target reaping. The distance reference scale is derived from the workspace calibration results of the robotic arm (e.g., The queue length reference scale is derived from the queue capacity allowed by the control system (e.g., The distance weight and queue length weight are determined by historical valid task statistics and on-site initialization configuration (e.g., [number] tasks). , ).

[0096] For example, using U1, both the lower robotic arm A and the middle robotic arm B can reach U1, but the upper robotic arm C cannot. The average distances from U1 to the bases of robotic arms A and B are 420mm and 300mm, respectively, and the remaining task lengths of the current picking queues for robotic arms A and B are 4 and 1, respectively. Substituting into formula (3) yields... , Therefore, robotic arm B is designated as the target harvesting robotic arm, and the task identifier of U1, the member harvesting point, and the member execution order are added to the harvesting operation queue of robotic arm B. If the overall costs are the same, the selection is made in the order of shorter queue length, closer base distance, and preset robotic arm order.

[0097] In this second embodiment, after the accessibility screening and deterministic comprehensive cost calculation are completed, and before the collaborative operation of the picking operation queue described in step S600, the task allocation within the overlapping operation area can be further performed in step S500 as follows:

[0098] S550: The joint states of each picking robotic arm, the three-dimensional spatial coordinates of the cluster picking task unit, the end positions of adjacent picking robotic arms and the occupancy status of the overlapping work area are combined into a joint state. The task unit selection and picking operation queue update are combined into a joint action. Reward information is constructed based on the end distance, task completion status, multi-arm parallel status and operation time.

[0099] S560. Train a multi-agent policy model using the joint state, joint action, and reward information, determine the task association degree through a self-attention mechanism, and determine the target picking robot arm by combining spatial accessibility, base distance, and picking queue length.

[0100] In this second embodiment, the multi-agent policy model is trained offline using an independent proximal policy optimization method. Three harvesting robotic arms maintain policy networks with identical structures and independently updated parameters. Joint states in the joint state are obtained by the robotic arm encoder. The three-dimensional spatial coordinates of the clustered harvesting task units are derived from the aforementioned clustering results. The end-effector positions of adjacent harvesting robotic arms are calculated using forward kinematics. The occupancy status of overlapping work areas is maintained by the task allocation module. Joint actions correspond to task unit selection and harvesting queue updates. Reward information is based on end-effector distance, task completion status, multi-arm parallel state, and work time, without adding reward items unrelated to the task allocation in this case.

[0101] The self-attention mechanism generates a candidate order based on the correlation between candidate clustered picking task units and the states of each picking robotic arm. The candidate results still need to be verified by reachability, base distance, and picking queue length. For clustered picking task units that have been assigned to each robotic arm, the robotic arm controller forms the local execution order of member picking points according to the online inference results of the model; in the absence of a strategy model, the member picking points are sorted according to the spatial traversal order from the outside to the inside and from the top to the bottom, or by using the shortest path planning algorithm.

[0102] For example, using U1, the remaining task lengths of the picking queues for robotic arms A, B, and C in the model input are 4, 1, and 2 respectively, and the degree of association between the three outputs of the self-attention mechanism and the tasks of U1 are 0.31, 0.62, and 0.07 respectively. The reachability check first excludes robotic arm C, the model candidate result points to robotic arm B, and then the check is completed by the comprehensive cost of 0.440 obtained by formula (3). Therefore, U1 is still added to the picking queue of robotic arm B as a whole. When U1 includes P1 and P2, robotic arm B first executes P1 located outside the crown, then executes P2, and writes this local order into the member execution order of U1.

[0103] When the input state of the multi-agent strategy model is missing, the model output points to an unreachable robotic arm, or the task association result cannot pass the comprehensive cost verification, the model output is not used directly, but the deterministic allocation result formed in step S500 is used instead.

[0104] like Figure 3 As shown, when each harvesting robotic arm operates collaboratively according to the harvesting operation queue, the collaborative control module triggers online modulation based on the end-effector spacing of adjacent harvesting robotic arms and the movement trend into the overlapping work area, and completes task snapshot saving, avoidance execution, recovery determination, task snapshot reading, and continuation of unfinished harvesting tasks in chronological order. Specifically, in this embodiment two, the following steps are further performed in step S700:

[0105] S710. Obtain the motion status of adjacent picking robotic arms according to the control cycle, and calculate the end-effector distance and the motion trend into the overlapping work area;

[0106] S720. Online modulation is triggered when the end-to-end spacing is less than the safety threshold or when adjacent harvesting robotic arms will enter the overlapping work area simultaneously.

[0107] S730. In the inverse kinematics solution, the end-effector pose deviation and inter-arm collision constraint are used to form a weighted cost function. The joint angle is used as the variable to be solved for nonlinear optimization to obtain the joint control quantity.

[0108] Wherein, steps S710 to S730 further include:

[0109] The collaborative control module reads the joint states, end effector positions, and current trajectories of adjacent harvesting manipulators according to the control cycle of the manipulator controller. The control cycle is determined by the real-time scheduling capability of the motion controller and the joint feedback frequency (e.g., a control cycle of 0.05 s). The prediction time period for entering the overlapping work area is determined by the nominal trajectory sampling interval and the size of the overlapping area (e.g., a prediction time period of 0.30 s). The safety threshold is jointly determined by the geometric envelope of the end effector of adjacent harvesting manipulators, structural calibration errors, and control margins (e.g., a safety threshold of...). ).

[0110] When the end-effector distance between adjacent harvesting robotic arms is less than a safety threshold, the collaborative control module triggers online modulation. When it is predicted that any related harvesting robotic arm will enter the overlapping work area within the aforementioned prediction period, the collaborative control module reads the occupancy status of the overlapping work area. If adjacent harvesting robotic arms have already occupied or will also enter the same overlapping work area within the prediction period, online modulation is triggered. During the inverse kinematics solution process, the weighted cost function composed of the end-effector pose deviation and the inter-arm collision constraint with the end-effector distance as the constraint quantity can be expressed by the following formula:

[0111] (4)

[0112] in, The weighted cost of the candidate joint angle combination vector is dimensionless. The candidate joint angle combination vector of the relevant harvesting robotic arm participating in online modulation, in rad; This refers to the set of allowed joint angle combinations for the joint limits of the relevant harvesting robotic arm, in rad. and These are, respectively, the candidate end-effector pose combinations and the online modulated target pose combinations of the dimensionless related harvesting robotic arm; For harvesting robotic arms With harvesting robotic arm The minimum spacing of the end effector geometric envelope under candidate joint angle combinations, i.e., the end effector spacing, in mm; and The weights are dimensionless, representing the end-effector pose deviation weights and inter-arm collision constraint weights. These two weights are determined by the trajectory tracking requirements and collision constraint priorities in the simulation (e.g., ...). , ).

[0113] For example, using robotic arms A and B, the system detects that the current end-effector distance between them is 170mm, which is less than the safety threshold of 180mm, thus triggering online modulation. Candidate joint solutions are obtained through nonlinear solving. and ,in, The corresponding dimensionless end-effector pose error is 0.04, and the candidate end-effector spacing is 160mm; The corresponding dimensionless end-effector pose error is 0.06, and the candidate end-effector spacing is 200mm. Substituting into formula (4) yields... , Therefore, The joint control quantity obtained through online modulation is then transmitted to the relevant robotic arm controller to perform obstacle avoidance.

[0114] In this second embodiment, after online modulation triggering, in order to maintain the continuity of unfinished harvesting tasks, step S700 further performs the following steps:

[0115] S740: When online modulation is triggered, save the current planned position of the relevant harvesting robotic arm and its harvesting operation queue status;

[0116] S750. Control at least one related harvesting robotic arm to perform avoidance according to the joint control amount, and continuously obtain the end-effector distance between adjacent harvesting robotic arms.

[0117] S760. When the end spacing recovers to above the safety threshold, exit online modulation, read the current planned position and the status of the picking operation queue, and connect the unfinished picking tasks to the corresponding picking operation queue.

[0118] Wherein, steps S740 to S760 further include:

[0119] The current planned position and the picking operation queue status are saved in the same task snapshot. The current planned position includes the cluster picking task unit being executed, the member picking point, and the corresponding trajectory position; the picking operation queue status includes the queue order, the current task identifier, the completed members, and the incomplete members. After the task snapshot is written and verified by the task identifier and the queue version, the robotic arm controller executes the joint control quantity obtained by formula (4).

[0120] During the avoidance process, the collaborative control module continues to acquire the end-effector distance between adjacent harvesting robotic arms according to the aforementioned control cycle. To avoid repeatedly entering and exiting online modulation near the safety threshold, the recovery margin and the number of consecutive effective cycles are determined by the structural calibration error, control cycle, and robotic arm braking response (for example, the recovery margin is 20mm, and the number of consecutive effective cycles is 3). That is, when the end-effector distance is detected to be no less than 200mm for three consecutive control cycles, and the aforementioned overlapping work area entry trigger condition is no longer met, the online modulation is exited and a task snapshot is read.

[0121] For example, after completing P1 in U1, robotic arm B, when preparing to execute P2, shows a tendency to enter the overlapping work area together with robotic arm A. The system saves the current planned position "before U1-P2" and the picking operation queue status "U1, U3", and performs avoidance according to the joint control quantity output by formula (4); the end distances detected in the subsequent three consecutive control cycles are 202mm, 205mm and 208mm, respectively, all of which are not less than 200mm, and the system confirms that the aforementioned overlapping work area entry trigger condition is no longer met, so it exits online modulation, reads the saved current planned position and picking operation queue status, and continues execution from P2.

[0122] If the task snapshot is incomplete, the task identifier is inconsistent, or the picking queue has been updated during the avoidance period, the old task snapshot will not be used to overwrite the current queue. Instead, the robotic arm will remain in a safe state, and the corresponding cluster picking task unit will be resubmitted to the task allocation module. If the robotic arm has already gripped the fruit, the end effector will remain closed and the robot will continue to move to the collection system without returning to the fruit stalk position to repeat the cutting.

[0123] After the harvesting robotic arm reaches the fruit stalk picking point, the drive motor of the integrated shearing and clamping end effector drives the rotating shaft and moving blade to rotate via a worm gear and worm wheel. The moving blade and fixed blade cut the fruit stalk and then hold the fruit in a closed position. The self-locking action of the worm gear transmission pair is used to maintain the closed state. The harvesting robotic arm transfers the fruit along a preset unobstructed path to the collection system located at the bottom or side of the portal frame. Before release, the end effector is adjusted to an open-up or flat position before releasing the fruit. A buffer guide structure located between the robotic arm and the collection system receives the released fruit.

[0124] For example, in the scenario where robotic arm B completes P2, after robotic arm B reaches the corresponding fruit stalk pose for P2, the end effector sequentially performs stalk cutting and fruit clamping, and moves along a preset unobstructed path to the collection system at the bottom of the frame. After the end effector switches to an upward-opening posture, it releases the fruit, and the buffer guide structure guides the fruit into the collection box. The completion status of P2 is written to U1. Once all members of U1 are completed, it is removed from the picking queue of robotic arm B, and robotic arm B continues to read U3.

[0125] In cases of missing image frames, asynchronous depth and color images, invalid calibration parameters, invalid composite similarity calculations, queue writing failures, interrupted robotic arm feedback, or joint control quantities exceeding joint limits, the system does not replace the core input with unconfirmed default values. Instead, it retains the current crown task identifier and the most recent stable clustering result, task allocation result, or queue state. If it can recover through re-acquisition, recalibration, rollback, deterministic allocation, or replanning, it can continue execution from the most recent stable state. If it cannot recover, it stops the relevant robotic arm and retains the corresponding cluster picking task unit.

[0126] Accordingly, this invention uses three-dimensional spatial coordinates and the main direction of the fruit stalk to form a composite similarity metric. Based on the consistency of local density and fruit stalk direction, the initial clustering units are separated and merged. The resulting clustered picking task units are then integrated into the robotic arm reachability screening, base distance and queue length joint allocation, multi-agent task association verification, online trajectory modulation and task state recovery. This ensures that the calculation results of each step are incorporated into the subsequent task allocation or control process, thereby completing the continuous collaborative picking of fruits within the canopy of unstructured fruit trees by multiple robotic arms.

[0127] Example 3:

[0128] Based on the same general inventive concept, the present invention also provides an operating system for a multi-arm cooperative harvesting robot, comprising:

[0129] The perception clustering module is used to acquire fruit crown images, identify fruit stalk picking points and extract their three-dimensional coordinates and main direction of the fruit stalk, and perform density clustering based on spatial distance and main direction difference of the fruit stalk; when the local density is higher than the unit average density and the consistency of the fruit stalk direction does not meet the clustering criteria, the corresponding region is separated; when the composite similarity measure of the boundary of adjacent units meets the merging condition, they are merged, and clustered picking task units are output.

[0130] The task allocation module is used to divide the overlapping work area according to the installation position of the picking robot arm, the reachable space and the task unit, filter the reachable robot arm, determine the target picking robot arm according to the distance from the task unit to the base of the reachable robot arm and the length of the picking operation queue, and add the task unit as a whole to its picking operation queue.

[0131] The collaborative control module is used to control each picking robotic arm to run according to the picking operation queue. When it enters the overlapping area or the arm spacing is less than the safety threshold, it modulates the motion trajectory online and saves the current planned position and the picking operation queue status. After the arm spacing is restored, it reads the saved status and continues the unfinished tasks.

[0132] The picking execution module is used to drive the picking robotic arm according to the joint control values, and control the end effector to complete the fruit stem cutting, fruit clamping, transfer and release.

[0133] like Figure 4 As shown, the operating system is mounted on a tracked mobile chassis. The mobile chassis moves within the orchard according to a preset map and supports a portal frame formed by connecting square aluminum profiles. The portal frame includes vertical columns and three layers of horizontal beams (upper, middle, and lower). Each layer of horizontal beams is equipped with a transverse movement screw sliding module, a horizontal linear guide rail, and a moving slide. A servo motor drives the moving slide to move along the horizontal linear guide rail, enabling the corresponding picking robotic arm to initially position the fruit at different horizontal levels.

[0134] Three robotic arms for harvesting are mounted on three layers of moving slides. Each robotic arm includes a base joint, intermediate joints, and end joints connected to the moving slides. Rotary motor drive components are installed at the joints. Each robotic arm is independently controlled by its corresponding controller and can extend and retract outwards from the portal frame. The robotic arm linkages employ a lightweight structure to reduce moment of inertia. The visual recognition system includes a first depth camera located in the middle of the frame and a second depth camera located in front of the upper crossbeam. Both cameras share the field of view to cover the target fruit tree canopy and transmit color and depth images to an onboard computer.

[0135] The onboard computer integrates image processing, task allocation, and collaborative control programs, and connects to the lateral movement lead screw sliding module, the servo motors of each robotic arm joint, and the drive motors of the end effector via a motion control interface. The image processing program generates the three-dimensional spatial coordinates of the fruit stalk picking point and the main direction of the fruit stalk. The task allocation program generates clustered picking task units and corresponding picking operation queues. The collaborative control program reads the joint status, end effector position, and overlapping work area occupancy status, and outputs nominal joint control values ​​or online modulated joint control values ​​to the robotic arm controller. Joint encoders, position encoders, and motor status feedback are returned to the onboard computer to update the robotic arm motion status and the picking operation queue status.

[0136] The integrated shearing and clamping end effector is connected to the end of the harvesting robotic arm via a flange. It includes a motor support base, a drive motor, a worm gear, a worm wheel, a rotating shaft, a moving blade, and a fixed blade. The drive motor is fixed within the housing of the motor support base, and its output shaft is connected to the worm gear. The worm gear meshes with the worm wheel fixed on the rotating shaft. The moving blade is fixedly connected to the rotating shaft and coaxially arranged with the worm wheel. The fixed blade is fixed relative to the motor support base. When the drive motor operates, the worm gear drives the worm wheel, rotating shaft, and moving blade to rotate. The moving blade and fixed blade form a shearing engagement and remain closed after shearing to clamp the fruit.

[0137] For example, after the mobile chassis reaches the target mango tree corresponding to task T-001, the first and second depth cameras acquire canopy images, and the perception clustering module forms U1, U2, and U3. The task allocation module writes the entire U1 into the picking operation queue of the middle-level robotic arm B. When robotic arms A and B have overlapping operation areas and conflict tend to occur, the collaborative control module saves the current planned position and picking operation queue status, and outputs the online modulated joint control quantities. After robotic arm B completes the avoidance and recovers, it continues to perform fruit stem cutting, fruit clamping, transfer to the collection system, and release from P2, and then returns the completion status to the onboard computer.

[0138] The perception clustering module, task allocation module, collaborative control module, and harvesting execution module transmit image data, clustered harvesting task units, harvesting operation queues, joint control quantities, and execution status according to the same crown width task identifier, so that visual recognition, task allocation, robotic arm control, and end-point harvesting remain consistent.

[0139] The above description is merely a preferred embodiment of the technical solution of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for operating a multi-arm collaborative harvesting robot, characterized in that, include: Acquire image data of the fruit canopy area, identify the fruit stalk picking point, and extract the three-dimensional spatial coordinates of each fruit stalk picking point and the main direction of the fruit stalk. A composite similarity metric is constructed based on the spatial distance between any two fruit stalk picking points and the difference in the main direction of the fruit stalk, and density clustering is performed. When the local density within the initial cluster unit is higher than its average density and the consistency of the fruit stalk direction does not meet the clustering criteria, the corresponding region is separated into sub-cluster units. When the composite similarity measure at the boundary of adjacent cluster units meets the merging condition, they are merged to obtain clustered picking task units. The overlapping work areas are defined based on the installation location of the harvesting robot arm, the accessible space, and the spatial range of the task unit. Accessibility screening is performed on task units within the overlapping area to exclude unreachable picking robotic arms. Target picking robotic arms are determined based on the distance from the task unit to the base of each reachable picking robotic arm and the length of its picking queue. The task unit is then added to the picking queue of the target picking robotic arm. Control each harvesting robotic arm to operate in coordination according to its corresponding harvesting task queue; When it is predicted that any picking robotic arm will enter the overlapping area or the arm spacing is less than the safety threshold, the relevant motion trajectory is modulated online and the current planned position and picking operation queue status are saved. The avoidance is performed according to the modulated joint control amount. After the spacing is restored to above the safety threshold, the saved status is read and the unfinished picking task is continued. The control end effector completes the fruit stem cutting, fruit clamping, transfer to the collection system and release.

2. The operating method of the multi-arm cooperative harvesting robot according to claim 1, characterized in that, The extraction of the three-dimensional spatial coordinates and main direction of the fruit stalk at each fruit stalk picking point includes: Extract the candidate fruit region and the fruit stalk picking point from the image data; Based on the depth information of the image data, the picking points of each fruit stalk are transformed to the same spatial coordinate system; Extract the fruit stalk features in the local area corresponding to each fruit stalk picking point, and fit the main direction vector of the fruit stalk based on the fruit stalk features.

3. The operating method of the multi-arm cooperative harvesting robot according to claim 2, characterized in that, The construction of the composite similarity metric and the performance of density clustering include: Calculate the three-dimensional Euclidean distance between any two fruit stalk picking points and the angle between the corresponding principal direction vectors of the fruit stalks; The three-dimensional Euclidean distance and the included angle are weighted and fused to obtain a composite similarity metric. Based on the composite similarity metric, density clustering is performed on the fruit stalk picking points to form multiple initial clustering units.

4. The operating method of the multi-arm cooperative harvesting robot according to claim 3, characterized in that, The determination of the clustering criteria and merging conditions includes: The local density of fruit stalk picking points within each initial cluster unit is statistically analyzed to obtain the corresponding average density. The consistency of the fruit stalk direction is determined by the distribution of the included angles between the principal direction vectors of the fruit stalk. The neighborhood parameters of density clustering are adjusted based on the average density and the consistency of the fruit stalk direction to form the clustering criterion; Calculate the composite similarity measure between fruit stalk picking points at the boundaries of adjacent cluster units, compare it with the merging threshold, and form the merging condition.

5. The operating method of the multi-arm cooperative harvesting robot according to claim 1, characterized in that, The division of work areas with overlapping adjacent boundaries includes: The core working area of ​​each harvesting robot arm is determined based on its installation position on the frame and the corresponding accessible space. The boundaries of adjacent core working areas are expanded according to the spatial range of the clustered picking task unit to form an overlapping working area that can be reached by adjacent picking robotic arms. Clustered picking task units located in non-overlapping work areas are assigned to the corresponding picking robotic arms.

6. The operating method of the multi-arm cooperative harvesting robot according to claim 5, characterized in that, The target-determining harvesting robotic arm includes: Obtain distance data between the fruit stalk picking point and each reachable picking robotic arm base within each cluster-shaped picking task unit; Obtain the length data of the current harvesting queue for each reachable harvesting robotic arm; The comprehensive cost of each reachable harvesting robotic arm is determined based on the distance and length data. The robot arm with the lowest overall cost is selected as the target harvesting robot arm, and the clustered harvesting task unit is added as a whole to its harvesting operation queue.

7. The operating method of the multi-arm cooperative harvesting robot according to claim 6, characterized in that, The target-determining harvesting robotic arm includes: The joint states of each picking robotic arm, the three-dimensional spatial coordinates of the cluster picking task unit, the end positions of adjacent picking robotic arms, and the occupancy status of overlapping work areas constitute a joint state. The selection of task units and the updating of the picking work queue constitute a joint action. Reward information is constructed based on the end distance, task completion status, multi-arm parallel status, and work time. The multi-agent policy model is trained using the joint state, joint action, and reward information. The degree of task association is determined through a self-attention mechanism, and the target harvesting robot arm is determined by combining spatial reachability, base distance, and harvesting queue length.

8. The operating method of the multi-arm cooperative harvesting robot according to claim 1, characterized in that, The online modulation includes: According to the control cycle, the motion state of adjacent picking robotic arms is obtained, and the end-effector spacing and the motion trend into the overlapping work area are calculated. Online modulation is triggered when the end-to-end spacing is less than the safety threshold or when adjacent harvesting robotic arms enter the overlapping work area simultaneously. In the inverse kinematics solution, the end-effector pose deviation and inter-arm collision constraint are used to form a weighted cost function. The joint angle is used as the variable to be solved for nonlinear optimization to obtain the joint control quantity.

9. The operating method of the multi-arm cooperative harvesting robot according to claim 8, characterized in that, The continuation of unfinished harvesting tasks includes: When online modulation is triggered, the current planned position of the relevant harvesting robotic arm and its harvesting queue status are saved; The joint control quantity is used to control at least one related harvesting robotic arm to perform avoidance, and the end-effector distance between adjacent harvesting robotic arms is continuously acquired. When the end-to-end spacing recovers to above the safety threshold, online modulation is exited, the current planned position and the status of the picking operation queue are read, and unfinished picking tasks are added to the corresponding picking operation queue.

10. A working system for a multi-arm collaborative harvesting robot, characterized in that, include: The perception clustering module is used to acquire fruit crown images, identify fruit stalk picking points and extract their three-dimensional coordinates and main direction of the fruit stalk, and perform density clustering based on spatial distance and main direction difference of the fruit stalk; when the local density is higher than the unit average density and the consistency of the fruit stalk direction does not meet the clustering criteria, the corresponding region is separated; when the composite similarity measure of the boundary of adjacent units meets the merging condition, they are merged, and clustered picking task units are output. The task allocation module is used to divide the overlapping work area according to the installation position of the picking robot arm, the reachable space and the task unit, filter the reachable robot arm, determine the target picking robot arm according to the distance from the task unit to the base of the reachable robot arm and the length of the picking operation queue, and add the task unit as a whole to its picking operation queue. The collaborative control module is used to control each picking robotic arm to run according to the picking operation queue. When it enters the overlapping area or the arm spacing is less than the safety threshold, it modulates the motion trajectory online and saves the current planned position and the picking operation queue status. After the arm spacing is restored, it reads the saved status and continues the unfinished tasks. The picking execution module is used to drive the picking robotic arm according to the joint control values, and control the end effector to complete the fruit stem cutting, fruit clamping, transfer and release.

Images