Multi-view dual-arm cooperative picking method

Abstract

The application discloses a multi-eye-double-arm cooperative picking method, which utilizes a multi-eye-double-arm cooperative picking system to realize picking, and specifically, image acquisition and pretreatment are first performed; a fruit recognition model is constructed and trained, and the image is detected to determine the fruit stem and the maturity of the fruit; the right eye D R of the binocular camera is detected through the fruit recognition model; the detection result is processed and analyzed, the maturity of the fruit and the pickability are determined, and the fruit string set is divided into four types, i.e., pickable, unpickable, no fruit stem found, and no part of the fruit found; and the fruit is picked based on the divided fruit string set type. The scheme can effectively improve the fruit recognition capability, solves the problem of missed picking of the fruit caused by light change or leaf shielding by combining the binocular camera and the visual camera, effectively reduces the damage to the fruit in the picking process by cooperating with the leaf supporting structure, and has high practical application and popularization value.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of agricultural picking robots, and particularly relates to a multi-view dual-arm cooperative picking method. BACKGROUND

[0002] To meet the needs of agricultural production, improve work efficiency, reduce labor costs, and reduce losses, picking robots have been developing and evolving since their inception. The rise of deep learning has triggered a revolution in the field of computer vision, and agricultural robots have also been innovated and optimized in combination with new technologies. For example, YOLO (You Only Look Once) is a popular target detection algorithm that uniquely converts the target detection task into a single regression problem, predicting the location and class of the target through a neural network, which can also achieve the accuracy and real-time requirements of picking and recognition to some extent.

[0003] In recent years, there has been a problem of fruit missing in the field of picking robots. This is because most current picking robots rely on visual systems, and if the fruit is blocked, the camera cannot obtain information about the fruit and cannot pick it, and currently only manual checking can be relied on. To solve this problem, the prior art takes appropriate measures, one of which is to use a blowing system to disturb the obstruction to reduce the possibility of fruit missing, however, this method will interfere with the target positioning and picking task, especially in the case of large leaves, it may not be able to successfully detect the blocked fruit; another method is to record the number and location of fruits during plant growth, which is feasible but also increases the workload in the early stage.

[0004] In addition, for the maturity determination of string-shaped fruits, the method based on image color is generally used, such as RGB-HSI, but this method has a higher requirement for light, and is prone to misjudgment under backlight or strong light. It is also a key problem to obtain target position information in a complex background. In the prior art, binocular camera positioning is a commonly used method, which captures image features for stereo matching and uses the parallax information of the dual camera to infer the depth and position of the target. In this process, the algorithm needs to handle complex scenes, therefore, a targeted algorithm is urgently needed to deal with problems such as light changes and obstructions. SUMMARY

[0005] To solve the problem of fruit missing caused by light changes or obstructions in the prior art, a multi-view dual-arm cooperative picking method is proposed, which combines the GD-YOLOv8P algorithm with dual robotic arms, and cooperates with binocular cameras, vision cameras and fill light to improve the maturity discrimination ability of fruits, solve the problem of fruit missing caused by leaf obstruction, and effectively reduce the damage to fruits during picking.

[0006] The application is implemented by adopting the following technical scheme: a multi-eye-double-arm cooperative picking method, which realizes picking by using a multi-eye-double-arm cooperative picking system, the multi-eye-double-arm cooperative picking system comprising a mobile chassis vehicle, a fruit collection box, a left mechanical arm assembly, a right mechanical arm assembly, a controller and a control panel; a vision camera and a leaf supporting structure are connected to the end of the mechanical arm of the left mechanical arm assembly, and a binocular camera, a picking mechanical clamp and a light supplementing lamp are connected to the end of the mechanical arm of the right mechanical arm assembly; a computer program corresponding to the multi-eye-double-arm cooperative picking method is loaded or called in the controller, and the picking method comprises the following steps:

[0007] Step A, image acquisition and preprocessing;

[0008] Step B, constructing a fruit recognition model based on GD-YOLOv8P and training, detecting the image, and judging the fruit stem and fruit maturity: the binocular camera right eye D R transmits back the detected image; and the detection result is processed and analyzed, the fruit string set is divided into four types of CP, NP, NS and NH;

[0009] Step C, fruit picking based on the divided fruit string set type:

[0010] Step C1, picking the CP type: through multi-point positioning, the corresponding fruit stem picking point distance is calculated, the controller sends the three-dimensional coordinates of the fruit stem picking point to the right mechanical arm assembly, combines the binocular camera positioning, calls the right mechanical arm to perform picking work, and places the picked fruit in the fruit collection box;

[0011] Step C2, picking the NS type and the NH type: the left mechanical arm assembly is controlled to move to the vicinity of the target, the vision camera is used to detect the obstruction, the left mechanical arm assembly moves according to the detection result, the leaf supporting structure is used to support the leaf, the fruit is identified, and if the identification result is the CP type, picking is performed;

[0012] Step C3, when only the NP type of fruit is left in the current area: the left mechanical arm assembly is controlled to perform fixed cruising, moves from the ground upwards, supports the leaf to detect the missed fruit, if the missed fruit is detected, the picking work cycle is entered, the leaf is pushed away by the left mechanical arm assembly and the right mechanical arm assembly is assisted to position and pick; if no missed fruit is detected, the mobile chassis vehicle automatically drives to the next area to work.

[0013] Further, in step B, the fruit recognition model training process is as follows:

[0014] (1) Image acquisition and processing: Construct dataset and store in different folders according to strong light, normal and weak light picking environment, label the images in each folder, the label includes fruit stem, high maturity, medium maturity and low maturity, and mark the category and position, create training set, validation set and test set;

[0015] (2) Model training: Train different picking environment files respectively, and quantize the trained model, configure training parameters including but not limited to learning rate, batch size, training time, data enhancement strategy, during training, the model constantly updates the weight to adapt to the target detection task;

[0016] (3) Model verification: Use the validation set to evaluate the performance of the model, including accuracy, recall rate, F1 score, adjust the model parameters and training strategy to improve the performance.

[0017] Further, in step B, the fruit maturity and pickable determination principle is as follows:

[0018] (1) Extract the information of all detection boxes, including category, position;

[0019] (2) Calculate the edge length, area and center of the detection box with high maturity, medium maturity and low maturity respectively;

[0020] (3) Calculate the longitudinal center line L of the detection result as Stem: (u ga , v ga ) is the top left corner coordinate of Stem detection box, (u gb , v gb ) is the right bottom corner coordinate of Stem detection box, the starting point coordinate of center line L is (U, V), and the ending point coordinate is Wherein, V = vga, is the general length of fruit stem;

[0021] (4) Shift the longitudinal center line L left and right by a certain distance to get L' and L'', determine the fruit detection box contacted by the longitudinal center line L during the translation process by judging whether the longitudinal center line L falls within the fruit maturity detection box, and mark it as set A;

[0022] (5) Remove the fruit in set A that is not connected to the fruit stem: remove the maximum and minimum area in set A, and calculate the average value μ s of the remaining detection box area, calculate the deviation σ sn of all detection box area, and calculate the ratio Pr n of average value and deviation:

[0023]

[0024] Pr n The fruit frame with a value greater than 0.1 is considered not to belong to the fruit connected by the pedicel, and is excluded;

[0025] (6) Analyzing the detection frame set:

[0026] The detection frame set containing low maturity type is recorded as not picking NP;

[0027] The detection frame set not containing low maturity type, but the high maturity type accounts for no more than 75% of the total number of detection frame sets, is recorded as not picking NP;

[0028] The detection frame set not containing low maturity type, and the high maturity type accounts for more than 75% of the total number of detection frame sets, is recorded as picking CP;

[0029] The detection frame set not being crossed by the longitudinal center line L, i.e., the type of detection frame set fails to find a matching pedicel detection frame, is recorded as no pedicel NS;

[0030] The difference between the number of left and right pedicel detection frames is greater than 60%, and the length of the pedicel detection frame in the v direction is less than That is, there is a problem of partial fruit being blocked, recorded as no partial fruit NH.

[0031] Further, in step B, the fruit recognition model based on GD-YOLOv8P includes an input layer, a backbone network, a Neck, and a detection head;

[0032] The backbone network adopts yolov8n architecture, including convolution modules, residual blocks, and SPPF, wherein the convolution module uses CBS structure, the CBS structure is composed of two-dimensional convolution Conv2d, BatchNorm2d, and SiLU activation function, the residual block uses C2f structure, the C2f structure is composed of a convolution module Conv, Split, n BottleNeck in series, and a connection layer Concat to fuse the upper features, and the SPPF is composed of a CBS convolution layer, three Maxpooling in series, and a connection layer Concat to fuse the upper features;

[0033] And the GD mechanism is introduced into the fruit recognition model, which includes a feature alignment module, an information fusion module, and an information injection module;

[0034] In the detection head, MPDIoU is used as the loss function of the boundary box regression instead of the default CIoU loss function; the output results of the detection head include pedicel, high maturity, medium maturity, and low maturity;

[0035] Further, in step C, when picking, multi-point binocular positioning is used, and the principle is as follows:

[0036] (1) Collect target area images through visual camera and binocular camera, identify and determine maturity of the image of right eye D R of binocular camera;

[0037] (2) Visual camera S C corrects, and the correction matrix is as follows:

[0038]

[0039] Wherein, R SC is a rotation matrix, T 3×1 is a translation matrix, and θ is a rotation angle;

[0040] (3) Perform stereo matching on [S C , D L ], [S C , D R ], and [D L , D R ] respectively to obtain depth maps;

[0041] (4) Select a CP type fruit that can be picked as a picking target, and obtain three groups of fruit stem picking point P point depths dist1, dist2, and dist3 from T3 depth map, and calculate the average of the three groups of depths as follows:

[0042]

[0043] Calculate the absolute value of the deviation of the three groups of depths from the average value respectively and compare the sizes:

[0044]

[0045] Take the depth dist′ measured by the camera combination corresponding to the minimum absolute value of deviation as the depth calibration parameter;

[0046] (5) Multi-point positioning is converted into binocular positioning: transform the D R coordinate system into the fruit stem picking point P point coordinate system, and the transformation formula is as follows:

[0047]

[0048] Wherein, f DR is the focal length of D R camera.

[0049] Further, in step A, the worker first selects the corresponding picking mode on the control panel according to the picking environment, at this time, the left and right mechanical arm assemblies remain standby state, and the moving chassis vehicle automatically drives to the working area; wake up the left and right mechanical arm assemblies, turn on the light supplementing lamp, and turn on the visual camera, take the current area through the binocular camera, and perform classification labeling and preprocessing operation on the collected image.

[0050] Compared with the prior art, the advantages and positive effects of the present application are that a multi-view dual-arm cooperative picking method combines YOLO algorithm and dual mechanical arms, mainly applied to picking of fruit clusters, and most of the traditional methods are single fruit picking, which has low picking efficiency. In the present scheme, a set of single fruits on the same cluster is established for ripeness determination, and the mature fruit cluster can be directly picked, which can more quickly and efficiently complete the picking task, and the dual mechanical arms work cooperatively to ensure clear vision, which helps to reduce misoperation and improve picking accuracy. In addition, multi-point target positioning corrects the target distance by calculating correction parameters, which can improve the target point positioning accuracy, so as to accurately cut off the fruit stem and collect, and reduce the fruit loss.

[0051] The present scheme proposes a GD-YOLOv8P fruit recognition model, which labels the fruits and fruit stems in the data set respectively by establishing a data set of different environmental light, so as to accurately obtain the target position information in a complex background, effectively improve the fruit recognition ability, and solve the fruit missing problem caused by light changes or leaf shading by combining binocular cameras and vision cameras, etc. In addition, through the cooperation of the leaf supporting structure, the damage to the fruits during picking is effectively reduced, and the practical application and popularization value are high. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 It is a dual-arm cooperative picking method flow diagram based on the GD-YOLOv8P algorithm of the embodiment of the present application;

[0053] Figure 2 It is a GD-YOLOv8P algorithm structure diagram of the embodiment of the present application;

[0054] Figure 3 It is a GD-YOLOv8P algorithm Head layer structure diagram of the embodiment of the present application;

[0055] Figure 4 It is a fruit cluster ripeness and pickable determination method diagram;

[0056] Figure 5 It is a vision camera correction diagram of the embodiment of the present application;

[0057] Figure 6 It is a left mechanical arm assembly end element structure diagram;

[0058] Figure 7 It is a right mechanical arm assembly end element structure diagram;

[0059] 1, leaf supporting structure; 2, vision camera; 3, light supplementing lamp; 4, binocular camera; 5, picking mechanical clamp. DETAILED DESCRIPTION

[0060] To better understand the above-described objects, features, and advantages of the present invention, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the present invention; however, the present invention may be practiced in other ways than those described herein, and therefore, the present invention is not limited to the specific embodiments disclosed below.

[0061] An embodiment of a multi-eye-dual-arm collaborative harvesting method utilizes a multi-eye-dual-arm collaborative harvesting system. This system includes a mobile chassis, a fruit collection box, a left robotic arm assembly, a right robotic arm assembly, a controller, and a control panel. The main body of the left robotic arm assembly is composed of six electrically connected servo motors. Figure 6 and Figure 7 As shown, the end of the robotic arm is connected to a vision camera 2 and a leaf-shaped structure 1. The main body of the right robotic arm assembly is also composed of six servo motors electrically connected together. The end of the right robotic arm is connected to a binocular camera 4, a picking mechanical clamp 5, and a supplementary light 3. The control panel is used to display the machine status, camera images, and select the picking mode. The controller loads or calls the computer program corresponding to the multi-view-dual-arm collaborative picking method to generate control commands to control the left / right robotic arm assembly to complete the picking work. The picking modes include strong light, normal, and weak light working modes.

[0062] In this embodiment, the visual camera works in conjunction with the binocular camera during the ranging process to detect missed fruits during the missed harvesting inspection. The leaf-supporting structure is an arc-shaped aluminum component fixed to the end of the robotic arm, with rubber covering the upper side. During the harvesting process and missed harvesting inspection, it lifts the leaves to expose the fruit without damaging the leaves. Simultaneously, the visual camera assists the binocular camera in acquiring depth information and locating image information, obtaining images more suitable for harvesting control, i.e., less obstruction, and at least the fruit stem harvesting point can be seen. The binocular camera has fruit recognition and depth ranging functions to acquire real-time image and depth information for subsequent construction of a 3D voxel map of the harvesting point area. The harvesting mechanical clamp has a double-layer structure, with a scissor-like structure on the upper layer and a clamping and fixing structure on the lower layer, which can simultaneously perform fruit stem separation and clamping operations. The supplementary light is used to supplement the image color light source in the fruit ripeness determination, and the type and color of the supplementary light can be changed according to different types of target fruits. The harvesting system has the advantages of flexible harvesting operations and high intelligence in response to different environments. The staff selects the corresponding harvesting mode on the control panel according to the harvesting environment, combined with... Figure 1 As shown, the harvesting method includes the following steps:

[0063] Step A: Select the picking mode and acquire images:

[0064] According to the picking environment, select the corresponding picking mode on the control panel, at this time, the left and right mechanical arm assemblies remain in standby state, the moving chassis vehicle automatically drives to the working area; wake up the left and right mechanical arm assemblies, turn on the light supplementing lamp, and start the visual camera, take the current area through the binocular camera, and perform classification labeling and preprocessing operations on the collected images;

[0065] Step B, build a fruit recognition model to detect the image, judge the fruit stem and fruit maturity;

[0066] Through the GD-YOLOv8P fruit recognition model, the right eye D R The returned picture is detected; and the detection result is processed and analyzed, and the fruit string set is divided into pickable CP, non-pickable NP, no stem NS, and no part of fruit NH;

[0067] Step C, fruit picking based on the divided fruit string set result:

[0068] Step C1, pick the pickable fruit (CP) type;

[0069] Through multi-point positioning, the corresponding fruit stem picking point distance is calculated and obtained, and the recognition result and distance are displayed on the control panel; the controller sends the three-dimensional coordinates of the fruit stem picking point to the right mechanical arm assembly, calls the right mechanical arm for picking work, and combines the binocular camera positioning to ensure the accuracy of the picking point position. The end of the right mechanical arm assembly clamps and separates the fruit and the plant, and places the fruit in the fruit storage box; repeat the above steps until the picking work of the pickable fruit (CP) type in the current area is completed;

[0070] Step C2, pick the no stem (NS) type and no part of fruit (NH) type

[0071] Control the left mechanical arm assembly to move to the target vicinity, enable the visual camera to detect the obstruction, and the left mechanical arm assembly moves according to the detection result, supports the leaf blade through the leaf supporting structure, identifies the fruit, and if the identification result is mature fruit, picks it, until the picking task of this type is completed;

[0072] Step C3, when the current area is only the non-pickable (NP) type fruit;

[0073] The left mechanical arm assembly is controlled to perform fixed cruising, moves upward from the ground, supports the blade to detect missed fruits, and the missed fruit detection also uses the GD-YOLOv8P recognition model. If a missed fruit is detected, the left mechanical arm assembly pushes away the leaves and assists the right mechanical arm assembly to perform positioning work, and then the right mechanical arm performs picking; if no missed fruit is detected, the left mechanical arm assembly waits, the moving chassis vehicle automatically drives to the next area, and the above steps are repeated.

[0074] In the picking process of the present example, the pressure sensor on the moving chassis vehicle monitors the weight of the harvested fruits in real time. If the weight reaches a critical value, it is determined that the fruit collection box is full, and the device will automatically return to the starting point. When the worker unloads the harvested fruits, the device starts the next round of picking.

[0075] The moving chassis vehicle is equipped with an electric track chassis, a battery, a navigation module, and a pressure sensor, and has automatic driving and quality detection functions. The fruit collection box is detachably connected to the top of the moving chassis vehicle. The mechanical arm body of the left mechanical arm assembly is composed of six electrically connected servos. The end of the mechanical arm is connected with a visual camera and a leaf supporting structure. The visual camera cooperates with the binocular camera during the distance measurement process and detects missed fruits during the missed fruit detection operation. The leaf supporting structure is an arc-shaped aluminum part fixed at the end of the mechanical arm, with a rubber cover on the upper side. It supports the leaves during the picking process and the missed fruit detection operation without causing damage to the leaves. The mechanical arm body of the right mechanical arm assembly is also composed of six electrically connected servos. The end of the mechanical arm is connected with a binocular camera, a picking mechanical clamp, and a fill light. The binocular camera has fruit recognition and depth measurement functions. The picking mechanical clamp has a double-layer structure, with a scissors-like structure on the upper layer and a clamping and fixing structure on the lower layer, which can simultaneously separate the fruit stem and clamp. The fill light is used for image color light source supplement in fruit maturity determination, and the type and color of the fill light can be changed according to different target fruits.

[0076] In step B, in combination with the GD-YOLOv8P fruit recognition model training process shown in Figure 2 and Figure 3 , the GD-YOLOv8P fruit recognition model training process is as follows:

[0077] Image acquisition and processing: Obtain inter-row images of fruits from the industrial park, establish a dataset, store them in different folders according to the picking environment (strong light, normal, weak light), label the images under each file, the labels include Stem (stem), High_Harvest (high maturity), Half_Harvest (medium maturity), Low_Harvest (low maturity), and mark the category and position of each object; Organize the image and label data into a format suitable for the GD-YOLOv8P model, create training set, validation set and test set for model training and evaluation;

[0078] The GD-YOLOv8P recognition model includes an input layer (Input), a backbone network (Backbone), a Neck architecture and a detection head (Head);

[0079] The backbone network (Backbone) adopts yolov8n architecture, including convolution module ConvModule, residual block Residual block and SPPF (Spatial Pyramid Pooling-Fast), wherein the convolution module ConvModule uses CBS structure, the CBS structure is composed of two-dimensional convolution Conv2d, BatchNorm2d and SiLU activation function, the residual block Residual block uses C2f structure, the C2f structure is composed of convolution module Conv, Split, n BottleNeck series, connection layer Concat and upper level features, and the SPPF (Spatial Pyramid Pooling-Fast) is composed of CBS convolution layer, three Maxpooling series and connection layer Concat and upper level features;

[0080] The GD-YOLOv8P recognition model introduces a GD (gather-and-distribute) mechanism, which mainly includes three modules: FAM (Feature Alignment Module), IFM (Information Fusion Module) and Inject (Information Injection Module);

[0081] The MPDIoU (Minimum Points Distance IoU) is used as a loss function for boundary box regression in the detection head (Head) instead of the default CIoU loss function, simplifying the model calculation process, more accurately reflecting the relative relationship and size relationship between targets, and improving the accuracy and efficiency of boundary box regression. The MPDIoU loss function formula is as follows:

[0082]

[0083]

[0084]

[0085] wherein, are the coordinates of the top-left and bottom-right corners of frame A, respectively, are the coordinates of the top-left and bottom-right corners of frame B, respectively, and w and h are the input image width and height, respectively.

[0086] The detection head output result includes Stem, High_harvest, Half_harvest, and Low_harvest.

[0087] Model training: based on the preprocessed data set, the GD-YOLOv8P model is trained, the files of different picking environments are trained respectively, and the trained model is quantized and deployed; configure the training parameters, including learning rate, batch size, training duration, data augmentation strategy, etc. Using the prepared data and the configured model, training is performed on a computer supporting GPU. During training, the model will continuously update the weights to adapt to the target detection task.

[0088] Model verification: use the validation set to evaluate the performance of the model, including precision, recall, F1 score, etc. Adjust the model parameters and training strategy to improve performance, so as to obtain a usable weight model for the next step of target detection operation.

[0089] In step B, when the system is working, the fruit maturity and pickability determination method principle is as shown in Figure 4 The detection frame information is extracted, it is analyzed whether the fruit stem and the adjacent fruits on both sides of the fruit stem belong to the same fruit string, the fruits not belonging to the same fruit string are excluded, the fruit string set is established, and the maturity proportion of the fruits in the set is analyzed to determine whether it meets the picking conditions, laying a foundation for picking operation.

[0090] After the image is identified by the model, different types of detection frames will be generated. The position information of the detection frame, i.e. the coordinates of the top-left corner and the bottom-right corner, can be extracted and saved as a data type file, such as Figure 4 (2) in the fruit stem detection frame (uga , v ga ), (u gb , v gb ), (u1a, v1a), (u1b, v1b) of the fruit detection frame, pre-process these data by type, including calculating the side length, fruit detection frame area (S1, S2, etc.), center coordinates and longitudinal center line L1; during the left and right translation of the longitudinal center line L1, the points on the line will fall into some specific detection frames, thereby preliminarily screening the fruit detection frame, and screening out the fruit set that may be connected to a fruit stem, such as the 7 fruit frames shown in Figure 4 (3); again, because the same fruit string is in the associated three-dimensional space, the size of the fruit should be approximately the same when observing the fruit string from the camera point, and the mean and deviation calculation can eliminate data that do not belong to a certain fruit set, such as fruit 7 shown in Figure 4 (3) which needs to be eliminated, thereby determining the fruit stem and its real connected fruit set, such as the set shown in Figure 4 (4) which is {fruit stem, fruit 1, fruit 2, …, fruit 6}, in actual application, there may also be the problem of occlusion of the same fruit string, so the proportion of maturity determination is set to reduce the influence of such problems;

[0091] When analyzing the detection frame of the set, the set containing low maturity is considered unnecessary for picking, and if the matching fruit stem fruit set contains low maturity fruit, the area where the set is located will not be included in the detection range in this round of picking task; only when the proportion of high maturity fruit in the matching fruit stem fruit set exceeds 75% will the fruit string be considered mature; if the fruit is a small target type and the number of fruit in a string is generally greater than 10, the proportion needs to be adjusted according to the actual situation; fruit stems that do not meet the fruit number requirement or fruit that do not match the stem are considered to be occluded, because there are two cases of fruit occlusion and leaf occlusion, so the fruit string that meets the conditions needs to be picked first to exclude part of the fruit occlusion;

[0092] Use the GD-YOLOv8P recognition model to detect fruit and fruit stems, and then calculate the probability of the detection result to obtain a single fruit string set, on this basis, common occlusion possibilities are distinguished, and whether the fruit string meets the picking conditions is inferred according to the proportion of mature fruit individuals in the fruit string, and the set that has matured is picked first, and the fruit string maturity and pickable determination method is as follows:

[0093] Step 1: Extract the information of all detection frames, including category, position;

[0094] Step 2: Calculate the side length, area, and center of the detection frame for High_harvest, Half_harvest, and Low_harvest detection results, respectively;

[0095] Step 3: Calculate the longitudinal center line L of the Stem detection result:

[0096] (u ga , v ga ) is the top-left vertex coordinate of the Stem detection box, (u gb , v gb ) is the bottom-right vertex coordinate of the Stem detection box, the starting point coordinate of the center line L is (U, V), and the ending point coordinate is wherein, V = vga, is the general length of the peduncle;

[0097] Step 4: (Take peduncle 1 as an example) Shift the longitudinal center line L1 left and right by a certain distance to get L1' and L1'', respectively. By judging whether the longitudinal center line L1 falls within the fruit maturity detection box, determine the fruit detection box that the longitudinal center line L1 contacts during the shifting process, and record it as set A1.

[0098] Step 5: Remove the fruit that is not connected to the peduncle from set A1: remove the maximum and minimum values of the detection box area, and calculate the average value μ s of the remaining detection box area, calculate the deviation σ sn of all detection box areas (including the detection box with the maximum and minimum area), and calculate the ratio Pr n of the average value and the deviation:

[0099]

[0100] Pr n > 0.1 is considered not to belong to the fruit connected to peduncle 1, and is removed;

[0101] Step 6: Analyze the detection box set (take string-shaped fruit as an example):

[0102] a. The detection box set containing Low_harvest type is recorded as NP (not picked);

[0103] b. The detection box set does not contain Low_harvest type, but the High_harvest type accounts for no more than 75% of the total number of detection boxes in the set, recorded as NP (not picked);

[0104] c. The detection box set does not contain Low_harvest type, and the High_harvest type accounts for more than 75% of the total number of detection boxes in the set, recorded as CP (pickable);

[0105] d. The detection box set that is not crossed by the longitudinal center line L (established by the intersection of detection boxes), i.e., the detection box set of this type cannot find a matching peduncle detection box, recorded as NS (no peduncle found);

[0106] e. The number of left and right detection boxes of the fruit stem is greater than 60%, and the length of the fruit stem detection box in the v direction is less than That is, there is a problem of partial fruit being blocked, denoted as NH (not found partial fruit).

[0107] This method does not need to be separately labeled for fruit strings, saving time and cost in the early stage, and can quickly obtain the fruit string aggregation result when used, and quickly determine the maturity of the fruit string. This method can also be applied to spindle-shaped fruit picking, which is relatively simple compared to string-shaped fruit. Replace the expression of the longitudinal center line L of the fruit stem with the general length of the spindle-shaped fruit, and the intersection of the fruit stem detection box and the fruit detection box is used to determine the fruit stem and its connected fruit, and the maturity detection is determined according to the recognition model result.

[0108] In step C, when picking fruits, the binocular camera and the vision camera Sc work together as follows:

[0109] Camera parameter acquisition and calibration. In actual use, when the vision camera assists the binocular camera in distance measurement, the two sets of cameras should be kept relatively static at a fixed angle and position, and in the initial work, [S C ,D L ], [S C ,D R ], and [D L ,D R ] are respectively calibrated and calibrated to determine the camera internal and external parameters.

[0110] The vision camera correction is shown in Figure 5 , where f SC , f DL , and f DR are the focal lengths of the vision camera, the left eye of the binocular camera, and the right eye of the binocular camera, respectively, θ is the rotation angle of the vision camera correction, and S C ' is the corrected position.

[0111] After obtaining the camera parameters, the coordinate system is rotated and translated through an algorithm to achieve good camera correction effect. The vision camera correction formula is shown in the formula:

[0112]

[0113] where R SC is the rotation matrix, T 3×1 is the translation matrix, and θ is the rotation angle.

[0114] Stereo matching, due to the distance between two cameras, will result in slightly different positions for the same object in the images captured by the two cameras. By comparing the disparity between corresponding pixels in the two images, the depth information of the object can be calculated, generating a disparity map. Each pixel in this map corresponds to the disparity value at a corresponding position in the image. Using the disparity map and camera parameters, a depth map can be calculated, where each pixel corresponds to the depth distance at a corresponding position in the scene. In actual operation, three sets of stereo matching are performed to obtain three distance data points. The distance calibration parameters are determined based on the average distance and the deviation results, using the following formula:

[0115]

[0116]

[0117] After determining the distance calibration parameters, the coordinate system transformation is achieved through an algorithm, as shown in the formula:

[0118]

[0119] Among them, f DR D R Camera focal length.

[0120] Table 1. Results of the comparative experiment between multi-point localization and binocular localization:

[0121]

[0122] The visual camera Sc used in Table 1, and its rotation matrix R SC :

[0123]

[0124] Translation matrix T 3×1 :

[0125] T 3×1 = [511.9058 -15.2909 -198.7242] T

[0126] As shown in Table 1, overall, the visual camera-assisted binocular camera positioning effectively improves the accuracy to a certain extent.

[0127] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments for application in other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A multi-view-dual-arm collaborative harvesting method, utilizing a multi-view-dual-arm collaborative harvesting system for harvesting, the multi-view-dual-arm collaborative harvesting system comprising a mobile chassis, a fruit collection box, a left robotic arm assembly, a right robotic arm assembly, a controller, and a control panel; the left robotic arm assembly has a vision camera and a leaf-supporting structure connected to its end arm, and the right robotic arm assembly has a binocular camera, a harvesting mechanical clamp, and a supplementary light connected to its end arm; the controller loads or calls a computer program corresponding to the multi-view-dual-arm collaborative harvesting method, characterized in that... The harvesting method includes the following steps: Step A: Image acquisition and preprocessing; Step B: Construct and train a fruit recognition model based on GD-YOLOv8P, and detect images to determine the fruit stalk and fruit maturity. The fruit recognition model is then used to analyze the right eye of the stereo camera (D). R The transmitted images are inspected; the inspection results are processed and analyzed to determine the fruit's maturity and harvestability. The principle for determining fruit maturity and harvestability is as follows: (1) Extract information from all bounding boxes, including category and location; (2) Calculate the side length, area, and center of the detection frame for those with high maturity, medium maturity, and low maturity, respectively; (3) Calculate the longitudinal centerline L of the detection result Stem: Let (u ga v ga ) represents the coordinates of the top-left vertex of the Stem detection box, (u gb v gb Let (U, V) be the coordinates of the bottom right vertex of the Stem detection box, and let (U, V) be the starting coordinates of the center line L, and (U, V+) be the ending coordinates. ),in, , , This refers to the typical length of the fruit stalk; (4) The longitudinal center line L is translated to the left and right by a certain distance to obtain L' and L', respectively. By judging whether the longitudinal center line L falls within the fruit maturity detection frame, the fruit detection frame that the longitudinal center line L contacts during the translation process is determined and denoted as set A; (5) Remove fruits in set A that are not connected by the fruit stalk: Remove the maximum and minimum area values ​​in set A, and calculate the average area of ​​the remaining detection boxes. Calculate the deviation of the area of ​​each detection box. Calculate the ratio of the mean to the deviation. : ； The fruit frame is considered not to belong to the fruit connected by the fruit stalk and is removed; (6) Analyze the set of detection boxes: The set of detection boxes containing low-maturity types is denoted as NP (Not to be picked). Types that do not contain low maturity but whose high maturity accounts for no more than 75% of the total number of types in their respective detection frames are denoted as NPs that are not picked. A type that does not contain low-maturity types and whose high-maturity types account for more than 75% of the total number of the detection frames is denoted as a harvestable CP. The set of detection boxes that are not crossed by the longitudinal center line L, i.e., the detection boxes for which no matching fruit stalks were found, is denoted as fruit stalk not found NS; The difference in the number of detection frames on the left and right sides of the fruit stalk is greater than 60%, and the length of the fruit stalk detection frame in the v-direction is less than 0.

6. This means that some fruits are partially obscured, and is recorded as NH where some fruits were not found. Step C: Harvesting fruits based on the type of the partitioned fruit cluster set: Step C1: Harvest the CP type: Through multi-point positioning, calculate the distance of the corresponding fruit stem picking point, and send the three-dimensional coordinates of the fruit stem picking point to the right robotic arm component. At the same time, combined with the binocular camera positioning, call the right robotic arm to carry out the picking work and place the picked fruit in the fruit collection box. Step C2: Harvest NS type fruits for which no fruit stalks were found and NH type fruits for which no fruits were found: Control the left robotic arm component to move to the vicinity of the target, activate the vision camera to detect obstructions, and the left robotic arm component moves according to the detection results. It lifts the leaves through the leaf-supporting structure and identifies the fruits. If the identification result is that the fruit is of the CP type that can be harvested, then it is harvested. Step C3: When only NP type fruits remain to be harvested in the current area: Control the left robotic arm component to perform fixed cruise, move upward from the ground, lift the leaves to detect missed fruits. If missed fruits are detected, the harvesting cycle begins, with the left robotic arm component parting the leaves and assisting the right robotic arm component in positioning and harvesting. If no missed fruits are detected, the moving chassis vehicle automatically travels to the next area to work.

2. The multi-eye-dual-arm cooperative harvesting method according to claim 1, characterized in that: In step B, the training process for the fruit recognition model is as follows: (1) Image acquisition and processing: Construct a dataset and store it in different folders according to the picking environment of strong light, normal and weak light. Label the images in each folder, including fruit stalk, high maturity, medium maturity and low maturity, and indicate their category and location. Create training set, validation set and test set. (2) Model training: The files of different picking environments are trained separately, and the trained models are deployed in a quantitative manner. The training parameters include learning rate, batch size, training time and data augmentation strategy. During training, the model continuously updates the weights to adapt to the target detection task. (3) Model validation: Use the validation set to evaluate the performance of the model, including precision, recall, and F1 score, and adjust model parameters and training strategies to improve performance.

3. The multi-eye-dual-arm cooperative harvesting method according to claim 1, characterized in that: In step B, the fruit recognition model constructed based on GD-YOLOv8P includes an input layer, a backbone network, a Neck, and a detection head. The backbone network adopts the yolov8n architecture, which includes convolutional modules, residual blocks, and SPPF. The convolutional modules use the CBS structure, which consists of two-dimensional convolution Conv2d, BatchNorm2d, and SiLU activation function. The residual blocks use the C2f structure, which consists of convolutional modules Conv, Split, n BottleNecks in sequence, and a concat connection layer to fuse upper-level features. The SPPF consists of CBS convolutional layers, three Maxpooling layers in sequence, and a concat connection layer to fuse upper-level features. Furthermore, the GD mechanism is introduced into the fruit recognition model. The GD mechanism includes three modules: feature alignment module, information fusion module, and information injection module. The detection head uses MPDIoU as the loss function for bounding box regression instead of the default CIoU loss function; the output of the detection head includes fruit stalk, high maturity, medium maturity and low maturity.

4. The multi-eye-dual-arm cooperative harvesting method according to claim 1, characterized in that: In step C, multi-point binocular positioning is used during harvesting, and its principle is as follows: (1) Acquire images of the target area using a visual camera and a binocular camera, and then process the images using the right eye D of the binocular camera. R Image recognition and maturity assessment; (2) Visual camera S C Correction, the correction matrix is ​​as follows: ； in, Let be a rotation matrix. It is a translation matrix. The rotation angle; (3) Regarding [S] C D L ]、[S C D R ]、[D L D R Perform stereo matching separately to obtain depth maps; (4) Select a harvestable CP type fruit as the harvesting target. The depths of three fruit stalk harvesting points P, dist1, dist2, and dist3, are obtained from the T3 depth map. The average value of the three depths is as follows: ， Calculate the absolute values ​​of the deviations of the three depths from the mean and compare their magnitudes: ， The depth measured by the camera combination corresponding to the minimum absolute value of the deviation. As a depth calibration parameter; (5) Converting multi-point positioning to stereo positioning: Convert D R The coordinate system is transformed to the coordinate system of the fruit stalk picking point P. The transformation formula is as follows: ； in, D R The camera's focal length.

5. The multi-eye-dual-arm cooperative harvesting method according to claim 1, characterized in that: In step A, the staff first selects the corresponding picking mode on the control panel according to the picking environment. At this time, the left and right robotic arm components remain in standby mode, and the mobile chassis automatically drives to the work area. The left and right robotic arm components are then awakened, the supplementary lights are turned on, and the vision camera is activated. The binocular camera is used to take pictures of the current area, and the collected images are classified, labeled, and preprocessed.

Images