Fruit and vegetable string picking method and device

Abstract

The invention is suitable for the field of agricultural automation, and provides a fruit and vegetable string picking method and device, and the method comprises the steps: obtaining fruit and vegetable image data; inputting the fruit and vegetable image data into a pre-trained target detection model to obtain fruit stem information and fruit information; the incidence relation between the fruit stems and the fruits is obtained; according to the incidence relation between the fruit stem and the fruit, the fruit category and a preset picking rule, determining a pickable string, and obtaining pickable string information; obtaining a three-dimensional picking coordinate under a mechanical arm base coordinate system according to the fruit stem center coordinate of the pickable string and a preset coordinate conversion matrix; and the mechanical arm is controlled to move to the corresponding picking position, and the end effector is controlled to execute string picking operation. According to the method, efficient, low-loss and selective bunching harvesting of bunching picking can be achieved, meanwhile, mistaken picking is reduced, and mature fruit and vegetable bunches are accurately harvested while the bunching harvesting efficiency advantage is reserved by combining fruit and vegetable maturity judgment with automatic picking.

Description

Technical Field

[0001] This application belongs to the field of agricultural automation technology, and in particular relates to a method and equipment for harvesting bunches of fruits and vegetables. Background Technology

[0002] In recent years, with the development of machine vision and artificial intelligence algorithms, harvesting robots have been increasingly widely used in agricultural production. Cherry tomatoes, as a high-value-added fruit and vegetable crop, are widely welcomed by the market due to their unique flavor and rich nutrition. However, cherry tomatoes ripen at different times and have dense clusters, and traditional harvesting methods mainly rely on manual bunch harvesting or single-fruit harvesting. Bundle harvesting is more efficient, but immature fruits are easily picked along with the fruit, affecting the quality of marketable fruit; while single-fruit harvesting allows for selective harvesting, it is inefficient, labor-intensive, and difficult to meet the needs of large-scale planting.

[0003] Currently, there is a lack of existing technologies that can balance harvesting efficiency with selective fruit ripeness. While some mechanized harvesting equipment can improve efficiency, its sorting accuracy is insufficient, easily causing fruit damage, especially for cherry tomatoes with significant differences in ripeness. In addition, manual harvesting relies on experience to judge ripeness, resulting in inconsistent standards, and labor costs are rising year by year. Summary of the Invention

[0004] This application provides a method and equipment for harvesting fruits and vegetables in bunches, which can solve the above-mentioned problems.

[0005] In a first aspect, embodiments of this application provide a method for harvesting bunches of fruits and vegetables, including: Acquire fruit and vegetable image data; wherein, the fruit and vegetable image data is image data corresponding to bunches of fruits and vegetables; The fruit and vegetable image data is input into a pre-trained target detection model to obtain fruit stem information and fruit information; wherein, the fruit stem information includes at least the fruit stem bounding box coordinates, and the fruit information includes at least the fruit bounding box coordinates and the fruit category; The association between the fruit stalk and the fruit is obtained based on the fruit stalk bounding box coordinates, the fruit bounding box coordinates, and a preset same-string matching algorithm. Based on the relationship between the fruit stalk and the fruit, the fruit category, and the preset harvesting rules, a harvestable bunch is determined to obtain harvestable bunch information; wherein, the harvestable bunch information includes at least the center coordinates of the fruit stalk of the harvestable bunch; Based on the center coordinates of the fruit stalk of the harvestable string and the preset coordinate transformation matrix, the three-dimensional harvesting coordinates in the robotic arm base coordinate system are obtained. Based on the three-dimensional picking coordinates, the robotic arm is controlled to move to the corresponding picking position, and the end effector is controlled to perform a series of picking operations.

[0006] Secondly, embodiments of this application provide a fruit and vegetable bunch-harvesting device, comprising: The first acquisition unit is used to acquire fruit and vegetable image data; wherein, the fruit and vegetable image data is image data corresponding to a bunch of fruits and vegetables; The first processing unit is used to input the fruit and vegetable image data into a pre-trained target detection model to obtain fruit stem information and fruit information; wherein, the fruit stem information includes at least the fruit stem bounding box coordinates, and the fruit information includes at least the fruit bounding box coordinates and the fruit category; The second processing unit is used to obtain the association relationship between the fruit stalk and the fruit based on the coordinates of the fruit stalk bounding box, the coordinates of the fruit bounding box, and a preset same-string matching algorithm. The third processing unit is used to determine the harvestable bunches based on the relationship between the fruit stalk and the fruit, the fruit category and the preset picking rules, and obtain harvestable bunch information; wherein, the harvestable bunch information includes at least the center coordinates of the fruit stalk of the harvestable bunch; The fourth processing unit is used to obtain the three-dimensional picking coordinates in the robotic arm base coordinate system based on the center coordinates of the fruit stalk of the pickable string and a preset coordinate transformation matrix. The fifth processing unit is used to control the robotic arm to move to the corresponding picking position according to the three-dimensional picking coordinates, and to control the end effector to perform a series of picking operations.

[0007] Thirdly, embodiments of this application provide a fruit and vegetable bunch-picking device, including: a control host, a robot, an end effector, and a depth camera; The control host is connected to the robot, the end effector, and the depth camera respectively; The robot includes a robotic arm and a robot body; The end effector includes a clamping assembly and a scissor assembly; The clamping assembly, the scissor assembly, and the depth camera are mounted on the end of the robotic arm; The control host and the robotic arm are mounted on the robot body; The control host is used to perform the steps in the bunch-harvesting method for fruits and vegetables as described in the first aspect above.

[0008] Fourthly, embodiments of this application provide a control host, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0009] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0010] In this embodiment, fruit and vegetable image data is acquired; the image data is input into a pre-trained target detection model to obtain stem and fruit information; the association between stem and fruit is obtained based on the stem bounding box coordinates, fruit bounding box coordinates, and a preset matching algorithm; harvestable bunches are determined based on the association between stem and fruit, fruit category, and preset harvesting rules, obtaining harvestable bunch information; three-dimensional harvesting coordinates in the robotic arm's base coordinate system are obtained based on the stem center coordinates of the harvestable bunch and a preset coordinate transformation matrix; the robotic arm is controlled to move to the corresponding harvesting position based on the three-dimensional harvesting coordinates, and the end effector is controlled to perform bunch harvesting operations. This method achieves efficient, low-loss, and selective bunch harvesting of fruits and vegetables, while reducing the misharvesting of immature fruits. By combining fruit and vegetable maturity assessment with automated harvesting, it accurately harvests mature fruit and vegetable bunches while retaining the efficiency advantage of bunch harvesting, solving the problem of balancing harvesting quality and efficiency in existing technologies. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic flowchart of a method for harvesting bunches of fruits and vegetables provided in the first embodiment of this application; Figure 2 This is a schematic flowchart of steps S1021 to S1023 in a method for harvesting bunches of fruits and vegetables provided in the first embodiment of this application; Figure 3 This is a schematic flowchart of steps S107 to S110 in a method for harvesting bunches of fruits and vegetables provided in the first embodiment of this application; Figure 4 This is a schematic flowchart of steps S1031 to S1033 in a method for harvesting bunches of fruits and vegetables provided in the first embodiment of this application; Figure 5 This is a schematic flowchart of steps S1041 to S1042 in a method for harvesting bunches of fruits and vegetables provided in the first embodiment of this application; Figure 6This is a schematic flowchart of steps S1061 to S1062 in a method for harvesting bunches of fruits and vegetables provided in the first embodiment of this application; Figure 7 This is a schematic diagram of the fruit and vegetable bunch harvesting device provided in the second embodiment of this application; Figure 8 This is a schematic diagram of the fruit and vegetable bunch harvesting device provided in the third embodiment of this application; Figure 9 This is a schematic diagram of the control host provided in the fourth embodiment of this application. Detailed Implementation

[0013] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0014] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0015] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0016] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0017] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0018] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0019] Please see Figure 1 , Figure 1 This is a schematic flowchart illustrating a method for harvesting bunches of fruits and vegetables according to the first embodiment of this application. In this embodiment, the executing entity of the method for harvesting bunches of fruits and vegetables is a control host. Figure 1 The methods for harvesting fruits and vegetables in bunches as shown may include: S101: Acquire fruit and vegetable image data; wherein, the fruit and vegetable image data is image data corresponding to a bunch of fruits and vegetables.

[0020] In this embodiment, a high-resolution camera can be used to photograph the target fruits and vegetables to ensure that sufficient image data is obtained under different lighting conditions.

[0021] A combination of timed shooting and real-time monitoring can be used to capture the state of the fruit at different times.

[0022] For example, a total of 321 JPG images with a resolution of 2736×3648 can be collected in a greenhouse under sunny and cloudy weather conditions.

[0023] S102: Input the fruit and vegetable image data into a pre-trained target detection model to obtain fruit stem information and fruit information; wherein, the fruit stem information includes at least the fruit stem bounding box coordinates, and the fruit information includes at least the fruit bounding box coordinates and the fruit category.

[0024] Pre-trained object detection models are stored in advance, and these models can identify information about fruit stems and fruits.

[0025] In one embodiment, the target detection model is an improved version of the YOLOv8s model. The target detection model includes a backbone network, a neck network, and a detection head. The backbone network introduces a CoordAtt attention mechanism and an SPPFCSPC feature extraction layer. The CoordAtt attention mechanism is used to capture the spatial information of the fruit and the fruit stalk, and the SPPFCSPC feature extraction layer is used to enhance the linear features of the fruit stalk.

[0026] The fruit stalk information includes at least the coordinates of the fruit stalk bounding box, and the fruit information includes at least the coordinates of the fruit bounding box and the fruit category.

[0027] The bounding box coordinates in the fruit stalk information refer to the position and size of the detected fruit stalk in the image, usually represented by a rectangle. This rectangle is defined by four coordinate points: Top-left corner coordinates (x_min, y_min) The coordinates of the bottom right corner (x_max, y_max) These coordinates determine the exact location of the fruit stalk in the image, which helps subsequent processing steps (such as associating fruit and stalk) to accurately identify and locate the stalk.

[0028] The bounding box coordinates in the fruit information are also rectangular boxes, used to represent the position and size of the fruit in the image. It is also defined by four coordinate points (x_min, y_min, x_max, y_max).

[0029] By using the bounding box coordinates of the fruit, the specific location of each fruit in the image can be determined, which facilitates subsequent processing and analysis.

[0030] Fruit category refers to the type or variety of each detected fruit. Fruit categories can be classified according to different degrees of ripeness, quality, or other characteristics.

[0031] For example, fruits might be categorized as "ripe," "immature," and "diseased." This helps enforce harvesting rules, ensuring that only fruits that meet specific criteria are picked.

[0032] In one embodiment, the target detection model includes a backbone network, a neck network, and a detection head. S102 may include S1021~S1023, such as... Figure 2 As shown, S1021~S1023 are as follows: S1021: Input the fruit and vegetable image into the backbone network to obtain a first feature map; wherein, the first feature map is a feature map extracted by the backbone network at different levels.

[0033] First, the original fruit and vegetable images are input into the backbone network of the model. The goal of the backbone network is to extract useful features from the images, which can include color, shape, texture, etc.

[0034] The backbone network is responsible for extracting multi-level features from the input image, providing rich feature representations for subsequent detection tasks. Its input is a fixed-size RGB image, and its output is three feature maps at different resolutions. The backbone network processes the image through multiple convolutional layers and activation functions, generating an output called the first feature map. This feature map contains information at different levels (such as low-level edge features and high-level semantic features), reflecting the different features of the fruit and its stem in the image.

[0035] S1022: Input the first feature map into the neck network to obtain a second feature map; wherein, the second feature map is a multi-scale feature map of the neck network after fusing the first feature map.

[0036] The first feature map is input into the neck network. The main function of the neck network is to further process and fuse the features from the backbone network.

[0037] The neck network is responsible for multi-scale feature fusion to improve the model's ability to detect targets of different sizes. The neck network combines feature maps from different levels through techniques such as feature fusion and feature pyramids to generate a second feature map. This fusion process preserves multi-scale features, enhancing the model's ability to detect targets of different sizes, especially when dealing with objects of varying shapes such as fruits and stems, where it can better capture their features.

[0038] S1023: Input the second feature map into the detection head to obtain the fruit stalk information and the fruit information.

[0039] The second feature map is input into the detection head. The detection head is responsible for converting the previously extracted and fused features into specific detection results.

[0040] The detection head is responsible for generating the final detection results (bounding boxes, categories, and confidence scores). In this stage, the detection head outputs information about the fruit stem (such as location and category) and the fruit (also including location and category) through classification and regression operations. This step is the final result of object detection; the model outputs the bounding boxes of the detected fruit and stem, along with the corresponding confidence scores and category information.

[0041] In this embodiment, the entire target detection model can extract important features from the original fruit and vegetable images and gradually transform these features into usable detection information. The backbone network is responsible for feature extraction, the neck network is responsible for feature fusion, and the detection head is responsible for final classification and localization. It can effectively identify and locate fruits and stems, providing reliable data support for subsequent automated processing.

[0042] In one embodiment, before S102, S107-S110 may also be included, such as... Figure 3 As shown, S107~S110 are as follows: S107: Collect training image data of fruits and vegetables; wherein the training image data of fruits and vegetables is collected under different weather conditions.

[0043] Shoot under different weather conditions (such as sunny, cloudy, rainy, foggy, etc.) to ensure that the model can learn the appearance changes of fruits and vegetables in various environments.

[0044] Consider acquiring images at different times of day (such as morning, noon, and evening) to capture the appearance of fruits and vegetables under different lighting conditions.

[0045] The same bunch of fruits and vegetables was photographed from different angles to enhance the model's ability to recognize the target.

[0046] High-resolution cameras, drones, or mobile phones can be used to capture images, ensuring image quality.

[0047] S108: Preprocess the fruit and vegetable training image data to obtain data-enhanced fruit and vegetable training images.

[0048] Preprocessing the fruit and vegetable training image data involves using data augmentation algorithms such as transpose, scaling, random brightness, random contrast, random saturation, Gaussian noise, and salt-and-pepper noise to augment the data, resulting in data-enhanced fruit and vegetable training images.

[0049] S109: Divide the data-enhanced fruit and vegetable training images to obtain a training dataset, a validation dataset, and a test dataset.

[0050] The augmented fruit and vegetable training images are divided into three groups, typically 70% for training, 20% for validation, and 10% for testing.

[0051] Images are randomly selected from the enhanced image dataset to avoid data bias.

[0052] When splitting the dataset, try to keep the number of samples in each category balanced to prevent the model from being biased towards certain categories.

[0053] S110: Train the initialized target detection model according to the training dataset, the validation dataset, the test dataset, and the preset model training algorithm to obtain the pre-trained target detection model.

[0054] Choose an appropriate object detection model based on the task requirements, such as YOLO, Faster R-CNN, SSD, etc.

[0055] Configure hyperparameters such as learning rate, batch size, and number of iterations, and adjust these parameters based on feedback from the validation set.

[0056] You can choose to perform transfer learning on models pre-trained on other large-scale datasets (such as COCO and Pascal VOC) to shorten training time and improve performance.

[0057] Use a loss function to monitor the training process and adjust the learning rate or other parameters in a timely manner to prevent training from failing to converge or overfitting.

[0058] In this embodiment, through an effective training process, a pre-trained target detection model with high accuracy and good recall is obtained, which can accurately identify fruits and stems and improve the efficiency of automated harvesting.

[0059] S103: Based on the coordinates of the fruit stalk bounding box, the coordinates of the fruit bounding box, and a preset same-string matching algorithm, obtain the association relationship between the fruit stalk and the fruit.

[0060] Based on the fruit stem bounding box and the fruit bounding box, association is performed using a preset same-string matching algorithm (such as IoU (Intersection over Union) matching or distance matching).

[0061] The algorithm is based on the spatial distribution pattern of fruits and pedicels for pairing.

[0062] This algorithm can determine which fruits belong to the same cluster based on the spatial relationship between the fruit stalk and the fruit.

[0063] In one embodiment, S103 may include S1031~S1033, such as Figure 4 As shown, S1031~S1033 are as follows: S1031: Establish a dynamic search region based on the coordinates of the fruit stalk bounding box; wherein the dynamic search region is an elliptical search region generated according to the direction of the main axis of the fruit stalk.

[0064] The principal axis direction is calculated based on the bounding box coordinates of the fruit stem. The principal axis direction can be determined by calculating the aspect ratio of the bounding box; typically, the longer side of the bounding box is chosen as the principal axis.

[0065] Centered on the center point of the fruit stalk's bounding box, an elliptical search region is generated according to the principal axis direction. The major axis of the ellipse is aligned with the principal axis direction of the fruit stalk, while the minor axis is perpendicular to it.

[0066] The size of the ellipse can be adjusted according to the size of the fruit and harvesting requirements. For example, the major axis can be set to twice the diameter of the fruit, and the minor axis can be set to once the diameter of the fruit.

[0067] The size and shape of the ellipse can also be dynamically adjusted based on ambient lighting, climate conditions, etc., to improve the effectiveness of the search.

[0068] Multiple elliptical regions of different sizes can be created within the elliptical search area to enable multi-level searching of fruits of different sizes or types.

[0069] S1032: Determine the fruits after primary screening based on the fruit bounding box coordinates and the dynamic search area.

[0070] In this embodiment, primary screening can be performed using distance thresholds and angle constraints.

[0071] The intersection of the fruit's bounding box coordinates with the dynamic search area can be used for judgment. If the fruit's bounding box intersects with the dynamic search area, the fruit is considered as a preliminary target.

[0072] Geometric algorithms (such as calculating the intersection area of ​​two rectangles) can be used to quickly determine this.

[0073] Initial screening can also be conducted based on the fruit's color, size, and ripeness, with decisions made in conjunction with visual characteristics.

[0074] Machine learning algorithms can also be used to analyze historical data and build fruit selection models to further improve the accuracy of selection.

[0075] S1033: Based on the fruit bounding box coordinates corresponding to the fruits after the initial screening and the preset fruit clustering algorithm, the association relationship between the fruit stalk and the fruit is obtained.

[0076] Choose a suitable clustering algorithm (such as K-means, DBSCAN, hierarchical clustering, etc.) to perform cluster analysis on the fruits after the initial screening.

[0077] Using the bounding box coordinates of the fruit as input, the fruit is divided into different groups according to a clustering algorithm. Fruits in each group are associated with the same fruit stalk.

[0078] The group information obtained after clustering is used to establish the association between the fruit stalk and related fruits using the Hungarian algorithm. It can usually be represented as: Fruit stalk ID → {Fruit ID 1, Fruit ID 2, ...}.

[0079] By comparing the effects of different clustering algorithms, the optimal clustering method can be selected to ensure that the association between the fruit stalk and the fruit is accurate and reliable.

[0080] If the harvesting process is dynamic, the time factor can be considered, a time series data model can be created, and the cluster dynamics of fruit changes over time can be analyzed.

[0081] In this embodiment, intelligent screening and clustering methods can significantly improve the efficiency of fruit and vegetable harvesting and reduce labor costs. Precise harvesting operations can reduce damage to the fruit and improve the overall quality of fruits and vegetables. This method can be adjusted according to the characteristics of different fruits and vegetables and their growing environment, and has strong adaptability.

[0082] S104: Based on the relationship between the fruit stalk and the fruit, the fruit category, and the preset picking rules, determine the harvestable bunches and obtain harvestable bunch information; wherein, the harvestable bunch information includes at least the center coordinates of the fruit stalk of the harvestable bunch.

[0083] Based on the correlation and fruit category, select harvestable fruit clusters using preset harvesting rules (such as maturity, fruit size, etc.).

[0084] Record the information of strings that meet the criteria (including the coordinates of the fruit stalk center).

[0085] In one embodiment, S104 may include S1041~S1042, such as Figure 5 As shown, S1041~S1042 are as follows: S1041: Determine the number of mature fruits in each bunch of fruits and vegetables based on the relationship between the fruit stalk and the fruit and the fruit category.

[0086] Fruit category refers to the type or variety of each detected fruit. Fruit categories can be classified according to different degrees of maturity, quality, or other characteristics. Fruits are divided into "ripe" and "immature" categories. The number of ripe fruits in each bunch is determined based on the relationship between the fruit stalk and the fruit and the fruit category.

[0087] S1042: If the number of mature fruits is not less than a preset threshold, determine that the bunch of fruits and vegetables is a harvestable bunch and obtain the harvestable bunch information.

[0088] Based on actual harvesting needs, a threshold for the number of mature fruits should be set (such as 5, 8, etc.). This threshold needs to be set reasonably based on market demand and production plans.

[0089] After collecting the number of ripe fruits in each bunch, a conditional judgment is made: if the number of ripe fruits is greater than or equal to a set threshold, the bunch is considered harvestable. Otherwise, it is marked as unharvestable.

[0090] In one embodiment, this embodiment may further include: if the number of mature fruits is less than a preset threshold, determining that the bunch of fruits and vegetables is an unharvestable bunch, and obtaining unharvestable bunch information.

[0091] In this embodiment, the identification mechanism for the number of mature fruits can intelligently determine which fruit and vegetable bunches can be harvested, thereby avoiding the harvesting of immature or unsuitable fruits and vegetables and improving the harvesting quality.

[0092] S105: Based on the center coordinates of the fruit stalk of the harvestable string and the preset coordinate transformation matrix, obtain the three-dimensional harvesting coordinates in the base coordinate system of the robotic arm.

[0093] The coordinates of the center of the fruit stem in the harvestable string are transformed to the robot's base coordinate system using a preset coordinate transformation matrix. The coordinate transformation matrix is ​​typically a 4x4 transformation matrix that contains rotation and displacement information. This matrix transforms a point in one coordinate system (e.g., the stem coordinate system) to another coordinate system (e.g., the robot's base coordinate system).

[0094] This matrix is ​​determined by the robot arm's pose data and coordinate system relationships, ensuring precise positioning.

[0095] S106: Based on the three-dimensional picking coordinates, control the robotic arm to move to the corresponding picking position, and control the end effector to perform a series of picking operations.

[0096] In automated harvesting systems, three-dimensional harvesting coordinates refer to the target position of the robotic arm's end effector (such as a gripper or suction cup) in the robotic arm's base coordinate system. These coordinates are calculated through previous coordinate transformation steps and represent the precise position of the fruit stem or fruit.

[0097] Based on the three-dimensional picking coordinates, the robotic arm is driven to move to the corresponding position using a control algorithm (such as PID control). Once the robotic arm has moved to the target position, the end effector (such as a gripper or suction cup) needs to be controlled to perform the picking operation.

[0098] The control system uses end effectors (such as grippers or suction cups) to perform the harvesting operation, ensuring minimal damage to the plant during the harvesting process. If grippers are used, the control system sends a signal to close the grippers, securing the fruit or stem. If suction cups are used, the control system activates a vacuum pump to generate suction, causing the suction cups to adhere firmly to the fruit surface.

[0099] When performing harvesting operations, ensure that damage to the plant is minimized. A gentle control strategy can be used with the clamps or suction cups to ensure that the force applied to the fruit or plant is appropriate, avoiding excessive pressure that could damage the fruit.

[0100] In one embodiment, the end effector integrates a clamping assembly and a scissor assembly; in another embodiment, S106 may include S1061~S1062, such as... Figure 6 As shown, S1061~S1062 are as follows: S1061: Based on the three-dimensional picking coordinates and the preset path planning algorithm, control the robotic arm to move to the corresponding picking position.

[0101] Use a preset path planning algorithm (such as RRT* algorithm, A* algorithm, Dijkstra algorithm, etc.) to plan a safe and efficient movement path for the robotic arm.

[0102] Based on the calculated three-dimensional harvesting coordinates and the planned path, the robotic arm is controlled to move towards the target location. The robotic arm should move smoothly to avoid damaging the surrounding plants.

[0103] S1062: Control the clamping assembly to pre-clamp the stem of the harvestable bunch, and control the scissor assembly to cut along the target plane; wherein, the target plane is a plane perpendicular to the axial direction of the stem of the harvestable bunch.

[0104] Once the robotic arm reaches the target position, the control gripping components pre-grip the stems of the harvestable bunches. This step ensures that the fruits and vegetables do not easily slip during the cutting process.

[0105] Control the shear assembly to cut along the target plane (the plane perpendicular to the axis of the fruit stalk of the harvestable bunch). This cutting action should be accurate and clean to minimize damage to the fruit.

[0106] In one embodiment, after S1062, the method further includes: controlling the robotic arm to move the harvested bunches of fruits and vegetables above the fruit basket, and controlling the clamping assembly to perform a releasing action.

[0107] After the cutting is completed, the robotic arm is controlled to move the harvested bunches of fruits and vegetables to the top of the fruit basket.

[0108] Above the fruit basket, control the clamping components to release the fruit and vegetable bunches gently into the basket.

[0109] In this embodiment, fruit and vegetable image data is acquired; the image data is input into a pre-trained target detection model to obtain stem and fruit information; the association between stem and fruit is obtained based on the stem bounding box coordinates, fruit bounding box coordinates, and a preset matching algorithm; harvestable bunches are determined based on the association between stem and fruit, fruit category, and preset harvesting rules, obtaining harvestable bunch information; three-dimensional harvesting coordinates in the robotic arm's base coordinate system are obtained based on the stem center coordinates of the harvestable bunch and a preset coordinate transformation matrix; the robotic arm is controlled to move to the corresponding harvesting position based on the three-dimensional harvesting coordinates, and the end effector is controlled to perform bunch harvesting operations. This method achieves efficient, low-loss, and selective bunch harvesting of fruits and vegetables, while reducing the misharvesting of immature fruits. By combining fruit and vegetable maturity assessment with automated harvesting, it accurately harvests mature fruit and vegetable bunches while retaining the efficiency advantage of bunch harvesting, solving the problem of balancing harvesting quality and efficiency in existing technologies.

[0110] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0111] Please see Figure 7 , Figure 7 This is a schematic diagram of a fruit and vegetable bunch-harvesting device provided in the second embodiment of this application. The fruit and vegetable bunch-harvesting device includes: a control host 71, a robot 72, an end effector, and a depth camera 74; The control host is connected to the robot, the end effector, and the depth camera respectively; The robot includes a robotic arm and a robot body; The end effector includes a clamping assembly 731 and a scissor assembly 732; The clamping assembly, the scissor assembly, and the depth camera are mounted on the end of the robotic arm; The control host and the robotic arm are mounted on the robot body; The control host is used to execute the steps in the fruit and vegetable bunch harvesting method described in the first embodiment above.

[0112] The control host, as the core control unit of the equipment, is responsible for coordinating the work of various components, processing data from the depth camera, performing path planning, and controlling the movement of the robotic arm.

[0113] It communicates in real time with the robot, end effector and depth camera via a data bus.

[0114] The robot's main body comprises a robotic arm and a support structure, providing a stable mobile platform. It carries the control unit and robotic arm, and possesses mobility (such as wheels or tracks) to move freely within the orchard.

[0115] The robotic arm consists of multiple joints and links, possessing multiple degrees of freedom to achieve flexible movement. It positions, moves, and operates according to instructions from the control unit, capable of extending, rotating, and adjusting angles to precisely reach the target fruits and vegetables.

[0116] The end effector includes: The clamping component is used to grip the fruit and vegetable bunches, ensuring the stability of the fruits and vegetables during the cutting process.

[0117] The scissor assembly is used to cut the stems of ripe fruits, thus completing the harvesting of fruits and vegetables.

[0118] Both the gripping and scissor assemblies are mounted at the end of the robotic arm, enabling integrated operation.

[0119] Depth cameras are used to capture real-time 3D image data of fruits and vegetables, identify ripe fruits, and calculate their spatial position. Mounted at the end of a robotic arm, they ensure accurate image information is acquired during harvesting.

[0120] The depth camera captures image data of the fruit and vegetable area in real time and transmits it to the control host.

[0121] The control host analyzes images using image processing algorithms to identify the type and maturity of fruits and vegetables, and calculates their three-dimensional spatial coordinates.

[0122] The control unit determines which fruits are ripe and prepares to be harvested based on the identified fruit and vegetable information.

[0123] If a bunch of fruits and vegetables is found to have insufficient number of ripe fruits, the control unit will mark it as an unharvestable bunch and plan a reasonable moving path.

[0124] The host computer instructs the robotic arm to move along a predetermined path to the target fruit and vegetable location.

[0125] Once it reaches the target location, the robotic arm performs a pre-gripping action to grasp the target fruit stem, ensuring that the fruit and vegetables will not fall off during cutting.

[0126] After the clamping component holds the fruit and vegetables, the control host commands the scissor component to cut the fruit stem.

[0127] After successful cutting, the robotic arm moves the harvested fruit and vegetable bunches above the fruit basket and releases the clamping components to gently place the fruits and vegetables into the fruit basket.

[0128] After harvesting a bunch of fruits and vegetables, the equipment returns to the control host to perform the next round of image data acquisition and path planning, and continues to perform the harvesting task.

[0129] In this embodiment, automated control reduces the time and labor intensity of manual harvesting. The combination of depth cameras and image processing technology enables the equipment to intelligently determine the ripeness of fruits and vegetables, improving harvesting quality. The high flexibility and precise control of the robotic arm enhances harvesting efficiency and reduces the risk of fruit damage. The equipment can adapt to the harvesting needs of various fruits and vegetables and to different orchard environments.

[0130] Please see Figure 8 , Figure 8 This is a schematic diagram of a fruit and vegetable bunch-harvesting device provided in the third embodiment of this application. The included units are used for performing... Figures 1-6 The steps in the corresponding embodiments. Please refer to the details. Figures 1-6 The relevant descriptions in the corresponding embodiments are shown below. For ease of explanation, only the parts relevant to this embodiment are shown. See also... Figure 8 The fruit and vegetable bunch-picking device 8 includes: The first acquisition unit 810 is used to acquire fruit and vegetable image data; wherein, the fruit and vegetable image data is image data corresponding to a bunch of fruits and vegetables; The first processing unit 820 is used to input the fruit and vegetable image data into a pre-trained target detection model to obtain fruit stem information and fruit information; wherein, the fruit stem information includes at least the fruit stem bounding box coordinates, and the fruit information includes at least the fruit bounding box coordinates and the fruit category; The second processing unit 830 is used to obtain the association relationship between the fruit stalk and the fruit based on the coordinates of the fruit stalk bounding box, the coordinates of the fruit bounding box, and a preset same-string matching algorithm. The third processing unit 840 is used to determine the harvestable bunches based on the relationship between the fruit stalk and the fruit, the fruit category and the preset picking rules, and obtain harvestable bunch information; wherein, the harvestable bunch information includes at least the center coordinates of the fruit stalk of the harvestable bunch; The fourth processing unit 850 is used to obtain the three-dimensional picking coordinates in the robotic arm base coordinate system based on the center coordinates of the fruit stalk of the pickable string and a preset coordinate transformation matrix. The fifth processing unit 860 is used to control the robotic arm to move to the corresponding picking position according to the three-dimensional picking coordinates, and to control the end effector to perform a series of picking operations.

[0131] Furthermore, the target detection model is an improvement based on the YOLOv8s model. The target detection model includes a backbone network, a neck network, and a detection head. The backbone network introduces a CoordAtt attention mechanism and an SPPFCSPC feature extraction layer. The CoordAtt attention mechanism is used to capture the spatial information of the fruit and the fruit stalk, and the SPPFCSPC feature extraction layer is used to enhance the linear features of the fruit stalk.

[0132] Furthermore, the target detection model includes a backbone network, a neck network, and a detection head, and the first processing unit is specifically used for: The fruit and vegetable images are input into the backbone network to obtain a first feature map; wherein, the first feature map is a feature map extracted by the backbone network at different levels; The first feature map is input into the neck network to obtain a second feature map; wherein, the second feature map is a multi-scale feature map of the neck network after fusing the first feature map; The second feature map is input into the detection head to obtain the fruit stalk information and the fruit information.

[0133] Furthermore, the fruit and vegetable bunch-picking device also includes: The sixth processing unit is used to collect fruit and vegetable training image data; wherein the fruit and vegetable training image data is collected under different weather conditions; The seventh processing unit is used to preprocess the fruit and vegetable training image data to obtain data-enhanced fruit and vegetable training images; The eighth processing unit is used to divide the data-enhanced fruit and vegetable training images to obtain a training dataset, a validation dataset, and a test dataset. The ninth processing unit is used to train the initialized target detection model based on the training dataset, the validation dataset, the test dataset, and a preset model training algorithm to obtain the pre-trained target detection model.

[0134] Furthermore, the second processing unit is specifically used for: A dynamic search region is established based on the coordinates of the fruit stalk bounding box; wherein, the dynamic search region is an elliptical search region generated according to the direction of the main axis of the fruit stalk; Based on the fruit bounding box coordinates and the dynamic search area, determine the fruits after the initial screening; Based on the fruit bounding box coordinates corresponding to the fruits after the initial screening and the preset fruit clustering algorithm, the association relationship between the fruit stalk and the fruit is obtained.

[0135] Furthermore, the third processing unit is specifically used for: The number of mature fruits in each bunch of fruits and vegetables is determined based on the relationship between the fruit stalk and the fruit and the fruit category. If the number of mature fruits is not less than a preset threshold, the bunch of fruits and vegetables is determined to be a harvestable bunch, and the harvestable bunch information is obtained.

[0136] Furthermore, the fruit and vegetable bunch-picking device also includes: The tenth processing unit is used to determine that the bunch of fruits and vegetables is unharvestable if the number of mature fruits is less than a preset threshold, and to obtain unharvestable bunch information.

[0137] Furthermore, the fifth processing unit is specifically used for: Based on the three-dimensional picking coordinates and the preset path planning algorithm, the robotic arm is controlled to move to the corresponding picking position; The clamping assembly is controlled to pre-clamp the stems of the harvestable bunch, and the scissor assembly is controlled to cut along the target plane; wherein, the target plane is a plane perpendicular to the axial direction of the stems of the harvestable bunch.

[0138] Figure 9 This is a schematic diagram of the control host provided in the fourth embodiment of this application. Figure 9 As shown, the control host 9 of this embodiment includes: a processor 90, a memory 91, and a computer program 92 stored in the memory 91 and executable on the processor 90, such as a fruit and vegetable bunch harvesting program. When the processor 90 executes the computer program 92, it implements the steps in the above-described fruit and vegetable bunch harvesting method embodiments, for example... Figure 1 Steps 101 to 106 are shown. Alternatively, when the processor 90 executes the computer program 92, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 6 The functions of modules 810 to 860 are shown.

[0139] For example, the computer program 92 can be divided into one or more modules / units, which are stored in the memory 91 and executed by the processor 90 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 92 in the control host 9. For example, the computer program 92 can be divided into a first acquisition unit, a first processing unit, a second processing unit, a third processing unit, a fourth processing unit, and a fifth processing unit, with the specific functions of each unit as follows: The first acquisition unit is used to acquire fruit and vegetable image data; wherein, the fruit and vegetable image data is image data corresponding to a bunch of fruits and vegetables; The first processing unit is used to input the fruit and vegetable image data into a pre-trained target detection model to obtain fruit stem information and fruit information; wherein, the fruit stem information includes at least the fruit stem bounding box coordinates, and the fruit information includes at least the fruit bounding box coordinates and the fruit category; The second processing unit is used to obtain the association relationship between the fruit stalk and the fruit based on the coordinates of the fruit stalk bounding box, the coordinates of the fruit bounding box, and a preset same-string matching algorithm. The third processing unit is used to determine the harvestable bunches based on the relationship between the fruit stalk and the fruit, the fruit category and the preset picking rules, and obtain harvestable bunch information; wherein, the harvestable bunch information includes at least the center coordinates of the fruit stalk of the harvestable bunch; The fourth processing unit is used to obtain the three-dimensional picking coordinates in the robotic arm base coordinate system based on the center coordinates of the fruit stalk of the pickable string and a preset coordinate transformation matrix. The fifth processing unit is used to control the robotic arm to move to the corresponding picking position according to the three-dimensional picking coordinates, and to control the end effector to perform a series of picking operations.

[0140] The control host may include, but is not limited to, a processor 90 and a memory 91. Those skilled in the art will understand that... Figure 9 This is merely an example of the control host 9 and does not constitute a limitation on the control host 9. It may include more or fewer components than shown, or combine certain components, or different components. For example, the control host may also include input / output devices, network access devices, buses, etc.

[0141] The processor 90 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0142] The memory 91 can be an internal storage unit of the control host 9, such as a hard disk or memory of the control host 9. The memory 91 can also be an external storage device of the control host 9, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the control host 9. Furthermore, the control host 9 can include both its internal storage unit and external storage devices. The memory 91 is used to store the computer program and other programs and data required by the control host. The memory 91 can also be used to temporarily store data that has been output or will be output.

[0143] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0144] This application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0145] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0146] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0147] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0148] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0149] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0150] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0151] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0152] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for harvesting fruits and vegetables in bunches, characterized in that, Including the following steps: Acquire fruit and vegetable image data; wherein, the fruit and vegetable image data is image data corresponding to bunches of fruits and vegetables; The fruit and vegetable image data is input into a pre-trained target detection model to obtain fruit stem information and fruit information; wherein, the fruit stem information includes at least the fruit stem bounding box coordinates, and the fruit information includes at least the fruit bounding box coordinates and the fruit category; The association between the fruit stalk and the fruit is obtained based on the fruit stalk bounding box coordinates, the fruit bounding box coordinates, and a preset same-string matching algorithm. Based on the relationship between the fruit stalk and the fruit, the fruit category, and the preset harvesting rules, a harvestable bunch is determined to obtain harvestable bunch information; wherein, the harvestable bunch information includes at least the center coordinates of the fruit stalk of the harvestable bunch; Based on the center coordinates of the fruit stalk of the harvestable string and the preset coordinate transformation matrix, the three-dimensional harvesting coordinates in the robotic arm base coordinate system are obtained. Based on the three-dimensional picking coordinates, the robotic arm is controlled to move to the corresponding picking position, and the end effector is controlled to perform a series of picking operations.

2. The method for harvesting fruits and vegetables in bunches according to claim 1, characterized in that, The target detection model is an improvement based on the YOLOv8s model. The target detection model includes a backbone network, a neck network, and a detection head. The backbone network introduces a CoordAtt attention mechanism and an SPPFCSPC feature extraction layer. The CoordAtt attention mechanism is used to capture the spatial information of the fruit and the fruit stalk, and the SPPFCSPC feature extraction layer is used to enhance the linear features of the fruit stalk.

3. The method for picking tea buds based on depth vision according to claim 1, characterized in that, The target detection model includes a backbone network, a neck network, and a detection head. The step of inputting the fruit and vegetable image data into the pre-trained target detection model to obtain fruit stem information and fruit information includes the following steps: The fruit and vegetable images are input into the backbone network to obtain a first feature map; wherein, the first feature map is a feature map extracted by the backbone network at different levels; The first feature map is input into the neck network to obtain a second feature map; wherein, the second feature map is a multi-scale feature map of the neck network after fusing the first feature map; The second feature map is input into the detection head to obtain the fruit stalk information and the fruit information.

4. The method for harvesting fruits and vegetables in bunches according to any one of claims 1 to 3, characterized in that, Before inputting the fruit and vegetable image data into a pre-trained target detection model to obtain fruit stem and fruit information, the following steps are included: Collect training image data of fruits and vegetables; wherein, the training image data of fruits and vegetables is collected under different weather conditions; The fruit and vegetable training image data is preprocessed to obtain data-enhanced fruit and vegetable training images; The data-enhanced fruit and vegetable training images are divided to obtain a training dataset, a validation dataset, and a test dataset. The initialized target detection model is trained using the training dataset, the validation dataset, the test dataset, and a preset model training algorithm to obtain the pre-trained target detection model.

5. The method for harvesting fruits and vegetables in bunches according to any one of claims 1 to 3, characterized in that, The step of obtaining the association between the fruit stalk and the fruit based on the fruit stalk bounding box coordinates, the fruit bounding box coordinates, and a preset same-string matching algorithm includes the following steps: A dynamic search region is established based on the coordinates of the fruit stalk bounding box; wherein, the dynamic search region is an elliptical search region generated according to the direction of the main axis of the fruit stalk; Based on the fruit bounding box coordinates and the dynamic search area, determine the fruits after the initial screening; Based on the fruit bounding box coordinates corresponding to the fruits after the initial screening and the preset fruit clustering algorithm, the association relationship between the fruit stalk and the fruit is obtained.

6. The method for harvesting fruits and vegetables in bunches according to any one of claims 1 to 3, characterized in that, The step of determining harvestable bunches and obtaining harvestable bunch information based on the relationship between the fruit stalk and the fruit, the fruit category, and preset harvesting rules includes the following steps: The number of mature fruits in each bunch of fruits and vegetables is determined based on the relationship between the fruit stalk and the fruit and the fruit category. If the number of mature fruits is not less than a preset threshold, the bunch of fruits and vegetables is determined to be a harvestable bunch, and the harvestable bunch information is obtained.

7. The method for harvesting fruits and vegetables in bunches according to claim 6, characterized in that, The method further includes the following steps: If the number of mature fruits is less than a preset threshold, the bunch of fruits and vegetables is determined to be an unharvestable bunch, and unharvestable bunch information is obtained.

8. The method for harvesting fruits and vegetables in bunches according to any one of claims 1 to 3, characterized in that, The end effector integrates a gripping assembly and a scissor assembly. Based on the three-dimensional picking coordinates, the robotic arm is controlled to move to the corresponding picking position, and the end effector is controlled to perform a series of picking operations, including the following steps: Based on the three-dimensional picking coordinates and the preset path planning algorithm, the robotic arm is controlled to move to the corresponding picking position; The clamping assembly is controlled to pre-clamp the stems of the harvestable bunch, and the scissor assembly is controlled to cut along the target plane; wherein, the target plane is a plane perpendicular to the axial direction of the stems of the harvestable bunch.

9. The method for harvesting fruits and vegetables in bunches according to claim 8, characterized in that, After controlling the scissor assembly to perform a cutting action along the target plane, the method further includes the following steps: The robotic arm is controlled to move the harvested bunches of fruits and vegetables to the top of the fruit basket, and the gripping assembly is controlled to perform a releasing action.

10. A fruit and vegetable bunch-harvesting device, characterized in that, include: Control unit, robot, end effector, and depth camera; The control host is connected to the robot, the end effector, and the depth camera, respectively. The robot includes a robotic arm and a robot body; The end effector includes a clamping assembly and a scissor assembly; The clamping assembly, the scissor assembly, and the depth camera are mounted on the end of the robotic arm; The control host and the robotic arm are mounted on the robot body; The control host is used to perform the steps in the method for harvesting fruits and vegetables in bunches as described in any one of claims 1 to 9.

Images