Methods and apparatus for continuous harvesting of fruits and vegetables

Abstract

This invention belongs to the field of automated fruit and vegetable harvesting technology, and provides a method and apparatus for continuous fruit and vegetable harvesting. The method includes: calculating the harvestability of multiple fruits and vegetables to be harvested, whereby the harvestability characterizes the harvesting success rate when obstacles exist around each fruit and vegetable; determining a first total harvesting cost based on the harvestability, missed harvesting rate, and harvesting path length of each fruit and vegetable; constructing and solving an optimization problem with the first total harvesting cost as the optimization objective to obtain the harvesting order of each fruit and vegetable, so that a robotic arm can continuously harvest multiple fruits and vegetables according to the harvesting order. The method and apparatus of this invention improve the harvesting success rate and efficiency of fruits and vegetables in dynamic clustered growth environments.

Description

Technical Field

[0001] This invention relates to the field of automated fruit and vegetable harvesting technology, and in particular to a method and apparatus for continuous fruit and vegetable harvesting. Background Technology

[0002] For fruits and vegetables that grow in dynamic clusters, optimizing the sequence of continuous harvesting is a core element that directly affects the success rate and efficiency of harvesting.

[0003] In related technologies, the continuous harvesting problem is usually modeled as the classic Traveling Salesman Problem (TSP) or its variants for solution, with minimizing path length as the core optimization objective. However, when using robotic arms for harvesting, the shortest path is not always optimal when dealing with delicate crops like strawberries that often grow in clusters. In particular, during the harvesting process, the harvesting difficulty of the surrounding fruits changes after each fruit is picked, resulting in inaccurate path planning results for continuous harvesting of strawberries in complex clustered scenarios using existing technologies. Consequently, the continuous harvesting efficiency of robotic arms for clustered fruits and vegetables is low. Summary of the Invention

[0004] This invention provides a method and apparatus for continuous harvesting of fruits and vegetables, which solves the defects of the prior art when continuously harvesting clustered fruits and vegetables, which only focuses on minimizing the path length as the core optimization objective, resulting in inaccurate path planning results and low continuous harvesting efficiency; the method of this invention improves the success rate and harvesting efficiency of fruits and vegetables in dynamic clustered growth environments.

[0005] This invention provides a method for continuous harvesting of fruits and vegetables, comprising:

[0006] Calculate the harvestability of multiple fruits and vegetables to be harvested, whereby the harvestability is used to characterize the harvesting success rate when there are obstacles around each fruit and vegetable;

[0007] The first total harvesting cost is determined based on the ease of harvesting, the missed harvesting rate, and the length of the harvesting path for each fruit and vegetable. An optimization problem is then constructed and solved using the first total harvesting cost as the optimization objective to obtain the harvesting order of each fruit and vegetable, so that the robotic arm can continuously harvest the multiple fruits and vegetables according to the harvesting order.

[0008] According to a method for continuous harvesting of fruits and vegetables provided by the present invention, the calculation of the harvestability of multiple fruits and vegetables to be harvested includes:

[0009] The multiple fruits and vegetables are processed based on the fruit and vegetable harvestability prediction model to obtain the first harvestability.

[0010] The fruit and vegetable harvestability prediction model is trained through the following steps:

[0011] Target detection is performed on the sample fruit and vegetable images to obtain the detection results for each fruit and vegetable;

[0012] For each fruit and vegetable, a region of interest is determined based on the detection results, and the convolutional deep learning network is trained using the region of interest as training samples. Under the condition of satisfying the iteration conditions, the fruit and vegetable harvestability prediction model is obtained.

[0013] According to a method for continuous harvesting of fruits and vegetables provided by the present invention, the calculation of the harvestability of multiple fruits and vegetables to be harvested further includes:

[0014] The second harvestability is obtained by calculating the harvestability of each fruit and vegetable using the following formula:

[0015] ;

[0016] in, For the second ease of acquisition, k b 、k c and k u These represent the harvesting difficulty coefficients corresponding to obstacles at the bottom, middle, and top of fruits and vegetables, respectively. n, m, o This indicates the number of obstacles at the corresponding location. The bottom of the fruit and vegetable i The distance between the obstacle and the fruits and vegetables The middle part of fruits and vegetables j The distance between the obstacle and the fruits and vegetables The top of the fruit and vegetable k The distance between the obstacle and the fruits and vegetables; q The value of is determined based on the distribution of obstacles relative to the target.

[0017] According to a method for continuous harvesting of fruits and vegetables provided by the present invention, after obtaining the second harvestability, the method further includes:

[0018] The target harvestability is determined from the first harvestability and the second harvestability of each fruit and vegetable.

[0019] According to a method for continuous harvesting of fruits and vegetables provided by the present invention, after calculating the harvestability of multiple fruits and vegetables to be harvested, the method further includes:

[0020] The second total harvest cost is determined based on the first total harvest cost and the grading information of each fruit and vegetable. A hierarchical optimization problem is constructed and solved with the second total harvest cost as the optimization objective to obtain the harvesting order.

[0021] The hierarchical optimization problem includes an outer optimization problem and an inner optimization problem. The outer optimization problem is used to optimize the harvesting order of all fruits and vegetables of different grades, while the inner optimization problem is used to optimize the harvesting order of all fruits and vegetables of the same grade.

[0022] According to a method for continuous harvesting of fruits and vegetables provided by the present invention, after calculating the harvestability of multiple fruits and vegetables to be harvested, the method further includes:

[0023] If the harvestability of a target fruit or vegetable is lower than the harvestability threshold, the harvesting of the target fruit or vegetable shall be cancelled; wherein the target fruit or vegetable belongs to at least one of the plurality of fruits or vegetables.

[0024] The present invention also provides a continuous fruit and vegetable harvesting device, comprising:

[0025] The calculation module is used to calculate the harvestability of multiple fruits and vegetables to be harvested, wherein the harvestability is used to characterize the harvesting success rate when there are obstacles around each fruit and vegetable.

[0026] The first harvesting planning module is used to determine the first total harvesting cost based on the ease of harvesting, missed harvesting rate and harvesting path length of each fruit and vegetable, and to construct and solve an optimization problem with the first total harvesting cost as the optimization objective, so as to obtain the harvesting order of each fruit and vegetable, so that the robotic arm can harvest the multiple fruits and vegetables continuously according to the harvesting order.

[0027] According to the continuous fruit and vegetable harvesting device provided by the present invention, the method further includes:

[0028] The second harvesting planning module is used to determine the second total harvesting cost based on the first total harvesting cost and the grading information of each fruit and vegetable after calculating the harvestability of the multiple fruits and vegetables to be harvested, and to construct and solve a hierarchical optimization problem with the second total harvesting cost as the optimization objective to obtain the harvesting order.

[0029] The hierarchical optimization problem includes an outer optimization problem and an inner optimization problem. The outer optimization problem is used to optimize the harvesting order of all fruits and vegetables of different grades, while the inner optimization problem is used to optimize the harvesting order of all fruits and vegetables of the same grade.

[0030] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the continuous fruit and vegetable harvesting method described above.

[0031] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the continuous fruit and vegetable harvesting method as described above.

[0032] The continuous fruit and vegetable harvesting method and apparatus provided by the present invention calculates the harvestability of multiple fruits and vegetables to be harvested, determines the first total harvesting cost based on the harvestability, missed harvesting rate and harvesting path length of each fruit and vegetable, constructs and solves an optimization problem with the first total harvesting cost as the optimization objective, obtains the harvesting order of each fruit and vegetable, and continuously harvests multiple fruits and vegetables according to the harvesting order, thereby improving the success rate and efficiency of harvesting fruits and vegetables in dynamic cluster growth environments. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0034] Figure 1 This is one of the flowcharts illustrating the continuous fruit and vegetable harvesting method provided by this invention.

[0035] Figure 2 This is a diagram showing the changes in harvesting targets and obstacles during the harvesting planning process provided by the present invention.

[0036] Figure 3 This is a schematic diagram of the process of obtaining a fruit and vegetable harvestability prediction model through two stages, as provided by the present invention.

[0037] Figure 4 This is the second flowchart of the continuous fruit and vegetable harvesting method provided by the present invention.

[0038] Figure 5 This is a schematic diagram of the structure of the continuous fruit and vegetable harvesting device provided by the present invention.

[0039] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0041] The following is combined Figures 1-5 The present invention describes a method and apparatus for continuous harvesting of fruits and vegetables.

[0042] Figure 1This is one of the flowcharts illustrating the continuous fruit and vegetable harvesting method provided by the present invention, such as... Figure 1 As shown, this method is applied to fruit and vegetable harvesting robots or robotic arms, and includes the following steps:

[0043] Step 110: Calculate the harvestability of multiple fruits and vegetables to be harvested. Harvesting ability is used to characterize the harvesting success rate when there are obstacles around each fruit and vegetable.

[0044] In this step, fruits and vegetables include, but are not limited to, strawberries, grapes, kiwis, and bell peppers; this embodiment uses strawberries as an example to illustrate the continuous harvesting method for multiple strawberries.

[0045] It should be noted that for each mature strawberry growing in a cluster, the success rate of picking each strawberry varies depending on the complexity of the surrounding obstacles. Quantitatively predicting the ease of picking each strawberry in the cluster before picking will help select the picking order with the lower overall picking difficulty, thereby improving the picking success rate.

[0046] In this step, a high ease of picking for fruits and vegetables such as strawberries indicates that there are few obstacles around the strawberry, making it easy for the picking robot to approach and complete the picking; conversely, a low ease of picking indicates that there are many obstacles around the strawberry, making picking difficult and prone to collisions or picking failures. The ease of picking provided in this embodiment is an important basis for planning the picking sequence.

[0047] In this step, obstacles refer to objects that exist around the strawberries and prevent the picking robot from picking them smoothly; these obstacles may include the leaves and stems of the strawberry plant, other adjacent strawberry fruits, or even the structures supporting the plant; the number, location, and shape of obstacles will affect the ease of picking the strawberries.

[0048] In this step, for individual strawberries growing in clusters, the harvestability of each strawberry can be evaluated by establishing a multi-factor weighted combination method. These factors include the distance or orientation of different parts of an individual strawberry to surrounding obstacles.

[0049] In this step, deep learning technology can also be used to build a strawberry harvestability prediction model. For example, by training a convolutional neural network or other regression model, the harvesting success rate of individual strawberries can be predicted, and the corresponding harvestability can be determined based on the harvesting success rate; the higher the harvesting success rate of strawberries, the greater the harvestability.

[0050] Step 120: Determine the first total harvesting cost based on the ease of harvesting, missed harvesting rate, and harvesting path length of each fruit and vegetable. Construct and solve an optimization problem with the first total harvesting cost as the optimization objective to obtain the harvesting order of each fruit and vegetable, so that the robotic arm can harvest multiple fruits and vegetables continuously according to the harvesting order.

[0051] In this step, the missed harvest rate refers to the proportion of strawberries that could not be successfully harvested in a single harvesting process due to various reasons (such as excessive harvesting difficulty, robot malfunction, etc.). Reducing the missed harvest rate can improve harvesting efficiency and profits.

[0052] In this step, the picking path length refers to the total distance traveled by the picking robot when picking multiple strawberries consecutively; shortening the picking path length can reduce the robot's movement time and improve picking efficiency.

[0053] In this step, the first total harvesting cost is a quantitative assessment of the total cost of a harvesting process after comprehensively considering three factors: ease of harvesting, missed harvesting rate, and harvesting path length. The first total harvesting cost refers to various losses in the harvesting process, including the risk of harvesting failure, wasted time and energy, etc.

[0054] The optimization problem in this embodiment is to find an optimal picking order that minimizes the initial total picking cost. The picking order refers to the order in which the picking robot picks multiple strawberries. Different picking orders will result in different picking path lengths, missed picking rates, and total picking costs.

[0055] In the optimal harvesting path technical route provided in this embodiment, the optimal harvesting path can be adjusted according to the importance of ease of harvesting, harvesting efficiency, and missed harvesting rate. This is achieved by changing the weighting coefficients of ease of harvesting, path length, and missed harvesting rate. , and To achieve this. For example, when When the size is large, optimization focuses more on the difficulty of harvesting. When the size is large, optimization focuses more on the picking path, finding the shortest path and improving picking efficiency; specific adjustments to the weights will be made in simulation or model environment testing.

[0056] In this embodiment, in order to cope with possible environmental changes and unexpected situations, the present invention considers using machine learning algorithms to automatically adjust weights based on historical data.

[0057] It should be noted that in this embodiment, when calculating the ease of picking the next strawberry after each strawberry is picked, the shading effect of the already picked strawberries on the current strawberry needs to be removed.

[0058] For example, a background-filling image processing strategy (filling with the average color of the surrounding area) can be used to subtract the picked strawberries and recalculate the pickability of the remaining strawberries on a new image.

[0059] Figure 2 This is a diagram showing the changes in harvesting targets and obstacles during the harvesting planning process provided by the present invention. Figure 2In the illustrated embodiment, the picking target and obstacles are dynamically changing during the strawberry picking planning. When strawberry No. 1 in (a) is picked, strawberry No. 5, which was blocked in (b), becomes visible, which is beneficial for picking strawberry No. 5. When strawberry No. 2 in (b) is picked, the position of the green strawberry next to strawberry No. 2 in (c) also changes.

[0060] In this embodiment, since the difficulty of picking the surrounding fruits will change when each strawberry is picked, it is necessary to plan the optimal picking order before picking and find the picking path with the least total difficulty. The whole process is divided into the following two key stages: (1) establishing a reliable deduction and difficulty recalculation mechanism, and (2) constructing a path optimization framework.

[0061] For stage (1): Use the background-filling image processing strategy (fill with the average color of the surrounding area) to subtract the picked strawberries and recalculate the pickability of the remaining strawberries on the new image;

[0062] For phase (2): After the deduction mechanism is verified, create all possible collection sequences.

[0063] For example, path A: 1→2→3→4 (when sampling 2, subtract 1; when sampling 3, subtract 1+2); path B: 3→4→1→2 (when sampling 4, subtract 3; when sampling 1, subtract 3+4). We need to compare all possible paths and use an optimization algorithm to find the optimal path.

[0064] The continuous fruit and vegetable harvesting method provided in this invention calculates the ease of harvesting multiple fruits and vegetables, determines the first total harvesting cost based on the ease of harvesting, missed harvesting rate, and harvesting path length of each fruit and vegetable, constructs and solves an optimization problem with the first total harvesting cost as the optimization objective, obtains the harvesting order of each fruit and vegetable, and continuously harvests multiple fruits and vegetables according to the harvesting order, thereby improving the success rate and efficiency of harvesting fruits and vegetables in dynamic clustered growth environments.

[0065] In some embodiments, calculating the harvestability of multiple fruits and vegetables to be harvested includes: processing multiple fruits and vegetables based on a fruit and vegetable harvestability prediction model to obtain a first harvestability; wherein, the fruit and vegetable harvestability prediction model is trained through the following steps: performing target detection on sample fruit and vegetable images to obtain detection results for each fruit and vegetable; for each fruit and vegetable detection result, determining the region of interest based on the detection result, and using the region of interest as training samples to perform regression training on a convolutional deep learning network, and obtaining the fruit and vegetable harvestability prediction model under the condition of satisfying the iteration condition.

[0066] In this embodiment, an end-to-end method based on deep learning can be used to train a convolutional deep learning network to learn the relationship between the scene and the ease of harvesting, thereby obtaining a corresponding fruit and vegetable ease of harvesting prediction model.

[0067] For example, a point cloud-based method can be used to predict the picking difficulty by analyzing the spatial layout of strawberries in the point cloud, and a physical model of the ease of picking can be established and quantitatively calculated. The end-to-end method evaluates the picking success rate under different conditions through multi-scenario testing in a simulation environment, and obtains stable easy-to-pick prediction values ​​through extensive training.

[0068] It should be noted that the above end-to-end method does not require explicit analysis and modeling of obstacles around the strawberry. The model only needs to input the scene information of the strawberry (such as RGB image or RGB point cloud) to output the corresponding easy-to-collect prediction value.

[0069] In this embodiment, during the training sample preparation stage, the harvesting success rate under different environments is first tested. The harvesting success rate can be represented as the harvesting ease value. Then, the harvesting ease value obtained in the corresponding scenario (expressed as an RGB image or RGB point cloud) is calibrated to train the deep learning model, thereby achieving the purpose of predicting the harvesting ease under different scenarios.

[0070] Figure 3 This is a schematic diagram illustrating the process of obtaining a two-stage prediction model for the harvestability of fruits and vegetables provided by the present invention. Figure 3 In the illustrated embodiment, the process of obtaining a fruit and vegetable harvestability prediction model through two-stage deep learning is as follows:

[0071] Phase 1: Object Detection and ROI Extraction; A strawberry detection model is trained using a convolutional neural network. The original input image is fed into the object detection network, which outputs object detection boxes. Simultaneously, the bounding boxes of the target strawberries are expanded, and a specific region of interest (ROI) is extracted. Figure 3 (This shows a schematic diagram illustrating the expanded ROI for the Strawberry AG). Figure 3 Solid lines represent ROIs, and dashed lines represent the corresponding expanded boxes.

[0072] Phase 2: Difficulty Value Prediction Regression; Improvements to the Convolutional Deep Learning Network; The extracted ROI is used as the input for regression training to output the corresponding strawberry picking difficulty value; Specifically, by adjusting the output layer of the convolutional deep learning network, the classification head is changed to the regression head, and the corresponding loss function is modified. Then, the training samples obtained in the training sample preparation phase are input into the improved network for iterative training to obtain the corresponding fruit and vegetable harvestability prediction model.

[0073] In the embodiment for calculating the second ease of collection, firstly, target detection is performed on the fruit and vegetable images captured in the harvesting scene to locate and identify the specific location and bounding box of each fruit and vegetable for subsequent analysis; then, based on the detection results, the region of interest (ROI) of each fruit and vegetable is cropped from the original image, and these ROIs are used as input samples for subsequent model training and inference; then, for each fruit and vegetable's ROI, a convolutional neural network model is constructed and trained to map the input image information to the ease of collection prediction value of that fruit and vegetable (note that this process is a regression task, unlike classification, and the output is a continuous value); next, the network parameters are iteratively optimized on a large amount of sample data until the convergence condition is met, resulting in an ease of collection prediction model with better performance and strong generalization ability; finally, the fruit and vegetable images to be detected are input into the model for ease of collection prediction, and the corresponding second ease of collection is output.

[0074] Furthermore, the scarcity of diverse picking scenarios and their ease of collection is a key bottleneck for this method. To overcome this limitation, this embodiment employs a simulated picking platform based on the PyBullet physics engine, integrating a strawberry scenario with a robot system. By leveraging the simulation environment's ability to randomly generate scenarios and the joint simulation framework of PyBullet and the Robot Operating System (ROS), it automatically completes the perception, recognition, and control closed loop, generating batches of picking success rate training data. This simulation strategy can achieve large-scale data accumulation at a cost far lower than on-site collection, thereby training a model with greater robustness and generalization performance.

[0075] The continuous harvesting method for fruits and vegetables provided in this invention processes multiple fruits and vegetables by training a fruit and vegetable harvestability prediction model to obtain a second harvestability score, thereby improving the calculation efficiency of harvestability score and further optimizing the continuous harvesting strategy to achieve efficient and low-damage fruit and vegetable harvesting.

[0076] In some embodiments, calculating the harvestability of multiple fruits and vegetables to be harvested further includes: calculating the harvestability of each fruit and vegetable using the following formula to obtain a second harvestability:

[0077] ;

[0078] in, For the second ease of collection, k b 、k c and k u These represent the harvesting difficulty coefficients corresponding to obstacles at the bottom, middle, and top of fruits and vegetables, respectively. n, m, o This indicates the number of obstacles at the corresponding location. The bottom of the fruit and vegetable i The distance between the obstacle and the fruits and vegetables The middle part of fruits and vegetables jThe distance between the obstacle and the fruits and vegetables The top of the fruit and vegetable k The distance between the obstacle and the fruits and vegetables; q The value of is determined based on the distribution of obstacles relative to the target.

[0079] In this embodiment, q The value is determined based on the distribution of obstacles. For example, it is more difficult to pick up the fruit when the obstacle is in front of the target than when it is behind it. Therefore, when the obstacle is in front of the target... q The value is larger than the value at the end.

[0080] For example, when the obstacle is behind the target, q =1, while when the obstacle is in front of the target, q A larger value can be selected because obstacles in front will cause the target's point cloud to be incomplete and will also cause distortion of the target's point cloud, thus affecting the accuracy of perception.

[0081] In this embodiment, based on the point cloud distribution data of multiple strawberries growing in clusters, the distribution of obstacles in the region of interest around each strawberry is calculated and predicted. Furthermore, the ease of harvesting is constructed by considering the possibility that obstacles in front of the target will increase the difficulty of harvesting. The mathematical calculation formula.

[0082] In one feasible embodiment, in a strawberry planting area, three-dimensional point cloud data of multiple strawberry plants around the target area were acquired by laser scanning; for a specific strawberry target, the data of obstacles at its bottom, middle, and top were analyzed as follows:

[0083] There are two obstacles at the bottom, with distances of 4cm and 6cm respectively.

[0084] There are 3 obstacles in the middle, at distances of 3cm, 5cm and 7cm respectively.

[0085] There is one obstacle at the top, 9cm away.

[0086] The difficulty level of picking is set as follows: k b =0.5, k c =0.3, k u =0.2, q All values ​​are set to 1 (assuming all obstacles are located behind the target). The ease of harvesting of the bottom, middle and top parts of the strawberry are calculated according to the above formula. The three ease of harvesting are then added together to obtain the total ease of harvesting, which is the ease of harvesting of a single strawberry.

[0087] Based on this, if the ease of picking is calculated separately for the same batch of strawberries, the robot can prioritize picking strawberries with lower ease of picking in order to improve the success rate of picking.

[0088] The continuous fruit and vegetable harvesting method provided in this invention calculates the ease of harvesting by quantitatively calculating the situation of obstacles at the bottom, middle and top of the fruit and vegetables. This achieves an accurate description of the impact of obstacles around the fruit and vegetables, enabling the harvesting robot to scientifically assess the ease of harvesting each fruit and vegetable.

[0089] In some embodiments, after obtaining the second harvestability, the continuous harvesting method for fruits and vegetables further includes determining a target harvestability from the first and second harvestability of each fruit and vegetable.

[0090] In this embodiment, the picking effects (including picking success rate, picking accuracy, algorithm running time, and other indicators) of the first and second ease of picking are compared and tested in an actual picking environment through picking experiments or simulations.

[0091] Among them, the ease of harvest prediction accuracy is used to compare the degree of agreement between the two ease of harvest and the actual harvest success rate, and to evaluate the reliability of the prediction; the algorithm running time is used to evaluate the time consumed by using different ease of harvest calculation methods, so as to support the needs of real-time harvesting.

[0092] In this embodiment, based on the performance evaluation results and the actual application requirements, it is decided to use either the first ease of harvesting or the second ease of harvesting as the target ease of harvesting, so as to provide a unified input for subsequent harvesting path optimization.

[0093] In other embodiments, a weighted fusion of the first and second ease of acquisition may be used as the target ease of acquisition.

[0094] The continuous harvesting method for fruits and vegetables provided in this invention improves the reliability of the harvestability of each strawberry by scientifically evaluating and comparing different harvestability indices and determining the target harvestability from the first and second harvestability of each fruit and vegetable.

[0095] In some embodiments, after calculating the ease of harvesting of multiple fruits and vegetables to be harvested, the continuous harvesting method further includes: determining a second total harvesting cost based on a first total harvesting cost and the grading information of each fruit and vegetable, and constructing and solving a hierarchical optimization problem with the second total harvesting cost as the optimization objective to obtain the harvesting order; wherein, the hierarchical optimization problem includes an outer optimization problem and an inner optimization problem, the outer optimization problem is used to optimize the harvesting order of all fruits and vegetables of different grades, and the inner optimization problem is used to optimize the harvesting order of all fruits and vegetables of the same grade.

[0096] It's important to note that manual strawberry picking typically involves grading strawberries by size after harvesting, which is not only time-consuming and labor-intensive but also increases fruit loss. If robots could automatically grade strawberries during the picking process and place strawberries of the same grade in designated areas, picking efficiency would be significantly improved and damage reduced. Compared to alternating picking and placing different grades of fruit, a robot capable of continuously picking strawberries of the same grade, collecting them in its picking hand, and then placing them uniformly would greatly enhance operational efficiency.

[0097] To achieve the above objectives, this embodiment comprehensively considers multiple factors such as picking path length, ease of picking, strawberry grade, and missed picking rate to construct a two-layer optimization model to solve the outer layer optimization problem and the inner layer optimization problem.

[0098] The two-layer optimization model includes an inner optimization model and an outer optimization model. The inner optimization model is used to find the optimal picking order of all strawberries within each given grade. The outer optimization model is used to determine the optimal sorting of different strawberry grades.

[0099] Specifically, the goal of addressing the aforementioned inner-layer optimization problem is to optimize the picking order within each strawberry grade, thereby minimizing the total picking cost within that grade.

[0100] Based on the ease of harvesting each target strawberry, the path length, and the missed harvest rate. The objective function of the inner optimization problem, obtained by superposition, can be expressed as a linear weighted model, which can be represented by the following equation:

[0101] ;

[0102] ;

[0103] ;

[0104] in, , and They represent ease of collection, respectively. Picking path length and missed sampling rate Weighting coefficients; For level The set of all possible strawberry picking orders; and They represent in In the ranking, the first The difficulty and path length for picking each strawberry; This indicates ensuring that at the level Each strawberry is picked only once along the picking path. This indicates the total number of strawberries in the workspace. This indicates the number of strawberries that were voluntarily abandoned due to the difficulty of picking them; For the order of picking, For level The cost of the optimal harvesting route For level The number of strawberries contained, In the first m Within each level, the first The order in which the strawberries are picked.

[0105] To reduce the missed sampling rate This embodiment takes into account the strategy of the robot actively abandoning harvesting when the target is difficult to pick. This not only increases the continuous picking time and improves picking efficiency, but also reduces fruit damage. Therefore, the embodiment sets an upper limit on the ease of picking strawberries ( This serves as a boundary condition, ensuring the robot only picks targets it is relatively confident of picking, skipping targets whose difficulty exceeds the upper limit. However, skipping targets exceeding the upper limit of ease of picking may affect the missed picking rate. ,therefore, It is also one of the optimization goals.

[0106] In this embodiment, after the inner-layer optimization problem is completed, the objective of the outer-layer optimization problem is to determine the optimal harvesting order among different levels based on the optimal harvesting path cost for each level; the objective function of the outer-layer optimization problem can be expressed as:

[0107] ;

[0108] in, The strawberry grade is represented by a vector of permutations and combinations, where each... Representing the Each level It is the set of all possible strawberry grade picking orders. This refers to the total number of grades of strawberries; regarding the picking order... Picking target of and All calculations will remove the previous ones. The existing harvested targets were identified to better adapt to the impact of changes in harvesting targets and obstacles on strawberry harvestability.

[0109] In this embodiment, after the outer optimization problem is completed, the optimal picking order can be output to guide the picking robot to pick multiple strawberries growing in clusters in sequence.

[0110] The continuous fruit and vegetable harvesting method provided in this invention effectively balances the harvesting path and harvesting success rate by introducing grading information to achieve tiered optimization, while ensuring the quality and efficiency of tiered harvesting, and further improving the accuracy and efficiency of continuous fruit and vegetable harvesting.

[0111] In some embodiments, after calculating the ease of harvesting of a plurality of fruits and vegetables to be harvested, the continuous harvesting method further includes: canceling the harvesting of the target fruit and vegetable if the ease of harvesting of the target fruit and vegetable is lower than the ease of harvesting threshold; wherein the target fruit and vegetable belongs to at least one of a plurality of fruits and vegetables.

[0112] In this embodiment, the ease of sampling threshold can be set in advance according to user needs; for example, the ease of sampling threshold is determined by historical experimental data and operational experience, and is generally set within a reasonable range to avoid excessive missed sampling or erroneous sampling.

[0113] In this embodiment, the ease of harvesting threshold can also be dynamically adjusted based on different batches or harvesting experience to balance harvesting coverage and harvesting success rate.

[0114] In this embodiment, picking is abandoned when the ease of picking of a strawberry exceeds a threshold. For example, in a continuous strawberry picking demonstration area, the target ease of picking was calculated for 20 tested fruits, and the ease of picking threshold was set to 0.4. For fruit A, the target ease of picking is 0.65, which meets the threshold and can be picked normally. For fruit B, the target ease of picking is 0.35, which is below the threshold and is automatically removed from the picking plan. For fruit C, the target ease of picking is 0.25, which is below the threshold and is automatically removed from the picking plan. That is, in the actual picking process, the robot skips fruits B and C and directly jumps to the next fruit with a qualified ease of picking, thereby avoiding wasting time and causing bad operations on difficult-to-pick fruits.

[0115] The continuous fruit and vegetable harvesting method provided in this invention effectively ensures the stability and economic benefits of the continuous fruit and vegetable harvesting process by canceling the harvesting of target fruits and vegetables when their harvestability is lower than the harvestability threshold.

[0116] Figure 4 This is the second flowchart illustrating the continuous fruit and vegetable harvesting method provided by this invention. Figure 4 In the illustrated embodiment, pickable strawberries are first determined, and two different pickability prediction methods (point cloud distribution-based method and end-to-end method) are used to calculate two pickability scores respectively. Then, the optimal pickability prediction method is evaluated, and the corresponding pickability score is output. The point cloud distribution-based method includes establishing a physical model of pickability and calculating pickability. The end-to-end method includes establishing a simulation environment and testing the mapping relationship between picking success rate and environment.

[0117] Next, the missed harvest rate of each strawberry is obtained through harvestability prediction; the distance between multiple harvestable strawberries and the robotic arm is calculated based on the harvesting path length of each harvestable strawberry, and the shortest path is selected; strawberries of the same grade are prioritized for harvesting based on their harvestability (corresponding to weights). ), picking path length (corresponding weight) ) and missed sampling rate (corresponding weight) With strawberry grading as the constraint, the algorithm is substituted into a multi-objective optimization algorithm and solved to output the optimal picking path.

[0118] The continuous fruit and vegetable harvesting device provided by the present invention is described below. The continuous fruit and vegetable harvesting device described below can be referred to in correspondence with the continuous fruit and vegetable harvesting method described above.

[0119] Figure 5 This is a schematic diagram of the continuous fruit and vegetable harvesting device provided by the present invention, as shown below. Figure 5 As shown, the continuous fruit and vegetable harvesting device includes: a calculation module 510 and a first harvesting planning module 520.

[0120] The calculation module 510 is used to calculate the harvestability of multiple fruits and vegetables to be harvested. The harvestability is used to characterize the harvesting success rate when there are obstacles around each fruit and vegetable.

[0121] The first harvesting planning module 520 is used to determine the first total harvesting cost based on the ease of harvesting, missed harvesting rate and harvesting path length of each fruit and vegetable, and to construct and solve an optimization problem with the first total harvesting cost as the optimization objective, so as to obtain the harvesting order of each fruit and vegetable, so that the robotic arm can harvest multiple fruits and vegetables continuously according to the harvesting order.

[0122] The continuous fruit and vegetable harvesting device provided by this invention calculates the ease of harvesting multiple fruits and vegetables to be harvested, then determines the first total harvesting cost based on the ease of harvesting, the missed harvesting rate, and the harvesting path length of each fruit and vegetable. An optimization problem is constructed and solved with the first total harvesting cost as the optimization objective to obtain the harvesting order of each fruit and vegetable. Multiple fruits and vegetables are harvested continuously according to the harvesting order, thereby improving the success rate and efficiency of harvesting fruits and vegetables in dynamic cluster growth environments.

[0123] In some embodiments, the continuous fruit and vegetable harvesting device further includes a second harvesting planning module.

[0124] The second harvesting planning module is used to calculate the ease of harvesting of multiple fruits and vegetables to be harvested, determine the second total harvesting cost based on the first total harvesting cost and the grading information of each fruit and vegetable, and construct and solve a hierarchical optimization problem with the second total harvesting cost as the optimization objective to obtain the harvesting order.

[0125] The hierarchical optimization problem includes an outer optimization problem and an inner optimization problem. The outer optimization problem is used to optimize the harvesting order of all fruits and vegetables of different grades, while the inner optimization problem is used to optimize the harvesting order of all fruits and vegetables of the same grade.

[0126] The continuous fruit and vegetable harvesting device provided in this invention effectively balances the harvesting path and harvesting success rate by introducing grading information to achieve tiered optimization, while ensuring the quality and efficiency of tiered harvesting, and further improving the accuracy and efficiency of continuous fruit and vegetable harvesting.

[0127] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640. The processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a continuous fruit and vegetable harvesting method. This method includes: calculating the harvestability of multiple fruits and vegetables to be harvested, where harvestability characterizes the harvesting success rate when obstacles exist around each fruit and vegetable; determining a first total harvesting cost based on the harvestability, missed harvesting rate, and harvesting path length of each fruit and vegetable; constructing and solving an optimization problem with the first total harvesting cost as the optimization objective to obtain the harvesting order of each fruit and vegetable, so that the robotic arm can continuously harvest multiple fruits and vegetables according to the harvesting order.

[0128] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0129] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the continuous fruit and vegetable harvesting method provided by the above methods. The method includes: calculating the harvestability of multiple fruits and vegetables to be harvested, wherein the harvestability is used to characterize the harvesting success rate when there are obstacles around each fruit and vegetable; determining a first total harvesting cost based on the harvestability, missed harvesting rate and harvesting path length of each fruit and vegetable; constructing and solving an optimization problem with the first total harvesting cost as the optimization objective to obtain the harvesting order of each fruit and vegetable, so that the robotic arm can continuously harvest multiple fruits and vegetables according to the harvesting order.

[0130] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the continuous fruit and vegetable harvesting method provided by the above methods. The method includes: calculating the harvestability of multiple fruits and vegetables to be harvested, wherein the harvestability is used to characterize the harvesting success rate when there are obstacles around each fruit and vegetable; determining a first total harvesting cost based on the harvestability, missed harvesting rate and harvesting path length of each fruit and vegetable; constructing and solving an optimization problem with the first total harvesting cost as the optimization objective to obtain the harvesting order of each fruit and vegetable, so that a robotic arm can continuously harvest multiple fruits and vegetables according to the harvesting order.

[0131] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0132] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method of continuous picking of fruits and vegetables, characterized in that, include: Calculate the harvestability of multiple fruits and vegetables to be harvested, whereby the harvestability is used to characterize the harvesting success rate when there are obstacles around each fruit and vegetable; The calculation of the harvestability of multiple fruits and vegetables to be harvested includes: The multiple fruits and vegetables are processed based on the fruit and vegetable harvestability prediction model to obtain the first harvestability. The fruit and vegetable harvestability prediction model is trained through the following steps: Target detection is performed on the sample fruit and vegetable images to obtain the detection results for each fruit and vegetable; For each fruit and vegetable, a region of interest is determined based on the detection results, and the convolutional deep learning network is trained using the region of interest as training samples. Under the condition of satisfying the iteration conditions, the fruit and vegetable harvestability prediction model is obtained. The first total harvesting cost is determined based on the ease of harvesting, the missed harvesting rate, and the length of the harvesting path for each fruit and vegetable. An optimization problem is constructed and solved with the first total harvesting cost as the optimization objective to obtain the harvesting order of each fruit and vegetable, so that the robotic arm can harvest the multiple fruits and vegetables continuously according to the harvesting order. The first total harvesting cost is determined by linear weighting of the ease of harvesting of each fruit and vegetable and its corresponding weight coefficient, the missed harvesting rate and its corresponding weight coefficient, and the harvesting path length and its corresponding weight coefficient.

2. The method for continuously picking fruits and vegetables according to claim 1, wherein The calculation of the harvestability of multiple fruits and vegetables to be harvested also includes: The second harvestability is obtained by calculating the harvestability of each fruit and vegetable using the following formula: ； in, For the second ease of acquisition, k b 、k c and k u These represent the harvesting difficulty coefficients corresponding to obstacles at the bottom, middle, and top of fruits and vegetables, respectively. n, m, o This indicates the number of obstacles at the corresponding location. The bottom of the fruit and vegetable i The distance between the obstacle and the fruits and vegetables The middle part of fruits and vegetables j The distance between the obstacle and the fruits and vegetables The top of the fruit and vegetable k The distance between the obstacle and the fruits and vegetables; q The value of is determined based on the distribution of obstacles relative to the target.

3. The method for continuous harvesting of fruits and vegetables according to claim 2, characterized in that, After obtaining the second ease of acquisition, the method further includes: The target harvestability is determined from the first harvestability and the second harvestability of each fruit and vegetable.

4. The method for continuous harvesting of fruits and vegetables according to claim 1, characterized in that, After calculating the harvestability of the multiple fruits and vegetables to be harvested, the method further includes: The second total harvest cost is determined based on the first total harvest cost and the grading information of each fruit and vegetable. A hierarchical optimization problem is constructed and solved with the second total harvest cost as the optimization objective to obtain the harvesting order. The hierarchical optimization problem includes an outer optimization problem and an inner optimization problem. The outer optimization problem is used to optimize the harvesting order of all fruits and vegetables of different grades, while the inner optimization problem is used to optimize the harvesting order of all fruits and vegetables of the same grade.

5. The method for continuous harvesting of fruits and vegetables according to claim 1, characterized in that, After calculating the harvestability of the multiple fruits and vegetables to be harvested, the method further includes: If the harvestability of a target fruit or vegetable is lower than the harvestability threshold, the harvesting of the target fruit or vegetable shall be cancelled; wherein the target fruit or vegetable belongs to at least one of the plurality of fruits or vegetables.

6. A continuous fruit and vegetable harvesting device, employing the continuous fruit and vegetable harvesting method as described in claim 1, characterized in that, include: The calculation module is used to calculate the harvestability of multiple fruits and vegetables to be harvested, wherein the harvestability is used to characterize the harvesting success rate when there are obstacles around each fruit and vegetable. The first harvesting planning module is used to determine the first total harvesting cost based on the ease of harvesting, missed harvesting rate and harvesting path length of each fruit and vegetable, and to construct and solve an optimization problem with the first total harvesting cost as the optimization objective, so as to obtain the harvesting order of each fruit and vegetable, so that the robotic arm can harvest the multiple fruits and vegetables continuously according to the harvesting order.

7. The continuous fruit and vegetable harvesting device according to claim 6, characterized in that, The method further includes: The second harvesting planning module is used to determine the second total harvesting cost based on the first total harvesting cost and the grading information of each fruit and vegetable after calculating the harvestability of the multiple fruits and vegetables to be harvested, and to construct and solve a hierarchical optimization problem with the second total harvesting cost as the optimization objective to obtain the harvesting order. The hierarchical optimization problem includes an outer optimization problem and an inner optimization problem. The outer optimization problem is used to optimize the harvesting order of all fruits and vegetables of different grades, while the inner optimization problem is used to optimize the harvesting order of all fruits and vegetables of the same grade.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the continuous fruit and vegetable harvesting method as described in any one of claims 1 to 5.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the continuous fruit and vegetable harvesting method as described in any one of claims 1 to 5.

Images