Closed-loop servo fruit stem shearing and picking system and closed-loop picking method thereof
By using a closed-loop servo fruit stalk shearing system, combined with visual recognition and environmental monitoring, precise positioning, stable alignment, and intelligent decision-making for fruit picking are achieved. This solves the problems of unstable positioning, uncontrollable open-loop operation, and unintelligent decision-making in indoor agricultural picking, thereby improving the success rate of picking and the reliability of operations.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HARBIN INST OF TECH AT WEIHAI
- Filing Date
- 2026-02-09
- Publication Date
- 2026-05-12
AI Technical Summary
Existing indoor agricultural fruit picking technologies face problems such as unstable positioning, uncontrollable open-loop operation, unintelligent decision-making, and untraceable operation in complex environments, resulting in low picking success rate, poor stability, and inability to achieve unattended continuous operation.
A closed-loop servo fruit stem shearing system is adopted. The system acquires fruit and stem information through a visual recognition module and combines temperature and humidity parameters from an environmental monitoring module to achieve shearing point generation, visual servo closed-loop alignment, shearing status determination and error correction, and environmental and maturity fusion scheduling.
It improved the success rate of harvesting, reduced the rate of incorrect cutting, enhanced harvesting efficiency and traceability, and strengthened the reliability and continuity of unattended operations.
Smart Images
Figure CN122004045A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of agricultural intelligent equipment and agricultural robot technology, specifically relating to a closed-loop servo fruit stalk cutting and harvesting system and its closed-loop harvesting method. Background Technology
[0002] With the large-scale development of facility agriculture, plant factories, and greenhouse cultivation, the requirements for operational efficiency, stability, and traceability in fruit and vegetable harvesting are constantly increasing. Traditional fruit harvesting mainly relies on manual labor, which suffers from high labor intensity, efficiency significantly affected by personnel skill levels, seasonal labor shortages, poor operational consistency, and high management costs. Against this backdrop, automated harvesting equipment based on mobile platforms, machine vision, and end effectors has gradually become a focus of research and industry attention.
[0003] Existing indoor agricultural harvesting equipment typically uses cameras to acquire images, identifies fruit locations through target detection or traditional image processing, and drives a moving platform and robotic arm / gripper / cutter to complete the harvesting. However, in real-world indoor agricultural operations, the spatial distribution of fruits and imaging conditions are significantly complex: fruits are often obscured, overlapping, or partially obscured by branches and leaves; greenhouse environments present challenges such as backlighting, reflections, localized shadows, and changes in light intensity over time; furthermore, fruit skins are fragile, fruit stalks vary greatly in strength, and plant branches exhibit significant flexibility and swaying. These factors lead to insufficient stability in the "detection-alignment-harvesting" process during engineering applications.
[0004] Most existing technologies only output two-dimensional information such as the center and bounding box dimensions of the fruit detection frame, and use this information to estimate the target position and perform proximity and shearing. Under conditions of leaf occlusion, fruit overlap, and changes in lighting, the detection frame is prone to jitter, drift, or false detections, causing the target position to be unstable with frame changes. More importantly, the fruit detection frame cannot directly characterize the spatial orientation and shearable position of the fruit stalk. If open-loop alignment is performed solely based on the fruit frame coordinates, it is easy to cause the shearing point to deviate from the fruit stalk, damage the fruit peel, or accidentally prune branches and leaves, making it difficult to guarantee the success rate and consistency of harvesting.
[0005] Existing technologies mostly employ open-loop strategies: after identifying a target once, they execute movement and shearing actions, lacking online judgment and feedback control for "whether alignment is correct," "whether a cut has occurred," and "whether an incorrect cut has occurred." When there are changes in fruit stalk thickness, interference from branches and leaves, or plant swaying, abnormalities such as incomplete shearing, missed shearing, jamming, and plant pulling often occur. After these abnormalities occur, manual intervention to reset or reposition is often required, leading to operation interruptions and failing to meet the engineering requirements of unattended, continuous, and automated harvesting in indoor agriculture. Therefore, the lack of a closed-loop mechanism of "alignment-cutting-judgment-error correction" is a significant reason restricting the practical application of existing solutions.
[0006] Indoor agriculture exhibits significant spatial microenvironmental differences. Temperature, humidity, ventilation, and light conditions may vary in different areas within the same greenhouse / corridor, thus affecting the fruit's ripening speed and uniformity. Existing harvesting methods typically determine the harvesting targets and order based solely on a single visual recognition result, lacking a decision-making mechanism that integrates environmental information such as temperature and humidity with ripeness assessments. This leads to unreasonable harvesting path planning, repetitive back-and-forth trips, and low harvesting volume per unit time. Furthermore, it is difficult to create process records linked to fruit quality, hindering subsequent quality traceability, refined planting management, and data-driven closed-loop optimization.
[0007] In summary, existing indoor agricultural fruit harvesting technologies still face problems such as "unstable positioning, uncontrollable open-loop operation, unintelligent decision-making, and untraceable operation" in complex environments. To improve the success rate, stability, and continuous unattended operation capability of harvesting, there is an urgent need to provide a system and its working method for automated indoor agricultural fruit harvesting that can achieve precise generation of cutting points, visual servo closed-loop alignment, cutting status determination and anomaly correction, and integrate microenvironment and maturity scheduling. Summary of the Invention
[0008] This invention provides a closed-loop servo fruit stem shearing and harvesting system and its closed-loop harvesting method to solve problems such as unstable positioning in water accumulation, uncontrollable open-loop operation, unintelligent decision-making, and untraceable operation.
[0009] This invention is achieved through the following technical solution: The beneficial effects of this invention are: This invention maintains stable positioning under occlusion and reflection conditions by "cutting point generation + multi-view fusion", reducing the false cut rate by 48%.
[0010] This invention achieves millimeter-level alignment consistency at the end point through visual servo closed-loop alignment, improving the harvesting success rate by 32%.
[0011] This invention achieves self-judgment and error correction of shearing completion through current criterion, thereby improving the reliability of unattended continuous operation.
[0012] This invention improves the harvest yield per unit time and enhances traceability by integrating environment and maturity levels and using multi-objective path scheduling. Attached Figure Description
[0013] Figure 1 This is a flowchart of the method of the present invention.
[0014] Figure 2 This is the logic diagram for determining the shear state based on current feedback in this invention. Detailed Implementation
[0015] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.
[0016] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.
[0017] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.
[0018] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0019] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.
[0020] Implementation Method 1 This embodiment provides a closed-loop servo fruit stem shearing and harvesting control system, the harvesting system including a visual recognition module, an environmental monitoring module and a core control system; The visual recognition module is used to acquire information related to the target fruit and its stem, and output the shearing point and uncertainty / confidence. The environmental monitoring module is used to collect parameters such as temperature and humidity to form spatial distribution characteristics; The core control system is used to perform shear point generation, visual servo closed-loop alignment, shear state judgment and error correction, and scheduling strategies based on maturity and environment integration.
[0021] Implementation Method 2 This embodiment provides a closed-loop harvesting method for a closed-loop servo stalk shearing and harvesting system, such as... Figure 1 As shown, the closed-loop harvesting method uses a closed-loop servo stem shearing harvesting system as described in Embodiment 1. The closed-loop harvesting method includes the following steps: Step 1: Target perception and candidate generation; Step 2: Construct a shear point generation model based on the candidate generation in Step 1; Step 3: Based on the shearing points formed in Step 2, use visual servo closed-loop alignment; Step 4: Perform shearing based on the alignment in Step 3; Step 5: Determine the cutting status in Step 4. If the determination is not completed, perform a second cut, revert to realign, or abandon and record the result. Step 6: Integrate environment and maturity level and schedule updates.
[0022] Furthermore, step 1 specifically involves acquiring images and depth information to obtain a set of candidate fruit targets and their confidence levels.
[0023] The visual recognition module uses a target detection and instance segmentation network to output key point information of the fruit region and pedicel, so as to obtain the pedicel direction and the coordinates of the cutting point.
[0024] First, target detection and depth acquisition are performed on the input image frames. By reasoning, we can obtain the first... Detection box for each fruit Instance Mask With confidence level Depth maps are obtained based on stereo / depth cameras. , pixel coordinates Mapped to 3D points in the camera coordinate system:
[0025] in, This is the camera intrinsic parameter matrix.
[0026] Furthermore, step 2 specifically involves estimating the fruit stalk orientation vector for the first... i The goal is to obtain three-dimensional points at the base of the fruit stalk. Three-dimensional point of the fruit center unit vector of fruit stalk direction for:
[0027] in, To prevent division by zero by extremely small constants; simultaneously, a shearing point is generated, which is not equivalent to the root of the fruit stalk, but rather offset by an adaptive distance along the direction of the fruit stalk. shear point Defined as a point offset along the direction of the fruit stalk:
[0028] Among them, the bias amount The settings are adaptively configured based on fruit size and variety parameters. This can be achieved using the masking area or estimated fruit diameter. The resulting function:
[0029] in, , , and By setting or calibrating parameters, the above-mentioned shearing point generation mechanism can significantly reduce the risk of accidental pruning of branches and abrasion of fruit peel.
[0030] Finally, a robust multi-view verification strategy for occlusion is implemented. When the target confidence is low or the variance of the shear point estimation is large, the chassis is controlled to perform a small lateral movement to obtain a second view. The two estimated shear points are then weighted and fused:
[0031] The weights can be chosen from: or ( To estimate uncertainty, thus maintaining stable positioning under conditions of occlusion and reflection.
[0032] By generating the above-mentioned cutting points, the technical problem of "not being able to determine the cutting position with only the fruit frame coordinates" is solved.
[0033] Furthermore, to ensure precise alignment between the shearing mechanism and the shearing point, this invention employs a two-stage control system combining coarse positioning and fine alignment. Coarse positioning involves the chassis reaching the vicinity of the target based on the global coordinates of the shearing point; the fine alignment stage enters a visual servo closed loop, performing omnidirectional and vertical fine adjustments based on image errors.
[0034] Step 3 specifically involves setting the shear point. Projected onto image coordinates ( , The projected coordinates of the center of the scissor guide opening are ( , Define image error:
[0035] The control system generates the chassis omnidirectional speed and lifting speed based on the error (the example uses proportional control, which can be extended to PI / model predictive control):
[0036] in For equivalent focal length, To control the gain. The system satisfies And target confidence level The shearing action is triggered in real time. This closed-loop mechanism significantly improves the success rate and consistency of alignment compared to open-loop alignment, ensuring alignment stability.
[0037] Furthermore, to avoid problems such as "uncutting or pulling the plant" or "empty cutting," this invention utilizes servo motor current estimation to achieve online determination of the shearing process. Specifically, step 5 involves establishing a shearing completion criterion and collecting servo motor current. Construct window energy index :
[0038] in, The sliding window length is the number of sampling points used to calculate the energy index. The sample number within the sliding window. For the current sampling time, The current sampling period is the time interval between two adjacent samples. For the servo motor The current sampling value at that time is used to characterize the load change during the shearing process; When a contact peak occurs and then falls back to the threshold within the time window, the following condition is met:
[0039] And continue The above indicates that the cutting is complete; among which... These are threshold and time-series parameters, which can be calibrated according to product category.
[0040] Furthermore, the anomaly correction strategy, if within the maximum shear time... If the completion criteria are not met, an error correction action is triggered: "micro-revert + repositioning + secondary trimming" is performed; or if the target confidence level decreases, the target is abandoned and the reason for failure is recorded, and the process moves on to the next target. This mechanism ensures the reliability and safety of unattended continuous operation.
[0041] Furthermore, to improve harvesting efficiency and traceability, this invention integrates environmental data such as temperature and humidity from multiple points with visual maturity assessment to construct harvesting priorities and path planning objectives. Specifically, step 6 involves designing a maturity fusion score and a visual maturity probability... With environmental feature vectors (Temperature, humidity, and their rate of change) are mapped to obtain the environmental maturity prior. The integration yielded a comprehensive score. :
[0042] in It can be calibrated or updated adaptively.
[0043] Path scheduling objective function for candidate target set Plan access sequence With the goals of "prioritizing high maturity, low migration cost, and low risk," a scheduling objective function is constructed based on scoring and migration cost. :
[0044] in, As the weight of the maturity item, The target index indicates the i-th candidate target in the sequence π. The weight of the move cost penalty term, The cost of moving to reach the i-th target. As the weight of risk penalty items, For the risk of occlusion (estimated by confidence level or uncertainty); By maximizing By obtaining the harvesting order and route, efficiency can be improved and risks reduced, and each harvesting event can be traced and recorded. And upload it for quality traceability and management decision-making.
[0045] Implementation Method 3 This embodiment provides a closed-loop servo fruit stalk shearing and harvesting control system as described in Embodiment 1, which is applied in facility agriculture, plant factories, and greenhouse cultivation.
[0046] Implementation Method 4 The memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the methods described in this invention is intended to include, but is not limited to, these and any other suitable types of memory.
Claims
1. A closed-loop servo fruit stalk shearing and harvesting control system, characterized in that, The harvesting system includes a visual recognition module, an environmental monitoring module, and a core control system; The visual recognition module is used to acquire information related to the target fruit and its stem, and output the shearing point and uncertainty / confidence. The environmental monitoring module is used to collect parameters such as temperature and humidity to form spatial distribution characteristics; The core control system is used to perform shear point generation, visual servo closed-loop alignment, shear state judgment and error correction, and scheduling strategies based on maturity and environment integration.
2. A closed-loop harvesting method for a closed-loop servo fruit stalk shearing and harvesting system, characterized in that, The closed-loop harvesting method uses a closed-loop servo stem shearing and harvesting system as described in claim 1, and the closed-loop harvesting method includes the following steps: Step 1: Target perception and candidate generation; Step 2: Construct a shear point generation model based on the candidate generation in Step 1; Step 3: Based on the shearing points formed in Step 2, use visual servo closed-loop alignment; Step 4: Perform shearing based on the alignment in Step 3; Step 5: Determine the cutting status in Step 4. If the determination is not completed, perform a second cut, revert to realign, or abandon and record the result. Step 6: Integrate environment and maturity level and schedule updates.
3. The closed-loop harvesting method according to claim 2, characterized in that, Step 1 specifically involves acquiring images and depth information to obtain a set of candidate fruit targets and their confidence levels.
4. The closed-loop harvesting method according to claim 2, characterized in that, Specifically, step 2 involves... i The goal is to obtain three-dimensional points at the base of the fruit stalk. Three-dimensional point of the fruit center unit vector of fruit stalk direction for: Among them, the shear point Defined as a point offset along the direction of the fruit stalk: Among them, the bias amount According to fruit size Adaptive determination: in, , , and These are preset or calibration parameters.
5. The closed-loop harvesting method according to claim 3, characterized in that, Step 3 specifically involves projecting the shear point onto image coordinates ( , The projected coordinates of the center of the scissor guide opening are ( , Define image error: The control system generates the chassis omnidirectional speed and lifting speed based on the error: When satisfied And visual confidence Time-triggered shearing ensures alignment stability.
6. The closed-loop harvesting method according to claim 1, characterized in that, Step 5 specifically involves collecting the servo motor current. Construct window energy index : When a contact peak occurs and then falls back to the threshold within the time window, the following condition is met: And continue If the above conditions are met, the cut is considered complete; if the criteria are not met within T_{max}, an error correction strategy is triggered.
7. The closed-loop harvesting method according to claim 6, characterized in that, The triggering error correction strategy is as follows: if the maximum shearing time is... If the completion criteria are not met, an error correction action is triggered: "micro-backtracking + repositioning + secondary trimming" is executed; or if the target confidence decreases, the target is abandoned and the reason for failure is recorded, and the next target is moved on.
8. The closed-loop harvesting method according to claim 7, characterized in that, Step 6 specifically involves visual maturity probability. Prior to the environment The integration yielded a comprehensive score. : A scheduling objective function is constructed based on score and movement cost. : By maximizing By determining the harvesting order and route, efficiency can be improved and risks reduced.
9. A closed-loop servo fruit stem shearing and harvesting control system as described in claim 1, characterized in that, The closed-loop servo fruit stalk shearing and harvesting control system is used in facility agriculture, plant factories, and greenhouse cultivation.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 2-8.