Automatic pruning device, method and robot for greenhouse tomatoes

Abstract

The invention discloses an automatic pruning device, method and robot for greenhouse tomatoes, and relates to the field of pruning. The device comprises a robot and an upper computer, wherein the robot comprises a push-broom camera, a tail end camera, a tail end actuator, a mechanical arm, a movable chassis, a lifting mechanism and a control system; the tail end camera and the tail end actuator are both arranged at the tail end of the mechanical arm. In the process that the mobile chassis drives the robot to move along the tomato row in the greenhouse, the position of the tomato plant without pruning and the position of the stem are identified by using the RGB image and the depth image of the tomato row collected by the push-broom camera in real time; the mechanical arm moves from bottom to top along the identified main stem position, meanwhile, the RGB image and the depth image, collected by the tail end camera in real time, of the unpruned tomato plant are used for identifying lateral buds, and cutting points are obtained; the end effector moves to a cutting point, and lateral buds are cut off; and when the terminal bud is identified, marking the tomato plant with a pruned branch. According to the invention, high-precision automatic tracking pruning can be carried out on tomato plants in a greenhouse.

Description

Technical Field

[0001] This application relates to the field of pruning, and in particular to an automatic pruning device, method and robot for greenhouse tomatoes. Background Technology

[0002] In tomato production in greenhouses and other facility agriculture, pruning (removing lateral buds) is a crucial management practice. Timely removal of lateral buds prevents excessive nutrient dissipation from the tomato plant, thereby improving yield and fruit quality. However, currently, the removal of tomato lateral buds largely relies on manual labor, which presents the following problems: Tomatoes require multiple pruning and side-bud removal operations throughout their growth period. Manual labor is required for extended periods in the high-temperature and high-humidity environment of greenhouses, resulting in high labor intensity, limited efficiency, and inconsistent work quality. The quality of manual side-bud removal often depends on the worker's experience and skill level, easily leading to incomplete removal of lateral buds or accidental damage to the main stem, affecting normal tomato growth and economic benefits. Due to changes in the rural labor force structure and a decrease in young workers, the reliance on manual lateral bud removal faces challenges in terms of both sustainability and economic viability.

[0003] To address these challenges, some research and equipment have attempted to prune tomato plants using mechanized or semi-automated methods. However, these solutions still have many shortcomings in terms of accuracy, stability, and adaptability to greenhouse environments. Simple mechanical devices cannot accurately locate lateral buds at different growth stages of tomatoes, requiring pre-fixed heights or positions, which limits flexibility and versatility. Some systems rely on single-machine vision or simple sensors, resulting in slow image acquisition speeds and sensitivity to ambient lighting and occlusion, leading to low recognition accuracy. Frequent manual adjustments to the shooting position and posture are required, making true automated patrol operations impossible. Furthermore, applying deep learning models (such as object detection and image segmentation) to locate tomato lateral buds often involves steps such as camera pose estimation and 3D spatial position inference. If the acquisition angle is limited or the error is large, it will lead to problems such as low accuracy in subsequent pose estimation, slow algorithm training and inference speeds, and difficulty in coping with the changing growth environment and constantly changing plant height in greenhouses. Summary of the Invention

[0004] The purpose of this application is to provide an automatic pruning device, method, and robot for greenhouse tomatoes, which can perform high-precision automatic tracking and pruning of tomato plants in greenhouses.

[0005] To achieve the above objectives, this application provides the following solution.

[0006] In the first aspect, this application provides an automatic pruning device for greenhouse tomatoes, including: a robot and a host computer; The robot includes: a push-broom camera, an end-effector camera, an end effector, a robotic arm, a mobile chassis, a lifting mechanism, and a control system; the end-effector camera and end effector are both located at the end of the robotic arm. As the mobile chassis moves the robot along the rows of tomatoes in the greenhouse, the push-sweep camera collects RGB and depth images of the tomato rows in real time. The host computer is used to identify the location of the unequaled tomato plants and the position of the main stem based on the real-time collected RGB and depth images of the tomato rows. The control system is used to drive the mobile chassis according to the location of the unequaled tomato plants, so that the robot moves in front of the unequaled tomato plants and controls the lifting mechanism to raise the robotic arm to the preset height. The control system is also used to control the robotic arm to move from bottom to top along the identified main stem position. During the movement, the end-effector camera is controlled to acquire RGB and depth images of the unequal tomato plant in real time. The host computer also uses an image segmentation algorithm to detect the main stem, lateral buds and apical buds in the real-time acquired RGB images of the unequal tomato plant. When a lateral bud is identified, the control system controls the robotic arm to drive the end-effector camera to acquire RGB and depth images of the lateral bud at different angles. Based on the RGB and depth images of the lateral bud at different angles, the host computer uses multi-frame image mask segmentation and RGB-D depth information fusion to obtain the three-dimensional coordinates of the lateral bud, and determines the cutting point based on the three-dimensional coordinates of the lateral bud. The control system is also used to move the end effector to the cutting point by controlling the robotic arm to remove side shoots; The host computer is also used to mark tomato plants as pruned when it identifies terminal buds in RGB images of undone tomato plants acquired in real time.

[0007] Secondly, this application provides an automatic pruning robot for greenhouse tomatoes, comprising: the automatic pruning robot for greenhouse tomatoes is the robot in the aforementioned automatic pruning device for greenhouse tomatoes.

[0008] Thirdly, this application provides an automatic pruning method for greenhouse tomatoes, wherein the automatic pruning method for greenhouse tomatoes utilizes the aforementioned automatic pruning device for greenhouse tomatoes, and the automatic pruning method for greenhouse tomatoes includes: The robot automatically finds its way to the row of tomatoes to be processed and moves in a straight line along the row, while simultaneously acquiring RGB and depth images of the row of tomatoes to be processed in real time. Based on the real-time acquisition of RGB and depth images of tomato rows, the location of unpruned tomato plants and the location of the main stem are identified; When an unpruned tomato plant is detected, the robotic arm is controlled to move from bottom to top along the detected main stem position, while simultaneously acquiring RGB and depth images of the unpruned tomato plant during the movement. An image segmentation algorithm was used to detect the main stem, lateral buds, and terminal buds in RGB images of unpruned tomato plants acquired in real time. When a lateral bud is detected, RGB and depth images of the lateral bud are acquired at different angles. Based on the RGB and depth images of the lateral buds at different angles, the three-dimensional coordinates of the lateral buds are obtained by multi-frame image mask segmentation and RGB-D depth information fusion. The cutting point is determined based on the three-dimensional coordinates of the lateral bud, and the end effector is moved to the cutting point to remove the lateral bud; When the terminal bud is identified, the tomato plant is marked as having been pruned.

[0009] According to the specific embodiments provided in this application, this application has the following technical effects.

[0010] This application provides an automatic pruning device, method, and robot for greenhouse tomatoes. It enables continuous operation of multiple rows of tomato plants through a moving chassis. For each unpruned tomato plant, the lifting mechanism and robotic arm can flexibly adjust the working height to achieve automatic tracking and pruning of tomato plants of different heights and growth stages in the greenhouse. By using multi-frame image mask segmentation and RGB-D depth information fusion, it can obtain the precise three-dimensional coordinates of lateral buds, making the robotic arm more accurate in positioning the cutting point, thereby completing the automatic pruning of lateral buds with high precision. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of the robot's workflow in an automatic greenhouse tomato pruning device provided in an embodiment of this application.

[0013] Figure 2 This is a schematic diagram of the image recognition result of the pushbroom camera provided in an embodiment of this application.

[0014] Figure 3 This is a schematic diagram of the image recognition result of the main stem climbing provided in an embodiment of this application.

[0015] Figure 4 This is a schematic diagram of apical bud image recognition provided in an embodiment of this application.

[0016] Figure 5 This is a schematic diagram simulating the movement of a robotic arm along the main stem, provided in an embodiment of this application.

[0017] Figure 6 This is a schematic diagram simulating the perspective of the end camera provided in an embodiment of this application.

[0018] Figure 7 This is a schematic diagram of depth image alignment provided in an embodiment of this application.

[0019] Figure 8 This is a flowchart illustrating an automatic pruning method for greenhouse tomatoes provided in an embodiment of this application. Detailed Implementation

[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] In one exemplary embodiment, an automatic pruning device for greenhouse tomatoes is provided, comprising: a robot and a host computer. The robot includes: a push-broom camera, an end effector, an end effector, a robotic arm, a mobile chassis, a lifting mechanism, and a control system; the end effector and the end effector are both located at the end of the robotic arm.

[0023] As the mobile chassis moves the robot along the rows of tomatoes in the greenhouse, the push-sweep camera collects RGB and depth images of the tomato rows in real time. The host computer identifies the location of the unequaled tomato plants and the position of the main stem based on the real-time collected RGB and depth images of the tomato rows. The control system drives the mobile chassis according to the location of the unequaled tomato plants, moves the robot to the front of the unequaled tomato plants, and controls the lifting mechanism to raise the robotic arm to a preset height.

[0024] The control system also controls the robotic arm to move upwards along the identified main stem position. During the movement, the end effector simultaneously captures real-time RGB and depth images of the unerected tomato plant using a camera. The host computer uses an image segmentation algorithm to detect the main stem, lateral buds, and terminal buds in the real-time captured RGB images of the unerected tomato plant. When a lateral bud is detected, the control system controls the robotic arm to move the end effector to capture RGB and depth images of the lateral bud at different angles. Based on these images, the host computer uses multi-frame image masking and RGB-D depth information fusion to obtain the lateral bud's three-dimensional coordinates and determines the cutting point. The control system also controls the robotic arm to move the end effector to the cutting point to remove the lateral bud.

[0025] The host computer is also used to mark tomato plants as pruned when it identifies terminal buds in RGB images of undone tomato plants acquired in real time.

[0026] The workflow of the robot in the automatic pruning device for greenhouse tomatoes is as follows: Figure 1 As shown, the working process of the automatic greenhouse tomato pruning device of this application is summarized as follows.

[0027] ① Moving Push-Sweep Mode: The push-sweep camera operates to identify and locate "unpruned" tomato plants. The host computer can also identify lateral buds from the RGB and depth images of the tomato rows, with the identification results as follows: Figure 2 As shown.

[0028] ② Main stem climbing mode: The robotic arm drives the end-effector camera to climb along the main stem, acquiring images to identify the central plant. The main stem climbing image recognition results are as follows: Figure 3 As shown.

[0029] ③ Lateral bud search and sorting: The program iterates through the identification results and determines the order of lateral buds to be processed from bottom to top.

[0030] ④ Robotic arm motion planning: Call the inverse kinematics (IK) algorithm to convert the three-dimensional coordinates of the lateral bud into the angles of each joint of the robotic arm.

[0031] ⑤ End effector pruning: After the robotic arm is in position, a small shearing blade quickly prunes the lateral buds; at the same time, force sensors or visual monitoring prevent accidental damage to the main stem.

[0032] ⑥ Marking and Cycling: Once all lateral buds have been processed and the terminal bud has been identified, the "Completed" or "Pruned" marker for the plant is updated, and the robot moves on to the next tomato plant to repeat the pruning process. Terminal bud image recognition is as follows: Figure 4 As shown.

[0033] This application provides a complete and feasible solution for the efficient removal of tomato lateral buds by linking two RGB-D (color + depth) cameras with the end effector of a robotic arm. Two RGB-D cameras are used to locate the main stem of the tomato plant and identify lateral buds from multiple perspectives. The robot utilizes a self-propelled chassis to perform continuous operation on multiple plants. Based on target detection and 3D positioning algorithms, it can quickly identify the main stem, lateral buds, and terminal buds of the tomato plant in a greenhouse environment, thereby completing the automatic pruning of lateral buds with high precision, reducing manual intervention and significantly improving work efficiency and pruning quality.

[0034] As an optional implementation method, in obtaining the three-dimensional coordinates of the lateral bud by using multi-frame image mask segmentation and RGB-D depth information fusion based on the RGB and depth images of the lateral bud at different angles, the host computer includes the following steps 1 to 4.

[0035] Step 1: Use an image segmentation algorithm to detect lateral buds from the RGB image at each angle and output them as target masks to obtain the segmentation mask of the lateral buds at each angle.

[0036] In one example, the image segmentation algorithm can use YOLOv11 as the main detection algorithm, or it can use other deep learning models (such as Faster-RCNN, Mask-RCNN, etc.) or traditional machine vision algorithms (such as Hough transform to detect stems), as long as it can perform fast instance mask segmentation of lateral buds.

[0037] Step 2: Based on the segmentation mask of the lateral bud at each angle and the depth image of the lateral bud at each angle, filter the local 3D surface point cloud of the lateral bud in the camera coordinate system at each angle.

[0038] For example, the filtering process for local 3D surface point clouds is as follows: Based on the depth images of the lateral buds at each angle, the formula is used. This allows us to obtain the three-dimensional coordinates of the lateral bud segmentation mask in the camera coordinate system at each angle.

[0039] In the formula, The three-dimensional coordinates of the lateral bud segmentation mask in the camera coordinate system. , , These represent the coordinates of the lateral bud segmentation mask on the x, y, and z axes in the three-dimensional coordinate system of the camera coordinate system. The effective pixels of the segmentation mask for lateral buds. These are the x and y coordinates of the effective pixels in the segmentation mask of the lateral bud, respectively. The values ​​are the original measurements from the depth sensor (unit: meters). For neighborhood depth compensation, , The original depth sensor measurement value of the i-th neighboring point is... For the number of neighboring points, , These are the u-axis and v-axis components of the camera focal length, respectively. , These are the x and y coordinates of the camera's optical center, respectively. Used to eliminate surface roughness noise. , , , All units are pixels.

[0040] Based on the three-dimensional coordinates of the lateral bud segmentation mask in the camera coordinate system at each angle, the formula is used... Filter the local 3D surface point cloud of the lateral bud in the camera coordinate system at each angle.

[0041] In the formula, This represents the local 3D surface point cloud of the lateral bud in the camera coordinate system. It is a binary mask. , These are the minimum and maximum effective working distances, respectively.

[0042] Step 3: Convert the local 3D surface point cloud of the lateral bud at each angle from the camera coordinate system to the world coordinate system to obtain the local 3D surface point cloud of the lateral bud at each angle in the world coordinate system.

[0043] The formula used to transform the local 3D surface point cloud of the lateral bud at each angle from the camera coordinate system to the world coordinate system is: ; ; ; ; In the formula, This represents the local 3D surface point cloud of the lateral bud in the world coordinate system. This represents the local 3D surface point cloud of the lateral bud in the camera coordinate system. Let be the homogeneous transformation matrix from the robot arm's base coordinate system to the world coordinate system. Let be the homogeneous transformation matrix from the end effector coordinate system to the robot arm base coordinate system. For the joint angle of the robotic arm, This is the homogeneous transformation matrix from the camera coordinate system to the end effector coordinate system; This is the rotation matrix from the camera coordinate system to the end effector coordinate system (obtained through hand-eye calibration). Let be the translation vector from the camera coordinate system to the end coordinate system. For the first Transformation matrices of each robotic arm joint For the first Each robotic arm joint angle The number of joints in the robotic arm. Let be the rotation matrix from the robot arm's base coordinate system to the world coordinate system. This represents the translation of the base origin in the world coordinate system.

[0044] Step 4: Using the Kalman filter method, multi-frame fusion correction is performed on the local 3D surface point cloud of the lateral bud in the world coordinate system at different angles to obtain the final corrected coordinates of the lateral bud, and these coordinates are determined as the 3D coordinates of the lateral bud.

[0045] Multi-frame fusion correction uses Kalman filtering to fuse multiple unstable visual detection results into a stable and accurate optimal position estimate.

[0046] 1. Build a dynamic model (including its position and velocity) for each lateral bud. Before each frame arrives, predict the most likely position of the lateral bud at the current moment based on the dynamic model.

[0047] The expression for the dynamic model is: x k =F·x k-1 +w k ; Where, x k =[x,y,z,v x ,v y ,v z ]ᵀ,x k This represents the current state vector (position + velocity), where (x, y, z) represents the position, and (v... x ,v y ,v z () indicates velocity. k-1 Let w represent the state vector from the previous time step. k Represents process noise. F represents the state transition matrix, F , It is the identity matrix. It is a zero matrix. For time intervals.

[0048] 2. Once the new vision system detects the coordinates of the lateral bud, it compares them with the predicted value from the previous step (the most likely location of the lateral bud at the current moment).

[0049] 3. The Kalman filter intelligently balances the reliability of the predicted and observed values, generating an optimal estimate that combines information from both and has less bias, which serves as the final corrected coordinate for the current frame.

[0050] As an optional implementation, in determining the cutting point based on the three-dimensional coordinates of the lateral bud, the host computer includes: determining the growth point of the lateral bud on the main stem based on the three-dimensional coordinates of the lateral bud; using the growth point as the starting point, extending a preset safe distance along the growth direction of the lateral bud as the cutting point, so that the end effector and the main stem maintain a safe distance during cutting. The preset safe distance can be defined as 2cm.

[0051] As an optional implementation, the robot also includes a force sensor. The force sensor is located at the end effector of the robotic arm; it is used to measure the force on the end effector in real time during the process of the end effector pruning side buds. The control system compares the real-time measured force on the end effector with a safety force threshold; if the real-time measured force on the end effector exceeds the safety force threshold, the control system stops the end effector from operating.

[0052] During the closing process of the end effector, the force sensor continuously and rapidly reads the force data from the end effector. A critical safety force threshold is pre-learned and set through experiments. This threshold is typically significantly higher than the force required to shear soft lateral buds, but lower than the force required to cause crushing damage to the tougher main stem. Real-time force data is compared with the safety force threshold, and logical judgments are made to promptly interrupt erroneous operations.

[0053] In addition to using force sensors to monitor and avoid accidentally damaging the main stem, visual monitoring can also be performed using end cameras to avoid accidentally damaging the main stem.

[0054] As an alternative implementation, in some relatively enclosed greenhouse environments, the mobile chassis can be a self-propelled mobile chassis, a track-mounted mobile chassis, a mobile system equipped with LiDAR, or a mobile system equipped with visual SLAM (simultaneous localization and mapping).

[0055] The lifting mechanism is a hydraulic lifting mechanism. If the planting height is fixed or can directly cover the entire plant, a fixed lifting platform or an electric lifting device can be used to simplify the structure and reduce costs.

[0056] The end effector can use different types of cutting tools, such as electric cutters, ultrasonic cutters, etc., and can also be equipped with small scissors with integrated camera to ensure cutting efficiency and safety.

[0057] The more detailed working process of the automatic pruning device for greenhouse tomatoes in this application is as follows.

[0058] After startup, the robot moves in a straight line along the rows of tomatoes in the greenhouse; the push-sweep camera continuously collects images and depth information of the tomato rows during the robot's movement and sends them to the host computer for preliminary identification; the preliminary identification mainly determines the location of the "unpruned" tomato plants (when pruning is completed, the tomatoes at this location will be marked with an electronic tag that says "pruned"), and marks the approximate area of ​​the main stem.

[0059] When an "unpruned" tomato plant is detected, the control system drives the self-propelled chassis to stop in front of the tomato plant; the hydraulic lifting platform rises to a preset height to ensure that the end camera can have a relatively complete field of view of the entire tomato plant.

[0060] At this point, the end-effector camera activates, capturing clearer, closer-up images of the tomato's main stem, lateral buds, and terminal buds. The robotic arm moves upwards along the identified main stem location to obtain images suitable for subsequent 2D segmentation. A simulation of the robotic arm moving along the main stem is shown below. Figure 5 As shown.

[0061] After identifying the lateral bud, the robotic arm controls the end-effector camera to automatically adjust its angle and position within a small range, acquiring images of the lateral bud from different angles to ensure the accuracy of depth information. The acquired data (RGB images and depth information) is stored in a designated folder on the host computer for subsequent image segmentation and depth fusion processing. A simulation of the end-effector camera's perspective is shown below. Figure 6 As shown.

[0062] The image segmentation algorithm is run on the host computer or robot control computer. YOLOv11 (You Only Look Once version 11) is used to detect four types of targets in the real-time acquired RGB images: main stem, lateral bud, top bud, and branch cut. The detection results are output in the form of a precise target mask to provide a basis for subsequent pose estimation.

[0063] In the synchronously acquired depth image, each pixel contains a depth value relative to the camera coordinate system; after detecting a region of lateral buds, the target is positioned in the image coordinate system based on the camera's intrinsic and extrinsic parameters. The coordinates are converted to (X, Y, Z) in the world coordinate system; this yields the three-dimensional pose of the lateral bud, providing precise coordinates for the robotic arm to perform lateral bud pruning. Figure 7 This is a schematic diagram of depth image alignment.

[0064] Pixel coordinate system → Camera coordinate system (with depth compensation): Converts 2D pixels into 3D points in the camera coordinate system by combining depth information. The compensated depth... This can reduce surface undulation errors. For each effective pixel in the segmentation mask. The corresponding 3D coordinates of the camera coordinate system are: .

[0065] Based on binary mask (segmentation mask) Filter out the local 3D surface point cloud of the target object (such as lateral branches, main stem, etc.): ; in, , Effective working distance, eliminating background interference.

[0066] Camera coordinate system → End effector coordinate system (hand-eye calibration): Transform the point cloud from the camera coordinate system to the end effector coordinate system to reflect the fixed installation relationship between the camera and the end effector (the camera is mounted above the gripper). The formula is: ; in, This indicates the fixed calibration relationship from the camera to the end point. The three-dimensional coordinates of the lateral bud segmentation mask in the end effector coordinate system. , , These are the coordinates of the lateral bud segmentation mask in the three-dimensional coordinates of the end effector coordinate system on the x-axis, y-axis, and z-axis, respectively.

[0067] End effector coordinate system → Robot arm base coordinate system (forward kinematics): Calculate the position and orientation of the robot arm's end effector relative to the base coordinate system based on real-time joint angles. When the robot arm joint angle is... At that time, the end-effector pose is: ; in, The first one , No. Each robotic arm joint angle. Calculated from the forward kinematics of the robotic arm. The three-dimensional coordinates of the lateral bud segmentation mask in the robot arm's base coordinate system are given. , , These are the coordinates of the lateral bud segmentation mask on the x-axis, y-axis, and z-axis in the three-dimensional coordinate system of the robotic arm's base coordinate system.

[0068] Robotic arm base coordinate system → World coordinate system: Calculate the position of the point in the global coordinate system based on the robot's movement in the global map. ; in, It indicates the position of the base in the world. The three-dimensional coordinates of the lateral bud segmentation mask in the world coordinate system. , , These are the coordinates of the lateral bud segmentation mask on the x-axis, y-axis, and z-axis in the three-dimensional coordinates of the world coordinate system.

[0069] The complete formula chain from pixel coordinates to world coordinates can be summarized as follows: .

[0070] Spatial filtering or multi-frame fusion correction is performed on the coordinates of the identified lateral buds to filter out noise or errors caused by leaf swaying, etc.; secondary detection and depth reconfirmation are performed on targets that may have overlapping edges to reduce false detections and missed detections.

[0071] After the lateral buds are identified and located, the application enables automated removal by a robotic arm and end effector.

[0072] Compared with existing manual removal or simple semi-automatic equipment, this application has the following technical effects and advantages.

[0073] 1. Improve pruning efficiency and reduce labor costs: Continuous operation is achieved through a linear self-propelled chassis, coupled with automatic computer detection and control, enabling the rapid removal of lateral buds from one or more rows of tomato plants. This significantly reduces the number of workers required for the same work area.

[0074] 2. Stable and high-precision pruning quality: A method is proposed to obtain the precise three-dimensional coordinates of lateral buds by multi-frame image mask segmentation and RGB-D depth information fusion, which makes the robotic arm positioning more accurate and the end effector pruning point more reliable. This reduces operational deviations caused by human emotions, fatigue, and other factors, ensuring the consistency of pruning position and time.

[0075] 3. Covers the entire growth period of tomatoes: The working height can be flexibly adjusted through a hydraulic lifting mechanism according to the height changes of tomato plants at different growth stages, so that lateral buds from the seedling stage to the vigorous growth stage can be accurately identified and removed, making it suitable for continuous use in large-scale facility agriculture production.

[0076] 4. High degree of automation and strong adaptability: It realizes automatic tracking and pruning of tomato plants of different heights and growth stages. By modifying the algorithm parameters or the type of end effector, it can also be applied to other greenhouse crop management processes that require targeted cutting.

[0077] The above advantages mainly stem from the following technical points: 1. Fully automatic driving of the mobile self-propelled chassis and adaptive adjustment of the hydraulic lifting mechanism; 2. Image acquisition and depth information fusion from dual RGB-D cameras improve the accuracy of target detection and positioning; 3. The YOLOv11 algorithm is used to identify the main stem, lateral buds and terminal buds of tomatoes in real time, which has high accuracy and speed. 4. The robotic arm, in conjunction with the end effector, enables precise and rapid pruning of lateral buds.

[0078] In one exemplary embodiment, an automatic greenhouse tomato pruning robot is provided, which is the robot in the above-mentioned automatic greenhouse tomato pruning device.

[0079] Based on the same inventive concept, this application also provides an automatic pruning method for greenhouse tomatoes using the aforementioned automatic pruning device. The solution provided by this method is similar to the solution described in the aforementioned device; therefore, the specific limitations in one or more embodiments of the automatic pruning method for greenhouse tomatoes provided below can be found in the limitations of the automatic pruning device for greenhouse tomatoes described above, and will not be repeated here.

[0080] In one exemplary embodiment, such as Figure 8 As shown, an automatic pruning method for greenhouse tomatoes includes the following steps 101 to 108.

[0081] Step 101: The robot automatically finds its way to the row of tomatoes to be processed and moves in a straight line along the row of tomatoes to be processed, while simultaneously acquiring RGB images and depth images of the row of tomatoes to be processed in real time.

[0082] Step 102: Based on the real-time acquired RGB and depth images of the tomato rows, identify the location of the unpruned tomato plants and the location of the main stem.

[0083] Step 103: When an unpruned tomato plant is detected, control the robotic arm to move from bottom to top along the detected main stem position, and simultaneously acquire RGB and depth images of the unpruned tomato plant during the movement.

[0084] Step 104: Use an image segmentation algorithm to detect the main stem, lateral buds, and terminal buds in the RGB image of the unpruned tomato plant acquired in real time.

[0085] Step 105: When a lateral bud is identified, acquire RGB and depth images of the lateral bud at different angles.

[0086] Step 106: Based on the RGB and depth images of the lateral buds at different angles, the three-dimensional coordinates of the lateral buds are obtained by multi-frame image mask segmentation and RGB-D depth information fusion.

[0087] Step 107: Determine the cutting point based on the three-dimensional coordinates of the lateral bud, and move the end effector to the cutting point to remove the lateral bud.

[0088] Step 108: When the terminal bud is identified, mark the tomato plant as having been pruned.

[0089] As an optional implementation, moving the end effector to the cutting point includes: using an inverse kinematics algorithm to convert the coordinates of the cutting point into the angles of each joint of the robotic arm; and controlling the robotic arm to move the end effector to the cutting point according to the angles of each joint of the robotic arm.

[0090] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0091] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An automatic pruning device for greenhouse tomatoes, characterized in that, include: Robots and host computers; The robot includes: a push-broom camera, an end-effector camera, an end effector, a robotic arm, a mobile chassis, a lifting mechanism, and a control system; the end-effector camera and end effector are both located at the end of the robotic arm. As the mobile chassis moves the robot along the rows of tomatoes in the greenhouse, the push-sweep camera collects RGB and depth images of the tomato rows in real time. The host computer is used to identify the location of the unequaled tomato plants and the position of the main stem based on the real-time collected RGB and depth images of the tomato rows. The control system is used to drive the mobile chassis according to the location of the unequaled tomato plants, so that the robot moves in front of the unequaled tomato plants and controls the lifting mechanism to raise the robotic arm to the preset height. The control system is also used to control the robotic arm to move from bottom to top along the identified main stem position. During the movement, the end-effector camera is controlled to acquire RGB and depth images of the unequal tomato plant in real time. The host computer also uses an image segmentation algorithm to detect the main stem, lateral buds and apical buds in the real-time acquired RGB images of the unequal tomato plant. When a lateral bud is identified, the control system controls the robotic arm to drive the end-effector camera to acquire RGB and depth images of the lateral bud at different angles. Based on the RGB and depth images of the lateral bud at different angles, the host computer uses multi-frame image mask segmentation and RGB-D depth information fusion to obtain the three-dimensional coordinates of the lateral bud, and determines the cutting point based on the three-dimensional coordinates of the lateral bud. The control system is also used to move the end effector to the cutting point by controlling the robotic arm to remove side shoots; The host computer is also used to mark tomato plants as pruned when it identifies terminal buds in RGB images of undone tomato plants acquired in real time.

2. The automatic pruning device for greenhouse tomatoes according to claim 1, characterized in that, To obtain the three-dimensional coordinates of the lateral buds based on their RGB and depth images at different angles, using multi-frame image mask segmentation and RGB-D depth information fusion, the host computer includes: The image segmentation algorithm is used to detect lateral buds from the RGB image at each angle and output them as target masks to obtain the segmentation mask of the lateral buds at each angle. Based on the segmentation mask of the lateral bud at each angle and the depth image of the lateral bud at each angle, filter the local 3D surface point cloud of the lateral bud in the camera coordinate system at each angle. The local 3D surface point cloud of the lateral bud at each angle is transformed from the camera coordinate system to the world coordinate system to obtain the local 3D surface point cloud of the lateral bud at each angle in the world coordinate system. The Kalman filter method is used to perform multi-frame fusion correction on the local three-dimensional surface point cloud of the lateral bud in the world coordinate system at different angles to obtain the final corrected coordinates of the lateral bud, which are then determined as the three-dimensional coordinates of the lateral bud.

3. The automatic pruning device for greenhouse tomatoes according to claim 2, characterized in that, Based on the segmentation mask of the lateral bud at each angle and the depth image of the lateral bud at each angle, the local 3D surface point cloud of the lateral bud in the camera coordinate system at each angle is filtered, including: Based on the depth images of the lateral buds at each angle, the formula is used. The three-dimensional coordinates of the lateral bud segmentation mask in the camera coordinate system at each angle are obtained; where, The three-dimensional coordinates of the lateral bud segmentation mask in the camera coordinate system. , , These represent the coordinates of the lateral bud segmentation mask on the x, y, and z axes in the three-dimensional coordinate system of the camera coordinate system. The effective pixels of the segmentation mask for lateral buds. , These are the x and y coordinates of the effective pixels in the segmentation mask of the lateral bud, respectively. These are the raw measurement values ​​from the depth sensor. For neighborhood depth compensation, , The original depth sensor measurement value of the i-th neighboring point is... For the number of neighboring points, , These are the u-axis and v-axis components of the camera focal length, respectively. , These are the x and y coordinates of the camera's optical center, respectively. Based on the three-dimensional coordinates of the lateral bud segmentation mask in the camera coordinate system at each angle, the formula is used... Filter the local 3D surface point cloud of the lateral bud in the camera coordinate system at each angle; where, This represents the local 3D surface point cloud of the lateral bud in the camera coordinate system. It is a binary mask. , These are the minimum and maximum effective working distances, respectively.

4. The automatic pruning device for greenhouse tomatoes according to claim 2, characterized in that, The formula used to transform the local 3D surface point cloud of the lateral bud at each angle from the camera coordinate system to the world coordinate system is: ； ； ； ； In the formula, This represents the local 3D surface point cloud of the lateral bud in the world coordinate system. This represents the local 3D surface point cloud of the lateral bud in the camera coordinate system. Let be the homogeneous transformation matrix from the robot arm's base coordinate system to the world coordinate system. Let be the homogeneous transformation matrix from the end effector coordinate system to the robot arm base coordinate system. For the joint angle of the robotic arm, This is the homogeneous transformation matrix from the camera coordinate system to the end effector coordinate system; Let be the rotation matrix from the camera coordinate system to the end effector coordinate system. Let be the translation vector from the camera coordinate system to the end coordinate system. For the first Transformation matrices of each robotic arm joint For the first Each robotic arm joint angle The number of joints in the robotic arm. Let be the rotation matrix from the robot arm's base coordinate system to the world coordinate system. This represents the translation of the base origin in the world coordinate system.

5. The automatic pruning device for greenhouse tomatoes according to claim 1, characterized in that, In determining the cutting point based on the three-dimensional coordinates of the lateral bud, the host computer includes: Determine the growth point of the lateral bud on the main stem based on the three-dimensional coordinates of the lateral bud; Starting from the growth point, the point extending a predetermined safe distance along the lateral bud growth direction is the cutting point.

6. The automatic pruning device for greenhouse tomatoes according to claim 1, characterized in that, The robot also includes: a force sensor; A force sensor is installed at the end of the robotic arm; Force sensors are used to measure the force on the end effector of a robotic arm in real time during the process of the end effector pruning side shoots; The control system compares the force measured in real time at the end of the robotic arm with a safety force threshold. If the force measured in real time at the end of the robotic arm is greater than the safety force threshold, the end effector is controlled to stop working.

7. The automatic pruning device for greenhouse tomatoes according to claim 1, characterized in that, The mobile chassis can be a self-propelled mobile chassis, a track-mounted mobile chassis, a mobile system equipped with lidar, or a mobile system equipped with visual SLAM. The lifting mechanism is a hydraulic lifting mechanism, a fixed lifting platform, or an electric lifting device; The end effector is a cutting tool or scissors.

8. An automatic pruning robot for greenhouse tomatoes, characterized in that, The automatic greenhouse tomato pruning robot is the robot in the automatic greenhouse tomato pruning device according to any one of claims 1-7.

9. An automatic pruning method for greenhouse tomatoes, characterized in that, The automatic pruning method for greenhouse tomatoes uses the automatic pruning device for greenhouse tomatoes according to any one of claims 1-7, and the automatic pruning method for greenhouse tomatoes includes: The robot automatically finds its way to the row of tomatoes to be processed and moves in a straight line along the row, while simultaneously acquiring RGB and depth images of the row of tomatoes to be processed in real time. Based on the real-time acquisition of RGB and depth images of tomato rows, the location of unpruned tomato plants and the location of the main stem are identified; When an unpruned tomato plant is detected, the robotic arm is controlled to move from bottom to top along the detected main stem position, while simultaneously acquiring RGB and depth images of the unpruned tomato plant during the movement. An image segmentation algorithm was used to detect the main stem, lateral buds, and terminal buds in RGB images of unpruned tomato plants acquired in real time. When a lateral bud is detected, RGB and depth images of the lateral bud are acquired at different angles. Based on the RGB and depth images of the lateral buds at different angles, the three-dimensional coordinates of the lateral buds are obtained by multi-frame image mask segmentation and RGB-D depth information fusion. The cutting point is determined based on the three-dimensional coordinates of the lateral bud, and the end effector is moved to the cutting point to remove the lateral bud; When the terminal bud is identified, the tomato plant is marked as having been pruned.

10. The automatic pruning method for greenhouse tomatoes according to claim 9, characterized in that, Moving the end effector to the cutting point includes: The inverse kinematics algorithm is used to convert the coordinates of the cutting point into the angles of each joint of the robotic arm; Based on the angles of each joint of the robotic arm, control the robotic arm to move the end effector to the cutting point.

Images