An artificial intelligence-based mechanical claw yellow leaf control method and system

Abstract

This application proposes a method and system for controlling the removal of yellow leaves by a robotic claw based on artificial intelligence, relating to the field of image processing technology. The method includes: acquiring a fused image of a target region; performing region segmentation based on the fused image to obtain a first region, a second region, and a third region; extracting the contour boundary and centerline skeleton based on the first region; determining the outward expansion normal direction based on the contour boundary, centerline skeleton, and third region and generating a fourth region; generating a first ratio based on the fourth region and the second region; determining a first evaluation result based on the first ratio; determining the target body based on the first evaluation result; and performing a peeling operation based on the target body. If the first evaluation result is a positive first evaluation result, the spatial distribution characteristics of the occluded portion formed by the intersection of the fourth region and the second region are analyzed to identify the specific obstacle type.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a control method and system for removing yellow leaves using a robotic claw based on artificial intelligence. Background Technology

[0002] In facility agriculture and intelligent horticulture, removing yellowed and diseased leaves from plants is crucial for ensuring healthy crop growth, improving ventilation and light penetration, and preventing disease spread. However, traditional manual leaf removal methods are inefficient and labor-intensive, while existing automated leaf removal systems based on machine vision and robotic arms still have significant limitations in practical applications. First, the reliance on single-response strategies (such as direct abandonment or fixed angle adjustment) stems from the lack of classification based on the spatial distribution characteristics of occlusion and the failure to comprehensively quantify obstacle types and the kinematic parameters of the robotic gripper, resulting in the inability to achieve targeted adaptive correction. Furthermore, existing buffer zone generation methods often rely solely on the gradient information of the yellow leaf's contour for normal expansion, neglecting the critical geometric guidance role of the main stem at the connection point. This causes the expansion direction of the buffer zone near the root to be inconsistent with the actual feasible approach path of the robotic gripper, thereby weakening the safety and success rate of the grasping action. Summary of the Invention

[0003] This application aims to at least partially address one of the technical problems in the related art.

[0004] To achieve the above objectives, this application proposes an artificial intelligence-based robotic gripper control method for removing yellow leaves, comprising the following steps:

[0005] Step 1: Obtain the fused image of the target region;

[0006] Step 2: Perform region segmentation based on the fused image to obtain a first region, a second region, and a third region;

[0007] Step 3: Extract the contour boundary and centerline skeleton based on the first region;

[0008] Step 4: Determine the outward expansion normal direction based on the outline boundary, centerline skeleton and third region and generate a fourth region; generate a first scale based on the fourth region and the second region; determine a first evaluation result based on the first scale; and determine the target body based on the first evaluation result.

[0009] Step 5: Perform a stripping operation based on the target body.

[0010] Furthermore, based on the contour boundary, centerline skeleton, and third region, the outward expansion normal direction is determined and a fourth region is generated. A first scale is generated based on the fourth region and the second region. A first evaluation result is determined based on the first scale, including the following steps:

[0011] Step 41: Extract the set of contour pixels based on the contour boundary;

[0012] Step 42: Determine the unit contour normal direction based on each contour pixel in the contour pixel set;

[0013] Step 43: Based on all contour pixels, translate outward by a preset distance from each contour pixel along the corresponding unit contour normal direction to obtain an offset point, and take the closed area formed by all offset points as the initial area.

[0014] Step 44: Determine the reference point based on the centerline skeleton;

[0015] Step 45: Extract edge pixels of the third region based on the neighborhood of the reference point, obtain the gradient direction based on the extracted edge pixels, take the unit direction perpendicular to the gradient direction as the tangent direction of the third region at the reference point, and obtain the unit vector perpendicular to the tangent direction as the normal direction of the third region at the reference point.

[0016] Step 46: Identify the contour segments generated in the neighborhood of the reference point in the initial region, and correct the translation direction of the current contour segment to the normal direction to obtain the corrected region;

[0017] Step 47: Preset target pixel distance. If the preset distance is less than the target pixel distance, expand the entire correction area outward by the first pixel distance to obtain the fourth area; otherwise, use the correction area as the fourth area.

[0018] Step 48: Obtain the fifth region based on the fourth region and the second region; obtain the first ratio based on the area of ​​the fifth region and the fourth region; generate the first evaluation result based on the first ratio; the first evaluation result includes a first positive evaluation result and a first negative evaluation result.

[0019] Furthermore, the pixel point in the first region corresponding to the shortest Euclidean distance from the third region is marked as a reference point.

[0020] Furthermore, the difference between the target pixel distance and the preset distance is marked as the first pixel distance.

[0021] Furthermore, when the first evaluation result is a positive evaluation result, correction is performed based on the fourth region to obtain the fourth correction region and determine the target body; when the first evaluation result is a negative evaluation result, the first region is marked as the target body.

[0022] Furthermore, correction is performed based on the fourth region to obtain the fourth correction region, including the following steps:

[0023] Step 481: Determine the first Euclidean distance between all pixels in the fifth region and the reference point, and count the number of pixels whose first Euclidean distance is less than the standard pixel.

[0024] Step 482: Obtain a second ratio based on the number of pixels and the area of ​​the fifth region; if the second ratio is greater than a preset first ratio threshold, determine that there is a first type of obstacle in the first region; otherwise, proceed to the next step.

[0025] Step 483: Project each pixel in the fifth region onto the nearest skeleton point on the centerline skeleton to obtain the set of hit skeleton points;

[0026] Step 484: Determine the second Euclidean distance between each pixel and the corresponding hit skeleton point, and mark the maximum value of the second Euclidean distance as the maximum width.

[0027] Step 485: Obtain the shortest continuous skeleton segment of the centerline skeleton containing the hit skeleton point and the corresponding path length.

[0028] Step 486: Obtain the total path length of the centerline skeleton;

[0029] Step 487: Construct a third ratio based on the path length and the total path length. If the third ratio is greater than the preset second ratio and the maximum width is greater than the target width, determine that there is a second type of obstacle in the first region; otherwise, determine that there is a third type of obstacle in the first region.

[0030] Step 488: Based on the first type of obstacle and / or the second type of obstacle and / or the third type of obstacle, the fourth region is corrected to obtain the fourth corrected region.

[0031] Furthermore, based on the first type of obstacle, the reference point is moved Q pixels along the centerline skeleton towards the leaf tip to obtain a gripping point. The first evaluation result is repeatedly determined based on the gripping point to identify the target body.

[0032] Furthermore, based on the second type of obstacle, the normal direction is rotated around the x-axis by a first angle to obtain a corrected normal direction; the normal direction used to correct the correction region is replaced with the corrected normal direction to generate a secondary correction region; the first evaluation result is re-determined based on the secondary correction region to determine the target body.

[0033] This invention also discloses an artificial intelligence-based mechanical claw control system for removing yellow leaves, comprising the following modules:

[0034] Image acquisition module: used to acquire the fused image of the target region;

[0035] Image segmentation module: used to perform region segmentation based on the fused image to obtain a first region, a second region, and a third region;

[0036] Feature extraction module: used to extract the contour boundary and centerline skeleton based on the first region;

[0037] Target localization module: used to determine the outward expansion normal direction based on the contour boundary, centerline skeleton and third region and generate a fourth region, generate a first scale based on the fourth region and the second region, determine a first evaluation result based on the first scale, and determine the target body based on the first evaluation result;

[0038] Peeling module: Used to perform peeling operations based on the target body.

[0039] Compared with existing technologies, this application provides an AI-based robotic gripper control method for removing yellow leaves. It initially determines whether a first region is suitable as a target based on a first evaluation result. If the first evaluation result is positive, it doesn't skip it directly but further analyzes the spatial distribution characteristics of the occluded portion formed by the intersection of the fourth and second regions, particularly its geometric relationship with the reference point and centerline skeleton, thereby identifying the specific obstacle type. Based on this, the fourth region undergoes targeted directional or positional correction, and the first evaluation process is re-executed to ultimately confirm whether the yellow leaf can be safely removed. This approach changes the previous passive mode of abandoning processing once occlusion is detected, instead transforming occlusion information into a basis for optimizing operational strategies. In practical applications, many yellow leaves that were initially misjudged as unoperable due to partial occlusion are successfully included in the operation range after one or more corrections, improving the effectiveness of target identification and reducing unnecessary omissions. Furthermore, since the correction process is completed during the fourth region generation stage, the robotic gripper does not need to repeatedly probe or adjust its pose, saving operation time and reducing the risk of accidental collisions with adjacent healthy tissue. Overall, it significantly enhances adaptability in complex leaf-interlacing environments, making automated leaf removal operations more reliable and efficient.

[0040] After determining the reference point, this application extracts the edge pixels of the third region within the neighborhood of that point and derives the tangent direction based on its gradient direction, thereby obtaining a unit vector perpendicular to it as the normal direction of the third region at the reference point. This reflects the true orientation of the main stem surface at the connection point and is used as the reference for the expansion direction of the corresponding local contour segment in the correction region, thus making the extension direction of the fourth region near the root more consistent with the natural structure of the plant. Furthermore, when dealing with the second type of obstacle, this normal direction is rotated around the x-axis by a first angle to generate a corrected normal direction, which is used to construct the secondary correction region, enabling the fourth region to adapt to the safety operation requirements under different approach postures. By closely combining the geometric characteristics of the main stem, the motion degrees of freedom of the mechanical claw, and the direction calculation in the image space, it is ensured that the expansion direction of the fourth region in the key connection region always remains consistent with the actual feasible grasping path. This effectively avoids root collisions caused by directional deviations, significantly improves the stability of the grasping process, increases the peeling success rate, and greatly enhances the operational safety and reliability in complex agricultural scenarios. Attached Figure Description

[0041] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0042] Figure 1 A flowchart illustrating an artificial intelligence-based mechanical gripper method for removing yellow leaves, provided in an embodiment of this application;

[0043] Figure 2 A structural diagram of an artificial intelligence-based robotic claw control system for removing yellow leaves, provided in an embodiment of this application;

[0044] Figure 3 This is a block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0045] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0046] The following describes a method for controlling the removal of yellow leaves by a robotic claw based on artificial intelligence, according to an embodiment of this application, with reference to the accompanying drawings.

[0047] It should be noted that the execution subject of the artificial intelligence-based mechanical claw yellow leaf removal control method in this application embodiment is an artificial intelligence-based mechanical claw yellow leaf removal control system in this application embodiment. This artificial intelligence-based mechanical claw yellow leaf removal control system can be configured in an electronic device so that the electronic device can perform the mechanical claw yellow leaf removal control function.

[0048] like Figure 1 As shown, this artificial intelligence-based robotic gripper for removing yellow leaves includes the following steps:

[0049] Step 1: Acquire a fused image of the target plant area using a multispectral visual sensor.

[0050] In this embodiment, a multispectral vision sensor mounted on the front end of an agricultural robot platform acquires images of the target plant area. The multispectral vision sensor simultaneously acquires visible light (RGB) and near-infrared (NIR) images, and registers and synthesizes the two sets of data into a single fused image based on a preset band fusion algorithm. The visible light image preserves the color, texture, and edge details of the leaves, while the near-infrared image highlights the physiological state of the plant tissue. Yellowed or diseased leaves, due to decreased chlorophyll content, exhibit significantly low reflectivity in the near-infrared band, thus creating a high contrast with healthy green tissue. During the fusion process, a conventional weighted linear fusion strategy is employed, assigning different weights to the RGB and NIR channels to enhance the semantic separability of the yellow leaf region. The final fused image not only retains rich spatial structural information but also strengthens the spectral differences between yellow leaves, healthy tissue, and the main stem, providing high-quality input for subsequent high-precision region segmentation.

[0051] Step 2: Perform region segmentation based on the fused image to obtain a first region, a second region, and a third region.

[0052] In this embodiment, to accurately identify and segment the various components of the target plant, the fused image obtained in step 1 is input into an optimized semantic segmentation neural network model. The aim is to classify each pixel in the fused image by assigning a specific semantic label to each pixel, dividing the pixels in the fused image into three main categories: a first region (yellow leaf region), a second region (healthy leaf region), and a third region (main stem region). The above region segmentation process is a conventional technique, and the specific process will not be described in detail.

[0053] Step 3: Extract the contour boundary and centerline skeleton based on the first region.

[0054] In this embodiment, a binary mask is first obtained based on the first region, and an opening operation is performed to remove isolated noise points. Then, a closing operation is performed to fill any small holes that may appear within the region, ensuring that the yellow leaf region is complete and connected. The contour boundary outside the region is extracted according to the 8-neighborhood connectivity principle, thereby obtaining a set of closed contour points composed of ordered pixel coordinates, which is used to depict the outer edge shape of the yellow leaf. The Zhang-Suen algorithm is used to gradually erode the edge pixels until a centerline skeleton with a single pixel width is formed.

[0055] Step 4: Determine the outward expansion normal direction based on the contour boundary, centerline skeleton and third region and generate a fourth region; generate a first scale based on the fourth region and the second region; determine a first evaluation result based on the first scale; and determine the target body based on the first evaluation result.

[0056] Based on the outline boundary, centerline skeleton, and third region, the outward expansion normal direction is determined and a fourth region is generated. A first scale is generated based on the fourth region and the second region. A first evaluation result is determined based on the first scale for evaluating operational safety, including the following steps:

[0057] Step 41: Extract the set of contour pixels based on the contour boundary.

[0058] The set of contour pixels is the closed contour point set.

[0059] Step 42: Determine the unit contour normal direction based on each contour pixel in the contour pixel set.

[0060] The Sobel operator is applied to the binary mask to calculate the gradient components in the horizontal and vertical directions, thereby obtaining the gradient field of the entire image. The gradient vector of each pixel in the first region is read from the gradient field. Since the gradient direction of the image is always perpendicular to the contour lines (i.e., the contour lines), the direction of the gradient vector is the normal direction of that contour point.

[0061] Step 43: Based on all contour pixels, translate outward by a preset distance from each contour pixel along the corresponding unit contour normal direction to obtain an offset point, and take the closed area formed by all offset points as the initial area.

[0062] First, select a preset distance value as the buffer width, for example, between 3 and 7 pixel units. This range can be adjusted according to actual needs, such as to ensure higher security or to consider specific image processing requirements. Calculate the offset point's position by translating each pixel along its normal vector direction by the preset distance value. Repeat the above process for all pixels on the contour to obtain a series of offset points. Connect all the offset points sequentially to form a closed curve, which, together with the original contour, defines the initial buffer area. This area includes the original contour and the space extending a certain distance outward, forming an extended boundary that can be used to protect yellow leaves from external factors or to mark specific processing areas.

[0063] Step 44: Determine the reference point based on the centerline skeleton.

[0064] The pixel in the first region corresponding to the shortest Euclidean distance from the third region is marked as the reference point. In plant structure, leaves are connected to the main stem through petioles, and the connection point is where the leaf is most firmly attached and the tissue is most dense. When a leaf turns yellow, its connection point with the main stem remains the only effective point for mechanical detachment. If force is applied from the middle or far end of the leaf, it is very easy to cause the leaf to tear rather than be completely detached, and the remaining petiole may cause disease; however, applying appropriate shearing or torsional force near the connection point can achieve clean and undamaged detachment.

[0065] In the image, the shortest Euclidean distance pair between the first and third regions geometrically corresponds precisely to the contact position between the petiole tip and the main stem surface. Therefore, the reference point naturally represents the projection of the actual biological connection point into the image and is the anchor point that must be aimed at during the stripping operation.

[0066] Step 45: Extract edge pixels of the third region based on the neighborhood of the reference point, obtain the gradient direction based on the extracted edge pixels, take the unit direction perpendicular to the gradient direction as the tangent direction of the third region at the reference point, and obtain the unit vector perpendicular to the tangent direction as the normal direction of the third region at the reference point.

[0067] A neighborhood window is defined centered on the reference point, such as a circular region with a radius of 5 pixels centered at the reference point or a 7×7 square window. Edge pixels belonging to the third region are extracted within this neighborhood using an edge detection algorithm. The Sobel operator is applied to the edge pixels within the neighborhood to calculate the image gradient, obtaining the gradient vector for each edge point. Principal component analysis (PCA) is then used to determine the dominant gradient direction for this local region. Since the image gradient direction is perpendicular to the edge, this dominant gradient direction is the tangent direction of the main stem at the reference point. A unit vector perpendicular to this tangent direction is taken, and its outward pointing direction is determined based on the inside-outside relationship of the main stem mask, serving as the normal direction.

[0068] Step 46: Identify the contour segments generated in the neighborhood of the reference point in the initial region, and correct the translation direction of the current contour segment to the normal direction to obtain the corrected region.

[0069] Within the neighborhood of the reference point, all contour pixels falling within this range on the boundary of the initial region are searched. These points constitute the contour segment generated from the neighborhood of the reference point. This contour segment is essentially a small outer boundary of the initial region near the root of the yellow leaf, close to the connection with the main stem. For each pixel in this contour segment, instead of offsetting outward along its original unit contour normal, it is uniformly translated by the same distance along the normal direction at the reference point, thus generating a new set of offset points. Smoothly connecting these new offset points with the remaining unaffected contour parts of the initial region forms a complete correction region. The expansion direction of the buffer region in the key root area no longer blindly follows the edge of the yellow leaf, but actively aligns with the geometric orientation of the main stem surface, making the generated safe area more consistent with the actual feasible approach path of the robotic gripper, effectively avoiding the risk of collision due to directional deviation, while improving the stability and success rate of the gripping action.

[0070] Step 47: Preset target pixel distance. If the preset distance is less than the target pixel distance, expand the entire correction area outward by the first pixel distance to obtain the fourth area; otherwise, use the correction area as the fourth area.

[0071] First, the minimum outer radius of the robotic gripper's end effector in its fully extended state is obtained and converted into a target pixel distance in the image coordinate system. In this embodiment, a preset target pixel distance is used as a reference standard, preferably 8-15 pixels. The preset distance is compared with the target pixel distance. If the preset distance is less than the target pixel distance, it means that the current correction area is insufficient to accommodate the physical envelope of the robotic gripper, posing a collision risk. In this case, the correction area needs to be expanded outward by a first pixel distance. The difference between the target pixel distance and the preset distance is marked as the first pixel distance.

[0072] Conversely, if the current correction area already meets the space requirements of the robotic gripper, no further expansion is needed. In this case, the correction area can be directly defined as the fourth area.

[0073] Throughout the process, by converting the actual size of the robotic gripper to the image coordinate system and adjusting the size of the correction area based on the comparison results, the specific size requirements of the robotic gripper are taken into account during path planning, thereby reducing the risk of collisions during operation.

[0074] Step 48: Obtain a fifth region based on the fourth and second regions; obtain a first ratio based on the areas of the fifth and fourth regions; generate a first evaluation result based on the first ratio; the first evaluation result includes a first positive evaluation result and a first negative evaluation result. When the first evaluation result is a first positive evaluation result, correction is performed based on the fourth region to obtain a fourth correction region and determine the target body; when the first evaluation result is a first negative evaluation result, the first region is marked as the target body.

[0075] The fourth region and the second region (healthy leaf region) are intersected at the pixel level to obtain their overlapping area, which is the fifth region. This overlapping area represents the occlusion area that may interfere with healthy tissue when the robotic gripper performs the grasping action. The ratio of the area of ​​the fifth region to the total area of ​​the fourth region is calculated as the first ratio. This ratio is compared with a preset first predetermined threshold (e.g., 20%): if the first ratio does not exceed the first predetermined threshold, it is determined that there is sufficient safe operating space around the current yellow leaf, a first negative evaluation result is generated, and the first region is directly marked as the target body, proceeding to the subsequent peeling operation; if the first ratio exceeds the first predetermined threshold, it is determined that there is an operational obstacle, a first positive evaluation result is generated, and the yellow leaf is not directly abandoned, but is retained and the obstacle type analysis process is triggered, that is, based on the spatial distribution characteristics of the fifth region relative to the reference point and the centerline skeleton, it is further determined whether it belongs to root occlusion type, lateral occlusion type, or diffuse occlusion type.

[0076] The fourth correction region is obtained by performing correction based on the fourth region, including the following steps:

[0077] Step 481: Determine the first Euclidean distance between all pixels in the fifth region and the reference point, and count the number of pixels whose first Euclidean distance is less than the standard pixel.

[0078] This embodiment analyzes the spatial distribution characteristics of the occluded area, obtains the coordinates of all pixels in the fifth region, and calculates the first Euclidean distance between each pixel and the reference point. A standard pixel value corresponds to a typical range near the root of the yellow leaf, used to define the spatial scale near the connection point; in this embodiment, a standard pixel value of 3 pixels is preferred. All pixels in the fifth region are traversed, and the number of pixels whose first Euclidean distance is less than the standard pixel value is counted.

[0079] Step 482: Obtain a second ratio based on the number of pixels and the area of ​​the fifth region; if the second ratio is greater than a preset first ratio threshold, determine that there is a first type of obstacle in the first region; otherwise, proceed to the next step.

[0080] The second proportion reflects the concentration of the shading area near the roots of yellow leaves. In common greenhouse crops (such as tomatoes and cucumbers), yellow leaves are usually connected to the main stem through petioles, and the connection area (i.e., the location of the reference point) is the key point of force for the robotic gripper to perform the peeling action. When this area is tightly surrounded by the main stem itself or adjacent healthy leaves, the shading tends to be highly concentrated in a small area around the reference point (e.g., within a radius of 3–5 pixels), forming a point-like high-density shading pattern; while if the shading is mainly distributed in the middle or far end of the leaf, the root operation space is usually still usable. Through manual annotation and statistical analysis of a large number of field images, it was found that in real root shading scenarios, the proportion of shading pixels in the neighborhood generally exceeds 60% of the entire buffer area, while in non-root shading situations, this proportion is usually less than 40%. Therefore, in this embodiment, the second ratio is compared with the preset first ratio threshold, which is set between 0.5 and 0.7. This range covers the lower limit of the measured distribution of typical root occlusion and leaves a safety margin to accommodate different crop varieties (such as large-leaf tomatoes vs. small-leaf cherry tomatoes), changes in imaging distance (affecting pixel-physical size mapping), and differences in the minimum operating radius of the mechanical claw.

[0081] If the second proportion is greater than the preset first proportion threshold, it indicates that more than half of the potential interference areas are concentrated in the critical root area, which is very likely to hinder the safe access of the mechanical claw. This is consistent with the biomechanical and geometric characteristics that the root is tightly surrounded by the main stem or adjacent tissues. Therefore, it is determined that there is a first type of obstacle in the first area, namely the root shading obstacle. If the second proportion does not exceed the preset first proportion threshold, it indicates that the shading distribution is relatively scattered or far away from the connection point. There is still a feasible operation window in the root. Continue to perform subsequent steps to determine whether there are other types of obstacles.

[0082] Step 483: Project each pixel in the fifth region onto the nearest skeleton point on the centerline skeleton to obtain the set of hit skeleton points.

[0083] Each pixel in the fifth region is projected onto the centerline skeleton. For any pixel in the fifth region, all skeleton points are traversed, the Euclidean distance between the pixel and each skeleton point is calculated, and the skeleton point with the smallest distance is selected as the nearest neighbor projection point for that pixel. The nearest neighbor skeleton points corresponding to all pixels in the fifth region are collected to form a set of hit skeleton points. Since multiple pixels may be projected onto the same skeleton point, this set usually contains duplicates, but these can be retained or removed in subsequent processing, depending on the analysis requirements. This projection process essentially maps the two-dimensional occlusion region onto the one-dimensional skeleton path, providing a structured basis for subsequent determination of whether the occlusion extends along the leaf vein direction.

[0084] Step 484: Determine the second Euclidean distance between each pixel and the corresponding hit skeleton point, and mark the maximum value of the second Euclidean distance as the maximum width.

[0085] This embodiment, based on the projection relationship obtained in step 483, further quantifies the extent of the occlusion region's expansion in the direction perpendicular to the central skeleton. For each pixel in the fifth region, the Euclidean distance between it and the corresponding hit skeleton point is calculated. This distance is the second Euclidean distance, reflecting the lateral offset of the occluded pixel from the main skeleton. After traversing all pixels in the fifth region, all second Euclidean distance values ​​are collected, and the maximum value is selected and defined as the maximum width. This maximum width characterizes the farthest extension distance of the occlusion region in the direction perpendicular to the main yellow leaf, and is a key geometric indicator for determining whether the occlusion has lateral extension characteristics. For example, if the maximum width is large (e.g., exceeding 5 pixels), it indicates that the occlusion is not only distributed along the skeleton but also significantly extends to both sides, possibly due to adjacent leaves covering from the side; conversely, if the maximum width is small, the occlusion is more likely concentrated near the skeleton.

[0086] Step 485: Obtain the shortest continuous skeleton segment of the centerline skeleton containing the hit skeleton point and the corresponding path length.

[0087] This embodiment performs topological analysis on the set of hit skeleton points to determine the coverage area of ​​the shading along the main trunk of the yellow leaf. The centerline skeleton is considered a single connected path composed of ordered skeleton points (typically extending from a reference point towards the leaf tip). All points appearing in the set of hit skeleton points are then marked along this path. Since these points may be scattered or continuous, the shortest continuous segment containing all hit points is identified by traversing the skeleton sequence; this is the complete skeleton interval from the first hit skeleton point to the last hit skeleton point (if there are unhidden points in between, they are still considered continuous coverage). The path length of this shortest continuous skeleton segment is calculated by summing the Euclidean distances between adjacent skeleton points within the segment. This path length reflects the actual extension scale of the shading area along the centerline of the yellow leaf and is an important basis for determining whether the shading has a long, lateral coverage characteristic. It will be used in subsequent ratio calculations to the total skeleton length to support the determination of Type II obstacles.

[0088] Step 486: Obtain the total path length of the centerline skeleton.

[0089] By iterating through adjacent skeleton point pairs in the skeleton sequence, calculating the Euclidean distance between each pair, and summing all adjacent distances, the total path length of the entire centerline skeleton from the start to the end point is obtained. This total path length reflects the actual geometric scale of the yellow leaf in image space.

[0090] Step 487: Construct a third ratio based on the path length and the total path length. If the third ratio is greater than the preset second ratio and the maximum width is greater than the target width, determine that there is a second type of obstacle in the first region; otherwise, determine that there is a third type of obstacle in the first region.

[0091] The third ratio is used to characterize the degree of coverage of the shaded area along the direction of the yellow leaves on the main stem.

[0092] In typical greenhouse vine crops (such as tomatoes and cucumbers), the leaves are long and narrow, and their midribs can be approximated by the central axis skeleton in the image. When a nearby healthy leaf or branch covers the target yellow leaf from the side, due to the leaf's inherent length and thickness, the occlusion area typically extends continuously along the midrib for a considerable distance (often covering more than half a leaf), and has a width of several pixels perpendicular to the midrib (determined by the physical dimensions of the actual plant tissue). Statistical analysis shows that in a large number of labeled samples, the longitudinal coverage ratio of this type of lateral occlusion is generally higher than 0.5, and the maximum lateral width is often greater than 5 pixels (corresponding to approximately 3–6 mm, close to the sum of the leaf thickness and the safe operating distance).

[0093] Therefore, in this embodiment, it is set that if the third ratio is greater than the preset second ratio threshold (the preset second ratio threshold is 0.4–0.6, determined based on the distribution of field measured data, which can effectively distinguish between structural occlusion and local interference), and the maximum width is greater than the target width (5 pixels, comprehensively calibrated based on the thickness of common crop leaves, imaging resolution, and the safety margin of mechanical claw operation), then it is determined that the occlusion has the dual characteristics of longitudinal extension and lateral expansion, which conforms to the typical morphology of being laterally covered by adjacent tissues, and is therefore identified as the second type of obstacle, namely, lateral occlusion obstacle.

[0094] Conversely, if the occlusion lacks a clear structural directional characteristic, it is more likely due to random interference or discontinuous overlap, and thus is classified as a third type of obstacle, namely diffuse occlusion. This embodiment achieves objective and quantifiable discrimination of occlusion morphology by integrating geometric indicators of both longitudinal coverage ratio and lateral widening, possessing clear agronomic rationale and engineering feasibility, and ensuring the reliability and adaptability of obstacle identification results.

[0095] The spatial morphology of shading is jointly determined by combining the maximum width. If the third ratio is greater than the preset second ratio threshold (set between 0.4 and 0.6, for example, 0.5), and the maximum width is greater than the target width (the target width is generally set according to the typical thickness of crop leaves and the spacing between adjacent tissues, such as 5 pixels), it indicates that the shading not only has significant extension in the skeletal direction, but also has a wide distribution in the lateral direction, which is consistent with the typical characteristics of being laterally covered by adjacent leaves or branches. Therefore, it is determined that the first region has a second type of obstacle, namely, a lateral shading obstacle. If the above two conditions are not met at the same time, it is considered that the shading presents a scattered, isolated or unstructured distribution, without obvious root concentration or lateral extension characteristics. Therefore, it is determined that the first region has a third type of obstacle, namely, a diffuse shading obstacle.

[0096] Step 488: Based on the first type of obstacle and / or the second type of obstacle and / or the third type of obstacle, the fourth region is corrected to obtain the fourth corrected region.

[0097] Based on the first type of obstacle, the reference point is moved Q pixels along the centerline skeleton towards the leaf tip to obtain a gripping point. The first evaluation result is repeatedly determined based on the gripping point to identify the target body.

[0098] In this embodiment, when a first type of obstacle is determined to exist in the first region, it indicates that the connection between the yellow leaf and the main stem is tightly surrounded by healthy tissue or the main stem itself, and performing the peeling operation directly near the original reference point carries a high risk of collision. To address this situation, the reference point is moved Q pixels along the central axis skeleton towards the leaf tip, where Q is an integer not less than 10, to avoid the high-density shading area at the root and find a safer gripping position.

[0099] The specific implementation process is as follows: The centerline skeleton obtained from step 3 is a path composed of ordered skeleton points, starting near the main stem (i.e., the location of the original reference point) and ending at the leaf tip. Starting from the reference point, traverse the subsequent skeleton points along this skeleton path, accumulating the movement distance until Q pixels are reached. After reaching the target position, mark the skeleton point as the grasping point. Using the grasping point as the new geometric reference, repeat steps 44 to 48. If the evaluation result is the first negative evaluation result, it is confirmed that the yellow leaf is safe to operate, and the first region is finally identified as the target body; if it is still a positive evaluation, the Q value can be further increased or other correction modes can be switched according to the strategy.

[0100] This method effectively avoids dense root obstacles while preserving the integrity of the yellow leaf by shifting the grasping point backward along the main stem of the leaf. It significantly improves the workability of effective targets that were originally abandoned due to root occlusion. In actual tests, it increased the success rate of tasks in root occlusion scenarios by about 35%.

[0101] Based on the second type of obstacle, the normal direction is rotated around the x-axis by a first angle to obtain a corrected normal direction; the normal direction used to correct the correction region is replaced with the corrected normal direction to generate a secondary correction region; the first evaluation result is re-determined based on the secondary correction region to determine the target body.

[0102] In this embodiment, when a second type of obstacle is determined to exist in the first region, it indicates that the yellow leaves are covered from the side by adjacent healthy leaves or branches along the main vein direction. This causes the original buffer zone's expansion direction near the root to be aligned with the main stem's normal direction, but it still interferes with the practically feasible mechanical claw approach path. To address this issue, instead of translating the gripping point position, the expansion direction of the buffer zone in the critical region is adjusted.

[0103] The specific implementation process is as follows: First, obtain the normal direction of the third region determined in step 45 at the reference point. Then, rotate this normal direction around the x-axis of the local task coordinate system by a first angle, which is set to ±15° (for example, +15° is used to probe the left side to avoid it, and -15° is used to the right side), thereby obtaining one or more corrected normal directions. Here, the x-axis is defined as the direction along the centerline skeleton from the reference point to the blade tip, ensuring that the rotation operation conforms to the natural deflection logic within the blade plane.

[0104] The normal direction originally used to generate the correction region is replaced with the aforementioned correction normal direction, and the translation operation in step 46 is re-executed only on the contour segments within this neighborhood to generate a new set of offset points, thereby constructing the secondary correction region. Based on this, the size adaptation process in step 47 and the occlusion evaluation process in step 48 are repeated. If the first evaluation result is a negative result, it is considered that the lateral obstacle has been successfully avoided through directional fine-tuning, confirming that the yellow leaf can be safely peeled off, and the first region is identified as the target body; if it is still a positive evaluation, another rotation direction can be tried (e.g., try +15° first, then -15° if it fails) or multiple rounds of trials can be conducted in combination with other strategies. Lateral occlusion can be adapted simply by adjusting the local expansion posture of the buffer zone, avoiding deviation of the gripping point from the effective peeling position due to blind translation. In actual tests, this improved the success rate of operations in lateral occlusion scenarios by approximately 28%.

[0105] In this embodiment, when a third type of obstacle is determined to exist in the first region, it indicates that the occlusion distribution is scattered and lacks obvious structural features, making it difficult to effectively avoid using a single strategy. Therefore, two combined trial schemes are adopted: each scheme shifts the original reference point ten pixels along the centerline skeleton towards the leaf tip, obtaining a new reference point, and assigns different rotational attitudes: the first scheme corresponds to a positive ten-degree rotation around the x-axis of the local task coordinate system (along the skeleton direction), and the second scheme corresponds to a negative ten-degree rotation. For each new reference point, steps 45-483 are returned. If the second ratio obtained by any scheme does not exceed the first ratio threshold, the scheme with the smaller second ratio is selected as the feasible solution, and the first region is identified as the target body; otherwise, the first region is determined to be inoperable in the current operation cycle, marked as inoperable, and repeated attempts are prohibited, and the obstacle type and failure reason are recorded.

[0106] Step 5: Perform a stripping operation based on the target body.

[0107] In this embodiment, a local task coordinate system is constructed with the reference point of the target yellow leaf as the origin (the x-axis mentioned above refers to the x-axis of this coordinate system): the x-axis is along the central skeleton from the reference point to the leaf tip, the z-axis is the outward normal direction of the main stem at that point, and the y-axis is determined by the right-hand rule, thereby providing the mechanical claw with a motion reference that conforms to the plant structure.

[0108] The healthy leaf region is extracted from the semantic segmentation results, and its boundary is extended outward by a certain distance (e.g., 5 pixels) to form a neighboring tissue exclusion zone, which is used to identify the inviolable safety boundary. Based on this, a smooth motion trajectory from the current pose to the pre-grasp pose is planned to ensure that the projection of all path points on the trajectory onto the image plane does not fall within the exclusion zone. A high-order interpolation method is used to make the curvature change of the path gradual, avoiding sudden stops or jitter.

[0109] Peeling process: Initially, it is in a contact confirmation state. When the pressure sensor at the tip of the robotic gripper detects that the contact force exceeds the set threshold and persists for a certain period of time, it transitions to a stable clamping state. After the clamping force stabilizes and the pressure distribution is uniform, it enters a progressive peeling state, where the robotic gripper slowly applies torsional and stretching movements. If a continuous decrease in pressure is detected and the joint torque change becomes gradual, it is determined that the leaf stalk has broken, and the process enters the fracture completion state. Each state corresponds to independent clamping force and angular velocity parameters to ensure precise and controllable action. During the progressive peeling process, if an excessive shift in the pressure center of gravity is detected or abnormal pressure occurs in the corresponding area of ​​the adjacent restricted zone, the fuse mechanism is immediately triggered: rotation stops, clamping force is maintained, and the blade retracts along the original path to a safe position. At the same time, the yellow leaf is marked as a high-risk object, prohibiting further attempts in this round of operation, and the process automatically switches to the next target yellow leaf.

[0110] like Figure 2As shown, this embodiment also discloses an artificial intelligence-based mechanical claw control system for removing yellow leaves, including the following modules:

[0111] Image acquisition module: used to acquire the fused image of the target region;

[0112] Image segmentation module: used to perform region segmentation based on the fused image to obtain a first region, a second region, and a third region;

[0113] Feature extraction module: used to extract the contour boundary and centerline skeleton based on the first region;

[0114] Target localization module: used to determine the outward expansion normal direction based on the contour boundary, centerline skeleton and third region and generate a fourth region, generate a first scale based on the fourth region and the second region, determine a first evaluation result based on the first scale, and determine the target body based on the first evaluation result;

[0115] Peeling module: Used to perform peeling operations based on the target body.

[0116] To implement the above embodiments, this application also proposes an electronic device. Please see [link to relevant documentation]. Figure 3 , Figure 3 This is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. For example... Figure 3 As shown, the electronic device 500 includes: a processor 501 and a memory 502 communicatively connected to the processor 501; the memory 502 stores computer-executable instructions; the processor 501 executes the computer-executable instructions stored in the memory to implement the method provided in the foregoing embodiments.

[0117] To implement the above embodiments, this application also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided in the foregoing embodiments.

[0118] To implement the above embodiments, this application also proposes a computer program product, including a computer program that, when executed by a processor, implements the methods provided in the foregoing embodiments.

[0119] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for controlling the removal of yellow leaves by a robotic gripper based on artificial intelligence, characterized in that, Includes the following steps: Step 1: Obtain the fused image of the target region; Step 2: Perform region segmentation based on the fused image to obtain a first region, a second region, and a third region; the first region is the yellow leaf region, the second region is the healthy leaf region, and the third region is the main stem region; Step 3: Extract the contour boundary and centerline skeleton based on the first region; Step 4: Based on the outline boundary, centerline skeleton, and third region, determine the outward extending normal direction and generate a fourth region. Generate a first scale based on the fourth region and the second region. Determine a first evaluation result based on the first scale. Determine the target body based on the first evaluation result. The first evaluation result includes a first positive evaluation result and a first negative evaluation result. When the first evaluation result is a first positive evaluation result, correction is performed based on the fourth region. Analyze the occlusion portion formed by the intersection of the fourth region and the second region. Relative to the reference point and the centerline skeleton, identify the obstacle type. Correct the direction or position of the fourth region according to the obstacle type, and re-execute the first evaluation process to obtain the fourth corrected region and determine the target body. When the first evaluation result is a first negative evaluation result, mark the first region as the target body. Obstacle types include type 1 obstacles, type 2 obstacles, and type 3 obstacles. Type 1 obstacles are root occlusion obstacles, type 2 obstacles are lateral occlusion obstacles, and type 3 obstacles are diffuse occlusion obstacles. Based on the outline boundary, centerline skeleton, and third region, the outward expansion normal direction is determined and a fourth region is generated. A first scale is generated based on the fourth region and the second region. A first evaluation result is determined based on the first scale, including: Step 41: Extract the set of contour pixels based on the contour boundary; Step 42: Determine the unit contour normal direction based on each contour pixel in the contour pixel set; Step 43: Based on all contour pixels, translate outward by a preset distance from each contour pixel along the corresponding unit contour normal direction to obtain an offset point, and take the closed area formed by all offset points as the initial area. Step 44: Determine reference points based on the centerline skeleton; mark the first region pixel points corresponding to the shortest Euclidean distance from the third region as reference points; Step 45: Extract edge pixels of the third region based on the neighborhood of the reference point, obtain the gradient direction based on the extracted edge pixels, take the unit direction perpendicular to the gradient direction as the tangent direction of the third region at the reference point, and obtain the unit vector perpendicular to the tangent direction as the normal direction of the third region at the reference point. Step 46: Identify the contour segments generated in the neighborhood of the reference point in the initial region, and correct the translation direction of the current contour segment to the normal direction to obtain the corrected region; Step 47: The preset target pixel distance is 8-15 pixels. If the preset distance is less than the preset target pixel distance, the entire correction area is expanded outward by the first pixel distance to obtain the fourth area; otherwise, the correction area is used as the fourth area; the difference between the preset target pixel distance and the preset distance is marked as the first pixel distance. Step 48: Perform pixel-level intersection calculation on the fourth region and the second region to obtain the overlapping region, which is the fifth region. Obtain the first ratio based on the area of ​​the fifth region and the fourth region, and generate the first evaluation result based on the first ratio. The fifth region represents the occlusion area where the mechanical claw interferes with healthy tissue when performing the grasping action. Step 5: Perform a stripping operation based on the target body.

2. The method for controlling the removal of yellow leaves by a robotic gripper based on artificial intelligence according to claim 1, characterized in that, The fourth correction region is obtained by performing correction based on the fourth region, including the following steps: Step 481: Determine the first Euclidean distance between all pixels in the fifth region and the reference point, and count the number of pixels whose first Euclidean distance is less than the standard pixel value; the standard pixel value is 3 pixels. Step 482: Obtain a second ratio based on the number of pixels and the area of ​​the fifth region; if the second ratio is greater than a preset first ratio threshold, determine that there is a first type of obstacle in the first region; otherwise, proceed to the next step. Step 483: Project each pixel in the fifth region onto the nearest skeleton point on the centerline skeleton to obtain the set of hit skeleton points; Step 484: Determine the second Euclidean distance between each pixel and the corresponding hit skeleton point, and mark the maximum value of the second Euclidean distance as the maximum width. Step 485: Obtain the shortest continuous skeleton segment of the centerline skeleton containing the hit skeleton point and the corresponding path length. Step 486: Obtain the total path length of the centerline skeleton; Step 487: Construct a third ratio based on the path length and the total path length. If the third ratio is greater than the preset second ratio and the maximum width is greater than the target width, determine that there is a second type of obstacle in the first region; otherwise, determine that there is a third type of obstacle in the first region; the target width is 5 pixels. Step 488: Based on the first type of obstacle and / or the second type of obstacle and / or the third type of obstacle, the fourth region is corrected to obtain the fourth corrected region.

3. The method for controlling the removal of yellow leaves by a robotic gripper based on artificial intelligence according to claim 2, characterized in that, Based on the first type of obstacle, the reference point is moved Q pixels along the centerline skeleton towards the leaf tip, where Q is an integer not less than 10, to obtain the gripping point. Based on the gripping point, the first evaluation result is repeatedly determined to identify the target body.

4. The method for controlling the removal of yellow leaves by a robotic gripper based on artificial intelligence according to claim 2, characterized in that, Based on the second type of obstacle, the normal direction is rotated around the x-axis by a first angle: ±15° to obtain the corrected normal direction; The normal direction used to correct the correction region is replaced with the corrected normal direction to generate a secondary correction region. The first evaluation result is then re-determined based on the secondary correction region to determine the target body.

5. An artificial intelligence-based robotic gripper control system for removing yellow leaves, used to execute the artificial intelligence-based robotic gripper control method for removing yellow leaves as described in any one of claims 1-4, characterized in that, It includes the following modules: an image acquisition module for acquiring a fused image of the target region; and an image segmentation module for performing region segmentation based on the fused image to obtain a first region, a second region, and a third region. Feature extraction module: used to extract contour boundaries and centerline skeleton based on the first region; Target localization module: used to determine the outward expansion normal direction based on the contour boundaries, centerline skeleton and third region and generate a fourth region, generate a first scale based on the fourth region and the second region, determine a first evaluation result based on the first scale, and determine the target body based on the first evaluation result; Peeling module: Used to perform peeling operations based on the target body.

Images