Method and system for identifying and positioning mulberry leaf picking point

Abstract

The invention discloses a mulberry leaf picking point identification and positioning method and system, and relates to the technical field of computer vision. The method comprises the following steps: in a target area, carrying out acquisition and preprocessing actions of multi-modal environment data to obtain an image set; a pre-constructed improved YOLOv11 model is operated, an image set is input, a group of bounding boxes and confidence scores thereof are output, and coordinates of a center point of each bounding box are candidate mulberry leaf picking point two-dimensional coordinates; constructing a picking point feasibility scoring function; according to the technical key points, the scheme adopts an improved lightweight YOLOv11 model, so that the detection precision and robustness of mulberry leaf picking points in a complex mulberry field environment are remarkably improved, and the problems of missing detection and false detection caused by uneven illumination, branch and leaf shielding and background interference in a traditional visual algorithm are effectively solved; a high-quality and high-confidence candidate picking point set is provided for mulberry leaf picking operation, and a basis is provided for subsequent decision making.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a method and system for identifying and locating mulberry leaf picking points. Background Technology

[0002] Currently, mulberry planting and silkworm rearing rely primarily on manual labor, with lagging development of related machinery and equipment. While some mulberry pruning machines exist, their operation involves first mechanically pruning the mulberry branches, then using defoliation machinery to collect the leaves from the remaining branches. Existing technology does not consider the specific requirements of leaf harvesting in actual production. For example, it doesn't account for the fact that in some regions, mulberry leaves are harvested at least three times a year, and in warmer, rainier areas in the south, seven to eight harvests can be completed. This necessitates ensuring that the mulberry branches are not damaged during harvesting, and that 4 to 6 leaves are retained at the top to inhibit axillary bud sprouting, ensuring continued photosynthesis and branch development. Retaining too few leaves will result in insufficient branches, affecting the yield of mulberry leaves the following spring. Moreover, some of the tender leaves at the top need to be taken for feeding silkworms. The second type is direct leaf-harvesting equipment, which harvests leaves directly without cutting branches. However, these devices have low leaf-harvesting efficiency and mostly still use blades and other tools for harvesting, which can easily damage branches. For mulberry leaf harvesting, robots with robotic arms are used, along with cameras with computer vision recognition capabilities for location-based harvesting. However, the actual environment of mulberry orchards is complex, with dense and overlapping mulberry trees. Light intensity also affects the detection effect. Research on real-time and accurate identification and location of mulberry leaf harvesting points in complex natural environments is still lacking.

[0003] Existing technologies generally rely on general target detection models, which are difficult to reliably capture key harvesting locations such as petioles in scenarios with intertwined branches and leaves and drastic changes in light. This results in a high rate of missed detections and large positioning errors. At the same time, the current recognition systems only focus on whether mulberry leaves are detected, lacking comprehensive consideration of agronomic standards, such as the retention of terminal functional leaves, the judgment of leaf maturity, and the feasibility of mechanical operations, such as whether the space allows for the safe entry of robotic arms. This leads to problems such as damage to branches during harvesting, which restricts the development of automated mulberry leaf harvesting. Summary of the Invention

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] A method for identifying and locating mulberry leaf picking points, the method comprising:

[0006] Within the target area, multimodal environmental data is collected and preprocessed to obtain an image set;

[0007] Run the pre-built improved YOLOv11 model. The input is an image set, and the output is a set of bounding boxes and their confidence scores. The coordinates of the center point of each bounding box are the two-dimensional coordinates of the candidate mulberry leaf picking point.

[0008] Construct a feasibility scoring function for picking sites, perform multi-dimensional evaluation of the two-dimensional coordinates of candidate mulberry leaf picking sites, and output a feasibility score Pr(p i The results are compared with the preset feasibility threshold Pr_th to obtain the effective picking point set Pr_final.

[0009] Based on the effective picking point set Pr_final, a three-dimensional positioning operation strategy is executed to output three-dimensional spatial coordinates, complete the picking point identification and positioning operation, and provide the robotic arm with the information to plan the picking path.

[0010] Furthermore, the multimodal environment data includes at least: color images and depth images; the image sets correspondingly include: a first image set and a second image set;

[0011] The data acquisition and preprocessing process is as follows: Data acquisition: Simultaneously acquire color images and depth images through the vision system; Preprocessing: Use camera calibration technology to pre-obtain the intrinsic and extrinsic parameter matrices of the two types of cameras in the vision system; Map each effective pixel in the depth image to the corresponding position in the color image to generate a spatially aligned second image set; Perform adaptive histogram equalization (CLAHE) on the color image to generate a first image set.

[0012] Furthermore, the improvements to the pre-built improved YOLOv11 model are based on the following: implementing embedded mechanisms in the backbone network of standard YOLOv11, and the implemented embedded mechanisms include at least: CBAM attention mechanism and C2f_DWR multi-scale feature fusion module.

[0013] Furthermore, the CBAM attention mechanism includes: inserting a CBAM module after the key feature layer of the backbone network, and the CBAM module contains two sub-modules: channel attention and spatial attention. Channel attention generates two 1D channel descriptors through global average pooling and max pooling, respectively, which are then summed after MLP and activated by Sigmoid to obtain the weights of each channel; spatial attention performs average and max pooling in the channel dimension, concatenates them, and then convolves to obtain a spatial weight map; the C2f_DWR multi-scale feature fusion module includes: replacing the C2PSA module in the original YOLOv11, C2f_DWR decomposes feature processing into region residual (RR) and semantic residual (SR). RR captures multi-scale contextual information through dilated convolution, and SR extracts high-level semantics through lightweight convolution; after drawing multi-scale contextual information, multiple outputs are aggregated.

[0014] Furthermore, the set of two-dimensional coordinates of the candidate mulberry leaf picking points is: {pi =(x i y i ), s i};

[0015] Among them, s i ∈[0,1], representing the confidence score; p i Let x be the coordinates of the center point of the i-th bounding box. i y i ), which is the two-dimensional coordinate of the i-th candidate mulberry leaf picking point, i=1,2,...,N, where N is the total number of coordinates.

[0016] Furthermore, the process of running the feasibility scoring function for picking sites is as follows:

[0017] Define a feasibility scoring function: multiply the constraints of each class in the constraint set, and multiply the result by the confidence score to obtain the required feasibility score Pr(p i The constraint set includes at least the top-reserved constraint factor C_1(p). i ), spatial operability constraint factor C_2(p i ) and leaf maturity constraint factor C_3(p i );

[0018] The valid picking point set Pr_final is derived based on the following: Pr_final = {p i |Pr(p i )≥Pr_th}.

[0019] Furthermore, the spatial operability constraint factor C_2(p i The setup process is as follows:

[0020] Run the preset decay function model, based on:

[0021] C_20(p i )=exp(-k×σ_d(p i ));

[0022] In the formula, C_20(p i ) is the initial spatial operability constraint factor, k is the positive adjustment coefficient, and σ_d(p) is the initial spatial operability constraint factor. i ) indicates p i Within a circular neighborhood centered at r, align the standard deviation of the depth values ​​corresponding to the second image set;

[0023] Set trigger conditions; if the following conditions are met:

[0024] Th_low≤C_20(p i )<Th_high;

[0025] This triggers the depth space reachability analysis mechanism, outputting a new spatial operability constraint factor C_2. x (p i And determine the final spatial operability constraint factor C_2(p) i =C_2 x (p i );

[0026] If the condition is not met, a second judgment operation will be performed:

[0027] When C_20(p i If ) < Th_low, then the final spatial operability constraint factor C_2(p) is determined. i )=0;

[0028] When C_20(p i When )≥Th_high, then the final spatial operability constraint factor C_2(p) is determined. i )=C_20(p i );

[0029] In the formula, Th_high is the high confidence threshold and Th_low is the low confidence threshold.

[0030] Furthermore, the operation process of the depth spatial reachability analysis mechanism is as follows:

[0031] Constructing the local 3D environment of the picking point: For each candidate mulberry leaf picking point, the 2D coordinates p i Using the aligned second image set and the camera intrinsic parameter matrix, a pinhole camera model is used to generate three-dimensional coordinates P in the camera coordinate system. i ; with P i Centered on a given point cloud, within a set search radius, all neighboring 3D points are extracted from the reconstructed point cloud to form a local point cloud region, denoted as LP(P). i );

[0032] Setting the safety operating envelope at the end effector of the harvesting robotic arm: Using a bounding box model of an axial cylinder, a cylindrical safety operating envelope, denoted as OE(P), is obtained based on a pre-acquired set of envelope parameters. i The envelope parameter set includes at least the picking direction vector, envelope height, and envelope radius.

[0033] Perform collision detection and update constraint factors: in LP(P i In the process of semantic segmentation, points belonging to the target mulberry leaf itself are excluded by semantic segmentation masking. The remaining points are considered potential obstacle points, and it is then checked whether any obstacle points fall into the OE (P) field. iIf at least one obstacle point is encountered, the picking point is determined to be unreachable, and a new spatial operability constraint factor C_2 is set. x (p i If )=0; if it does not exist, it is determined to be reachable, and a new spatial operability constraint factor C_2 is set. x (p i =1.

[0034] Furthermore, the 3D positioning operation strategy is as follows: query the depth value d corresponding to each valid picking point in the valid picking point set Pr_final in the second image set. i Combined with the corresponding camera intrinsic parameter matrix of the acquired color image, the three-dimensional coordinates of the point in the camera coordinate system are calculated by back-projection through the pinhole imaging model; the three-dimensional coordinates are transformed to the robot arm world coordinate system by using the pre-calibrated camera-to-robot base coordinate transformation matrix to obtain the required three-dimensional spatial coordinates.

[0035] A mulberry leaf picking point identification and positioning system, the system comprising:

[0036] Image acquisition module: Within the target area, it performs multimodal environmental data acquisition and preprocessing to obtain an image set;

[0037] Initial Coordinate Analysis Module: Runs a pre-built improved YOLOv11 model. The input is an image set, and the output is a set of bounding boxes and their confidence scores. The coordinates of the center point of each bounding box are the two-dimensional coordinates of the candidate mulberry leaf picking point.

[0038] Location filtering module: Constructs a feasibility scoring function for picking points, performs multi-dimensional evaluation of the two-dimensional coordinates of candidate mulberry leaf picking points, and outputs a feasibility score Pr(p i The results are compared with the preset feasibility threshold Pr_th to obtain the effective picking point set Pr_final.

[0039] Coordinate output module: Based on the valid picking point set Pr_final, execute the three-dimensional positioning operation strategy, output three-dimensional spatial coordinates, complete the picking point identification and positioning operation, and provide the robotic arm with the information to plan the picking path.

[0040] This invention provides a method and system for identifying and locating mulberry leaf picking points, which has the following beneficial effects:

[0041] (1) This scheme significantly improves the detection accuracy and robustness of mulberry leaf picking points in complex mulberry orchard environments by adopting an improved lightweight YOLOv11 model. It effectively solves the problems of missed detection and false detection caused by uneven lighting, foliage occlusion and background interference in traditional visual algorithms. It provides a high-quality and high-confidence set of candidate picking points for dealing with mulberry leaf picking operations, and provides a basis for subsequent decision-making.

[0042] Adding the CBAM attention mechanism to the backbone network of YOLOv11 effectively weakens invalid information and increases the neural network's attention to the mulberry leaf picking point in the image. This can effectively improve the detection speed, network feature extraction capability, and detection accuracy of the training model. It can also be added as a multi-dimensional attention mechanism module to the channel attention module that focuses on the shape features of the image and the spatial attention module that focuses on the spatial location features.

[0043] In addition, the original C2PSA structure in YOLOv11 is replaced with the C2f_DWR structure, and its module structure diagram is as follows. Figure 3 As shown, this structure decomposes the original single-step method into two steps: Region Residualization (RR) and Semantic Residualization (SR). After drawing multi-scale contextual information, multiple outputs are aggregated. Specifically, all feature maps are concatenated. Next, batch normalization (BN) is performed on the maps, and then pointwise convolution is used to merge features to form the final residual. Finally, the final residual is added to the input features to build a stronger and more comprehensive feature representation, thereby improving the algorithm's ability to extract multi-scale features of mulberry leaves in complex natural environments.

[0044] By lightweighting the YOLO framework model, the number of parameters and computational cost of the YOLO backbone network are reduced, the model memory size is decreased, and the detection speed is improved, thus achieving a lightweight model. Furthermore, the backbone network is optimized, and multi-scale feature fusion technology is used to fuse feature information of different sizes, thereby compensating for the accuracy loss caused by lightweighting. This not only enables effective identification and positioning of mulberry leaf picking points in complex natural environments, but also improves the model speed.

[0045] (2) This solution upgrades the indirect evaluation based on image statistics to the direct accessibility verification based on three-dimensional point cloud and physical size by running a deep spatial accessibility analysis mechanism. It simulates the safe operating space required by the end of the robotic arm when picking the point and detects whether the space is occupied by branches. It can not only clearly distinguish between the boundaries of uncollectible and collectible points, but also significantly reduce the risk of collision, reduce work interruption and equipment damage. At the same time, it can still output reliable picking points stably when dealing with scenarios where branches are highly dense and severely intertwined.

[0046] (3) This scheme introduces a feasibility constraint function, which quantifies the requirements for actual needs such as top leaf retention, space accessibility and leaf maturity into calculable constraint factors, and performs multi-dimensional screening and optimization of visual detection results. This fundamentally solves the problem of blind operation that is common in existing picking equipment, which only identifies leaves without judging whether they should be picked. It ensures that the final output picking point not only exists, but is also pickable and should be picked.

[0047] (4) This solution relies on a precisely calibrated vision system and coordinate transformation operation to map the optimized two-dimensional picking points to the world coordinate system of the robotic arm, generating three-dimensional spatial instructions that can be directly executed by the robotic arm, thereby ensuring a collision-free and highly efficient picking action. Attached Figure Description

[0048] Figure 1 This is a structural diagram of the improved YOLOv11 in this invention;

[0049] Figure 2 This is a structural diagram of the CBAM in this invention;

[0050] Figure 3 This is a design diagram of the C2f_DWR module in this invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0052] Example 1:

[0053] Please see Figures 1-3 This embodiment provides a method for identifying and locating mulberry leaf picking points. The general overview of this method is as follows:

[0054] A system that deeply integrates prior physiological knowledge with an improved computer vision model is used to identify and locate mulberry leaf picking points. After acquiring data through sensors, an improved lightweight YOLOv11 model is used for detection. The first improvement is that the CBAM attention mechanism is added to the backbone network of the YOLOv11 model, and the original C2PSA structure of YOLOv11 is replaced with the C2f_DWR structure. The second improvement is that a feasibility constraint function for mulberry leaf picking points is introduced to filter and optimize the original detection results of the model, ensuring that the picking points not only ensure the mulberry leaves but also guarantee their pickability. The picking point position is then output for the robotic arm to perform the picking action.

[0055] The specific steps of this positioning method include:

[0056] S1. Within the target area, perform multimodal environmental data acquisition and preprocessing to obtain an image set;

[0057] The target area mentioned here refers to the area captured by the vision system deployed at the front end of the picking robot, which is usually the area above the marked picking position of the mulberry tree; the marked picking area is the blurry area of ​​the mulberry branches on the mulberry tree; the vision system refers to a binocular vision system that integrates a high-resolution RGB color camera and a high-precision depth camera.

[0058] Multimodal environmental data includes at least: color images and depth images;

[0059] The image set includes a first image set and a second image set.

[0060] The preprocessing steps are as follows:

[0061] Data acquisition: Color and depth images are acquired simultaneously through a vision system;

[0062] Preprocessing: Camera calibration technology is used to pre-acquire the intrinsic and extrinsic parameter matrices of the two types of cameras in the vision system; each effective pixel in the depth image is mapped to the corresponding position in the color image to generate a spatially aligned second image set; adaptive histogram equalization (CLAHE) is performed on the color image to enhance the texture of mulberry leaves under uneven illumination to generate a first image set; in addition, bilateral filtering can be applied to the second image set to suppress noise while preserving edges.

[0063] S2. Run the pre-built improved YOLOv11 model. The input is an image set, and the output is a set of bounding boxes and their confidence scores. The coordinates of the center point of each bounding box are the two-dimensional coordinates of the candidate mulberry leaf picking point.

[0064] The improvement of the modified YOLOv11 model is based on the following:

[0065] The embedded mechanisms are implemented in the backbone network of standard YOLOv11, and the implemented embedded mechanisms include at least: CBAM attention mechanism and C2f_DWR multi-scale feature fusion module;

[0066] Specifically, the CBAM attention mechanism includes: inserting a CBAM module after the key feature layer of the backbone network, and the CBAM module contains two sub-modules: channel attention and spatial attention. Channel attention generates two 1D channel descriptors through global average pooling and max pooling, respectively. After passing through MLP, they are summed and activated by Sigmoid to obtain the weights of each channel, which are used to highlight the unique green spectral channels of mulberry leaves. Spatial attention performs average and max pooling in the channel dimension, and after convolution, a spatial weight map is obtained, which focuses on the outline of mulberry leaves and the petiole connection area.

[0067] The operation of the CBAM attention mechanism can effectively suppress background interference such as dense branches and weeds in mulberry orchards;

[0068] The C2f_DWR multi-scale feature fusion module replaces the C2PSA module or structure in the original (standard) YOLOv11. C2f_DWR decomposes feature processing into Region Residualization (RR) and Semantic Residualization (SR). RR captures multi-scale contextual information through dilated convolutions with different dilation rates, while SR extracts high-level semantics through lightweight convolutions. After drawing multi-scale contextual information, multiple outputs are aggregated. Specifically, all feature maps are concatenated, batch normalization (BN) is performed, and feature merging is performed using pointwise convolutions to form the final residual. The final residual is added to the input features to construct a stronger and more comprehensive feature representation, thereby improving the algorithm's ability to extract multi-scale features of mulberry leaves in complex natural environments. This design greatly enhances the model's robustness to mulberry leaves of different sizes and degrees of occlusion.

[0069] The process of running a pre-built improved YOLOv11 model also includes implementing lightweighting and training strategies;

[0070] The lightweighting and training strategy involves: using pre-deployed edge computing devices to lightweight the model, specifically by replacing some standard convolutions with depthwise separable convolutions and removing redundant filters using channel pruning techniques; the training dataset must contain a large number of mulberry leaf images under different seasons, lighting conditions, and mulberry varieties, and the picking point must be precisely and effectively defined as the connection between the mulberry leaf petiole and the branch, with sufficient space below the picking point for the robotic arm to operate; the loss function used is EIoU, which not only considers the overlap area between the predicted box and the ground truth box but also optimizes the center point distance and aspect ratio, making the localization more accurate;

[0071] The set of two-dimensional coordinates of the candidate mulberry leaf picking points is as follows:

[0072] {p i =(x i y i ), s i}, where s i ∈[0,1], representing the confidence score; p i Let x be the coordinates of the center point of the i-th bounding box. i y i ), which is the two-dimensional coordinate of the i-th candidate mulberry leaf picking point, i=1,2,...,N, where N is the total number of coordinates.

[0073] By using an improved lightweight YOLOv11 model, the detection accuracy and robustness of mulberry leaf picking points in complex mulberry orchard environments are significantly enhanced. This effectively solves the problems of missed detections and false detections caused by uneven lighting, foliage occlusion, and background interference in traditional vision algorithms. It provides a high-quality, high-confidence set of candidate picking points for mulberry leaf picking operations, laying the foundation for subsequent decision-making. Specifically, attention mechanisms have been widely used in computer vision, especially in object detection tasks. In this embodiment, the CBAM attention mechanism is added to the backbone network of YOLOv11 to effectively reduce invalid information and improve the neural network's attention to mulberry leaf picking points in the image. This can effectively improve the detection speed, network feature extraction capability, and detection accuracy of the training model. It is also added as a multi-dimensional attention mechanism module to the channel attention module that focuses on image shape features and the spatial attention module that focuses on spatial location features.

[0074] In addition, the original C2PSA structure in YOLOv11 is replaced with the C2f_DWR structure, and its module structure diagram is as follows. Figure 3 As shown, this structure decomposes the original single-step method into two steps: Region Residualization (RR) and Semantic Residualization (SR). After drawing multi-scale contextual information, multiple outputs are aggregated. Specifically, all feature maps are concatenated. Next, batch normalization (BN) is performed on the maps, and then pointwise convolution is used to merge features to form the final residual. Finally, the final residual is added to the input features to build a stronger and more comprehensive feature representation, thereby improving the algorithm's ability to extract multi-scale features of mulberry leaves in complex natural environments.

[0075] By lightweighting the YOLO framework model, the number of parameters and computational cost of the YOLO backbone network are reduced, the model's memory size is decreased, and the detection speed is improved, thus achieving a lightweight model. Furthermore, the backbone network is optimized, and multi-scale feature fusion technology is used to fuse feature information of different sizes, thereby compensating for the accuracy loss caused by lightweighting. This not only enables effective identification and positioning of mulberry leaf picking points in complex natural environments but also improves the model's speed.

[0076] S3. Construct a feasibility scoring function for picking sites, evaluate the two-dimensional coordinates of candidate mulberry leaf picking sites from multiple dimensions, and output a feasibility score Pr(p i The set of valid picking points, Pr_final, is then compared and filtered against a preset feasibility threshold, Pr_th. The process of running the picking point feasibility scoring function is as follows:

[0077] S3.1 Define the feasibility scoring function: Multiply each type of constraint factor in the constraint set, and multiply the result by the confidence score to obtain the required feasibility score Pr(p i The specific function used is as follows:

[0078] Pr(p i )=s i ×C_1(p i )×C_2(p i )×C_3(p i );

[0079] In the formula, C_1(p i C_2(p) is a constraint factor for apical retention, ensuring that the apical functional leaves of the mulberry tree are not harvested. i C_3(p) represents the spatial operability constraint factor, which assesses whether the robotic arm used for picking mulberry leaves can safely approach the area. i The function ) represents the leaf maturity constraint factor, ensuring that the harvested mulberry leaves are of the required maturity. The function is in product form, which means that if any constraint is not met, the overall feasibility is 0, reflecting the strict requirements for mulberry leaf harvesting.

[0080] S3.2, Top retention constraint factor C_1(p i The basis for setting ) is:

[0081] ;

[0082] In the formula, y i H_top represents the ordinate of the i-th candidate mulberry leaf picking point in the two-dimensional coordinate system, while H_top represents the dynamically set threshold for the top forbidden zone. In this embodiment, it is usually taken as the first 20% to 30% of the image height, or it can be adaptively adjusted according to the average canopy height of the mulberry tree. It will not be elaborated here, and can be selected according to actual needs. It should be noted that mulberry trees are harvested 4 to 8 times a year, and 4 to 6 functional leaves at the top must be retained to maintain photosynthesis, strengthen branches, and inhibit premature sprouting of axillary buds. If the tender leaves at the top are picked, it will lead to weak budding and reduced leaf yield in the following spring. This constraint is directly derived from the mulberry cultivation standard. By forcibly filtering the high-confidence detection results located at the top of the canopy, the situation of mispicking that is visually correct but has agronomic errors is avoided, ensuring that the mulberry tree maintains long-term healthy growth.

[0083] S3.3, Spatial operability constraint factor C_2(p i The setup process is as follows:

[0084] S3.3.1 First, run the preset decay function model, based on:

[0085] C_20(p i )=exp(-k×σ_d(p i ));

[0086] In the formula, C_20(p i) represents the initial spatial operability constraint factor, k represents the positive adjustment coefficient used to control the penalty intensity, and its value range is greater than 0, which can be calibrated experimentally; σ_d(p i Then it means that p i Within a circular neighborhood centered at a radius of r, the standard deviation of the depth values ​​of the corresponding second image set is aligned; where the radius r can be determined according to the actual situation; using an exponential decay function instead of a hard threshold can achieve smooth penalty, moderately reduce the weight of slightly crowded areas, and almost eliminate extremely crowded areas, so as to maintain a balance between robustness and recall.

[0087] S3.3.2 Next, set the trigger conditions. If the following conditions are met:

[0088] Th_low≤C_20(p i )<Th_high;

[0089] This triggers the depth space reachability analysis mechanism, outputting a new spatial operability constraint factor C_2. x (p i And determine the final spatial operability constraint factor C_2(p) i =C_2 x (p i );

[0090] If the condition is not met, a second judgment operation will be performed:

[0091] When C_20(p i If ) < Th_low, then the final spatial operability constraint factor C_2(p) is determined. i )=0;

[0092] When C_20(p i When )≥Th_high, then the final spatial operability constraint factor C_2(p) is determined. i )=C_20(p i );

[0093] In the formula, Th_high is the high confidence threshold, which requires no further verification, and Th_low is the low confidence threshold, which is directly determined as unacceptable; for Th_low≤C_20(p i The interval formed by ) < Th_high represents the region to be verified;

[0094] The operation process of the depth spatial reachability analysis mechanism is as follows:

[0095] S3.3.2.1 Constructing the local three-dimensional environment of the picking point: For each candidate mulberry leaf picking point, the two-dimensional coordinates p iUsing the aligned second image set and the camera intrinsic parameter matrix, a pinhole camera model is used to generate three-dimensional coordinates P in the camera coordinate system. i ; with P i Centered on a given point cloud, within a set search radius, all neighboring 3D points are extracted from the reconstructed point cloud to form a local point cloud region, denoted as LP(P). i ); where the search radius is preset according to the working range of the robotic arm; S3.3.2.2, setting the safety operation envelope of the end effector of the harvesting robotic arm: using the bounding box model of an axial cylinder, based on the pre-acquired envelope parameter set, a cylindrical safety operation envelope is obtained, denoted as OE(P i ); where the envelope parameter set includes at least the picking direction vector, envelope height, and envelope radius; specifically, the picking direction vector: taken as the direction of the camera's optical axis, i.e., the direction directly facing the mulberry leaf; the envelope height: the required clearance length along the picking direction; the envelope radius: the required safe distance laterally; this safe operation envelope indicates that if there are no obstacles inside the cylinder, the robotic arm can safely and without collisions complete the picking action; S3.3.2.3, perform collision detection and update constraint factors: in LP(P i In the process of semantic segmentation, points belonging to the target mulberry leaf itself are excluded by semantic segmentation masking. The remaining points are considered potential obstacle points, and it is then checked whether any obstacle points fall into the OE (P) field. i If at least one obstacle point is encountered, the picking point is determined to be unreachable, and a new spatial operability constraint factor C_2 is set. x (p i If )=0; if it does not exist, it is determined to be reachable, and a new spatial operability constraint factor C_2 is set. x (p i )=1;

[0096] The original solution indirectly assesses the crowding around a candidate picking point by calculating the standard deviation of the depth values ​​in the neighborhood, and then defines a spatial operability constraint factor. This method is simple and efficient, but it has certain limitations. It cannot determine whether the obstacle is in front of or behind the target mulberry leaf, nor can it be correlated with the actual size of the robotic arm's end effector, which can easily lead to misjudgment. Therefore, by running a depth spatial accessibility analysis mechanism, the indirect assessment based on image statistics is upgraded to a direct accessibility verification based on 3D point cloud and physical size. The core point is to simulate the safe operating space required by the robotic arm end effector when picking the point and detect whether the space is occupied by branches. This not only clearly distinguishes between unharvestable and harvestable areas, but also significantly reduces the risk of collisions, reduces work interruptions and equipment damage. At the same time, it can still stably output reliable picking points when dealing with scenarios where branches are highly dense and severely intertwined.

[0097] S3.4, Leaf maturity constraint factor C_3(p iThe setup process is as follows:

[0098] S3.4.1 Obtain the maturity index M(p) through weighted calculation. i The process is as follows:

[0099] M(p i )=q1×S_avg(p i )+q2×(1-V_avg(p i ));

[0100] In the formula, S_avg(p i ) and V_avg(p i ) respectively represent p i Within the neighborhood, the average saturation and brightness of the RGB image after conversion to the HSV color space. Here, the RGB image refers to the first image set after CLAHE enhancement. The HSV conversion is performed on this basis to ensure that the saturation and brightness features of mulberry leaves can still be accurately extracted under uneven lighting conditions. q1 and q2 represent weighting coefficients, both of which take values ​​in the range of [0, 1].

[0101] S3.4.2. Set the secondary constraint settings, based on the following:

[0102] ;

[0103] In the formula, Mth represents the maturity threshold. According to the definition based on historical data, because overly tender mulberry leaves are light in color and bright, they have low nutritional value, while aged mulberry leaves are withered and yellow and have poor palatability, only moderately mature leaves are suitable for feeding or use. Therefore, the above-mentioned secondary constraints are set to ensure harvesting quality, avoid ineffective harvesting, and improve the utilization efficiency of mulberry leaves.

[0104] In the final step of S3, when deriving the effective picking point set Pr_final, the basis is: Pr_final = {p i |Pr(p i The effective picking point set Pr_final will simultaneously satisfy the conditions under the relevant constraints such as high confidence, non-apex, spatial reachability and leaf maturity, ensuring that the finally determined picking points meet the requirements of being pickable and should be picked.

[0105] By introducing a feasibility constraint function based on prior knowledge of mulberry physiology, the requirements for actual needs, such as top leaf retention, spatial accessibility, and leaf maturity, are quantified into calculable constraint factors. Furthermore, the visual inspection results are screened and optimized in multiple dimensions. This fundamentally solves the problem of blind operation that is common in existing picking equipment, which only identifies leaves without judging whether they should be picked. It ensures that the final output picking points not only exist but are also ready to be picked.

[0106] S4. Based on the effective picking point set Pr_final, execute the three-dimensional positioning operation strategy, output the three-dimensional spatial coordinates, complete the picking point identification and positioning operation, and provide the robotic arm with a collision-free and highly efficient picking path.

[0107] The execution of the three-dimensional positioning operation strategy is as follows:

[0108] S4.1 Query the depth value d corresponding to each valid pick point in the valid pick point set Pr_final in the second image set. i ;

[0109] S4.2. Combine the corresponding camera intrinsic parameter matrix obtained from the color image and calculate the three-dimensional coordinates of the point in the camera coordinate system by back projection through the pinhole imaging model.

[0110] The specific basis for S4.2 is as follows:

[0111] ;

[0112] In the formula, P_cam i K_rgb represents the three-dimensional coordinates in the camera coordinate system. -1 This represents the inverse of the intrinsic parameter matrix, used to backproject the image coordinates into the normalized camera coordinate system;

[0113] S4.3. Using a pre-calibrated camera-to-robotic arm base coordinate transformation matrix, the three-dimensional coordinates are transformed to the robotic arm's world coordinate system to obtain the required three-dimensional spatial coordinates.

[0114] The basis for transforming three-dimensional coordinates to the robot arm's world coordinate system is as follows:

[0115] ;

[0116] In the formula, P_world i This represents the final three-dimensional coordinates of the picking point in the robotic arm's world coordinate system, i.e., the required three-dimensional spatial coordinates. ; where T_base cam R represents the homogeneous transformation matrix from the camera coordinate system to the robot arm base coordinate system. bc T represents a 3×3 rotation matrix describing the pose of the camera coordinate system relative to the robot arm base. bcThis represents a 3×1 translation vector, describing the position of the camera coordinate system origin in the robot arm base coordinate system. It should be noted that the points in the camera coordinate system are transformed to the robot arm base coordinate system through rotation and translation. Homogeneous coordinates can be used to uniformly express rotation and translation. The motion control of the robot arm is based on its base coordinate system. Therefore, the visual perception results must be transformed to this unified reference system. This transformation matrix is ​​precisely calibrated in advance through hand-eye calibration to ensure that the positioning accuracy meets the requirements of the harvesting operation.

[0117] The above solution relies on a precisely calibrated vision system and coordinate transformation to map the optimized two-dimensional picking points to the robotic arm's world coordinate system, generating three-dimensional spatial commands that the robotic arm can directly execute, thus ensuring collision-free and highly efficient picking actions.

[0118] Example 2:

[0119] Based on Example 1, this embodiment also provides a mulberry leaf picking point identification and positioning system, which includes:

[0120] Image acquisition module: Within the target area, it performs multimodal environmental data acquisition and preprocessing to obtain an image set;

[0121] Initial Coordinate Analysis Module: Runs a pre-built improved YOLOv11 model. The input is an image set, and the output is a set of bounding boxes and their confidence scores. The coordinates of the center point of each bounding box are the two-dimensional coordinates of the candidate mulberry leaf picking point.

[0122] Location filtering module: Constructs a feasibility scoring function for picking points, performs multi-dimensional evaluation of the two-dimensional coordinates of candidate mulberry leaf picking points, and outputs a feasibility score Pr(p i The results are compared with the preset feasibility threshold Pr_th to obtain the effective picking point set Pr_final.

[0123] Coordinate output module: Based on the valid picking point set Pr_final, execute the three-dimensional positioning operation strategy, output three-dimensional spatial coordinates, complete the picking point identification and positioning operation, and provide the robotic arm with the information to plan the picking path.

[0124] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0125] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0126] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for identifying and locating mulberry leaf picking points, characterized in that, The method includes: Within the target area, multimodal environmental data is collected and preprocessed to obtain an image set; Run the pre-built improved YOLOv11 model. The input is an image set, and the output is a set of bounding boxes and their confidence scores. The coordinates of the center point of each bounding box are the two-dimensional coordinates of the candidate mulberry leaf picking point. Construct a feasibility scoring function for picking sites, perform multi-dimensional evaluation of the two-dimensional coordinates of candidate mulberry leaf picking sites, and output a feasibility score Pr(p i The results are compared with the preset feasibility threshold Pr_th to obtain the effective picking point set Pr_final. Based on the effective picking point set Pr_final, a three-dimensional positioning operation strategy is executed to output three-dimensional spatial coordinates, complete the picking point identification and positioning operation, and provide the robotic arm with the information to plan the picking path.

2. The method for identifying and locating mulberry leaf picking points according to claim 1, characterized in that: Multimodal environmental data includes at least: color images and depth images; the image sets correspond to: a first image set and a second image set; The data acquisition and preprocessing process is as follows: Data acquisition: Simultaneously acquire color images and depth images through the vision system; Preprocessing: Use camera calibration technology to pre-obtain the intrinsic and extrinsic parameter matrices of the two types of cameras in the vision system; Map each effective pixel in the depth image to the corresponding position in the color image to generate a spatially aligned second image set; Perform adaptive histogram equalization (CLAHE) on the color image to generate a first image set.

3. The method for identifying and locating mulberry leaf picking points according to claim 1, characterized in that: The improvements to the pre-built improved YOLOv11 model are based on the following: the implementation of embedded mechanisms in the backbone network of standard YOLOv11, and the implemented embedded mechanisms include at least the CBAM attention mechanism and the C2f_DWR multi-scale feature fusion module.

4. The method for identifying and locating mulberry leaf picking points according to claim 3, characterized in that: The CBAM attention mechanism includes: inserting a CBAM module after the key feature layer of the backbone network, and the CBAM module contains two sub-modules: channel attention and spatial attention. Channel attention generates two 1D channel descriptors through global average pooling and max pooling, respectively. After passing through MLP, they are summed and activated by Sigmoid to obtain the weights of each channel. Spatial attention performs average and max pooling in the channel dimension, concatenates them, and then convolves them to obtain a spatial weight map. The C2f_DWR multi-scale feature fusion module includes: replacing the C2PSA module in the original YOLOv11. C2f_DWR decomposes feature processing into region residual (RR) and semantic residual (SR). RR captures multi-scale contextual information through dilated convolution, and SR extracts high-level semantics through lightweight convolution. After drawing multi-scale contextual information, multiple outputs are aggregated.

5. The method for identifying and locating mulberry leaf picking points according to claim 1, characterized in that: The set of two-dimensional coordinates of the candidate mulberry leaf picking points is: {p i =(x i y i ), s i }; Among them, s i ∈[0,1], representing the confidence score; p i Let x be the coordinates of the center point of the i-th bounding box. i y i ), that is, the two-dimensional coordinates of the i-th candidate mulberry leaf picking point, i=1,2,...,N, where N is the total number of coordinates.

6. The method for identifying and locating mulberry leaf picking points according to claim 1, characterized in that: The process of running the feasibility scoring function for picking sites is as follows: Define a feasibility scoring function: multiply the constraints of each class in the constraint set, and multiply the result by the confidence score to obtain the required feasibility score Pr(p i The constraint set includes at least the top-reserved constraint factor C_1(p). i ), spatial operability constraint factor C_2(p i ) and leaf maturity constraint factor C_3(p i ); The valid picking point set Pr_final is derived based on the following: Pr_final = {p i |Pr(p i )≥Pr_th}.

7. The method for identifying and locating mulberry leaf picking points according to claim 6, characterized in that: Spatial operability constraint factor C_2(p i The setup process is as follows: Run the preset decay function model, based on: C_20(p i )=exp(-k×σ_d(p i )); In the formula, C_20(p i ) is the initial spatial operability constraint factor, k is the positive adjustment coefficient, and σ_d(p) is the initial spatial operability constraint factor. i ) indicates p i Within a circular neighborhood centered at r, align the standard deviation of the depth values ​​corresponding to the second image set; Set trigger conditions; if the following conditions are met: Th_low≤C_20(p i )<Th_high; This triggers the depth space reachability analysis mechanism, outputting a new spatial operability constraint factor C_2. x (p i And determine the final spatial operability constraint factor C_2(p) i =C_2 x (p i ); If the condition is not met, a second judgment operation will be performed: When C_20(p i If ) < Th_low, then the final spatial operability constraint factor C_2(p) is determined. i )=0; When C_20(p i When )≥Th_high, then the final spatial operability constraint factor C_2(p) is determined. i )=C_20(p i ); In the formula, Th_high is the high confidence threshold and Th_low is the low confidence threshold.

8. The method for identifying and locating mulberry leaf picking points according to claim 7, characterized in that: The operation process of the depth spatial reachability analysis mechanism is as follows: Constructing the local 3D environment of the picking point: For each candidate mulberry leaf picking point, the 2D coordinates p i Using the aligned second image set and the camera intrinsic parameter matrix, a pinhole camera model is used to generate three-dimensional coordinates P in the camera coordinate system. i ; with P i Centered on a given point cloud, within a set search radius, all neighboring 3D points are extracted from the reconstructed point cloud to form a local point cloud region, denoted as LP(P). i ); Setting the safety operating envelope at the end effector of the harvesting robotic arm: Using a bounding box model of an axial cylinder, a cylindrical safety operating envelope, denoted as OE(P), is obtained based on a pre-acquired set of envelope parameters. i The envelope parameter set includes at least the picking direction vector, envelope height, and envelope radius. Perform collision detection and update constraint factors: in LP(P i In the process of semantic segmentation, points belonging to the target mulberry leaf itself are excluded by semantic segmentation masking. The remaining points are considered potential obstacle points, and it is then checked whether any obstacle points fall into the OE (P) field. i If at least one obstacle point is encountered, the picking point is determined to be unreachable, and a new spatial operability constraint factor C_2 is set. x (p i If )=0; if it does not exist, it is determined to be reachable, and a new spatial operability constraint factor C_2 is set. x (p i =1.

9. The method for identifying and locating mulberry leaf picking points according to claim 1, characterized in that: The 3D positioning operation strategy is as follows: Query the depth value d corresponding to each valid picking point in the valid picking point set Pr_final in the second image set. i Combined with the corresponding camera intrinsic parameter matrix of the acquired color image, the three-dimensional coordinates of the point in the camera coordinate system are calculated by back-projection through the pinhole imaging model; the three-dimensional coordinates are transformed to the robot arm world coordinate system by using the pre-calibrated camera-to-robot base coordinate transformation matrix to obtain the required three-dimensional spatial coordinates.

10. A mulberry leaf picking point identification and positioning system, characterized in that, The system includes: Image acquisition module: Within the target area, it performs multimodal environmental data acquisition and preprocessing to obtain an image set; Initial Coordinate Analysis Module: Runs a pre-built improved YOLOv11 model. The input is an image set, and the output is a set of bounding boxes and their confidence scores. The coordinates of the center point of each bounding box are the two-dimensional coordinates of the candidate mulberry leaf picking point. Location filtering module: Constructs a feasibility scoring function for picking points, performs multi-dimensional evaluation of the two-dimensional coordinates of candidate mulberry leaf picking points, and outputs a feasibility score Pr(p i The results are compared with the preset feasibility threshold Pr_th to obtain the effective picking point set Pr_final. Coordinate output module: Based on the valid picking point set Pr_final, execute the three-dimensional positioning operation strategy, output three-dimensional spatial coordinates, complete the picking point identification and positioning operation, and provide the robotic arm with the information to plan the picking path.

Images