Flexible swallowing type pepper multi-posture picking robot and picking method

Abstract

This invention discloses a flexible swallowing-type multi-pose pepper harvesting robot and harvesting method, belonging to the field of harvesting robot technology. The method acquires RGB images and depth maps of pepper plants through a visual perception system, processes the RGB images based on a multi-task deep learning model to identify pepper clusters and branches, and outputs a target mask. Based on the target mask, the method segments the point cloud of pepper clusters from the depth map and uses principal component analysis to fit the principal axis and central axis of the pepper clusters, calculating the three-dimensional spatial pose parameters of the pepper. Based on the three-dimensional spatial pose parameters and preset harvesting orientation constraints, the method plans the motion trajectory of the multi-degree-of-freedom robotic arm and the feed parameters of the flexible swallowing-type end effector. The method controls the multi-degree-of-freedom robotic arm to move the flexible swallowing-type end effector to the harvesting pose, achieving precise positioning and harvesting trajectory planning for pepper clusters in different spatial poses, and improving the operational adaptability and success rate of the flexible swallowing-type end effector.

Description

Technical Field

[0001] This invention relates to the field of harvesting robot technology, and more specifically to a flexible swallowing-type multi-posture pepper harvesting robot and harvesting method. Background Technology

[0002] Sichuan pepper, a widely cultivated specialty economic crop and important condiment, continues to see growing market demand. However, the harvesting of Sichuan pepper currently relies entirely on manual labor, which is labor-intensive, inefficient, and accounts for up to one-third of the total production cost, severely hindering the sustainable development of the industry. The biological characteristics of Sichuan pepper plants, such as spreading branches, irregular growth, numerous thorns, small fruits, and short stalks, present significant challenges to mechanized and automated harvesting.

[0003] Currently, the main technical bottlenecks in research and equipment for Sichuan pepper harvesting are as follows:

[0004] 1) Complex working environment and severe mechanical damage: Sichuan pepper branches are thorny and grow randomly, and the fruit surface is rich in oil cells, making it extremely easy to break. Traditional rigid end effectors (such as scissor-type or clamp-type) are difficult to avoid interference from branches and thorns when operating in unstructured field environments, which can easily cause fruit drop or epidermal damage, resulting in a decline in harvest quality.

[0005] 2) Low recognition and positioning accuracy, and difficulty in attitude estimation: Under complex backgrounds such as changes in natural lighting and leaf occlusion, existing vision systems struggle to achieve high-precision pixel-level segmentation and three-dimensional spatial attitude estimation of pepper fruit clusters. The lack of accurate fruit cluster orientation information prevents the robotic arm from planning the optimal harvesting path, thus limiting the success rate of the operation.

[0006] 3) Limited harvesting strategy and poor adaptability: Existing harvesting devices mostly adopt a fixed linear harvesting pattern, lacking adaptive strategies for fruit clusters with different spatial postures. When facing randomly distributed pepper clusters, collisions between the robotic arm and the plant are likely to occur, leading to harvesting failure.

[0007] Therefore, in order to address the above-mentioned technical problems, there is an urgent need to develop an intelligent robot and harvesting method that integrates three-dimensional pose estimation, adaptive harvesting strategy and flexible non-destructive harvesting, so as to solve the problems of low efficiency, large damage and poor adaptability in the existing technology. Summary of the Invention

[0008] In view of the above problems, the present invention is proposed to provide a flexible swallowing-type multi-posture pepper harvesting robot and harvesting method that overcomes or at least partially solves the above problems.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, embodiments of the present invention provide a flexible, swallowing, multi-posture pepper-harvesting robot, comprising: A multi-degree-of-freedom robotic arm is mounted on the robot's mobile chassis or base; A flexible swallowing end effector, connected to the end flange of the multi-degree-of-freedom robotic arm, is used to envelop and separate clusters of pepper fruits. A visual perception system is used to acquire RGB images and depth maps of Sichuan pepper plants in the field. The control system is electrically connected to the multi-degree-of-freedom robotic arm, the flexible swallowing end effector, and the vision perception system, respectively. The control system is configured to perform the following functions: The RGB image is processed using a multi-task deep learning model to identify clusters of pepper fruits and branches, and to output a target mask. Based on the target mask, the point cloud of pepper fruit clusters is segmented from the depth map, and the principal component analysis method is used to fit the principal axis and the central axis of the pepper fruit clusters to calculate the three-dimensional spatial pose parameters of the pepper. Based on the three-dimensional spatial posture parameters and the preset picking orientation constraints, the motion trajectory of the multi-degree-of-freedom robotic arm and the feed parameters of the flexible swallowing end effector are planned. The multi-degree-of-freedom robotic arm is controlled to move the flexible swallowing end effector to the picking position, and the flexible swallowing end effector is driven to perform a swallowing picking action.

[0010] Preferably, the flexible swallowing end effector includes a flexible transmission mechanism and a shearing mechanism; The shearing mechanism includes an upper ratchet, an upper pawl, a cutting gear, a spring, and a blade. The cutting gear directly meshes with the drive gear of the drive motor, serving as the power input component of the shearing mechanism. The fixed blade is fixedly mounted on the end face of the cutting gear, and the moving blade is hinged to the cutting gear, forming a shearing pair with the fixed blade to cut off the peppercorn stem. The upper pawl engages with the motor shaft through its lower hexagonal shaft structure. The upper pawl has a built-in spring, and it meshes with the upper ratchet in one direction. The flexible transmission mechanism includes a lower ratchet, a lower pawl, and a rotating drum. The lower end of the rotating drum is rotatably connected to the flange base through a bearing. The lower ratchet is fixed at the connection between the motor shaft and the upper pawl shaft. The regular hexagon inside the lower pawl forms an interference fit with the upper pawl shaft and also forms a one-way anti-reverse structure with the lower ratchet.

[0011] Preferably, the rotating drum has a hollow cylindrical structure, and its inner wall is fixed with a spiral guide tube to guide the sheared peppercorn clusters to slide down along the spiral trajectory.

[0012] Secondly, embodiments of the present invention provide a method for harvesting Sichuan pepper using multiple postures, including: S1: Acquire RGB images and depth maps of Sichuan pepper plants through a visual perception system, process the RGB images based on a multi-task deep learning model, identify Sichuan pepper fruit clusters and branches, and output target masks. S2: Based on the target mask, the point cloud of pepper fruit clusters is segmented from the depth map, and the principal component analysis method is used to fit the principal axis and central axis of the pepper fruit clusters to calculate the three-dimensional spatial pose parameters of the pepper. S3: Based on the three-dimensional spatial posture parameters and the preset picking orientation constraints, plan the motion trajectory of the multi-degree-of-freedom robotic arm and the feed parameters of the flexible swallowing end effector; S4: Control the multi-degree-of-freedom robotic arm to move the flexible swallowing end effector to the picking position, and drive the flexible swallowing end effector to perform a swallowing picking action.

[0013] Preferably, the RGB image is processed based on a multi-task deep learning model to identify pepper fruit clusters and branches, and to output a target mask, including: The RGB image is preprocessed and then input into a shared encoder. Multi-scale features of Sichuan pepper plants are extracted using an efficient long-distance attention network (E-ELAN) structure. Group convolution design enables different layer weights to learn diverse features of Sichuan pepper plants. The features of the pepper plant are input into a splicing-based lightweight feature fusion neck network, and a spatial pyramid pooling (SPP) module is introduced to output a fused multi-scale feature map. Based on the fused multi-scale feature map and backbone network features, parallel processing is performed using the pepper target detection head, pepper segmentation head, and branch segmentation head, respectively: The pepper target detection head receives low-level, mid-level, and high-level features from the fused multi-scale feature map. The cross-scale propagation capability of high- and low-level localization features is further enhanced through the path aggregation network (PAN) structure. Finally, the bounding box coordinates and category confidence information of pepper fruit clusters and branches are output. The high-resolution C3 feature map, which is downsampled by 8 times and output by the backbone network E-ELAN, is used by the pepper segmentation head to receive the rich detail information to achieve accurate pixel-level segmentation of small fruit clusters. The branch segmentation head receives the mid-level features from the fused multi-scale feature map and uses the integrated global context information to achieve continuous and complete segmentation of slender branches. The pepper leaf segmentation head and the branch segmentation head respectively output pixel-level masks for the corresponding categories.

[0014] Preferably, segmenting the pepper fruit cluster point cloud from the depth map based on the target mask includes: After completing the pixel-level alignment of the depth map with the RGB image, based on the pixel-level mask of the pepper fruit clusters and branches, the mask area is back-projected to the three-dimensional space through the pinhole camera model to obtain the depth value corresponding to the pixel, and the three-dimensional coordinates of the pepper fruit clusters and branches in the camera coordinate system are calculated. Based on the three-dimensional coordinates, the effective pixels within the target area are converted into a three-dimensional point cloud to obtain the initial point cloud set of pepper fruit clusters and the point cloud set of branches. For the point cloud set of pepper fruit clusters and the target point cloud set of branches, the depth values ​​of all points are collected and a depth histogram is constructed. The center of the bin with the highest frequency is defined as the depth peak. The effective depth interval is determined by the proportional windowing method with the peak as the center. The 3D point cloud with the depth value falling within the interval is retained, and the noise points with the depth value outside the effective depth interval are removed to obtain the filtered point cloud along the depth direction.

[0015] Preferably, the step of fitting the principal axis and central axis of the Sichuan pepper fruit cluster using principal component analysis to calculate the three-dimensional spatial orientation parameters of the Sichuan pepper specifically includes: The point cloud filtered along the depth direction is used as the input data for principal component analysis. The geometric center of the point cloud is defined as the center point of the fruit cluster ellipsoid model. The point cloud is then subjected to mean-reduction processing based on the center point to construct the covariance matrix. The covariance matrix is ​​decomposed into eigenvalues ​​to obtain three eigenvalues ​​and their corresponding unit vectors. The eigenvalues ​​are sorted according to their size. The unit vector corresponding to the largest eigenvalue is determined as the major axis direction of the fruit cluster ellipsoid model, the unit vector corresponding to the second largest eigenvalue is determined as the median axis direction, and the unit vector corresponding to the smallest eigenvalue is determined as the minor axis direction, thus forming the direction matrix of the fruit cluster ellipsoid model. The mean-free point cloud is projected onto the principal axis coordinate system consisting of the major axis, the median axis, and the minor axis. The lengths of the three semi-axes of the ellipsoid are calculated to obtain the fruit cluster ellipsoid model, which is uniquely determined by the ellipsoid center, the direction matrix, and the semi-axe lengths. The three orthogonal principal directions of the fruit cluster ellipsoid model are constructed as rotation matrices and represented by robot Euler angles. Combined with the position of the ellipsoid center, the position and orientation of the pepper cluster in three-dimensional space are fully described.

[0016] Preferably, it further includes constructing a parametric geometric model of the branches, specifically including: Principal component analysis was performed on the effective point cloud of a single preprocessed branch to obtain its global principal extension direction. Ordered sampling is performed along the global main extension direction through local neighborhood clustering. The effective point cloud is divided into multiple local subsets and the geometric center of each subset is calculated. The discrete skeleton center point sequence representing the direction of the branch center is extracted. The discrete skeleton center point sequence is smoothly fitted using a cubic B-spline curve to obtain the branch space curve, and the unit tangent vector at any position is calculated based on the branch space curve.

[0017] Preferably, based on the three-dimensional spatial posture parameters and preset picking orientation constraints, the motion trajectory of the multi-degree-of-freedom robotic arm and the feed parameters of the flexible swallowing end effector are planned, including: Based on the geometric relationship between the center of the pepper cluster ellipsoid and the spatial curve of the branch, the point on the branch with the closest Euclidean distance to the center of the fruit cluster is identified as the spatial correlation point. The unit tangent vector of the spatial correlation point on the fitted curve of the branch is extracted as the local tangential basis vector, and the offset vector pointing from the center of the pepper cluster to the spatial correlation point is normalized as the radial unit vector. A right-handed orthogonal coordinate system is constructed based on the tangential basis vector and the radial unit vector to determine the target feed direction of the flexible swallowing end effector. Based on the preset picking orientation constraint, the radial unit vector is determined as the optimal feeding direction of the flexible swallowing end effector, so that the opening plane of the flexible swallowing end effector is perpendicular to the direction of the fruit stalk, so as to avoid physical interference with the main body and branches of the pepper cluster. Perform a dot product operation between the unit vector of the optimal feed direction and the three orthogonal principal axis unit vectors of the fruit cluster ellipsoid model, and obtain the principal axis direction with the largest absolute value of the dot product; select the semi-axis length corresponding to this principal axis as the feed depth of the fruit cluster in the optimal feed direction; Based on the center position of the pepper cluster ellipsoid, the optimal feed direction, and the feed depth of the fruit cluster in the optimal feed direction, a smooth joint space motion trajectory is generated by combining the inverse kinematics algorithm of a multi-degree-of-freedom robotic arm.

[0018] As can be seen from the above technical solution, compared with the prior art, this invention discloses a flexible swallowing-type multi-posture pepper harvesting robot and harvesting method. This invention achieves non-destructive wrapping and harvesting of pepper clusters through the collaboration of a flexible swallowing end effector and a multi-task deep learning model. Based on the spatial posture parameters calculated by the PCA algorithm, the robotic arm can adapt to peppers in different growth postures, effectively avoiding physical interference, improving harvesting success rate and work efficiency, and reducing fruit damage rate. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0020] Figure 1This is a schematic diagram of a flexible swallowing-type multi-posture pepper-harvesting robot provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the flexible swallowing end effector structure provided in an embodiment of the present invention; Figure 3 This is a diagram illustrating the action of the flexible swallowing end effector during retrieval, as provided in this embodiment of the invention. Figure 4 This is a flowchart of the multi-posture harvesting method for Sichuan pepper provided in this embodiment of the invention; Figure 5 This is a diagram of the YOLOP2 algorithm structure provided in this embodiment of the invention; Figure 6 This is a dataset of Sichuan pepper and branches provided in an embodiment of the present invention; Figure 7 This is an image showing the effect of identifying and segmenting pepper clusters and branches in a complex environment, as provided in an embodiment of the present invention. Figure 8 This is a schematic diagram of the depth peak filtering process provided in an embodiment of the present invention. (a) Original RGB image; (b) Original target point cloud generated by mask constraint; (c) Target point cloud depth histogram and peak adaptive window boundary; (d) Output point cloud after peak filtering. Figure 9 This is a comparison image of the mask and point cloud before and after depth main peak filtering provided in this embodiment of the invention; Figure 10 For the PCA solution of the images of pepper and branches provided in the embodiments of the present invention, (a) original image, (b) fused image, (c) predicted segmentation image, (d) principal component analysis results of branches, and (e) principal component analysis results of pepper; Figure 11 The diagram below illustrates the principle of pose construction for a Sichuan pepper cluster based on PCA ellipsoid fitting in this embodiment of the invention. (a) Point cloud of the Sichuan pepper cluster and its ellipsoid fitting; (b) Schematic diagram of PCA principle; (c) Pose matrix construction and Euler angle representation. Figure 12 This is a flowchart of the point cloud spline curve fitting process for pepper branches provided in this embodiment of the invention; Figure 13 This is a schematic diagram illustrating the principle of the picking posture of the pepper cluster relative to the branch provided in an embodiment of the present invention; Figure 14 This is a schematic diagram illustrating the selection of the feed direction of the end effector provided in an embodiment of the present invention; Figure 15 This is a schematic diagram of the end effector feed attitude and feed depth matching provided in the embodiment of the present invention, (a) end effector feed direction, (b) feed depth matching.

[0021] Among them, 1. Multi-degree-of-freedom robotic arm, 2. Flexible swallowing end effector, 3. Upper ratchet, 4. Upper pawl, 5. Cutting gear, 6. Spring, 7. Blade, 701. Fixed blade, 702. Moving blade, 8. Lower ratchet, 9. Lower pawl, 10. Rotary roller, 11. Bearing, 12. Flange base, 13. Spiral guide cylinder, 14. Drive motor, 15. Camera bracket. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] This invention discloses a flexible, swallowing, multi-pose pepper-harvesting robot, such as... Figure 1 As shown, it includes: A multi-degree-of-freedom robotic arm 1 is mounted on the robot's mobile chassis or base; The flexible swallowing end effector 2 is connected to the end flange of the multi-degree-of-freedom robotic arm 1 and is used to wrap and separate the pepper fruit clusters. A visual perception system is used to collect RGB images and depth maps of pepper plants in the field. It can be set on top of the flexible swallowing end effector 2 via a camera bracket 15. The control system is electrically connected to the multi-degree-of-freedom robotic arm 1, the flexible swallowing end effector 2, and the vision perception system, respectively. The control system is configured to perform the following functions: The RGB image is processed using a multi-task deep learning model to identify clusters of pepper fruits and branches, and the target mask is output. The point cloud of pepper fruit clusters was segmented from the depth map based on the target mask, and the principal component analysis method was used to fit the principal axis and the central axis of the pepper fruit clusters to calculate the three-dimensional spatial pose parameters of the pepper. Based on the three-dimensional spatial posture parameters and the preset picking orientation constraints, the motion trajectory of the multi-degree-of-freedom robotic arm 1 and the feed parameters of the flexible swallowing end effector 2 are planned. The multi-degree-of-freedom robotic arm 1 drives the flexible swallowing end effector 2 to the picking position and drives the flexible swallowing end effector 2 to perform swallowing picking action.

[0024] The visual perception system employs a high-precision depth camera to acquire images and transmit the data to the control terminal via image transmission. After completing target recognition and positioning, the control terminal sends control commands to the PLC via TPC / IP protocol and WiFi communication. The PLC, acting as an intermediate control unit, receives high-level control commands from the control terminal and converts them into low-level control signals for the multi-degree-of-freedom robotic arm 1 and the flexible swallowing end effector 2. The PLC sends motion commands to the multi-degree-of-freedom robotic arm 1 via I / O signals to achieve precise control of joint position and speed; simultaneously, it uses high and low level signals and FG feedback signals to decouple the shearing and transmission mechanisms of the flexible swallowing end effector 2 in time, driving the execution of shearing, transmission, and other actions. The multi-degree-of-freedom robotic arm 1 feeds back joint position, speed, and other status information to the PLC in real time, while the flexible swallowing end effector 2 also transmits its action completion status back via FG signals, forming a closed-loop communication of command issuance, status feedback, and dynamic adjustment, ensuring coordination and synchronization between visual recognition, control commands, and execution actions.

[0025] More specifically, to improve the flexibility and efficiency of the pepper-harvesting robot and achieve a compact, integrated shearing and transmission structure for the end effector, this invention employs a modular design for the end effector structure. The flexible swallowing end effector 2 is divided into two main functional modules: a flexible transmission mechanism and a shearing mechanism. Through decoupling, the design focuses on the shearing and transmission actions during the pepper-harvesting process. Figure 2 As shown.

[0026] The shearing mechanism includes an upper ratchet 3, an upper pawl 4, a cutter gear 5, a spring 6, and a blade 7, with the following structure: Figure 2As shown in (b). The cutter gear 5 directly meshes with the drive gear of the drive motor 14; the cutter gear 5 and the upper ratchet 3 are external meshing gears, with their teeth closely meshing on the same plane; the upper ratchet 3 serves as a power input component, driving the cutter gear 5 to rotate in the opposite direction through meshing transmission. The blade 7 includes a fixed blade 701 and a moving blade 702. The fixed blade 701 is fixedly installed on the end face of the cutter gear 5 and remains stationary, while the moving blade 702 is hinged to the cutter gear 5 and rotates synchronously with it, forming a shearing pair with the fixed blade 701 to cut the peppercorn stem; the upper pawl 4 engages with the motor shaft through its lower hexagonal shaft portion, while the upper pawl 3 has a built-in spring 6 providing pre-tightening and reset force, keeping the upper pawl 3 and the upper ratchet 4 in unidirectional meshing, realizing unidirectional power transmission from the shearing mechanism to the drum collecting mechanism. Once the robotic arm positions the pepper cluster, the flexible swallowing end effector 2 begins the cutting operation. The motor rotates forward, driving the coaxial upper ratchet 3 to engage and rotate. Power is transmitted through the upper ratchet 3 and the cutting gear 5 to the double cycloidal cutting edge, rotating the blade 7 to perform the cutting. After cutting is completed, the motor reverses, and the blade 7, under the tension of the tension spring 6, opens to its original position in preparation for the next cycle.

[0027] The transmission mechanism includes a lower ratchet 8, a lower pawl 9, and a rotating roller 10, such as... Figure 2 As shown in (c). The rotating drum 10 is a hollow cylindrical structure, with its lower end rotatably connected to the flange base 12 via a bearing 11. A spiral guide cylinder 13 is fixed to the inner wall of the rotating drum 10 to guide the cut pepper clusters down the spiral path, preventing them from getting stuck. The lower ratchet 8 is fixed at the connection between the motor shaft and the upper pawl shaft. The regular hexagon inside the lower pawl 9 forms an interference fit with the upper pawl shaft and, together with the lower ratchet 8, forms a one-way anti-reverse structure to prevent the drum from reversing during harvesting, causing the peppers to spill. After the peppers are cut, the spiral flexible conveyor mechanism conveys the pepper clusters into the collection device. The lower ratchet 8 drives the flexible spiral guide cylinder 13 inside the rotating drum to transmit the pepper clusters. The spiral blades in the spiral guide cylinder 13 adopt a single-head left-hand spiral surface and are made of flexible material to increase the contact friction between the pepper grains and the conveyor cylinder and to disperse local stress through elastic deformation.

[0028] In Sichuan pepper harvesting robots, preventing peppercorns from falling off and the oil cells from cracking are critical issues that need to be addressed. Therefore, good buffering performance and flexibility of the transmission mechanism are essential for stable transport and low-damage harvesting. Through a double ratchet system, the two are decoupled in both spatial and kinematic dimensions, avoiding mechanical interference and ensuring independent optimization of shearing force and transmission efficiency. This achieves highly efficient "rigid shearing - flexible transmission" operation, significantly reducing fruit damage. A diagram illustrating the actual harvesting process of the flexible swallowing end effector is shown below. Figure 3 As shown.

[0029] Based on the aforementioned robot, embodiments of the present invention provide a multi-pose harvesting method for Sichuan peppercorns, such as... Figure 4 As shown, it includes: S1: Acquire RGB images and depth maps of Sichuan pepper plants through a visual perception system, process the RGB images based on a multi-task deep learning model, identify Sichuan pepper fruit clusters and branches, and output target masks. S2: Based on the target mask, the point cloud of pepper fruit clusters is segmented from the depth map, and the principal component analysis method is used to fit the principal axis and the central axis of the pepper fruit clusters to solve the three-dimensional spatial pose parameters of the pepper. S3: Based on the three-dimensional spatial posture parameters and the preset picking orientation constraints, plan the motion trajectory of the multi-degree-of-freedom robotic arm and the feed parameters of the flexible swallowing end effector; S4: Control the multi-degree-of-freedom robotic arm to move the flexible swallowing end effector to the picking position, and drive the flexible swallowing end effector to perform the swallowing picking action.

[0030] During actual field harvesting, the visual perception system simultaneously acquires RGB images and depth maps of the pepper plants. The RGB images are used to train a multi-task model to perform target recognition and segmentation, while the depth maps are used to perform 3D point cloud back projection based on the output target mask. This is then used for subsequent 3D pose calculation of fruit clusters and planning of the robotic arm's harvesting trajectory.

[0031] In this embodiment, only RGB images of Sichuan peppercorns with a resolution of 3024×4032 are needed during the deep learning model training phase. After multiple cropping and filtering processes, 941 images were selected. The experimental dataset consists of two data types: object detection and semantic segmentation. Therefore, LabelImg and LabelMe were used to label the datasets respectively. The object detection dataset contains 1111 Sichuan peppercorn targets in the 941 images, such as... Figure 6 As shown in the first row, 1441 negative sample images for object detection were prepared, with a positive-to-negative sample ratio of approximately 1:1.5. Figure 6 The third row shows a segmented dataset of 941 images of Sichuan pepper and its branches. Figure 6 As shown in the second row, the input size is 1280×720. Within the working range of the robotic arm, all datasets for the experiment are labeled only with foreground targets.

[0032] like Figure 5As shown, the YOLOP2 multi-task model is used as a multi-task deep learning model to complete the simultaneous identification and segmentation of pepper fruit clusters and branches. Its core feature is that the detection and segmentation tasks share the same encoder. The specific execution process is as follows: First, the preprocessed RGB image of the pepper field is input into the shared encoder. The backbone of the shared encoder adopts an efficient long-range attention network (E-ELAN), which is composed of 4 E-ELAN blocks and 3 C2f modules stacked alternately. The input feature channels are divided into 4 parallel groups through group convolution design. Each group uses convolution kernels with two different receptive fields, 3×3 and 5×5, to extract features independently. The convolution stride is 1 and the padding is 1. Then, the outputs of each group are spliced ​​and fused along the channel dimension. The encoder outputs 3 feature maps of different scales in sequence: C3 feature map with 8x downsampling (size 60×90, number of channels 256), C4 feature map with 16x downsampling (size 80×45, number of channels 512), and C5 feature map with 32x downsampling. The feature map (size 40×23, number of channels 1024) enables the weights of different layers to learn more diverse features of the pepper plant, including the detailed texture of fruit clusters and the overall outline of branches.

[0033] Next, a lightweight feature fusion neck network is used, employing a stitched path aggregation network (PAN) structure. A spatial pyramid pooling (SPPF) module is introduced to perform multi-scale pooling on the C5 feature map, with pooling kernel sizes of 5×5, 9×9, and 13×13, outputting features that fuse global context information. The neck network performs two 2x upsampling operations to concatenate high-level features with corresponding low-level features at different scales, followed by feature purification using a C2fPAN module (outputting 512 and 256 channels respectively). Subsequently, two 2x downsampling operations are performed to again concatenate and fuse low-level and high-level features, ultimately outputting three fused multi-scale feature maps: low-level feature P3 (160×90, 256 channels), mid-level feature P4 (80×45, 512 channels), and high-level feature P5 (40×23, 1024 channels), effectively addressing the multi-scale differences in Sichuan pepper fruit clusters.

[0034] Finally, the signal enters the dual-branch decoder: the detection branch receives fused features from three scales, P3, P4, and P5, and after enhancing local feature representation through RFAConv convolutional layers, it is input into the Dyhead detection head. Each detection head contains three 3×3 convolutional layers, and finally outputs the bounding box coordinates (4D), class confidence (2D), and target presence confidence (1D) of the pepper fruit clusters and branches. The semantic segmentation branch adopts a dual-decoder head parallel architecture. The pepper fruit cluster segmentation head receives the high-resolution C3 feature map output by the backbone network, and after passing through three 3×3 convolutional layers and one 1×1 convolutional layer, it outputs a pixel-level mask of fruit clusters with a size of 1280×720. The branch segmentation head receives the P4 feature map output by the neck network, and after passing through convolutional layers of the same structure, it outputs a pixel-level mask of branches with a size of 1280×720. The two segmentation heads respectively achieve accurate segmentation of small-sized fruit clusters and continuous and complete segmentation of slender branches.

[0035] The multi-task model YOLOP2 of this invention is an existing model and does not involve innovation. For specific implementation details, please refer to the existing technology, which will not be repeated here.

[0036] The target detection algorithm is evaluated using precision, recall, and average precision (AP), and the mean AP across all categories is the mean AP. This invention uses the mAP at an IoU threshold of 0.5 as the evaluation standard for detection accuracy. The total number of target categories, For the first Average precision of the categories. In instance segmentation tasks, the Intersection over Union (IoU) of the masks measures the accuracy of the predicted mask set. With the set of real masks Spatial overlap: (1) (2) The YOLOP2 model can reliably identify pepper clusters and branches under complex backgrounds, fluctuating natural lighting, and moderate viewing angle variations, especially performing well in extracting the main area of ​​the pepper cluster. Its results are as follows: Figure 7 As shown.

[0037] In this embodiment, to obtain the point cloud of pepper clusters and branches, after the pepper fruit clusters and branches are identified and segmented using the multi-task model YOLOP2, and fruit cluster masks and branch masks are obtained, pixel-level alignment of the depth image and the color RGB image is first performed. Based on the segmented fruit cluster mask and branch mask, the mask region is back-projected into three-dimensional space using a pinhole camera model to obtain the depth value corresponding to the pixel. Then, the three-dimensional coordinates of the target point cloud of the pepper fruit cluster and branches in the camera coordinate system are obtained. Let the mask of the pepper fruit cluster be... Branch mask is , Let be the pixel coordinates of the RGB image. In the aligned depth map, it satisfies... or Pixels with valid depth values ​​are selected as candidate pixels for the 3D point cloud. The target point cloud of pepper fruit clusters and branches can be represented as follows: (3) (4) In the formula, This represents the set of target point clouds for the clusters of Sichuan pepper fruits; This represents the set of target point clouds representing the branches of the Sichuan pepper plant. Raw readings of the depth image; For depth scale factor; and The focal length of the camera; The horizontal and vertical coordinates of the main point; , and These are the three-dimensional coordinates in the camera coordinate system.

[0038] This method uses the fruit cluster and branch masks output by semantic segmentation as constraints, combined with depth back projection operations, to convert only the effective pixels in the target area into a 3D point cloud. It achieves the separation of the foreground target from the background environment from the data generation stage, effectively avoiding the problem of background noise mixing when extracting point clouds from the whole depth map, and greatly improving the purity and geometric integrity of the target point cloud, providing a basic input for subsequent point cloud denoising and 3D modeling.

[0039] To address the issue of background noise in the point cloud of foreground pepper clusters and branches, this invention employs a depth-based main peak extraction method to segment the pepper points in the original image, such as... Figure 8 As shown in (a), the point cloud of the noisy peppercorns is obtained, as follows: Figure 8 As shown in (b). Next, the depth values ​​of all points in the target point cloud of the pepper fruit clusters and branches were collected, and a depth histogram was constructed, as shown in [image]. Figure 8 As shown in (c), the bin center with the highest frequency is defined as the depth peak. This represents the primary spatial location of the pepper cluster or branch in the current field of view. Subsequently, a proportionally expanded windowing method centered on this primary peak is used to determine the effective depth range. During the dynamic expansion of the window, the growth rate of the number of points within two adjacent windows is calculated. Expansion stops when the growth rate falls below a preset threshold, ultimately yielding the effective depth range, i.e. in, This is the lower limit of the point cloud depth threshold. Upper limit of point cloud depth threshold. Retaining all 3D point cloud values ​​falling within this range yields the filtered point cloud along the depth direction. ,like Figure 8 As shown in (d), this process effectively removes foreground and background overlap points as well as long-tailed noise in the depth direction. After extracting the same depth peak, background stray noise in the branch point cloud is effectively removed, leaving the target subject intact and clean, achieving an ideal denoising effect, as shown in (d). Figure 9 As shown.

[0040] In this embodiment, the complex natural environment and unpredictable growth patterns of Sichuan pepper make its harvesting research particularly difficult. Based on the positional relationship of the branches growing from the Sichuan pepper cluster, the clusters are divided into five orientations, such as... Figure 10 As shown. The pose estimation method based on 2D image PCA can effectively determine the orientation of pepper clusters in the image plane, but it only provides 2D projection information, which cannot meet the precise requirements of the robotic arm for six-degree-of-freedom pose during harvesting. Since both 2D and 3D PCA extract the principal direction through eigenvalue decomposition of the covariance matrix in their mathematical essence, this idea can be extended from the image pixel space to the 3D point cloud space. After obtaining a clean target point cloud, it is necessary to further construct a parameterized geometric model of the fruit clusters and branches to provide a unified geometric basis for subsequent solution of the spatial relationship between fruit and branches and pose estimation, realizing a unified framework from 2D orientation classification to 3D pose parameterization. Given the significant differences in morphology between fruit clusters and branches, this invention uses ellipsoidal models and spline curves for modeling, respectively. The above-mentioned point cloud filtered along the depth direction. This refers to the effective point cloud of a single fruit cluster after preprocessing. These two terms refer to the same data object at different technical stages. Define the geometric center of the pepper cluster point cloud. As the center point of the ellipsoid model, and construct the covariance matrix , Let be the total number of valid points in the fruit cluster point cloud. First, define the set of pure fruit cluster point clouds obtained after depth filtering using formula (5). It is determined that the point cloud contains m valid 3D points and the camera coordinate system coordinates of each point, which serve as the basic input for subsequent calculations. Then, based on this point cloud set, (6) the arithmetic mean of the X, Y, and Z coordinates of all points is taken to calculate the centroid coordinates of the fruit cluster point cloud. The centroid is both the geometric center of the fruit cluster ellipsoid model and the reference point for subsequent mean-reduction processing; based on this, (8) the covariance matrix of the fruit cluster point cloud is constructed. .in This indicates the three-dimensional center position of the pepper fruit cluster in the camera coordinate system. The point coordinate vector after mean reduction is the result of subtracting the centroid coordinates from the coordinates of the k-th original point. This step translates the entire point cloud to the origin, eliminating the interference of positional offset on subsequent main direction extraction and providing an important geometric benchmark for subsequent fruit branch spatial matching and harvesting location determination. By sequentially calculating the outer product of all demeaned vectors and taking the arithmetic mean, the covariance matrix is ​​finally obtained. This matrix fully characterizes the distribution and main extension direction of the fruit cluster point cloud in three-dimensional space, and is the core data foundation for subsequent extraction of the major axis, median axis and minor axis of the fruit cluster ellipsoid through eigenvalue decomposition.

[0041] (5) (6) (7) (8) For covariance matrix Eigenvalue decomposition yields three eigenvalues ​​and their corresponding unit vectors. The direction matrix that constitutes the fruit cluster ellipsoid As shown in equation (9). Based on the magnitude of the eigenvalues, it can be known that... Corresponding to the major axis of the ellipsoid, Corresponding to the central axis direction, Corresponding to the minor axis direction. After determining the ellipsoidal direction, the centered point cloud is projected onto the principal axis coordinate system, and the projection vector is set as follows: Then the lengths of the three semi-axes of the ellipsoid can be obtained. Assume the three principal axis vectors are... Then the major axis ( ), central axis ( ) and short axis ( The semi-axis lengths are respectively .in This is the original coordinate vector of the k-th 3D point of the pepper fruit cluster in the camera coordinate system after depth backprojection and denoising. These represent the three coordinate components of the i-th point cloud along the major, median, and minor axes of the ellipsoid in the principal axis coordinate system of the fruit cluster. Therefore, the fruit cluster ellipsoid model can be derived from the parameter set... The only certainty.

[0042] (9) (10) (11) The implementation principle of the prickly ash cluster pose estimation algorithm based on principal component analysis is as follows: Figure 11 As shown. First, the point cloud P of the pepper cluster along the depth direction is acquired using an RGB-D camera. ZAfter denoising, an effective three-dimensional point set is obtained. Then, an ellipsoidal fitting is performed on the point cloud, with the geometric center of the ellipsoid as the centroid of the pepper cluster. Principal component analysis is used to extract the three principal directions of the point cloud, which correspond to the major axis of the ellipsoid (…). ), central axis ( ) and short axis ( Finally, these three orthogonal principal directions are constructed as rotation matrices. The position and orientation of the pepper cluster in three-dimensional space are fully described by using robot Euler angles.

[0043] (12) (13) (14) (15) in, The total rotation matrix for the local pose of the fruit cluster is obtained by multiplying the three basic rotation matrices in sequence, which fully describes the orientation of the fruit cluster in three-dimensional space. Let X be the rotation matrix about the X-axis. The roll angle describes the angle of rotation of the fruit cluster around its own long axis; Let Y be the rotation matrix about the Y-axis. The pitch angle describes the angle at which the fruit cluster swings up and down around its own central axis. Let Z be the rotation matrix about the Z-axis. The yaw angle describes the left and right yaw angle of the fruit cluster about its minor axis. This algorithm combines discrete point clouds with a continuous ellipsoidal geometric model and estimates the posture of the peppercorns using the PCA algorithm, providing a unified kinematic benchmark for subsequent robotic arm harvesting pose planning.

[0044] To quantitatively evaluate the accuracy of the above ellipsoid fitting method in describing the three-dimensional morphology of pepper clusters, the mean absolute error was used. Root mean square error Mean absolute percentage error and deviation As an evaluation indicator. and Used to characterize the level of absolute error Used to reflect the magnitude of relative error This is used to determine whether the model exhibits systematic overestimation or underestimation. The specific calculation formulas are shown in equations (16) to (19). Where, This indicates that the ellipsoidal fit value of the s-th sample was estimated by the model constructed in this study. This represents the corresponding manually measured value. This indicates the total number of fruit samples.

[0045] (16) (17) (18) (19) In this embodiment, the principal component analysis results of the pepper and branch images provide a feasibility pre-verification for the pepper cluster pose estimation algorithm. Based on this, the algorithm flow can directly output the pepper cluster pose parameters required for harvesting, providing a kinematic benchmark for the robotic arm's harvesting planning. The extraction results of the branch's principal direction also support subsequent multi-pose harvesting strategies, realizing the joint calculation of pepper pose and relative pose. Combined with quantitative evaluation indicators, the accuracy and robustness of the method are fully verified. However, since the pepper clusters are densely distributed on the branches, relying solely on their spatial pose information is insufficient to ensure the robot's harvesting success rate. To avoid the risk of collision between the robotic arm and the branches, it is necessary to further obtain the spatial orientation relationship of the pepper clusters relative to the branches, thereby constructing a three-dimensional collision avoidance model to achieve an efficient pepper harvesting strategy.

[0046] Locating the tangent vector of the point on the branch closest to the center of the fruit cluster provides crucial information for subsequently describing the spatial orientation of the fruit cluster relative to the branch. This invention employs a central skeleton extraction and cubic B-spline curve fitting method to construct a parametric geometric model of the branch. First, the preprocessed effective point cloud of a single branch is analyzed. Principal component analysis (PCA) is performed to obtain the global principal extension direction; along this direction, ordered sampling by local neighborhood clustering is used to divide the point cloud into multiple local subsets and calculate the geometric centers, extracting the discrete skeleton center point sequence representing the direction of the branch centers. The discrete center point sequence of the branch skeleton obtained by equation (20) is obtained by using cubic B-spline curves. Perform smooth fitting to obtain the branch space curve. The curve is The points in the middle are used as control points Composed of control points and cubic B-spline basis functions Common definition. Based on fitted curves. Calculate its first derivative with respect to parameter u. Based on the fitted curve and normalized by Euclidean norm, the unit tangent vector at any position on the curve is obtained. , where n is the number of sampling points.

[0047] (20) (twenty one) (twenty two) Because the robot's depth camera captures the target from different angles, the position of the pepper clusters above the branches varies. Therefore, based on the relative position of the pepper clusters and the branches they grow on, the pepper cluster postures are categorized into five types: upward, downward, leftward, rightward, and forward. This invention establishes the spatial relationship between the pepper clusters and the branches through geometric shortest distance and constructs a locally orthogonal posture reference system to achieve a structured posture solution for the pepper clusters, such as... Figure 13 As shown. Solve for the distance function from the center of the pepper cluster ellipsoid to the spatial curve of the branch. The optimal parameters are obtained at the minimum point in the parameter interval [0,1]. Then, the optimal parameters are substituted into the defined equation of the cubic B-spline branch curve. The spatial correlation point on the branch that is closest to the center of the pepper fruit cluster in Euclidean distance is obtained through equation (23). Finally (24) the centroid coordinates of the fruit cluster were used. Subtracting the coordinates of the spatially associated point, the offset vector pointing from the fruit cluster center to the branch is calculated. This provides the core geometric basis for the subsequent construction of the local attitude reference system of the fruit cluster and the determination of the optimal feed direction of the end effector.

[0048] (twenty three) (twenty four) (25) Based on the above correlation results, the local tangential unit vector of the branch is further extracted. and radial unit vector The local tangential unit vector of the branch. As the first basis vector The radial unit vector of the pepper cluster relative to the branch As the third basis vector is defined as The second basis vector, which together with the two forms the right-handed coordinate system, is To ensure that the three basis vectors are strictly orthogonal, for Perform a reorthogonalization process to construct the local pose rotation matrix of the fruit cluster. The construction of this rotation matrix is ​​a crucial geometric basis for the robotic arm to achieve precise feeding: its third column basis vector directly defines the optimal feeding direction perpendicular to the fruit stalk direction of the end effector, fundamentally avoiding lateral interference with thorny branches and frontal collisions with the main body of the fruit cluster during the feeding process; at the same time, relying on the principal axis information of the local coordinate system of the fruit cluster provided by it, the feeding depth matching the actual geometric size of the fruit cluster can be accurately calculated, avoiding missed harvesting or fruit oil cell rupture caused by excessive feeding; it also serves as the target posture matrix of the end effector, enabling the robotic arm to automatically adjust the rotation angle according to the different orientations of the pepper clusters, achieving adaptive feeding of "one cluster, one posture"; finally, combined with the center coordinates of the fruit cluster ellipsoid, the six-degree-of-freedom target pose of the end effector can be completely defined, and a smooth, collision-free feeding trajectory can be generated through inverse kinematics algorithm, comprehensively ensuring the success rate and operation quality of multi-posture pepper harvesting.

[0049] (26) If the feeding direction is improper during the harvesting of Sichuan peppercorns, the end effector may collide with the peppercorn cluster or branches, causing peppercorns to fall off, the actuator to jam, or even damage to the branches. Therefore, when the end effector approaches the target peppercorn cluster, a feeding direction strategy needs to be planned to effectively reduce plant damage and improve the harvesting success rate. Figure 14 As shown, when the actuator feeds along direction A, its opening plane is parallel to the direction of the fruit stalk, which easily causes lateral interference with the branch, making it difficult for the shearing mechanism to reach the base of the fruit stalk. When the actuator feeds along direction B, its opening plane forms a large angle with the direction of the fruit stalk, making it easy to collide head-on with the main body of the fruit cluster. However, when feeding along direction C, that is, when the actuator's opening plane is perpendicular to the direction of the fruit stalk, it can accurately align with the base of the fruit stalk, avoiding physical interference while providing sufficient operating space for the shearing mechanism. Therefore, feeding along direction C is the optimal feeding strategy.

[0050] In the C direction, the actuator cavity can naturally enclose the entire fruit cluster, avoiding collision with the main body of the fruit cluster. Based on the geometric parameterization of the pepper cluster and branches, the radial unit vector of the pepper cluster relative to the branch is... The direction is basically the same as that of the fruit stalk, therefore the feed direction of the end effector is set. Determined as As shown in equation (26). After determining the feed direction, based on the local tangential unit vector of the branch. The radial unit vector of the branch relative to the pepper cluster This allows us to solve for the pose of the robot's end effector during assembly, such as... Figure 15 As shown in (a).

[0051] (27) The end effector attitude calculation only considers the actuator opening plane being perpendicular to the feed direction. To achieve successful shearing of the pepper clusters, the feed depth of the end effector also needs to be determined. .like Figure 15 In section (b), this invention optimizes feed length calculation by determining the spindle direction matching degree. Let the optimal feed direction be a unit vector. The three orthogonal axes of the cluster ellipsoid are unit vectors. The corresponding semi-axis length satisfies . The angle between the spindle and the feed direction is used to calculate the effective extension length of the fruit cluster in the feed direction based on this angle and the half-axis length of the corresponding spindle, providing a reliable feed termination position reference for the end effector. and The angles between the other two main axes and the feed direction provide a basis for setting the opening width of the end effector, correcting the actuator's posture around the feed axis, and avoiding interference with fruit clusters or branches during the shearing process. Calculate separately. The absolute value of the dot product with each principal axis is used, and the principal axis with the largest absolute value of the dot product is selected as the one with respect to the principal axis. The closest principal axis direction.

[0052] (28) (29) like Then the principal axis is considered With feed direction The closest and most directly adopted method is to use the axis length corresponding to this main axis as the feed depth of the fruit cluster in the feed direction. This ensures the feed depth matches the actual geometric dimensions of the fruit cluster. It also considers the center position of the fruit cluster. and feed depth The final picking position of the end effector can be obtained.

[0053] In this embodiment, to verify the operational reliability and parameter rationality of the end effector, mechanical modeling and dynamic analysis are performed on the shearing mechanism and the rotary conveying mechanism, respectively, such as... Figure 4 As shown.

[0054] First, a mechanical analysis is performed on the shearing mechanism. The shearing mechanism employs a gear-driven eccentric blade structure, where the shearing force is transmitted from the drive motor through the driving gear and the cutting gear to the moving blade. The shearing resistance of the pepper stalk is determined by the maximum shear strength of the stalk, and the required minimum shearing force... The calculation formula is as shown in equation (29): (30) in, The maximum shear strength of the pepper fruit stalk. This refers to the cross-sectional area of ​​the fruit stalk. Field trials determined that the maximum shear strength of a mature Sichuan pepper fruit stalk is approximately 1.2 MPa, and the average cross-sectional area is approximately 4.19 mm². 2 The minimum shear force required to cut the fruit stalk was calculated. Approximately 5.03 N. In this design, the output torque of the drive motor is amplified through gear transmission, resulting in the actual shearing force acting on the blade edge. The value is 7.31, which satisfies the requirement. > The design requirements ensure reliable cutting of fruit stalks of varying thicknesses while avoiding excessive shearing force that could damage the branches.

[0055] Next, a dynamic analysis of the rotary conveyor mechanism is performed: The rotary conveyor mechanism adopts a hollow drum structure with a flexible spiral guide cylinder fixed to the inner wall. The pepper fruit clusters inside the drum are subjected to the combined effects of gravity mg, centrifugal force, the normal force N of the spiral guide cylinder, and frictional force f, and are conveyed downwards along a spiral trajectory. A rigid body dynamic model of the fruit clusters on the spiral guide cylinder is established, and its rotational angular acceleration is... It satisfies the rigid body rotation law, as shown in equation (30): (31) Let be the equivalent rotational inertia of a single pepper cluster. The net external torque acting on the fruit cluster. Combining the geometric parameters of the spiral guide cylinder and the force relationship, the formula for calculating the angular acceleration of the fruit cluster moving along the spiral guide cylinder is further derived, as shown in equation (31): (32) in The coefficient of friction between the pepper fruit clusters and the flexible spiral guide cylinder. The equivalent radius of the pepper fruit cluster, It is the acceleration due to gravity. The overall tilt angle of the roller installation. The rotational angular velocity of the drum. Let be the inner radius of the drum. The force balance equation for the fruit cluster along the axial direction of the spiral guide cylinder is shown in equation (32): (33) in The inertial force component of the Sichuan pepper fruit cluster along the axial direction of the spiral guide cylinder. The component of the normal supporting force of the inner wall of the drum on the fruit cluster in the axial direction of the spiral guide cylinder. This represents the normal positive pressure exerted by the inner wall of the spiral guide cylinder on the fruit cluster. To ensure that the fruit cluster can overcome friction and be smoothly conveyed downwards, avoiding blockage, the core conveying conditions must be met as shown in equation (33): (34) The above dynamic analysis verified the rationality of the parameters of the rotary conveyor mechanism, and the helical guide cylinder lift angle was optimized. Drum speed The friction coefficient of the flexible guide tube enables stable and unobstructed transport of peppercorn clusters; at the same time, the elastic deformation of the flexible spiral guide tube can disperse local stress, effectively preventing the fruit oil cells from rupturing due to compression, thus ensuring harvest quality.

[0056] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

Claims

1. A flexible, swallowing, multi-posture pepper-harvesting robot, characterized in that, include: A multi-degree-of-freedom robotic arm is mounted on the robot's mobile chassis or base; A flexible swallowing end effector, connected to the end flange of the multi-degree-of-freedom robotic arm, is used to envelop and separate clusters of pepper fruits. A visual perception system is used to acquire RGB images and depth maps of Sichuan pepper plants in the field. The control system is electrically connected to the multi-degree-of-freedom robotic arm, the flexible swallowing end effector, and the vision perception system, respectively. The control system is configured to perform the following functions: The RGB image is processed using a multi-task deep learning model to identify clusters of pepper fruits and branches, and to output a target mask. Based on the target mask, the point cloud of pepper fruit clusters is segmented from the depth map, and the principal component analysis method is used to fit the principal axis and the central axis of the pepper fruit clusters to calculate the three-dimensional spatial pose parameters of the pepper. Based on the three-dimensional spatial posture parameters and the preset picking orientation constraints, the motion trajectory of the multi-degree-of-freedom robotic arm and the feed parameters of the flexible swallowing end effector are planned. The multi-degree-of-freedom robotic arm is controlled to move the flexible swallowing end effector to the picking position, and the flexible swallowing end effector is driven to perform a swallowing picking action.

2. The harvesting robot as described in claim 1, characterized in that, The flexible swallowing end effector includes a flexible transmission mechanism and a shearing mechanism; The shearing mechanism includes an upper ratchet, an upper pawl, a cutting gear, a spring, and a blade. The cutting gear directly meshes with the drive gear of the motor, serving as the power input for the shearing mechanism. The cutting gear meshes with the upper ratchet via gears. The fixed blade is fixedly mounted on the end face of the cutting gear, and the moving blade is hinged to the cutting gear, forming a shearing pair with the fixed blade to cut the peppercorn stem. The upper pawl engages with the motor shaft via its lower hexagonal shaft structure. The upper pawl has a built-in spring, and it meshes with the upper ratchet in one direction. The flexible transmission mechanism includes a lower ratchet, a lower pawl, and a rotating drum. The lower end of the rotating drum is rotatably connected to the flange base through a bearing. The lower ratchet is fixed at the connection between the motor shaft and the upper pawl shaft. The regular hexagon inside the lower pawl forms an interference fit with the upper pawl shaft and also forms a one-way anti-reverse structure with the lower ratchet.

3. The harvesting robot as described in claim 2, characterized in that, The rotating drum is a hollow cylindrical structure with a spiral guide tube fixed to its inner wall to guide the sheared peppercorn clusters to slide down along the spiral trajectory.

4. A method for multi-posture harvesting of Sichuan pepper based on a harvesting robot as described in any one of claims 1-3, characterized in that, include: S1: Acquire RGB images and depth maps of Sichuan pepper plants through a visual perception system, process the RGB images based on a multi-task deep learning model, identify Sichuan pepper fruit clusters and branches, and output target masks. S2: Based on the target mask, the point cloud of pepper fruit clusters is segmented from the depth map, and the principal component analysis method is used to fit the principal axis and central axis of the pepper fruit clusters to calculate the three-dimensional spatial pose parameters of the pepper. S3: Based on the three-dimensional spatial posture parameters and the preset picking orientation constraints, plan the motion trajectory of the multi-degree-of-freedom robotic arm and the feed parameters of the flexible swallowing end effector; S4: Control the multi-degree-of-freedom robotic arm to move the flexible swallowing end effector to the picking position, and drive the flexible swallowing end effector to perform a swallowing picking action.

5. The method as described in claim 4, characterized in that, The RGB image is processed using a multi-task deep learning model to identify pepper fruit clusters and branches, and a target mask is output, including: The RGB image is preprocessed and then input into a shared encoder. Multi-scale features of Sichuan pepper plants are extracted using an efficient long-distance attention network (E-ELAN) structure. Group convolution design enables different layer weights to learn diverse features of Sichuan pepper plants. The features of the pepper plant are input into a splicing-based lightweight feature fusion neck network, and a spatial pyramid pooling (SPP) module is introduced to output a fused multi-scale feature map. Based on the fused multi-scale feature map and backbone network features, parallel processing is performed using the pepper target detection head, pepper segmentation head, and branch segmentation head, respectively: The pepper target detection head receives low-level, mid-level, and high-level features from the fused multi-scale feature map. The cross-scale propagation capability of high- and low-level localization features is further enhanced through the path aggregation network (PAN) structure. Finally, the bounding box coordinates and category confidence information of pepper fruit clusters and branches are output. The high-resolution C3 feature map, which is downsampled by 8 times and output by the backbone network E-ELAN, is used by the pepper segmentation head to receive the rich detail information to achieve accurate pixel-level segmentation of small fruit clusters. The branch segmentation head receives the mid-level features from the fused multi-scale feature map and uses the integrated global context information to achieve continuous and complete segmentation of slender branches. The pepper leaf segmentation head and the branch segmentation head respectively output pixel-level masks for the corresponding categories.

6. The method as described in claim 5, characterized in that, The step of segmenting the pepper fruit cluster point cloud from the depth map based on the target mask includes: After completing the pixel-level alignment of the depth map with the RGB image, based on the pixel-level mask of the pepper fruit clusters and branches, the mask area is back-projected to the three-dimensional space through the pinhole camera model to obtain the depth value corresponding to the pixel, and the three-dimensional coordinates of the pepper fruit clusters and branches in the camera coordinate system are calculated. Based on the three-dimensional coordinates, the effective pixels within the target area are converted into a three-dimensional point cloud to obtain the initial point cloud set of pepper fruit clusters and the point cloud set of branches. For the point cloud set of pepper fruit clusters and the target point cloud set of branches, the depth values ​​of all points are collected and a depth histogram is constructed. The center of the bin with the highest frequency is defined as the depth peak. The effective depth interval is determined by the proportional windowing method with the peak as the center. The 3D point cloud with the depth value falling within the interval is retained, and the noise points with the depth value outside the effective depth interval are removed to obtain the point cloud after filtering along the depth direction.

7. The method as described in claim 6, characterized in that, The method of fitting the principal axis and central axis of the Sichuan pepper fruit cluster using principal component analysis to calculate the three-dimensional spatial orientation parameters of the Sichuan pepper specifically includes: The point cloud filtered along the depth direction is used as the input data for principal component analysis. The geometric center of the point cloud is defined as the center point of the fruit cluster ellipsoid model. The point cloud is then subjected to mean-reduction processing based on the center point to construct the covariance matrix. The covariance matrix is ​​decomposed into eigenvalues ​​to obtain three eigenvalues ​​and their corresponding unit vectors. The eigenvalues ​​are sorted according to their size. The unit vector corresponding to the largest eigenvalue is determined as the major axis direction of the fruit cluster ellipsoid model, the unit vector corresponding to the second largest eigenvalue is determined as the median axis direction, and the unit vector corresponding to the smallest eigenvalue is determined as the minor axis direction, thus forming the direction matrix of the fruit cluster ellipsoid model. The mean-free point cloud is projected onto the principal axis coordinate system consisting of the major axis, the median axis, and the minor axis. The lengths of the three semi-axes of the ellipsoid are calculated to obtain the fruit cluster ellipsoid model, which is uniquely determined by the center of the fruit cluster ellipsoid, the direction matrix, and the lengths of the semi-axes. The three orthogonal principal directions of the fruit cluster ellipsoid model are constructed as rotation matrices and represented by robot Euler angles. Combined with the position of the ellipsoid center, the position and orientation of the pepper cluster in three-dimensional space are fully described.

8. The method as described in claim 7, characterized in that, This also includes constructing a parametric geometric model of the branches, specifically including: Principal component analysis was performed on the effective point cloud of a single preprocessed branch to obtain its global principal extension direction. Ordered sampling is performed along the global main extension direction through local neighborhood clustering. The effective point cloud is divided into multiple local subsets and the geometric center of each subset is calculated. The discrete skeleton center point sequence representing the direction of the branch center is extracted. The discrete skeleton center point sequence is smoothly fitted using a cubic B-spline curve to obtain the branch space curve, and the unit tangent vector at any position is calculated based on the branch space curve.

9. The method as described in claim 8, characterized in that, Based on the three-dimensional spatial posture parameters and the preset picking orientation constraints, the motion trajectory of the multi-degree-of-freedom robotic arm and the feed parameters of the flexible swallowing end effector are planned, including: Based on the geometric relationship between the center of the pepper cluster ellipsoid and the spatial curve of the branch, the point on the branch with the closest Euclidean distance to the center of the fruit cluster is identified as the spatial correlation point. The unit tangent vector of the spatial correlation point on the fitted curve of the branch is extracted as the local tangential unit vector, and the offset vector from the center of the pepper cluster to the spatial correlation point is normalized as the radial unit vector. A right-handed orthogonal coordinate system is constructed based on the local tangential unit vector and the radial unit vector to determine the target feed direction of the flexible swallowing end effector. Based on the preset picking orientation constraint, the radial unit vector is determined as the optimal feeding direction of the flexible swallowing end effector, so that the opening plane of the flexible swallowing end effector is perpendicular to the direction of the fruit stalk, so as to avoid physical interference with the main body and branches of the pepper cluster. Perform a dot product operation between the radial unit vector and the three orthogonal principal axis unit vectors of the fruit cluster ellipsoid model, and obtain the principal axis direction with the largest absolute value of the dot product; select the semi-axis length corresponding to this principal axis as the feed depth of the fruit cluster in the optimal feed direction; Based on the center position of the pepper cluster ellipsoid, the optimal feed direction, and the feed depth of the fruit cluster in the optimal feed direction, a smooth joint space motion trajectory is generated by combining the inverse kinematics algorithm of a multi-degree-of-freedom robotic arm.

Images