A fruit and vegetable picking system and picking method based on visual and origami dual modal

Abstract

The application discloses a fruit and vegetable picking system and picking method based on vision and origami dual modalities, and the system comprises a perception module, a decision and control module and an execution module; a mechanical arm end of the execution module is provided with a picker, an origami grabbing mechanism and a cutting unit are integrated on a support of the picker, the origami grabbing mechanism is composed of a three-leaf origami structure and a picking driving mechanism, the three-leaf origami structure can realize two kinds of grabbing modes of supporting and enveloping through driving, and the cutting unit completes a cutting action corresponding to a fruit stem position; the perception module collects RGB-D image data of a working area through a depth camera and transmits the image data to the decision and control module; after the module pre-processes the image data, a target fruit and vegetable two-dimensional mask, three-dimensional geometric features and a fruit stem pose vector are obtained through an image recognition model, a grabbing mode is adaptively selected in combination with a fruit and vegetable skin hardness rating, and then a control instruction is generated according to information such as a fruit and vegetable spatial position and is sent to the execution module, so that adaptive and non-damage precise picking of the fruit and vegetable is realized.

Description

Technical Field

[0001] This invention relates to the field of agricultural robot technology, specifically to a fruit and vegetable harvesting system and method based on a dual-modal approach of vision and origami. Background Technology

[0002] With the continuous advancement of agricultural modernization, the structural shortage of rural labor is becoming increasingly prominent. The automation and intelligent upgrading of fruit and vegetable harvesting has become a core bottleneck and urgent need for the development of smart agriculture. Developing agricultural robots that can replace manual labor and achieve efficient and damage-free harvesting is of significant practical importance and economic value for ensuring consistent fruit and vegetable quality, reducing production costs, and enhancing the overall competitiveness of the fruit and vegetable industry.

[0003] From a technical perspective, a complete fruit and vegetable harvesting robot system typically consists of four core modules working together: a mobile chassis, a multi-degree-of-freedom robotic arm, a vision perception system, and an end effector. Among these, the end effector, as the terminal execution component that directly interacts with the fruit, directly determines the success rate, harvesting efficiency, and the degree of protection of the fruit's commercial value through its structural design, grasping performance, and operational adaptability. It is the key to the technological development and performance breakthroughs of fruit and vegetable harvesting robots. Currently, the mainstream designs of end effectors in the industry mainly revolve around three technical routes: rigid mechanics, negative pressure adsorption, and soft, compliant designs.

[0004] Rigid mechanical grippers, typically represented by parallel two-finger grippers and humanoid multi-finger dexterous hands, rely on high-precision position control to achieve grasping operations, possessing technical advantages such as controllable gripping force and high positioning accuracy. However, this type of structure has obvious inherent limitations when dealing with fruits and vegetables with significant differences in shape, size, and hardness under natural growth conditions: its fixed grasping trajectory and contact method are difficult to adapt to the irregular shape and contour of fruits and vegetables, easily generating excessive contact pressure in local areas of the fruit, thus causing harvesting damage such as peel breakage and soft tissue bruising; at the same time, for different types of fruits and vegetables or different fruit shapes of the same type, the gripper structure needs to be redesigned and parameters adjusted, resulting in insufficient system versatility and difficulty in achieving rapid deployment and multi-scenario adaptation.

[0005] To reduce harvesting damage, soft grippers and pneumatic network actuators based on flexible materials have become research hotspots. These end effectors can achieve flexible wrapping of fruits through passive deformation, completing gentle gripping through surface contact, thereby effectively reducing localized stress on the fruits. However, their technical shortcomings are also prominent: first, their gripping and load-bearing capacity is limited, making it difficult to stably grip large, heavy fruits; second, the controllability of the deformation process is poor and the response speed is slow, making it impossible to efficiently coordinate with high-speed, precise fruit stem cutting actions; third, most soft grippers have not achieved an integrated "grip-cut" structural design, requiring an additional cutting mechanism to separate the fruit stem after the fruit is wrapped, which can easily cause operational interference and increase the complexity of the system structure and the time required for operation.

[0006] Visual perception systems provide harvesting robots with environmental and target recognition capabilities. Based on traditional computer vision and deep learning target detection algorithms, effective identification and spatial positioning of fruits and vegetables can be achieved. However, in most existing harvesting robot systems, the visual perception module and the end effector module are loosely coupled: the visual perception module only outputs two-dimensional or three-dimensional spatial coordinate information of the fruits and vegetables, while the end effector performs grasping actions according to a preset fixed program. It lacks real-time perception and feedback on the physical characteristics of the fruit, such as its rigidity, fragility, and optimal grasping orientation, and cannot autonomously adjust the grasping strategy, working posture, and grasping force according to the actual state of the target fruit (or target fruit and vegetable). In real orchard operation scenarios with densely packed fruits and complex foliage obstruction, this rigid operation mode is prone to problems such as grasping failure, branch collisions, and damage to adjacent fruits, thus severely restricting the harvesting effect.

[0007] In summary, existing end-effector technologies for fruit and vegetable harvesting robots face significant technical challenges: rigid actuators offer high control precision but are prone to fruit damage and have poor environmental adaptability; flexible actuators provide gentle gripping but lack stability and coordination; furthermore, the lack of intelligent linkage and closed-loop feedback between the sensing system and the actuator makes it difficult to adapt to complex orchard operation scenarios. Therefore, there is an urgent need to develop a novel harvesting robot system that integrates intelligent environmental sensing, autonomous gripping strategy decision-making, and controllable rigid-flexible execution to achieve high success rates, low damage rates, and highly adaptable automated harvesting of various fruit and vegetable varieties, meeting the demands of large-scale, standardized operations in smart agriculture. Summary of the Invention

[0008] In order to overcome the shortcomings of the existing technology, the present invention provides a fruit and vegetable harvesting system based on vision and origami dual-modality. The fruit and vegetable harvesting system can automatically harvest target fruits and vegetables in complex scenarios, and the harvesting efficiency is higher and the damage rate of target fruits and vegetables is lower.

[0009] The second objective of this invention is to provide a fruit and vegetable harvesting method based on a visual and origami dual-modal approach.

[0010] The technical solution of the present invention to solve the above-mentioned technical problems is:

[0011] A fruit and vegetable harvesting system based on a dual-modal approach of vision and origami includes a perception module, a decision-making and control module, and an execution module.

[0012] The execution module includes a robotic arm and a harvester mounted on the robotic arm; the harvester includes a support, a paper-folding gripping mechanism mounted on the support, and a cutting unit; the support is installed at the end of the robotic arm; the paper-folding gripping mechanism includes a Miura origami structure mounted on the support and a harvesting drive mechanism; the Miura origami structure is mounted on a connecting frame, and the connecting frame is mounted on the support; the Miura origami structure includes multiple sets of Miura origami units mounted on the connecting frame; the multiple sets of Miura origami units are arranged curled along the central axis, with their ends... Adjacent Miura origami units are interconnected to form a cylindrical structure for wrapping target fruits and vegetables; the picking drive mechanism is used to drive the upper and / or lower layers of the Miura origami structure to unfold or retract; when only the lower layer of the Miura origami structure retracts while the upper layer remains unfolded, the Miura origami structure is in a supporting mode; when both the upper and lower layers of the Miura origami structure retract, the Miura origami structure is in an enveloping mode; the cutting unit is located on the upper or lower side of the origami gripping mechanism, corresponding to the fruit stem attachment position, and is used to cut the fruit stems of the target fruits and vegetables;

[0013] The sensing module includes a depth camera mounted on the support frame; the depth camera is used to acquire RGB-D image data of the target work area in real time and transmit the acquired RGB-D image data to the decision and control module.

[0014] The decision and control module preprocesses the received RGB-D image data and identifies the preprocessed RGB-D image data based on the built-in image recognition model to obtain the two-dimensional mask, three-dimensional geometric features, and spatial pose vector of the fruit stem of the target fruit and vegetable. Combining the extracted three-dimensional geometric features of the target fruit and vegetable with the skin hardness rating of the target fruit and vegetable, the module adaptively selects the grasping mode of the origami grasping mechanism. At the same time, based on the spatial position of the target fruit and vegetable, the spatial pose vector of the fruit stem, and the selected grasping mode of the origami grasping mechanism, the module generates control commands and sends them to the execution module.

[0015] Preferably, it also includes a multi-sensor detection module; the multi-sensor detection module is used to detect sensor information during the grasping process and transmit the sensor information to the decision and control module; the decision and control module processes the sensor information to determine whether the fruit and vegetable picking is successful; if the picking is successful, it controls the robotic arm to move the picker to the preset fruit placement position to complete the fruit placement; if the picking is unsuccessful, it controls each module to reset and re-execute the picking process.

[0016] A fruit and vegetable harvesting method based on a visual and origami dual-modal approach, employing the aforementioned visual and origami dual-modal fruit and vegetable harvesting system, comprises the following specific steps:

[0017] S1: Real-time acquisition of RGB-D image data of the target work area via a depth camera, and transmission of the acquired RGB-D image data to the decision and control module;

[0018] S2: The decision and control module preprocesses the received RGB-D image data, and then calls the image recognition model trained and optimized by a multi-category fruit and vegetable dataset to perform image segmentation and recognition on the preprocessed RGB-D image data, and obtain the two-dimensional mask, three-dimensional geometric features and fruit stem spatial pose vector of the target fruit and vegetable.

[0019] S3: Combining the extracted three-dimensional geometric features of the target fruits and vegetables, and the skin hardness rating of the target fruits and vegetables obtained through texture analysis or a preset database, the gripping mode of the origami gripping mechanism is adaptively selected by referring to the preset feature-action mapping rule library; at the same time, based on the spatial position of the target fruits and vegetables, the spatial pose vector of the fruit stem, and the selected gripping mode, the motion trajectory of the robotic arm is planned, and the motion control instructions of the robotic arm, the drive control instructions of the origami gripping mechanism, and the control instructions of the cutting unit are generated and output, and sent to the execution module.

[0020] S4: After receiving various control commands from the decision and control module, the execution module first drives the robotic arm to move along the planned trajectory, moving the harvester to the preset grasping position of the target fruit and vegetable. Based on the relative spatial pose of the fruit and stem, the global working posture of the harvester is adjusted to adapt to the harvesting requirements. Then, the paper-folding grasping mechanism is controlled to perform the corresponding grasping action. After the target fruit and vegetable is grasped stably, the cutting unit is driven to adjust the cutting angle according to the spatial pose vector of the stem and accurately cut the stem along the optimal path to achieve non-destructive separation of the target fruit and vegetable from the plant.

[0021] S5: Based on the sensor information fed back by the multi-sensor detection module, the decision and control module determines whether the fruit and vegetable picking is successful. If the picking is successful, the module controls the robotic arm to move the picker to the preset fruit placement position and drives the paper-folding gripping mechanism to reset and unfold to complete the fruit placement process. If the picking fails, the module controls each module to reset and re-executes the picking process of steps S1 to S5.

[0022] Preferably, in step S2, the preprocessing includes image distortion correction, grayscale equalization, and noise filtering.

[0023] Preferably, in step S2, the image recognition model is an instance segmentation model based on YOLOv8-Seg.

[0024] Preferably, in step S2, after obtaining the two-dimensional mask of the target fruit and vegetable, the two-dimensional mask is first aligned with the depth map in the RGB-D image data to generate a dense three-dimensional point cloud of the target fruit and vegetable; then, Euclidean clustering and statistical filtering are performed on the dense three-dimensional point cloud to remove noise points; then, the minimum bounding cube of the denoised dense three-dimensional point cloud is calculated to obtain the spatial coordinates, three-dimensional dimensions, and principal axis direction of the target fruit and vegetable; finally, the distribution characteristics of the denoised dense three-dimensional point cloud are analyzed, and the aspect ratio and sphericity of the target fruit and vegetable are calculated through principal component analysis; the spatial coordinates, three-dimensional dimensions, principal axis direction, aspect ratio, and sphericity of the target fruit and vegetable together constitute the three-dimensional geometric features of the target fruit and vegetable.

[0025] Preferably, in step S2, based on the denoised dense 3D point cloud, an attention-enhanced branch network is used to perform key region attention weighting and local feature enhancement on the dense 3D point cloud of the top region of the target fruit and vegetable. The specific steps are as follows: extract the normal distribution features of the dense 3D point cloud of the top region of the target fruit and vegetable, and simultaneously extract the color and texture features of the top region of the target fruit and vegetable; perform fusion analysis on the normal distribution features and color and texture features through the branch network to locate the attachment points of the fruit stalk and the fruit; and calculate the spatial pose vector of the fruit stalk based on the spatial coordinates of the attachment points and the normal distribution pattern of the dense 3D point cloud of the top region of the target fruit and vegetable.

[0026] Preferably, in step S3, the selection rule for the crawling modality of the feature-action mapping rule base is as follows:

[0027] When the sphericity of the target fruit or vegetable is less than a preset first threshold or the skin hardness rating is less than a preset second threshold, the decision and control module selects the support mode.

[0028] When the sphericity of the target fruit or vegetable is greater than or equal to a preset first threshold or the skin hardness rating is greater than or equal to a preset second threshold, the decision and control module selects the envelope mode.

[0029] Preferably, in step S3, the origami gripping mechanism drive control command includes a preset shrinkage diameter of the Miura origami structure and a smooth drive curve of the picking drive mechanism; the robotic arm motion control command includes optimal approach posture parameters of the robotic arm that integrate the target fruit and vegetable attachment potential.

[0030] Preferably, in step S5, the multi-sensor feedback information includes the picking pressure feedback, the current feedback of the driving component, the cutting completion feedback of the cutting unit, and the fruit posture feedback of the sensing module.

[0031] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0032] 1. This invention, through the adoption of an origami structure design inspired by Miura creases, can achieve highly adaptive gripping of fruits and vegetables of different diameters and irregular shapes, and completes a gentle and uniform enveloping clamping through surface contact, thereby significantly reducing mechanical damage during the harvesting process. At the same time, relying on the inherent crease guiding characteristics of the Miura origami structure, and with a simple driving method (such as rope drive), the diameter of the Miura origami structure can be continuously and smoothly adjusted with only a single driving input, and can be flexibly switched between two gripping modes of support and envelopment, thereby meeting the needs of different fruit postures and harvesting conditions.

[0033] 2. This invention, through the adoption of an origami structure design inspired by Miura creases, integrates the controllability, structural stability, and load-bearing capacity of rigid mechanisms with the passive adaptability, low-damage characteristics, and surface contact advantages of flexible mechanisms. This effectively solves the technical contradiction of the difficulty in achieving both performance in traditional rigid and flexible mechanisms. In addition, the origami panels based on thin sheet materials have natural lightweight characteristics. Combined with a compact transmission system (such as a rope transmission system), the fruit and vegetable harvesting system of this invention is lightweight and highly integrated, making it easy to assemble with mobile platforms such as robotic arms. This meets the requirements of agricultural robots for low inertia and high portability at the end effector.

[0034] 3. This invention solves the technical problem of existing harvesting robots having limited end effector functionality and being unable to adapt to fruits and vegetables with significant differences in shape, size, and hardness through the collaborative design of deep vision recognition and origami gripping mechanism. This invention can automatically identify and adapt to various types of fruits and vegetables such as dragon fruit, eggplant, apple, and orange with a single fruit and vegetable harvesting system, without the need to replace the end effector, thereby greatly expanding the operating range of the harvesting robot, improving the economy and practicality of the equipment, and reducing the equipment investment cost for large-scale harvesting.

[0035] 4. This invention establishes an adaptive mapping rule for "visual features - grasping modality," which solves the technical shortcomings of traditional picking methods, such as the separation of perception and execution and reliance on manually preset parameters. The fruit and vegetable picking system of this invention can automatically decide the optimal grasping strategy (supporting modality or envelope modality) based on the three-dimensional geometric features of the target fruits and vegetables acquired in real time, and generate precise control commands to complete a fully automatic intelligent closed loop from "fruit and vegetable recognition" to "grasping strategy decision" to "execution of grasping," without the need for human intervention throughout the process, which significantly improves the autonomy and intelligence level of the picking robot.

[0036] 5. This invention integrates the advantages of rigid and flexible mechanisms. Utilizing the predictable large deformation characteristics of the Miura origami structure, it designs an end effector (i.e., a harvester) that combines high controllability, high load capacity, passive adaptability, and low damage. This solves the technical contradictions of existing rigid grippers that easily damage fruits and purely soft grippers that have weak load capacity and complex control. Through simple drive, it achieves compliant surface contact gripping, thereby reducing the squeezing and friction on the surface of fruits and vegetables during the gripping process, effectively reducing the harvesting damage rate of fruits and vegetables, and ensuring the quality of harvested fruits.

[0037] 6. This invention integrates a closed-loop process of visual recognition, feature analysis, strategy decision-making, compliant grasping, and collaborative cutting. It introduces a multi-sensor real-time status judgment and feedback mechanism, which solves the technical problems of incomplete operation process, poor stability in complex farmland environments (such as light changes and foliage shading), and low single-harvest success rate of existing harvesting systems. This effectively improves the single-harvest success rate and harvesting efficiency, enhances the overall robustness of the system and its adaptability to complex farmland environments, and ensures efficient and stable operation of harvesting operations in complex scenarios. Attached Figure Description

[0038] Figure 1 This is a structural block diagram of the fruit and vegetable harvesting system based on a dual-modal approach of vision and origami according to the present invention.

[0039] Figure 2 and Figure 3 This is a structural diagram of the execution module from two different perspectives.

[0040] Figure 4 This is a structural diagram of the origami gripping mechanism and the cutting unit.

[0041] Figure 5 This is a structural diagram of the Miura Origami structure.

[0042] Figure 6 This is a schematic diagram of the elastic reset structure.

[0043] Figure 7 This is a schematic diagram of the Miura Origami structure in the support mode.

[0044] Figure 8 This is a schematic diagram of the Miura Origami structure in the envelope mode.

[0045] Figure 9 This is a schematic diagram of the Miura Origami unit in its unfolded state.

[0046] Figure 10 This is a schematic diagram of the Miura Origami unit in a folded state.

[0047] Figure 11 This is a spatial geometric model diagram of the Miura Origami unit.

[0048] Figure 12 This is a diagram of the creases in a Miura origami structure.

[0049] Figure 13 This is a schematic diagram of the structure of a Kevlar rope.

[0050] Figure 14 This is a diagram of the YOLACT network structure.

[0051] Figure 15 A flowchart illustrating the logic for selecting the capture mode for the decision-making and control module.

[0052] Figure 16 This is a logic flowchart of the fruit and vegetable harvesting method based on a visual and origami dual-modal approach according to the present invention.

[0053] Figure 17 This is a schematic diagram of the fruit and vegetable picking system based on vision and origami dual-modality of the present invention before (left) and after (right) grabbing a pineapple.

[0054] Figure 18 This is a schematic diagram of the fruit and vegetable picking system based on vision and origami dual-modality of the present invention before (left) and after (right) grasping an apple.

[0055] Figure 19 This is a schematic diagram of the fruit and vegetable picking system based on vision and origami dual-modality of the present invention before (left) and after (right) grasping an eggplant.

[0056] Figure 20 This is a schematic diagram of the fruit and vegetable picking system based on vision and origami dual-modality of the present invention before (left) and after (right) picking a bell pepper.

[0057] Figure 21 This is a schematic diagram of the fruit and vegetable picking system based on vision and origami dual-modality of the present invention before (left) and after (right) grabbing a pumpkin.

[0058] Figure 22 This is a schematic diagram of the fruit and vegetable picking system based on vision and origami dual-modality of the present invention before (left) and after (right) picking an orange.

[0059] Figure 23 This is a schematic diagram of the fruit and vegetable picking system based on vision and origami dual-modality of the present invention before (left) and after (right) picking a tomato.

[0060] Figure 24 This is a schematic diagram of the fruit and vegetable picking system based on vision and origami dual-modality of the present invention before (left) and after (right) picking dragon fruit. Detailed Implementation

[0061] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0062] Example 1

[0063] See Figures 1-24 The fruit and vegetable harvesting system based on a dual-modal approach of vision and origami of the present invention includes a perception module, a decision-making and control module, and an execution module, wherein...

[0064] The execution module includes a robotic arm 1 and a picker mounted on the robotic arm 1; the picker includes a support 10, a paper-folding gripping mechanism mounted on the support 10, and a cutting unit 4; the support 10 is mounted on the end flange of the robotic arm 1; the paper-folding gripping mechanism (i.e., Figure 16 The "origami gripping structure" includes a Miura origami structure 2 and a picking drive mechanism mounted on a support 10; the Miura origami structure 2 is mounted on a connecting frame 11 and includes multiple sets of Miura origami units mounted on the connecting frame 11, wherein the connecting frame 11 is mounted on the support 10; the multiple sets of Miura origami units are arranged curled along a central axis, and adjacent Miura origami units are connected to each other to form a cylindrical structure for wrapping target fruits and vegetables; each set of Miura origami units includes an origami panel and creases between adjacent origami panels, wherein the origami panel is made of polypropylene film; the creases include mountain creases and valley creases (such as...). Figure 12 As shown, the blue dashed lines represent valley folds, and the red solid lines represent mountain folds. The mountain folds and valley folds are arranged alternately according to the Miura fold topology, so that each origami panel can rotate relative to the corresponding fold. The picking drive mechanism is used to drive the upper and / or lower layers of the Miura origami structure to unfold or retract. When only the lower layer of the Miura origami structure 2 is retracted and the upper layer remains unfolded, the Miura origami structure 2 is in the support mode. When both the upper and lower layers of the Miura origami structure 2 are retracted, the Miura origami structure 2 is in the envelope mode. The cutting unit 4 is located on the upper or lower side of the origami gripping mechanism, corresponding to the fruit stem attachment position, and is used to cut the fruit stem of the target fruit and vegetable.

[0065] The sensing module includes a depth camera 3 mounted on the support 10; the depth camera 3 is used to acquire RGB-D image data of the target work area in real time and transmit the acquired RGB-D image data to the decision and control module.

[0066] The decision and control module preprocesses the received RGB-D image data and identifies the preprocessed RGB-D image data based on the built-in image recognition model to obtain the two-dimensional mask, three-dimensional geometric features, and spatial pose vector of the fruit stem of the target fruit and vegetable. Combining the extracted three-dimensional geometric features of the target fruit and vegetable with the skin hardness rating of the target fruit and vegetable, the module adaptively selects the grasping mode of the origami grasping mechanism. At the same time, based on the spatial position of the target fruit and vegetable, the spatial pose vector of the fruit stem, and the selected grasping mode of the origami grasping mechanism, the module generates control commands and sends them to the execution module.

[0067] In addition, the fruit and vegetable harvesting system of the present invention also includes a multi-sensor detection module, which is used to detect sensor information during the grasping process and transmit the sensor information to the decision and control module; the decision and control module processes the sensor information to determine whether the fruit and vegetable harvesting is successful; if the harvesting is successful, it controls the robotic arm 1 to move the harvester to the preset fruit placement position to complete the fruit placement; if the harvesting is unsuccessful, it controls each module to reset and re-execute the harvesting process.

[0068] In this embodiment, the Miura origami structure 2 has two adaptively switchable grasping modes: support and envelopment. Its core is a cylindrical origami actuator based on the Miura crease structure principle, driven by two independently controlled Kevlar ropes in the upper and lower structures. This allows for the use of suitable grasping modes for fruits and vegetables with different skin hardness and shapes. Specifically, the support mode is suitable for grasping fruits and vegetables with fragile skin or irregular shapes (such as peaches, dragon fruit, and tomatoes). Figure 7 As shown; in this support mode, the Miura origami structure 2 only retracts the Kevlar rope in the lower structure. The lower part of the Miura origami structure 2 contracts radially, while the upper part remains unfolded, thus forming an upward "bowl-shaped" support surface. This balances the weight of the fruit by relying on the supporting force, thereby avoiding the squeezing damage that may be caused by traditional lateral clamping. The envelope mode is suitable for fruits and vegetables with regular shapes or hard skins (such as pineapples, apples, pumpkins, etc.). Figure 8 As shown, by synchronously contracting the upper and lower layers of Kevlar ropes, the Miura Origami Structure 2 is uniformly radially contracted as a whole, thereby achieving adaptive flexible enveloping clamping of the fruit. The enveloping clamping allows the Miura Origami Structure 2 to adaptively conform to the surface of the fruit, thereby forming a large-area, low-pressure surface contact, and the clamping force is evenly distributed. This can avoid the pressure and puncture damage caused by point contact or line contact during picking, and thus effectively protect the fruit.

[0069] See Figures 1-24The origami panels include a first origami panel 201, a second origami panel 202, a third origami panel 203, a fourth origami panel 204, a fifth origami panel 205, a sixth origami panel 206, a seventh origami panel 207, and an eighth origami panel 208. The first origami panel 201 and the second origami panel 202, the third origami panel 203 and the fourth origami panel 204, the fifth origami panel 205 and the sixth origami panel 206, and the seventh origami panel 207 and the eighth origami panel 208 are arranged in pairs side-by-side. The third origami panel 203, the fourth origami panel 204, the fifth origami panel 205, and the sixth origami panel 206 are parallelogram structures, while the first origami panel 201, the second origami panel 202, the seventh origami panel 207, and the eighth origami panel 208 are triangular structures.

[0070] Valley creases are provided between the lower edge of the first origami panel 201 and the upper edge of the third origami panel 203, and between the lower edge of the second origami panel 202 and the upper edge of the fourth origami panel 204, respectively; mountain creases are provided between the side edge of the first origami panel 201 and the side edge of the second origami panel 202 in the adjacent Miura origami unit, and between the side edge of the second origami panel 202 and the side edge of the first origami panel 201 in the adjacent Miura origami unit, respectively.

[0071] Mountain folds are provided between the lower edge of the third origami panel 203 and the upper edge of the fifth origami panel 205, and between the lower edge of the fourth origami panel 204 and the upper edge of the sixth origami panel 206, respectively; valley folds are provided between the side edge of the third origami panel 203 and the side edge of the fourth origami panel 204 in the adjacent Miura origami unit, and between the side edge of the fourth origami panel 204 and the side edge of the third origami panel 203 in the adjacent Miura origami unit, respectively.

[0072] Valley creases are provided between the lower edge of the fifth origami panel 205 and the upper edge of the seventh origami panel 207, and between the lower edge of the sixth origami panel 206 and the upper edge of the eighth origami panel 208, respectively; mountain creases are provided between the side edge of the fifth origami panel 205 and the side edge of the sixth origami panel 206 in the adjacent Miura origami unit, and between the side edge of the sixth origami panel 206 and the side edge of the fifth origami panel 205 in the adjacent Miura origami unit, respectively.

[0073] Among them, by Figures 9-11 From the geometric relationship of Miura Origami Structure 2, it can be seen that the following relationship is satisfied during the folding process:

[0074] ;

[0075] ;

[0076] ;

[0077] ;

[0078] ;

[0079] ;

[0080] ;

[0081] ;

[0082] In the formula: n is the number of Miura Origami units along the central axis; and θ represents the lengths of the short and long sides of the origami panel of the parallelogram structure in each Miura origami unit; θ is the acute angle of the origami panel of the parallelogram structure in each Miura origami unit. and The crease angle during the folding process; and These are the folding angles during the folding process; , and These are the height, width, and length of each Miura origami unit; This is the inner diameter of the Miura origami structure, i.e., the maximum clamping radius; The outer diameter of the Miura origami structure;

[0083] The size of the Miura origami structure 2 can be adjusted according to the diameter of the object being grasped by changing the parameters of a single Miura origami unit and the total number of Miura origami units.

[0084] In this embodiment, fruits and vegetables with varying skin hardness, stem toughness, and size, such as pineapple, apple, orange, pumpkin, dragon fruit, tomato, bell pepper, and eggplant, were selected as harvesting targets (e.g., Figures 17-24 As shown in the figure, the fruit and vegetable product with the largest longitudinal diameter is the eggplant, with a longitudinal diameter of 180mm-250mm, while the one with the largest transverse diameter is the pineapple. Taking the pineapple grown in Shenwan Town, Zhongshan City, Guangdong Province as an example, the transverse diameter is 90mm-130mm. Since the origami gripping mechanism in this invention can use both support mode and envelope mode for picking and holding, and considering that the transverse diameter is the main design parameter, it should be made so that... ≥65mm; therefore, set r1=65mm. =45°, n=12, a=34mm, b=50mm, use 13 Miura origami units to create Miura origami structure 2, the crease diagram is as follows. Figure 12 As shown.

[0085] The thickness selection of the Miura Origami Structure 2 needs to balance folding performance and load-bearing stiffness. Excessive thickness can easily lead to plastic tearing during folding, while insufficient thickness can cause instability after unfolding due to insufficient stiffness. Therefore, to determine the optimal thickness, parametric pre-studies were conducted using ABAQUS to simulate the folding process of polypropylene films of different thicknesses (0.1mm-0.5mm) under the same crease pattern. Simulations showed that when the thickness was greater than 0.4mm, the equivalent plastic strain at the crease of the Miura Origami Structure 2 began to exceed the material's yield limit, thus posing a risk of tearing. Conversely, when the thickness was less than 0.2mm, the stiffness of the Miura Origami Structure 2 in the unfolded state was insufficient, making it prone to instability. Therefore, the simulation results provided an optimal range for the thickness selection of the Miura Origami Mechanism: 0.2mm-0.35mm. Considering commercial material specifications, 0.3mm thick polypropylene film is readily available; therefore, 0.3mm thick polypropylene material was selected to fabricate the Miura Origami Structure 2.

[0086] Therefore, the Miura origami structure 2 in this invention uses a polypropylene film with a length × width of 625.08mm × 124.04mm and a thickness of 0.3mm, pressed as follows: Figure 12 The fold lines shown are folded into a cylindrical shape. To effectively improve the rigidity of the structure while maintaining overall lightweight, a 0.1mm thick stainless steel sheet is covered on the polypropylene film to enhance the rigidity of the Miura origami structure 2. The stainless steel sheet is slightly smaller than the corresponding origami panel and has a 0.1mm gap between it and the corresponding fold line. Each parallelogram structure module (i.e., the third origami panel 203, the fourth origami panel 204, the fifth origami panel 205, and the sixth origami panel 206) has a 4mm diameter hole at its geometric center. The hollow rivet 6 has a through hole for mounting the hollow rivet 6, which has a through hole inside for the Kevlar rope to pass through. By placing the hollow rivet 6 at the center of each parallelogram-shaped origami panel, the stainless steel sheet is installed in the origami panel through the hollow rivet 6. This increases the rigidity of the structure and effectively avoids interference between the two materials (stainless steel sheet and polypropylene film) at the crease during movement. That is, near the crease line, the two layers of materials can separate slightly, thus effectively avoiding movement interference and deformation jamming. Finally, the Miura origami structure 2 made in the above manner weighs 16.7g. In subsequent tests, the prototype underwent more than 200 continuous gripping and releasing cycles. Throughout the process, the origami gripping mechanism moved smoothly without any jamming, which indirectly proves the reliability of the above design.

[0087] See Figures 1-24Elastic reset structures are provided between the third origami panel 203 and the fourth origami panel 204, between the fifth origami panel 205 and the sixth origami panel 206 in the adjacent Miura origami unit, and between the sixth origami panel 206 and the fifth origami panel 205 in the adjacent Miura origami unit. The elastic reset structure includes two sets of mounting seats 7 and a spring 5 installed between the two sets of mounting seats 7. The two sets of mounting seats 7 are respectively installed on the corresponding origami panels by hollow rivets 6. The two ends of the spring 5 are respectively fixedly installed on the two sets of mounting seats 7.

[0088] In this embodiment, there are two sets of picking drive mechanisms, which are respectively arranged on the upper and lower layers of the Miura origami structure 2. Each set of picking drive mechanisms includes a Kevlar rope and a winding drive mechanism for winding and releasing the Kevlar rope. The winding drive mechanism includes two sets, each set of winding drive mechanism including a winding motor 8 and a winding wheel 9 arranged on the main shaft of the winding motor 8. The winding wheel 9 is provided with a winding groove. The two ends of the Kevlar rope are respectively connected to the winding wheels 9 in the two sets of winding drive mechanisms, and the middle part passes through the through holes in the hollow rivets 6 on the third origami panel 203 and the fourth origami panel 204, or the fifth origami panel 205 and the sixth origami panel 206 in sequence along the circumferential direction of the Miura origami structure 2. The part of the Kevlar rope located on the outside of the Miura origami structure 2 passes through the spring 5.

[0089] With the above settings, the Miura origami structure 2 remains in an unfolded state under the force of the spring 5, at which time the diameter of the Miura origami structure 2 is relatively large. The winding motor 8 drives the winding wheel 9 to rotate, thereby winding the Kevlar rope and pulling the Miura origami structure 2 to achieve a retracting action. At this time, the diameter of the Miura origami structure 2 decreases accordingly. When the winding force of the Kevlar rope is removed, the restoring force of the spring 5 can drive the Miura origami structure 2 to automatically return to the unfolded state, thereby achieving an adaptive increase in the diameter of the Miura origami structure 2. Among them, the two Kevlar ropes can be independently wound and unwound. By independently controlling the winding and unwound of the two Kevlar ropes, the unfolding and unwound control of the upper and lower layers of the Miura origami structure 2 can be achieved.

[0090] See Figures 1-24The cutting unit 4 includes an electric saw 401 mounted on the support 10 and a cutting drive mechanism 402 for driving the electric saw 401 to move along an axis perpendicular to the Miura origami structure 2. The electric saw 401 can be implemented with reference to existing devices. The cutting drive mechanism 402 can be a combination of a motor and a lead screw transmission mechanism to drive the electric saw 401. The depth camera 2 identifies and locates the target fruits and vegetables, and then the winding motor 8 drives the winding wheel 9 to wind and unwind the Kevlar rope, thereby controlling the unfolding and retraction of the Miura origami structure 2. Finally, it works in conjunction with the cutting unit 4 to complete the tasks of autonomously identifying, locating, enveloping and grasping, and cutting and harvesting the target fruits and vegetables. Through a series of grasping tests, it can be confirmed that the Miura origami structure 2 of this invention can adaptively grasp fruits and vegetables with irregular shapes, different diameters (42-130mm), and different skin hardness. The grasping success rate of various fruits and vegetables reaches 96%, with an average time of 7 seconds.

[0091] In addition, to further improve the versatility of the harvester, the harvester adopts a modular design; depending on the relative position of the fruit stalk and the fruit (the fruit is above or below the fruit stalk), the harvester can be installed at the end of the robotic arm 1 via the bracket 10 in a forward position or after rotating 180°, for example, the harvester is located above or below the cutting unit 4; after installation, in order to maintain the normal orientation of the depth camera 3, the depth camera 3 can be reinstalled and adjusted back to its original forward position; furthermore, the harvester can also be adapted and replaced according to the physical characteristics of the fruit stalks of different crops.

[0092] In summary, the Miura Origami Structure 2 of this invention can flexibly switch between two grasping modes: support and envelopment. It can adaptively grasp fruits and vegetables with irregular shapes, varying diameters, and different skin hardness, thereby effectively improving the versatility and adaptability of grasping. The harvester in this invention adopts a reconfigurable modular design, which can be flexibly set according to the different fruit and stem attachment postures and differences in the physical characteristics of the stems, thereby significantly improving the overall versatility and engineering practicality of this invention. In addition, the Miura Origami Structure 2 of this invention weighs only 16g. Weighing only 7g, this machine boasts excellent gripping performance, enabling stable and damage-free gripping of fruits and vegetables with diameters ranging from 42 to 130mm. Test results show that the prototype has a maximum load capacity of 1500g, a load-to-weight ratio of 89.82, and a 96% success rate in gripping test subjects, thus achieving a good balance between lightweight and high performance. Therefore, this invention enables damage-free gripping and precise cutting during fruit and vegetable harvesting, and has advantages such as lightweight structure, adaptability to different fruit shapes, gentle gripping, and flexible operation, making it suitable for automated harvesting scenarios of various fruits and vegetables.

[0093] Finally, compared with the prior art, the fruit and vegetable harvesting system of the present invention has the following advantages:

[0094] Firstly, the fruit and vegetable harvesting system of the present invention realizes the transformation of the grasping method from "line / point contact" to "surface contact", thereby significantly reducing the pressure on the fruit and vegetable skin to avoid mechanical damage to the fruit and vegetable; in addition, the fruit and vegetable harvesting system of the present invention can adapt to a wide range of fruit and vegetable diameters, and can also be adapted to grasping irregularly shaped fruits and vegetables, with better grasping versatility and adaptability.

[0095] Secondly, the fruit and vegetable harvesting system of the present invention uses a very simple drive and transmission scheme (for example, a single winding motor, a single rope, or symmetrically arranged double ropes) to replace the traditional complex multi-joint or multi-drive system, thereby significantly reducing the manufacturing cost, control complexity, and overall weight of the mechanism, while simultaneously achieving continuous adjustment of the gripping diameter and flexible switching of the gripping mode, thus adapting to different harvesting conditions.

[0096] Thirdly, the fruit and vegetable harvesting system of the present invention solves the core contradictions of traditional rigid clamps being prone to damaging fruits and vegetables and traditional flexible clamps having weak load capacity and being difficult to control from the perspective of structural principle. It achieves a comprehensive performance breakthrough in terms of lightweight, high load capacity, low damage and high adaptability, thereby improving the practicality of the fruit and vegetable harvesting system of the present invention. The fruit and vegetable harvesting system of the present invention can simultaneously achieve an extremely high load ratio (e.g., >89:1), an extremely low self-weight (e.g., <20g), a high grasping success rate (e.g., >96%) and a low damage rate.

[0097] See Figures 1-24 The present invention provides a fruit and vegetable harvesting method based on a visual and origami dual-modal approach, comprising the following steps:

[0098] S1: Real-time acquisition of RGB-D image data of the target work area via a depth camera, and transmission of the acquired RGB-D image data to the decision and control module;

[0099] S2: The decision and control module preprocesses the received RGB-D image data, including image distortion correction, grayscale equalization and noise filtering; then it calls the image recognition model trained and optimized by a multi-category fruit and vegetable dataset to perform image segmentation and recognition on the preprocessed RGB-D image data to obtain the two-dimensional mask, three-dimensional geometric features and fruit stem spatial pose vector of the target fruit and vegetable.

[0100] In this embodiment, the decision and control module preprocesses the received RGB-D image data and then calls the YOLOv8-seg instance segmentation model, trained and optimized using a multi-category fruit and vegetable dataset, to segment and identify the preprocessed RGB-D image data to obtain a two-dimensional mask of the target fruit or vegetable. The YOLOv8-seg instance segmentation model is an extension of the YOLOv8 object detection model, capable of simultaneously performing object detection, semantic segmentation, and category recognition. Its overall structure consists of three parts: a backbone network, a neck feature fusion layer, and a head prediction layer.

[0101] The backbone network is used to extract image features. The resolution of the input image gradually decreases while the number of channels gradually increases. It mainly includes the CBS convolutional group (Conv+BN+SiLU), the C2f multi-branch feature extraction module with residual branches, and the SPPF spatial pyramid pooling module. Among them, the CBS convolutional group is responsible for convolution calculation and nonlinear mapping, the C2f multi-branch feature extraction module keeps the input and output channels consistent, and the SPPF spatial pyramid pooling module is used to enhance the receptive field without changing the output size.

[0102] The Neck feature fusion layer is used to fuse 80×80, 40×40, and 20×20 multi-scale features output by the Backbone backbone network. It constructs a feature pyramid structure through upsampling and Concat operations. The C2f multi-branch feature extraction module it contains does not contain residual branches and has a variable number of channels.

[0103] The Head prediction layer adds a Mask branch to the regular detection branch. Specifically, it includes a Box branch (3 scales) for predicting the location of the target box, a Cls branch (3 scales) for predicting the fruit and vegetable category, a MaskCoefficients branch for outputting mask coefficients (outputting (1,32,H,W) mask coefficients per scale), and a Prototype branch for generating the prototype mask (generating a (1,32,160,160) prototype mask from an 80×80 feature map, which is then upsampled to match the resolution of the original image).

[0104] The segmentation principle of the YOLOv8-seg instance segmentation model in this embodiment is based on the YOLACT method. Specifically, a 32-channel prototype mask is generated through the Prototype branch, and 32 weights are assigned to each detection box through the MaskCoefficients branch. The prototype mask is multiplied by the corresponding weights and then added to obtain the prediction mask. The prediction mask is then cropped and binarized to obtain the instance segmentation mask of the target fruit and vegetable, thus providing a foundation for subsequent 3D geometric feature extraction. Figure 14The diagram shows the structure of the YOLACT network. The YOLOv8-seg model uses the principle shown in the YOLACT network structure to segment instances and obtain a two-dimensional mask of the target fruits and vegetables.

[0105] After obtaining the two-dimensional mask of the target fruits and vegetables, extract the three-dimensional geometric features according to the following process:

[0106] First, the 2D mask is aligned with the depth map in the RGB-D image data to generate a dense 3D point cloud of the target fruit and vegetable. Next, Euclidean clustering and statistical filtering are performed on the dense 3D point cloud to remove noise points. Then, the minimum bounding cube of the denoised dense 3D point cloud is calculated to obtain the spatial coordinates, 3D dimensions, and principal axis orientation of the target fruit and vegetable. Finally, the distribution characteristics of the denoised dense 3D point cloud are analyzed, and the aspect ratio and sphericity of the target fruit and vegetable are calculated through principal component analysis. The spatial coordinates, 3D dimensions, principal axis orientation, aspect ratio, and sphericity of the target fruit and vegetable together constitute its 3D geometric features.

[0107] Based on the denoised dense 3D point cloud, an attention-enhanced branch network is used to focus on and enhance the features of the dense 3D point cloud in the top region of the target fruit and vegetable. The specific steps are as follows: extract the normal distribution features of the dense 3D point cloud in the top region of the target fruit and vegetable, and then simultaneously extract the color and texture features of the top region of the target fruit and vegetable; perform fusion analysis on the normal distribution features and color and texture features through the branch network to locate the attachment points of the fruit stalk and the fruit; and calculate the spatial pose vector of the fruit stalk based on the spatial coordinates of the attachment points and the normal distribution pattern of the dense 3D point cloud in the top region of the target fruit and vegetable.

[0108] S3: Combining the extracted three-dimensional geometric features of the target fruits and vegetables, and the skin hardness rating of the target fruits and vegetables obtained through texture analysis or a preset database, the gripping mode of the origami gripping mechanism is adaptively selected by referring to the preset feature-action mapping rule library; at the same time, based on the spatial position of the target fruits and vegetables, the spatial pose vector of the fruit stem, and the selected gripping mode, the motion trajectory of the robotic arm is planned, and the motion control instructions of the robotic arm, the drive control instructions of the origami gripping mechanism, and the control instructions of the cutting unit are generated and output, and sent to the execution module.

[0109] In this embodiment, the crawling modality selection rule of the feature-action mapping rule base is as follows:

[0110] When the sphericity of the target fruit or vegetable is less than a preset first threshold or the skin hardness rating is less than a preset second threshold, the decision and control module selects the support mode; when the sphericity of the target fruit or vegetable is greater than or equal to the preset first threshold or the skin hardness rating is greater than or equal to the preset second threshold, the decision and control module selects the envelope mode.

[0111] The grasping modality selection rules in the aforementioned feature-action mapping rule base can significantly improve the robustness and adaptability of the system, making the grasping strategy more closely match the physical characteristics of the target fruits and vegetables; for example... Figure 15 As shown, the decision-making process is clear and rigorous, and the decision results will be transformed into specific control commands in real time, including the preset shrinkage diameter of the Miura origami structure, the smooth drive curve of the winding motor, and the optimal approach posture parameters of the robotic arm that integrates the ecological potential of the target fruits and vegetables, thereby forming a collaborative closed loop from perception, decision-making to execution.

[0112] S4: After receiving various control commands from the decision and control module, the execution module first drives the robotic arm to move along the planned trajectory, moving the harvester to the preset grasping position of the target fruit and vegetable. Based on the relative spatial pose of the fruit and stem, the global working posture of the harvester is adjusted to adapt to the harvesting requirements. Then, the paper-folding grasping mechanism is controlled to perform the corresponding grasping action. During the compliant grasping process, the current feedback of the winding motor needs to be monitored in real time to achieve adaptive force control. After the target fruit and vegetable is grasped stably, the depth camera accurately locates the position of the stem, drives the cutting unit to adjust the cutting angle according to the spatial pose vector of the stem, and accurately cuts the stem along the optimal path to achieve non-destructive separation of the target fruit and vegetable from the plant.

[0113] The cutting unit of this invention consists of a protective cover, a high-speed micro electric saw housed within the protective cover, and a cutting drive mechanism for driving the high-speed micro electric saw. It can adjust the cutting angle based on the spatial pose vector of the fruit stem provided by a sensing module (i.e., a depth camera) to achieve precise separation. Crucially, the entire end effector of the robotic arm can adaptively adjust its global posture based on the relative spatial pose of the fruit and stem (e.g., pineapple fruit on top or apple fruit on the bottom), driven by the robotic arm (i.e., switching between "grasping on top, cutting on bottom" or the opposite working posture). This enables universal, non-destructive, and efficient harvesting of fruits and vegetables with various growth postures. Figure 16 This is a flowchart illustrating the overall operation of the origami gripping mechanism.

[0114] S5: Combining the sensor information fed back by the multi-sensor detection module, the decision and control module determines whether the fruit and vegetable picking is successful. If the picking is successful, the module controls the robotic arm to move the picker to the preset fruit placement position and drives the paper-folding gripping mechanism to reset and unfold to complete the fruit placement process. If the picking fails, the module controls each module to reset and re-executes the picking process of steps S1 to S5.

[0115] In this embodiment, the multi-sensor feedback information includes the gripping pressure feedback of the end effector, the current feedback of the drive component, the cutting completion feedback of the cutting unit, and the fruit posture feedback of the sensing module.

[0116] Finally, the entire harvesting process of this invention is managed by a state machine, which enables multi-task coordination and exception handling, thereby ensuring the robustness of the system in complex environments. This invention deeply integrates advanced machine vision, intelligent decision-making and innovative mechanism design, realizing the intelligentization of the entire process of fruit and vegetable harvesting from perception to execution, that is, forming an intelligent closed loop of perception-decision-execution-feedback.

[0117] This invention, by constructing an intelligent closed-loop system of "visual recognition - dual-modal adaptive grasping - collaborative operation," effectively solves the technical defects of existing fruit and vegetable harvesting robots, such as poor versatility, high damage rate, weak autonomy, and insufficient operational reliability. Compared with existing technologies, the fruit and vegetable harvesting method of this invention has the following advantages:

[0118] Firstly, this invention solves the technical problem of existing harvesting robots having limited end effector functionality and being unable to adapt to fruits and vegetables with significant differences in shape, size, and hardness through the collaborative design of depth vision recognition and origami gripping mechanism. This invention can automatically identify and adapt to various types of fruits and vegetables such as dragon fruit, eggplant, apple, and orange with a single fruit and vegetable harvesting system, without the need to replace the end effector, thereby greatly expanding the operating range of the harvesting robot, improving the economy and practicality of the equipment, and reducing the equipment investment cost for large-scale harvesting.

[0119] Secondly, this invention establishes an adaptive mapping rule of "visual features - grasping modality", which solves the technical shortcomings of traditional picking methods, such as the separation of perception and execution and reliance on manually preset parameters. The fruit and vegetable picking system of this invention can automatically decide the optimal grasping strategy (supporting modality or envelope modality) based on the three-dimensional geometric features of the target fruits and vegetables acquired in real time, and generate precise control commands to complete a fully automatic intelligent closed loop from "fruit and vegetable recognition" to "grasping strategy decision" to "execution grasping", without the need for human intervention throughout the process, which significantly improves the autonomy and intelligence level of the picking robot.

[0120] Thirdly, this invention integrates the advantages of rigid and flexible mechanisms. Utilizing the predictable large deformation characteristics of the Miura origami structure, it designs an end effector that combines high controllability, high load capacity, passive adaptability, and low damage. This solves the technical contradictions of existing rigid grippers that easily damage fruits and purely soft grippers that have weak load capacity and complex control. Through simple drive, it achieves compliant surface contact gripping, thereby reducing the squeezing and friction on the surface of fruits and vegetables during gripping, effectively reducing the harvesting damage rate of fruits and vegetables, and ensuring the quality of harvested fruits.

[0121] Fourth, this invention integrates a closed-loop process of visual recognition, feature analysis, strategy decision-making, compliant grasping, and collaborative cutting. It introduces a multi-sensor real-time status judgment and feedback mechanism, which solves the technical problems of incomplete operation process, poor stability in complex farmland environments (such as light changes and foliage shading), and low single-harvest success rate of existing harvesting systems. This effectively improves the single-harvest success rate and harvesting efficiency, enhances the overall robustness of the system and its adaptability to complex farmland environments, and ensures efficient and stable operation of harvesting operations in complex scenarios.

[0122] The above are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above content. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A fruit and vegetable harvesting system based on a dual-modal approach of vision and origami, characterized in that, It includes a perception module, a decision-making and control module, and an execution module, among which, The execution module includes a robotic arm and a harvester mounted on the robotic arm; the harvester includes a support, a paper-folding gripping mechanism mounted on the support, and a cutting unit; the support is installed at the end of the robotic arm; the paper-folding gripping mechanism includes a Miura origami structure mounted on the support and a harvesting drive mechanism; the Miura origami structure is mounted on a connecting frame, and the connecting frame is mounted on the support; the Miura origami structure includes multiple sets of Miura origami units mounted on the connecting frame; the multiple sets of Miura origami units are arranged curled along the central axis, with their ends... Adjacent Miura origami units are interconnected to form a cylindrical structure for wrapping target fruits and vegetables; the picking drive mechanism is used to drive the upper and / or lower layers of the Miura origami structure to unfold or retract; when only the lower layer of the Miura origami structure retracts while the upper layer remains unfolded, the Miura origami structure is in a supporting mode; when both the upper and lower layers of the Miura origami structure retract, the Miura origami structure is in an enveloping mode; the cutting unit is located on the upper or lower side of the origami gripping mechanism, corresponding to the fruit stem attachment position, and is used to cut the fruit stems of the target fruits and vegetables; The sensing module includes a depth camera mounted on the support frame; the depth camera is used to acquire RGB-D image data of the target work area in real time and transmit the acquired RGB-D image data to the decision and control module. The decision and control module preprocesses the received RGB-D image data and identifies the preprocessed RGB-D image data based on the built-in image recognition model to obtain the two-dimensional mask, three-dimensional geometric features, and spatial pose vector of the fruit stem of the target fruit and vegetable. Combining the extracted three-dimensional geometric features of the target fruit and vegetable with the skin hardness rating of the target fruit and vegetable, the module adaptively selects the grasping mode of the origami grasping mechanism. At the same time, based on the spatial position of the target fruit and vegetable, the spatial pose vector of the fruit stem, and the selected grasping mode of the origami grasping mechanism, the module generates control commands and sends them to the execution module.

2. The fruit and vegetable harvesting system based on a dual-modal approach of vision and origami as described in claim 1, characterized in that, It also includes a multi-sensor detection module; the multi-sensor detection module is used to detect sensor information during the grasping process and transmit the sensor information to the decision and control module; the decision and control module processes the sensor information to determine whether the fruit and vegetable picking is successful; if the picking is successful, it controls the robotic arm to move the picker to the preset fruit placement position to complete the fruit placement. If the harvesting is deemed unsuccessful, the control modules will be reset and the harvesting process will be re-executed.

3. A fruit and vegetable harvesting method based on a visual and origami dual-modal approach, characterized in that, The specific steps of the fruit and vegetable harvesting system based on a dual-modal approach of vision and origami as described in claim 2 are as follows: S1: Real-time acquisition of RGB-D image data of the target work area via a depth camera, and transmission of the acquired RGB-D image data to the decision and control module; S2: The decision and control module preprocesses the received RGB-D image data, and then calls the image recognition model trained and optimized by a multi-category fruit and vegetable dataset to perform image segmentation and recognition on the preprocessed RGB-D image data, and obtain the two-dimensional mask, three-dimensional geometric features and fruit stem spatial pose vector of the target fruit and vegetable. S3: Combining the extracted three-dimensional geometric features of the target fruits and vegetables, and the skin hardness rating of the target fruits and vegetables obtained through texture analysis or a preset database, the gripping mode of the origami gripping mechanism is adaptively selected by referring to the preset feature-action mapping rule library; at the same time, based on the spatial position of the target fruits and vegetables, the spatial pose vector of the fruit stem, and the selected gripping mode, the motion trajectory of the robotic arm is planned, and the motion control instructions of the robotic arm, the drive control instructions of the origami gripping mechanism, and the control instructions of the cutting unit are generated and output, and sent to the execution module. S4: After receiving various control commands from the decision and control module, the execution module first drives the robotic arm to move along the planned trajectory, moving the harvester to the preset grasping position of the target fruit and vegetable. Based on the relative spatial pose of the fruit and stem, the global working posture of the harvester is adjusted to adapt to the harvesting requirements. Then, the paper-folding grasping mechanism is controlled to perform the corresponding grasping action. After the target fruit and vegetable is grasped stably, the cutting unit is driven to adjust the cutting angle according to the spatial pose vector of the stem and accurately cut the stem along the optimal path to achieve non-destructive separation of the target fruit and vegetable from the plant. S5: Based on the sensor information fed back by the multi-sensor detection module, the decision and control module determines whether the fruit and vegetable picking is successful. If the picking is successful, the module controls the robotic arm to move the picker to the preset fruit placement position and drives the paper-folding gripping mechanism to reset and unfold to complete the fruit placement process. If the picking fails, the module controls each module to reset and re-executes the picking process of steps S1 to S5.

4. The fruit and vegetable harvesting method based on a dual-modal approach of vision and origami as described in claim 3, characterized in that, In step S2, the preprocessing includes image distortion correction, grayscale equalization, and noise filtering.

5. The fruit and vegetable harvesting method based on a visual and origami dual-modal approach according to claim 3, characterized in that, In step S2, the image recognition model is an instance segmentation model based on YOLOv8-Seg.

6. The fruit and vegetable harvesting method based on a visual and origami dual-modal approach according to claim 3, characterized in that, In step S2, after obtaining the two-dimensional mask of the target fruit and vegetable, the two-dimensional mask is first aligned with the depth map in the RGB-D image data to generate a dense three-dimensional point cloud of the target fruit and vegetable. Then, Euclidean clustering and statistical filtering are performed on the dense three-dimensional point cloud to remove noise points. Next, the minimum bounding cube of the denoised dense three-dimensional point cloud is calculated to obtain the spatial coordinates, three-dimensional dimensions, and principal axis orientation of the target fruit and vegetable. Finally, the distribution characteristics of the denoised dense three-dimensional point cloud are analyzed, and the aspect ratio and sphericity of the target fruit and vegetable are calculated through principal component analysis. The spatial coordinates, three-dimensional dimensions, principal axis orientation, aspect ratio, and sphericity of the target fruit and vegetable together constitute the three-dimensional geometric features of the target fruit and vegetable.

7. The fruit and vegetable harvesting method based on a visual and origami dual-modal approach according to claim 3, characterized in that, In step S2, based on the denoised dense 3D point cloud, an attention-enhanced branch network is used to perform key region attention weighting and local feature enhancement on the dense 3D point cloud of the top region of the target fruit and vegetable. The specific steps are as follows: extract the normal distribution features of the dense 3D point cloud of the top region of the target fruit and vegetable, and simultaneously extract the color texture features of the top region of the target fruit and vegetable; perform fusion analysis on the normal distribution features and color texture features through the branch network to locate the attachment points of the fruit stalk and the fruit; and calculate the spatial pose vector of the fruit stalk based on the spatial coordinates of the attachment points and the normal distribution pattern of the dense 3D point cloud of the top region of the target fruit and vegetable.

8. The fruit and vegetable harvesting method based on a visual and origami dual-modal approach according to claim 3, characterized in that, In step S3, the selection rule for the crawling modality in the feature-action mapping rule base is as follows: When the sphericity of the target fruit or vegetable is less than a preset first threshold or the skin hardness rating is less than a preset second threshold, the decision and control module selects the support mode. When the sphericity of the target fruit or vegetable is greater than or equal to a preset first threshold or the skin hardness rating is greater than or equal to a preset second threshold, the decision and control module selects the envelope mode.

9. The fruit and vegetable harvesting method based on a visual and origami dual-modal approach according to claim 3, characterized in that, In step S3, the origami gripping mechanism drive control command includes the preset shrinkage diameter of the Miura origami structure and the smooth drive curve of the picking drive mechanism; the robotic arm motion control command includes the optimal approach posture parameters of the robotic arm that integrate the target fruit and vegetable attachment potential.

10. The fruit and vegetable harvesting method based on a visual and origami dual-modal approach according to claim 3, characterized in that, In step S5, the multi-sensor feedback information includes the picking pressure feedback, the current feedback of the driving component, the cutting completion feedback of the cutting unit, and the fruit posture feedback of the sensing module.

Images