Robotic fruit harvesting system and method

Abstract

Embodiments generally relate to a robotic fruit picking device and method for picking fruit. An exemplary device includes a chassis, a robotic arm supported by the chassis and having an end effector, the end effector comprising a plurality of grippers and an extendable suction element, a vision system carried by the chassis and configured to identify fruits for picking, and a control system carried by the chassis and in communication with the vision system to control the robotic arm to pick the identified fruits using the end effector. The control system may be configured to operate in a first mode to control the extendable suction element to extend the suction element towards the fruit. The control system may be further configured to operate in a second mode after the first mode to maintain contact with the fruit by applying a suction force while allowing the suction element to freely extend or retract based on a movement of the fruit.

Description

[Technical field]

[0001] Embodiments relate generally to robotic devices for harvesting fruit. In particular, embodiments relate to end effectors, image processing, and position determination for robotic fruit picking devices. [Background technology]

[0002] The adoption of robotic devices for fruit harvesting can provide economic benefits and reduce wear and tear on human workers under harsh working conditions. For example, robotic devices can increase production levels in orchards due to reduced labor costs. However, existing devices and methods for fruit harvesting tend to be inefficient and unreliable.

[0003] It would be desirable to address or ameliorate one or more of the shortcomings or disadvantages associated with conventional devices and methods for robotic fruit harvesting, or at least provide a useful alternative thereto.

[0004] Throughout this specification, the word "comprise" or variations such as "comprises" or "comprising" will be understood to mean the inclusion of a stated element, component or step, or group of elements, components or steps, but not the exclusion of any other elements, components or steps, or group of elements, components or steps.

[0005] Any discussion of documents, acts, materials, devices, articles or the like which has been included in this specification should not be construed as an admission that all or any part of such content formed part of the prior art document existing prior to the priority date of each of the appended claims or was common general knowledge in the art relevant to the present disclosure. Summary of the Invention

[0006] Some embodiments relate to a robotic fruit picking device that includes a chassis, a robotic arm supported by the chassis and having an end effector, which may include a plurality of grippers and an extendable suction element, a vision system carried by the chassis, which may be configured to identify fruit for picking, and a control system carried by the chassis and in communication with the vision system to control the robotic arm to pick the identified fruit using the end effector.

[0007] In some embodiments, the control system is configured to operate in a first mode to control the extendable suction element to extend the suction element towards the fruit, and after the first mode, the control system may be configured to operate in a second mode to maintain contact with the fruit by applying suction force while allowing the suction element to freely extend or retract based on the movement of the fruit.

[0008] In some embodiments, the plurality of grippers may include gripper fingers, each having an inner wall and an outer wall coupled to the inner wall by a plurality of tendons. Each gripper finger may include a plurality of stiffening plates to stiffen each of the tendons. The gripper fingers may be elastically deformable when gripping the fruit. The gripper may include three, four, or five gripper fingers.

[0009] In some embodiments, the suction element comprises a suction opening defined by an end face of the suction element and an end portion that may be configured to permit bending relative to a longitudinal axis of the suction element, and the end portion may be compressible along the longitudinal axis of the suction element.

[0010] In some embodiments, the robotic fruit picking device further comprises a suction line and a pressure sensor for sensing pressure in the suction line, the suction line may pass through the suction element, the pressure sensor may be configured to provide a pressure output signal to the control system, and the control system may be configured to determine whether the suction element has applied suction force to the fruit based on the pressure output signal.

[0011] In some embodiments, the control system is configured to pneumatically operate the suction element.

[0012] The vision system may be configured to identify fruit for picking from the captured and processed images, sort the fruit into an order for picking, and provide output to the control system. The vision system may be used in pose estimation to determine a path for the robotic arm.

[0013] The imaging subsystem may be configured to identify the interfering object in the image and output interfering object identification information to the control unit. The control unit may be configured to determine a movement of the robot arm and end effector to avoid collision with the interfering object based on the interfering object identification information. The control system may be configured to operate the suction element in a retraction mode after either the extended mode or the passive mode to retract the suction element towards one end of the robot arm.

[0014] In some embodiments, the control system is configured to control the gripper to adopt an extended position in which the gripper fingers are spread about a longitudinal axis of the end effector and a retracted position in which the gripper fingers are retracted about the longitudinal axis of the end effector. The control system may be configured to operate the robotic arm to rotate the end effector about the longitudinal axis of the end effector to pick a fruit when the gripper is in the retracted position.

[0015] The robotic fruit harvester may further include a locomotion system coupled to the body to facilitate movement of the body relative to the ground.

[0016] Some embodiments relate to a robotic fruit harvester. The robotic fruit harvester may include a body, a robotic arm coupled to the body, an end effector coupled to a first end of the robotic arm, the end effector including a gripper with a plurality of gripper portions for gripping a fruit and a suction member extendable from the end effector, and a control unit carried by the body, the control unit configured to operate the suction member in an extension mode to apply a suction force and extend the suction member from the end effector towards the fruit, and after the extension mode, to operate the suction member in a passive mode. In the passive mode, the suction member is free to move relative to the gripper while applying a suction force.

[0017] In some embodiments, the robotic fruit harvesting apparatus may comprise a vision system coupled to the body, the vision system comprising an image capture device, the vision system may be configured to identify fruit for picking using the image capture device, hi some embodiments, the vision system may be further configured to determine a collision free path for the apparatus to pick the fruit.

[0018] Some embodiments relate to a computer-implemented method for picking fruit. The computer-implemented method for picking fruit may include operating an end effector of a robot in an extension mode to extend a suction element towards the fruit, where the suction element may be configured to apply a suction force to the fruit, operating the end effector in a passive mode, where the suction element is free to move longitudinally along a longitudinal axis of the end effector while applying a suction force by the suction element to the fruit, and orienting the end effector towards the fruit to pick the fruit.

[0019] The computer-implemented method for picking fruit may further include actuating a gripper coupled to the end effector to grasp the fruit for picking and rotating the end effector about a longitudinal axis of the end effector to remove the fruit.

[0020] The computer-implemented method for picking fruit may further include orienting the end effector toward the storage container, operating the end effector in a retraction mode to retract a suction element toward the end effector, where the suction element stops applying suction force to the fruit, and actuating a gripper of the end effector to release the fruit into the storage container.

[0021] In some embodiments, the computer-implemented method further includes generating a software-defined fruit map based at least in part on the data received by the vision system, determining a harvesting order based at least in part on the fruit map, and determining an approach angle for the end effector for each fruit in the fruit map based at least in part on the fruit map, the harvesting order, and the pose estimate.

[0022] Embodiments are described in more detail below, by way of example, with reference to the accompanying drawings, in which: [Brief description of the drawings]

[0023] [Figure 1] FIG. 1 is a block diagram of a first control system for a robotic fruit picking device, according to some embodiments. [Diagram 2] FIG. 2 is a block diagram of an alternative second control system for the robotic fruit picking device of FIG. [Diagram 3] FIG. 1 is a schematic diagram of a pneumatic system according to some embodiments. [Figure 4] FIG. 1 is a schematic diagram of a pneumatic system according to some embodiments. [Diagram 5]FIG. 2 illustrates a perspective view of an end effector of a robotic fruit picking device, according to some embodiments. [Figure 6] FIG. 6 is a perspective view of a gripper portion of the end effector of FIG. 5 according to some embodiments. [Figure 7] FIG. 1 is a process flow diagram of a computer-implemented method for harvesting fruit. [Figure 8A] FIG. 2 illustrates the reference frame of the end effector. [Figure 8B] FIG. 2 shows a frame of reference for a fruit. [Figure 9A] FIG. 13 is a diagram of an example result of a depth-first search, according to some embodiments. [Figure 9B] FIG. 13 is an example of a cluster-first search result according to some embodiments. [Figure 10] 11 is a flow chart illustrating a cluster-first search process according to some embodiments. [Figure 11] FIG. 1 is a diagram of a workspace boundary of a robotic fruit picking device, according to some embodiments. [Figure 12] FIG. 6 illustrates an impact cylinder around the end effector of FIG. 5 according to some embodiments. [Figure 13A] FIG. 6 is a side view of the retracted gripper of the end effector of FIG. [Figure 13B] FIG. 6 is an end view of the retracted gripper of the end effector of FIG. 5. [Figure 14A] FIG. 6 is a side view of the unfolded gripper of the end effector of FIG. 5. [Figure 14B] FIG. 6 is an end view of the unfolded gripper of the end effector of FIG. 5. [Figure 15-1] FIG. 6 illustrates a control sequence for the end effector of FIG. 5 according to some embodiments. [Figure 15-2] FIG. 6 illustrates a control sequence for the end effector of FIG. 5 according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] FIG. 1 is a block diagram of a first control system of a robotic fruit picking machine (RFPA) 1000, according to some embodiments. The RFPA 1000 comprises several subsystems configured to communicate with each other to harvest fruit. As shown in FIG. 1, the RFPA 1000 comprises or is coupled to a chassis 1050. In some embodiments, the chassis 1050 may include a frame or enclosed structure suitable for containing, holding, carrying, or otherwise housing the subsystems RFPA 1000. In some embodiments, the subsystems of the RFPA 1000 may include separate housings that are coupled to form the chassis 1050. The RFPA 1000 comprises a vision subsystem 1100 for capturing and processing images to determine fruit to be picked and the path of the equipment, a machine operation subsystem 1300 for controlling the movement of the equipment, a power subsystem 1400 for controlling and providing power to the equipment, a movement subsystem 1500 for enabling movement of the equipment relative to the working environment, and a communication subsystem 1600 for providing a means of external communication with the RFPA 1000 and for controlling the subsystems of the RFPA 1000. Each subsystem communicates with each other via an internal network 1200.

[0025] The vision subsystem 1100 comprises a processor 1110 and a memory 1130 accessible by the processor 1110. The processor 1110 may be configured to access data stored in the memory 1130, execute instructions stored in the memory 1130, and read and write data to and from the memory 1130. The processor 1110, and any processors defined herein below, unless otherwise specified, may include one or more microprocessors, microcontrollers, central processing units (CPUs), application specific instruction set processors (ASIPs), or other processors capable of reading and executing instruction code.

[0026] The memory 1130, and any memory defined herein below, unless otherwise specified, may include one or more volatile or non-volatile memory types, such as, for example, RAM, ROM, EEPROM, or Flash. The memory 1130 may be configured to store executable applications for execution by the processor 1110. For example, the memory 1130 may store at least one sorting module 1134 configured to sort the identified fruits into an order for picking.

[0027] To facilitate communication with other subsystems within the RFPA 1000, such as the machine operation subsystem 1300, the vision subsystem 1100 further comprises a communication module 1120. The communication module 1120 may enable wired and / or wireless communication between the vision subsystem 1100 and internal systems. The communication module 1120, and any communication modules defined herein below, may facilitate communication via, for example, a direct connection, Bluetooth, USB, Wi-Fi, Ethernet, or a telecommunications network, unless otherwise specified. According to some embodiments, the communication module 1120 may facilitate communication with internal devices and systems via the internal network 1200.

[0028] The internal network 1200 may include one or more communication methods that facilitate communication between elements of the device 1000. The internal network 1200 may facilitate communication via communication modules in each subsystem of the RFPA 1000. The communication modules of the subsystems of the RFPA 1000 may communicate by methods described herein.

[0029] The vision subsystem 1100 further comprises input and output peripherals (I / O) 1140, which allows a user to communicate with the RFPA 1000 and enables the vision subsystem 1100 to capture images. The I / O 1140 may comprise at least one image capture device 1141, which in some embodiments may be, for example, a webcam, a compact digital camera, or an action camera. In some embodiments, the at least one image capture device 1141 may include, for example, an Intel RealSense computer vision system. In some embodiments, the image capture device 1141 may provide point cloud data of the environment. The I / O 1140 may comprise a control device 1142, which in some embodiments may be, for example, a touch screen display, and one or more of a keyboard, a mouse, a camera, a microphone, a speaker, a button, a slider, and an LED.

[0030] The machine operation subsystem 1300 comprises a processor 1310 and a memory 1330 accessible by the processor 1310. The processor 1310 may be configured to access data stored in the memory 1130, execute instructions stored in the memory 1330, and read and write data to and from the memory 1330. The memory 1330 may be configured to store executable applications for execution by the processor 1310. For example, the memory 1330 may store at least one movement module 1332 configured to control the physical movement of the RFPA 1000. The memory 1330 may also store data in a data storage location, such as an internal storage 1333. According to some embodiments, the internal storage 1333 may store, for example, current poses or positions of each portion of the robot arm 1342 in three-dimensional space accessible to the arm operation module 1331, as well as historical arm positions, optionally accessible to the arm operation module 1331. Here, the three-dimensional space has six degrees of freedom, including translation along three perpendicular axes, and rotation about each of the three axes.

[0031] To facilitate communication with other subsystems within the RFPA 1000, such as the vision subsystem 1100, the machine operation subsystem 1300 further comprises a communications module 1320. The communications module 1320 may enable wired and / or wireless communications between the machine operation subsystem 1300 and internal systems. According to some embodiments, the communications module 1320 may facilitate communication with internal devices and systems via the internal network 1200.

[0032] The machine operation subsystem 1300 further comprises input and output peripherals 1340 to allow the subsystem to communicate with the outside world. The I / O 1340 may comprise an end effector 1341 (FIG. 5). The I / O 1340 may comprise a robotic arm 1342, which in some embodiments may be, for example, off-the-shelf or privately manufactured. In some embodiments, the robotic arm 1342 may enable rotational and / or translational movement. The I / O 1340 may comprise a drive system 1343, which may be a motor control board configured to communicate with and control the locomotion subsystem 1500. The I / O 1340 may comprise a suction system 1344, which in some embodiments may be, for example, off-the-shelf or privately manufactured.

[0033] The locomotion subsystem 1500 includes at least one motor and drive assembly. In some embodiments, the drive assembly may include, for example, at least one wheel, track, or leg.

[0034] The power subsystem 1400 includes a processor 1410 and a memory 1430 accessible by the processor 1410. The processor 1410 may be configured to access data stored in the memory 1430, execute instructions stored in the memory 1430, and read and write data to and from the memory 1430. The memory 1430 may be configured to store executable applications for execution by the processor 1410. For example, the memory 1430 may store data in an internal memory 1431 configured for access by the processor 1410.

[0035] To facilitate communication with other subsystems within RFPA 1000, such as machine operation subsystem 1300, power subsystem 1400 further comprises a communications module 1420. Communications module 1420 may enable wired and / or wireless communications between power subsystem 1400 and internal systems. According to some embodiments, communications module 1420 may facilitate communication with internal devices and systems via internal network 1200.

[0036] The power subsystem 1400 may further include a battery array 1440 including at least one battery 1441. In some embodiments, the battery array 1440 may include a plurality of batteries 1441-144n. The battery 1441 may include a rechargeable battery, such as a nickel metal hydride battery, a lithium ion battery, a lead acid battery, or a nickel cadmium battery. To recharge the battery array 1440, the power subsystem 1400 may further include a charging module 1450 including a power port for connecting to an external power source, and a PCB for monitoring and controlling the input of electrical energy to the battery array 1440. The RFPA 1000 may utilize the charging module 1450 and the connected external power source to provide power directly to the subsystems of the RFPA 1000. The power subsystem 1400 may also include a first power supply link 1460, a second power supply link 1470, and a third power supply link 1480 that provide electrical connections to the internal subsystems of the device 1000. This allows the vision subsystem 1100, the machine operation subsystem 1300, and the locomotion subsystem 1500 to receive the electrical energy necessary to perform their respective functions from the power subsystem 1400. In some embodiments, the RFPA 1000 may omit the battery array 1440 and the battery 1441 and utilize a charging module 1450 connected to an external power source to directly power the systems of the RFPA 1000. The external power source may be, for example, a generator or a connection to the electrical mains.

[0037] The communications subsystem 1600 comprises a processor 1610 and a memory 1630 accessible by the processor 1610. The processor 1610 may be configured to access data stored in the memory 1630, execute instructions stored in the memory 1630, and read and write data to and from the memory 1630. The memory 1630 may be configured to store executable applications for execution by the processor 1610. For example, the memory 1630 may store data in an internal memory (not shown) configured for access by the processor 1610.

[0038] To facilitate communication with other subsystems within RFPA 1000, such as machine operation subsystem 1300, as well as external systems, communications subsystem 1600 further comprises communications module 1620. Communications module 1620 may enable wired and / or wireless communications between communications subsystem 1600 and internal systems, and in some embodiments, external computing devices and components. According to some embodiments, communications module 1620 may facilitate communication with internal devices and systems via internal network 1200.

[0039] FIG. 2 is a block diagram of an alternative second control system 200 for RFPA 1000, described further below.

[0040] 3 shows a schematic of a pneumatic system 300 including a pneumatic cylinder 302, a suction element 316, an end 306, and an air valve 308. The pneumatic cylinder 302 includes a first chamber 310, a second chamber 312, and a piston 314. The air valve 308 includes a two-position, four-way, five-port valve, allowing the pneumatic cylinder 302 to be in an extended state or a retracted state. The two positions corresponding to the extended and retracted states of the air valve 308 are determined by a solenoid 304 and a solenoid 305. When the solenoid 304 is activated, the pneumatic system 300 is in an extended state. When the solenoid 305 is activated, the pneumatic system 300 is in a retracted state. At any one time, only one solenoid, either the solenoid 304 or the solenoid 305, may be activated.

[0041] When the pneumatic system 300 is in an extended state, the suction element 316 extends to the right in FIG. 3 away from the pneumatic cylinder 302. This is accomplished by closing the exhaust port connected to the first chamber 310 and opening the exhaust port connected to the second chamber 312 to supply pressurized gas to the first chamber 310, thereby exerting a greater force on the piston 314 in the first chamber 310 than the force exerted on the piston 314 in the second chamber 312. The supply of pressurized gas to the first chamber 310 is then cut off and the pneumatic cylinder 302 remains in the extended state. When the pneumatic system 300 is in a retracted state, the suction element 316 retracts into the pneumatic cylinder 302 to the left in FIG. 3. This is accomplished by closing the exhaust port connected to the second chamber 312 and opening the exhaust port connected to the first chamber 310 to supply pressurized gas to the second chamber 312, thereby exerting a greater force on the piston 314 in the second chamber 312 than the force exerted on the piston 314 in the first chamber 310. The supply of pressurized gas to the second chamber 312 is cut off and the pneumatic cylinder 302 remains in a retracted state.

[0042] 4 is a pneumatic system 400 including a pneumatic cylinder 302, a suction element 316, an end 306, and an air valve 402. The air valve 402 may include a three-position, four-way, five-port open center valve, which allows the pneumatic cylinder 302 to be in one of three states: extended, retracted, or passive, where the extended and retracted states are the same as those described with respect to the pneumatic system 300. In some embodiments, the air valve 302 may include a three-position exhaust center valve or a three-position, five-port normally open valve. The three positions of the air valve 308 are determined by the solenoid 304, the solenoid 305, and the solenoid 403. When the solenoid 304 is activated, the pneumatic system 300 is in the extended state. When the solenoid 305 is activated, the pneumatic system 300 is in the retracted state. When solenoid 403 is activated, pneumatic system 300 is in a passive state. Only one of solenoid 304, solenoid 305, or solenoid 403 may be activated at any one time. When pneumatic system 400 is in a passive state, the exhaust ports connected to first chamber 310 and second chamber 312 are open, the supply of pressurized gas to either chamber is cut off, and neither chamber exerts a force on piston 314. This allows piston 314 to move freely within pneumatic cylinder 302 under an externally applied force via suction element 316.

[0043] 5 illustrates an example of an end effector 1341 of the robotic fruit picking device 1000, according to some embodiments. The end effector 1341 includes a gripper 291, a suction system 292, and a pneumatic system 293. The gripper 291 of the end effector 1341 functions to grasp and hold fruit.

[0044] In various embodiments, the gripper may also function to redirect branches and obstacles in the path of the end effector 1341 (FIG. 13A). In some embodiments, the end effector 1341 is coupled to the robot arm 1342 via a mounting plate 535. The mounting plate 535 may be coupled to lift the plate 540 to allow access to the mounting points on the mounting plate 535. According to some embodiments, the gripper 291 may comprise at least three appendages or gripper portions (shown in FIG. 5 as three gripper portions 505a, 505b, and 505c, but generalized as 505a-505n), a gripper portion base 510, a push bar 515, a support beam 520, a cylinder cap 560, a stage 1 piston 570, a push bar base 575, and a rigid ring 585.

[0045] According to some embodiments, to actuate the gripper portions 505a-505n, each gripper portion 505 is coupled to a gripper portion base 510. The base 510 of each gripper portion is then coupled to a push bar 515 and a support beam 520. That is, the base 510 of each gripper portion is coupled to the support beam 520 at an outer point further away from the longitudinal axis 599 of the end effector 1341 than the inner point coupled to the push bar 515, and is configured to rotate about the outer point. The support beam 520 of each gripper portion 505a-505n is coupled to the same distance from the longitudinal axis 599 of the end effector 1341 via a rigid ring 585. The support beam 520 and the rigid ring 585 are coupled to the cylinder cap 560 and are considered static components. That is, the support beam 520, the rigid ring 585, and the cylinder cap 560 move depending on the movement of the end effector 1341. Each push bar 515 is coupled to a push bar base 575 which is coupled to a stage 1 piston 570. The at least three gripper portions 505a-505n, each gripper portion base 510, each push bar 515, stage 1 piston 570, and push bar base 575 are considered dynamic components, i.e., the at least three gripper portions 505a-505n, each gripper portion base 510, each push bar 515, stage 1 piston 570, and push bar base 575 move dependent on the end effector 1341 and can also move independently of the end effector 1341.

[0046] The stage 1 piston 570 is pneumatically actuated by the pneumatic system 293 along the longitudinal axis 599 of the end effector 1341. This actuation causes the push bar base 575 to move, which in turn causes the respective push bars 515 of each gripper segment 505a-505n to move. The movement of each push bar 515 coupled to each finger base 510 in turn causes a rotational movement of each segment base 510, which in turn causes a movement of each gripper segment 505a-505n. That is, rotation is induced as the stage 1 piston 570 is actuated and extended along the longitudinal axis 599 of the end effector 1341 away from the cylinder cap 560. This rotational movement of the gripper segment base 510 of each gripper segment 505a-505n causes the gripper segments 505a-505n to assume an extended position. In the extended position, the gripper portions 505a-505n are spread apart about the longitudinal axis 599 of the end effector 1341. The stage 1 piston 570 may also be actuated to move in a direction opposite to that required by the extended position. A rotation is induced when the stage 1 piston is actuated and retracts along the longitudinal axis 599 of the end effector 1341 toward the cylinder cap 560. This rotational movement of each gripper portion 505a-505n at the gripper portion base 510 causes the gripper portions 505a-505n to adopt a retracted position. In the retracted position, the gripper portions 505a-505n are retracted about the longitudinal axis 599 of the end effector 1341.

[0047] The suction system 292 may include a stage 2 piston 580, a suction damper 590, and an end 306, which may be referred to as a suction tip or suction cup. As described above in Figures 3 and 4, the suction element 316 is controlled to be in an extended, retracted, or passive mode depending on the configuration of the stage 2 piston 580. The end 306 is coupled to the suction damper 590, which in turn is coupled to the suction element 316. In some embodiments, the end face of the end 306 is thin and made from a soft material, such as a rubber or silicone material, to further increase efficiency in creating a suction seal.

[0048] In some embodiments, the end 306 may be a bellows-like design. The bellows-like design may serve the need to accommodate various shapes of fruits and adapt to their exterior surfaces to apply sufficient suction. In some embodiments, the bellows-like design may comprise multiple convolutions, which may serve to increase the positioning tolerance in all three dimensions. The increased positioning tolerance may allow the end 306 to be misaligned with the longitudinal axis 599 of the end effector 1341. That is, the end may bend such that the end face of the end 306 is not perpendicular to the longitudinal axis 599 of the suction element 316 and the end effector 1341. The multiple convolutions may serve to absorb (dampen) shocks when contacting the fruit. The cushioning inherent in the bellows-like design allows the end face of the end 306 to adapt to different fruit surfaces to create an adequate suction seal. For example, the end face of the end 306 may be initially flat and adopt a concave shape when contacting the fruit.

[0049] The radius of the bellows-like design at its narrowest point may not be smaller than the radius of the suction element 316. When the gripper 291 is in the retracted configuration and the suction element 316 is retracted, the diameter of the bellows-like design at its widest point may be less than or equal to the diameter of a circle centered on the longitudinal axis 599 and having a radius equal to the narrowest distance between the longitudinal axis 599 and each of the gripper portions 505a-505n, where the circle centered on the longitudinal axis 599 is in the same plane as the end face of the end portion 306. That is, the bellows-like design is not so large that it interferes with any of the gripper portions 505a-505n when the gripper 291 is in the retracted configuration and the suction element 316 is retracted. In some embodiments, the material of the bellows-like design may have a durometer of 25° to 75° Shore, or 30° to 60° Shore, for example. In some embodiments, the bellows-like design material may have a durometer ranging from 25° to 75° Shore, or a combination of 30° to 60° Shore, for example.

[0050] The end 306 may include an opening on its end face. That is, the face of the end 306 opposite the face that connects to the suction damper 590 may include an opening in the center of the end face. This opening provides a passage within the end 306 that connects internally to the suction damper 590. This internal passage may be connected to the suction system 1344, for example, via plastic tubing. The suction system 1344 may create a vacuum within the internal passage of the end 306 and the suction damper 590 and provide suction to the fruit through the opening in the end face of the end 306. In some embodiments, the vacuum pressure range may be determined by the type of fruit being picked by the robot. The vacuum pressure range may be between 0.6 megapascals and 1 megapascal for apples, for example.

[0051] In some embodiments, the pneumatic system 293 may include a flow control valve 525, an air cylinder 302, an acrylonitrile butadiene styrene (ABS) block 550, a cosmetic tube 555, and a stopper 565. In some embodiments, the pneumatic system 293 may actuate the stage 1 piston 570 in the manner described for the pneumatic system 300. The flow control valve 525 may include a two-position, four-way, five-port valve. In some embodiments, the pneumatic system 293 may actuate the stage 2 piston 580 in the manner described for the pneumatic system 400. The flow control valve 525 may include a three-position, four-way, five-port open center valve. In some embodiments, the pneumatic system 293 may actuate both the stage 1 piston 570 and the stage 2 piston 580 simultaneously, independently of each other. In some embodiments, the stopper 565 may limit the retraction of the stage 1 piston 570 by blocking its path of travel in the direction of the longitudinal axis 599 of the end effector 1341. The air cylinder 302 includes a cylinder cap 560 and a cylinder base 545 to create a sealed cylinder for use in the pneumatic system 293. The ABS block 550 provides an attachment point for a cosmetic tube 555 to be coupled. The cosmetic tube 555 may not be necessary, but may provide protection from external elements that could damage the internal components of the pneumatic system 293. In some embodiments, the cosmetic tube 555 may enclose the components of the pneumatic system 293.

[0052] As shown in FIG. 6, each gripper portion 505 may comprise a first inner wall 610, a second outer wall 620, and a number of tendons 630a-n. The inner wall 610 is coupled to the outer wall 620 via a number of tendons 630. The gripper portion 505 may be generally considered to have a finger-like appearance, where the gripper portion 505 gradually narrows from a wide base at 640 to a pointed (but not sharp) tip at 645. The inner wall 610 and the outer wall 620 comprise a shape-adaptive flexible material, such as, for example, a thermoplastic polyurethane (TPU), a thermoplastic elastomer (TPE), or a thermoplastic copolyester (TPC). The shape-adaptive flexible material may enable the gripper portion 505 to at least partially conform to the shape of the fruit when gripped. In some embodiments, the inner wall 610 may comprise a silicone-based surface finish to enhance the gripping friction of each gripper portion 505.

[0053] The tendons 630 may comprise a lightweight and rigid metallic material such as, for example, aluminum, an aluminum alloy, titanium, or a titanium alloy. The tendons 630 act to reduce moment forces within the cross section of the gripper portion 505 to prevent undesired rotation along the longitudinal axis 599 when gripping a fruit. This undesired rotation may result in a loss of grip on the fruit. In some embodiments, given the finger-like appearance of the gripper portion 505, the tendons 630a-630n may be gradually larger in length and height from the tip to the base of the gripper portion 505. That is, the tendons 630a toward the tip may be smaller than the tendons 630b away from the tip, which in turn may be smaller than 630c, for example, and similarly for 630c-630n. This particular structure may allow the gripper portion 505 to have a natural curvature, as shown in FIG. 6. The aforementioned tendon structure may also allow for greater bending of the gripper portion 505 upon actuation.

[0054] FIG. 7 illustrates a computer-implemented method 700 for locating fruit, determining a path to reach the located fruit, and picking fruit within a designated workspace of the RFPA 1000. In some embodiments, the method may be performed by the control system 200. At 702 of the computer-implemented method 700, in some embodiments, the processor 1310 executes the movement module 1332 to move the RFPA 1000 to a new workspace via the movement subsystem 1500. The workspace is typically, for example, a plot of an orchard with fruit ready to be picked that is within reach of the robotic fruit picking device. At 704, the processor 1110 executes the image processing module 1131 to obtain visual data of the workspace. In some embodiments, the visual data is, for example, a stream of captured images. At 706, the image processing module 1131 is still executed by the processor 1110. The captured images are analyzed and the location of the fruit is determined. Position data for each fruit's center of gravity location is stored in memory 1130 by processor 1110. In some embodiments, the fruit locations are added to a software defined fruit map, thereby recording the position of the fruit relative to the robot.

[0055] At 708, the processor 1110 executes the pose estimation module 1133 to determine approach angles to the located fruits. Pose estimation is applied to the known fruits to maximize the number of fruits that can be reached within the workspace. Pose estimation includes determining the appropriate approach angle for each fruit. For example, fruits that are high in the tree canopy should be approached and picked from below rather than horizontally because doing so may overextend the robotic arm. This overextension may move the robotic arm outside of its operating workspace. The output of the pose estimation module 1133 is subject to optimization of several factors. These factors include, for example, the arm workspace, speed constraints, fruit surface occlusions, and collisions with hard branches and canopy structures. Fruit surface occlusions may include, for example, leaves and soft branches.

[0056] The pose estimation module 1133 utilizes numerical optimization to calculate the approach angles to the fruit. This numerical optimization maximizes the workspace of the RFPA 1100 while ensuring that the inspection, grasping, and extraction trajectories remain feasible. The standard optimization problem for a fruit is: Minimize x f(x) (1) Constraint: g i (x)≦0, i = 1, ..., m (2) h j (x) = 0, j = 1, ..., n (3) where Equations 2 and 3 represent general functions of the variable x.

[0057] For a fruit, a function f(x) in Equation 1 is defined such that when minimized with the constraints in Equation 2 and Equation 3, an optimized approach angle for the fruit is found. This optimized approach angle satisfies the kinematic constraints as well as the path and collision constraints. It has been determined that some of the constraints can be relaxed to increase the rate of determining a successful picking path for difficult-to-reach fruits.

[0058] Within the pose estimation module 1133 and the path planning module 1132, via frames are defined as various points of interest within the workspace of the RFPA 1000. G and Fruit Frame F A The fruit frame F represents the fruit within the tree canopy. A is considered as the via frame. When picking a fruit, the final gripper rotation is mostly arbitrary. This allows the pose estimation module 1133 to estimate the gripper frame F G For harvesting, the y- and z-axes of the fruit frame F can be used to determine the approach angle, which is not strictly constrained. A When it coincides with the gripper frame F G The following constraints arise: Constraint:P G =P A (4)

number

[0059] Here, the constraint in Equation 4 states that the origins of the gripper frame and the fruit frame are equal. The constraint in Equation 5 states that the gripper frame F G and fruit frame F A Both x-axes of are in the same direction. The processor 1110 executing the pose estimation module 1133 utilizes these constraints to distinguish between fruits that can be picked and fruits that cannot be picked within the workspace of the RFPA 1000.

[0060] At 710, the processor 1110 executes the sorting module 1134. To date, only basic sorting methods are utilized by modern robotic harvesters. One example is depth-first sorting, where the picking order is defined based on distance from a reference point. The depth-first search approach is simple, but minimizes disturbance to other fruits by harvesting from the outside in. However, the depth-first search approach is not considered time-optimal for tree canopies that prefer clustered growth, such as apples. As shown in FIG. 9A, depth-first sorting results in a fruit picking order that includes a haphazard path for the gripper, which travels long distances between fruits. This reduces harvesting efficiency and can disturb large areas of the orchard canopy.

[0061] The sorting module 1134 of the RFPA 1000, when executed by the processor 1110, utilizes a greedy search approach, with additional optimization constraints added, resulting in a cluster output as shown in FIG. 10. The greedy approach is a method that follows a problem-solving heuristic that makes locally optimal choices at each stage. At 1012, the processor 1110 executes the sorting module 1134 to establish a seed fruit data object that is initially determined by the closest depth of the fruit that is determined to be pickable, and a new list to store the fruit data object. As shown in FIG. 8A, the closest fruit that can be picked is marked with a 1, which is the initial seed fruit data object. From the location of the initial seed fruit data object, the search radius is extended in increments at 1014. At each increment, the sorting module 1134 checks whether a new fruit data object is within the extended search radius at 1016. If a new fruit data object is found, it is added to the list at 1018. If no new fruit data objects are found, a process executed by processor 1110 determines at 1020 that all fruit data objects in the current list are clusters.

[0062] At 1022, the sorting module 1134 checks whether there are any known fruit data objects remaining in the workspace but not in a cluster. If so, at 1024, the closest fruit data object that is not in a cluster is selected as the new seed fruit data object and a new list is created. The processor 1110 then repeats the above process until all fruit data objects in the workspace are in clusters, as shown in Figure 9B, and returns to 1014. Once it is determined that all clusters have been found, the sorting is complete and the processor 1110 proceeds to perform path planning at 1026. An example of pseudocode for the cluster-first search approach is provided below: [Table 1]

[0063] The cluster-first search method offers two main advantages. The first is the opportunity to optimize the robot's trajectory when considering clusters of fruits. That is, when a cluster of fruits is harvested, the robot continuously harvests in the same area. The paths calculated for fruits in the same cluster are similar. This condition can be exploited to apply the optimized trajectory of the path to one fruit to all neighboring fruits in the same cluster. This significantly reduces the computational redundancy when calculating the trajectory for harvesting each fruit, allowing the robot to move efficiently between fruits. The second advantage is the reduction of canopy disturbance when picking fruits. In standard robotic harvesting, the canopy of an orchard is often disturbed, causing fruits to undesirably change position or fall off, reducing the efficiency of the harvester. Harvesting in clusters limits disturbance to the cluster area and leaves the remaining workspace undisturbed. This limitation of disturbance reduces the number of visual updates required to re-determine the location of fruits for additional movements. The result of using this new cluster-first search approach is a more energy and time efficient approach, reducing costs and / or improving picking efficiency.

[0064] Following execution of the sorting module 1134, the processor 1110 executes a path planning module 1132. The path planning module 1132 determines a data set that outputs movement commands to the machine manipulation subsystem 1300, including the end effector 1341, the robotic arm 1342, and the suction system 1344, to harvest the fruit in the order determined by the sorting module 1134. The path planning module 1134 determines optimal paths for the mechanical elements of the RFPA 1000 by enforcing a number of kinematic and collision constraints.

[0065] The path planning module 1132 calculates the fruit frame F A The rotation of the fruit frame F is optimized to minimize the amount of rotation required. This optimization is subject to constraints such as ensuring no collisions and the existence of a valid path.A is essentially the proposed gripper frame F G Therefore, the final gripper rotation position is arbitrary. This allows us to solve the optimization problem by choosing the gripper frame F G We can omit the rotation around the x-axis of , and obtain the objective function f(x)γ 2 +β 2 (6) where x=[γ β] T and γ and β are the rotations of the fruit about the z-axis and y-axis, respectively.

[0066] Further constraints are added to ensure that the optimized path is an inverse kinematic solution.

number

number

number

[0067] The path planning module 1132 calculates the fruit frame F such that the grasping is optimized while satisfying the kinematic constraints and additional constraints. A0 The aim is to find the rotation (equation 7) to be applied to the new fruit frame F A Equation 9 shows the rotation matrix of F A0and the applied rotation in Equation 7. The end effector position is used to constrain the path through the kinematics. As shown in FIG. 11, we introduce an inner workspace boundary 191 and an outer workspace boundary 192. Here, the inner workspace boundary 191 is defined by the minimum area that the robot arm 1342 and the end effector 1341 can move. The inner workspace boundary 191 has a radius defined by r. Furthermore, the outer workspace boundary 192 is defined by the maximum area that the robot arm 1342 and the end effector 1341 can move. The outer workspace boundary 192 has a radius defined by R. The RFPA workspace 193 is defined by the inner workspace boundary 191 and the outer workspace boundary 192, such that the ring of the RPFA workspace 193 is Δ=Rr. For both the inner workspace boundary 191 and the outer workspace boundary 192, the RFPA workspace 193 is relatively uniform, so that the end effector frame F of the RFPA 1000 E At least one inverse kinematic solution exists when the end effector frame F is within the boundary of the workspace. E and gripper frame F G does not match, so the transformation matrix

number

number

[0068] When Equation 1 is optimized and Equations 10 and 11 satisfy the constraints in Equation 2, it is guaranteed that a kinematic solution for the path of the end effector 1341 exists. Furthermore, the fruit grasping trajectory may follow a series of via frames. For example, P V1 =P A →P G =P A →P V2 =P A (12) In the formula, PV1 and P V2 is F G is the position of the via frame attached to, which results in further inequality constraints.

number

[0069] The path planning module 1132 also considers collisions when optimizing the paths of the robot arm 1342 and the end effector 1341. The first cause of collisions considered is the workspace environment, which may include, for example, branches, building structures, or leaves. The second cause of collisions considered is the RFPA 1000. In other words, the robot arm 1342 and the end effector 1341 may collide with any of the elements of the RFPA 1000 and / or the chassis 1050. This form of collision is called a self-collision. The self-collision check is performed by the processor 1110 via the path planning module 1132 after optimization. The design of the robot arm 1342 allows for a minimum number of kinematic solutions to safely avoid self-collisions for most end effector positions. Checking for self-collisions after optimization reduces computational costs. In some embodiments, the processor 1110 executing the path planning module 1132 may check the path to determine whether a self-collision occurs after determining the end effector path for harvesting fruit. If a self-collision occurs, the path planning module 1132 executed by the processor 1110 alters the path that includes the collision to avoid the self-collision.

[0070] To prevent collision of the end effector with the workspace environment, when the path planning module 1132 is executed by the processor 1110, collision with the environment is taken into account during optimization.

number

number

number

number

number

[0071] Using the above, P G and P E If is derived from x, then equations 15 and 17 can be expressed as inequality constraints. g (3,3n-2) (x)=-r(x) (19) g (3,3n-1) (x)=r(x)-1 (20) g (3,3n)(x)=d(x)-R G (twenty one) As a result, for every n-th point, c ∈ C B It is.

[0072] Therefore, point c is inside the hypothetically defined collision cylinder. In other words, point c is considered to collide with the end effector 1341 if any of Equation 19, Equation 20, or Equation 21 does not satisfy the condition in Equation 2. For a path to be determined to be collision-free and satisfy the constraints described herein, C B All points within must be outside the collision cylinder. The output of the path planning module 1132, when executed by the processor 1110, is a data set input for the machine manipulation subsystem 1300. The processor 1310 may execute an arm manipulation module 1331 to process the data set input and manipulate the robot arm 1342, which in turn manipulates the end effector 1341.

[0073] 13A-13B and 14A-14B show the gripper 291 in both retracted and extended configurations, respectively. The end effector 1341 approaches the fruit with the gripper 291 in the retracted position as shown in FIG. 13A. This allows the end effector to form a narrow profile or point at its end, which allows the end effector to more effectively go through the tree canopy and push aside any twigs, leaves, or other small objects. Once the end effector reaches the fruit for harvesting, the gripper 291 is controlled by the arm operation module 1331 to assume an extended position as shown in FIG. 14A. This allows the gripper 291 to prepare to grip the fruit, but also helps to push aside twigs, leaves, or other small objects that are in close proximity to the gripper fingers 505a-505n. The method of utilizing the pointed gripper fingers of the end effector to go through the tree canopy and push aside objects is referred to as a "go-through strategy." This strategy is key for picking fruit in dense canopies where objects such as twigs and leaves are not considered as collision obstacles when determining the fruit harvesting path.

[0074] FIG. 15 shows the control sequence of the end effector 1341 and robotic arm 1342 when picking a fruit. In 15a, the robotic arm 1342 moves the end effector 1341 towards the fruit. As the end effector 1341 approaches the fruit, the gripper 291 is pneumatically actuated to adopt a retracted configuration to implement the "pass through strategy" described herein. In 15b, the end effector 1341 reaches the fruit for picking. In 15c, the gripper 291 is pneumatically actuated to adopt an extended configuration, pushing aside any objects in close proximity to the gripper fingers 505a-505n. In 15d, the suction element 316 extends from the end effector 1341 towards the fruit. The end 306 creates a vacuum as described above, contacts the fruit and applies suction to the fruit to hold it. The suction element 316 is now in a passive state, in which it is free to move along the longitudinal axis 599 of the end effector 1341 while maintaining the applied suction force. At 15e, the robotic arm 1342 moves the end effector 1341 towards the fruit, allowing the gripper 291 to surround the fruit. At 15f, the gripper 291 is pneumatically actuated to retract the gripper fingers 505a-505n around the fruit to grip the fruit. At 15g, the gripper 291 securely grips the fruit, with the gripper fingers 505a-505n conforming to the shape of the fruit. At 15h, the robotic arm 1342 rotates the end effector 1341 about its longitudinal axis 599 to detach the fruit from its stem while still holding the fruit. The robotic arm 1342 then retracts the end effector 1341 at 15i, removing the end effector 1341 and the fruit from the tree canopy.

[0075] In some embodiments, the RFPA 1000 can place fruit into a storage container according to the control sequence shown in FIG. 15. The storage container can be, for example, a bucket, bag, or bin. The robot arm 1342 can retract the end effector 1341 at 15i in FIG. 15 and then direct the end effector 1341 and the fruit towards the storage container. The end 306 then stops applying suction to the fruit and the suction element 316 retracts as described above. The end effector 1341 then pneumatically actuates the gripper 291 to adopt an extended configuration and release the fruit into the storage container.

[0076] Alternative Control System Architectures 2 illustrates an alternative second control system 200 of the RFPA 1000, including a central controller 205 and a gripper control unit 255, according to some embodiments. The central controller 205 comprises a processor 210 and a memory 230 accessible by the processor 210. The processor 210 is configured to access data stored in the memory 230, execute instructions stored in the memory 230, and read and write data to and from the memory 230. The memory 230 may be configured to store data and executable applications for execution by the processor 210. For example, the memory 230 may store data in an internal memory 230 configured for access by the processor 210.

[0077] The central controller 205 oversees the control of the vision subsystem 1100, the power subsystem 1400, and the locomotion subsystem 1500 and is capable of storing and retrieving data in and from the memory of each subsystem, as well as executing instructions stored in the memory of each subsystem. In some embodiments, program data, modules, and instructions for performing the functions of each subsystem, including the vision subsystem 1100, the power subsystem 1400, and the locomotion subsystem 1500, may be stored in the internal memory 235 and executable by the processor 210. In some embodiments, the central controller 205 may communicate operating instructions to each subsystem, including the vision subsystem 1100, the power subsystem 1400, and the locomotion subsystem 1500, for execution using their own processors and data stored in their own memory modules.

[0078] In some embodiments, the processor 210 of the central controller 205 may execute the image processing module 1131, the path planning module 1132, the pose estimation module 1133, and / or the sorting module 1134 of the vision subsystem 1100 described herein. The dataset output of the path planning module 1132 for the operation of the robotic arm 1342 and end effector 1341 may be input to the gripper control unit 255.

[0079] To facilitate communication with the gripper control unit 255, the central controller 205 further comprises a communications module 220. The communications module 220 may enable wired and / or wireless communications between the central controller 205 and the gripper control unit 255, as well as, in some embodiments, with external computing devices and components. According to some embodiments, the communications module 220 may facilitate communication with internal devices and systems via the internal network 1200.

[0080] Gripper control unit 255 includes a processor 260 and a memory 280 accessible by processor 260. Processor 260 may be configured to access data stored in memory 280, execute instructions stored in memory 280, and read and write data to memory 280. Memory 280 may be configured to store executable applications for execution by processor 260. For example, memory 280 may store an operations module 282 configured for access by processor 260.

[0081] To facilitate communication with the central controller 205, the gripper control unit 255 further comprises a communications module 270. The communications module 270 may enable wired and / or wireless communications between the gripper control unit 255 and the central controller 205. According to some embodiments, the communications module 270 may facilitate communication with internal devices and systems via the internal network 1200.

[0082] The gripper control unit 255 controls the operation of the end effector 1341. The gripper control unit 255 receives control signals for manipulation and pneumatic actuation of the end effector from the central controller 205 via the communications module 270. The gripper control unit 255 then processes the received control signals via the processor 260 to manipulate and / or actuate the systems of the end effector 1341 as desired by the central controller 205. In some embodiments, the processor 260 of the gripper control unit 255 may execute the manipulation module 282 to process the data set output of the path planning module 1132.

[0083] It will be understood by those skilled in the art that numerous variations and / or modifications may be made to the above-described embodiments without departing from the broad general scope of the present disclosure, and the present embodiments are therefore to be considered in all respects as illustrative and not restrictive.

Claims

1. 1. A robotic fruit harvesting device, comprising: Chassis and a robotic arm supported by the chassis and having an end effector, the end effector comprising a plurality of grippers and an extendable suction element; a vision system carried by the chassis and configured to identify fruit for picking; a control system carried by the chassis and in communication with the vision system to control the robotic arm to pick an identified fruit using the end effector; Equipped with the control system is configured to operate in a first mode to control the extendable suction element to extend the suction element towards fruit; the control system is configured to operate in a second mode after the first mode to maintain contact with the fruit by applying suction while allowing the suction element to freely extend or retract based on movement of the fruit.

2. The robotic fruit harvesting device.

2. The apparatus of claim 1 , wherein the plurality of grippers comprises gripper fingers.

3. The apparatus of claim 2 , wherein each of the gripper fingers has an inner wall and an outer wall coupled to the inner wall by a plurality of tendons.

4. The apparatus of claim 3 , wherein each gripper finger comprises a plurality of stiffening plates for stiffening each of the tendons.

5. 5. An apparatus according to claim 3 or claim 4, wherein the gripper fingers are elastically deformable when gripping the fruit.

6. An apparatus according to any one of the preceding claims, wherein the gripper comprises three, four or five gripper fingers.

7. The apparatus of any one of claims 1 to 6, wherein the suction element comprises a suction opening defined by an end face of the suction element.

8. The device of claim 7 , wherein the suction element includes ends configured to permit bending relative to a longitudinal axis of the suction element.

9. The device of claim 8 , wherein the end is compressible along the longitudinal axis of the suction element.

10. Further comprising a suction line and a pressure sensor for sensing the pressure of the suction line, the aspiration line passes through the aspiration element, and the pressure sensor is configured to provide a pressure output signal to the control system; The apparatus according to any one of claims 1 to 9, wherein the control system is configured to determine, based on the pressure output signal, whether the suction element has applied suction to the fruit.

11. Apparatus according to any one of the preceding claims, wherein the control system is configured to pneumatically operate the suction element.

12. 12. The apparatus of any one of claims 1 to 11, wherein the vision system is configured to identify fruit for picking from captured and processed images, sort the fruit in an order for picking, and provide the output to the control system.

13. Apparatus according to any one of claims 1 to 12, wherein the vision system is used for pose estimation to determine a path for the robot arm.

14. the imaging subsystem is configured to identify an interfering object in the image and output interfering object identification information to the control unit; The apparatus of any one of claims 1 to 13, wherein the control unit is configured to determine movements of the robot arm and the end effector based on the interfering object identification information to avoid collision with the interfering object.

15. The apparatus of any one of claims 1 to 14, wherein the control system is configured to operate the suction element in a retraction mode after the extension mode or passive mode to retract the suction element towards one end of the robotic arm.

16. The apparatus of any one of claims 1 to 15, wherein the control system is configured to control the gripper to adopt an extended position in which the gripper fingers are spread apart about a longitudinal axis of the end effector and a contracted position in which the gripper fingers are contracted about the longitudinal axis of the end effector.

17. 17. The apparatus of claim 16, wherein the control system is configured to operate the robotic arm to rotate the end effector about the longitudinal axis of the end effector to pick the fruit when the gripper is in the retracted position.

18. 20. The apparatus of claim 1 or 17, further comprising a movement system coupled to the body for facilitating movement of the body relative to a ground surface.

19. 1. A robotic fruit picking device comprising: The main body, a robot arm coupled to the body; an end effector coupled to a first end of the robotic arm, a gripper including a plurality of gripper portions for gripping the fruit; a suction member extendable from the end effector; The end effector comprising: a control unit carried by the body, the control unit configured to operate the suction member in an extension mode to apply a suction force and extend the suction member from the end effector towards the fruit, and after the extension mode, to operate the suction member in a passive mode, in which the suction member is free to move relative to the gripper while applying the suction force; and 5. The robotic fruit picking device according to claim 1 ,

20. a vision system coupled to the body; the vision system comprises an image capture device; the vision system is configured to use the image capture device to identify fruit for picking.

20. The apparatus of claim 19.

21. 21. The apparatus of claim 20, wherein the vision system is further configured to determine a collision-free picking path for the apparatus to pick fruit.

22. 1. A computer-implemented method for picking fruit, comprising: operating the robot end effector in an extension mode to extend a suction element towards a fruit, the suction element being configured to apply a suction force to the fruit; operating the end effector in a passive mode, in which the suction element is free to move longitudinally along a longitudinal axis of the end effector while applying a suction force by the suction element to the fruit; orienting the end effector towards the fruit to pick the fruit; The method comprising:

23. The method further comprising: actuating a gripper coupled to the end effector to grasp the fruit for picking; rotating the end effector about a longitudinal axis of the end effector to remove the fruit; 23. The computer-implemented method of claim 22, further comprising:

24. The method further comprising: orienting the end effector towards a storage container; operating the end effector in a retraction mode to retract the suction element towards the end effector, wherein the suction element ceases to apply a suction force to the fruit; actuating the gripper of the end effector to release the fruit into the storage container; 24. The computer-implemented method of claim 22 or claim 23, further comprising:

25. The method further comprising: generating a software defined fruit map based at least in part on the data received by the vision system; determining a harvest sequence based at least in part on the fruit map; and determining an approach angle for an end effector for each fruit in the fruit map based at least in part on the fruit map, the harvesting sequence, and a pose estimate; The computer-implemented method of any one of claims 22 to 24, further comprising:

26. Systems, subsystems, structures, apparatus, elements, processes, sub-processes, steps, features, and / or components disclosed herein or shown individually or collectively in the specification of this application, and any and all combinations of two or more of said systems, subsystems, structures, apparatus, elements, processes, sub-processes, steps, features, and / or components.

Images