Robotic fruit picking system

Abstract

The invention is a robotic fruit picking system. The system comprises: at least one picking arm; at least one picking head, mounted on the or each picking arm to (i) either cut a fruit or stalk or stem for a specific fruit or bunch of fruits or (ii) pluck that fruit or bunch; at least one picking head camera mounted on the or each picking head or the picking arm; and a survey subsystem that includes a survey camera module configured to obtain and to analyse image scenes containing fruit-producing crops. The survey camera module is configured to generate data for one or more of the following processes: (i) surveying fruits or fruit bunches; (ii) determining target fruit locations of some or all pickable fruits; (iii) determining approach vectors of some or all pickable fruits in the imaged scene; in each case being controlled to be positioned or oriented, at least in part, independently of a position or orientation of one or more of: the or each picking head camera, the picking arm or the picking head.

Description

[0001] ROBOTIC FRUIT PICKING SYSTEM

[0002] BACKGROUND OF THE INVENTION

[0003] 1. Field of the Invention

[0004] The field of the invention relates to systems and methods for robotic fruit picking.

[0005] A portion of the disclosure of this patent document contains material, which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.

[0006] 2. Description of the Prior Art

[0007] In general, fruit picking robots perform three functions: (i) finding ripe fruit, (ii) picking it, and (iii) packaging it. Finding ripe fruit means both localizing the fruit in a useful coordinate frame, and deciding whether or not it is ripe for picking. Picking means permanently severing the fruit from the plant and, usually, moving it to a different position. Some fruits are picked by grasping the body of the fruit, others by cutting and grasping the stalk. Finally packaging means transferring the fruit to a suitable container (e.g. a punnet) so that the picking end effector can pick the next fruit and all picked fruit can be transported easily. Sometimes picked fruit is inspected before packaging so that the robot can avoid packaging unsaleable fruit or implement supermarket packaging guidelines.

[0008] State of the art fruit picking robots generally perform all three of the functions described above using a single robotic positioning device, e.g. robot arm or flying drone.

[0009] For example, a strawberry picking robot uses a robot arm equipped with a stereo vision camera mounted near the end effector of a picking robot arm. • In the localization phase, the robot moves the stereo vision camera to one or more poses to obtain images of the scene, which are processed to determine the locations of ripe fruit.

[0010] • In the picking phase, the robot uses the same arm to position an end effector so as to cut and grasp the strawberry stalk.

[0011] • In the packaging phase, the robot uses the same arm to transfer the picked berry to an inspection chamber on board the robot where it is imaged to detect defects before being transferred to a punnet or waste chute as appropriate.

[0012] Although current solutions are equipped with multiple robot arms, each arm must perform all three of these functions in sequence.

[0013] As another example, a flying drone may also be equipped with a camera system and picking end effector to pick apples:

[0014] • In the localization phase, the drone moves the camera to obtain images from a variety of viewpoints, which are used to determine the location of ripe fruit.

[0015] • In the picking phase, the same drone is used to position an end effector capable of picking a target apple

[0016] • In the packaging phase the same drone transports the picked apple to a container in which it will be deposited.

[0017] There is therefore a need for a robotic fruit picking system that can perform multiple functions with improved efficiency.

[0018] The present invention addresses the above vulnerabilities and also other problems not described above.

[0019] SUMMARY OF THE INVENTION

[0020] An aspect of the invention is a robotic fruit picking system, the system comprising: at least one picking arm; at least one picking head, mounted on the or each picking arm to (i) either cut a fruit or stalk or stem for a specific fruit or bunch of fruits or (ii) pluck that fruit or bunch; at least one picking head camera mounted on the or each picking head or the picking arm; and a survey subsystem that includes a survey camera module configured to obtain and to analyse image scenes containing fruit-producing crops, in which the survey camera module is configured to generate data for one or more of the following processes: (i) surveying fruits or fruit bunches; (ii) determining target fruit locations of some or all pickable fruits; (iii) determining approach vectors of some or all pickable fruits; in each case being controlled to be positioned or oriented, at least in part, independently of a position or orientation of one or more of: the or each picking head camera, the picking arm or the picking head.

[0021] Another aspect is a robotic fruit picking system, the system comprising: at least one picking arm; at least one picking head, mounted on the or each picking arm to (i) either cut a fruit or stalk or stem for a specific fruit or bunch of fruits or (ii) pluck that fruit or bunch; at least one picking head camera mounted on the or each picking head or the picking arm; and a survey subsystem that includes a survey camera module configured to obtain and to analyse image scenes containing fruit-producing crops, in which the survey camera module is configured to generate data for one or more of the following processes: (i) surveying fruits or fruit bunches; (ii) determining target fruit locations of some or all pickable fruits; (iii) determining approach vectors of some or all pickable fruits; in each case being controlled to be positioned or oriented, at least in part, in parallel or at substantially the same time as the positioning or orientation of one or more of: the or each picking head camera, the picking arm or the picking head.

[0022] Another aspect is a robotic fruit picking system, the system comprising: at least one picking arm; at least one picking head, mounted on the or each picking arm to (i) either cut a fruit or stalk or stem for a specific fruit or bunch of fruits or (ii) pluck that fruit or bunch; at least one picking head camera mounted on the or each picking head or the picking arm; and a survey subsystem that includes a survey camera module configured to obtain and to analyse image scenes containing fruit-producing crops, in which the survey camera module is configured to generate data for one or more of the following processes: (i) surveying fruits or fruit bunches; (ii) determining target fruit locations of some or all pickable fruits; (iii) determining approach vectors of some or all pickable fruits; and in which the position of the picking head camera, the picking arm or the picking head is a function of a prior survey camera module position.

[0023] Another aspect is a robotic fruit picking system, the system comprising: at least one picking arm; at least one picking head, mounted on each picking arm to either cut a fruit or stalk or stem for a specific fruit or bunch of fruits or pluck that fruit or bunch; and a linear rail subsystem, in which the or each picking arm is configured to move along the linear rail subsystem.

[0024] Another aspect is a robotic fruit picking system, the system comprising: at least two picking arms; at least two picking heads, in which each picking head is mounted on each picking arm to either cut a fruit or stalk or stem for a specific fruit or bunch of fruits or pluck that fruit or bunch; and a linear rail subsystem, in which the picking arms are configured to move along the linear rail subsystem; and in which the system is configured to minimise time a picking arm is quiescent.

[0025] Another aspect is a robotic fruit picking system for localising, picking and packaging a fruit or bunch of fruit, in which the system comprises a survey subsystem, a picking subsystem and a packaging subsystem, and in which each subsystem is independent of the other subsystems and / or performs its functions in parallel to the other subsystems.

[0026] Another aspect is a robotic fruit picking system for picking and packaging a fruit or bunch of fruit, the system comprising: at least one picking arm; the picking arm being a fast robot arm, such as 6 degree- of-freedom robot arm; at least one picking head, mounted on each picking arm to either cut a fruit or stalk or stem for a specific fruit or bunch of fruits or pluck that fruit or bunch, and to transfer the picked fruit or bunch to a packaging arm; and the packaging arm being configured to transfer the picked fruit or bunch to an inspection chamber or storage container or punnet, in which the packaging arm is a simple low-cost arm, such as only a 1 or 2 degree-of-freedom robot arm.

[0027] Another aspect is a method of optimizing fruit yield mapping across a fruit farm or multiple fruit farms, by analysing images of fruits to be picked in an imaged scene using the robotic fruit picking system defined above.

[0028] Another aspect is a method of determining fruits suitability for picking by using the robotic fruit picking system defined above, and to determine the picked fruits suitability for retail.

[0029] Another aspect is a method of minimising time a picking arm is quiescent by using the robotic fruit picking system defined above.

[0030] Another aspect is a method of maximizing fruit shelf life by using the robotic fruit picking system defined above.

[0031] Another aspect is a method of picking fruit by using the robotic fruit picking system defined above. BRIEF DESCRIPTION OF THE FIGURES

[0032] Aspects of the invention will now be described, by way of example(s), with reference to the following Figures, which each show features of the invention:

[0033] Figure 1 shows a diagram including a set of approach vectors to pick a target fruit 11 in a horizontally oriented 2D plane.

[0034] Figure 2 shows a diagram illustrating a segment of a robot arm (21), showing the end effector (22), a camera (23) and the camera field of view (24).

[0035] Figure 3 shows an example of the robotic fruit picking system.

[0036] Figure 4 shows an example of the robotic fruit picking system, including a calibration target (41).

[0037] Figure 5 shows a top view of the robotic fruit picking system.

[0038] Figure 6 shows a diagram illustrating fixed working volumes of two picking arms mounted in astatic location.

[0039] Figure 7 shows a diagram showing a simple utilisation balancing system intended to ensure that two robotic picking arms remain fully utilised as they pick the berries.

[0040] Figure 8 shows a diagram illustrating the swept volumes of two picking arms.

[0041] Figure 9 shows a diagram illustrating the swept volumes of two picking arms, one arm being a leading arm, and one arm being a trailing arm.

[0042] Figure 10 shows an example of the robotic fruit picking system in which survey cameras are mounted to fixed geometry in the chassis coordinate frame

[0043] Figure 11 shows an example of the robotic fruit picking system in which survey cameras 38 are mounted to a structure 111 which moves with respect to the chassis 32 along the same linear axes (35) as the picking arms (33).

[0044] Figure 12 shows an example of the robotic fruit picking system in which survey cameras 38 mounted to a structure 121 which moves with respect to the chassis 32 along an additional linear axis 122.

[0045] Figure 13 shows an example of the robotic fruit picking system in which survey cameras 38 mounted to structure 131 which shares one DoF with the picking arm 33.

[0046] Figure 14 shows an example of the robotic fruit picking system in which survey cameras 38 are sharing more than one DoF with the picking arm 33. Figure 15 shows a robotic fruit picking system in which the picking arm is suspended below a linear axis mounted above the machine.

[0047] Figure 16 shows an alternative arrangement with picking arm(s) 33, running on linear axes 35 supported at the machine ends.

[0048] Figure 17 shows an alternative arrangement with all linear axes in pairs either side of the fruit trays, and a bridge structure between the picking arm linear axes supporting the picking arm above the central trays.

[0049] Figure 18 shows a diagram illustrating a robot fruit picking system moving along a crop row bordered by plants and supporting infrastructure.

[0050] Figure 19 shows diagram illustrating the robot fruit picking system with retractable guards.

[0051] Figure 20 shows diagrams illustrating different configurations of the robot fruit picking system.

[0052] Figure 21 shows another example of the robotic fruit picking system with an extended chassis structure 32 and the linear rail 35 being mounted or attached to the extended chassis 32.

[0053] Figure 22 shows a perspective view of the robotic fruit picking system shown in the diagram of Figure 21.

[0054] Figure 23 shows a perspective view of the robotic fruit picking system shown in the diagram of Figure 21, further including a calibration target 41.

[0055] Figure 24 shows a perspective view of the robotic fruit picking system shown in the diagram of Figure 21, further including a calibration target 41 and survey camera modules 38.

[0056] Figure 25 shows a perspective view of one picking arm 33 of the robotic fruit picking system of Figure 21.

[0057] Figure 26 shows a diagram illustrating an inspection chamber.

[0058] Figure 27 shows a diagram illustrating an adjustable reflective diffuser or barrier configured to control how the light illuminates the object.

[0059] Figure 28 shows another example of an inspection chamber for scattering or directing light onto an object in an automated inspection subsystem. DETAILED DESCRIPTION

[0060] Work Parallelization for Efficient Robotic Fruit Picking

[0061] One important limitation of using the same robot manipulator (whether a robot arm or drone or otherwise) to perform the three functions described above, namely finding a ripe fruit, picking it, and packaging it, is that a robot manipulator designed to perform all three functions cannot be so well suited to any one specific function as a robot manipulator designed expressly for that purpose. For example, a 6DOF picking arm capable of dexterously positioning a picking end effector so as to pick a strawberry stalk is likely to be unnecessarily complex (and therefore unnecessarily expensive, power hungry, unreliable, etc.) as a means of transferring fruit to an inspection system and from there to a storage container (or ‘punnet’). In consequence, state-of-the-art robotic fruit picking solutions are slow and achieve poor utilisation of sub-system components.

[0062] A robotic picking solution that achieves much higher speed and better hardware utilization by subdividing the picking work between multiple different sub-systems, each specialised for a particular function, is now described. This approach confers various advantages. Firstly, because there are multiple sub-systems, the work of localizing, picking, and packaging ripe fruit can be parallelised effectively, dramatically increasing picking speed. Secondly, because each sub-system can be designed for a specific function, each component sub-system can be more optimal for its function considering a wide range of design criteria such as complexity, reliability, cost, energy consumption, etc. For example, a simple arm with only 2 or 3 degrees of freedom might be used to transport picked fruit to punnets. In consequence it may be comparatively more reliable or inexpensive to manufacture.

[0063] The methods and systems described herein are applicable to a variety of different crops that grow on plants (like strawberries, tomatoes), bushes (like raspberries, blueberries, grapes), and trees (like apples, pears, logan berries). In this document, the term fruit shall include all fruits, vegetables, and other kinds of produce or waste products that are picked from plants (including e.g. nuts, seeds, vegetables, excess foliage, unwanted branches, etc.) and plant shall mean all kinds of fruit producing crop (including plants, bushes, trees). For fruits that grow in clusters or bunches (e.g. grapes, blueberries), fruit may refer to the individual fruit or the whole cluster. The word ripe shall be interpreted to mean desirable to pick. The techniques described here are adaptable to various growing systems, including, but not limited to, crops grown directly on the ground, in raised beds, on tabletops, or in vertical farming. Since in vertical farming the crop is sometimes moved relative to the ground, references to a robot or vehicle moving with respect to a crop row should also be taken to include the crop row moving relative to the robot.

[0064] Parallelizing Survey, Picking, & Packaging

[0065] The first stage of robotic picking is usually to find ripe fruit. This may be achieved using a combination of (i) one or more survey cameras positioned so as to image the crop and (ii) some image processing software capable of detecting fruit in the survey images, determining whether it is ripe, and localizing it in a suitable 3D coordinate frame. Various different types of cameras may be used for survey, including monocular cameras, stereo cameras, and active or passive depth sensing cameras. Typically, the survey camera is positioned using a robot positioning device such as a robot arm or flying drone.

[0066] In general, it is advantageous to survey the crop from more viewpoints because additional viewpoints may reveal fruit that would otherwise be hidden or outside of the field of view of the camera. Because cameras may be expensive, one advantage in attaching the camera to a robotic positioning device (e.g. a robot arm or a drone) is that more viewpoints can be covered with fewer cameras (or only a single camera). However, the time required to move the camera using the picking manipulator is likely so significantly to reduce the overall productivity of the robot as a whole as to nullify the cost benefit provided by using fewer cameras. This motivates a robot design that uses a greater number of cameras to survey the fruit.

[0067] One approach is to use an array of cameras in fixed position, e.g. with respect to the robot chassis. This approach has the advantage that survey images may be captured from multiple viewpoints with very low time cost, increasing the proportion of time in which the picking arm can pick productively. Another benefit is the possibility of parallelizing fruit localisation (including the imaging and image processing phases) with the work of picking. However, especially in narrow crop rows, a significant drawback of this approach is that the cameras are likely to need to occupy space that would otherwise be available for the movement of the picking arms, limiting the working volume available to the arm and therefore, potentially compromising its speed of operation or the proportion of the ripe fruit that it can reach or its mechanical complexity.

[0068] Another possibility is to use an array of survey cameras capable of being moved in a limited number of degrees of freedom. This approach achieves a compromise between the number of cameras required and the number of viewpoints that can be covered quickly, as well as allowing the survey cameras to be moved out of the way of the picking arm. For example, the array of survey cameras could be mounted:

[0069] • at the front of a robot vehicle so that the cameras image the crop row as the robot advances along it;

[0070] • to a linear rail so that the cameras can be translated relative to the robot chassis or some other part of the robot;

[0071] • to a rotating pole so that the cameras can be rotated relative to the fixed portion of the robot.

[0072] Multi-viewpoint survey for pick trajectory optimization

[0073] For many crops that can be picked by robot, capturing survey views from more viewpoints increases the chances of picking success. One reason is that a wider range of viewpoints increases the chances of revealing fruit that would otherwise be occluded (e.g. by foliage or other fruits) or outside of the field of view of the camera. Furthermore, especially for fruits such as strawberries, raspberries, tomatoes, peppers, cucumbers, etc. that are supported by geometrically complex stalk structures, imaging the crop from a wider range of viewpoints may facilitate identification of an approach vector, or multiple approach vectors, from which a target berry can be picked without the end effector colliding with leaves or other berries or stalks as it approaches the pick position. It is helpful to consider the picking action of a vision-guided robotic fruit picking arm as a stochastic process. Various sources of error in the robot’s hand-eye (i.e. end effector-camera) coordination chain can give rise to missed picks. For example,

[0074] • the target pick point might not be localised precisely in 3D due to noisy interpretation of the image data:

[0075] • inaccurate stereo camera calibration may cause the robot to over- or underestimate depth;

[0076] • the forward kinematic model of the robot arm might not exactly describe the relationship between joint angles and arm pose, e.g. because bearing surfaces are not machined precisely parallel;

[0077] • wind might disturb the position of the berry or other obstacles in between the time when the computer vision system determined the picking trajectory and the time when the end effector reaches the target fruit.

[0078] This stochastic model of the picking action motivates the idea of using an appropriate inference engine to predict the probability of picking success as a function of the intended motion of the robot arm. For example, if the action of the robot arm is informed by an image of the scene, then we might try to predict the probability of picking success as a function of the image and (optionally) a mask describing the target fruit or a point describing the intended point at which a stalk will be severed.

[0079] For robots equipped with survey cameras mounted to robotic actuators, there is usually a trade-off between visiting more viewpoints and time spent. Given the location of a target fruit, a simplistic approach is to move the camera to a sequence of viewpoints, imaging the target at each viewpoint to determine whether the corresponding trajectory has probability of picking success greater than some threshold value. If it does, then the robot should attempt to pick the target. If it does not, then the robot should move on to the next viewpoint or abandon the berry. In practice, this approach is sub-optimal because the robot might commit early to a picking trajectory with a lower probability of picking success than one of the as-yet unexplored trajectories. It is desirable to explore all approach trajectories before attempting to pick - but there is a trade-off between optimality of the picking trajectory and time spent. Figure 1 shows a diagram including a set of approach vectors to pick a target fruit 11 in a horizontally oriented 2D plane defined by the point at which the stalk will be cut or the fruit will be contacted by the end effector. To pick a target fruit 11, a robot arm’s end effector must approach along a given trajectory. For illustration, the diagram shows a set of approach vectors in which obstacles such as other berries 12 or foliage 13 may mean that picking is likely to succeed for some approach directions (Pickable) but not others (Unpickable).

[0080] One challenge inherent in using multiple survey views to infer the 3D positions of targets (e.g. ripe fruits) is to match detected targets from survey view to survey view - so that unique targets are detected precisely once.

[0081] For a survey system that uses cameras capable of determining the 3D position of fruit directly, such as stereo or depth cameras, a simple way of doing this is to map berry position estimates obtained by the camera into an appropriate shared 3D coordinate frame. Then a clustering algorithm can be used to determine which survey view detections correspond to the same target berry. The parameters of the clustering algorithm should be optimised such that detections corresponding to the same target fruit will tend to be clustered together and detections corresponding to different target fruits will tend to be clustered apart. For clustering purposes, berries may be characterised by feature vectors comprising their 3D position as well as other aspects of their appearance or shape, such as mean colour or volume, or relative position.

[0082] If the uncertainty associated with estimated 3D berry positions can be modelled, then clustering can be implemented using a pairwise berry-to-berry distance model that reflects this positional uncertainty. For example, a point reconstructed in a stereo camera frame has more uncertainty in the direction parallel to the optical axis of the cameras and less in the perpendicular directions.

[0083] For a survey system that uses monocular cameras, 3D berry positions can be estimated by matching corresponding berries from view to view in 2D and then using triangulation. Various cues can be used to facilitate inter- view matching, such as berry colour or shape, or the epipolar constraint. Since real computer vision systems are likely to generate some spurious target detections, a useful feature is to reject any detections that do not appear in more than some threshold number of survey views. This approach works because detected berry positions are likely to be wrong at random, so that wrong detections will not reinforce one another at the clustering phase.

[0084] The suitability for picking of a target fruit (typically a function of ripeness) is typically determined from the appearance or size of the fruit. If a target fruit has been matched over multiple survey viewpoints, then another useful feature is to determine picking suitability by fusing information from all available viewpoints. For example, for fruits like strawberries for which colour is indicative of ripeness, the mean colour might be computed over all available views. Position measurements obtained with respect to multiple different survey viewpoints might also be combined to improve the accuracy of overall target position estimates.

[0085] Some fruit picking robots use an end effector designed simultaneously to grip and to cut the stalk. Typically, a computer vision system is used (i) to detect the target fruit in an image, (ii) to confirm that the fruit is ripe, (iii) to associate the target fruit with a stalk, and (iv) to determine where on the stalk to cut. Generally, the robot control system must also determine a 3D trajectory by which the end effector can be moved to the stalk without collision, e.g. with other fruits, or stalks, or leaves. If the pick point can be seen from a particular viewpoint, then a useful approach is to move the end effector along the ray defined by the camera optical centre and the pick point; since the target pick point could be observed along this ray, the implication is that there are no sources of occlusion along the ray. In practice, fruits and stalks are often occluded, e.g. by other stalks or by foliage, and so it is highly advantageous to explore multiple viewpoints and multiple approach vectors. For a robot with a camera mounted onto the end of a robot end effector, the camera might be moved to multiple viewpoints until one is found that seems likely to give picking success.

[0086] This approach may allow a trade-off. Should the robot keep moving in the hope of finding a better approach vector or pick the berry along the best approach vector discovered to date? However, it is noteworthy that the chances of picking success are only increased by spending more time - which is undesirable from the perspective of robot productivity. This gives rise to a different approach to the identification of pick points, which is to capture images of the scene from an array of survey viewpoints.

[0087] The following scheme is used for picking:

[0088] 1. A survey system obtains images of the scene using suitable cameras (e.g. monocular or stereo or depth sensing cameras)

[0089] 2. Computer vision image processing is used to detect unique target fruits in the 3D camera coordinate frames.

[0090] 3. Since each target fruit may be observed from multiple survey viewpoints, each survey view in which the target appears is used to generate a pickability score for the target. This reflects the probability that an attempt to pick the target fruit along a trajectory defined as a function of the survey viewpoint (e.g. along the ray between the optical centre of the survey camera and the contact position on the fruit or its stalk) will succeed.

[0091] 4. A picking arm moves the picking end effector (possibly along this ray) so as to achieve a suitable vantage point for picking the target.

[0092] 5. Optionally, a camera equipped onto the picking arm, e.g. near the end effector, may be used to capture one or more intermediate images as the end effector approaches the target berry. These intermediate images may be used (i) to achieve more precise positioning to of the end effector with respect to the target (visual servo), (ii) to determine a optimal pick point, or (iii) to obtain a more accurate measure of the ripeness of the target berry, e.g. by using colour cues.

[0093] 6. The robot attempts to pick the target fruit.

[0094] 7. Optionally the robot might try to determine whether picking has been successful, e.g. by visual inspection of the end effector jaws or their contents. If picking was unsuccessful, then the robot might reasonably try to pick the same berry via a different approach trajectory, e.g. the one with the second highest pickability score, or move on to the next target.

[0095] One way to compute a pickability score in (3) above is to use machine learning to train a suitable model to predict the chances of picking success given an image of the target and optionally a segmentation mask that defines the location of important parts of the target, e.g. the body of the fruit of the stalk. In a robot equipped with an on-board quality control system, training data can be provided by validating whether actual pick attempts made by the robot were successful or not.

[0096] Camera calibration

[0097] To allow target fruits detected using survey cameras to be picked by a robotic picking arm, it is usually necessary to relate the positions of fruits detected in a survey camera coordinate frame into the robot arm coordinate frame. Then, given a target end effector pose computed based on the arm-frame position of the target fruit, the arm configuration (i.e. the set of joint angles or positions) required to achieve that pose may be determined via inverse kinematics. Because it is difficult to determine the geometric relationship between survey camera and arm coordinate frames by careful control of manufacturing tolerances alone, it is usually advantageous to determine the relationship by a simple computer vision calibration process. A natural way of doing this is to define a shared 3D coordinate frame in a chassis-relative way by determining camera pose with respect to a calibration target designed to be mounted temporarily to the robot chassis, e.g. during a camera commissioning phase of manufacture. Alternatively, the calibration target might be mounted to the crop row or the ground.

[0098] A suitable calibration target might be a planar target with fiducial markers designed to be easily detected by computer vision and known 2D position on the target plane. E.g. popular embodiments use chessboard or other grid-based calibration patterns (see e.g. Figure 4). A useful feature is to provide fixings on the calibration target and / or the robot chassis intended to allow the target quickly and conveniently to be attached to and removed from the robot chassis, and to ensure that the target frame has a reasonably predictable position with respect to the robot chassis. The camera calibration operation would typically need to be repeated when new survey cameras are attached to the robot or when survey camera position is changed.

[0099] Because the survey camera pose calibration process may be cumbersome or time consuming (e.g. because it requires a calibration target to be positioned near the robot or mounted to the robot), it may be desirable to avoid repeating the operation unnecessarily. This can be facilitated by introducing a means of validating that pose calibration is still sufficiently accurate without the need to repeat the pose calibration process. This can be achieved by using a smaller calibration target that is visible to the survey cameras but which does not occlude the scene. The system can check that the reconstructed positions of fiducial markers on the smaller target are consistent with their reconstructed positions immediately after calibration.

[0100] It may be desirable to allow the pose of survey cameras to be adjusted better to suit the geometry or condition of the crop. For example, for crops such as strawberries that are sometimes grown on tabletops, it may be useful to allow the height of the survey cameras to be easily adjusted better to suit the height of the tabletops on which the crop is supported. This might be achieved e.g. by allowing survey cameras to be slid up and down a pole. After survey camera pose is adjusted, it is advantageous if survey camera calibration can be conducted quickly. This motivates the idea of integrating the calibration target permanently with the robot. For example, the target might be accommodated using a part of the robot that is designed to be positioned temporarily in the field of view of the survey cameras.

[0101] Camera pose is typically expressed as the combination of a 3x3 rotation matrix R and translation T that allow 3D points in the camera frame to be mapped to homogenous 2D points u in the image frame: where ~ means equality up to an arbitrary scale factor. For a camera capable of determining the 3D position of a point (e.g. a stereo camera or a time-of-flight depth camera), camera pose relates a 3D position Xtin the calibration target frame to a 3D position Xsin the camera frame:

[0102] Xs= RXt+ T

[0103] In the case of a robot with survey cameras that move independently of the vehicle chassis, camera pose in the vehicle chassis frame will be a function of survey arm pose. For example, for a survey camera that moves along a straight line path relative to the chassis frame:

[0104] Xs= RXt+ T + Ad Here, 4 is a unit 3-vector reflecting the direction of travel of the camera in the camera frame and A reflects the distance travelled in this direction.

[0105] The parameter is set in practice by moving the camera by a certain distance, e.g. a distance determined from the number of revolutions made by a motor, and then obtaining an image. However, in the interests of the speed of the survey operation, it may be desirable to obtain images while the survey camera is still moving. For a moving camera, it may be challenging to determine A with sufficient accuracy. This gives rise to a useful feature, which is to allow the arm to provide a synchronisation signal to the camera. This might be accomplished e.g. by using an electrical output from the arm to synchronize a clock in the camera. Then images can be timestamped with respect to arm motion, which can often be determined very precisely as a function of time.

[0106] Fruit picking robots usually need to be low cost to be commercially useful. However, low-cost robots tend to have high manufacturing tolerances. Therefore it may be challenging to position an end effector with sufficient precision in the target frame to pick the fruit, particularly for robots that pick fruit by the stalk. This motivates the idea of equipping a camera onto a part of the picking arm (e.g. the end effector) so that visual servo can be used to adapt the precise position of the picking arm as it approaches the target fruit. In other words, as the arm approaches the target fruit, one or more images of the target is obtained and used to update the target pose estimate. By this means, the arm needs good relative accuracy instead of good absolute accuracy.

[0107] This necessitates obtaining an accurate estimate of the geometric relationship between the end effector and the arm-mounted camera. One way of achieving this is to ensure that the end effector is (at least partially) visible within the arm camera’s field of view. Then the image position of the end effector (or a part of the end effector) can be used to determine the geometric relationship between the end effector frame and the camera frame (Figure 2). The image position might be determined by manual annotation or computer vision.

[0108] The end effector is positioned so it is visible in the camera frame such that it provides a convenient way of calibrating the geometric relationship between the end effector and the camera frame, facilitating accurate positioning of the end effector at a pick point determined in the camera frame. Figure 2 shows a diagram illustrating a segment of a robot arm 21, showing the end effector 22, a camera 23 and the camera field of view 24. The field of view of the camera 24 includes the end effector, allowing part of the end effector, typically (but not necessarily) the tip 25 to be visible in images captured by the camera.

[0109] Figure 3 shows an example of the robotic fruit picking system. The system includes: a vehicular platform with tracks 31, a suspended chassis 32. Two picking arms 33 are supported by bridges 34 that are designed to move along a pair of linear rails 35. The bridges provide space between the linear rails for a packaging system comprising packaging arms 36 (to which the picking arms hand off picked berries), an inspection module, and trays 37 for storing picked fruit. Two groups of survey cameras 38 are mounted to the bridges via vertical poles 39 that can rotate about a vertical axis to allow both sides of the crop row to be imaged with a smaller number of cameras.

[0110] Figure 4 shows an example of the robotic fruit picking system, including a calibration target 41 mounted to the robot, static in the chassis frame of reference, to position targets to be viewed by the survey cameras.

[0111] Figure 5 shows a top view of the robotic fruit picking system, showing a picking arm base axis 51, Packaging arms 36, Fruit trays 37, Waste chutes 52, and Inspection subsystem 53.

[0112] Camera Colour Calibration

[0113] For crops such as strawberries and tomatoes, ripeness can be determined by the colour of the fruit in an image. However, the colour of incident daylight may change throughout the day, with changing weather conditions, or as a function of position in the scene, and individual cameras may have differing colour response. This means that making sufficiently accurate colour measurements using images obtained by ordinary colour (or multi spectral) camera necessitates accurate colour calibration. Colour calibration might be achieved by imaging a suitably controlled grey card target and adapting the camera’s channel gains so that the grey card appears grey in the resulting image. By this means, the imaging system can compensate for the colour temperature of the incident light. However, this approach presents several challenges in this context. If the camera in question is mounted in fixed position relative to the robot (e.g. a survey camera mounted to a robot’s chassis), it will be impossible to mount the grey card in a static position within the camera’s field of view without occluding part of that field of view. Additionally, it may be impossible to position the grey card both within the camera’s field of view and such that the light incident upon it is sufficiently similar in colour to that illuminating the target fruit or fruits to make an accurate enough reading of colour temperature (the light incident upon the fruit may have been reflected by green foliage or other surfaces that modify its colour temperature compared to daylight). This motivates an approach to colour calibration as follows:

[0114] 1. The colour response of one or more cameras is measured by an a priori calibration process, e.g. at manufacturing time using a light source with carefully controlled colour temperature.

[0115] 2. Target fruits are localised in a convenient shared coordinate frame using images obtained from one or more viewpoints.

[0116] 3. The colour temperature of the light incident upon the target fruit or fruits is measured using one or more spectrometers (e.g. low-cost, small, single chip spectrometers) positioned so as to be illuminated by light of a similar colour to that illuminating the target fruit. Multiple spectrometers may be deployed with known position in the shared coordinate frame. Then the spectrometer or spectrometers closest to or near to a particular target fruit or target fruits may be used to determine the colour temperature of the incident light at the target location. Alternatively, a moveable robot arm may be used to position a spectrometer mounted on that arm near to the target fruit. Unlike the grey card, the spectrometer does not need to be positioned in the camera’s field of view and can be positioned on the end of the robot’s moveable picking arm so that the robot can position the spectrometer close enough to the previously localised target fruit to make an accurate colour temperature reading of the light falling on it. However, this idea generalizes to an approach that uses grey cards too, as long as the grey cards are visible in the field of view of at least one colour calibrated camera. 4. Finally the colour of a particular target fruit may be measured accurately by compensating for the measured colour temperature of the incident light as well as the colour response of the camera in which the target is imaged, e.g. by applying appropriate channel gains.

[0117] Utilization Balancing for Multiple Picking Arms

[0118] Sometimes, it is optimal for a single mobile picking robot to share the work of picking between multiple robotic picking arms. The main motivation for using more than one arm is to amortise the large cost of the robot vehicle over more, productive picking capacity. However, several factors mean that there may be disadvantages in having too many picking arms. One is that too many picking arms might make robots less manoeuvrable. Another is that it may be difficult fully to utilise more than one picking arm because picking arms can only be active for as long as they have meaningful picking work to do. For illustration, consider a robot with two picking arms (as shown in Figure 6). When one arm has picked (or attempted to pick) all the berries that it can reach, then it must typically stop and wait while the other arm continues to pick the berries that it can reach. Otherwise, the robot could move along the crop row to bring more berries into reach but probably only at the expense of leaving some berries behind. Similar considerations apply whether the robot stops and starts as it moves along the row or moves continuously along the row. Note that this asymmetry arises purely because of the uneven spatial distribution of fruit along the crop row. For a strawberry robot with two arms in fixed position, and at representative berry density, we found in practice that the reduction in arm utilization arising because of statistical asymmetry in the number of berries available to front and back arms was around 30%, which significantly reduces the benefit provided by adding the second arm. Similar evaluation showed that it was not economic to add a third arm in this specific application because the increase in capacity was not enough to offset the reduction in overall utilization.

[0119] Figure 6 shows a diagram illustrating fixed working volumes (represented schematically by dotted rectangles 61 and 62) of two picking arms (which could be on the same side of the robot or on different sides) mounted at a static location. A ripe fruit 63 is also shown. The working volumes of the two arms typically would not overlap so as to ensure that the arms cannot collide with one another. There are fewer berries in the working volume of the first arm 61 than in that of the second arm 62. Thus, if both arms pick berries at about the same rate, the first arm will be unused while the righthand arm completes its work. Some fruit 64 may be inaccessible to either arm.

[0120] These challenges motivate the approach of picking arms that can travel backwards and forwards along the crop row independently of the robot vehicle. For a robot that moves along the crop row via a series of moves, this has the advantage that much higher utilization of both arms can be achieved (see Figure 7). For example, the two arms could start picking at opposite ends of the robot, picking detected berries one by one until the arms meet. If both arms have to pick the same number of berries, they will meet approximately in the middle of their travel. If one arm has to pick more berries than the other, it will progress more slowly than the other and so the meeting point will be shifted away from the middle. In each case, both arms pick continuously until all the fruit is picked.

[0121] Figure 7 is a diagram showing a simple utilisation balancing system intended to ensure that two robotic picking arms remain fully utilised as they pick the berries. The two picking arms start at opposite ends of the machine, and work towards the middle. In the figure, the first arm starts in step 1 by picking berry (A), and the second arm starts by picking berry (B). In Step 2 the arms have progressed to pick the next berry in an inwards direction. If berries are approximately evenly distributed, then the two arms will converge approximately in the middle of the machine. If berry distribution is biased towards one end of the machine, then the arms will converge away from the middle. In each case, both arms will be fully utilised until picking is complete (for an even number of berries).

[0122] In practice, the swept volume required by the arm to pick the berry requires that it is not always possible to pick two berries which are close together. As illustrated in the diagram of Figure 8, in order to pick a fruit (A), the first picking arm must move through swept volume (C), and the second arm must use swept volume (D) to pick berry (B). This does not pose a problem for the first three stages depicted, however in stage 4, the intersection of swept volumes prohibits both arms from picking concurrently, and one arm must wait for the other to complete a pick before starting itself to pick, reducing picking arm utilisation. In practice, for machines of a practical length, and arms sufficiently large to reach berries, this can affect the ability to pick a significant proportion of berries, and leads to poor arm utilisation.

[0123] In practice, real robot arms cannot meet without colliding because the arms require space in which to move during picking (see Figure 8). A mitigation is to pick the last berries with only one of the two arms, but this undermines the intended arm utilisation benefits. An alternative strategy is to have the two arms travel in the same direction during picking, maintaining broadly the same spacing between themselves. A leading arm can start picking approximately in the middle of the robot, a trailing arm starts picking at one end of the robot. The middle of the work is determined e.g. by assigning the first half of the berries to be picked to the leading arm and the second half of the berries to be picked to the trailing arm (see Figure 9). Note that this approach works well under the assumption that the working volume required by an arm for picking is smaller than the spacing between the two arms.

[0124] A system that can determine the central berry can achieve higher utilisation balancing between arms. As illustrated in the diagram of Figure 9, the first arm starts by picking berry (A) sweeping through volume (C). At the same time, the second arm starts to pick berry (B) (previously determined to be the middle berry) sweeping through volume (D). In this case all four steps can be completed, picking all berries without the swept volumes of the arms intersecting and interrupting picking. Both arms are therefore fully utilised.

[0125] A summary of our utilisation balancing approach is as follows:

[0126] 1 . The survey system determines the position of all target berries in a convenient coordinate frame, e.g. a chassis coordinate frame. The survey system also determines the approach trajectory for each berry as described above.

[0127] 2. The system orders the detected berries and assigns the first half to the leading arm and the second half to the trailing arm. Then the trailing arm starts picking at one end of the linear rail and the leading arm starts picking at the middle point (typically near the middle of the robot). Both arms progress along the rail from front to back, staying at an approximately constant distance apart and therefore usually avoiding collision without the need to interrupt the motion of either arm. 3. It is important that the picking arms should transfer picked fruits to storage containers or hand off picked fruits to separate packaging arms in outboard locations.

[0128] 4. Berries may be ordered in the order in which they appear in the chassis coordinate frame, e.g. from left to right. However, it may be more desirable to pick berries in order of the extent of the swept volume required for the arm to pick them. This approach allows for the fact that berries will be approached from various angles and the amount of 'elbow room' required will therefore differ berry by berry.

[0129] Note that although this scheme is described in the context of a two-arm system, all of the utililization balancing ideas described here generalize easily to robots with more than two arms.

[0130] To guarantee no collision between the two picking arms, neither picking arm may be allowed to move in the space required by the other picking arm. The space required for each arm to move during the course of picking the next target berry may be described by bounding box geometry or otherwise. Physics simulation of arm movement can be used to determine whether the arms will collide. In the event that there is imprecision in the timing of movement, for example because the two arms are controlled by separate threads, it will be desirable to allow a margin for error.

[0131] Collision between leading and trailing arms can be prevented by having the trailing arm reserve working volume and the leading arm free working volume as the two arms progress along the linear rail. For example, this might be achieved by having two software threads responsible for controlling the leading and trailing arms reserve and free working volume under a shared lock. As the leading arm moves from the middle of the work towards the leading end of the rail it will progressively release working volume, making it available to the following arm. As the trailing arm moves from the trailing end of the rail towards the middle of the work, it will lock working volume. If the trailing arm needs more working volume than has been released by the leading arm then picking with the trailing arm picking should be blocked until the leading arm has released enough working volume. Note that this scheme will work best if (4) is implemented. Working volume might be described using 3D voxels (cuboidal regions of 3D space) that can be locked and released independently by the leading and trailing arms or by a pair of planes oriented with normal perpendicular to the direction of travel of the leading and trailing arms. The position of the first plane describes the furthest extent of travel of the trailing arm as it picks the next berry, the position of the second plane describes the furthest extent of the leading arm as it picks the next berry. At the start of picking, the leading arm plane will be further forward than the trailing arm plane. If this constraint would be violated during picking, then the trailing arm would have to wait.

[0132] One problem with the two-arm utilisation balancing approach described above is that a bad initial estimate of the middle of the work will mean one arm finishes picking before the other, compromising arm utilization. This might happen e.g. because (i) we have to make an approximate estimate of the midpoint quickly to avoid delaying the start of picking or because (ii) picking speed is hard to model accurately (e.g. because working volume considerations means that the leading arm blocks picking by the trailing arm unpredictably). An alternative solution is to have the leading and trailing arms pick in multiple back and forth passes along the robot. For example, the arms might make a first pass travelling in a front-to-back direction and then a second pass in a back-to- front direction, picking only half the total available berries in each pass. The key insight here is that the trailing arm can turn around and start moving in the back-to-front direction a little earlier or later than originally planned according to the progress actually made by the two arms. For example, if the leading arm hasn't reached the end of the linear rail when the trailing arm has reached the middle of the work, the trailing arm can pick a little past the originally predicted midpoint to give the leading arm a chance to 'catch up'. Conversely, if the leading arm has reached the end of its travel before the trailing arm has reached the original predicted midpoint, then the trailing arm can turn around sooner.

[0133] One way of dividing the available berries between passes is to divide the left and the right hand sides of the robot (which has the obvious advantage that is apparent to users and developers what the robot is doing). Another advantage of this approach is that, with appropriately positioned survey cameras, it is possible to obtain survey images of one side of the robot as the arms pick on the other side. One could also divide the berries into more than two sets and have the arms make more than two passes. Provided there are enough berries that atomicity problems can be avoided (and ignoring the time cost of an increase in total lateral arm movement, which may be negligible if it can be overlapped with other movements), in the limit as the number of sets tends to infinity, the productivity loss tends to zero. In practice using two or three passes provides significant utilisation benefit.

[0134] Machine layout

[0135] Fruit picking robots typically use the same robot arm to perform the survey, picking, and packaging tasks on each berry. Multi-arm machines are typically configured to minimise intersection of working volumes between arms, allowing the arms to work substantially independently of one another.

[0136] A robot capable of parallelising survey, picking, and packaging might be equipped with modules designed for the completion for each function, with various degrees of independence (for example they might be completely independent, or share computer processing resources, or robotic degrees of freedom). In general terms, we speak of:

[0137] 1. A survey module, designed to gather images of the scene, and interpret the result

[0138] 2. A picking arm, designed efficiently to remove berries from the plant

[0139] 3. A packaging arm, designed to perform on-board tasks with picked berries. The packaging arm could be an arm with fewer DoF, designed to transport and inspect berries, and place them in punnets. Without loss of generality it could also apply to higher or lower DoF systems including multiple rotary DoF arms, or carousel based systems.

[0140] Two core challenges are inherent in designing the layout of a picking robot capable of parallelising fruit localisation, picking, and packaging:

[0141] 1. Firstly, it is desirable that the picking arm approaches the fruit by moving its end effector along a vector that at least approximately corresponds to the ray along which the survey camera(s) imaged the fruit, since approach vectors surveyed give the best available information about the pickability of the berry. In the confined space of the crop row, where cameras cannot be mounted further away due to collision with the adjacent row, the survey camera and picking arm must occupy the same region of space, avoiding a collision only due to a separation in time.

[0142] 2. Secondly, the working volumes of the picking and packaging arms must overlap to allow transfer of picked fruits between the two when picking is completed at at least one mutually accessible location. The machine must be designed to minimise the impact on the picking arm’s rate of picking by the need to visit a mutually accessible location, and to achieve sufficient positional precision at this point to allow the transfer of the fruit to be completed reliably.

[0143] There are a range of secondary drivers in the machine design. These include: Easy and safe access to trays of fruit by the operator; Minimisation of overall package height (to allow picking of fruit close to the ground, and to minimise the height of the centre of gravity); Separation of the linear rails across the machine; and proper guarding of the hazardous aspects of the machine to protect the operator from harm.

[0144] Relationship between survey system and picking arms

[0145] Several embodiments address the first challenge above - allowing both survey cameras and picking arms to access the same region space during picking.

[0146] Figure 10 shows an example of the robotic fruit picking system in which survey cameras are mounted to fixed geometry in the chassis coordinate frame, which traverses past the crop as the robot drives down the row. Survey cameras 38 mounted to a structure 39 fixed in a chassis 32 coordinate frame can survey picking volume during movement of the vehicle along the crop row. Picking arms 33 moving on a linear axis 35 subsequently pick identified fruit. Cameras are shown mounted to both ends of the machine but could also be mounted to only one to reduce camera costs and system complexity.

[0147] This class of solutions benefits from mechanical and control simplicity, by default separating survey and picking modules. However it suffers from several drawbacks: 1. The machine often drives along uneven ground, which considerably exacerbates the problem of relating points from the (moving) camera frame into the arm coordinate frame.

[0148] 2. Obtaining additional survey images of a section of row that the robot has already passed (e.g. because picking has subsequently revealed some hidden fruit) would necessitate reversing the machine, incurring a significant time penalty.

[0149] 3. The survey cameras are separated from the picking arms by a long chain of components, each with expected manufacturing tolerances and other deviations from nominal geometry. This reduces the reliability of pose determination between the survey cameras and picking arm, tending to reduce picking performance.

[0150] Considering these drawbacks, an alternative idea, shown in Figures 11 and 12, is to mount the cameras to one or more single- or multi-DOF survey arms. This addresses the first drawback, by allowing survey to progress while the chassis is stationary, and with the position of the survey cameras at the time of image capture well defined through the motion of robotic axes. The second drawback listed above is improved as survey arms can now be used to repeat survey parts of the scene without moving the chassis along the row.

[0151] Figure 11 shows an example of the robotic fruit picking system in which survey cameras 38 are mounted to a structure 111 which moves with respect to the chassis 32 along the same linear axes 35 as the picking arms 33. This approach makes it relatively straightforward to relate the positions of target fruits between survey camera and arm coordinate frames, but the shared linear axis increases system complexity.

[0152] Figure 12 shows an example of the robotic fruit picking system in which survey cameras 38 mounted to a structure 121 which moves with respect to the chassis 32 along an additional linear axis 122. Picking arms 33 mounted to a separate linear axis 35 pick fruit. Layout could be configured to allow survey cameras to pass picking arms.

[0153] However, this class of embodiments generally may still suffer from the third drawback listed above (since survey cameras and picking arms are still separated by variable geometry components), although to a lesser extent, and suffers from a further drawback: 4. The provision of additional robotic arms for survey camera control increases system cost and complexity. For machines of a length only several times longer than the characteristic length of the working volumes of their picking and survey arms, the drive to move survey cameras through the same volume as picking arms generally leads to poor utilisation: For a large proportion of time picking arms must remain unused while survey arms occupy working volume, and then the survey arms must leave the space and remain unused while the picking arms pick berries.

[0154] Another approach involves survey cameras sharing one or more degrees of freedom with the picking arm, possibly including additional dedicated degrees of freedom (see Figures 13 and 14). This class of embodiments avoids the complexity of additional survey arms and control architecture, better amortising the costs of at least one degree of freedom across both survey and picking functions. Secondly, the picking arm and survey cameras are now separated by fewer degrees of freedom and variable geometry components, simplifying the challenge of pose determination between survey cameras and picking arms. Further, during picking, the survey cameras are likely to move relative to the chassis. At low time cost, additional images might be captured from the survey cameras, increasing available data on remaining fruit within the working volume of the picking arm and improving pick performance.

[0155] Figure 13 shows an example of the robotic fruit picking system in which survey cameras 38 mounted to structure 131 which shares one DoF (in this case a linear axis 35) with the picking arm 33 and which moves relative to the chassis 32. The structure 131 might also provide one or more dedicated survey DoFs to the cameras. The structure 131 might be detachable from the picking arm base 131 to alleviate restrictions on working volume while picking.

[0156] Figure 14 shows an example of the robotic fruit picking system in which survey cameras 38 are sharing more than one DoF (in this case a rotary 141 and linear axis 35) with the picking arm 33. All of the above classes of embodiment might include lighting elements in a fixed position relative to the survey or arm cameras, better to illuminate the scene or to ensure control of the colour temperature of incident light. Fixing such lights with respect to the cameras better controls image consistency as the cameras traverse their working volume and so performance and consistency of berry detection and pick point determination. Lighting elements fixed in the chassis frame might be used in addition or alternatively to further improve lighting or reduce system cost and complexity. To reduce the power consumption of the lights and facilitate the use of lights with lower output power ratings, it may be desirable to use flash lighting, activating the lights only during the desired camera exposure window. Unfortunately, flash lighting can be distracting for nearby human workers and, at some frequencies, can even provoke epileptic seizures. This motivates the alternative idea of strobing the survey lighting continuously at a sufficiently high frequency (e.g. 40Hz) that the strobing is not perceptible to humans and timing camera exposures so as to coincide with one or more pulses. This requires much lower power than operating the lights continuously but still allows adequate exposure during the camera’s exposure window.

[0157] Minimising survey mast vibration

[0158] Several embodiments considered above include the provision of a mast - a rigid body such as a pole or robot arm to which cameras are mounted, and which can be driven in one or more degrees of freedom to capture survey images across the scene, better amortising camera costs. It is important to know the position of this mast and its cameras when images are taken, and vibration of the mast can compromise survey performance.

[0159] Normally, vibrations might be addressed through measures such as supporting the mast at either end, or increasing stiffness to reduce the amplitude of vibration. These solutions, however, typically reduce performance through added mass or cost. They may also have negative implications on the overall machine layout which increase the volume of mast assembly (making it more likely to collide with berries or infrastructure), or require, for example, the provision of additional linear axes or structure above the machine, raising the centre of gravity. Three useful improvements to the survey mast design can alleviate these problems:

[0160] (1) Stemming from the observation that the mast is likely to have a consistent centre of mass and stiffness, the addition of a tuned mass damper tuned to the resonant frequency of the mast can increase damping (and so reduce vibration) with minimal increased mass and cost.

[0161] (2) Further, excitation of vibration might be reduced through the avoidance of moves with frequency components at or close to the resonant frequency of the mast.

[0162] (3) Finally, it is possible to compensate for vibration by the addition of an inertial measurement unit to the system, measuring acceleration of the survey cameras, and applying a correction to the assumed position of the camera at the time of capturing the image.

[0163] Relationship between picking and packaging arms

[0164] As described in the second challenge above, “The working volumes of the picking and packaging arms must overlap to allow transfer of picked berries between the two when picking is completed at at least one mutually accessible location. The machine must be designed to minimise the impact on the picking arm’s rate of picking by the need to visit a mutually accessible location, and to achieve sufficient positional precision at this point to allow the transfer of the fruit to be completed reliably.”

[0165] It is also generally beneficial for the design to:

[0166] • Minimise the highest point of static hardware. This brings down the minimum height from the ground that fruit can be picked, and lowers the centre of gravity of the machine improving chassis behaviour and stability.

[0167] • Allow the packaging arms to traverse their full working volume without interfering with the picking arms. This increases the independence of picking and packaging arms, allowing asynchronous behaviour (other than when synchronisation is required to allow passing of a picked berry between arms)

[0168] • Maximise cross-body stiffness, and minimise backlash. This reduces the degree of deflection of system components as the robot arms move, improving accuracy and precision of movement, and improving picking and packaging performance.

[0169] • Allow trays to be unloaded from the (generally accessible) ends of machines. This improves operator efficiency, and is explored in more detail in a later section.

[0170] A solution to this challenge is required to enable the parallelisation of picking and packaging arms, a challenge not required by conventional fruit picking machines. Three approaches are now described which attempt to satisfy the challenges above.

[0171] An improvement to some designs already in the market, shown in Figure 15, suspends the picking arm below a linear axis mounted above the machine and accommodates packaging arms and fruit storage trays below. This has the advantage of spatially separating the lower (and generally more cumbersome) axes and joints of the picking and packaging arms, making collision avoidance less complex, and giving a wide range of mutually accessible locations between picking and packaging arms, permitting efficient operation.

[0172] Figure 15 shows a robotic fruit picking system arrangement with picking arm linear axes 35 supported at the machine ends and positioned above the fruit trays and supporting picking arm 33. The packaging arms 36 run along linear axes 151 either side of trays of fruit 37 fixed in the frame of reference of the chassis 32. There are few obstructions between the picking and packaging arms, making transfer of picked fruit efficient.

[0173] The above design has several drawbacks, however:

[0174] 1. First, the picking arm linear axis (generally the heaviest component in the picking arm) must be supported high up in the machine. This raises the centre of gravity of the machine significantly, reducing the usability of the robot chassis and making it more likely to tip when driving up / down / across a slope. The height of the axis also restricts activities such as loading the robot into vehicles (e.g. vans) with restrictive headspace.

[0175] 2. Second, the linear axis for the picking arms must be supported by the chassis. This requires the provision of structural components between the chassis and linear axis, that traverse the range of heights where the picking arms are moving. These support elements can be positioned within the working volume of the picking arms, in which case they impose restrictions on the picking arms, which need to avoid them while picking. Alternatively they can be positioned outside the ends of the working volume of the picking arms, increasing the overall machine length and increasing design complexity.

[0176] 3. Thirdly, the static mounding points for the linear axes of the picking and packaging arms are now far away from one another, and typically linked by a series of structural components. This makes it more challenging to achieve sufficient relative precision between the picking and packaging end effectors at the point of handing off picked fruit, and negatively influences reliability, machine cost, and / or complexity.

[0177] An alternative design, shown in figure 16 might arrange the linear axes for the picking arms above the central trays. This design significantly reduces the height of fixed hardware, improving the centre of gravity of the machine, and brings the linear axes for the picking and packaging arms closer, allowing better tolerance control between them, improving the reliability of handing picked fruit between them.

[0178] Figure 16 shows an alternative arrangement with picking arm(s) 33, running on linear axes 35 supported at the machine ends (fixed relative to the chassis (32)) and positioned above the fruit trays 37. Packaging arms 36 and their linear axes 151 run along the sides of the machine. The mass of the linear axis 35 is lowered, however obstructs access to the packaging arms by the picking arms (and vice versa).

[0179] However, this design may also suffer from various drawbacks: Firstly, the picking arm linear axis is now generally inside the working volume of the picking arm (to enable the picking arm to reach past it to access the packaging arm). To make efficient use of the picking arm hardware, it is also desirable to have substantial overlap between the working volume of the picking arm and the volume potentially occupied by target fruit. Therefore, it is likely that the linear axis for the picking arm will be at a similar height to a proportion of the target fruit, and it will be desirable for the picking arm to operate to either side of the linear axis while picking these fruits. In the context of a narrow row of fruit, this restricts the maximum width of the linear axis, and compromises crossbody stiffness and precision of movement of the picking arm, with negative consequences on picking performance and hand-off precision.

[0180] Second, the picking arm must now pass around its linear axis in order to access the packaging arm after picking a berry. This restricts the number of paths available, tending to prevent the use of the fastest path, and so slows the rate of picking.

[0181] An alternative on the above design, shown in figure 17, arranges all linear axes in pairs either side of the fruit trays, with a bridge structure between the picking arm linear axes supporting the picking arm above the central trays. This performs well against all of the metrics listed above, at the cost of increased dynamic mass and complexity of the linking bridge component.

[0182] Figure 17 shows a cross-sectional view of a machine configured to allow end-access to trays 37 along the chassis 32. Linear axes for the arms and packaging systems ((35and 151) are positioned along the side of the machine. A bridge (171) links the axes, supporting the subsequent DoFs of the picking arm (33) and providing space for the operation of the packaging system (36). Relatively unrestricted access is possible either side of the bridge component (out of the plane of the figure), and the close coupling of the picking and packaging arm linear stages facilitates a high degree of relative precision and reliable transfer of picked fruit.

[0183] Extending the shelf life of picked fruit

[0184] Various defects can compromise the shelflife of picked fruit, e.g. rots and moulds such as botrytis and mildew, bruising, insect damage, over-ripeness, etc. On-robot visual inspection of picked fruit is used to detect visual evidence of defects so that defective fruit can be disposed of in a waste container instead of being stored alongside healthy fruit - reducing the likelihood that disease pathogens will be transferred from fruit to fruit. However, some defects may not be visually apparent, e.g. the early stages of diseases like mildew and botrytis, and may therefore be difficult to detect using visual inspection alone. This motivates the idea of using an on-robot odour sensor (or ‘bionic nose’) to detect evidence of defects in picked fruit (and other quality characteristics such as ripeness). Odour sensors work by detecting volatile organic compounds that are characteristic of particular defects (and other quality characteristics) and are being implemented in a variety of small form-factor devices using various technologies.

[0185] A potential limitation of using odour detectors for on-robot defect detection is that they may be slow to detect VOCs in low concentration, and this could compromise the speed performance, potentially compromising the picking speed of the robot. One potential mitigation is to force air over the surface of the picked fruit and past or through the odour detector, increasing the rate at which relevant VOC molecules are encountered. This motivates the idea of positioning the picked fruit in an enclosed chamber (e.g. using the robotic picking arm or other positioning device) and using a suitable fan or pump to force air from a suitable intake over the surface of picked fruit and past the odour sensor. Sufficiently clean intake air may be obtained by positioning an air inlet farm from (e.g. higher than) the crop or by passing air through a suitable filter, such as a charcoal filter. A related innovation is to force air over the surface of the picked fruit and then contain it, e.g. by closing inlet and exhaust valves in an enclosed chamber containing the odour detector. The advantage of this approach is that the picking arm or other robotic positioning device can start to reposition the picked fruit whilst the odour detector is still analysing its VOC signature, allowing more time for analysis. A consequent increase in processing speed can be obtained e.g. by optimistically moving the picked fruit towards the storage container (on the assumption that fruit is nondefective with high probability) but redirecting movement towards the waste container in the event that the fruit is determined to be defective.

[0186] It may be beneficial to use an odour sensor in combination with visual inspection, since the orthogonal defect detection techniques are complementary. Visual inspection is typically achieved using an imaging chamber designed to exclude uncontrolled daylight and it is ideal if the same chamber can be used for odour-based defect detection.

[0187] Another way of extending the shelf life of picked fruit is to destroy disease pathogens before they can cause damage to the fruit or spread to other fruits. One way of doing this is to irradiate the fruit with ultraviolet (UV) light. UV irradiation of the developing crop is an established idea but existing approaches have the significant disadvantage that usually only one side of the fruit can be irradiated because, before picking, part of the surface of the fruit is generally by the fruit itself or other fruit or foliage. Alternatively, fruit may be irradiated after picking but additional handling is required to ensure all parts of the fruit are exposed to UV - and this is highly undesirable for fruits such as delicate berry fruits that are easily bruised by contact. Since harvesting robots much anyway handle fruit during picking, this motivates the alternative idea of irradiating the fruit whilst it is held by the robot positioning system but before it is deposited into the storage container, allowing irradiation of the entire surface of the fruit (or, at least, a significantly larger proportion of it) without additional handling. This approach can also be applied to other means of extending shelf life such as exposing picked fruit to ozone.

[0188] To protect workers from UV radiation it is desirable to irradiate the fruit whilst the berry is in an enclosed chamber.

[0189] Since moving fruit from place to place using a robot positioning system may be time consuming, it is desirable that the functions described above (i.e. any combination of visual inspection, odour-based defect detection, UV irradiation, etc.) can be be carried out in a single enclosure, potentially simultaneously or in quick succession.

[0190] Tray access

[0191] To enhance the shelf life of picked fruit, it is typically an operational requirement to minimise the time between fruit being picked and moved to a cold storage location. Typically, picked fruits are placed into trays (which here should be taken to mean any container) and trays are offloaded as soon as time since picking for any of the fruit they contain exceeds some threshold. It may be necessary for either a human operator or a secondary machine to interact with the picking robot to unload picked fruit while it is in the crop row, replacing full trays with empty ones. Most fruit picking robots (including those produced by Dogtooth to date) require access to the sides of the machine by the operator to replace the trays. However, the crop rows and infrastructure generally restrict operator access to the sides of the machine while it is in the row. Furthermore, a machine designed to pick fruit on both sides will create a greater safety hazard when an operator is required to access it from the side (where it is impractical to provide physical guards as since the picking arm must be able to reach the crop.

[0192] An obvious, and generally adopted solution is for the machine to drive out of the aisle to facilitate unloading of picked fruit. However, the time taken to drive out of the aisle and back in may compromise productivity, particularly in long aisles accessible from only one end. Additionally, the autonomous exit of machines from rows into areas commonly accessed by people creates in itself a safety hazard that must be mitigated, driving increased system complexity and cost.

[0193] These challenges give rise to a useful feature, which is to configure the robot so as to allow trays to be loaded or unloaded from one of the end or both ends of the machine while it is still in the aisle, possibly while the robot is still picking (see Figure 18).

[0194] Figure 18 shows a machine picking along a crop row in direction (181), typically bordered by the plants and supporting infrastructure (182) in which the machine is usefully configured to allow access to one or both ends of the machine (183) to allow easy removal and replacement of fruit trays (184). Without loss of generality, the robot could be configured to travel in either direction along the crop row, and to allow trays to be accessed from one or both ends.

[0195] It is generally beneficial for a robot to pick into more than one tray of fruit. Benefits include: Extended intervals between required fruit offloads when picking fast; and the ability to more optimally fill the first tray by providing additional placement options for berries as the first tray becomes more full. If the robot has only one tray, then unloading and reloading from either end of the robot is straightforward. If the robot has two or more trays, it may be beneficial to unload one tray before another (because it is full, or because a time limit has been reached since the first berry was placed in the tray).

[0196] One useful feature is for the picking robot to fill the trays in an order that means that the next tray to be unloaded will always be at the most accessible end of the robot.

[0197] To enable this behaviour, several useful machine designs might facilitate the routine positioning of the “next” tray of fruit at the most accessible end of the robot.

[0198] • A unloading robot which temporarily removes all trays from the picking robot, removing the full tray, and reloading existing and new trays to locate the “oldest” tray at the accessible end of the robot

[0199] • An embodiment allowing trays to pass above and below one another inside the robot envelope, rearranging their order to positing the “oldest” tray at the accessible end at the point of unloading.

[0200] • A picking robot which allows loading of new trays at one end, and unloading at the opposite end.

[0201] • A picking robot with a “magazine” of empty trays, allowing replacement of trays from within the robot envelope when full trays are removed.

[0202] Operator safety & guarding

[0203] When fruit picking robots are operated in crop rows, typically the crop itself and any supporting infrastructure on either side of the robot restricts human access to the volume of space through which potentially hazardous robot picking arms are moving. However, users may need to access the end(s) of the robot and the system design must protect them from hazardous moving machinery.

[0204] One way to prevent users from coming into contact with moving picking arms is to use dedicated rigid safety guards mounted in a static position with respect to the robot’s chassis. One disadvantage is that the presence of the guard restricts the working volume of the arms. Alternatively, positioning the guards far outboard to maximise working volume leads to a long overall machine length with negative impacts on manoeuvrability and storage requirements. One method of addressing this problem is to provide extending guards. When the robot is picking, the guards are extended, increasing the arm working volume. When not picking, the guards are retracted to reduce machine length and improve manoeuvrability and storage (see Figure 19).

[0205] Figure 19 shows diagrams illustrating the robot fruit picking system with retractable guards in two configurations: A robot chassis (32) is fitted with retractable guards (191) to protect people from the moving picking arms (33) and survey modules (38), which may be moving on a linear axis (35). When picking (A), guards (191) are extended to maximise machine length and arm working volume; When not picking (B), guards are retracted to minimise machine length, improving manoeuvrability and reducing storage requirements.

[0206] A useful observation is that the primary hazard to a users at one end of the picking robot is collision between the user and the higher-degrees of freedom of the picking arm (since these are generally faster moving, move along more complex paths, and are closer to the end-effector which poses a risk to eyes). In a machine layout comprising survey masts outboard of picking arms, an approach is to configure the survey masts to provide some guarding function between picking arms and the user, eliminating the need for at least some portion of the static guards. These survey masts might be designed to minimise the impact of a collision between mast and user, for example by adding cushioning elements as shown in figure 20.

[0207] It might also be beneficial to use the structure of the mast to provide a guarding function outside the survey cameras, to protect the user and also to provide functional benefits such as better protection for the survey cameras from impact, and a convenient location for mounting lighting elements. An example is shown in figure 20 part C.

[0208] Figure 20 shows diagrams illustrating different configurations of the robot fruit picking system. A typical machine (A) might protect a user from impact with a picking arm (33), moving along a linear axis (35) by providing a guard (201) fixed relative to the chassis (32). An alternative (B) is to use the survey mast to provide a guarding function, to which cushioning elements (202) could be added. Additionally (C) the support for the survey cameras (203) could be formed around the cameras (38), providing a guarding function while also better protecting the cameras from damage, and providing an advantageous location for the mounting of lighting to illuminate survey views.

[0209] Figure 21 shows another example of the robotic fruit picking system with an extended chassis structure 32 and the linear rail 35 being mounted or attached to the extended chassis 32. The two picking arms 33 mounted on the linear rail 35 move along the linear rail 35 and pick fruit.

[0210] Figure 22 shows a perspective view of the robotic fruit picking system shown in the diagram of Figure 21. Multiple trays 37 for storing picked fruits are also shown. Two inspection chambers 221 are also located at each end of the robot to be within easy reach of the two picking arms 33. In this configuration, the picking arm is used for inspection and packaging. Alternatively, a separate packaging arm may be used, as described above.

[0211] Figure 23 shows a perspective view of the robotic fruit picking system shown in the diagram of Figure 21, further including a calibration target 41.

[0212] Figure 24 shows a perspective view of the robotic fruit picking system shown in the diagram of Figure 21, further including an easily removable calibration target 41. Additional calibration targets 241 are also located at each end of the robot. The calibration targets 241 are fixed 2D computer vision calibration target that enables the calibration of the geometric relationship between the survey camera module 38 and the picking head camera(s) 252. One advantage in having a rotating survey camera array is that calibration targets can be permanently equipped onto the ends of the robot (so as to be visible for some orientations of the survey camera array) without impinging on the cameras field of view. By this means we obviate the need for demountable calibration targets.

[0213] Figure 25 shows a perspective view of one picking arm 33 of the robotic fruit picking system of Figure 21. The picking head positions a picking head or end effector 251 and picking head camera(s) 252, each mounted on the picking arm. In this configuration, the survey camera module 38 is built in to or attached to the picking arm 33, such as the "upper arm" portion of the picking arm. This approach is more compact than accommodating the survey camera array using for example a separate survey pole and allows to use a motor in one of the picking arm's existing revolute joints to rotate the survey camera array to face either the left or right side of the robot. Lighting elements 253 are also positioned on the picking arm 33 at fixed position relative to the built in survey camera module 38.

[0214] The picking head may include one or more picking head cameras, such as monocular camera, stereo camera, active or passive depth sensing cameras.

[0215] The survey camera module is configured to generate data for one or more of the following processes: (i) surveying fruits or fruit bunches; (ii) determining target fruit locations; (iii) determining pick vectors; in each case being controlled to be positioned or oriented, at least in part, independently of a position or orientation of one or more of: the or each picking head camera 252, the picking arm 33 or the picking head 251.

[0216] In some instances, the survey camera’s operation may be determined by an output from the picking head camera. As an example, the picking head camera’s view from several positions of a fruit may be blocked by some unripe berries and the picking head camera may therefore send a message to the survey camera to look at the fruit and build a model of the positions of the unripe fruits and the target fruit so a pick vector of the blocked berry can be determined or plotted.

[0217] A future position of the picking head camera, picking arm or picking head may also be a function of the earlier survey camera module position.

[0218] Appendix A: Quality Control Inspection Chamber

[0219] The robot fruit picking system, as described above, may also include an inspection subsystem including an inspection chamber to monitor the quality of fruit that has been picked and to grade that fruit according to size and / or quality. As an example, the packaging arm or any other transport mechanism may be used to move the picked fruit directly inside the inspection chamber.

[0220] The inspection subsystem includes an inspection chamber, the inspection chamber being located on-board of the robotic fruit picking system or externally to the robotic fruit picking system. The inspection chamber may also include one or more odour sensors, one or more UV light sources and one or more ozone sources.

[0221] The inspection chamber includes one or more light sources that direct or diffuse light onto a picked fruit. In particular, highly diffuse light source(s) are used in order to optimise defect detection and / or silhouette extraction, such as for mass assessment.

[0222] Figure 26 shows a diagram illustrating the inspection chamber. An object, such as a fruit, is lowered and positioned inside the inspection chamber via an aperture. The object is then illuminated by one or more light sources. One or more of the side walls of the inspection chamber may include adjustable reflective diffuser or barrier that help control how the light hits the picked fruit.

[0223] Figure 27 shows a diagram illustrating an adjustable reflective diffuser or barrier configured to control how the light illuminates the object, the barrier includes two baffles, the first baffle is configured to be moveable and the second baffle partially overall the first baffle. A movement controller is operably coupled to the first baffle to enable the first baffle to be moved. The movement controller may comprise an electromagnet and a controlling apparatus

[0224] One or more light sources strategically positioned inside the inspection chamber are then configured to direct light at the first and second baffles to provide illumination of the object. Moving the first baffle means that its position and / or angle may be adjusted. The first baffle initially intercepts the light from the light source and by changing its position and / or angle, the first baffle controls the amount and direction of light reflected.

[0225] Additionally, the second baffle may also be adjusted independently from the first baffle in order to further refine or diffuse the light that has been filtered by the first baffle, thereby allowing for a further control of the reflected light characteristics.

[0226] By automatically adjusting the position of the baffles, the quality control subsystem can ensure the object or picked fruit is evenly lit for quality inspection. Specific features or defect on the picked fruit may also be highlighted that would otherwise have been missed under less controlled lighting conditions.

[0227] By dynamically controlling the lighting conditions, the inspection subsystem is also configured to ensure that each picked fruit is inspected under the same conditions, ensuring that the picked fruit can meet the same standards. The inspection subsystem can also handle different shapes, sizes and other characteristics of the fruit to be inspected.

[0228] Optional features may include any one or more of the following:

[0229] • The first baffle comprises a set of first diffusion features and the second baffle comprises a set of second diffusion features

[0230] • The first set of diffusion features may black and transparent stripes and the second set of diffusion features are black and white stripes.

[0231] • The light is transmitted through at least one baffle.

[0232] • The light is reflected from at least one baffle.

[0233] • The movement of the first baffle relative to the second baffle to a first position causes the object to be illuminated with a diffuse light.

[0234] • The movement of the first baffle relative to the second baffle to a second position causes the object to be illuminated with a direct light.

[0235] • One or more of the baffles comprise a transmissive material.

[0236] • One or more of the baffles comprise a reflective material. The inspection chamber includes one or more odour sensors, one or more UV light sources and one or more ozone sources.

[0237] Figure 28 shows another example of an inspection chamber for scattering or directing light onto an object in an automated inspection subsystem. The fruit being inspected is shown as a dotted oval on the left. The fruit can be illuminated either via a point light source shining through a hold in the diffuser or a diffuse light source (which may an edge-illuminated diffuser). The point light source casts strong shadows, revealing surface shape and therefore defects that affect surface shape; the diffuse light source reveals surface albedo, revealing defects that affect surface albedo. The diagram also includes an example of a light-guide panel on the bottom left.

[0238] Optional features may include any one or more of the following:

[0239] • The inspection subsystem includes at least one switchable diffuser having one or more diffusion features.

[0240] • A first light source is configured to direct a first light beam through a first switchable diffuser.

[0241] • A second light source is configured to direct a second light beam in a direction substantially perpendicular to the first light beam through the first switchable diffuser.

[0242] • A light source controller is operably coupled to the first and second light sources to activate at least one light source.

[0243] • The first light beam is diffused by the one or more diffusion features and the second light beam is not diffused by the one or more diffusion features.

[0244] • The switchable diffuser comprises a light transmitting sheet having at least one face.

[0245] • The diffusion features comprise one or more surface decorations printed on at least one face of the light transmitting sheet.

[0246] • The diffusion features comprise one or more surface crenulations formed on at least one face of the light transmitting sheet

[0247] • The diffusion features comprise one or more voids within the light transmitting sheet

[0248] • The one or more surface decorations printed on at least one face of the light transmitting sheet comprise dots with a dimension of less than 5 mm. • The diffusion features are evenly distributed across the light transmitting sheet.

[0249] • The diffusion features are distributed across the light transmitting sheet at different effective lines of latitude to produce a spherical illumination effect.

[0250] • The diffusion features are randomly distributed across the light transmitting sheet.

[0251] Appendix B: Features Summary

[0252] This section summarises the most important high-level features; an implementation of the invention may include one or more of these high-level features, or any combination of any of these. Note that each feature is therefore potentially a stand-alone invention and may be combined with any one or more other feature or features; the actual invention defined in this particular specification is however defined by the appended claims.

[0253] There is inevitably a degree of overlap between these features. This approach to organising the features is therefore not meant to be a rigid demarcation, but merely a general high-level guide.

[0254] Feature A - Survey camera module is positioned or orientated, at least in part, independently from the picking head camera, the picking head or the picking arm. A robotic fruit picking system, the system comprising: at least one picking arm; at least one picking head, mounted on the or each picking arm to (i) either cut a fruit or stalk or stem for a specific fruit or bunch of fruits or (ii) pluck that fruit or bunch; at least one picking head camera mounted on the or each picking head or the picking arm; and a survey subsystem that includes a survey camera module configured to obtain and to analyse image scenes containing fruit-producing crops, in which the survey camera module is configured to generate data for one or more of the following processes: (i) surveying fruits or fruit bunches; (ii) determining target fruit locations of some or all pickable fruits; (iii) determining approach vectors of some or all pickable fruits; in each case being controlled to be positioned or oriented, at least in part, independently of a position or orientation of one or more of: the or each picking head camera, the picking arm or the picking head.

[0255] Alternatively, the picking head, designed to pick a target fruit, may be positioned using an end effector positioning device other than a traditional picking arm. As an example, linear actuators may be used to move a picking head in multiple directions. Aerial drones or other mobile robots may also be used to move the picking head.

[0256] Picking heads are designed to interact directly with the fruit or stalk or stem of a specific fruit or bunch of fruits. Many types of picking heads or other end effectors may be used. As an example, automated actuators, vacuum or suction type devices, shaking or vibrating devices may be used. Picking heads may also be used to perform other functions, such as husbandry tasks, in which case the picking head may not only cut or pluck a fruit but also other parts of a plant, such as runners or foliage.

[0257] Feature B - Parallelizing survey and picking operations

[0258] A robotic fruit picking system, the system comprising: at least one picking arm; at least one picking head, mounted on the or each picking arm to (i) either cut a fruit or stalk or stem for a specific fruit or bunch of fruits or (ii) pluck that fruit or bunch; at least one picking head camera mounted on the or each picking head or the picking arm; and a survey subsystem that includes a survey camera module configured to obtain and to analyse image scenes containing fruit-producing crops, in which the survey camera module is configured to generate data for one or more of the following processes: (i) surveying fruits or fruit bunches; (ii) determining target fruit locations of some or all pickable fruits; (iii) determining approach vectors of some or all pickable fruits; in each case being controlled to be positioned or oriented, at least in part, in parallel or at substantially the same time as the positioning or orientation of one or more of: the or each picking head camera, the picking arm or the picking head.

[0259] Feature C - Future position of the picking head camera, picking arm or picking head is a function of the earlier survey camera module position. A robotic fruit picking system, the system comprising: at least one picking arm; at least one picking head, mounted on the or each picking arm to (i) either cut a fruit or stalk or stem for a specific fruit or bunch of fruits or (ii) pluck that fruit or bunch; at least one picking head camera mounted on the or each picking head or the picking arm; and a survey subsystem that includes a survey camera configured to obtain and to analyse image scenes containing fruit-producing crops, in which the survey camera module is configured to generate data for one or more of the following processes: (i) surveying fruits or fruit bunches; (ii) determining target fruit locations of some or all pickable fruits; (iii) determining approach vectors of some or all pickable fruits; and in which the position of the picking head camera, the picking arm or the picking head is a function of a prior survey camera module position.

[0260] Feature D - Linear rail subsystem that picking arm moves along, optimising solutions to the inverse kinematics problem

[0261] A robotic fruit picking system, the system comprising: at least one picking arm; at least one picking head, mounted on each picking arm to either cut a fruit or stalk or stem for a specific fruit or bunch of fruits or pluck that fruit or bunch; and a linear rail subsystem, in which the or each picking arm is configured to move along the linear rail subsystem.

[0262] Feature E - Linear rail subsystem that two picking arms move along, minimising time a picking arm is quiescent

[0263] A robotic fruit picking system, the system comprising: at least two picking arms; at least two picking heads, in which each picking head is mounted on each picking arm to either cut a fruit or stalk or stem for a specific fruit or bunch of fruits or pluck that fruit or bunch; and a linear rail subsystem, in which the picking arms are configured to move along the linear rail subsystem; and in which the system is configured to minimise time a picking arm is quiescent.

[0264] Feature F - Parallelizing Survey, Picking, & Packaging

[0265] A robotic fruit picking system for localising, picking and packaging a fruit or bunch of fruit, in which the system comprises a survey subsystem, a picking subsystem and a packaging subsystem, and in which each subsystem is independent of the other subsystems and / or performs its functions in parallel to the other subsystems.

[0266] Each subsystem is configured to be easily moveable / detachable from the robotic fruit picking system.

[0267] Feature G - Picking arm that hands off picked berry to a different packaging arm, the picking arm being an expensive, fast arm for positioning the picking head, and the packaging arm being a simple low-cost arm

[0268] A robotic fruit picking system for picking and packaging a fruit or bunch of fruit, the system comprising: at least one picking arm; the picking arm being a fast robot arm, such as 6 degree- of-freedom robot arm; at least one picking head, mounted on each picking arm to either cut a fruit or stalk or stem for a specific fruit or bunch of fruits or pluck that fruit or bunch, and to transfer the picked fruit or bunch to a packaging arm; and the packaging arm being configured to transfer the picked fruit or bunch to an inspection chamber or storage container or punnet, in which the packaging arm is a simple low-cost arm, such as only a 1 or 2 degree-of-freedom robot arm.

[0269] Optional features include any one or more of the following, in any combination:

[0270] Approach vector

[0271] • An approach vector refers to the direction or path that the picking arm(s) follow when approaching a pickable fruit without the picking arm(s) and / or picking head(s) colliding, such as with leaves of other berries or other stalks or other fruits as it approaches a pick position. • A pick position is defined as a point at which a stalk will be cut or the fruit will be contacted by the picking head.

[0272] • Approach vector is modelled using a stochastic process that determines a probabilistic direction or path that the picking arm(s) follow when approaching a pickable fruit.

[0273] • Probabilistic direction or path is a function of an intended motion of the picking arm(s).

[0274] • Stochastic process takes into account one or more of the following: noisy interpretation of image data, inaccurate camera calibration, inaccurate forward kinematic model of the picking arm, wind or other obstacles in between a time when the imaging subsystem determines a picking direction or path and the time when the picking head reaches a target fruit.

[0275] • System is configured to determine a ‘pickability score’ associated with a probability of picking success and when the probability of picking success is above a certain threshold, then the robotic fruit picking system attempts to pick a target.

[0276] Survey subsystem

[0277] • Survey subsystem is configured to determine the location of some or all target fruits in a coordinate frame, such as a chassis coordinate frame.

[0278] • Survey subsystem uses an array of survey viewpoints to infer or estimate a 3D location of some or all target fruits.

[0279] • location uncertainty of a target fruit is determined using a pairwise fruit-to-fruit distance model that considers a relative distance for each pair of fruits from each survey viewpoint.

[0280] • Imaging sensor is moved to multiple viewpoints until one is found that provides a predefined pickability score.

[0281] • Survey subsystem includes one or more cameras, such as monocular camera, stereo camera, active or passive depth sensing cameras.

[0282] • Survey subsystem uses computer vision image processing to detect target fruits in the 3D camera coordinate frames.

[0283] • Each detected target fruit is associated with a pickability score, and each survey viewpoints is used to determine or update the pickability score. • Any detections that do not appear in more than a predefined threshold number of survey viewpoints is rejected.

[0284] • A detected fruit’s suitability for picking is determined by fusing information from all available viewpoints.

[0285] • Survey subsystem is configured to compute one or more fruit related parameters, such as geometry or colour, over all available viewpoints.

[0286] • Picking head is positioned to be at least partially visible in the survey camera module field of view.

[0287] • System includes light sources positioned at a fixed position relative to the survey camera module that illuminates fruits to reduce scene contrast in bright daylight, and also enable picking at dawn, dusk and at night.

[0288] • Light sources have a strobing frequency that is designed to not be perceptible to humans, such as less than 40Hz.

[0289] • Survey subsystem includes a synchronising mechanism that automatically synchronise the strobing frequency and the survey camera module shutters.

[0290] • System includes an inertial measurement unit to measure an acceleration of the survey camera module.

[0291] • System is configured to apply a correction to the assumed position of the camera module at the time of capturing an image.

[0292] • Picking arms are configured to output a signal to the survey camera module, that enables for any image data captured by the survey camera module to be timestamped with respect to the picking arm position.

[0293] • Survey camera module is moveable past the picking arm(s).

[0294] • Survey camera module is shaped as a large hoop that can pass over the picking arm(s).

[0295] • Survey subsystem includes one or more spectrometers.

[0296] • The spectrometers are positioned at position(s) to be illuminated by light of a similar colour to that of some or all target fruits.

[0297] • The spectrometers are positioned within known position(s) in a shared coordinate frame.

[0298] • System includes a moveable arm configured to position the one or more spectrometers. • The survey camera module shares one or more degrees of freedom with the picking head camera, picking arm or picking head.

[0299] • The survey camera module is attached or built-in onto the picking arm.

[0300] • Picking arm is driven by one or more motors, and at least one motor of the picking arm is used to position or orient the survey camera module.

[0301] • The survey camera module is positioned or orientated using a robot arm or flying drone.

[0302] • The survey camera module is positioned or orientated at a fixed position with respect to a feature of the robot, such as the chassis of the robot.

[0303] • The survey camera module is mounted to a moveable feature of the robot, such that the camera module is moveable relative to a fixed portion of the robot.

[0304] • The survey camera module is mounted to a rotating feature of the robot, such as a pole, so that the camera module is rotatable relative to a fixed portion of the robot.

[0305] • The survey camera module includes an array of cameras mounted at the front of the robot and / or at the back of the robot.

[0306] • The survey camera module moves along a linear rail.

[0307] • The survey camera module is adjustable based on the geometry or condition of the crop.

[0308] • Fruits are grown on tabletops and height of the survey camera module is adjustable to take into account a height of the tabletops.

[0309] • The survey camera module is mounted to one or more survey arms, such as a single or multi-degree of freedom robot arm.

[0310] Calibration target

[0311] • Survey camera module is first calibrated using a calibration target that is mounted at a known position onto the robot, such as using fixings on the robot chassis.

[0312] • Calibration target is planar target including fiducial markers.

[0313] • Calibration target is positioned such that it is located within the survey camera module field of view without occluding the imaged scene.

[0314] • Calibration target is permanently fixed onto the robot fruit picking system, while the robot fruit picking system is in use. System is configured to check that the reconstructed positions of fiducial markers on the calibration target are consistent with their reconstructed positions immediately after calibration.

[0315] Picking subsystem

[0316] • Picking head is user replaceable.

[0317] • Picking head includes a light source that illuminates fruits to reduce scene contrast in bright daylight, and also enable picking at dawn, dusk and at night.

[0318] • Picking subsystem is positioned above the packaging subsystem.

[0319] • Picking subsystem is positioned above storage containers or trays.

[0320] • The position of the picking head camera, picking arm or picking head is a function of a prior survey camera module position.

[0321] • Picking head includes one or more built-in cameras.

[0322] • The picking head is any device or end-effector configured to permanently sever fruit or bunches of fruit.

[0323] • The picking head is any device or end-effector configured to permanently sever one part of a plant from another part of the plant, such as runners or other foliage.

[0324] Packaging subsystem

[0325] • Packaging subsystem includes a simple robot arm, such as an arm with only two or three degrees of freedom.

[0326] • Packaging subsystem includes a buffer subsystem, such as conveyer belt or carousel or magazine.

[0327] • Packaging subsystem is positioned out of reach of the picking arm.

[0328] • Packaging arm or buffer subsystem is configured to deposit the picked fruit or bunch onto an inspection chamber or storage container(s) or tray(s).

[0329] • Inspection subsystem is configured to image a picked fruit from multiple relative viewpoints.

[0330] • Inspection subsystem is configured to monitor the size and quality of the picked fruit or bunch and to grade a picked fruit or bunch and determine its suitability for retail. Robot fruit picking system is configured to allow the container(s) or tray(s) to be loaded or unloaded from either end of the robot.

[0331] Linear rail subsystem

[0332] • Picking arms can travel backwards and forwards along a crop row independently of the robot.

[0333] • System includes two picking arms, one arm being a leading arm and the second being a trailing arm.

[0334] • Leading arm and trailing arm are configured to stay at up to an approximately minimum distance apart.

[0335] • Neither picking arm is allowed to move in a space required by the other picking arm.

[0336] • A working space for each arm to move while picking the next target berry is determined by a 3D bounding box geometry.

[0337] • System is configured to provide a shared working space for both picking arms as the picking arms move along a linear rail or path.

[0338] • Leading arm is configured to release a portion of the working space as it moves towards one end of the linear rail or path.

[0339] • Trailing arm can reserve working space as it progresses along the linear rail or path.

[0340] • If the trailing arm requires more working space that has been released by the leading arm, the trailing arm’s operation is delayed until sufficient working space is available.

[0341] • Robot fruit picking system includes a control subsystem, such as two control software threads under a shared lock, for managing the reservation and release of the working space.

[0342] • Leading and trailing arms are configured to pick in multiple back and forth passes along the robot.

[0343] Bionic nose

[0344] • Robot fruit picking includes at least one odour sensor configured to detect volatile organic compounds (VOCs) and / or other odours. • Robot fruit picking includes an air moving device, such as fan or pump, that draws air from an air intake and directs the air over a picked fruit surface and past the odour sensor.

[0345] • Robot fruit picking includes a test chamber with an opening, and the air moving device directs the air over a picked fruit surface, into the chamber, and past the odour sensor located inside the chamber.

[0346] • Valve is associated with the opening of the test chamber, the valve being operable to open and close the opening.

[0347] • Odour sensor is configured to analyse the detected VOCs and determine whether a picked fruit should be discarded.

[0348] • Test chamber is also configured to image a picked fruit from multiple viewpoints.

[0349] UV light source and / or ozone source

[0350] • Robot fruit picking system includes a UV light source configured to emit UV light and expose a picked fruit from multiple relative viewpoints.

[0351] • Control subsystem is configured to move the picking or packaging arm such that the angle and direction of the UV light relative to the picked fruit is adjusted.

[0352] • Control subsystem is also configured to control the exposure time of the picked fruit.

[0353] • UV light source and / or ozone source are arranged to enable the irradiation of substantially the entire surface of the picked fruit.

[0354] • Robot fruit picking system includes an ozone source configured to emit ozone and expose a picked fruit from multiple relative viewpoints.

[0355] Inspection chamber

[0356] • Inspection chamber is configured to house a UV light source and / or an ozone source and the picked fruit.

[0357] • Inspection chamber is made of materials that are opaque to UV radiation and resistance to ozone.

[0358] • The picked fruit is treated for sterilization or inspection purposes. • Inspection chamber houses an odour sensor, a UV light source, an ozone source, imaging sensors and one or more light sources.

[0359] • Inspection chamber includes one or more light sources that direct or diffuse light onto a picked fruit.

[0360] • Inspection chamber uses highly diffuse light source(s) in order to optimise defect detection and / or silhouette extraction, such as for mass assessment.

[0361] Method or applications

[0362] In this section, we summarise features which relate to methods or applications of the system including any of the features or optional features listed above.

[0363] A method of optimizing fruit yield mapping across a fruit farm or multiple fruit farms, by analysing images of fruits to be picked in an imaged scene using the robotic fruit picking system defined above.

[0364] A method of determining fruits suitability for picking by using the robotic fruit picking system defined above, and to determine the picked fruits suitability for retail.

[0365] A method of minimising time a picking arm is quiescent by using the robotic fruit picking system defined above.

[0366] A method of maximizing fruit shelf life by using the robotic fruit picking system defined above.

[0367] A method of picking fruit by using the robotic fruit picking system defined above.

[0368] A final aspect is the fruit when picked using the robotic fruit picking system defined above.

[0369] The fruit can be a crop that grow on plants, such as strawberry or tomato, bushes such as raspberry, blueberry or grape, and trees such as apple, pear or logan berry. The fruit also include vegetable, and other kinds of produce or waste product that are picked from plants, such as nut, seed, vegetable, excess foliage or unwanted branch. Note

[0370] It is to be understood that the above-referenced arrangements are only illustrative of the application for the principles of the present invention. Numerous modifications and alternative arrangements can be devised without departing from the spirit and scope of the present invention. While the present invention has been shown in the drawings and fully described above with particularity and detail in connection with what is presently deemed to be the most practical and preferred example(s) of the invention, it will be apparent to those of ordinary skill in the art that numerous modifications can be made without departing from the principles and concepts of the invention as set forth herein.

Claims

CLAIMS1. A robotic fruit picking system, the system comprising: at least one picking arm; at least one picking head, mounted on the or each picking arm to (i) either cut a fruit or stalk or stem for a specific fruit or bunch of fruits or (ii) pluck that fruit or bunch; at least one picking head camera mounted on the or each picking head or the picking arm; and a survey subsystem that includes a survey camera module configured to obtain and to analyse image scenes containing fruit-producing crops, in which the survey camera module is configured to generate data for one or more of the following processes: (i) surveying fruits or fruit bunches; (ii) determining target fruit locations of some or all pickable fruits; (iii) determining approach vectors of some or all pickable fruits in the imaged scene; in each case being controlled to be positioned or oriented, at least in part, independently of a position or orientation of one or more of: the or each picking head camera, the picking arm or the picking head.

2. The system of claim 1, in which the survey camera module is in each case being controlled to be positioned or oriented, at least in part, in parallel or at substantially the same time as the positioning or orientation of one or more of: the or each picking head camera, the picking arm or the picking head.

3. The system of claim 1 or 2, in which the position of the picking head camera, the picking arm or the picking head is a function of a prior survey camera module position.

4. The system of any preceding claim, in which an approach vector refers to the direction or path that a picking arm follows when approaching a pickable fruit without the picking arm and / or its picking head colliding, such as with leaves of other berries or other stalks or other fruits as it approaches a pick position.

5. The system of claim 4, in which a pick position is defined as a point at which a stalk will be cut or the fruit will be contacted by the picking head.

6. The system of any preceding claim, in which an approach vector is modelled using a stochastic process that determines a probabilistic direction or path that a picking arm follows when approaching a pickable fruit.

7. The system of claim 6, in which the probabilistic direction or path is a function of an intended motion of the or each picking arm.

8. The system of any of claims 6-7, in which the stochastic process takes into account one or more of the following: noisy interpretation of image data, inaccurate camera calibration, inaccurate forward kinematic model of the or each picking arm, wind or other obstacles in between a time when the imaging subsystem determines a picking direction or path and the time when the or each picking head reaches a target fruit.

9. The system of any preceding claim, in which the survey subsystem is configured to determine a ‘pickability score’ associated with a probability of picking success and when the probability of picking success is above a certain threshold, then the robotic fruit picking system attempts to pick a target.

10. The system of any preceding claim, in which the survey subsystem is configured to determine the location of some or all target fruits in a coordinate frame, such as a chassis coordinate frame.

11. The system of any preceding claim, in which the survey subsystem is configured to use an array of survey viewpoints to infer or estimate a 3D location of some or all target fruits.

12. The system of any preceding claim, in which positional uncertainty of a target fruit is determined using a pairwise fruit-to-fruit distance model that considers a relative distance for each pair of fruits from each survey viewpoint.

13. The system of any preceding claim, in which the survey camera module is configured to move to multiple viewpoints until one is found that provides a predefined pickability score.

14. The system of any preceding claim, in which the survey subsystem includes one or more cameras, such as monocular camera, stereo camera, active or passive depth sensing cameras.

15. The system of any preceding claim, in which the survey subsystem is configured to associate each detected target fruit with a pickability score, and to determine or update the pickability score using each survey viewpoints.

16. The system of any preceding claim, in which the survey subsystem is configured to reject any detections that do not appear in more than a predefined threshold number of survey viewpoints.

17. The system of any preceding claim, in which the survey subsystem is configured to determine a detected fruit’s suitability for picking by fusing information from all available viewpoints.

18. The system of any preceding claim, in which the survey subsystem is configured to compute one or more fruit related parameters, such as geometry or colour, over all available viewpoints.

19. The system of any preceding claim, in which the or each picking head is positioned to be at least partially visible in the survey camera module field of view.

20. The system of any preceding claim, in which the system includes light sources positioned at a fixed position relative to the survey camera module that illuminates fruits to reduce scene contrast in bright daylight, and also enable picking at dawn, dusk and at night.

21. The system of claim 20, in which the light sources have a strobing frequency that is designed to not be perceptible to humans, such as less than 40Hz.

22. The system of any of claims 20-21, in which the survey subsystem includes a synchronising mechanism that automatically synchronise the strobing frequency and the survey camera module shutters.

23. The system of any preceding claim, in which the system includes an inertial measurement unit to measure an acceleration of the survey camera module.

24. The system of any preceding claim, in which the system is configured to apply a correction to the assumed position of the survey camera module at the time of capturing an image.

25. The system of any preceding claim, in which the picking arms are configured to output a signal to the survey camera module, that enables for any image data captured by the survey camera module to be timestamped with respect to the picking arm position or orientation or with respect to the picking head position or orientation.

26. The system of any preceding claim, in which the survey camera module is moveable past the or each picking arm.

27. The system of any preceding claim, in which the survey camera module is shaped as a large hoop that is configured to pass over the or each picking arm.

28. The system of any preceding claim, in which the survey subsystem includes one or more spectrometers.

29. The system of claim 28, in which the one or more spectrometers are positioned at position(s) to be illuminated by light of a similar colour to that of the one or more target fruits.

30. The system of any of claims 28-29, in which the one or more spectrometers are positioned within known position(s) in a shared coordinate frame.

31. The system of any of claims 28-30, in which the system includes a moveable arm configured to position the one or more spectrometers.

32. The system of any preceding claim, in which the survey camera module shares one or more degrees of freedom with the picking head camera, picking arm or picking head.

33. The system of any preceding claim, in which the survey camera module is attached or built-in onto the picking arm.

34. The system of any preceding claim, in which the picking arm is driven by one or more motors, and at least one motor of the picking arm is used to position or orient the survey camera module.

35. The system of any preceding claim, in which the survey camera module is positioned or orientated using a robot arm or flying drone.

36. The system of any preceding claim, in which the survey camera module is positioned or orientated at a fixed position with respect to a feature of the robot, such as the chassis of the robot.37 The system of any preceding claim, in which the survey camera module is mounted to a moveable feature of the robot, such that the camera module is moveable relative to a fixed portion of the robot.

38. The system of any preceding claim, in which the survey camera module is mounted to a rotating feature of the robot, such as a pole, so that the camera module is rotatable relative to a fixed portion of the robot.

39. The system of any preceding claim, in which the survey camera module includes an array of cameras mounted at the front of the system and / or at the back of the system.

40. The system of any preceding claim, in which the survey camera module moves along a linear rail.

41. The system of any preceding claim, in which the survey camera module is adjustable based on the geometry or condition of the crop.

42. The system of any preceding claim, in which fruits are grown on tabletops and height of the survey camera module is adjustable to take into account a height of the tabletops.

43. The system of any preceding claim, in which the survey camera module is mounted to one or more survey arms, such as a single or multi -degree of freedom robot arm.

44. The system of any preceding claim, in which the survey camera module is first calibrated using a calibration target that is mounted at a known position onto the system, such as using fixings on a chassis.

45. The system of any preceding claim, in which the system includes a calibration target that is a planar target including fiducial markers.

46. The system of any preceding claim, in which the system includes a calibration target that is positioned such that it is located within the survey camera module field of view without occluding the imaged scene.

47. The system of any preceding claim, in which the system includes a calibration target that is permanently fixed onto the robotic fruit picking system, while the robotic fruit picking system is in use.

48. The system of claim 45-47, in which the system is configured to check that the reconstructed positions of fiducial markers on the calibration target are consistent with their reconstructed positions immediately after calibration.

49. The system of any preceding claim, in which the or each picking head is user replaceable.

50. The system of any preceding claim, in which the or each picking head includes a light source that illuminates fruits to reduce scene contrast in bright daylight, and also enables picking at dawn, dusk and at night.

51. The system of any preceding claim, in which the position of the picking head camera, picking arm or picking head is a function of a prior survey camera module position.

52. The system of any preceding claim, in which the picking head includes one or more built-in cameras.

53. The system of any preceding claim, in which the system further comprises a picking subsystem and a packaging subsystem, and in which the survey subsystem, the picking subsystem and the packaging subsystem are configured to operate independently of each other and / or each subsystem performs its functions in parallel to the other subsystems.

54. The system of claim 53, in which each subsystem is configured to be easily moveable / detachable from the robotic fruit picking system.

55. The system of any preceding claim, in which the picking subsystem is positioned above the packaging subsystem.

56. The system of any preceding claim, in which the picking subsystem is positioned above storage containers or trays.

57. The system of any preceding claim, in which the packaging subsystem includes a simple robot arm, such as an arm with only two or three degrees of freedom.

58. The system of any preceding claim, in which the packaging subsystem includes a buffer subsystem, such as conveyer belt or carousel or magazine.

59. The system of any preceding claim, in which the packaging subsystem is positioned out of reach of the or each picking arm.

60. The system of any preceding claim, in which the packaging subsystem is configured to deposit the picked fruit or bunch onto an inspection chamber or storage container(s) or tray(s).

61. The system of any preceding claim, in which the system further comprises an inspection subsystem that is configured to image a picked fruit from multiple relative viewpoints.

62. The system of claim 61, in which the inspection subsystem is configured to monitor the size and quality of the picked fruit or bunch and to grade a picked fruit or bunch and determine its suitability for retail.

63. The system of any preceding claim, in which the system is configured to allow the container(s) or tray(s) to be loaded or unloaded from either end of the robot.

64. The system of any preceding claim, in which the system further comprises a linear rail subsystem, in which the or each picking arm is configured to move along the linear rail subsystem.

65. The system of any preceding claim, in which the system comprises at least two picking arms; and at least two picking heads; in which the two picking arms are configured to move along the linear rail subsystem; and in which the system is configured to minimise time a picking arm is quiescent.

66. The system of claim 65, in which the two or more picking arms can travel backwards and forwards along a crop row independently of the robot.

67. The system of any of claims 65-66, in which a first arm is a leading arm and the second arm is a trailing arm.

68. The system of claim 67, in which the leading arm and trailing arm are configured to stay at up to an approximately minimum distance apart.

69. The system of any of claims 65-68, in which neither picking arm is allowed to move in a space required by the one or more other picking arm.

70. The system of any of claims 65-69, in which a working space for each arm to move while picking the next target berry is determined by a 3D bounding box geometry.

71. The system of any of claims 65-70, in which the system is configured to provide a shared working space for both picking arms as the picking arms move along a linear rail or path.

72. The system of any of claims 65-71, in which the leading arm is configured to release a portion of the working space as it moves towards one end of the linear rail or path.

73. The system of any of claims 65-72, in which the trailing arm can reserve working space as it progresses along the linear rail or path.

74. The system of any of claims 65-73, in which if the trailing arm requires more working space that has been released by the leading arm, the trailing arm’s operation is delayed until sufficient working space is available.

75. The system of any of claims 65-74, in which the system includes a control subsystem, such as two control software threads under a shared lock, for managing the reservation and release of the working space.

76. The system of any of claims 65-75, in which the leading and trailing arms are configured to pick in multiple back and forth passes along the robot.

77. The system of any preceding claim, in which the system includes at least one odour sensor configured to detect volatile organic compounds (VOCs) and / or other odours.

78. The system of any preceding claim, in which the includes an air moving device, such as fan or pump, that is configured to draw air from an air intake and to direct the air over a picked fruit surface and past the odour sensor.

79. The system of any preceding claim, in which the includes a test chamber with an opening, and the air moving device is configured to direct the air over a picked fruit surface, into the chamber, and past the odour sensor located inside the chamber.

80. The system of any preceding claim, in which a valve is associated with the opening of the test chamber, the valve being operable to open and close the opening.

81. The system of any preceding claim, in which the odour sensor is configured to analyse the detected VOCs and determine whether a picked fruit should be discarded.

82. The system of any preceding claim, in which the test chamber is also configured to image a picked fruit from multiple viewpoints.

83. The system of any preceding claim, in which the system includes a UV light source configured to emit UV light and expose a picked fruit from multiple relative viewpoints.

84. The system of any preceding claim, in which the picked fruit is exposed when the fruit is held by the packaging arm, and a control subsystem is configured to move the packaging arm such that the angle and direction of the UV light relative to the picked fruit is adjusted.

85. The system of any preceding claim, in which the control subsystem is also configured to control the exposure time of the picked fruit.

86. The system of any preceding claim, in which the UV light source and / or ozone source are arranged configured to enable the irradiation of substantially the entire surface of the picked fruit.

87. The system of any preceding claim, in which the system includes an ozone source configured to emit ozone and expose the target from multiple relative viewpoints.

88. The system of any preceding claim, in which an inspection chamber houses the UV light source and / or the ozone source and the picked fruit.

89. The system of any preceding claim, in which an inspection chamber is made of materials that are opaque to UV radiation and resistance to ozone.

90. The system of any preceding claim, in which some or all picked fruits are treated for sterilization or inspection purposes.

91. The system of any preceding claim, in which the system includes an inspection chamber that houses an odour sensor, a UV light source, an ozone source and imaging sensors.

92. The system of any preceding claim, in which the system includes an inspection chamber that includes one or more light sources that direct or diffuse light onto a picked fruit.

93. The system of any preceding claim, in which the system includes an inspection chamber that uses highly diffuse light source(s) in order to optimise defect detection and / or silhouette extraction, such as for mass assessment.

94. The system of any preceding claim, in which the packaging subsystem includes a packaging arm being configured to transfer the picked fruit or bunch to an inspection chamber or storage container or punnet, in which the packaging arm is a simple, low- cost arm, such as only a 1 or 2 degree-of-freedom robot arm.

95. The system of any preceding claim, in which the picking head is any device or end-effector configured to permanently sever fruit or bunches of fruit.

96. The system of any preceding claim, in which the picking head is any device or end-effector configured to permanently sever one part of a plant from another part of the plant, such as runners or other foliage.

97. A method of optimizing fruit yield mapping across a fruit farm or multiple fruit farms, by analysing images of fruits to be picked in an imaged scene using the robotic fruit picking system defined in any of claims 1-96.

98. A method of determining fruits suitability for picking by using the robotic fruit picking system defined in any of claims 1-96, and to determine the picked fruits suitability for retail.

99. A method of minimising time a picking arm is quiescent by using the robotic fruit picking system defined in any of claims 1-96.

100. A method of maximizing fruit shelflife by using the robotic fruit picking system defined in any of claims 1-96.

101. A method of picking fruit, comprising the step of operating the robotic fruit picking system defined in any of claims 1-96.

Images