Canopy detection system and implement positioning system

Abstract

Advantages are described relating to accurate positioning of implements in a canopy or tree row of production field, such as a vineyard or orchard. In one aspect a method and system are described for detecting a canopy in an orchard or vineyard comprising a plurality of rows of plants, the method comprising, obtaining spatial data from a sensor, the spatial data representing at least portion of one row of plants; determining a subset of the spatial data, the subset comprising at least a portion of the canopy; determining a canopy surface from the determined subset of spatial data.

Description

Canopy detection system and implement positioning system.TECHNICAL FIELD

[0001] The present disclosure generally relates to accurate positioning of implements in a canopy or tree row of production field, such as a vineyard or orchard. In part it relates to a canopy detection method where a face or surface of the canopy is determined, wherein a desired implement position is determined based on one or more faces or surfaces of the canopy.BACKGROUND

[0002] Vineyards (grapes) and orchards (apples, citrus, nuts, pip fruit) often now rely on mechanized tasks to manage the trees or canopies to maintain fruit quality, reduce pest and disease pressure and / or improve yield. The mechanized tasks include activities like trimming, leaf removal, trunk desuckering, fungicide spraying, mechanized pruning and harvesting. Typically, these tasks involved an implement, or at least the head of an implement, which interact with the canopy of each row of plants in the vineyard or orchard. These activities may require a high level of accuracy relative to the plant or tree. In some cases this is the canopy in others it is specific pieces of the plant such as the trunk or cordon in order for the activity to be effective.

[0003] The implements are mounted to a tractor or other vehicle capable of providing power to the implement and navigating the row. A machinery operator may be required to drive the vehicle and simultaneously control the implement. This requires a significant amount of concentration and skill to ensure precise work and avoid damage to the implements or plants and infrastructure.

[0004] Attempts have been made by specific implement suppliers to add mechanized or mechanical sensors to control and maintain an implement position. For example, US11778934B2 shows an agricultural lane following system which uses range data to detect crop rows and, based on the position of a crop row control a position of an autonomous vehicle. Bounding boxes are calculated for plants, for example using a neural net. However, merely controlling the position of an autonomous vehicle does not enable control of the implement attached to the vehicle.

[0005] Further developments have been made with navigational autonomy in orchard and vineyard environments, with self-drive tractors and robots solutions becoming more prevalent.However, the control of implements attached to these vehicles has been lacking. In particular where the actions of the implement may affect the movement or control of the autonomous vehicle.

[0006] It is the intention of the present invention to describe a canopy detection system or implement control system that substantially mitigates one or more of the above limitations.SUMMARY

[0007] Aspects of the invention address one or more of the above problems.

[0008] In a first aspect the invention may broadly be said to consist in a method of detecting a canopy in an orchard or vineyard comprising a plurality of rows of plants, the method comprising: Obtaining spatial data from a sensor, the spatial data representing at least portion of one row of plants; Determining a subset of the spatial data, the subset comprising at least a portion of the canopy; Determining a canopy surface from the determined subset of spatial data.

[0009] Optionally the spatial data comprised a plurality of points. Optionally

[0010] comprising the step of filtering the spatial data to identify at least one of the plurality of rows; Optionally filtering the spatial data comprises removing at least one of ground and ground cover features from the spatial data. Optionally removing at least one of ground and ground cover features comprises any one or more of: A height cut off; A plane estimation of the ground plane; A plane estimation of the sky; An environment filter; Determination of a region tangential to the vertical axis. Optionally removing at least one of ground and ground cover features comprises using a difference of normal to determine the ground or ground cover. Optionally in filtering the spatial data comprises filtering noise. Optionally filtering noise comprises any one or more of: Radius outlier removal; Statistical outlier removal; A passthrough filter. Optionally filtering noise comprises using one or more of an intensity filter, a reflectivity filer and a size of material filter. Optionally the filtering is performed independently of the sensor.

[0011] Optionally the sensor comprises one or more noise filters. Optionally the one or more noise filters are configured or configurable based on the sensor. Optionally the one or more noise filters comprise one or more of an intensity filter, a reflectivity filer and a size of material filter. Optionally the sensor comprises any one or more of LIDAR, radar, depth sensor,or RGB-D camera. Optionally the sensor comprises any one or more of electromagnetic, magnetic, capacitance, laser, thermal and temperature sensors.

[0012] Optionally determining the subset of spatial data comprises segmenting the spatial data into one or more objects. Optionally at least one of the objects comprises at least a portion of the canopy. Optionally at least one of the objects comprises stationary infrastructure and / or dynamic obstacles. Optionally the segmenting is performed by machine learning, optionally wherein the machine learning comprises a neural net.

[0013] Optionally determining the canopy surface comprises: determining a relative orientation of each of a plurality of points in the subset of data; and grouping or selecting points with substantially parallel relative orientation. Optionally the relative orientation is a normal estimation of each of the plurality of points. Optionally the substantially parallel relative orientation is parallel a horizontal axis. Optionally selecting points with substantially parallel relative orientation comprises using the difference of normals. Optionally the canopy surface is contiguous. Optionally the canopy surface is a vertical or horizontal canopy face. Optionally the canopy surface is substantially planar. Optionally the plurality of points are clustered to determine the canopy surface. Optionally the canopy surface comprises a horizontal and / or a vertical face.

[0014] Optionally an implement engagement region is determined by at least one window applied to the canopy surface. Optionally each of the at least one window is configured to set a size or accuracy of the implement engagement region. Optionally each of the at least one window is positioned at an average and / or best fit location relative to the canopy surface. Optionally the at least one window comprises at least two windows of different sizes. Optionally the at least one window is 0.3 metres to 5 metres. Optionally the size of one or more of the windows is configured based on an implement characteristic, such as speed, or function. Optionally the size of the window is dynamic. Optionally a centre of the row is estimated based on the canopy detection of a near face of the canopy and a far face of the canopy. Optionally the centre of the canopy is determined by detection of infrastructure. Optionally the infrastructure is determined by the segmentation of the spatial data. Optionally the near face and far face of the canopy are determined from the spatial data. Optionally the system monitors multiple rows at one time.

[0015] Optionally the spatial data is transformed into a frame of reference different to the frame of reference of the sensor. Optionally the frame of reference is relative to animplement carrying vehicle and / or an implement. Optionally the spatial data is correlated to a map, the correlation comprising linking the position of the spatial data with a location in the map. Optionally the map is a GPS referenced persistent map. Optionally the map comprises historic spatial data. Optionally the method comprises comparing the spatial data and the historical spatial data. Optionally the map is stored at a first resolution and used by a or the implement at a second resolution. Optionally the second resolution is lower than the first resolution, optionally wherein the map is down-sampled or vectorised. Optionally the map is stored in a first size and used by the implement in a second size. Optionally the second size represents an area around one or more of a or the vehicle or a or the implement. Optionally the second size represents a rectangular portion of a row.

[0016] In a further aspect the invention may be said to consist in a method of controlling an implement for an orchard or vineyard comprising at least one row of plants, the method comprising: Obtaining spatial data from a sensor, the spatial data representing at least one row of plants; Determining a canopy face from spatial data; Transforming the frame of reference of the sensor data to a second frame of reference of, or associated with, the implement; and Controlling the implement to follow the canopy face.

[0017] Optionally the implement control comprising at least one linear actuator to control the position of the implement. Optionally the implement is controlled in at least a vertical and / or horizontal direction. Optionally the implement is mounted to a vehicle. Optionally comprising the steps of determining a characteristics (such as the speed and / or direction) of the vehicle and / or the implement and controlling the implement dependent on the determined characteristic. Optionally comprising a persistent map, wherein the determined canopy face is added to the persistent map and the implement is positioned based on the canopy location in the persistent map. Optionally the implement is configured to follow the outer edge of the canopy and / or a to follow a set distance from the center of the canopy.

[0018] Optionally the implement is any one or more of a trimmer, a sprayer, a harvester, a pruner, a desucker and a defoliator. Optionally the method comprises reference data, the reference data specifying the desired position of the implement relative to the horizontal and / or vertical face of the canopy. Optionally the method comprises receiving, from the implement, a signal determining the correct functioning of the implement. Optionally the received signal can stop / vary operation of the implement. Optionally the implement is attached to an automated vehicle. Optionally the automated vehicle is configured to automatically movealong the rows of the vineyard or orchard. Optionally comprising determining a required position of the implement by considering the expected or actual movement of the vehicle relative to the position of the sensor when the point cloud data was obtained.

[0019] Optionally the implement comprises a position sensor (GPS), optionally the GPS sensor confirms the expected location of the implement and the canopy.

[0020] Optionally the implement comprises an image detector, the image detector configured to confirm a location of the canopy. Optionally the canopy face has a minimum spacing from a row center. Optionally the canopy face is detected as below the minimum width the implement is withdrawn from the row. Optionally the implement is mounted on a towed vehicle. Optionally the sensor is mounted on the towing vehicle. Optionally the implement is mounted on a trailer. Optionally the vehicle and or trailer have a GPS tracker.

[0021] In a further aspect the invention may be said to consist in a persistent map of an orchard or vineyard comprising a plurality of rows of trees, shrubs or vines, the persistent map comprising, for each of the rows: A left hand side canopy surface; A right hand side canopy surface; and A canopy centre.

[0022] Optionally the persistent map further comprising a top canopy surface.

[0023] Optionally the canopy surfaces are produced by the described method, or have the features and / or characteristics of the described method.

[0024] In a further aspect the invention may be said to consist in a system for an orchard or vineyard comprising a plurality of rows of trees, shrubs or vines, the system comprising: A vehicle associate with a spatial sensor; An implement moveably attached to the vehicle; A controller configured to: Receive the spatial information from the sensor; Determine a canopy face of one or more of the rows; Control the implement to track the canopy face.

[0025] Optionally the accuracy of determining the canopy face is controllable, such that the implement can be configured to closely or crudely follow the canopy face. Optionally the sensor data and the implement are positioned relative to a frame of reference of the vehicle. Optionally the position of the implement relative to the canopy is configured based on the vehicle movement and / or expected movement. Optionally the vehicle is an autonomous vehicle configured to move along each of the rows of the orchard. Optionally the controller is configured to implement any one or more of the methods claimed herein.

[0026] In a further aspect the invention may be said to consist in a method of detecting a canopy in an orchard or vineyard comprising a plurality of rows of plants, the methodcomprising: Obtaining spatial data from a sensor, the spatial data representing at least portion of one row of plants; Determining a subset of the spatial data, the subset comprising at least a portion of an individual plant in the row of plants; Determining one or more portions of each of the individual plants based on the determined subset of spatial data.

[0027] Optionally the one or more portions comprise a canopy or a cordon of the plant.

[0028] Optionally comprising any one or more of the steps of the methods described herein. Optionally the steps are configured to refer to calculation or detection of the portion of the plant, such as the cordon, instead of the canopy. For example, machine learning may be used to identify cordons and the implement may track the position of neighbouring cordons along the rows.

[0029] Features from one or more embodiments or configurations may be combined with features of one or more other embodiments or configurations.

[0030] As used herein the term "(s)" following a noun means the plural and / or singular form of that noun.

[0031] As used herein the term "and / or" means "and" or "or", or where the context allows both.

[0032] The term "comprising" as used in this specification means "consisting at least in part of". When interpreting each statement in this specification that includes the term "comprising", features other than that or those prefaced by the term may also be present. Related terms such as "comprise" and "comprises" are to be interpreted in the same manner.

[0033] It is intended that reference to a range of numbers disclosed herein (for example, 1 to 10) also incorporates reference to all rational numbers within that range (for example, 1, 1.1, 2, 3, 3.9, 4, 5, 6, 6.5, 7, 8, 9 and 10) and also any range of rational numbers within that range (for example, 2 to 8, 1.5 to 5.5 and 3.1 to 4.7) and, therefore, all sub-ranges of all ranges expressly disclosed herein are hereby expressly disclosed. These are only examples of what is specifically intended and all possible combinations of numerical values between the lowest value and the highest value enumerated are to be considered to be expressly stated in this application in a similar manner.

[0034] This disclosure may also be said broadly to consist in the parts, elements and features referred to or indicated in the specification of the application, individually or collectively, and any or all combinations of any two or more said parts, elements or features.

[0035] Where specific integers are mentioned herein which have known equivalents in the art to which this disclosure relates, such known equivalents are deemed to be incorporated herein as if individually set forth.

[0036] The disclosure consists in the foregoing and also envisages constructions of which the following gives examples only.BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The invention will now be described by way of example only and with reference to the drawings in which:

[0038] Figure 1 shows a diagram of a vehicle with a sensor and tools moving along rows in an orchard or vineyard.

[0039] Figure 2 shows a flow chart of example steps in an implement control process.

[0040] Figure 3 shows a flow chart of example steps in a further implement control process,

[0041] Specific embodiments and modifications thereof will become apparent to those skilled in the art from the detailed description herein having reference to the figures that follow.

[0042] Figure 1 shows an overview of a canopy detection system. A vehicle 10 has at least one sensor 11 attached to obtain a spatial map of the environment surrounding it. The environment is a vineyard or orchard with a plurality of rows 15 of vines or trees, of which two are shown in Figure 1. At least one implement 13 (also referred to as a tool) is attached to the vehicle 10 or configured to be moved by the vehicle 10. As the vehicle 10 moves along the row the implements 13 are configured to track the surface of the canopy. Depending on the implement 13 the surface may comprise the side surface (a vertical surface along the side of the canopy), the top surface (a horizontal surface along the top of the canopy) and / or an angled surface to obtain a particular effect. Where plants are discussed herein these refer to plants having foliage, including trees, shrubs or vines. The plants are typically grown in rows for ease of production.

[0043] For example, the implements 13 may comprise implements for activities like trimming, leaf removal (defoliation), trunk desuckering, fungicide spraying, mechanized pruning and crop harvesting. The implement 13 may have at least one implement head. Each implement head may be configured to interact with a single surface of the canopy. The surface may be planar or shaped or may be configured to shape the canopy of the vine. For example,a trimmer may be configured to trim a top or side of the canopy with planar horizontal and / or vertical surfaces. The surface may be referred to as an interaction surface, typically in the direction of travel (referred to as horizontal) with a vertical orientation. In some cases, they also interact with the top of the canopy with a horizontal orientation. The location of the implement relative to the surface may be in contact or spaced at a distance from the surface. For example, trimming uses contact, while leaf removal or defoliation may be spaced at a distance from the canopy to ensure appropriate coverage of the canopy. In some situation the interaction is relative to the canopy, while other interactions may use a reference from the canopy centre. In some cases, the implements may interact with multiple canopy planes, or multiply canopy faces in different rows. In some cases, the implement may straddle the entire canopy and interact on both sides of the canopy.

[0044] The implements may be moveable relative to their mounting, such as relative to the vehicle. This enables the implement to be moved independently of the vehicle so as to better track the canopy position during operation. The implement may be moved by one or more actuators, such as linear actuators. The actuators may be electrically or hydraulically driven. The speed and / or operational characteristics of the implement may also be controlled. In some cases, the movement of the implement may be achieved by mounting the implement to a moveable frame. The implement may be moveable in one, two or three dimensions. In one dimension the implement may be moveable in and out so as to control the closeness of the implement to the canopy. In two dimensions the implement may also be moveable up and down so as to track canopy height. In three dimensions the implement slope or curve may also be adjustable so as to better follow the canopy (either horizontally or vertically). The speed of movement of the implement may be controller and / or limited based on the speed and / or direction of the vehicle and / or the required movement to get to the next engagement region of the canopy. The implement may have one or more characteristics which are provided to the system to decide, or limit, the performance or operation of the system. For example, the characteristics may include a speed or movement and / or a position to be placed relative to the canopy.

[0045] The interaction planes are typically in the direction travel on a vertical orientation however some also interact with the top of the plant or tree in a horizontal axis. In some situation the interaction is relative to the canopy itself such as leaf removal and others require general offsets from the canopy centre such as trimming. These implements can havemultiple heads so that they can interact with multiple canopy planes in multiple rows with each row pass.

[0046] The present disclosure describes how the system is able to track and / or improve the tracking of the canopy to allow the implements to better follow the canopy to perform the required task. It has been found that prior art system which simply identify rows or perhaps the presence of a canopy does not provide the necessary accuracy for performing mechanical tasks, which is available by determining one or more faces or surfaces of the canopy.

[0047] In overview the system first obtains spatial data of at least a portion of a row in the vineyard or orchard, it then processes the spatial data to at least determine a canopy portion. Once the canopy portion is determined at least one face of the canopy portion is determined. The face of the canopy portion can then be intersected with an implement engagement window to determine where an implement should be placed to properly interact with the canopy. Advantageously this system ensures that the implement can accurately follow the canopy face, rather than a general area where the canopy may be present.

[0048] Figure 2 shows example steps that may be included in the present system. As shown the system first obtains spatial data 201 of the vineyard or orchard. As shown in Figure 1 this may be obtained from a sensor 11 arranged on the vehicle 10 supporting the implement 13. Alternatively, the sensor 11 may be attached to a different vehicle, or the spatial data may be obtained separately. The spatial data may be obtained from a LIDAR, radar, depth sensor or other suitable device such as one or more magnetic, electromagnetic or temperature sensors that define a particular characteristic of the entire canopy or point in the canopy that interactive decisions may be made from.

[0049] The spatial data may be saved as a plurality of points (for example, a point cloud map), each point representing a location in the vineyard or orchard and combining to form at least a portion of a row of plants. In some cases, further data may be associated with each of the points, such as colour data based on an image. The spatial data may be transformed or stored in an alternative format for efficiency, to save space, or based on preference. For example, the spatial data may be represented as a 2D colour or pixel map. Spatial data can be provided by a number of sensors 11, whether depth detection sensors like LIDAR, stereo cameras, or estimated date based on series of images. The presentation of 3D space is obtained as a point cloud which may also be represented as a 2D colour or pixel map. In some cases,the spatial data may be combined with image data to create or assign colour (RGB) data to each of the points in the point cloud

[0050] The spatial data is then filtered to identify at least one of the plurality of rows. This may comprise removing and cleaning the data.

[0051] By removing features above and below a chosen height in the point cloud data unnecessary or unwanted features (such as ground features can be excluded). Known filtering techniques may be used, such as dependent on movement, location or known to the controller based on mapping. For example, the sensor 11 may apply a filter, such as a noise filter. The noise filter may comprise one or more of an intensity filter, a reflectivity filter and / or a size filter. The choice of filter may be dependent on the sensor 11 and / or the type of canopy or environment. For example, RADAR sensing is sensitive to the level of absorption of radio frequencies in the environment so the reflectivity may be managed, while LIDAR is sensitive to the level of reflectivity of the environment so intensity may be managed. The most effective filtering may depend on how the sensor responds to the environment of the vineyard or orchard. In some cases, the filtering, or additional noise filtering, is performed on the signal from the sensor, or by the controller after receiving the spatial data. Noise filters may also include any one or more of radius outlier removal, statistical outlier removal and / or a passthrough filter. These filters may attempt to remove errors in the spatial data, for example.

[0052] The spatial data may be filtered 203 to reduce the area to be reviewed. In particular, there may be a concentration on the canopy area with the filters configured to reduce or remove other features. In some cases, the filter may remove non-canopy features and / or identify known environment features. Removing non-canopy features may include one or more height cutoffs - for example an upper height cutout to identify a sky portion, and a lower height cutoff to determine a ground portion or ground cover portion of the spatial data. Filters are available for this purpose, including a height cut-off filter, a plane estimation of the ground plane, a plane estimate of the sky, an environmental filter, or determination of regions tangential to the vertical axis. A difference of normals technique may be used to identify and remove the ground plane. The most effective filter may be selected or tested based on, for example, the type of ground cover present. In maintained environments there may be little ground cover, so that the ground plane is easily determined. In contrast, wilder environments may include longer grass and / or branches on the ground which requires greater processing for removal. In some cases, detection of infrastructure or markings in the vineyard or orchardmay be used to remove unneeded spatial data. For example, detection of posts or wires may be used to determine the position of the canopy.

[0053] Once filtered the spatial data may be segmented 206 or divided into different components. Segmenting the spatial data enables the system to concentrate on the areas of interest - the canopy and or canopy infrastructure such as wires, posts, trunks and cordons - and pass the other data to the controller for navigation, object avoidance or general processing if required.

[0054] Segmentation could be performed by filter or shape fitting for known infrastructure, based on positional data, or based on machine learning, neural net classification or other techniques. Segmentation allows that spatial data to be divided into the desired areas and the canopy potion to be better identified. Segmentation may also identify one or more initial faces of the canopy. For example, the spatial data may have obtained points at both a near and far side of the canopy (as the canopy is only partially occluded, allowing some sight through to the far side of the canopy). Segmentation may identify which points belong to which side of the canopy, or this may be processes later. Segmentation may also identify and / or remove static or dynamic objects which are not canopy. For example, humans and / or animals present in the orchard or vineyard, infrastructure, or other objects. The segmenting may be based on assumptions of what is expected to be seen in the vineyard / orchard, or alternatively by machine learning, such a neural net processing of the environment.

[0055] After segmenting the spatial data a canopy portion has been determined. The system now determines at least one canopy face, so as to allow the control of an implement on the canopy face. This improves over prior art systems which simply provide a coarse estimate of the canopy face, or simply the general location of a row. By determining a canopy face or surface or surface portion, instead of merely the presence of the canopy or a general area of foliage, the surface of the canopy can be used to, for example, allow an implement to track the surface of the canopy and perform an agricultural task on the canopy. This also helps to avoid outliers or extensions from the canopy. In contrast row tracking, or coarse estimation does not provide an accurate canopy position, or varies widely as, for example, overgrown vines or uneven canopies occur. Without an accurate canopy position the control of implements is not sufficient.

[0056] Determining a face of the canopy 207 comprises identifying and grouping neighbouring points in the spatial data which are identified as canopy. This could be performedby machine learning, or a neural net, or specific techniques may be used. For example, a relative orientation of each point in the spatial map may be determined. One method of doing so is determining a normal estimation of each of the points. The normal estimation is typically estimated based on the nearby points and relative angles between them. Then, by comparing the relative orientation of neighbouring points a canopy face can be constructed by linking points with similar orientations, or where the orientations follow a predicted curve. For example, where the canopy face is substantially planar and rectangular each of the canopy points along the row will have a normal perpendicularto the face of the canopy, and the canopy face can be formed by connecting each of these points. In other cases, for example at end of rows, or where gaps are present in the rows the curvature of the canopy may increase. Here the system can look for small changes in the relative orientation.

[0057] In some cases, a threshold is used to determine whether neighbouring points have substantially parallel relative orientation, and therefore form the canopy face. The threshold may consider points within 5-10% to be consistent with a contiguous canopy. The threshold may be dependent on the required accuracy for the implement, or the apparent curvature of the canopy. For example, where previous data is suggestive of a curving canopy the threshold may increase to allow for the greater expected curvature. The threshold may also depend on the type of foliage or leaves present in the canopy.

[0058] There are known methods of determining a continuous or contiguous surface based on the estimated normal vectors. For example, the system may use a normal estimation to estimate a normal vector at a point based on the surrounding points. By calculating at each point and comparing between neighbouring points contiguous surfaces can be determined. Small and / or large-scale search algorithms can determine these features by looking at the difference of normals. Where substantially parallel relative orientation is discussed, it should be understood as referring to neighbouring points being similar enough to form the canopy. Due to the roughness of canopy edges (i.e., because of individual leaves) there will be differences present. For example, the similarity may be up to 20%, up to 10%, or up to 5%.

[0059] The canopy face may be determined 207 in at least a first direction. The first direction may be along the row in the direction of movement of the tool or implement. The canopy may be estimated in a second dimension, representing the height or vertical length of the canopy. In some cases, the canopy face is determined in first and second dimensions, so as to closely map the 3D shape of the canopy. The canopy face can be determined separatelyfor the canopy side (i.e., the vertical canopy wall) and the canopy top (i.e., the horizontal canopy wall) or these may be determined as a single canopy. The canopy face may be determined as a single contiguous structure, or the canopy may be determined as a series of canopy sections, either adjacent, overlapping or spaced apart depending on the spatial data available. In some cases, the canopy face is detected by clustering points, and then linking adjacent clusters. Alternatively, a region growing approach may be used.

[0060] The step of identifying a canopy face 207 minimises false detection of outliers and / or non-canopy features while accurately identifying the canopy. This is because the canopy should extend from the trunk of the plants to provide a clear vertical feature in the vineyard or orchard row, and one that extends along the row.

[0061] Once the canopy face has been detected 207 the system may save the canopy face for later use or process the use in real-time. In real-time the canopy face may be processed and directed to a canopy implement, or a control mechanism for a canopy implement, to move an implement to the canopy face. To achieve this the system may determine a canopy engagement region 208. Canopy engagement regions 14, 15 and 16 are shown in Figure 1. The canopy engagement regions from a window or area of the canopy surface. The desired size of the window may be dependent on any one or more of the size of the implement head, the speed of movement of the vehicle, the precision or tolerance required for the implement operation. In some cases, the size of window may be selected from a plurality of preselected window size. In some case the size of the window is selected or configured based on an implement, environment and / or movement characteristic. In some cases, the window size is dynamic based on any one or more of the characteristics. In some cases, the window covers the full height of the canopy, with the size controlled by the extent of the window along the row. Possible sizes include between 30cm and 5m (along the row). The window is discussed as rectangular, but circular, elliptical or other shaped windows may be sued, for example where the shape of the canopy is better suited to a non-rectangular window. Changing the size of the window impacts the averaging of the data and responsiveness to localized changes for the implement. Larger windows mean smoother data but less response to small changes.

[0062] Once a window size has been selected the system compares the window with the detected canopy surface. This may comprise overlaying or arranging the window on the canopy surface to provide the best agreement. In some cases, the window is centred on an average location of the canopy surface across the window. For example, a least squares fit maybe used to find the location which reduces any discrepancy between the canopy surface and the window the most. In other cases, the window may be aligned to an approximate middle of the canopy, or a maximum or minimum of the canopy. The apposition or alignment of the window may be at an offset to the canopy surface - for example trimming may target 5-10cm below the surface of the canopy. In some cases, the alignment is to the centre of the canopy (i.e., a set distance from the center of the canopy).

[0063] In one example the implement is a trimmer. The trimmer could use two windows: A precise trim may require a relatively narrow window so as to closely follow the contour of the canopy, while the implement may also use the wider setting to track the general curve of the canopy. In some cases, instead of single, substantially vertical, windows sized for the size of the implement head, there may be multiple vertical windows. In some cases, the windows may be tilted, for example, to provide an angled prune, or concentrate spray in a portion of the canopy.

[0064] Once the implement engagement region has been detected the implement can be controlled 209 to the engagement region. Typically, this will occur as the vehicle supporting the implement moves along the row. There is generally a spacing between the sensor and the implement. This provides a time for the implement to be moved 209 (i.e., by a linear actuator) so that the implement intersects with the determined engagement region when the vehicle has moved forward. Where the system is operating continuously this should mean that the implement moves from implement engagement region to implement engagement region smoothly as the vehicle moves along the row. In cases where there are gaps between regions the implement may be withdrawn from the canopy to avoid unwanted interactions. In some cases, there is a change in reference frame 204 before implement operation. This may be performed on the spatial data before it is processed, or the spatial data may be processed and then the output transformed to the new reference frame. The change in reference frame may be used to adjust for differences in the position and / or movement of the sensor and the implement. For example, the frame of reference may be adjusted to the vehicle or implement location, so as to simplify control signals sent to the implement.

[0065] As mentioned previously in some cases the spatial data is able to determine more than simply the near face of the canopy and / or the top of the canopy. By using captured spatial information, the far side of the canopy may also be detected, or at least enough points on the far side of the canopy to determine an approximate position of the canopy if not a farside canopy surface. The spatial data may, for example by the gaps between foliage in the canopy and the many different angles obtained during operation, provide an estimate of the near and far faces of the canopy. In this way the system can accommodate the occlusions normally expected from the far side of the canopy. Detecting an approximate position of the far side of the canopy allows the system to calculate a canopy center based on, for example, a midpoint between the near and far canopy sizes. In some cases, implements are positioned dependent on the detected centre of the canopy. In some cases, detection of infrastructure, such as posts or wires may assist the detection of the centre of the canopy (as the centre should be aligned with this infrastructure). In some cases, a cluster centroid with a bias in the y direction is used to estimate a center of the canopy. This may be used in combination with alignment to other segmented features such as posts.

[0066] In some cases, the system uses a map 205 to store and / or correlation the spatial data. The map may store the data as received from the sensor, so as to allow the canopy to be determined with all available data, or the map may store the processed canopy data to be more easily used by the implement. For example, the map may record a left-hand side surface of the canopy, a right-hand side surface of the canopy and a canopy centre. Additionally, a top canopy surface may be mapped. Correlation comprises linking the position of the spatial data to a position on the map. This may be based on a position measurement (i.e., GPS) of the sensor, vehicle or implement, or by recognizing features or infrastructure already on the map. Using the map allows for data to be combined or compared to historical data, as well as allowing processing of multiple rows at once (using multiple row implements).

[0067] In some cases, the map is provided to the implement or implement controller directly. However, the implement controller may receive only a portion of the map, a down- sampled map, or a limited resolution of map to reduce the processing required. In this way the map may provide a full resolution store of all the data from the orchard (i.e., pre and post implement performance data), and the implement may be provided with an area restricted or region restricted portion of the map to be able to quickly process the required implement engagement region. A persistent map having at least both canopy surfaces and a center of the canopy allows an implement to be automated to pass along each row and perform any one or more actions without requiring independent sensing. This improves on a simple mapping scheme which contains too much detail to usefully control an implement and requires additional processing on the implement.

[0068] In some cases, an objective of the system is to build up a map of the information intended to be used to control the implement relative to the canopy. For example, in the case of implemented mounted sensors all you are concerned about is the canopy. In the case of a vehicle, you are also focused on objects and navigation of the vehicle. The use of the map allows the same spatial information to be processed through different pipelines. The use of a map also allows the combination of sensor data (from different sensors, or over time). As each sensor has limited field of view or coverage as it moves in space the map can collect and coordinate a lot more information than the single sensor. This can provide a more detailed view of the world especially when talking about objects like canopies and trees. The map may be a persistent map, or a GPS referenced persistent map to improve usability.

[0069] The disclosure has discussed generic implement attached to a vehicle. However, implement may be attached directly to the vehicle, to a second vehicle (such as a trailer) attached to the vehicle, or to a separate vehicle. Where different vehicles are used a position indication (such as GPS) can help to ensure correct alignment between the two vehicles, or between the spatial data and the implement. The implement may be configured to determine a path to move between engagement region windows with the least movement, or with a substantially continuous profile. In some cases, the implement has one or more sensors. The sensors (e.g., an image sensor, a depth sensor or a pressure or force sensor) may be used to confirm the expected position of the canopy is correct and / or to stop or prevent operation of the implement where incorrect operation or an error is detected.

[0070] As shown in Figure 1 the system includes a sensor 11 mounted to a vehicle or otherwise able to move along rows of the orchard. Multiple sensors may be used to provide additional information, either spatial information or, for example, colour information. Additional sensors may include position sensors, such as GPs, or RTK GPS for improved accuracy or an IMU (inertial measurement unit) to measure force on the implement or control for movement of the sensor. The system has a controller, such as a processor or microprocessor for processing the spatial information into the canopy surface information. Multiple processors may be used. The controller(s) is configured to process the spatial information as discussed. In some cases, the processing may occur remotely, and the control instructions provided back to the implement.

[0071] In some cases, the processing is performed on board one or more of the vehicle, sensor or implement. Communication between the sensors, vehicle and implement may beused to ensure the location of the vehicle and or implement is accounted for during movement of either. The communication around the vehicle may use CANbus or other communication protocol. The controller may have an input to receive details on the implement attachment, implement characteristics, or desired task. The input may be on the vehicle or the implement or entered remotely. In some cases, the implement has an indicator that provides an input to the controller when connected to the vehicle. The vehicle and / or implement may have at least one sensor to detect improper operation of the implement, or operation of the vehicle and / or implement outside of one or more specified parameters.

[0072] Figure 3 shows a process flow of the system when controlling an implement. In a first process the spatial data is obtained, used to determine the canopy surface by, for example, comparison of points with neighboring points. At the same (or separately) time the characteristics of an implement are obtained to determine a frame of reference required by the implement and a preferred location of the implement relative to the canopy. The characteristics may also be used to determine a suitable size of window (this may also depend on speed or vehicle movement). When the implement is in place at the canopy an engagement region or contact interface is determined, indicating where the tool should be placed to perform the required action on the canopy. The implement can then be controlled to move to that contact interface and follow a path of a series of contact interfaces as it is moved along the row.

[0073] Although certain embodiments and examples are disclosed herein, inventive subject matter extends beyond the specifically disclosed embodiments to other alternative embodiments and / or uses, and to modifications and equivalents thereof. Thus, the scope of the claims or embodiments appended hereto is not limited by any of the particular embodiments described herein. For example, in any method or process disclosed herein, the acts or operations of the method or process may be performed in any suitable sequence and are not necessarily limited to any particular disclosed sequence. Various operations may be described as multiple discrete operations in turn, in a manner that may be helpful in understanding certain embodiments; however, the order of description should not be construed to imply that these operations are order dependent. Additionally, some structures described herein may be embodied as integrated components or as separate components. For purposes of comparing various embodiments, certain aspects and advantages of these embodiments are described. Not necessarily all such aspects or advantages are achieved by any particular embodiment. Thus, for example, various embodiments may be carried out in amanner that achieves or optimizes one advantage or group of advantages as taught herein without necessarily achieving other aspects or advantages as may also be taught or suggested herein.

[0074] In some cases, the canopy is not the feature of interest. For example, in winter pruning the implement may be located relative to a position of a trunk, or a cordon. A cordon refers to a limb of the plant or tree which is trained or attached to infrastructure so as to guide it into a preferred position. The use of horizontal cordons is widespread in, for example, vineyards to form clear rows and allow easier processing of the vines. Mechanical pruning of the cordon is performed in winter so as to provide for the new vine growth in spring and summer. Precise accuracy is required to prevent damage to the cordon and / or infrastructure of the vineyard. However, prior art solutions do not provide adequate identification around the cordon.

[0075] Returning to Figure 2, the canopy face detection step 207 may be replaced by a cordon detection step. In the cordon detection step the face or location of the cordon is determined. This may be achieved in a similar method to the canopy - the normal of neighbouring points may be determined and then classified to identify neighbouring points, which are grouped together to form regions corresponding to each cordon. Alternatively, an expected surface, in the case of a cordon a cylinder or semi-cylinder surrounding the cordon could be mapped to the determined points. In a further example the infrastructure may be used to determine the cordon. Typically, the cordon is attached to or follows the wires, which are strung or attached to posts and create infrastructure along each row of the vineyard or orchard. Next, by determining a tool engagement region, defined by a window around the cordon (or the wire if detected), the cordon can be pruned. The size of the window may be predetermined or may be determined based on the size and / or shape of the detected cordon. As described above the size of the window may be dependent on the implement acting on the cordon. This may allow the system to determine both broad or precise location or features of the cordon, if required. In some cases, a known shape or configuration of the cordon may be used to determine the location of the cordon. For example, the canopy face detection used the known contiguous shape of the canopy to determine the canopy face, whereas here the system may use the known shape (along wires) or configuration (cylindrical with branching points) to determine the cordon.

[0076] In other cases, different features of the plants and / or trees may be determined. In each case, after segmentation of the spatial data has identified the plants or trees of interest a second level of determination is used to determine, at a predefined or selected scale, the face or surface of the feature of the plant or tree. The face or surface detection may use a normal estimation or classification method to determine the neighbouring points or locations in the plant or tree. After detection of the face or surface a window method is used to determine the implement engagement region. By controlling the size of the window, the preciseness of the window can be controlled to a tolerance required for the implement and / or task.

[0077] In some cases, the implement is not attached directly to a vehicle, but instead is towed behind a vehicle. For example, a tow behind harvester may be sued. In current systems the tow behind harvester is controlled by a person in a second vehicle, or a tractor cab, or walking behind the harvester. The present disclosure may be applied to this example, where the spatial data, possibly from a sensor attached to the towing vehicle, is then processed, optionally transformed into reference frame of the implement (or towed vehicle) and then the implement is actuated or controlled to follow the required feature, such as the canopy face. In such an example a positioning device, such as a GPS, on the towing vehicle, towed vehicle or implement may be used to ensure the correct adjustments are made between the spatial data (i.e., the location from which the spatial data was recorded) and the location of the implement. This may be important due to the relative movement of the vehicle and towed vehicle meaning that compensation is required for the position of the implement.

[0078] It should be emphasized that many variations and modifications may be made to the embodiments described herein, the elements of which are to be understood as being among other acceptable examples. All such modifications and variations are intended to be included herein within the scope of this disclosure and protected by the following claims. Further, nothing in the foregoing disclosure is intended to imply that any particular component, characteristic or process step is necessary or essential.

Claims

CLAIMS1. A method of detecting a canopy in an orchard or vineyard comprising a plurality of rows of plants, the method comprising:Obtaining spatial data from a sensor, the spatial data representing at least portion of one row of plants;Determining a subset of the spatial data, the subset comprising at least a portion of the canopy;Determining a canopy surface from the determined subset of spatial data.

2. The method as claimed in claim 1 wherein the spatial data comprised a plurality of points.

3. The method as claimed in claim 1 or 2 comprising the step of filtering the spatial data to identify at least one of the plurality of rows;4. The method as claimed in claim 3 wherein filtering the spatial data comprises removing at least one of ground and ground cover features from the spatial data.

5. The method as claimed in claim 4 wherein removing at least one of ground and ground cover features comprises any one or more of:A height cut off;A plane estimation of the ground plane;A plane estimation of the sky;An environment filter;Determination of a region tangential to the vertical axis.

6. The method as claimed in any one of claims 4 or 5 wherein removing at least one of ground and ground cover features comprises using a difference of normal to determine the ground or ground cover.

7. The method as claimed in any one of claims 3 to 6 wherein filtering the spatial data comprises filtering noise.

8. The method as claimed in claim 7 wherein filtering noise comprises any one or more of:Radius outlier removal;Statistical outlier removal;A passthrough filter.

9. The method as claimed in claim 7 or 8 wherein filtering noise comprises using one or more of an intensity filter, a reflectivity filer and a size of material filter.

10. The method as claimed in any one of claims 3 to 9 wherein the filtering is performed independently of the sensor.

11. The method as claimed in any one of claims 1 to 10 wherein the sensor comprises one or more noise filters.

12. The method as claimed in claim 11 wherein the one or more noise filters are configured or configurable based on the sensor.

13. The method as claimed in any one of claims 11 to 12 wherein the one or more noise filters comprise one or more of an intensity filter, a reflectivity filer and a size of material filter.

14. The method as claimed in any one of claims 1 to 13 wherein the sensor comprises any one or more of LIDAR, radar, depth sensor, or RGB-D camera.

15. The method as claimed in any one of claims 1 to 14 wherein the sensor comprises any one or more of electromagnetic, magnetic, capacitance, laser, thermal and temperature sensors.

16. The method as claimed in any one of claims 1 to 15 wherein determining the subset of spatial data comprises segmenting the spatial data into one or more objects.

17. The method as claimed claim 16 wherein at least one of the objects comprises at least a portion of the canopy.

18. The method as claimed in any one of claims 16 to 17 wherein at least one of the objects comprises stationary infrastructure and / or dynamic obstacles.

19. The method as claimed in any one of claims 16 to 18 wherein the segmenting is performed by machine learning, optionally wherein the machine learning comprises a neural net.

20. The method as claimed in any one of claims 1 to 19 wherein determining the canopy surface comprises: determining a relative orientation of each of a plurality of points in the subset of data; and grouping or selecting points with substantially parallel relative orientation.

21. The method as claimed claim 20 wherein the relative orientation is a normal estimation of each of the plurality of points.

22. The method as claimed in any one of claims 20 to 21 wherein the substantially parallel relative orientation is parallel a horizontal axis.

23. The method as claimed in any one of claims 20 to 22 wherein selecting points with substantially parallel relative orientation comprises using the difference of normals.

24. The method as claimed in any one of claims 20 to 23 wherein the canopy surface is contiguous.

25. The method as claimed in any one of claims 20 to 24 wherein the canopy surface is a vertical or horizontal canopy face.

26. The method as claimed in any one of claims 20 to 25 wherein the canopy surface is substantially planar.

27. The method as claimed in any one of claims 20 to 22 wherein the plurality of points are clustered to determine the canopy surface.

28. The method as claimed in any one of claims 20 to 27 wherein the canopy surface comprises a horizontal and / or a vertical face.

29. The method as claimed in any one of claims 1 to 28 wherein an implement engagement region is determined by at least one window applied to the canopy surface.

30. The method as claimed claim 29 wherein each of the at least one window is configured to set a size or accuracy of the implement engagement region.

31. The method as claimed in any one of claims 29 to 30 wherein each of the at least one window is positioned at an average and / or best fit location relative to the canopy surface.

32. The method as claimed in any one of claims 29 to 31 wherein the at least one window comprises at least two windows of different sizes.

33. The method as claimed in any one of claims 29 to 32 wherein the at least one window is 0.3 metres to 5 metres.

34. The method as claimed in any one of claims 29 to 33 wherein the size of one or more of the windows is configured based on an implement characteristic, such as speed, or function.

35. The method as claimed in any one of claims 29 to 34 wherein the size of the window is dynamic.

36. The method as claimed in any one of claims 1 to 35 wherein a centre of the row is estimated based on the canopy detection of a near face of the canopy and a far face of the canopy.

37. The method as claimed claim 36 wherein the centre of the canopy is determined by detection of infrastructure.

38. The method as claimed claim 37 wherein the infrastructure is determined by the segmentation of the spatial data.

39. The method as claimed in any one of claims 36 to 38 wherein the near face and far face of the canopy are determined from the spatial data.

40. The method as claimed in any one of claims 1 to 39 wherein the method monitors multiple rows at one time.

41. The method as claimed in any one of claims 1 to 40 wherein the spatial data is transformed into a frame of reference different to the frame of reference of the sensor.

42. The method as claimed claim 41 wherein the frame of reference is relative to an implement carrying vehicle and / or an implement.

43. The method as claimed in any one of claims 1 to 41 wherein the spatial data is correlated to a map, the correlation comprising linking the position of the spatial data with a location in the map.

44. The method as claimed claim 43 wherein the map is a GPS referenced persistent map.

45. The method as claimed in any one of claims 43 to 44 wherein the map comprises historic spatial data.

46. The method as claimed claim 45 wherein the method comprises comparing the spatial data and the historical spatial data.

47. The method as claimed in any one of claims 43 to 46 wherein the map is stored at a first resolution and used by a or the implement at a second resolution.

48. The method as claimed claim 47 wherein the second resolution is lower than the first resolution, optionally wherein the map is down-sampled or vectorised.

49. The method as claimed in any one of claims 43 to 48 wherein the map is stored in a first size and used by the implement in a second size.

50. The method as claimed in claim 49 wherein the second size represents an area around one or more of a or the vehicle or a or the implement.

51. The method as claimed in claim 49 or 50 wherein the second size represents a rectangular portion of a row.

52. A method of controlling an implement for an orchard or vineyard comprising at least one row of plants, the method comprising:Obtaining spatial data from a sensor, the spatial data representing at least one row of plants;Determining a canopy face from spatial data;Transforming the frame of reference of the sensor data to a second frame of reference of, or associated with, the implement; andControlling the implement to follow the canopy face.

53. The method of claim 52 wherein the implement control comprising at least one linear actuator to control the position of the implement.

54. The method of claim 53 wherein the implement is controlled in at least a vertical and / or horizontal direction.

55. The method of any one of claims 52 to 53 wherein the implement is mounted to a vehicle.

56. The method of any one of claims 52 to 55 comprising the steps of determining a characteristics (such as the speed and / or direction) of the vehicle and / or the implement and controlling the implement dependent on the determined characteristic.

57. The method of any one of claims 52 to 56 comprising a persistent map, wherein the determined canopy face is added to the persistent map and the implement is positioned based on the canopy location in the persistent map.

58. The method of any one of claims 52 to 57 wherein the implement is configured to follow the outer edge of the canopy and / or a to follow a set distance from the center of the canopy.

59. The method of any one of claims 52 to 58 wherein the implement is any one or more of a trimmer, a sprayer, a harvester, a pruner, a desucker and a defoliator.

60. The method of any one of claims 52 to 59 wherein the method comprises reference data, the reference data specifying the desired position of the implement relative to the horizontal and / or vertical face of the canopy.

61. The method of any one of claims 52 to 61 wherein the method comprises receiving, from the implement, a signal determining the correct functioning of the implement.

62. The method of claim 61 wherein the received signal can stop / vary operation of the implement.

63. The method of any one of claims 52 to 53 wherein the implement is attached to an automated vehicle.

64. The method of claim 63 wherein the automated vehicle is configured to automatically move along the rows of the vineyard or orchard.

65. The method of any one of claims 52 to 64 comprising determining a required position of the implement by considering the expected or actual movement of the vehicle relative to the position of the sensor when the point cloud data was obtained.

66. The method of any one of claims 52 to 65 wherein the implement comprises a position sensor (GPS), optionally the GPS sensor confirms the expected location of the implement and the canopy.

67. The method of any one of claims 52 to 66 wherein the implement comprises an image detector, the image detector configured to confirm a location of the canopy.

68. The method of any one of claims 52 to 67 wherein the canopy face has a minimum spacing from a row center.

69. The method of claim 68 wherein the canopy face is detected as below the minimum width the implement is withdrawn from the row.

70. The method of any one of claims 52 to 69 wherein the implement is mounted on a towed vehicle.

71. The method of claim 70 wherein the sensor is mounted on the towing vehicle.

72. The method of any one of claim 70 to 71 wherein the implement is mounted on a trailer.

73. The method of claim 72 wherein the vehicle and or trailer have a GPS tracker.

74. A persistent map of an orchard or vineyard comprising a plurality of rows of trees, shrubs or vines, the persistent map comprising, for each of the rows,A left hand side canopy surface;A right hand side canopy surface; andA canopy centre.

75. The method as claimed claim 74 wherein the persistent map further comprising a top canopy surface.

76. A system for an orchard or vineyard comprising a plurality of rows of trees, shrubs or vines, the system comprising:A vehicle associate with a spatial sensor;An implement moveably attached to the vehicle;A controller configured to:Receive the spatial information from the sensor;Determine a canopy face of one or more of the rows;Control the implement to track the canopy face.

77. The system as claimed claim 76 wherein the accuracy of determining the canopy face is controllable, such that the implement can be configured to closely or crudely follow the canopy face.

78. The system as claimed in any one of claims 76 to 77 wherein the sensor data and the implement are positioned relative to a frame of reference of the vehicle.

79. The system as claimed in any one of claims 76 to 77 wherein the position of the implement relative to the canopy is configured based on the vehicle movement and / or expected movement.

80. The system as claimed in any one of claims 76 to 79 wherein the vehicle is an autonomous vehicle configured to move along each of the rows of the orchard.

81. The system as claimed in any one of claims 76 to 80 wherein the controller is configured to implement any one or more of the methods claimed in claims 1 to 75.

82. A method of detecting a canopy in an orchard or vineyard comprising a plurality of rows of plants, the method comprising:Obtaining spatial data from a sensor, the spatial data representing at least portion of one row of plants;Determining a subset of the spatial data, the subset comprising at least a portion of an individual plant in the row of plants;Determining one or more portions of each of the individual plants based on the determined subset of spatial data.

83. The method as claimed in claim 82 wherein the one or more portions comprise a canopy or a cordon of the plant.

84. The method as claimed in claim 83 comprising any one or more of the steps of claims 1 to 75.

Images