Method for controlling an automatic harvesting device for 5 plant elements of interest

Abstract

Method for controlling an automatic harvesting device for plant elements of interest. The present invention relates to a method for controlling a device (10) for the automatic harvesting of plant elements of interest, the method comprising, over time, the following steps: the detection and localization of plant elements of interest in the harvesting area based on an image of the harvesting area and an associated depth map; the detection and localization of obstacles for harvesting the detected plant elements of interest based on the depth map and plant element data; the determination of a harvesting order and a harvesting angle for the detected plant elements based on the plant element data and obstacle data; and the determination of a trajectory for harvesting the plant elements by the harvesting device based on the determined harvesting order and harvesting angles. Figure for the abstract: 1

Description

Title of the invention: Method for controlling an automatic harvesting device for plant elements of interest

[0001] The present invention relates to a method for controlling an automatic harvesting device for plant elements of interest. The present invention also relates to an associated computer program and an associated assembly.

[0002] The harvesting of plant elements was formerly done by hand, but is increasingly mechanized, which allows for a gain in time and productivity.

[0003] Existing mechanical tools are not always precise, however, particularly when harvesting small plant elements such as cherry tomatoes. It is therefore common for such a plant element or one of its neighbors to be damaged, or for the harvesting attempt to fail. Thus, the intervention of an operator is generally required.

[0004] There is therefore a need for a means of improving the harvesting of plant elements by means of a harvesting device, particularly in the case of small plant elements, such as cherry tomatoes.

[0005] To this end, the present description relates to a method for controlling an automatic harvesting device for plant elements of interest, such as tomatoes, the method being implemented by computer and comprising over time the following steps: a. obtaining an image of a harvesting area, called an area image, and an associated depth map, the area image and the depth map having been acquired by an imager of the harvesting device, b. the detection and localization of plant elements of interest in the harvesting area based on the area image and depth map obtained, the detection and localization data obtained being referred to as plant element data, c. the detection and localization of obstacles for the harvesting of plant elements of interest detected based on the depth map and plant element data, the detection and localization data obtained being called obstacle data, d. determining a harvesting order for the detected plant elements based on plant element and obstacle data, e. determining a harvesting angle for each plant element to be harvested based on plant element and obstacle data, and f. determining a trajectory for harvesting plant elements by the harvesting device as a function of the harvesting order and harvesting angles determined.

[0006] According to particular embodiments, the process comprises one or more of the following features, taken individually or in all technically possible combinations:

[0007] - the method includes controlling the harvesting device according to the tra determined jectory;

[0008] - the step of detecting and locating plant elements includes the detection, by a detection model, of shapes assimilated to plant elements on the area image, the detection model having been previously trained to detect plant elements on images according to a database of labeled images;

[0009] - the step of detecting and locating plant elements includes filtering the shapes detected according to at least one filtering criterion, the or at least one filtering criterion stipulating that the retained shapes have a hue greater than or equal to a predetermined hue;

[0010] - the step of detecting and locating plant elements includes the determination calculation of the dimensions of the shapes obtained and the location of the shapes obtained for the plant elements according to the depth map associated with the area image, the dimensions and location obtained being data of plant elements;

[0011] - the step of detecting and locating plant elements includes filtering the shapes obtained having dimensions outside a predetermined size range and / or whose distance from the harvesting device is greater than a predetermined distance;

[0012] - when the plant elements of interest include a stem, the detection step and localization of plant elements includes the determination, from the area image, of the stem orientation of the detected plant elements, the possible orientation determined for the stem being a plant element data;

[0013] - the obstacle detection and localization step includes:

[0014] - the preprocessing of the depth map to remove vegetation elements detected, and

[0015] - the extraction of the main lines from the pre-processed depth map as a function of a line identification model, the extracted main lines being considered as obstacles, the line identification model preferably implementing a RANSAC algorithm;

[0016] - the extraction of the main lines from the pre-processed depth map includes:

[0017] - obtaining a point cloud corresponding to the pre-processed depth map,

[0018] - processing the point cloud so as to retain only the points in border of elements,

[0019] - the application of the line identification model so as to identify the sets of points corresponding to the same line, and

[0020] - for each identified set of points, the grouping of points located next to each other from others at a distance less than a predetermined distance, the points of each grouping allow us to obtain a position and an orientation for the obstacle corresponding to a grouping;

[0021] - a confidence score is associated with each detected obstacle, the detection step and obstacle localization including:

[0022] - the comparison of the position of the detected obstacles between the current area image and at least one previous area image,

[0023] - the merging of obstacles appearing both on the current area image and on the image from the previous zone and whose displacement between the two images is less than a predetermined distance,

[0024] the confidence score of each obstacle detected for the current area image being updated according to the comparison and any fusion performed;

[0025] - the obstacles include at least one plant structure, such as a plant of tomatoes, supporting the plant elements of interest, the step of determining a harvest order including the determination of an order of approach of the plant structures for the harvesting of the plant elements of interest, the order of approach being, preferably, determined according to at least one of the following characteristics: a. the number of plant elements in each plant structure that have a shade greater than a predetermined shade, b. The distance between each plant structure is the harvesting device, and c. the difficulty of accessing the plant structure;

[0026] - the step of determining a harvest order includes determining a harvesting order of plant elements on each plant structure once the approach order has been determined for the plant structures, the harvesting order being determined according to the alignment of the plant elements on the plant structures;

[0027] - the step of determining a harvesting angle for each plant element of interest includes determining the harvesting angle by applying a set of restrictive rules from among the set of possible harvesting angles, the restrictive rules including one or more of the following rules:

[0028] - maintaining the gripper of the harvesting device at a distance greater than a predetermined distance from neighboring plant elements of interest to the plant element of interest under consideration,

[0029] - the priority choice of lateral orientations in relation to a possible structure plant supporting the plant element of interest in question,

[0030] - the avoidance of the stem of the plant element of interest when the stem has been pre- detected,

[0031] - avoidance of obstacles detected at a distance less than a predetermined distance completed from the plant element of interest in question, and

[0032] - the choice of the angle that minimizes the movement of the harvesting device.

[0033] This description also relates to a computer program product comprising program instructions recorded on a computer-readable medium, for the execution of a process as described above when the computer program is executed on a computer.

[0034] This description also relates to a readable information medium on which a computer program product as previously described is stored.

[0035] This description also relates to an assembly comprising: a. an automatic harvesting device for plant material of interest, such as tomatoes, comprising: i. an imager capable of acquiring over time an image of a harvesting area, called an area image, and an associated depth map, ii. at least one gripper for harvesting plant elements of interest, preferably the gripper being a suction gripper, iii. at least one movable element capable of moving the gripper, b. a controller configured to control the harvesting device as described above.

[0036] Other features and advantages of the invention will become apparent from the following description of embodiments of the invention, given by way of example only and with reference to the drawings which are:

[0037] [Fig. 1], [Fig. 1], a schematic view of an example of an automatic harvesting device for plant elements of interest and a controller configured to operate such a device, and

[0038] [Fig.2], [Fig.2], an organizational chart of an example of the implementation of a control method for an automatic harvesting device for plant elements of interest.

[0039] An automatic harvesting device 10 for plant elements of interest is illustrated in [Fig. 1]. Such a device 10 is controlled by a controller 12.

[0040] The plant elements of interest are typically fruits or vegetables. Preferably, the plant elements of interest have a substantially spherical shape. The plant elements of interest are, for example, tomatoes, or even cherry tomatoes.

[0041] The device 10 is configured to automatically harvest plant material of interest upon receiving commands from the controller 12. The Device 10 is a harvesting robot.

[0042] The device 10 includes an imager 20 and a gripper 22. Preferably, as illustrated by [Fig.1], the device 10 includes a support 24 on which the imager 20 and the gripper 22 are mounted, and a platform 26 on which the support 24 is mounted. The device 10 also includes a tracking element 28 mounted on the support 24 or the platform 26.

[0043] The imager 20 comprises at least one camera. Each camera is a stereo camera designed to acquire an image of a scene (intensity map) and a depth map associated with the acquired image. The depth map thus provides distance information for the elements of the scene relative to the camera.

[0044] The camera or cameras is preferably a color camera, such as an RGBD camera (“Red Green Blue Depth”).

[0045] In one embodiment, the imager 20 comprises at least two cameras, namely:

[0046] - a first camera positioned on the support 24 (at the top of the support 24) for the pre-collection of information, and

[0047] - a second camera closer to the gripper 22 for collecting information more precise.

[0048] The gripper 22 is an element configured to harvest the plant elements of interest.

[0049] In one embodiment, the gripper 22 is a suction unit configured to collect plant elements of interest by suction. In particular, once positioned in front of the plant element of interest to be harvested, the gripper 22 is configured to perform a linear forward-reverse movement controlled by the support 24, enabling it to suction the plant element of interest and then return to its original position. In one example, the gripper 22 is also configured to perform a rotational movement to facilitate the detachment of the plant element of interest.

[0050] In one variant, the gripper 22 is a clamp or any other grasping member.

[0051] The support 24 includes at least one movable part 30 on which the gripper 22 is mounted, allowing the gripper 22 to move. Optionally, the imager 20 or at least one camera of the imager 20 is also mounted on the movable part. The movable part 30 is, for example, motorized. The movable part 30 is, for example, an instrumented arm.

[0052] The platform 26 is suitable for moving the support 24, and thereby the imager 20 and the gripper 22.

[0053] Platform 26 is, for example, a motorized mobile trolley.

[0054] The tracking element 28 is configured to track the movements of the harvesting device 10, i.e., both the movements of the platform 26 and those of the moving part 30 of the support 24. Thus, the tracking element 28 makes it possible to track the movements of the gripper 22. The tracking element 28 includes, for example, a computer for performing the tracking based on information relating to a joint coding of the device 10 and information relating to the dimensioning of the gripper 22. Preferably, the tracking element 28 also includes one or more motion sensors fixed on the harvesting device 10, and whose measurements are used by the computer of the tracking element 28 to perform the tracking.

[0055] The controller 12 is configured to control the device 10, and in particular to control the movements of the device 10. In the example described above, the controller 12 is configured to control the movements of the platform 26 (allowing for rough positioning of the gripper 22 relative to the plant element of interest), as well as the movements of the support 24 (allowing for fine positioning of the gripper 22 relative to the plant element of interest). The controller 12 is, for example, connected by wire or wireless (Wi-Fi) to the imager 20 so as to receive the data acquired by the imager 20, and to the tracking element 28 so as to receive the position of the harvesting device 10.

[0056] In one embodiment, the controller 12 is integrated into the harvesting device 10 (i.e. is positioned on the platform 26), thus forming an all-in-one system.

[0057] In one embodiment, the controller 12 is a computer interacting with a computer program product.

[0058] The controller 12 typically includes a processor comprising a data processing unit, memories and an information support reader.

[0059] The computer program product includes an information carrier.

[0060] The information storage medium is a medium readable by the controller 12, usually by the data processing unit. The readable information storage medium is a medium suitable for storing electronic instructions and capable of being connected to a bus of a computer system. For example, the information storage medium is a USB flash drive, an optical disc, a CD-ROM, a magneto-optical disc, a ROM, a RAM, an EPROM, an EEPROM, a magnetic card, an optical card, or a punched card.

[0061] The computer program comprising program instructions is stored on the information medium.

[0062] The computer program can be loaded onto the data processing unit of the controller 12 and is adapted to drive the implementation of a control method for the harvesting device 10.

[0063] In another embodiment, at least part of the functions of the controller 12 is implemented in the form of printed circuits or integrated circuits, such as FPGAs (Field Programmable Gate Arrays).

[0064] A method for controlling the harvesting device 10 will now be described with reference to [Fig. 2], which illustrates a flowchart of the process. Such a method is implemented by the controller 12, that is to say, it is implemented by computer.

[0065] The steps of the process are repeated over time, and advantageously in real time, so as to generate the trajectory determined for the harvesting of plant elements of interest by the harvesting device 10, and to allow the harvesting of plant elements of interest in different harvesting areas.

[0066] The control method includes a step 100 of obtaining an image of a harvesting area, called the IMZ area image, and an associated depth map D.

[0067] The IMZ zone image and the depth map D were acquired by the imager 20 of the harvesting device 10.

[0068] The control method includes a step 110 of detecting and locating plant elements of interest in the harvesting zone based on the IMZ zone image and the depth map D obtained. The detection and location data obtained are called plant element data.

[0069] Note that the location information in the plant element data has been converted from the imager 20 reference frame to correspond to the terrestrial reference frame, which corresponds to the starting point of the harvesting device 10 (gripper 22 on support 24 and platform 26).

[0070] In an example implementation, step 110 includes the detection, by a detection model, of shapes similar to plant elements on the IMZ zone image. The shapes are, for example, boxes encompassing the elements similar to plant elements.

[0071] The detection model has been previously trained to detect plant elements in images based on a database of labeled images. The detection model is, for example, a neural network, such as the YOLO neural network or Faster RCNN. Such a detection model has, for example, been obtained following training, testing, and validation steps on a set of approximately 1000 labeled images (including or excluding the plant elements of interest).

[0072] Preferably, at the output of the detection model, shapes considered inconsistent with the plant elements of interest are eliminated by the controller 12. A shape is considered inconsistent, for example, when the shape is too small compared to a standard size of the plant elements of interest, or non-compliant (for example, in the in the case of tomatoes, a very stretched rectangle: the identified object is probably not a tomato), or associated with a confidence score that is too low (when a confidence score is assigned to detections).

[0073] Preferably, step 110 includes filtering the detected shapes according to at least one filtering criterion.

[0074] At least one filtering criterion stipulates that the retained shapes have a hue greater than or equal to a predetermined hue.

[0075] In this case, the hue of the detected shapes is evaluated by converting the RGB pixels of an area at the center of the detected shape to the HSV color space (Hue, Saturation, Brightness) and analyzing the H (hue) channel. For the example of tomatoes, this allows the ripeness of the tomatoes to be taken into account, and only those deemed sufficiently ripe to be harvested.

[0076] Preferably, step 110 includes determining the dimensions of the shapes obtained and locating the shapes obtained for the plant elements based on the depth map D associated with the IMZ zone image. The dimensions and location obtained form part of the plant element data.

[0077] In one example, the determination is carried out by analyzing the points of the depth map D located in a sub-region of the shape. Assuming a spherical object (including in the case of "elongated" tomatoes), this analysis gives the physical size of the plant element (the radius of the sphere) and the 3D position of its center in the frame of the imager 20.

[0078] Preferably, step 110 includes filtering out the obtained shapes whose dimensions fall outside a predetermined size range (shapes that are too small or too large) and / or whose distance from the harvesting device 10 (in particular from the imager 20) exceeds a predetermined distance (shapes that are too far away). The predetermined distance is preferably such that the gripper 22 cannot physically reach the plant element of interest identified by its shape. The shape is therefore located beyond a possible harvesting perimeter for the gripper 22.

[0079] Preferably, when the plant elements of interest include a stem, step 110 includes determining, from the IMZ area image, the stem orientation of the detected plant elements. The orientation determined for the stem, if any, forms part of the plant element data.

[0080] In the case of tomatoes, the stem is a peduncle, the orientation of the stem being indicative of the orientation of the tomato.

[0081] In one embodiment, the determination of the rod and its orientation is performed by a keypoint detection model. The detection model Key points include, for example, a neural network, such as a CNN Hourglass neural network. Such a detection model has, for example, been previously trained to detect the stems of plant elements of interest in images based on a database of labeled images. This detection model has, for example, been obtained following training, testing, and validation steps on a set of approximately 1000 labeled images (including or excluding the plant elements of interest and their stems).

[0082] When the stem is successfully detected, it is located in the 3D coordinate system of the imager 20, assuming it is a spherical object. Conversely, if it could not be detected, it is assumed to be invisible to the imager 20, and a "default" orientation for the plant element of interest is used instead.

[0083] The control method includes a step 120 of detecting and locating obstacles for harvesting the plant elements of interest detected according to the depth map D and the plant element data. The detection and location data obtained are called obstacle data.

[0084] The following describes, for each IMZ zone image, an example of the implementation of step 120 of obstacle detection. This provides the existing CnA collection of obstacles detected for the previous zone image (empty if it is the first image).

[0085] In this example, step 120 includes preprocessing the depth map D to remove detected plant elements and extract the area of ​​interest.

[0086] Step 120 then involves extracting the principal lines from the pre-processed depth map D, these principal lines being considered as obstacles. For example, these lines are plants from which the plant elements of interest originate, such as tomato plants.

[0087] Preferably, the extraction is carried out by obtaining a point cloud corresponding to the pre-processed depth map D.

[0088] The point cloud is then processed so as to retain only the points at the edges of elements. In particular, points that form flat areas, such as leaves, are not taken into account. Plants are thus overrepresented among the points in the cloud, which facilitates the application of the RANSAC algorithm (short for RANdom SAmple Consensus), as described below.

[0089] Preferably, the point cloud is then voxelized so as to reduce the total number of points and to retain only one point in each cell of a 3D grid, the size of the cells on side dc being parameterizable. This preprocessing has the effect of regularizing the point cloud, reducing the number of points and significantly accelerating the execution of the RANSAC algorithm.

[0090] Then, a line identification model is applied so as to identify the sets of points corresponding to the same line.

[0091] Preferably, the line identification model implements a RANSAC algorithm. Thus, the RANSAC algorithm is applied, using a line model, with a threshold distance dr (for example, less than or equal to 0.05 meters) and a maximum number of iterations nr (for example, greater than or equal to 10,000). Once a line is detected, the associated points are removed from the point cloud and the RANSAC algorithm is run again. The operation is performed a certain number of times (for example, greater than or equal to 5).

[0092] For each identified set of points, points located less than a predetermined distance dcl from each other (for example, less than or equal to 0.1 meters) are grouped (clustered). This prevents distant points from being grouped together in the same line. The points in the same group form an oriented shape (bounding box) that can be considered an obstacle. In particular, the shape is obtained, for example, by principal component analysis.

[0093] Preferably, a confidence score is assigned to each detected obstacle. The confidence score has a maximum value (e.g., Pæ) equal to 0.8. Step 120 involves comparing the position of the detected obstacles between the current area image and at least one previous area image. Obstacles appearing in both the current and previous area images, and whose displacement between the two images is less than a predetermined distance, are merged. The confidence score of each detected obstacle for the current area image is then updated based on the comparison and any merging performed.

[0094] In an example implementation, for each obstacle a of the collection Gi-i, newly added obstacles o! (obtained for the current image) are compared and pairs of obstacles whose distance is less than a predetermined distance dft|^ (for example less than or equal to 0.1 meters) are merged, using the Hungarian algorithm.

[0095] The new pose is then calculated using linear interpolation between the previous pose and the new pose. The confidence level of the obstacles in collection x is updated according to one of the following formulas:

[0096] Where corresponds to the number of consecutive images during which the obstacle has has been detected and merged. If there is no candidate for merging for , then it is added to the collection i which then becomes the collection .. Obstacles in the collection whose confidence has become less than a confidence score v?™ (Pæ" example equal to 0.3) are removed.

[0097] Preferably, after detection steps 110 and 120, all the plant element data from the current time n are merged with those from the previous time n-1 using a tracking algorithm (Kalmann) and stored in a database. For each plant element, its 3D position, size, color, and the number of times it has been seen are thus stored. Mapping allows the retrieval of information on plant elements in a specific region and also in the form of clusters via clustering techniques (DBscan). Obstacle data are also stored in a database.

[0098] The control method includes a step 130 of determining a harvest order of the detected plant elements based on plant element data and obstacle data.

[0099] In one embodiment, the obstacles include at least one plant structure, such as a tomato plant, supporting the plant elements of interest. Step 130 involves determining an order in which the plant structures should be approached for harvesting the plant elements of interest.

[0100] The order of approach is preferably determined based on at least one of the following characteristics:

[0101] - the number of plant elements in each plant structure that have a su tint above a predetermined shade. The predetermined shade indicates that the plant element of interest is ready to be harvested (sufficiently ripe).

[0102] - the distance between each plant structure and the harvesting device 10, and

[0103] - the difficulty of accessing the plant structure (row depth, density) obstacles).

[0104] In particular, in the case of tomatoes, information (positions and colors) of the tomato clusters in the robot arm's workspace is obtained. The tomatoes to be harvested are identified by filtering their color. The harvesting order per cluster is then determined, for example, based on the aforementioned characteristics.

[0105] In one example of implementation, step 130 includes determining a harvesting order for plant elements on each plant structure once the approach order has been determined for the plant structures.

[0106] Preferably, the harvest order is determined according to the alignment of plant elements on plant structures.

[0107] For example, the harvest order of intra-cluster plant elements (tomatoes) is determined according to the shape of the bunch. For this reason, the bunches are treated differently depending on whether:

[0108] - (i) the plant elements are aligned vertically in a single column,

[0109] - (ii) the plant elements form a cluster of two vertical columns, and

[0110] - (iii) the plant elements are organized according to any other arrangement (clusters forked, heaps...).

[0111] If the plant elements are aligned in a vertical column, the harvest order is defined from top to bottom (among the mature plant elements).

[0112] If the plant elements form a cluster of two vertical columns: the main plane of the cluster is extracted using singular value decomposition (SVD). Within this plane, the harvesting order is defined using a heuristic whose logic is a column-by-column approach and top-down harvesting within each column.

[0113] In any other situation, the principal plane of the cluster of plant elements is extracted using singular value decomposition (SVD). Within this plane, the plant elements are grouped into virtual columns of width equal to the characteristic size of these elements. The harvesting order is then defined as follows: column by column starting from one end, then from top to bottom within each column.

[0114] In summary, the general idea is to proceed column by column, starting with an outermost column if there are more than two, and working from top to bottom within each column. If the observed arrangement does not naturally occur in one or two columns, then the elements are grouped into vertical columns located in the principal plane of the cluster / cluster, the width of the columns being equal to the characteristic size. The classification into (i) one column, (ii) two columns, or (iii) another situation is based on geometric criteria derived from SVD.

[0115] The control method includes a step 140 of determining a harvesting angle for each plant element to be harvested (part of a plant structure counted in the order of approach) based on plant element data and obstacle data.

[0116] In an example implementation, the harvesting angle is determined by applying a set of restrictive rules from among the set of possible harvesting angles. The restrictive rules include, for example, one or more of the following rules:

[0117] - i) maintaining the gripper 22 of the harvesting device 10 at a greater distance at a predetermined distance from neighboring plant elements of interest. This avoids collisions that would repel the target plant element, and also prevents harvesting an untargeted plant element.

[0118] - ii) the priority choice of lateral orientations over a possible The plant structure supporting the plant element of interest. This prevents interference from the grape vine stem.

[0119] - iii) avoidance of the stem of the plant element of interest when the stem has been pre- detected. Indeed, in the case of harvesting tomatoes by suction, the harvest does not work well if the gripper 22 approaches directly by the peduncle (peduncle to be avoided).

[0120] - iv) avoidance of obstacles detected at a distance less than a predetermined distance The end of the plant element of interest in question. This prevents collision with a plant structure (tomato plant) or another plant element (another tomato).

[0121] - v) the choice, as a priority, of the angle minimizing the displacements of the device harvest 10. For this, the position of the harvesting device 10 is taken into account and the angle is determined for example by taking the angle closest to the perpendicular to the harvest row among the intersection of all the previous criteria.

[0122] In one variant, instead of i) and ii), all possible orientations that have a margin of 100° with the nearby elements of interest are determined. If this represents more than 15% of the possible orientations, the targeted plant element (tomatoes) is considered to be located at one end of the cluster, and these orientations are used in the intersection of v) in place of those of i) and ii).

[0123] The control method includes a step 150 of determining a trajectory for the harvesting of plant elements by the harvesting device 10 according to the harvesting order and the determined harvesting angles.

[0124] In particular, the starting and ending points are determined according to the position of the harvesting device 10 and the determined harvesting orders and angles (e.g., Origin position towards Cluster, Cluster towards Tomato, Tomato towards Tomatoes2 ...).

[0125] Preferably, the algorithm used operates by random sampling with optimization in the joint space (e.g., ABITstar). The parameter common to the different possible algorithms is the planning time. Plant elements other than the target are considered obstacles. Plant structures (and any other obstacle, the robot itself, hard obstacles, etc.) detected using the "obstacle detection" block are also taken into account for trajectory planning in order to avoid collisions and ensure successful harvesting.

[0126] The control method includes a step 160 of controlling the harvesting device 10 according to the determined trajectory.

[0127] In particular, during the control step, the controller 12 commands the movement of the platform 26, then of the support 24, so that the gripper 22 is in a position allowing the harvesting of the plant element in question (typically in face of this element). The gripper 22 then harvests this plant element of interest. The procedure is repeated for the other plant elements of interest in the determined order.

[0128] Thus, the present method enables the control of a harvesting device 10 for the automatic harvesting of plant elements. In particular, the plant element detection step and the obstacle detection step allow for a better determination of the trajectories to be followed by the harvesting device 10. Indeed, the trajectories are calculated specifically to avoid detected obstacles, thereby reducing the risk of damage to the plant element or one of its neighbors, or of the harvest failing. Such a method is particularly well-suited for harvesting small plant elements, such as cherry tomatoes.

[0129] A person skilled in the art will understand that the embodiments and variants described above can be combined to form new embodiments provided that they are technically compatible.

Claims

1. Claims Method for controlling a device (10) for automatically harvesting plant elements of interest, such as tomatoes, the method being implemented by computer and comprising over time the following steps: a. obtaining an image of a harvesting area, called a zone image (IMZ), and an associated depth map (D), the zone image (IMZ) and the depth map (D) having been acquired by an imager (20) of the harvesting device (10), b. the detection and localization of plant elements of interest in the harvesting area based on the obtained area image (IMZ) and depth map (D), the obtained detection and localization data being called plant element data, c. detecting and localizing obstacles for harvesting the detected plant elements of interest based on the depth map (D) and the plant element data, the obtained detection and localization data being called obstacle data, the obstacle detection and localization comprising: i. preprocessing the depth map (D) to remove detected plant elements, and ii. extracting the main lines from the preprocessed depth map (D) based on a line identification model, the extracted main lines being considered to be obstacles, d. determining a harvesting order for the detected plant elements based on the plant element data and obstacle data, e. determining a harvesting angle for each plant element to be harvested based on plant element data and obstacle data, and f. determining a trajectory for harvesting the plant elements by the harvesting device (10) based on the harvesting order and the harvesting angles determined.

2. The method of claim 1, wherein the method comprises controlling the harvesting device (10) based on the determined trajectory.

3. Method according to claim 1 or 2, in which the step of detecting and locating plant elements comprises the detection, by a detection model, of shapes assimilated to plant elements on the zone image (IMZ), the detection model having been previously trained to detect the plant elements on images according to a database of labeled images.

4. The method of claim 3, wherein the step of detecting and locating plant elements comprises filtering the detected shapes according to at least one filtering criterion, the at least one filtering criterion stipulating that the preserved shapes have a hue greater than or equal to a predetermined hue.

5. Method according to claim 3 or 4, in which the step of detecting and locating plant elements comprises determining the dimensions of the shapes obtained and locating the shapes obtained for the plant elements according to the depth map (D) associated with the zone image (IMZ), the dimensions and the location obtained being plant element data.

6. A method according to claim 5, wherein the step of detecting and locating plant elements comprises filtering out the obtained shapes having dimensions outside a predetermined dimension range and / or whose distance from the harvesting device (10) is greater than a predetermined distance.

7. Method according to any one of claims 1 to 6, wherein when the plant elements of interest comprise a stem, the step of detecting and locating plant elements comprises determining, from the zone image (IMZ), the orientation of the stem of the detected plant elements, the possible orientation determined for the stem being a piece of plant element data.

8. A method according to any one of claims 1 to 7, wherein the line identification model implements a RANSAC algorithm.

9. Method according to any one of claims 1 to 8, in which the extraction of the main lines from the preprocessed depth map (D) comprises: - obtaining a point cloud corresponding to the depth map pre-processed depth (D), - processing the point cloud so as to keep only the points on the edge of elements, - applying the line identification model so as to identify the sets of points corresponding to the same line, and - for each set of points identified, grouping the points located from each other at a distance less than a predetermined distance, the points of each grouping make it possible to obtain a position and an orientation for the obstacle corresponding to a grouping.

10. Method according to any one of claims 1 to 9, in which a confidence score is associated with each obstacle detected, the step of detecting and locating obstacles comprising: - comparing the position of the obstacles detected between the current zone image and at least one previous zone image, - merging the obstacles appearing on both the current zone image and the previous zone image and whose displacement between the two images is less than a predetermined distance, the confidence score of each obstacle detected for the current zone image being updated as a function of the comparison and the possible merging carried out.

11. A method according to any one of claims 1 to 10, wherein the obstacles comprise at least one plant structure, such as a tomato plant, supporting the plant elements of interest, the step of determining a harvesting order comprising determining an order of approach of the plant structures for harvesting the plant elements of interest, the order of approach preferably being determined as a function of at least one of the following characteristics: a. the number of plant elements of each plant structure that have a shade greater than a predetermined shade, b. the distance between each plant structure and the harvesting device (10), and c. the difficulty of accessibility of the plant structure.

12. The method of claim 11, wherein the step of determining a harvesting order comprises determining an order of harvesting the plant elements on each plant structure after the order determined approach for plant structures, the harvesting order being determined based on the alignment of plant elements on the plant structures.

13. A method according to any one of claims 1 to 12, wherein the step of determining a harvesting angle for each plant element of interest comprises determining the harvesting angle by applying a set of restrictive rules from among the set of harvesting angle possibilities, the restrictive rules comprising one or more of the following rules: - maintaining the gripper (22) of the harvesting device (10) at a distance greater than a predetermined distance from the plant elements of interest neighboring the plant element of interest considered, - choosing as a priority the lateral orientations with respect to a possible plant structure supporting the plant element of interest considered, - avoiding the stem of the plant element of interest when the stem has been previously detected, - avoiding the obstacles detected at a distance less than a predetermined distance from the plant element of interest considered,and - the priority choice of the angle minimizing the movements of the harvesting device (10).,

14. A computer program product comprising program instructions recorded on a computer-readable medium, for executing a method according to any one of claims 1 to 13 when the computer program is executed on a computer.

15. Assembly comprising: a. a device (10) for automatically harvesting plant elements of interest, such as tomatoes, comprising: i. an imager (20) capable of acquiring over time an image of a harvesting zone, called a zone image (IMZ), and an associated depth map (D), ii. at least one gripper (22) for harvesting the plant elements of interest, preferably the gripper (22) being a suction gripper, iii. at least one movable element (30) capable of moving the gripper (22), b. a controller (12) configured to control the device harvesting (10) according to a method according to any one of 1 to 13.

Images