Image-based identification and classification of plants

Abstract

The invention relates to a computer-implemented method for the image-based identification and classification of at least one plant in an agricultural field. The invention also relates to a device for dispensing a liquid onto at least one part of an agricultural field on the basis of the identified and classified information about the plants, and to a computer-readable storage medium on which the computer-implemented method is encoded.

Description

[0001] BCS243033 Abroad

[0002] IMAGE-BASED RECOGNITION AND CLASSIFICATION OF PLANTS

[0003] The present invention relates to a computer-implemented method for image-based recognition and classification of at least one plant in an agricultural field. The invention also relates to a device for applying a liquid to at least a part of an agricultural field based on the recognized and classified plant information. A further aspect of the invention relates to a computer-readable storage medium on which the computer-implemented method is encoded.

[0004] In precision agriculture, it is crucial to accurately identify objects such as companion plants and crops in partially overlapping images captured while an agricultural machine is in motion. Current methods rely on deep learning-based image analysis techniques, which, while capable of recognizing objects with a certain probability, also have a certain error rate. Misclassification of objects can be attributed to various factors, such as unfavorable perspectives during image capture, uneven lighting (especially in the corners of the images), and the fact that machine learning processes, even with optimal image quality, always learn with a certain degree of error, which they then reproduce even in previously unseen images. Therefore, there is a need to reduce this error rate and improve the accuracy of object recognition.

[0005] The present invention addresses these problems.

[0006] A first object of the present invention is a computer-implemented method for image-based recognition and classification of at least one plant in an agricultural field comprising the steps:

[0007] -(a) Production of at least two partially overlapping images of the agricultural field, wherein at least one plant can be at least partially captured in each image,

[0008] -(b) Object tracking of at least one plant in each image capture,

[0009] -(c) Classification of the at least one plant in each of the at least two at least partially overlapping images by assigning the at least one plant to a plant species, whereby a probability value is generated in each case and each probability value corresponds to the confidence value of the assignment of the at least one plant to a plant species,

[0010] -(d) Use of the at least two probability values ​​to determine the species of the at least one plant.

[0011] Another object of the present invention is a device for applying a liquid to at least a part of an agricultural field, comprising a calculating and control unit, at least one reservoir for receiving the liquid, at least one die,

[0012] Means for requesting the fluidity from the at least one storage container towards the at least one die, at least one image acquisition unit that can be oriented towards the agricultural field, and is configured to produce at least two at least partially overlapping image captures, wherein the computing and control unit is configured (a) to cause the at least one image acquisition unit to produce at least two at least partially overlapping image captures of the agricultural field, wherein at least one plant can be at least partially captured on each image capture, wherein the computing and control unit is configured, (b) to perform object tracking of at least one plant on each image capture, wherein the computing and control unit is configured(c) to classify the at least one plant on each of the at least two at least partially overlapping images by assigning it to a plant species, generating a probability value in each case, each probability value corresponding to the confidence value of the assignment to a plant species, wherein the computing and control unit is configured, (d) to use the at least two probability values ​​from step (c) to determine the plant species of the at least one plant, wherein the computing and control unit is configured, (e) to activate the at least one die at least partially on the basis of the information from step (d) in order to apply the fluid.

[0013] Another object of the present invention is a non-volatile, computer-readable storage medium on which software instructions are stored which, when executed by a processor of a computer system, cause the computer system to perform the following steps:

[0014] -(a) Receiving at least two partially overlapping photographs of the agricultural field, wherein at least one plant is at least partially captured in each photograph,

[0015] -(b) Object tracking of at least one plant in each image capture,

[0016] -(c) Classification of the at least one plant in each of the at least two partially overlapping images by assigning it to a plant species, whereby a probability value is generated in each case and each probability value corresponds to the confidence value of the assignment to a plant species,

[0017] -(d) Use of the at least two probability values ​​to determine the species of the at least one plant.

[0018] The invention is explained in more detail below, without distinguishing between the subject matter of the invention (method, device, storage medium). Rather, the following explanations are intended to apply analogously to all subject matter of the invention, regardless of the context in which they are made (method, device, storage medium).

[0019] If the present description or the patent claims mention steps in a sequence, this does not necessarily mean that the invention is limited to that sequence. Rather, it is conceivable that the steps could also be carried out in a different sequence or even in parallel; unless one step builds upon another, which makes it imperative that the building step be carried out subsequently (which will be clear in the specific case). The sequences mentioned thus represent preferred embodiments of the invention.

[0020] The present invention relates to a computer-implemented method for the image-based recognition and classification of at least one plant in an agricultural field. The term "plant" refers to living organisms belonging to the flora category and capable of photosynthesis. This includes a wide variety of species, including cultivated plants and companion plants, which either occur naturally in a particular ecosystem or are specifically cultivated to meet human needs.

[0021] The term "cultivated plant" refers to a plant that is intentionally grown as a useful or ornamental plant through human intervention.

[0022] The term "companion plants" (often also referred to as weeds) refers to plants of the spontaneous accompanying vegetation (segetal flora) in cultivated plant stands, meadows or gardens, which are not specifically cultivated there and develop, for example, from the seed potential of the soil or via windfall.

[0023] The term "plant species" is defined as a group of plants that are characterized by shared morphological, genetic, and ecological traits and can interbreed under natural conditions to produce fertile offspring. In one example, the term refers to a classification as a cultivated plant or a companion plant. In another example, the term refers to a classification as a useful plant or an ornamental plant. In yet another example, the term encompasses the classification as a useful plant within a specific category, such as vegetables, cereals, field crops, fruits, etc. In another example, the term encompasses the classification as a specific type of cultivated plant, such as wheat, corn, soybeans, tomatoes, potatoes, etc. In yet another example, the term also encompasses the classification of a companion plant within a specific species, such as field foxglove, field bindweed, couch grass, goosefoot, cleavers, etc.

[0024] The term "agricultural field" refers to a spatially definable area of ​​the earth's surface that is used for agricultural purposes, in which crops are planted, possibly supplied with nutrients and harvested.

[0025] The term "at least partially overlapping image acquisitions" means that the successive images in an image sequence share at least some areas or features. This can mean that part of the observed object (i.e., at least one plant) overlaps between successive images, or that some visual features or structures are repeated in the images (preferably with at least part of the observed object being repeated). Technically, this means that the image areas representing the object being tracked are not completely isolated from one another, but rather that there is some overlap or continuity between the images. This allows the tracking of the object over time and ensures the consistency of visual features or structures between successive images.

[0026] In the computer-implemented method, in a first step (a) at least two at least partially overlapping image images of the agricultural field are generated, whereby at least one plant can be at least partially captured in each image. In one example, the at least one plant can be substantially completely captured in each image. In another example, the image capture is performed by a camera. The camera is, for example, attached to an agricultural device, such as a sprayer, and pointed towards the ground, and the agricultural device moves across the agricultural field. When capturing at least partially overlapping images with an agricultural device, an appropriate frame rate must be selected to ensure sufficient image overlap.In one example, images of an agricultural field are continuously captured by an agricultural device moving across it. The speed of movement of the agricultural device is preferably essentially constant and appropriate to ensure uniform overlap and avoid motion blur. For example, a desired overlap rate (e.g., "frame-to-frame overlap") of the at least one plant in the at least partially overlapping images of the agricultural field can be assumed in order to calculate the frame rate and speed of the agricultural device accordingly. Preferably, the time required for image analysis (e.g., object tracking and classification, etc.) is included in the overall calculation, as this affects the effective frame rate and speed.which can affect the speed of the agricultural device.

[0027] The camera used according to the invention can include an image capture sensor and optical elements. The image capture sensor is a device for capturing two-dimensional images of light electrically. These are usually semiconductor-based image capture sensors such as CCD (CCD = charge-coupled device) or CMOS sensors (CMOS = complementary metal-oxide-semiconductor). The optical elements (lenses, apertures, and the like) serve to create the sharpest possible image of an object on the image capture sensor. The image capture formats of the at least one image capture can be, for example, rectangular or square. Common rectangular image capture formats are, for example, 3:2, 4:3, or 16:9.

[0028] In a further step (b) of the inventive procedure, object tracking of at least one plant in each image is performed. Various methods are used for object tracking to monitor the movement and identification of objects in images or videos. These include, among others, optical flow, image-based object tracking, multiview object matching, multi-object tracking, deep learning-based approaches, feature-based methods, probabilistic models such as the Kalman filter or the particle filter, and histogram-based methods. The selection of the appropriate method depends on various factors such as the type of objects to be tracked, the quality of the input data, and the specific requirements of the application.

[0029] Object tracking in the inventive method in step (b) is preferably performed using optical flow, image-based object tracking, multi-target sequence tracking, or multi-view object matching. In optical flow, the movement of all pixels in image captures is analyzed. This method analyzes the intensity changes of pixels and estimates how they move over time. This yields a dense motion field representation that provides information about the entire image scene. These displacements are used to precisely track the position and extent of objects and to update the bounding boxes and / or polygons around the objects accordingly.

[0030] Image-based object tracking does not calculate every pixel displacement. Instead, it focuses on the displacements of specific objects, i.e., at least one plant (for example, represented by bounding boxes / polygons). The Fast Fourier Transform (FFT) is used to determine the total displacement of the image content, and this information is used to adjust the position of the bounding boxes / polygons.

[0031] Specifically, image-based object tracking comprises the steps of image registration, object tracking, and validation. Image registration begins with registering the current image acquisition in comparison to the previous one. At least individual pixel shifts in the x and y directions are determined to quantify the offset between the two images, preferably using FFT. Based on these determined offsets, the positions of the boundary boxes / polygons that enclose the objects (i.e., the at least one plant) in the previous image are adjusted in the current image. This adjustment is achieved through simple mathematical transformations of the coordinates. A loss function is required to validate the accuracy of the estimated boundary boxes / polygons. This function effectively calculates the difference between the predicted coordinates of the boundary boxes / polygons and the actual coordinates (i.e., the actual position of the object).the real boundary boxes / polygons) quantified. Suitable loss functions for localization are: Mean Squared Error (MSE), Intersection over Union (IoU), and their combinations. The described image-based object tracking is particularly advantageous because, unlike other methods such as optical flow, it requires less computational capacity.

[0032] Multi-object tracking involves tracking multiple objects within an image sequence, monitoring and analyzing their movements over time. This requires algorithms capable of assigning objects between successive images, even when the objects overlap or their appearance changes.

[0033] Multiview object matching enables the matching of objects in different views or perspectives, for example, in images captured by multiple cameras. This process requires the precise mapping of features or properties between the different views to enable consistent object tracking across multiple viewpoints.

[0034] In one embodiment of the inventive method, the image-based object tracking in step (b) comprises the following steps: (bl) identification and object localization of at least one plant on the first of the at least two at least partially overlapping image acquisitions, and (b2) calculation of at least individual pixel shifts of the successive second image acquisition compared to the first image acquisition, and (b3) position determination of the at least one plant on the successive second image acquisition of the at least two at least partially overlapping image acquisitions by at least partially using the information of the at least one plant identified and object localized in step (bl) and the pixel shift calculated in step (b2). In one example, the pixel shift is calculated in step (b2) by applying the FFT.Updating the boundary box or polygon to include the detected object in step (b3) is typically based on mathematical methods of geometric transformation. This involves transformations such as translation (shifting), scaling (changing the size), and rotation, which are applied to the original coordinates of the boundary box or polygon to calculate the updated position and extent of the object. As described above, a suitable loss function for localization can be used to verify the accuracy of the transformed boundary box / polygon.

[0035] In a further embodiment of the inventive method, the multi-target sequence tracking in step (b) comprises the following steps: (b1) identification and object localization of at least one (preferably at least two) plant(s) on the first of the at least two at least partially overlapping image recordings and (b2) tracking the movement and determining the position of the at least one (preferably at least two) plant(s) on the successive second image recording of the at least two at least partially overlapping image recordings.

[0036] In one implementation, the identification and object localization of the at least one plant in the at least one image is carried out in step (bl) (and preferably also in steps (b2) and (b3)) by generating a boundary box and / or a polygon. A "boundary box" can be a rectangular box defined by the coordinates of its corners, enclosing an object of interest within an image. The boundary box is characterized by its position, typically specified by the coordinates of the upper left corner (xi, yi) and the lower right corner (x2, y2), or alternatively by the center coordinates (ex, cy), as well as its width (w) and height (h). The boundary box serves as a spatial representation that delineates the object's extent and enables its identification and / or analysis. Boundary boxes are frequently used to mark objects in images.It should be noted that the bounding box does not necessarily have to be rectangular; other geometric shapes are also suitable for marking objects, e.g., circles, ellipses, hexagons, or other forms. In this sense, the term "polygons" encompasses such geometric shapes. As an alternative or in addition to the bounding boxes / polygons, at least one plant in an image can also be highlighted with color. It is also conceivable that the marking can be achieved by displaying parts of an image that do not contain plant(s) in a different color or in grayscale and / or with reduced brightness and / or reduced contrast, so that the plant(s) stand out in relation to the other parts of the image. Marking can also mean adding a label or an indicator (e.g., an arrow) that points to a localized plant.

[0037] Image object identification and localization are crucial tasks in machine vision and computer vision. Image object identification refers to a system's ability to recognize and identify a specific object within an image. Various methods are employed for this purpose, including convolutional neural networks (CNNs) based on deep learning. These networks can extract features from the image and learn to recognize different objects by training on extensive training data. Other image object identification methods include the use of feature descriptors such as SIFT (Scale-Invariant Feature Transform) or SURF (Speeded Up Robust Features) in combination with machine learning algorithms.

[0038] Image object localization refers to a system's ability to determine the precise position of a detected object within an image. Various techniques can be used for this purpose, including bounding-box regression, which allows for the precise localization of the bounding boxes around detected objects, and the application of region-based convolutional neural networks (R-CNNs) and their variants, such as Fast R-CNN and Faster R-CNN. These techniques enable the accurate localization of objects within the image and the drawing of bounding boxes and / or polygons around them. In addition to these methods, techniques such as region-based fully convolutional networks (R-FCN), single-shot multibox detectors (SSDs), and You Only Look Once (YOLO) can also be used for simultaneous object detection and localization.

[0039] In one example, during object identification and localization of the at least one plant in step (b), a probability value P(A) is generated that corresponds to the confidence value of the object identification and localization of the at least one plant. This is preferably done for each of the overlapping image acquisitions. The confidence value is typically used in methods based on machine learning, especially in deep learning models for object identification and localization. For example, when using CNNs for object identification and localization, a confidence value is typically generated that indicates how certain the network is that it has correctly identified the detected object and how accurate the object's localization is.In R-CNNs and their variants, such as Fast R-CNN and Faster R-CNN, Region Proposal Networks (RPNs) are used to identify regions in the image that potentially contain objects. These methods also generate a confidence score for the identification and localization of the detected objects. Methods like Single Shot Multibox Detector (SSD) and You Only Look Once (YOLO) are designed to efficiently identify and locate objects in an image. These methods also generate confidence scores for the detected objects to indicate the reliability of the identification and localization. In all these methods, the confidence score is used to quantify the certainty and accuracy of object identification and localization.This makes it possible to assess the reliability of the detected objects and to take appropriate measures based on the confidence value, such as displaying warnings or initiating follow-up actions.

[0040] In step (c) of the invention-based procedure, a classification of the at least one plant on each of the at least two at least partially overlapping image recordings is carried out by assigning the at least one plant to a plant species, whereby a probability value is generated in each case and each probability value corresponds to the confidence value of the assignment of the at least one plant to a plant species.

[0041] Various methods can be used to classify at least one plant into a plant species, such as machine learning and artificial intelligence or image processing and analysis.

[0042] Machine learning and artificial intelligence are frequently used to identify plants based on characteristics such as leaf shape, color, size, and texture. Various algorithms are employed, including neural networks such as Transformer or Convolutional Neural Networks (CNNs), which are specifically designed for image recognition and classification. These neural networks can extract features at different levels of the image and recognize complex patterns, making them a powerful method for plant classification.

[0043] In image processing and analysis, specific plant characteristics are extracted and analyzed from the generated images. Techniques such as the Hough transform for shape recognition, edge detection for identifying leaf structures, and color segmentation for differentiating between various plant parts can be used. Furthermore, machine learning methods such as support vector machines (SVMs) can be employed to classify the extracted features and identify the plants.

[0044] Generating a confidence score for the classification is done by evaluating the reliability of the classification based on the extracted features and the trained models. This confidence score can be based on probabilities, confidence intervals, or other statistical measures to indicate how certain the classification results are.

[0045] In another embodiment of the inventive method, the calculations in step (c) and step (d) are performed for each plant identified and object-located in step (b) on the at least two image recordings that overlap at least partially.

[0046] In a further step (d), the at least two probability values ​​from step (c) are used to determine the plant species of the at least one plant. In other words, after step (c), at least two probability values ​​are available: one for classifying the at least one plant in the first of the two at least partially overlapping images, and at least one further probability value for classifying the at least one plant in the second of the two at least partially overlapping images.

[0047] For the determination in step (d) various methods can be used such as a simple majority decision, a majority voting ensemble, a weighted decision, the Bayes rule, a stacking technique.

[0048] In a simple majority decision, at least two probability values ​​are considered, and the plant class with the highest probability value of at least two probability values ​​is selected.

[0049] In the majority decision ensemble, the class predicted by the majority of at least two probability values ​​is selected. In an example, steps (a) to (c) of the inventive procedure generate at least three probability values ​​(i.e., at least one plant is object-localized, identified, and classified in at least three at least partially overlapping image recordings).

[0050] The majority decision ensemble is demonstrated in an example using an odd number of probability values ​​generated in step (c) (e.g., at least three).

[0051] Probability values) are carried out (see also Figure 5). In the weighted decision, the at least two probability values ​​are combined with specific weights. The weighting can be done, for example, via the confidence value of object localization and identification in step (b). That is, in the object tracking of the at least one plant in step (b), a confidence value is created for each image acquisition, representing the accuracy and reliability of the object tracking of the at least one plant in the respective image acquisition. In step (c), the confidence value of the assignment to a plant species and the confidence value of the object tracking are then taken into account to generate a probability value (e.g., via a weighted average calculation). In other words, a confidence value for the accuracy and reliability of the object tracking and for the classification is generated for each plant in each image acquisition.The two calculated confidence values ​​can be used in step c) to ultimately determine a consolidated confidence value for assigning at least one plant to a plant species. This consolidated confidence value can then be used in step (d). Alternatively, for example, the confidence value for the accuracy and reliability of the object tracking can be included as a weighting parameter in the determination in step (d).

[0052] Step (d) of the procedure according to the findings can alternatively be carried out by applying Bayes' rule. Bayes' rule makes it possible to calculate the updated probability of a class based on the previous assumptions (a priori probabilities) and the probabilities with which the observed data occur (likelihoods). This allows for a precise estimation of class membership based on new information.

[0053] Another method applicable in step (d) is the "stacking technique," in which predictions from several base classifiers are used as input for a meta-classifier. The meta-classifier is then trained to combine the predictions of the base classifiers and make a final prediction. The stacking technique makes it possible to combine the strengths of different classifiers and potentially achieve better predictions. By using a meta-classifier, the predictions of the base classifiers can be combined and optimized at a higher level, which can lead to improved prediction performance.

[0054] The information in step (d) can, for example, be output to a user via a suitable output device such as a monitor, printer, mobile phone, speaker, projector, etc.

[0055] In another implementation of the computer-implemented method, in step (e) at least one nozzle of an agricultural device (e.g., an agricultural sprayer) is controlled, at least partially, based on information determined in step (d). The information determined in step (d) contains data on object-located, identified, and classified plants in the agricultural field, thus enabling its direct use for controlling spray nozzles of agricultural devices. For example, classified companion plants can be specifically treated with a herbicide. Similarly, classified crops can be specifically treated with a pesticide.

[0056] Steps (a) to (d) of the computer-implemented procedure are preferably performed iteratively ( ;The process is carried out as follows: In (a / ), at least one further image of the agricultural field is generated. This further image preferably overlaps at least partially with the previous image. The process is carried out with the new image data, and the data calculated using the first and second image acquisitions are updated. For example, in step (a), continuous (and preferably at least partially overlapping) image acquisitions of the agricultural field can be generated by an agricultural device (e.g., a sprayer) moving across it.A further aspect of the invention relates to a device for applying a liquid to at least a part of an agricultural field, comprising a computing and control unit, at least one reservoir for receiving the liquid, at least one nozzle, means for drawing the liquid from the at least one reservoir towards the at least one nozzle, and at least one image acquisition unit that can be oriented towards the agricultural field (e.g., the ground) and is configured to produce at least two at least partially overlapping image images. The computing and control unit is configured (a) to cause the at least one image acquisition unit to produce at least two at least partially overlapping image images of the agricultural field, wherein at least one plant can be at least partially captured in each image image.The computing and control unit is configured (b) to perform object tracking of at least one plant in each image. The computing and control unit is further configured (c) to classify the at least one plant in each of the at least two at least partially overlapping image images by assigning it to a plant species, generating a probability value in each case, with each probability value corresponding to the confidence value of the assignment to a plant species. The computing and control unit is further configured (d) to use the at least two probability values ​​from step (c) to determine the plant species of the at least one plant. The computing and control unit is further configured (e) to activate the at least one die, at least partially, based on the information from step (d), in order to apply the liquid.

[0057] The liquid may be water or a liquid solution or suspension. The liquid solution or suspension may contain one or more nutrients and / or one or more plant protection products and / or one or more seed treatment agents.

[0058] The term "nutrients" refers to those inorganic and organic compounds from which plants can obtain the elements that make up their bodies. These elements themselves are also often referred to as nutrients. They are mostly simple inorganic compounds such as nitrate (NO₃⁻) and phosphate (PO₄²⁻). 3 - 1 and potassium (K +In addition to the core elements of organic matter (C, O, H, N and P), the following are essential for life: K, S, Ca, Mg, Mo, Cu, Zn, Fe, B, Mn, Cl in higher plants, Co, Ni. Various compounds can be present for the individual nutrients; for example, nitrogen can be supplied as nitrate, ammonium or amino acid.

[0059] The term "plant protection product" refers to a substance used to protect plants or plant products from harmful organisms or to prevent their effects, to kill unwanted plants or plant parts, to inhibit or prevent unwanted plant growth, and / or to influence plant life processes in a way other than as nutrients (e.g., growth regulators). Examples of plant protection products include herbicides, fungicides, and other pesticides (e.g., insecticides).

[0060] Growth regulators are used, for example, to increase the lodging resistance of cereals by shortening the stem length (intermodium shorteners), to improve the rooting of cuttings, to reduce plant height through stunting in horticulture, or to prevent the germination of potatoes. Growth regulators can be, for example, phytohormones or their synthetic analogs.

[0061] The term "computing and control unit" refers to components of a computer or processor used to perform arithmetic and logical operations, as well as to control the execution of instructions and operations. In modern computers, the computing unit and the control unit are often closely integrated and together form the so-called "CPU" ("Central Processing Unit"). These two units work together to enable the processing and execution of instructions in a computer. It is also possible for at least certain functions to be performed via cloud-based computing. The liquid is stored in a reservoir before application. There may be several reservoirs. Several (different) liquids may be applied. The liquid is applied to at least part of the agricultural field via one or more nozzles.In a preferred embodiment, several nozzles are used, preferably encompassed by a beam or bar (hereinafter referred to as a "spray beam"). In a further preferred embodiment, the device according to the invention comprises a plurality of nozzles. The term plurality preferably means more than ten. The nozzles are preferably arranged such that each nozzle dispenses liquid in an area with a maximum lateral extent of less than 20 cm. The plurality of nozzles can, for example, be arranged side by side along a spray beam that extends transversely (e.g., at an angle of 90°) to the direction of movement of the agricultural device. At least one camera can be assigned to each nozzle. For dispensing, the liquid is conveyed from the at least one storage container towards the at least one nozzle by means of conveying means. For example, a pump can be used to convey the liquid.The at least one image acquisition unit, which can be oriented towards the agricultural field (e.g., the ground) and is configured to produce at least one image, is, for example, a camera as described above. The camera axis of the at least one camera, as part of the device according to the invention, can preferably be oriented substantially perpendicular (vertically) to the ground (i.e., the at least one camera points "downwards" or "straight ahead" with respect to the ground). Preferably, the at least one camera is located (in the direction of travel) in front of the at least one (spray) screen. This distance is, for example, between 30 cm and 100 cm.

[0062] The device according to the invention can also be equipped with a positioning unit. Such a system can, for example, include a receiver of a satellite navigation system (GNSS, Global Navigation Satellite System), colloquially also referred to as a GPS receiver. The Global Positioning System (abbreviated GPS), officially NAVSTAR GPS, is an example of a global satellite navigation system for determining position; other examples are GLONASS, Galileo, and Beidou. The object recognition technology described above can be used in conjunction with the data from a positioning unit to precisely locate and map at least one plant.

[0063] Another subject of the present disclosure is a non-volatile, computer-readable storage medium on which software instructions are stored which, when executed by a processor of a computer system, cause the computer system to perform the following steps:

[0064] -(a) Receiving at least two partially overlapping photographs of the agricultural field, wherein at least one plant is at least partially captured in each photograph,

[0065] -(b) Object tracking of at least one plant in each image capture,

[0066] -(c) Classification of the at least one plant in each of the at least two partially overlapping images by assigning it to a plant species, whereby a probability value is generated in each case and each probability value corresponds to the confidence value of the assignment to a plant species,

[0067] -(d) Use of the at least two probability values ​​to determine the species of the at least one plant.

[0068] Other implementation forms of the present invention are:

[0069] 1. Computer-implemented method (10) for image-based detection and classification of at least one plant in an agricultural field comprising the steps: -(a) generating at least two at least partially overlapping images of the agricultural field, wherein at least one plant can be at least partially detected in each image,

[0070] -(b) Object tracking of at least one plant in each image capture,

[0071] -(c) Classification of the at least one plant in each of the at least two at least partially overlapping images by assigning the at least one plant to a plant species, whereby a probability value is generated in each case and each probability value corresponds to the confidence value of the assignment of the at least one plant to a plant species,

[0072] -(d) Use of the at least two probability values ​​to determine the species of the at least one plant.

[0073] 2. Computer-implemented method (10) according to embodiment 1, wherein in step a) the at least one plant on each image can be substantially fully captured.

[0074] 3. Computer-implemented method (10) according to one of the previous embodiments, wherein in step (a) continuous image recordings of the agricultural field can be generated by an agricultural device moving thereon.

[0075] 4. Computer-implemented method (10) according to one of the previous embodiments, wherein the object tracking in step (b) of at least one plant in the agricultural field is carried out, preferably via optical flow, image-based object tracking, multi-object tracking or multi-view object matching.

[0076] 5. Computer-implemented method (10) according to implementation form 4, wherein the image-based object tracking in step (b) comprises the following steps:

[0077] -(bl) Identification and object localization of at least one plant in the first of at least two partially overlapping image recordings,

[0078] -(b2) Calculation of at least individual pixel shifts of the successive second image acquisition compared to the first image acquisition,

[0079] -(b3) Position determination of the at least one plant on the successive second image of the at least two at least partially overlapping image recordings by at least partially using the information of the at least one plant identified and object-located in step bl) and the pixel shift calculated in step b2).

[0080] 6. Computer-implemented method (10) according to implementation form 4, wherein the multi-target sequence tracking in step (b) comprises the following steps:

[0081] -(bl) Identification and object localization of at least one plant in the first of at least two partially overlapping image recordings,

[0082] -(b2) Tracking the movement and determining the position of at least one plant on the successive second image of at least two at least partially overlapping image recordings.

[0083] 7. Computer-implemented method (10) according to embodiment 5 or 6, wherein the identification and object localization of the at least one plant on the at least one image is carried out at least in step (bl), preferably also in step (b2) or (b3), by generating a bounding box and / or a polygon. 8. Computer-implemented method (10) according to one of the preceding embodiments, wherein the calculations in step (c) and step (d) are carried out for each plant identified and object localized in step (b) on the at least two at least partially overlapping image recordings.

[0084] 9. Computer-implemented method (10) according to one of the previous embodiments, wherein, for the object tracking of the at least one plant in step (b), a confidence value is created for each image acquisition, which represents the accuracy and reliability of the object tracking of the at least one plant on the respective image acquisition, and in step c) for the generation of a probability value, the confidence value of the assignment to a plant species and the confidence value of the object tracking are taken into account (e.g. via a weighted average calculation).

[0085] 10. Computer-implemented method (10) according to one of the previous implementations, wherein in step d) the determination of the plant species of the at least one plant is carried out by a majority decision ensemble.

[0086] 11. Computer-implemented method (10) according to embodiment 10, wherein in step a) at least three at least partially overlapping image recordings of the agricultural field, wherein at least one plant can be at least partially captured on each image recording, are generated, in steps b) and c) the three at least partially overlapping image recordings are used to generate at least three probability values ​​for step d).

[0087] 12. Computer-implemented method (10) according to one of the previous implementations, comprising in step:

[0088] -(e) Control of at least one part of an agricultural device, preferably an agricultural spraying device, at least partially on the basis of information determined in step (d).

[0089] 13. Computer-implemented method (10) according to one of the previous implementations, wherein steps (a) to (d) are performed iteratively.

[0090] 14. Device (100) for applying a liquid (F) to at least a part of an agricultural field (LF) comprising a calculating and control unit (110), at least one reservoir (120) for receiving the liquid (F), at least one nozzle (130),

[0091] Means (150) for requesting the fluid (F) from the at least one reservoir (120) towards the at least one die (130), at least one image acquisition unit (160) which can be oriented towards the agricultural field, and is configured to produce at least two at least partially overlapping image recordings, wherein the computing and control unit (110) is configured (a) to cause the at least one image acquisition unit (160) to produce at least two at least partially overlapping image recordings of the agricultural field, wherein at least one plant can be at least partially captured on each image recording, wherein the computing and control unit (110) is configured, (b) to perform object tracking of at least one plant on each image recording, wherein the computing and control unit (110) is configured,(c) to classify the at least one plant on each of the at least two at least partially overlapping images by assigning it to a plant species, generating a probability value in each case, each probability value corresponding to the confidence value of the assignment to a plant species, wherein the computing and control unit (110) is configured, (d) to use the at least two probability values ​​from step (c) to determine the plant species of the at least one plant, wherein the computing and control unit (110) is configured, (e) to activate the at least one device (130) at least partially on the basis of the information from step (d) in order to apply the liquid.

[0092] 15. Non-volatile, computer-readable storage medium (200) on which software instructions are stored which, when executed by a processor of a computer system (300), cause the computer system (300) to perform the following steps:

[0093] (a) Receiving at least two partially overlapping images of the agricultural field, wherein at least one plant is at least partially captured in each image,

[0094] -(b) Object tracking of at least one plant in each image capture,

[0095] -(c) Classification of the at least one plant in each of the at least two at least partially overlapping images by assigning it to a plant species, whereby a probability value is generated in each case and each probability value corresponds to the confidence value of the assignment to a plant species,

[0096] -(d) Use of the at least two probability values ​​to determine the species of the at least one plant.

[0097] The invention is explained in more detail below using figures and examples, without intending to limit the invention to the features and combinations of features mentioned in the figures and examples.

[0098] They show:

[0099] Figure 1 schematically illustrates the computer-implemented method 10 for image-based detection and classification of at least one plant in an agricultural field. The method comprises, in step (a), the generation of at least two at least partially overlapping image acquisitions of the agricultural field, whereby at least one plant can be at least partially detected in each image acquisition. In step (b), the object tracking of at least one plant in each image acquisition is performed. In step (c), the at least one plant in each of the at least two at least partially overlapping image acquisitions is classified by assigning the at least one plant to a plant species, whereby a probability value is generated in each case, and each probability value corresponds to the confidence value of the assignment of the at least one plant to a plant species.In step (d), at least two probability values ​​are used to determine the plant species of at least one plant. Step (b) can be divided into steps (bl), (b2), and (b2). Step (bl), when applying the image-based object tracking method for object tracking, describes the identification and object localization of at least one plant in the first of at least two partially overlapping image acquisitions. Step (b2) includes the calculation of at least individual pixel shifts of the subsequent second image acquisition compared to the first image acquisition.Step (b3) comprises determining the position of at least one plant in the successive second image of the at least two at least partially overlapping image acquisitions by at least partially using the information of the at least one plant identified and object-located in step bl) and the pixel shift calculated in step b2). When applying multi-target sequence tracking for object tracking, step (bl) describes the identification and object-localization of at least one plant in the first of the at least two at least partially overlapping image acquisitions. Step (b2) comprises tracking the movement and determining the position of the at least one plant in the successive second image of the at least two at least partially overlapping image acquisitions.Figure 1 schematically shows the optional step e) (dashed box) comprising the control of at least one nozzle of an agricultural device, preferably an agricultural sprayer, at least partially based on information determined in step (d). The upper right indicates that the process a) to d) or a) to e) can be carried out iteratively (a,) - d,) or a,) - e,).

[0100] Figure 2 schematically shows how the at least one image is analyzed by the computer-implemented method 10. In step a), at least two partially overlapping images of the agricultural field are generated (B1 and B2). The at least one plant (Pfl) is captured in each image: in B1, it is located at the top center of the image, and in B2, it is located at the bottom center of the image. In step b), the at least one plant is located and identified in each image by object tracking. In Figure 2, a dashed boundary box is shown around the plant in each image. In step c), the at least one plant in B1 and B2 is classified by assigning it to a plant species. Figure 1 shows the classification of Pfl to the plant species companion plants (BPfl). The probability value W(B1, Pfl) corresponds to the confidence value of the assignment of Pfl to BPfl for image 1 (B1).The probability value W(B2, Pfll) corresponds to the confidence value of the assignment of Pfll to BPfl for Figure 2 (B2). In step d), these probability values ​​are used to determine, based on the data available up to that point, whether Pfll 1 is a companion plant or not. The upper right indicates that the procedure a) to d) can be carried out iteratively. z ) - d,).

[0101] Figure 3 schematically shows the device 100 for applying a liquid F to at least a part (e.g., as shown, a plant) of an agricultural field LF. The device 100 comprises a calculating and control unit 110, at least one reservoir 120 for receiving the liquid F, at least one nozzle 130, means 150 for drawing the liquid F from the at least one reservoir 120 towards the at least one nozzle 130, and at least one image acquisition unit 160, which can be oriented towards the agricultural field F. In Figure 1, the device 100 moves from right to left, and the image acquisition unit 160 is located in the direction of movement in front of the at least one nozzle 130. The direction of the nozzle 130 is shown obliquely relative to the ground surface. However, the direction of the nozzle 130 can also be directed straight downwards relative to the ground surface.

[0102] Figure 4 schematically shows the functionality of the device 100 for applying a liquid F to at least a part of an agricultural field LF. The device 100 comprises a computing and control unit 110, at least one reservoir 120 for receiving the liquid F, at least one nozzle 130, means 150 for dispensing the liquid F from the at least one reservoir 120 towards the at least one nozzle 130, and at least one image acquisition unit 160, which can be oriented towards the agricultural field and is configured to generate at least two at least partially overlapping images.The computing and control unit 110 is configured in step (a) to cause the at least one image acquisition unit 160 to generate at least two at least partially overlapping image captures of the agricultural field, whereby at least one plant can be at least partially captured in each image capture. The computing and control unit 110 is configured in step (b) to perform object tracking of at least one plant in each image capture. The computing and control unit 110 is further configured in step (c) to perform a classification of the at least one plant in each of the at least two at least partially overlapping image captures by assigning it to a plant species, whereby a probability value is generated in each case and each probability value corresponds to the confidence value of the assignment to a plant species.The computing and control unit 110 is configured in step (d) to use at least two probability values ​​from step (c) to determine the plant species of at least one plant. The computing and control unit 110 is further configured in step (e) to activate at least one die 130, at least partially, based on the information from step (d), in order to apply the liquid F. As described for the procedure in Figure 1, step b) can be further subdivided into steps (b1), (b2), and (b3). The upper right indicates that steps a) to e) can be performed iteratively (a / ) - e,).

[0103] Figure 5 shows an example of the implementation of one form of the invention-based method using three overlapping images (B1, B2, and B3) of a sugar beet field. A plant (shown within a white circle in each image) is present in all images (each at a different position). (The boundary box around each plant is visible within the white circle around the respective plant.) The invention-based method determines a class for the plant in each image, and a majority decision (ensemble) procedure (step (d)) determines the final class for the plant. In this example, in image 1 (B1, at time t = 1), the plant was identified as a companion plant. In image 2 (B2, at time t = 2, where t = 2 is later than t = 1), the plant was identified as a sugar beet. In image 3 (B3, at time t=3, where t=3 is later than t=2) the plant has been identified as a companion plant.The result of the majority decision procedure for this input data is 2:1 in favor of the plant species companion plant.

Claims

PATENT CLAIMS 1. Computer-implemented method (10) for image-based detection and classification of at least one plant in an agricultural field comprising the steps: (a) Production of at least two at least partially overlapping images of the agricultural field, wherein at least one plant can be at least partially captured in each image, -(b) Object tracking of at least one plant in each image capture, -(c) Classification of the at least one plant in each of the at least two at least partially overlapping images by assigning the at least one plant to a plant species, whereby a probability value is generated in each case and each probability value corresponds to the confidence value of the assignment of the at least one plant to a plant species, -(d) Use of the at least two probability values ​​to determine the species of the at least one plant.

2. Computer-implemented method (10) according to claim 1, wherein in step a) the at least one plant can be substantially fully captured on each image.

3. Computer-implemented method (10) according to one of the previous claims, wherein in step (a) continuous image recordings of the agricultural field can be generated by an agricultural device moving thereon.

4. Computer-implemented method (10) according to one of the preceding claims, wherein the object tracking in step (b) is performed by at least one plant in the agricultural field, preferably fiber optical flow, image-based object tracking, multi-object tracking or multi-view object matching.

5. Computer-implemented method (10) according to claim 4, wherein the image-based object tracking in step (b) comprises the following steps: -(bl) Identification and object localization of at least one plant in the first of at least two partially overlapping image recordings, -(b2) Calculation of at least individual pixel shifts of the successive second image acquisition compared to the first image acquisition, -(b3) Position determination of the at least one plant on the successive second image of the at least two at least partially overlapping image recordings by at least partially using the information of the at least one plant identified and object-located in step bl) and the pixel shift calculated in step b2).

6. Computer-implemented method (10) according to claim 4, wherein the multi-target sequence tracking in step (b) comprises the following steps: -(bl) Identification and object localization of at least one plant in the first of at least two partially overlapping image recordings, -(b2) Tracking the movement and determining the position of at least one plant on the successive second image recording of at least two at least partially overlapping image recordings.

7. Computer-implemented method (10) according to claim 5 or 6, wherein the identification and object localization of the at least one plant on the at least one image acquisition in step (bl), preferably also in step (b2) or (b3), is carried out by generating a bounding box and / or a polygon.

8. Computer-implemented method (10) according to one of the previous claims, wherein the calculations in step (c) and step (d) are performed for each plant identified and object-located in step (b) on the at least two image recordings that overlap at least partially.

9. Computer-implemented method (10) according to one of the previous claims, wherein, for the object tracking of the at least one plant in step (b), a confidence value is created for each image acquisition, which represents the accuracy and reliability of the object tracking of the at least one plant on the respective image acquisition, and in step c) for generating a probability value, the confidence value of the assignment to a plant species and the confidence value of the object tracking are taken into account.

10. Computer-implemented method (10) according to one of the previous claims, wherein in step d) the determination of the plant species of the at least one plant is carried out by a majority decision ensemble.

11. Computer-implemented method (10) according to claim 10, wherein in step a) at least three at least partially overlapping image recordings of the agricultural field, wherein at least one plant can be at least partially captured on each image recording, are generated, in steps b) and c) the three at least partially overlapping image recordings are used to generate at least three probability values ​​for step d).

12. Computer-implemented method (10) according to any of the previous claims, comprising in step: -(e) Control of at least one part of an agricultural device, preferably an agricultural spraying device, at least partially on the basis of information determined in step (d).

13. Computer-implemented method (10) according to one of the previous claims, wherein steps (a) to (d) are performed iteratively.

14. Device (100) for applying a liquid (F) to at least a part of an agricultural field (LF) comprising a calculating and control unit (110), at least one storage container (120) for receiving the liquid (F), at least one nozzle (130), Means (150) for requesting the flow (F) from the at least one storage container (120) towards the at least one die (130), at least one image acquisition unit (160) which can be oriented towards the agricultural field and is configured to produce at least two at least partially overlapping image recordings, wherein the computing and control unit (110) is configured (a) to cause the at least one image acquisition unit (160) to produce at least two at least partially overlapping image recordings of the agricultural field, wherein at least one plant can be at least partially captured on each image recording, wherein the computing and control unit (110) is configured (b) to perform object tracking of at least one plant on each image recording, wherein the computing and control unit (110) is configured, (c) to perform a classification of the at least one plant on each of the at least two at least partially overlapping image recordings by assigning it to a plant species, generating a probability value in each case and each probability value corresponding to the confidence value of the assignment to a plant species, wherein the computing and control unit (110) is configured, (d) to use the at least two probability values ​​from step (c) to determine the plant species of the at least one plant, wherein the computing and control unit (110) is configured, (e) to activate the at least one device (130) at least partially on the basis of the information from step (d) in order to apply the liquid.

15. Non-volatile, computer-readable storage medium (200) on which software instructions are stored which, when executed by a processor of a computer system (300), cause the computer system (300) to perform the following steps: -(a) Receiving at least two partially overlapping images of the agricultural field, wherein at least one plant is at least partially captured in each image, -(b) Object tracking of at least one plant in each image capture, -(c) Classification of the at least one plant in each of the at least two at least partially overlapping images by assigning it to a plant species, whereby a probability value is generated in each case and each probability value corresponds to the confidence value of the assignment to a plant species, -(d) Use of the at least two probability values ​​to determine the species of the at least one plant.

Images