Detecting functional impairment of a device

Abstract

The devices, methods and computer programs disclosed herein refer to the automated detection of functional impairment of a device, which comprises an image capturing unit and an image recognition unit for plants, by methods of machine learning. The device is in particular a spraying device.

Description

[0001] Detecting functional impairments in a device

[0002] TECHNICAL FIELD

[0003] The devices, methods, and computer programs disclosed herein relate to the automated detection of functional impairments in a device having an image capture and image recognition unit of plants using machine learning methods. The device is, in particular, a spraying device.

[0004] INTRODUCTION

[0005] There is a general need to apply liquids to crops in a field with increasing precision to conserve resources, increase the efficiency of applied products, and prevent application in undesired areas. For example, when controlling companion plants that develop alongside crops in a field, there is a need to treat only the companion plants with a herbicide and not apply herbicide to the crops or to the soil between the plants.

[0006] WO2019 / 166497A1 discloses a spray treatment device in which at least one image of a field is provided to a processing unit. Historical details regarding the spray application of a weed control liquid and / or a pest control liquid are also provided to the processing unit. The processing unit analyzes the at least one image to determine at least one location within the field for the activation of at least one spray gun for weed control and / or for the activation of at least one spray gun for pest control.

[0007] W02019007894A1 discloses a device for weed control. At least one image of an environment is provided to the processing unit of the device. The processing unit analyzes the at least one image to determine, from a plurality of vegetation control technologies, at least one vegetation control technology to be used for weed control for at least a first portion of the environment. An output unit outputs information usable for activating the at least one vegetation control technology.

[0008] It is possible that a device with a plant image capture and image recognition unit may experience a functional impairment over time. For example, the camera's optical system may become dirty during operation. Another possible functional impairment is, for example, plant debris becoming caught in the device or—in the case of a spraying device—the spray nozzles becoming clogged. Further possible functional impairments are listed further down in the description.

[0009] A functional impairment means that the function intended to be performed by the device can no longer be performed or can no longer be performed sufficiently.

[0010] SUMMARY

[0011] This problem is solved by the subject matter of the independent claims of the present disclosure. Preferred embodiments can be found in the dependent claims, the description, and the drawings. The present disclosure describes means by which a functional impairment of a device with an image capture and image recognition unit of plants can be detected at an early stage.

[0012] A first subject of the present disclosure is a computer-implemented method for a device having an image capture and an image recognition unit of plants, comprising:

[0013] Receiving an image recording, wherein the received image recording shows at least a part of a field (and optionally shows at least a part of the device) and was recorded by the image acquisition unit of the device,

[0014] Providing a trained machine learning model, wherein the machine learning model is configured and has been trained on the basis of training data to distinguish image recordings of at least a part of a field (and optionally at least a part of the device) with a functional impairment of the device with an image capture and an image recognition unit of plants from image recordings of a part of a field (and optionally at least a part of the device) without a functional impairment of the device with an image capture and an image recognition unit of plants,

[0015] Feeding the received image to the trained machine learning model,

[0016] Receiving information from the machine learning model, the information indicating whether the image recording of at least part of a field (and optionally at least part of the device) indicates a functional impairment of the device with an image capture and image recognition unit of plants, in the event that the received image recording of at least part of a field (and optionally at least part of the device) indicates a functional impairment of the device with an image capture and image recognition unit of plants: issuing a notification that the received image recording indicates a

[0017] Impairment of the device with an image capture and a

[0018] Image recognition unit.

[0019] A further subject of the present disclosure is a computer system comprising: an input unit, a control and computing unit and an output unit, wherein the control and computing unit is configured to cause the input unit to receive an image recording, wherein the received image recording shows at least part of a field and optionally also at least part of a device with an image acquisition and image recognition unit for plants and was recorded by a device with an image acquisition and image recognition unit for plants, to feed the received image recording to a machine learning model, wherein the machine learning model is configured and has been trained on the basis of training data,To distinguish image recordings of at least part of a field and optionally at least part of a device with an image capture and image recognition unit for plants with a functional impairment of the device with an image capture and image recognition unit for plants from image recordings of at least part of a field and optionally at least part of a device with an image capture and image recognition unit for plants without a functional impairment of the device with an image capture and image recognition unit for plants, To receive information from the machine learning model, wherein the information indicates whether the received image recording represents a device with an image capture and image recognition unit for plants with a functional impairment, To cause the output unit to output a message, wherein the message indicatesthat the received image indicates a functional impairment of the device with an image capture and image recognition unit for plants, if the information output by the machine learning model indicates that the received image represents a device with an image capture and image recognition unit for plants with a functional impairment.

[0020] Another subject of the present disclosure is a non-transitory computer-readable storage medium having stored thereon software instructions that, when executed by a processor of a computer system, cause the computer system to perform the following steps:

[0021] Receiving an image recording, wherein the received image recording shows at least a part of a field (and optionally at least a part of the device) and was recorded by the image acquisition unit of the device,

[0022] Providing a trained machine learning model, wherein the machine learning model is configured and has been trained on the basis of training data to distinguish image recordings of at least a part of a field (and optionally at least a part of the device) with a functional impairment of the device with an image capture and an image recognition unit of plants from image recordings of a part of a field (and optionally at least a part of the device) without a functional impairment of the device with an image capture and an image recognition unit of plants,

[0023] Feeding the received image to the trained machine learning model,

[0024] Receiving information from the machine learning model, the information indicating whether the image recording of at least part of a field (and optionally at least part of the device) indicates a functional impairment of the device with an image capture and image recognition unit of plants, in the event that the received image recording of at least part of a field (and optionally at least part of the device) indicates a functional impairment of the device with an image capture and image recognition unit of plants: issuing a notification that the received image recording indicates a

[0025] Impairment of the device with an image capture and a

[0026] Image recognition unit.

[0027] Another subject of the present disclosure is a spraying device with an image capture and image recognition unit for plants comprising:

[0028] • a control unit,

[0029] • an analysis unit,

[0030] • an output unit

[0031] • at least one image acquisition unit, wherein the control unit is configured to cause the at least one image acquisition unit to generate one or more image recordings of at least a part of a field and optionally of at least a part of the spraying device, wherein the analysis unit is configured to distinguish the one or more generated image recordings of at least a part of the field and optionally of at least a part of the spraying device with a functional impairment of the spraying device from image recordings of at least a part of a field and optionally of at least a part of the spraying device without a functional impairment of the spraying device, wherein the analysis unit is configured to receive information from the machine learning model, wherein the information indicates whether the one or more image recordings represent the spraying device with a functional impairment,wherein the output unit is configured to output the information.,

[0032] BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Fig. 1 shows schematically and exemplarily the training of a machine learning model.

[0034] Fig. 2 schematically shows the use of a trained machine learning model to detect a functional impairment in a device with an image acquisition and an image recognition unit of plants.

[0035] Fig. 3 schematically shows another example of training a machine learning model.

[0036] Fig. 4 schematically shows another example of using a trained machine learning model to detect a functional impairment in a device having an image capture and an image recognition unit of plants.

[0037] Fig. 5 shows an example and schematically a computer-implemented method for training a machine learning model in the form of a flow chart.

[0038] Fig. 6 shows, by way of example and schematically, a computer-implemented method for detecting a functional impairment in a device with an image acquisition and an image recognition unit of plants in the form of a flow chart.

[0039] Fig. 7 shows an exemplary and schematic illustration of an embodiment of a computer system of the present disclosure.

[0040] Fig. 8 shows an exemplary and schematic illustration of another embodiment of a computer system of the present disclosure.

[0041] DETAILED DESCRIPTION

[0042] The invention is explained in more detail below, without distinguishing between the subject matters of the present disclosure (method, computer system, computer-readable storage medium, device). Rather, the following statements apply mutatis mutandis to all subject matters of the invention, regardless of the context in which they are described (method, computer system, computer-readable storage medium, device).

[0043] If steps are specified in a particular order in this description or in the claims, this does not necessarily mean that the invention is limited to the specified order. Rather, it is conceivable that the steps could also be performed in a different order or even in parallel, unless one step builds on another, which necessarily requires that the subsequent step be performed (although this will become clear in the individual case). The specified orders are therefore preferred embodiments of the present disclosure.

[0044] The invention is explained in more detail at some points with reference to drawings. The drawings depict specific embodiments with specific features and combinations of features, which primarily serve for illustrative purposes; the invention should not be understood as being limited to the features and combinations of features shown in the drawings. Furthermore, statements made in the description of the drawings regarding features and combinations of features are intended to apply generally, meaning they are also transferable to other embodiments and are not limited to the embodiments shown.

[0045] The present disclosure describes means for detecting early and automatically a functional impairment in a device having an image capture and image recognition unit of plants.

[0046] A device with an image capture and image recognition unit for plants is understood to be a device in which plants in a field are captured, for example, with a camera and are detected and / or identified by an image recognition algorithm. The device is preferably a spraying device and is designed, for example, to spray a companion plant with an agent for controlling the companion plant. The decision to do so is based at least partially on the result of the image analysis of the plants in the field. It is also possible for the image recognition algorithm to detect and / or identify pests or fungal infestation on the crop and for the affected crop to be sprayed with an appropriate plant protection agent based on the results of the image analysis.Another embodiment is the precise supply of nutrients with the spraying device based on the results of the image recognition of the plants (e.g. in the case of low growth of certain identified crops compared to other crops in the same field, or identification of a "visible" nutrient deficiency of crops) in the field.

[0047] The term "field" refers to a spatially defined area of ​​the earth's surface used for agricultural purposes, in which crops are planted, possibly supplied with nutrients, and harvested. A crop community does not necessarily have to be established at the time of application.

[0048] The term “cultivated plant” refers to a plant that is purposefully cultivated as a useful or ornamental plant through human intervention.

[0049] The term “companion plants” (often also referred to as weeds) refers to plants of the spontaneous accompanying vegetation (segetal flora) in crop stands, grassland or gardens, which are not deliberately cultivated there and develop, for example, from the seed potential of the soil or via migration.

[0050] The device with a plant image capture and image recognition unit comprises, for example, a control unit, an analysis unit, an output unit, and at least one image capture unit. The image capture unit preferably comprises at least one camera.

[0051] If the device is a spraying device, the device further comprises at least one storage container for holding a liquid agent, at least one spray nozzle and means for conveying the liquid agent in the direction of the at least one spray nozzle. The spraying device can, for example, be connected to an agricultural machine (e.g. as part of a mobile trailer) but can also be a direct part of a vehicle, e.g. a robot or an aircraft (e.g. a drone). Such a spraying device can move autonomously in or over a field or be controlled by a person. In one embodiment of this device with an image capture and an image recognition unit of plants, the control unit is configured to cause the at least one image capture unit to generate one or more recorded images of at least part of a field and optionally of the device.The analysis unit is configured to detect and / or identify plants in the field based on the images received from the camera (i.e., executes the image recognition algorithm). The output unit is configured to output information about the detected and / or identified plants in the field. If the device is a spraying device, the output unit is configured to activate the at least one spray nozzle based on the detected and / or identified plants. The spray nozzle sprays the detected and / or identified plant with the liquid agent.

[0052] The liquid agent may be water or an aqueous solution or suspension. The aqueous solution or suspension may contain one or more nutrients and / or one or more plant protection products and / or one or more seed treatment products.

[0053] The term "nutrients" refers to those inorganic and organic compounds from which plants can extract the elements from which their bodies are built. These elements themselves are often also referred to as nutrients. These are usually simple inorganic compounds such as nitrate (NO 3- ), phosphate (PO ) and potassium (K + In addition to the core elements of organic matter (C, O, H, N, and P), K, S, Ca, Mg, Mo, Cu, Zn, Fe, B, Mn, CI in higher plants, Co, and Ni are also essential for life. Different compounds can be present for the individual nutrients; for example, nitrogen can be supplied as nitrate, ammonium, or amino acid.

[0054] The term “plant protection product” refers to a product used to protect plants or plant products from pests or to prevent their effects, to destroy undesirable plants or parts of plants, to inhibit undesirable plant growth or to prevent such growth, and / or to influence plant life processes in a way other than through nutrients (e.g., growth regulators). Examples of plant protection products include herbicides, fungicides, and other pesticides (e.g., insecticides). Growth regulators are used, for example, to increase the stability of cereals by shortening stalk length (internode shorteners), to improve the rooting of cuttings, to reduce plant height by compression in horticulture, or to prevent the germination of potatoes. Growth regulators can be, for example, phytohormones or their synthetic analogues.

[0055] In the device with an image capture and an image recognition unit of plants, at least one camera is arranged and aligned such that it can generate images of at least part of the field and optionally of at least part of the device. In one embodiment, the device with an image capture and an image recognition unit of plants is a spraying device and the at least one camera is ideally arranged above at least one spray nozzle and directed towards the spray nozzle and the field soil below it so that an image can be generated which shows at least part of the spraying device (preferably part of the spray nozzle) and at least part of a field. In an alternative embodiment, the at least one camera is pivotable and can capture either part of the field or part of the spraying device.It is also conceivable that two or more cameras are used to collect the image data for the two areas mentioned.

[0056] A camera is a device that can create images in digital form and store them and / or make them available via an interface. A camera typically comprises an image sensor and optical elements. The image sensor is a device for electrically capturing two-dimensional images from light. These are typically semiconductor-based image sensors such as CCD (charge-coupled device) or CMOS (complementary metal-oxide-semiconductor) sensors. The optical elements (lenses, apertures, and the like) serve to create the sharpest possible image of the object, of which a digital image is to be created, on the image sensor.

[0057] The camera is used to generate digital images of at least part of the field and, optionally, of at least part of the device. The generated images can be used (i) to detect whether one or more plants are present in the field (plant detection), and (ii) to identify plants, i.e., to determine which plant (subclass, superorder, order, suborder, family, genus, species, growth stage, etc.) it is. The exact location of the plant can also be determined, for example, using RTK-GPS (as an additional sensor of the device) and knowing the distance of the camera from the recording area.

[0058] To image the image capture area on one or more image sensors, a light source is used to illuminate the image capture area so that light (electromagnetic radiation in the infrared, visible, and / or ultraviolet range of the spectrum) is scattered / reflected from the illuminated image capture area toward the camera. Daylight can be used for this purpose. However, it is also conceivable to use an illumination unit that provides defined illumination independent of daylight. This is preferably mounted to the side of the camera so that the camera does not cast a shadow on the image capture area. The image acquisition unit preferably comprises at least one illumination unit.

[0059] The term “image capture area” refers to the area to which the camera is aimed and with which at least part of the field and optionally at least part of the device can be captured.

[0060] It is also conceivable to use a lighting source attached to the device that illuminates the image recording area from the side, while a camera takes one or more images "from above".

[0061] It is conceivable that several light sources illuminate the image recording area from different directions.

[0062] The terms "light" and "illumination" should not imply that the spectral range is limited to visible light (approximately 380 nm to approximately 780 nm). It is also conceivable that electromagnetic radiation with a wavelength below 380 nm (ultraviolet light: 100 nm to 380 nm) or above 780 nm (infrared light: 780 nm to 1000 pm) is used for illumination. The image sensor and optical elements are usually adapted to the electromagnetic radiation used.

[0063] The device comprises a control unit. The control unit can be a component of the image capture unit, e.g., the camera, or a separate device that can communicate with the image capture unit via a wired or wireless connection (e.g., Bluetooth). The control unit is configured to cause the image capture unit to capture one or more images of at least a portion of the field and, optionally, at least a portion of the device. The control unit can be a computer system, as described further below in the description.

[0064] The one or more image recordings may be individual image recordings or sequences of image recordings (e.g. video recordings).

[0065] In a further embodiment, the device can comprise one or more sensors (other than the camera). A "sensor" is a technical component that can detect certain physical and / or chemical properties and / or the material composition of its environment qualitatively or quantitatively as a measured variable. These variables are detected by means of physical or chemical effects and converted into a further processable, usually electrical or optical signal. A sensor unit with at least one sensor can comprise further means for transmitting and / or forwarding signals and / or information (e.g., to the control unit). The at least one sensor can, for example, comprise a receiver of a satellite navigation system, colloquially also referred to as a GPS receiver. The Global Positioning System (abbreviationGPS (officially NAVSTAR GPS) is an example of a global satellite navigation system for positioning; other examples are GLONASS, Galileo, and Beidou. The satellites of such a satellite navigation system communicate their precise position and time via radio codes. To determine a position, a receiver (the "GPS receiver") must receive signals from at least four satellites simultaneously. The receiver measures the pseudo-signal propagation times, and uses these to determine the current position. Precise positioning is possible using a real-time kinematic method (RTK GPS).

[0066] In a further embodiment, the device can comprise a transmitting unit. Images and / or information can be transmitted to a separate computer system via the transmitting unit. Transmission preferably occurs via a radio network, for example, a mobile network. The transmitting unit can be a component of the control unit or a unit independent of the control unit.

[0067] The device may include a receiving unit to receive commands from a separate computer system.

[0068] The transmitting unit and / or the receiving unit may be components of a computer system as described further down in the description.

[0069] The images generated by the camera are typically analyzed automatically to detect and / or identify plants in the field. This analysis can be performed by an analysis unit that may be part of the device; however, this analysis can also be performed by an analysis unit that may be part of a separate computer system to which the images are transmitted via the device's transmission unit. The analysis unit can be part of a computer system as described further down in the description. The analysis unit can comprise a trained machine learning model that is configured and trained to detect and / or identify plants depicted in images.Details on the automated detection and / or identification of plants (and, for example, the differentiation of cultivated and companion plants) in image recordings are described in publications on this topic (see, for example, KK Thyagharajan et al., A Review of Visual Descriptors and Classification Techniques used in Leaf Species Identification, Archives of Computational Methods in Engineering, 26, 933-60, 2019). In a further embodiment, the analysis unit can also comprise a trained machine learning model that is configured and trained to detect and / or identify diseases and pests in plants depicted in image recordings. Details on this are described in publications on this topic (see, for example, Ju Liu et al., Plant Disease and Pests Detection based on Deep Learning: a Review, Plant Methods, 17, 22, 2021).

[0070] The images generated by the image acquisition unit can also be used to detect a functional impairment of the device.

[0071] A functional impairment describes a condition of the device that affects one or more components of the device or the device as a whole in such a way that one or more functions are no longer performed, no longer performed sufficiently, or no longer performed optimally. For example, a function may no longer be performed sufficiently or optimally if the impairment slows down or impedes the function, or the result is inferior or faulty.

[0072] A functional impairment can be an impairment that currently affects function or will affect function in the near future if no measures are taken to maintain function. Examples of functional impairments are listed below:

[0073] Restrictions to the field of view: It is possible that cobwebs in the device prevent the image capture unit from having a clear view of the image capture area. In addition to cobwebs, spiders that are, for example, in front of a camera lens can also pose a problem. Furthermore, insect constructs (e.g., pupae of larvae) and / or plant parts (e.g., pollen, flowers, twigs, leaves, roots) in the device can completely or partially obscure the image capture area from the perspective of the image capture unit.

[0074] The image capture unit and / or its optical elements are dirty: Deposits on a camera lens can cause impairment. It is possible that the camera's field of view is restricted as a result of deposits, and the entire image capture area is no longer captured. Deposits could result in blurred or partially blurred images. Water (e.g., rainwater) and / or another liquid could get onto a lens, restricting the field of view and / or causing blurred images.

[0075] The image capture unit, and in particular the camera, is defective: It is possible that the camera is defective and the resulting images are unsuitable for automated detection and / or identification of plants in a field. For example, the resulting images may be noisy, have a color cast, have a low contrast range, and / or be completely black or white.

[0076] Image capture unit does not generate images of at least part of a field: It is possible that the image capture unit generates images during maintenance of the device, e.g., in a workshop. Such images may be unsuitable for automated detection and / or identification of plants. Such images can also be identified and, for example, rejected using the means described in this description. Rejection may mean that a rejected image is not automatically analyzed to detect and / or identify plants in a field.

[0077] Illumination source(s) defective and / or dirty: If the device is equipped with one or more illumination sources (as part of the image acquisition unit), it is possible that one or more of these illumination sources emits no or less electromagnetic radiation, and / or that, due to contamination of one or more illumination sources, insufficient electromagnetic radiation reaches and illuminates the image acquisition area. The lack of or reduced illumination can lead to a loss of contrast and / or increased noise in the images, which in turn can complicate the automated detection and / or identification of plants.

[0078] Unwanted reflections: It is possible that reflections may be observed in the images at certain times. These could, for example, be caused by sunlight entering the image capture area at a specific angle. It is possible that sunlight enters the image capture area at certain times of day and / or year, causing unwanted reflections.

[0079] The following are examples of further functional impairments when the device is a spray device.

[0080] Changes in the position and / or orientation of components of the spraying device: It is conceivable that over time, and for example due to frequent use of the spraying device, there may be a change in the position and / or orientation and / or orientation of components of the spraying device (e.g., the spray nozzles, the orientation of the camera and / or the optical elements of the camera or the illumination source relative to the field ground, etc.). Such changes may, for example, be the result of wear or material fatigue and result in the originally intended image recording area no longer being imaged, or no longer being imaged in its entirety, and / or being imaged entirely or partially blurred.

[0081] Image capture unit fails to capture images of at least part of the spray device: It is possible that the camera captures images during maintenance of the spray device, e.g., when the spray nozzles are removed in a workshop. Such captures can be discarded as described above.

[0082] Plant debris has become trapped in the spraying device: It is possible that plant debris may become trapped on or in the device while the spraying device is in use in the field. For example, plant debris may become trapped around a spray nozzle, thereby impairing the nozzle's spraying function. It is also conceivable that plant debris may become trapped between the image capture unit and the spray nozzle during use, which could impair its functionality.

[0083] The spray cone of the spray nozzles of the spray device does not meet the specifications or the spray nozzles are defective: The image capture unit can, among other things, generate an image of at least part of a spray device. Preferably, the image capture unit records image data of the nozzle and / or the spray cone generated by the nozzle. Based on the image information from the nozzle and / or the change in the shape of the spray cone, it can be determined, for example, whether the nozzle is clogged.

[0084] The functional impairments and / or their effects are captured in images generated by the device's image acquisition unit. The images are used to train a machine learning model to automatically detect such functional impairments. The term "automatic" means without human intervention.

[0085] Such a "machine learning model" can be understood as a computer-implemented data processing architecture. The model can receive input data and produce output data based on this input data and model parameters. Through training, the model can learn a relationship between the input data and the output data. During training, model parameters can be adjusted to produce a desired output for a given input.

[0086] When training such a model, the model is presented with training data from which it can learn. The trained machine learning model is the result of the training process. The training data includes input data and the correct output data (target data) that the model is supposed to generate based on the input data. During training, patterns are recognized that map the input data to the target data.

[0087] During the training process, the input data of the training data is fed into the model, and the model generates output data. The output data is compared with the target data. Model parameters are modified so that the deviations between the output data and the target data are reduced to a (defined) minimum. An optimization method such as a gradient descent method can be used to modify the model parameters to reduce the deviations.

[0088] The deviations can be quantified using a loss function. Such an error function can be used to calculate an error (loss) for a given pair of output and target data. The goal of the training process may be to change (adjust) the parameters of the machine learning model so that the error is reduced to a (defined) minimum for all pairs of the training dataset. For example, if the output and target data are numbers, the error function may be the absolute difference between these numbers. In this case, a high absolute error may mean that one or more model parameters need to be changed significantly.

[0089] For example, for output data in the form of vectors, difference metrics between vectors such as the mean square error, a cosine distance, a norm of the difference vector such as a Euclidean distance, a Chebyshev distance, an Lp norm of a difference vector, a weighted norm, or another type of difference metric between two vectors can be chosen as the error function.

[0090] For higher-dimensional outputs, such as two-dimensional, three-dimensional, or higher-dimensional outputs, an element-wise difference metric can be used. Alternatively or additionally, the output data can be transformed, e.g., into a one-dimensional vector, before calculating an error value.

[0091] In this case, the machine learning model receives one or more image recordings as input data. The model can be trained to output information for each image recording, indicating whether the image recordings are one or more images of a device with a malfunction or one or more images of a device without a malfunction. In other words, the machine learning model can be trained to distinguish image recordings of a device with a malfunction(s) from image recordings of a device without a malfunction(s).

[0092] The machine learning model may be trained to assign the one or more image recordings to one of at least two classes, wherein at least a first class represents image recordings of devices that do not have functional impairments and at least a second class represents image recordings of devices that have a functional impairment.

[0093] The machine learning model is trained based on training data. The training data comprises a plurality of images of one or more devices (images showing at least part of a field and optionally at least part of a device). The term “plurality” means more than 10, preferably more than 100. The images serve as input data. Some of the images may show the one or more devices without any functional impairments, i.e., in a state in which they function properly. Another part of the images may show the one or more devices with a functional impairment. In addition to the input data, the training data may also comprise target data. The target data may indicate for each image whether the device depicted in the image has a functional impairment or whether it does not have any functional impairment.The target data may also include information about the type of functional impairment present in the individual case and / or how severe it is and / or what degree of severity it has.

[0094] The machine learning model can be trained to assign each image to exactly one of two classes, where exactly one class represents images of devices that do not exhibit any functional impairments, and the other class represents images of devices that exhibit one or more functional impairments. In other words, the machine learning model can be trained to perform binary classification. In such a case, it is sufficient that for each individual image in the training data, information is available as to whether the image represents a device with a functional impairment or whether the image represents a device without a functional impairment.In such a case, the machine learning model can be trained to detect devices with one (or more) functional impairments, regardless of the specific impairment(s). It is also possible to train a machine learning model to perform feature extraction for each image input to the machine learning model and generate a compressed representation of the image. The machine learning model can be trained to generate similar compressed representations for images that do not show a device with a functional impairment.If the trained machine learning model is fed an image of a device with a functional impairment, the trained machine learning model generates a compressed representation of the image of the device with the functional impairment that can be distinguished from the compressed representations of the images of devices without the functional impairment. Such training, in which the machine learning model is only trained to recognize whether a functional impairment or no functional impairment is present, can be useful if a user is only interested in knowing whether the device is functioning properly or whether intervention is required to eliminate a functional impairment (whatever it may be).

[0095] The machine learning model can also be trained to detect a specific functional impairment. The specific functional impairment can be one of the functional impairments described earlier in this description. A user may only be interested in knowing whether the specific functional impairment is present. The machine learning model can be trained to assign each image capture to one of two classes, where one class represents image captures of devices that have the specific functional impairment and the other class represents image captures of devices that do not have the specific functional impairment, i.e., where either no functional impairment is present or where a functional impairment other than the specific functional impairment is present.In such a case, the training data for each image acquisition includes information about whether the specific functional impairment is present or not present in the device.

[0096] The machine learning model can also be trained to assign each image to one of more than two classes, where the classes represent, for example, the severity of the specific functional impairment. A first class can, for example, represent images of devices in which the specific functional impairment does not occur (e.g., no contamination); a second class can represent images of devices in which the specific functional impairment occurs slightly (e.g., slight contamination); a third class can represent images of devices in which the specific functional impairment is clearly evident (e.g., significant contamination). A slight functional impairment can mean that the device is still sufficiently functional, but that maintenance will be necessary in the future to avoid further functional impairment.A significant or severe impairment may indicate that immediate maintenance is required. More than the three levels mentioned are also conceivable, e.g., four (e.g., no impairment, mild impairment, moderate impairment, severe impairment) or more and / or other levels. In such a case, the training data for each image acquisition includes information about whether the impairment is present in the device, and if so, how severe it is and / or with what severity.

[0097] However, the machine learning model can also be trained to recognize more than one specific functional impairment, i.e., to distinguish between different functional impairments. For example, the machine learning model can be trained to learn a number n of specific functional impairments, where n is an integer greater than 1. The machine learning model can be trained to assign each image recording to one of at least n+1 classes, where a first class represents image recordings of devices that do not have a functional impairment, and each of the at least n remaining classes represents image recordings of devices that show one of the n specific functional impairments. It is also possible for the machine learning model to be additionally trained to recognize two or more severity levels of one or more of the n specific functional impairments.In other words, the machine learning model can be trained to recognize the severity and / or level of severity of one or more of the n specific functional impairments. In such a case, the training data includes, for each image acquisition, information about whether a functional impairment is present, if a functional impairment is present, which specific functional impairment is present, and, for one or more of the specific functional impairments, the severity and / or level of severity of the impairment.

[0098] Existing devices with an image acquisition and an image recognition unit of plants can be used to generate the training data described in this description. Such devices can be operated for a period of time to generate image data, and the generated images can be analyzed by one or more experts. The one or more experts can annotate each image with one of the information required for training (annotations), which are then used as target data. The one or more experts can view the images and annotate each image with information indicating whether the respective image shows a device with or without functional impairment.If necessary for training the machine learning model, each image representing a device with a functional impairment can be annotated with information about the severity of the functional impairment and / or its occurrence. If necessary for training the machine learning model, each image representing a device with a functional impairment can be annotated with information about the specific functional impairment present.

[0099] When training the machine learning model, the image recordings are fed (sequentially) to the machine learning model. The machine learning model can be configured to assign each image recording to one of at least two classes. The class assignment can be output by the machine learning model, for example, in the form of a number. For example, the number 0 can represent image recordings of devices that show no functional impairment; the number 1 can represent image recordings of devices that have a first specific functional impairment; the number 2 can represent image recordings of devices that have a second specific functional impairment, etc.

[0100] It is also possible for the machine learning model to be configured to output a vector for each image recording, wherein the vector includes a number for each functional impairment at a coordinate of the vector, which number indicates whether the respective functional impairment is represented in the image recording (i.e., present in the device) or not represented (i.e., not present in the device). Such an approach has the advantage that different functional impairments that are present simultaneously in a device can also be detected alongside one another. In such a vector, the number 0 can indicate that a specific functional impairment is not present, and the number 1 can indicate that the specific functional impairment is present. The position in the vector (coordinate) at which the respective number occurs can provide information about which specific functional impairment is involved.

[0101] It is also possible for the machine learning model to be configured to specify a probability for one or more (specific) functional impairments that the (specific) functional impairment will occur in the respective device. The probability can, for example, be specified as a value in the range from 0 to 1, with the higher the value, the higher the probability.

[0102] It is also possible that the machine learning model is configured to output a severity level for one or more (specific) functional impairments with which the (specific) functional impairment occurs in the respective device.

[0103] The output (output data) produced by the machine learning model based on an input image acquisition can be compared with the target data. Deviations between the output data and the target data can be quantified using an error function. In an optimization procedure (e.g., a gradient descent method), the deviations can be reduced by modifying model parameters. If the deviations reach a (predefined) minimum or a plateau, training can be terminated. The trained machine learning model can be used to detect one or more functional impairments and, optionally, their severity in the device.

[0104] For this purpose, a new image from an image acquisition area can be fed to the machine learning model. The term "new" means that the corresponding image has not already been used to train the machine learning model. The trained machine learning model assigns the new image to one of the at least two classes that were used to train the machine learning model. The trained machine learning model outputs information about the class to which the machine learning model has assigned the image. It is possible that the trained machine learning model outputs information about the probability that one or more functional impairments are present and / or how severe they are and / or with what degree of severity they occur.

[0105] The output of the machine learning model may be displayed on a screen, printed on a printer, stored in a data storage device, and / or transmitted to a separate computer system (e.g., over a network).

[0106] The output of the machine learning model can be used to automatically filter out image recordings that represent devices with a functional impairment (or with multiple functional impairments). In such a case, an image recording of a device is analyzed in a first step according to the present disclosure for the presence of a functional impairment (or multiple functional impairments) before being analyzed in a subsequent second step to detect and / or identify plants in the field. It is possible that only those image recordings for which the analysis in the first step has shown that they do not have a functional impairment are fed to the second step for detecting and / or identifying plants in the field.It is also possible that only those images for which the analysis in the first step showed no functional impairment or only a functional impairment with a low degree of severity are sent to the second analysis for the detection and / or identification of plants in the field. Likewise, it is possible that only images with specific functional impairments or with specific functional impairments of a predefined degree of severity or a minimum number of different functional impairments are sorted out. It is possible for a user to specify in advance which images should be sorted out. The third step then optionally involves spraying a pesticide on the detected and / or identified plants.

[0107] If the trained machine learning model assigns an image to a class that represents image recordings with devices with a functional impairment, a notification can be issued to a user. Such a notification can inform the user that a functional impairment of a device exists. The notification can inform the user that an image recording will not be analyzed to detect and / or identify plants in the field because the device has a functional impairment. A notification to a user can include the following information: which device is affected (if several devices are in use, for examplea location of the affected device must be specified); location of the image capture unit (with which the received image of a device with a functional impairment was taken); what functional impairment exists; how severe is the functional impairment; what measures can be taken to restore the full functionality of the device; and when should the measures be taken to prevent further functional impairment. It is also possible that the image recording in which the trained machine learning model has detected a functional impairment is also displayed to the user so that the user can form their own opinion about the functional impairment.

[0108] If the output of the trained machine learning model is a probability value for the presence of a functional impairment, this probability value can be compared to a predefined threshold. If the probability value is greater than or equal to the threshold, a notification can be issued to a user regarding the presence of a functional impairment in the device. If the probability value is less than the threshold, the image can be analyzed to detect and / or identify plants in the field.

[0109] It is conceivable that there is more than one threshold with which the probability value is compared. For example, it is possible that there is an upper threshold and a lower threshold. If the probability value is below the lower threshold, the probability of a functional impairment is so low that the user does not need to be informed. The image can be submitted to an analysis to detect and / or identify plants in the field. If the probability value is above the upper threshold, the probability of a functional impairment is so high that a notification about the presence of a functional impairment is issued to the user. It is possible that the image is not submitted to an analysis to detect and / or identify plants in the field.If the probability value lies in the range from the lower threshold to the upper threshold, there is a certain degree of uncertainty as to whether or not a functional impairment exists. This uncertainty may result from the image recording being of comparatively low quality. It is possible that a command is transmitted to the control unit of the device to generate another image recording in order to also feed this additional image recording to the trained machine learning model for detecting a functional impairment. It is possible that parameters are changed when generating the additional image recording in order to improve the quality of the image recording. For example, the exposure time can be increased and / or the illumination of the image recording area can be increased by one or more illumination units and / or filters (color filters, polarization filters, and / or the like) can be used.Further image acquisition can then provide clarity as to whether or not a functional impairment exists. However, it is also possible that the uncertainty regarding the presence of a functional impairment results from the fact that a functional impairment is only just becoming apparent, i.e. that only a comparatively minor functional impairment exists (e.g. slight contamination). For example, it is possible that the device's control unit is prompted by a command to reduce the time interval between two consecutive image acquisitions. Images are then acquired at shorter intervals in order to detect further functional impairment and / or an increase in the severity of the functional impairment at an early stage. Threshold values ​​can be set by an expert based on their experience. But they can also be set by a user.It is possible for the user to decide for themselves whether they would like to be informed when the probability of a functional impairment is lower, or whether they would rather be informed when the probability of a functional impairment is comparatively high.

[0110] The machine learning model of the present disclosure may be or include an artificial neural network. An "artificial neural network" includes at least three layers of processing elements: a first layer with input neurons (nodes), a kth layer with at least one output neuron (node), and k-2 inner layers, where k is a natural number greater than 2.

[0111] The input neurons are used to receive the input representations. Typically, there is one input neuron for each pixel of an image input to the artificial neural network. Additional input neurons may be present for additional input values ​​(e.g., information about the image acquisition, the device, camera parameters, weather conditions during image generation, and / or the like).

[0112] The output neurons can be used to output information about which class the input image was assigned to and / or with which probability it was assigned to the class.

[0113] The processing elements of the layers between the input neurons and the output neurons are connected in a predetermined pattern with predetermined connection weights.

[0114] The neural network can be trained, for example, using a backpropagation method. The goal is to achieve the most reliable mapping of the input data to the target data for the network. The quality of the prediction is described by an error function. The goal is to minimize the error function. In the backpropagation method, an artificial neural network is trained by changing the connection weights.

[0115] In the trained state, the connection weights between the processing elements contain information regarding the relationship between image recordings and device functional impairments. This information can be used to detect a device functional impairment early on based on a new image recording. The term "new" means that the new image recording was not already used in training the artificial neural network.

[0116] A cross-validation method can be used to split the data into training and validation sets. The training set is used for backpropagation training of the network weights. The validation set is used to test the predictive accuracy of the trained network when applied to unknown (new) data.

[0117] The artificial neural network can be a so-called convolutional neural network (CNN for short) or it can include one.

[0118] A convolutional neural network (“CNN”) is able to process input data in the form of a matrix. This makes it possible to use images represented as a matrix (e.g. width x height x color channels) as input data. A neural network, e.g. in the form of a multi-layer perceptron (MLP), on the other hand, requires a vector as input. This means that in order to use an image as input, the image elements (pixels) of the image would have to be rolled out one after the other in a long chain. This means that multi-layer perceptrons are not able to recognize objects in an image regardless of the object's position in the image. The same object at a different position in the image would have a different input vector.

[0119] A CNN usually consists essentially of filters (convolutional layer) and aggregation layers (pooling layer), which repeat alternately, and at the end of one or more layers of fully connected neurons (dense / fully connected layer).

[0120] Numerous architectures of artificial neural networks are described in the scientific literature that are used to assign an image to a class (image classification). Examples are Xception (see e.g.: F. Chollet: 012110), EfficientNet (see e.g.: T. Mingxing Tan et al. '. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks, arXiv:1905.11946v5), DenseNet (see e.g.: G. Wang et al.'. Study on Image Classification Algorithm Based on Improved DenseNet , Journal of Physics: Conference Series, 2021, 1952, 022011), Inception (see e.g. J. Bankar et al. '.Convolutional Neural Network based Inception v3 Model for Animal Classification, International Journal of Advanced Research in Computer and Communication Engineering, 2018, Vol. 7, Issue 5) and others (see e.g.: M. Tripathi: Analysis of Convolutional Neural Network based Image Classification Techniques. Journal of Innovative Image Processing, 2021, 3, 100-117; K. He et al. '. Deep Residual Learning for Image Recognition, arXiv:1512.03385vl; M. Aamir et al.'. An Optimized Architecture of Image Classification Using Convolutional Neural Network, International Journal of Image, Graphics and Signal Processing, 2019, 11, 30-39).

[0121] The machine learning model of the present disclosure may have such or a similar architecture.

[0122] The machine learning model of the present disclosure may be or include a transformer. A transformer is a model that can translate one sequence of characters into another sequence of characters, taking into account dependencies between distant characters. Such a model can be used, for example, to translate text from one language to another. A transformer includes encoders and decoders connected in series. Transformers have been successfully used to classify images (see, for example, A. Dosovitskiy et al.: An image is worth 16 x 16 Words: Transformers for image recognition at scale, arXiv:2010.11929v2; A. Khan et al.: Transformers in Vision: A Survey, arXiv:2101.01169v5).

[0123] The machine learning model of the present disclosure may have a hybrid architecture in which, for example, elements of a CNN are combined with elements of a transformer.

[0124] The machine learning model of the present disclosure can be initialized using standard methods (random initialization, He initialization, Xavier initialization, etc.). However, it can also be pre-trained based on publicly available, already annotated images (see, for example, https: / / www.image-net.org). The training of the machine learning model can therefore be based on initialization or pre-training and can also include transfer learning, so that only parts of the weights / parameters of the machine learning model are retrained.

[0125] The machine learning model can feature an autoencoder architecture. An autoencoder is an artificial neural network that can be used to learn efficient data encodings in an unsupervised learning process. Generally, the task of an autoencoder is to learn a compressed representation for a dataset and thus extract essential features. This allows it to be used for dimensionality reduction by training the network to ignore "noise." An autoencoder comprises an encoder, a decoder, and a layer between the encoder and decoder that has a lower dimension than the encoder's input layer and the decoder's output layer.This layer (often referred to as bottleneck, encoding, or embedding) forces the encoder to generate a compressed representation of the input data that minimizes noise and is sufficient for the decoder to reconstruct the input data. The autoencoder can therefore be trained to generate a compressed representation of the input image for an input image. For example, the autoencoder can be trained exclusively on images that do not represent any device impairment. However, the autoencoder can also be trained on images that represent devices with and without any device impairment. Once the autoencoder is trained, the decoder can be discarded, and the encoder can be used to generate a compressed representation for each input image.When a new device is used, a first image recording of the device can be generated upon first use. This first image recording, which represents the device without any functional impairment, can be used as a reference. A compressed representation of the first image recording can be generated using the encoder of the trained autoencoder. This compressed representation is the reference representation. During operation of the device, compressed representations of images generated by the image acquisition unit of the device can be generated using the encoder. The more similar a compressed representation is to the reference image recording, the less likely it is that a functional impairment is present. The more a compressed representation differs from the reference image recording, the more likely it is that a functional impairment is present.The similarity of representations can be quantified using a similarity or distance measure. Examples of such similarity or distance measures are cosine similarity, Manhattan distance, Euclidean distance, Minkowski distance, Lp norm, and Chebyshev distance. If a distance measure exceeds or falls below a predefined threshold, which can be set by an expert or specified by a user, a notification can be issued indicating a functional impairment and / or the image capture can be discarded.

[0126] An example of an autoencoder architecture is the U-Net (see, for example, O. Ronneberger et al.-. U-net: Convolutional networks for biomedical image segmentation, International Conference on Medical image computing and computer-assisted intervention, pages 234-241, Springer, 2015, https: / / doi.org / 10.1007 / 978-3-319-24574-4_28).

[0127] It is also possible to use an architecture for the machine learning model of the present invention as described, for example, in the following publication: J. Dippel, S. Vogler, S. Höhne: Towards Fine-grained Visual Representations by Combining Contrastive Learning with Image Reconstruction and Attention-weighted Pooling, arXiv:2104.04323v2. The autoencoder described therein comprises, in addition to an encoder and a decoder, a strand (projection head) which, based on the compressed representation generated by the encoder, generates an output indicating whether the image recording represents a device with a functional impairment or without a functional impairment.In other words, the autoencoder is not only trained to generate a compressed representation of the input data and reconstruct the input data based on the compressed representation, but the autoencoder is also simultaneously trained to distinguish image recordings from devices with functional impairments from image recordings from devices without functional impairments (contrastive reconstruction). Once the autoencoder is trained, the encoder can be used, for example, to generate a compressed reference representation for a first image recording of a new device being used for the first time. This compressed reference representation is compared with compressed representations of image recordings generated during device operation, and in the event of a defined deviation, a notification is issued indicating a functional impairment.It is also possible to use the encoder together with the projection operator (projection head) directly for classification. Further techniques for classifying images are described, for example, in SVS Prasad et al.: Techniques in Image Classification - A Survey, Global Journal of Researches in Engineering (F), 2015, Volume XV, Issue VI, Version I, 17-32; K. Sanghvi et al.: A Survey on Image Classification Techniques, 2020, http: / / dx.doi.org / 10.2139 / ssrn.3754116 and can also be used to implement the present invention.

[0128] The invention is explained in more detail below with reference to drawings, without wishing to limit the invention to the features and combinations of features shown in the drawings.

[0129] Fig. 1 shows a schematic and exemplary training process for a machine learning model. The machine learning model is trained using training data TD. The training data TD comprises a plurality of image recordings. Each image recording I shows at least part of a field and optionally at least part of the device (not shown in Fig. 1). The training data TD further comprises, for each image recording I, information A indicating whether the device depicted in the image recording I has a functional impairment or whether it does not have a functional impairment. It is possible for the information A to comprise information about which specific functional impairment is present in the individual case and / or how severe the functional impairment is and / or the degree of severity of the functional impairment. In Fig.1, for the sake of clarity, only one training data set comprising an image I with information A is shown; however, the training data comprises a large number of such training data sets. The image I represents input data for the MLM machine learning model. The information A represents target data for the MLM machine learning model. The image I is fed to the MLM machine learning model. The MLM machine learning model assigns the image to one of at least two classes. The assignment is made based on the image I and on the model parameters MP. The MLM machine learning model outputs information O that indicates the class the image has been assigned to and / or the probability that the image has been assigned to one or more of the at least two classes. The output information O is compared with the information A.An error function LF is used to quantify the deviations between the information O (output) and the information A (target data). For each pair of information A and information O, an error value LV can be calculated. The error value LV can be reduced in an optimization procedure (e.g., a gradient method) by modifying model parameters MP. The goal of training can be to reduce the error value for all image acquisitions to a predefined minimum. Once the predefined minimum is reached, training can be terminated.

[0130] Fig. 2 schematically shows the use of a trained machine learning model to detect a functional impairment in a device with an image capture and image recognition unit of plants. The trained model MLM 1of machine learning may have been trained in a training procedure as described in relation to Fig. 1. The trained model MLM 1 of machine learning, a new image I* is fed. The new image I* shows at least part of a field and optionally at least part of the device. The model assigns the new image I* to one of the at least two classes for which the trained model MLM 1 of machine learning. The trained model MLM 1 The machine learning algorithm outputs information O that indicates which class the image was assigned to and / or the probability that the image was assigned to one or more of the at least two classes. The information O can be output to a user.

[0131] Fig. 3 schematically shows another example of training a machine learning model. The machine learning model has an autoencoder architecture. The autoencoder AE comprises an encoder E and a decoder D. The encoder is configured to generate a compressed representation CR for an image recording I based on model parameters MP. The decoder is configured to generate a reconstructed image recording RI based on the compressed representation CR and model parameters MP, which is as close as possible to the image recording I. An error function LF can be used to quantify deviations between the image recording I and the reconstructed image recording RI. The deviations can be minimized in an optimization process (e.g., in a gradient method) by modifying model parameters MP.The autoencoder AE is typically trained using an unsupervised learning process based on a large number of image recordings. Only one image recording I of the large number of image recordings is shown in Fig. 3. Each image recording of the large number of image recordings shows at least a portion of a field and optionally at least a portion of the device. The components of the device(s) may have one or more functional impairments or be free of functional impairments.

[0132] Fig. 4 schematically shows another example of using a trained machine learning model to detect a functional impairment in a device with an image acquisition and an image recognition unit of plants. The trained machine learning model can have been trained using a training method as described with reference to Fig. 3. The trained machine learning model can be an encoder E of an autoencoder. The encoder E is shown twice in Fig. 4; however, it is the same encoder in both cases; it is shown twice merely to illustrate the recognition method. In a first step, a first image recording Ii* of an image recording area is fed to the encoder E. The asterisk * indicates that the image recording L* was not used to train the machine learning model.The image recording L* is preferably an image recording that represents a device without any functional impairment, which can, for example, have been generated after the first use of a new device. The encoder is configured to generate a first compressed representation CRi for the first image recording L*. The first compressed representation CRi can be used as a reference representation. It can be stored in a data memory. During operation of the device, further image recordings of the image recording area are generated. Fig. 4 shows one of these further image recordings, the image recording L*. The image recording I2* is also fed to the encoder E. The encoder E generates a second compressed representation CR2 for the image recording I2*. The first representation CRi and the second representation CR2 are compared with each other in a next step.During this comparison, a distance measure D is calculated that quantifies the differences between the first representation CRi and the second representation CR2. In a next step, the distance measure is compared with a predefined threshold value T. If the distance measure is greater than the predefined threshold value T (, >"), a message M is output. The message M includes information that the device shown in the image recording I2* has a functional impairment. If the distance measure is not greater than the predefined threshold value T ("n"), the image recording I2* is subjected to an analysis DCI(l2*) in order to detect plants in the field in the image recording range of the device and / or to identify the plants located in the image recording range.

[0133] Fig. 5 shows an example and schematically a computer-implemented method for training a machine learning model in the form of a flow chart.

[0134] The training procedure (100) includes the following steps:

[0135] (HO) Receiving and / or providing training data, wherein the training data comprises input data and target data, o wherein the input data show a plurality of image recordings of at least part of a field and optionally show at least part of the device with an image acquisition and an image recognition unit of plants, o wherein the target data comprise a class assignment for each image recording, wherein the class assignment indicates which class of at least two classes the image recording is assigned to, wherein at least a first class represents image recordings of devices that do not have a functional impairment, and at least a second class represents image recordings of devices that have a functional impairment,

[0136] (120) Providing a machine learning model, wherein the machine learning model is configured to assign the image recording to one of the at least two classes based on an image recording and on the basis of model parameters,

[0137] (130) Training the machine learning model with the training data, wherein the training comprises for each image acquisition:

[0138] (131) Feeding the image capture to the machine learning model,

[0139] (132) receiving an output from the machine learning model, the output indicating which class the machine learning model has assigned the image recording to and / or with what probability the machine learning model has assigned the image recording to one or more of the at least two classes,

[0140] (133) Determining a deviation between the output and the class assignment,

[0141] (134) Minimizing the deviation by modifying the model parameters,

[0142] (140) storing and / or outputting the trained machine learning model and / or using the trained machine learning model to detect a functional impairment in a device having an image capture and an image recognition unit of plants.

[0143] Fig. 6 shows an example and schematically a computer-implemented method for detecting a functional impairment in a device with an image capture and an image recognition unit of plants.

[0144] The recognition process (200) comprises the following steps:

[0145] (210) receiving an image recording, wherein the received image recording shows at least a part of a field and optionally also shows at least a part of the device and was recorded by the image acquisition unit of the device,

[0146] (220) Providing a trained machine learning model, wherein the machine learning model is configured and trained on the basis of training data to assign image recordings to one of at least two classes, wherein the training data comprises input data and target data,

[0147] • wherein the input data comprises a plurality of image recordings, each showing at least part of a field and optionally showing at least part of the device,

[0148] • wherein the target data for each image recording comprises a class assignment, wherein the class assignment indicates which class of at least two classes the image recording is assigned to, wherein at least a first class represents image recordings from the device with an image capture and an image recognition unit of plants that do not have a functional impairment, and at least a second class represents image recordings from devices with an image capture and an image recognition unit of plants that have a functional impairment, (230) feeding the image recording to a trained machine learning model,

[0149] (240) Receiving information from the machine learning model as to which class the image recording was assigned,

[0150] (250) wherein the notification that the device with an image capture and an image recognition unit of plants has a functional impairment is output when the received image recording has been assigned to one of the at least one second class.

[0151] The steps, methods and / or functions described in this disclosure may be performed in whole or in part by a computer system.

[0152] A "computer system" is an electronic data processing system that processes data using programmable computing instructions. Such a system typically includes a "computer," the unit that includes a processor for performing logical operations, and peripherals.

[0153] In computer technology, "peripherals" refers to all devices connected to a computer that serve to control the computer and / or act as input and output devices. Examples include monitors, printers, scanners, mice, keyboards, drives, cameras, microphones, speakers, etc. Internal connectors and expansion cards are also considered peripherals in computer technology.

[0154] Today's computer systems are often divided into desktop PCs, portable PCs, laptops, notebooks, netbooks, and tablet PCs, as well as so-called handheld devices (e.g., smartphones); all of these systems can be used to implement the invention.

[0155] The term "computer" should be broadly interpreted to include any type of electronic device with data processing capabilities, including, as non-limiting examples, personal computers, servers, embedded cores, communications devices, processors (e.g., digital signal processors (DSPs), microcontrollers, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc.) and other electronic computing devices.

[0156] The term "process," as used above, is intended to encompass any type of calculation, manipulation, or transformation of data that is represented as physical, e.g., electronic, phenomena and that may occur or be stored, e.g., in registers and / or memory of at least one computer or processor. The term "processor" includes a single processing unit or a plurality of distributed or remote processing units.

[0157] Fig. 7 shows an exemplary and schematic representation of an embodiment of a computer system of the present disclosure. The computer system (1) comprises an input unit (10), a control and computing unit (20), and an output unit (30).

[0158] The control and computing unit (20) is configured to cause the input unit to receive an image recording, wherein the received image recording shows at least part of a field and optionally also at least part of a device with an image acquisition and image recognition unit for plants and was recorded by a device with an image acquisition and image recognition unit for plants, to feed the received image recording to a trained machine learning model, wherein the machine learning model is configured and has been trained on the basis of training data to assign image recordings to one of at least two classes, wherein the training data comprises input data and target data, o wherein the input data comprises a plurality of image recordings, each showing at least part of a field and optionally showing at least part of the device, o wherein the target data comprises a class assignment for each image recording,wherein the class assignment indicates which class of at least two classes the image recording is assigned to, wherein at least a first class represents image recordings of devices that do not have a functional impairment and at least a second class represents image recordings of devices that have a functional impairment, to receive information from the machine learning model as to which class of the at least two classes the image recording was assigned to and / or with what probability the image recording was assigned to one or more of the at least two classes, to cause the output unit to output a message that the device has a functional impairment if the image recording was assigned to one of the at least one second class.

[0159] Fig. 8 shows an exemplary and schematic illustration of another embodiment of a computer system of the present disclosure.

[0160] The computer system (1) comprises a processing unit (20) connected to a memory (50).

[0161] The processing unit (20) may comprise one or more processors alone or in combination with one or more memories. The processing unit (20) may be conventional computer hardware capable of processing information such as digital images, computer programs, and / or other digital information. The processing unit (20) typically consists of an arrangement of electronic circuits, some of which may be embodied as an integrated circuit or as multiple interconnected integrated circuits (an integrated circuit is sometimes referred to as a "chip"). The processing unit (20) may be configured to execute computer programs that may be stored in a main memory of the processing unit (20) or in the memory (50) of the same or another computer system.

[0162] The memory (50) may be ordinary computer hardware capable of storing information such as digital images (e.g., representations of the examination area), data, computer programs, and / or other digital information either temporarily and / or permanently. The memory (50) may comprise volatile and / or non-volatile memory and may be permanently installed or removable. Examples of suitable memories include RAM (Random Access Memory), ROM (Read-Only Memory), a hard disk, flash memory, a removable computer diskette, an optical disc, magnetic tape, or a combination of the above. Optical discs may include read-only compact discs (CD-ROM), read / write compact discs (CD-R / W), DVDs, Blu-ray discs, and the like.

[0163] In addition to the memory (50), the processing unit (20) can also be connected to one or more interfaces (11, 12, 30, 41, 42) for displaying, transmitting, and / or receiving information. The interfaces can comprise one or more communication interfaces (41, 42) and / or one or more user interfaces (11, 12, 30). The one or more communication interfaces (41, 42) can be configured to send and / or receive information, e.g., to and / or from other computer systems, networks, data storage devices, or the like. The one or more communication interfaces (41, 42) can be configured to transmit and / or receive information via physical (wired) and / or wireless communication connections. The one or more communication interfaces (41, 42) can include one or more interfaces for connecting to a network, e.g.,using technologies such as cellular, Wi-Fi, satellite, cable, DSL, fiber optic, and / or the like. In some examples, the one or more communication interfaces (41, 42) may include one or more short-range communication interfaces configured to connect devices using short-range communication technologies such as NFC, RFID, Bluetooth, Bluetooth LE, ZigBee, infrared (e.g., IrDA), or the like.

[0164] The user interfaces (11, 12, 30) may comprise a display (30). A display (30) may be configured to display information to a user. Suitable examples include a liquid crystal display (LCD), a light-emitting diode display (LED), a plasma display (PDP), or the like. The user input interface(s) (11, 12) may be wired or wireless and may be configured to receive information from a user into the computer system (1), e.g., for processing, storage, and / or display. Suitable examples of user input interfaces (11, 12) include a microphone, an image or video capture device (e.g., a camera), a keyboard or keypad, a joystick, a touch-sensitive surface (separate from or integrated with a touchscreen), or the like.In some examples, the user interfaces may include automatic identification and data capture (AIDC) technology for machine-readable information. This may include barcodes, radio frequency identification (RFID), magnetic stripes, optical character recognition (OCR), integrated circuit cards (ICC), and the like. The user interfaces may further include one or more interfaces for communicating with peripheral devices such as printers and the like.

[0165] One or more computer programs (60) may be stored in memory (50) and executed by the processing unit (20), which is thereby programmed to perform the functions described in this description. The retrieval, loading, and execution of instructions of the computer program (60) may occur sequentially, such that one instruction is retrieved, loaded, and executed at a time. However, the retrieval, loading, and / or execution may also occur in parallel.

Claims

Patent claims 1. A computer-implemented method for a device having an image capture and an image recognition unit of plants, comprising: Receiving an image recording, wherein the received image recording shows at least part of a field and was recorded by the image acquisition unit of the device, Providing a trained machine learning model, wherein the machine learning model is configured and trained on the basis of training data to distinguish image recordings of at least a part of a field with a functional impairment of the device with an image capture and an image recognition unit of plants from image recordings of a part of a field without a functional impairment of the device with an image capture and an image recognition unit of plants, Feeding the received image to the trained machine learning model, Receiving information from the machine learning model, the information indicating whether the image recording of at least part of a field indicates a functional impairment of the device with an image capture and image recognition unit of plants, in the event that the received image recording of at least part of a field indicates a functional impairment of the device with an image capture and image recognition unit of plants: outputting a notification that the received image recording indicates a functional impairment of the device with an image capture and image recognition unit.

2. A method according to claim 1, wherein the device having an image capture and an image recognition unit of plants is a spraying device.

3. A method according to claim 2, comprising: Receiving an image recording, wherein the received image recording shows at least part of the spray device and at least part of a field and was recorded by the image acquisition unit of the device, Providing a trained machine learning model, wherein the machine learning model is configured and trained on the basis of training data to distinguish image recordings of at least a part of the spraying device and at least a part of the field with a functional impairment of the spraying device from image recordings of at least a part of the spraying device and at least a part of the field without a functional impairment of the spraying device, Feeding the received image to the trained machine learning model, Receiving information from the machine learning model, the information indicating whether the image recording of at least part of the spray device and at least part of the field indicates a functional impairment of the spray device, in the event that the received image recording of at least part of the spray device and at least part of the field indicates a functional impairment of the spray device: issuing a notification that the spray device shown at least partially in the received image recording has a functional impairment.

4. The method according to claim 1 to 3, wherein the machine learning model is configured and has been trained on the basis of training data to assign image recordings to one of at least two classes, wherein the training data comprises input data and target data, o wherein the input data comprises a plurality of image recordings, each showing at least a part of a field and optionally showing at least a part of the device with an image capture and an image recognition unit of plants, o wherein the target data comprises a class assignment for each image recording, wherein the class assignment indicates which class of at least two classes the image recording is assigned to, wherein at least a first class represents image recordings from the device with an image capture and an image recognition unit of plants that do not have a functional impairment,and at least one second class represents image recordings from devices with an image capture and an image recognition unit of plants that have a functional impairment, wherein receiving information from the machine learning model comprises: o receiving information from the machine learning model as to which class the image recording was assigned, wherein the notification that the device with an image capture and an image recognition unit of plants has a functional impairment is output if the received image recording was assigned to one of the at least one second class.

5. The method of claim 4, wherein training the machine learning model for each image capture of the plurality of image captures comprises: Inputting the image into the machine learning model, wherein the machine learning model is configured to assign the image to one of at least two classes based on the input image and model parameters, Receiving an output from the machine learning model, the output indicating which of the at least two classes the input image recording was assigned to by the machine learning model, Quantifying a deviation between the output and the class assignment of the training data, Minimize the deviation by modifying model parameters.

6. The method according to any one of claims 1 to 5, wherein the machine learning model is configured and trained to assign the received image recording to one of a plurality of classes, wherein each class of a plurality of classes has a specific functional impairment, wherein outputting the notification comprises: outputting a notification which specific functional impairment is present in the device having an image capture and an image recognition unit of plants.

7. Method according to one of claims 1 to 6, wherein the machine learning model is configured and trained to output, based on the received image recording for one or more specific functional impairments, a probability with which the specific functional impairment occurs, wherein outputting the message comprises: outputting a message with which probability the one or more specific functional impairments are present in the device with an image capture and an image recognition unit of plants.

8. The method according to any one of claims 1 to 7, wherein the machine learning model is configured and trained to output a severity level for one or more specific functional impairments based on the received image recording, wherein outputting the message comprises: outputting a message indicating the severity level of the one or more specific functional impairments in the device having an image capture and an image recognition unit of plants.

9. The method according to any one of claims 1 to 8, wherein the functional impairment and / or the one or more specific functional impairments of the device are selected from the following list: Field of view of the image acquisition unit is limited, Image capture unit and / or its optical elements are dirty, Image capture unit is defective, Image acquisition unit does not produce images of at least part of a field, Lighting source(s) defective and / or dirty, unwanted reflections occur.

10. The method according to any one of claims 2 to 9, wherein the functional impairment and / or the one or more specific functional impairments of the device are selected from the following list: Image capture unit does not produce images of at least part of the spray device, Position and / or location of components of the spray device are changed, Plant parts have become caught in the spray device, The spray cone of the spray nozzles of the spray device does not meet the specifications or the spray nozzles are defective.

11. The method according to any one of claims 1 to 10, wherein the machine learning model is configured and trained to generate a compressed representation for the received image, the method further comprising: o Quantifying a similarity and / or a difference between the compressed representation and a reference representation by calculating a similarity measure and / or a distance measure, wherein outputting the notification comprises: outputting the notification that the device having an image capture and an image recognition unit of plants has a functional impairment if the similarity measure is below a predefined threshold and / or the distance measure is above a predefined threshold.

12. The method according to any one of claims 2 to 11, wherein the method further comprises: exclusively in the event that the received image recording shows a spraying device without functional impairment: detecting and / or identifying plants in the field on the basis of the received image recording with the image recognition unit for plants, spraying a plant protection agent onto the detected and / or identified plants with the spraying device.

13. The method according to any one of claims 1 to 12, wherein the notification further comprises one or more of the following information: Location of the image capture unit with which the received image of the device with a functional impairment was taken, Information about the functional impairment, Information about the severity of the impairment, Information on what measures can be taken to restore full functionality of the device, Information on when action should be taken to prevent further impairment of function, the received image recording.

14. A computer system comprising: an input unit, a control and computing unit and an output unit, wherein the control and computing unit is configured to cause the input unit to receive an image recording, wherein the received image recording shows at least part of a field and optionally also at least part of a device with an image acquisition and image recognition unit for plants and was recorded by a device with an image acquisition and image recognition unit for plants, to feed the received image recording to a machine learning model, wherein the machine learning model is configured and has been trained on the basis of training data, to receive image recordings of at least part of a field and optionally at least part of a device with an image acquisition and image recognition unit for plants with a To distinguish a functional impairment of the device with an image capture and image recognition unit for plants from image recordings of at least part of a field and optionally at least part of a device with an image capture and image recognition unit for plants without a functional impairment of the device with an image capture and image recognition unit for plants, To receive information from the machine learning model, wherein the information indicates whether the received image recording represents a device with an image capture and image recognition unit for plants with a functional impairment, To cause the output unit to output a message, wherein the message indicates that the received image recording indicates a functional impairment of the device with an image capture and image recognition unit for plants,if the information output by the machine learning model indicates that the received image represents a device with an image acquisition and image recognition unit for plants with a functional impairment., 15. A non-transitory computer-readable storage medium having stored thereon software instructions which, when executed by a processor of a computer system, cause the computer system to carry out the method according to any one of claims 1 to 13.

16. A spraying device with an image capture and image recognition unit for plants comprising: • a control unit, • an analysis unit, • an output unit • at least one image acquisition unit, wherein the control unit is configured to cause the at least one image acquisition unit to generate one or more image recordings of at least a part of a field and optionally of at least a part of the spraying device, wherein the analysis unit is configured to distinguish the one or more generated image recordings of at least a part of the field and optionally of at least a part of the spraying device with a functional impairment of the spraying device from image recordings of at least a part of a field and optionally of at least a part of the spraying device without a functional impairment of the spraying device, wherein the analysis unit is configured to receive information from the machine learning model, wherein the information indicates whether the one or more image recordings represent the spraying device with a functional impairment,wherein the output unit is configured to output the information.,

Images