Method, apparatus and system for assisting with the analysis of a plant, and method for providing an evaluation circuit therefor

Abstract

The invention relates to an apparatus (10) or to a system which can assist with the analysis of a plant or a plurality of plants (2). The apparatus (10) or the system has an image capturing device (21) which is designed to capture at least one image (60) of at least one part of the plant or the plurality of plants (2). The apparatus (10) or the system has a human-machine interface (24) which is designed to output information which is obtained on the basis of the at least one image (60) and which indicates at least one area suitable for a measurement for the purpose of analyzing the plant or the plurality of plants (2).

Description

[0001] Method, device and system for supporting an examination of a plant and method for providing an evaluation circuit therefor

[0002] TECHNICAL FIELD

[0003] The invention relates to devices, systems, and methods that can be used in connection with the examination of a plant. The invention particularly relates to such devices, systems, and methods that can support a more detailed examination of the plant based on images. The invention particularly relates to such devices, systems, and methods that can be used with crop plants.

[0004] BACKGROUND

[0005] Assessing the condition of plants is of great importance. It serves to ensure adequate nutrient supply and / or to detect biotic or abiotic stress.

[0006] Conventional methods for assessing plant health have traditionally relied on human expert assessment. Such techniques, which involve quality assessment by human experts, are also known as visual assessment. These techniques have the disadvantage of being prone to human error and not based on objective data. Techniques that use measuring devices to objectively and quantitatively analyze plant health are becoming increasingly important. For example, such techniques can be used to detect over- or under-supply of certain nutrients.

[0007] Optical sensors that can be used in the field offer the advantage of quickly providing analysis results. Examples of such devices are known, for example, from US 11 320 307 B2. The techniques described in US 11 320 307 B2 require a spectrometer and cannot assist a user in selecting one or more samples from a multitude of possible samples.

[0008] Thus, there is still a need for devices, systems and methods that offer improvements in the agriculturally and industrially important examination of plants.

[0009] SUMMARY

[0010] The invention is based on the object of providing improved devices, systems, and methods that support the examination of plants. In particular, the invention is based on the object of providing devices, systems, and methods that support a user in selecting one or more samples from a multitude of possible samples. According to the invention, devices, systems, and methods are provided as defined in the independent claims. The dependent claims define preferred and advantageous embodiments.

[0011] According to one aspect, the invention relates to a device or a system for supporting the examination of one or more plants. The device or the system comprises an image capture device configured to capture at least one image of at least part of the plant or of the multiple plants. The device or the system comprises a human-machine interface configured to output information determined from the at least one image, which indicates at least one area suitable for a measurement for the examination of the plant or of the multiple plants.

[0012] The device or system assists a user in selecting one or more samples from a multitude of possible samples. The examination of the plant(s) is thus supported, for example, at least partially guided, by the device.

[0013] The device or system may be configured such that the information indicates which measurement areas are suitable for a spectral analytical, camera-based or laboratory-based examination of the plant or of the plurality of plants.

[0014] This informs the user visually or in another way which samples (e.g. which leaves or which leaf areas) are suitable for the respective examination.

[0015] The device or system may be configured such that the information indicates which ranges are suitable for determining a nutrient concentration, an active ingredient concentration, and / or a health status of the plant.

[0016] This provides the user with visual or other information about which samples (e.g. which leaves or leaf areas) are suitable for the respective objective and / or measurement method of the investigation.

[0017] The device or system may be configured to determine and output the information depending on the examination to be performed.

[0018] This allows the user to be specifically supported with regard to the type of examination to be conducted (e.g., the nutrients to be examined and / or the type of biotic or abiotic stress to be tested with the examination, e.g., one or more diseases to be tested). The evaluation of the at least one image can be specifically carried out with regard to which samples (e.g., which leaves or which leaf regions) are suitable for the corresponding examination. The human-machine interface can be configured to receive user input that specifies the examination to be conducted (e.g., an objective of the examination to be conducted and / or a type of measurement).

[0019] This allows the user to receive targeted support with regard to the type of study to be conducted (e.g., an objective such as the nutrients to be examined and / or the type of biotic or abiotic stress to be tested, e.g., one or more diseases to be tested, and / or a measurement method such as image data acquisition or spectral analysis data). The evaluation of at least one image can be carried out specifically with regard to which samples (e.g., which leaves or leaf areas) are suitable for the study specified by the user.

[0020] The device or system may comprise an evaluation circuit configured to determine the information to be output based on the at least one image.

[0021] This allows for automatic, image-based, and thus non-destructive, determination of which samples (e.g., which leaves or leaf regions) are suitable for the respective examination. The device or system also allows for contactless determination of which samples (e.g., which leaves or leaf regions) are suitable for the respective examination. This is helpful in preventing the transmission of diseases.

[0022] The evaluation circuit can be configured to determine the information to be output depending on one, several or all of the following information: plant stage, leaf stage, plant species, plant variety, at least one nutrient to be analyzed, at least one active ingredient to be analyzed, at least the type of biotic or abiotic stress (e.g. disease(s)) to be tested with the test, position of the leaf on the plant, measurement method (e.g. image-based, spectral analysis or laboratory-based), evaluation technique of the measurement results.

[0023] This allows one, several or all of the above-mentioned information, which may influence the appropriate selection of samples, to be taken into account.

[0024] The evaluation circuit can be configured to determine the plant species, the plant variety, the plant stage, the leaf stage and / or the position of the leaf on the plant based on an evaluation of the at least one image.

[0025] This allows the evaluation circuit to determine the appropriate sample position(s) for the automatically determined parameters (such as the plant species, plant variety, plant stage, leaf stage, and / or the position of the leaf on the plant) and for the respective investigation (e.g., objective and / or measurement method). The evaluation circuit can be configured to determine, based on an evaluation of the at least one image, whether an abaxial or adaxial leaf side is depicted in the at least one image.

[0026] This allows the determination of the appropriate sample position(s) depending on how the leaves are positioned and whether the measurement should be taken on the abaxial or adaxial side for the respective study.

[0027] The evaluation circuit can be configured to control the human-machine interface to enable user input specifying one or more of the following information: plant stage, leaf stage, plant species, plant variety, at least one nutrient to be analyzed, at least one active ingredient to be analyzed, at least one type of biotic or abiotic stress (e.g., disease(s) to be tested or other health status tests) to be tested with the test, position of the leaf on the plant, measurement method (e.g., image-based, spectral analysis, or laboratory-based), evaluation technique of the measurement results.

[0028] This allows the user to define for which plants, leaves and tests to be carried out the appropriate sample position(s) for the measurement are to be determined.

[0029] The evaluation circuit can be configured to evaluate the at least one image with at least one data-driven trained image evaluation in order to determine the information.

[0030] This allows the appropriate sample positions to be determined using an objective, data-based image analysis.

[0031] The evaluation circuit can be configured to evaluate the at least one image with at least one trained machine learning model to determine the information.

[0032] This allows the appropriate sample positions to be determined using a data-based machine learning model. There is no need for image analysis to be performed by a human expert.

[0033] The at least one machine learning model can be configured to receive as input pixel values ​​of the at least one image or input values ​​determined from preprocessing of the pixel values.

[0034] This allows the appropriate sample positions to be determined in a simple manner, for example by processing an RGB image.

[0035] The evaluation circuit can be set up to evaluate the image at least one

[0036] Support vector machines, edge- or region-based segmentation, or other techniques can be used. This allows the appropriate sample positions to be determined using well-established image analysis techniques.

[0037] The evaluation circuit can be configured to carry out an automatic detection of leaves based on the at least one image in order to determine the information to be output.

[0038] This allows the boundaries of leaves to be determined automatically for examinations to be carried out on leaves.

[0039] The device or system can be configured such that the information indicates both leaves suitable for measurement and leaf regions of the leaves suitable for measurement. The indication of leaf regions can include an indication of suitable leaf sides (abaxial or adaxial sides).

[0040] This allows the sample areas to be determined for tests to be carried out on leaves and made available to the user.

[0041] The device or system may be configured to output the information as information superimposed on the at least one image in a visually perceptible manner via the human-machine interface.

[0042] This makes it easy to assign the appropriate sample areas to the real world in which the measurement for the investigation is carried out.

[0043] The device or system may be configured to visualize the at least one suitable region in different ways, for example as a filled region, as a bordered region, as a semi-transparent region, by markings (for example crosses, arrows), by alphanumeric information.

[0044] This allows for the assignment of suitable sample areas to the real world in a variety of ways.

[0045] The device or system may be configured to control the human-machine interface such that a captured image is displayed as a still image and the information is displayed superimposed on the still image.

[0046] This allows the sample areas to be assigned to the real world in which the data for the investigation is recorded in a computationally efficient manner.

[0047] The device or system can be configured to control the human-machine interface such that, upon movement of the image capture device relative to the plant or multiple plants, the image displayed on the human-machine interface is updated and the information is tracked. The human-machine interface can be continuously controlled by an evaluation circuit such that the information displayed on the human-machine interface is tracked according to the orientation and position of the image capture device relative to the plant, plants, and / or leaves. This allows the sample areas to be assigned to the real world in which the data for the examination is recorded in a particularly easy-to-understand manner. The risk of an erroneous measurement is reduced.

[0048] The device or system may comprise a housing in which the image capture device and the human-machine interface are mounted. The device may be designed as a movable, particularly manually portable, device.

[0049] This makes handling particularly easy.

[0050] The image capture device may comprise a camera.

[0051] This allows the determination of the at least one region using robust and compact components. In particular, no spectral analysis detection device is required to determine the at least one region suitable for subsequent measurement.

[0052] The camera can have two or more color channels. For example, the camera can have an RGB camera.

[0053] This allows the information contained in the multiple color channels (for example, about the reflection spectrum of the plant(s)) to be used to determine at least one area.

[0054] The device or system may include imaging optics for capturing the at least one image at a distance from the plant or the plurality of plants. The distance may be within a distance interval that may depend on the respective plant species.

[0055] This allows a larger area to be imaged and used to identify areas suitable for measurement. It is also possible to identify a suitable area within the sheet by controlling the imaging optics (e.g., by zooming).

[0056] The imaging optics can be controllable, for example for a zoom operation.

[0057] This can make it easier to identify a suitable area within a sheet.

[0058] The evaluation circuit can determine the information depending on further factors, optionally also independent of the at least one image. For example, the evaluation circuit can be configured to generate the information representing a measurement recommendation depending on environmental parameters (e.g., an ambient temperature), time, lighting conditions, and / or location coordinates (such as coordinates determined using a global navigation satellite system (GNSS)).

[0059] This allows additional factors influencing the measurement recommendation to be taken into account. This is particularly relevant if the measurement position for the respective study is influenced by such factors (e.g., the time of day or the ambient temperature). Determining the information representing the measurement recommendation based on location coordinates can, for example, facilitate the re-locating of a measurement point and / or the re-locating of a previously identified measurement point.

[0060] The device or system may further comprise a storage system. The storage system may be integrated into the housing containing the image capture device. The storage system may comprise parameters of image analysis techniques to be applied, with which plant regions (e.g., central leaf regions in the center of leaves, leaf edges, leaf regions surrounding leaf stems, leaves arranged at the edges or centrally in a group of leaves) are automatically identified in the at least one image. The storage system may comprise allocation data that specify which type of plant regions are suitable for measurement for various examinations. The evaluation circuit can access the allocation data stored in the storage system to determine which type of plant regions are suitable for the examination.The evaluation circuit can access the storage system to perform the image evaluation based on the parameters (e.g., parameters of a trained machine learning model) and depending on the type of plant areas. The evaluation circuit can generate the information indicating the at least one suitable area based on both the mapping data for the desired examination (e.g., specified by the user) and the result of the image evaluation.

[0061] This allows for efficient measurement recommendations to be made depending on the specific study. If the device or system is configured for use with multiple plant species or varieties, the mapping data and image analysis parameters can be stored in the memory system for each plant variety or species. Selection can be made automatically based on image recognition or based on user input.

[0062] The evaluation circuit can be configured to logically combine the results of multiple image evaluation steps, for example, using a logical AND or a logical AND NOT operation. This can be achieved by pixel-by-pixel multiplication of different image evaluation steps (which can, for example, be selected from a detection of central leaf areas in the center of leaves, a detection of leaf edges, a detection of leaf areas in the vicinity of leaf stems, or a detection of leaves arranged at the edges or centrally in a leaf group).

[0063] This allows different results of the image analysis steps to be logically linked if the assignment data specify that the suitability for conducting an investigation requires the cumulative presence of several properties (for example, a middle-sized arrangement of the leaf in a leaf group logically AND-linked with a measurement in the center or at the edge or near the leaf stem).

[0064] The device or system may be configured to output the information in real time via the human-machine interface.

[0065] This allows the measurement to be carried out while the user is still on site at the plant.

[0066] The device or system can be configured to capture the at least one image in response to a trigger. Image analysis and provision of the information can also occur at a later time.

[0067] This allows suitable measurement areas to be visualized at a later time, for example, shortly after the image has been captured in a more ergonomically comfortable position for the user. This design also has the effect of eliminating the need for continuous image analysis and provision (e.g., continuously or at repeated intervals), which reduces power consumption and thus increases the duration of application while simultaneously reducing operating costs.

[0068] The evaluation circuit of the device or system can be configured to selectively perform only those image evaluation steps that are required depending on user inputs defining the plant species and the examination to be carried out.

[0069] This allows the information to be identified and provided efficiently.

[0070] The system may further comprise a measuring device that is movable relative to the housing of the image capture device. The measuring device may be configured to take a measurement in the at least one suitable area.

[0071] This allows the system to automatically determine which area or areas within the field of view of the image capture device are suitable for measurement. This information is then used to measure and examine the plant. The measuring device can be controlled automatically or semi-automatically. For example, an actuator can be controlled to position the measuring device so that the measurement is taken in at least one suitable area.

[0072] The system may be configured to evaluate the measurements acquired by the measuring device to carry out an investigation with regard to: checking one or more nutrient concentrations, checking the presence or absence of one or more disease states, detecting biotic and / or abiotic stress.

[0073] This allows the examination to be completed. In this case, the system is a system configured to perform the examination. The measuring device can include a spectrometer, a hyperspectral camera, or another camera.

[0074] This allows for the detection of nutrient concentrations, diseases, biotic stress and / or abiotic stress.

[0075] The measuring device may include a device for physically taking samples from the appropriate area.

[0076] This allows for further analysis, optionally including laboratory analysis, based on the sample taken.

[0077] The device can be designed as a movable device.

[0078] This enables its use in the field on living plants.

[0079] The device can be designed as a manually held device.

[0080] This facilitates use in the field on living plants without necessarily having vehicle access to the plant.

[0081] The device can be attached to a vehicle, in particular an agricultural vehicle. Accordingly, according to one embodiment, a vehicle is provided with the device according to the invention mounted thereon.

[0082] This facilitates use in the field on living plants using a vehicle.

[0083] The system may comprise a robot (e.g., a multi-axis robot, e.g., a multi-axis robot arm) that positions the measuring device relative to the plant or the plurality of plants depending on the at least one area to perform the measurement.

[0084] This allows the measurement recommendation to be implemented automatically. The measuring robot can be controlled selectively based on user confirmation and / or a user selection from several suitable areas.

[0085] The device can be attached to a flying object or robot. Accordingly, according to one embodiment, a flying object or robot is provided that has the device according to the invention.

[0086] This facilitates use in the field on living plants using the flying object or robot.

[0087] The vehicle, robot or flying object may comprise a receiver for receiving remote control signals and at least one actuator for moving the vehicle, robot or flying object in response to the remote control signals for positioning the device.

[0088] This enables control from a control position remote from the device. The vehicle, robot, or flying object can have a controller configured to move the vehicle, robot, or flying object autonomously or semi-autonomously to position the device.

[0089] This achieves a higher degree of automation.

[0090] The vehicle, robot or flying object may have a gripper arm for gripping and positioning a plant part relative to the device.

[0091] This allows even parts of the plant that are difficult to access to be correctly positioned in relation to the device.

[0092] According to a further aspect of the invention, a method for supporting an examination of a plant or multiple plants is provided. The method comprises capturing at least one image of at least a portion of the plant or multiple plants. The method comprises evaluating the at least one image by an evaluation circuit to determine information indicating at least one area suitable for a measurement for examining the plant or multiple plants. The method comprises providing the information for output via a human-machine interface.

[0093] The method supports the user in selecting one or more samples from a multitude of possible samples. This supports, or at least partially guides, the examination of the plant(s).

[0094] The information may indicate which measurement areas are suitable for a spectral analytical, camera-based or laboratory-based examination of the plant or several plants.

[0095] This informs the user visually or in another way which samples (e.g. which leaves or which leaf areas) are suitable for the respective examination.

[0096] The information can indicate which areas are suitable for determining nutrient concentrations and / or determining the health status of the plant.

[0097] This provides the user with visual or other information about which samples (e.g. which leaves or leaf areas) are suitable for the objective and / or measurement method of the respective investigation.

[0098] The procedure allows the information to be determined and output depending on the examination to be carried out.

[0099] This allows the user to receive targeted support regarding the type of test to be conducted (e.g., the nutrients to be examined and / or the type of biotic or abiotic stress to be tested, e.g., one or more diseases to be tested). The evaluation of at least one image can be carried out specifically with regard to which samples (e.g., which leaves or leaf areas) are suitable for the respective test.

[0100] The method may include controlling the human-machine interface to receive a user input specifying the examination to be performed (e.g., an objective of the examination to be performed and / or a type of measurement).

[0101] This allows the user to receive targeted support with regard to the type of study to be conducted (e.g., an objective such as the nutrients to be examined and / or the type of biotic or abiotic stress to be tested, for example, one or more diseases to be tested, or a measurement method such as image data acquisition or spectral analysis data). The evaluation of at least one image can be carried out specifically with regard to which samples (e.g., which leaves or which leaf areas) are suitable for the study specified by the user.

[0102] In the method, the evaluation circuit can determine the information to be output depending on one, several or all of the following information: plant stage, leaf stage, plant species, plant variety, at least one nutrient to be analyzed, at least one active ingredient to be analyzed, the type of biotic or abiotic stress to be tested with the test, for example one or more diseases to be tested, position of the leaf on the plant, measurement method (for example image-based, spectral analysis or laboratory-based), evaluation technique of the measurement results.

[0103] This allows one, several or all of the above-mentioned information, which may influence the appropriate selection of samples, to be taken into account.

[0104] The evaluation circuit can determine the plant species, the plant variety, the plant stage, the leaf stage and / or the position of the leaf on the plant based on an evaluation of the at least one image.

[0105] This allows the evaluation circuit to determine the appropriate sample position(s) for the automatically determined parameters (such as the plant species, the plant variety, the plant stage, the leaf stage and / or the position of the leaf on the plant) and for the respective investigation (e.g. objective and / or measurement method).

[0106] In the method, it can be determined based on an evaluation of the at least one image whether an abaxial or adaxial leaf side is shown in the at least one image.

[0107] This allows the determination of the appropriate sample position(s) depending on how the leaves are positioned and whether the measurement should be carried out on the abaxial or adaxial side for the respective study.

[0108] The evaluation circuit can control the human-machine interface to enable user input specifying one or more of the following information: plant stage, leaf stage, plant species, plant variety, at least one nutrient to be analyzed, at least one active ingredient to be analyzed, at least one type of biotic or abiotic stress to be tested with the test, for example at least one disease to be tested, position of the leaf on the plant, measurement method (for example image-based, spectral analytical or laboratory-based), evaluation technique of the measurement results.

[0109] This allows the user to define for which plants, leaves and tests to be carried out the appropriate sample position(s) for the measurement are to be determined.

[0110] To determine the information, the evaluation circuit can evaluate the at least one image with at least one data-driven trained image evaluation.

[0111] This allows the appropriate sample positions to be determined using an objective, data-based image analysis.

[0112] To determine the information, the evaluation circuit can evaluate the at least one image with at least one trained machine learning model.

[0113] This allows the appropriate sample positions to be determined using a data-based machine learning model. There is no need for image analysis to be performed by a human expert.

[0114] The at least one machine learning model can receive as input pixel values ​​of the at least one image or input values ​​determined from preprocessing of the pixel values.

[0115] This allows the appropriate sample positions to be determined in a simple manner, for example by processing an RGB image.

[0116] The evaluation circuit can use at least one support vector machine, edge- or region-based segmentation or other techniques for image evaluation.

[0117] This allows the appropriate sample positions to be determined using well-established image analysis techniques.

[0118] The evaluation circuit can perform an automatic detection of leaves based on the at least one image in order to determine the information to be output.

[0119] This allows the boundaries of leaves to be determined automatically for examinations to be carried out on leaves.

[0120] The information can specify both leaves suitable for measurement and leaf regions suitable for measurement. The leaf regions can include a specification of suitable leaf sides (abaxial or adaxial sides).

[0121] This allows the sample areas to be determined for leaf examinations and made available to the user. The information can be visually output via the human-machine interface as information overlaid on at least one image.

[0122] This makes it easy to assign the appropriate sample areas to the real world in which the measurement for the investigation is carried out.

[0123] The at least one suitable region can be visualized in different ways, for example as a filled region, as a bordered region, as a semi-transparent region, by markings (e.g. crosses, arrows), by alphanumeric information.

[0124] This allows for the assignment of suitable sample areas to the real world in a variety of ways.

[0125] The information output for visualizing the at least one suitable area may further include a historical output indicating which suitable areas were previously identified and / or in which areas measurements were previously performed. For this purpose, stored data regarding previous measurement recommendations and / or previous measurement locations may be used in combination with position and orientation data.

[0126] This allows previous measurement recommendations and / or previous measuring points to be taken into account when supporting data collection.

[0127] The human-machine interface can be controlled so that a captured image is displayed as a still image and the information is displayed superimposed on the still image.

[0128] This allows the sample areas to be assigned to the real world in which the data for the investigation is recorded in a computationally efficient manner.

[0129] The human-machine interface can be controlled in such a way that, when the image capture device moves relative to the plant or multiple plants, the image displayed on the human-machine interface is updated and the information is tracked. The human-machine interface can be continuously controlled by an evaluation circuit in such a way that the information displayed on the human-machine interface is tracked according to the orientation and position of the image capture device relative to the plant, the plants, and / or the leaves.

[0130] This allows for a particularly easy-to-understand mapping of sample areas to the real world, where data for the study is collected. The risk of incorrect measurements is reduced.

[0131] The image capture device and the human-machine interface can be mounted in a housing. The device can be designed as a movable, particularly manually portable, device.

[0132] This makes handling particularly easy.

[0133] The image capture device can comprise a camera. This allows the at least one area to be determined using robust and compact components. In particular, no spectral analysis detection device is required to determine the at least one area suitable for subsequent measurement.

[0134] The camera can have two or more color channels. For example, the camera can have an RGB camera.

[0135] This allows the information contained in the multiple color channels (for example, about the reflection spectrum of the plant(s)) to be used to determine at least one area.

[0136] The at least one image can be captured using imaging optics to capture the at least one image at a distance from the plant or the plurality of plants. The distance can be within a distance interval that can depend on the respective plant species.

[0137] It is also possible to identify a suitable area within the sheet by controlling the imaging optics (for example by zooming).

[0138] The imaging optics can be adjusted, for example for a zoom operation.

[0139] This can make it easier to identify a suitable area within a sheet.

[0140] This allows a larger area to be imaged and used to determine areas suitable for measurement.

[0141] The information can be determined depending on other factors, optionally also independent of the at least one image. For example, the evaluation circuit can generate the information representing a measurement recommendation depending on environmental parameters (e.g., ambient temperature), time, lighting conditions, and / or location coordinates (such as coordinates determined using a GNSS).

[0142] This allows other factors influencing the measurement recommendation to be taken into account. This is particularly relevant if the measurement position for the respective examination is influenced by such factors (e.g., the time of day or the ambient temperature).

[0143] The image evaluation may include access to a storage system. The storage system may be integrated into the housing containing the image capture device. The storage system may include parameters of applicable image evaluation techniques with which plant regions (e.g., central leaf regions in the center of leaves, leaf edges, leaf regions surrounding leaf stems, leaves arranged at the edges or centrally within a group of leaves) are automatically identified in the at least one image. The storage system may include allocation data that specify which type of plant regions are suitable for measurement for various examinations. The evaluation circuit may access the allocation data stored in the storage system to determine which type of plant regions are suitable for the examination.The evaluation circuit can access the storage system to perform the image evaluation based on the parameters (e.g., parameters of a trained machine learning model) and depending on the type of plant areas. The evaluation circuit can generate the information indicating the at least one suitable area based on both the mapping data for the desired examination (e.g., specified by the user) and the result of the image evaluation.

[0144] This allows for efficient measurement recommendations to be made depending on the specific study. If the device or system is configured for use with multiple plant species or varieties, the mapping data and image analysis parameters can be stored in the memory system for each plant variety or species. Selection can be made automatically based on image recognition or based on user input.

[0145] The evaluation circuit can logically combine the results of multiple image evaluation steps, for example, using a logical AND or a logical AND NOT operation. This can be achieved by pixel-by-pixel multiplication of different image evaluation steps (which can, for example, be selected from the following: detection of central leaf areas in the center of leaves, detection of leaf edges, detection of leaf areas in the vicinity of leaf stems, or detection of leaves arranged at the edges or centrally within a leaf group).

[0146] This allows different results of the image analysis steps to be logically linked if the assignment data specify that the suitability for conducting an investigation requires the cumulative presence of several properties (for example, a middle-sized arrangement of the leaf in a leaf group logically AND-linked with a measurement in the center or at the edge or near the leaf stem).

[0147] The information can be output in real time via the human-machine interface.

[0148] This allows the measurement to be carried out while the user is still on site at the plant.

[0149] At least one image can be captured in response to a trigger. The image analysis and provision of the information can also be performed at a later time, for example, shortly after the capture in a position that is more ergonomically comfortable for the user. This design also has the effect that image analysis and provision are not performed continuously (for example, continuously or at repeated intervals), which reduces power consumption and thus increases the duration of application while simultaneously reducing operating costs.

[0150] This means that suitable measuring ranges can also be visualized at a later point in time.

[0151] The evaluation circuit of the device or system can be configured to selectively perform only those image evaluation steps that are required depending on user inputs defining the plant species and the examination to be carried out.

[0152] This allows the information to be identified and provided efficiently.

[0153] The method may further comprise performing the measurement using a measuring device.

[0154] This allows the information representing a measurement recommendation to be implemented during the measurement.

[0155] The measuring device can be movable relative to the housing of the image capture device. The measuring device can perform the measurement in the at least one suitable area.

[0156] This allows the system to automatically determine which area or areas within the field of view of the image capture device are suitable for measurement. This information is then used to measure and examine the plant. The measuring device can be controlled automatically or semi-automatically. For example, an actuator can be controlled to position the measuring device so that the measurement is taken in at least one suitable area.

[0157] The method may comprise evaluating the measurements acquired by the measuring device to carry out an investigation with regard to: checking one or more nutrient concentrations, checking the presence or absence of one or more disease states, detecting biotic and / or abiotic stress.

[0158] This allows the investigation to be completed. In this case, the system is a system configured to conduct the investigation.

[0159] The measuring device can comprise a spectrometer, a hyperspectral camera or another camera.

[0160] This allows for the detection of nutrient concentrations, diseases, biotic stress and / or abiotic stress.

[0161] The measuring device may include a device for physically taking samples from the appropriate area. The method may include laboratory analysis of the physically taken samples.

[0162] This allows for further analysis, optionally including laboratory analysis, based on the sample taken. The system can comprise a robot (e.g., a multi-axis robot, e.g., a multi-axis robot arm) that positions the measuring device relative to the plant or multiple plants depending on the at least one area in order to perform the measurement.

[0163] This allows the measurement recommendation to be implemented automatically. The measuring robot can be controlled selectively based on user confirmation and / or a user selection from several suitable areas.

[0164] The method may be performed by or with the apparatus or system according to an aspect or embodiment.

[0165] According to a further aspect of the invention, a method is provided for providing an evaluation circuit to support an examination of a plant. The method comprises generating a plurality of trained machine learning models, each of which is configured to receive as input pixel values ​​of at least one image of a plant or of a plurality of plants or input values ​​determined from the pixel values ​​by preprocessing. The method further comprises storing parameters of the plurality of trained machine learning models for use by the evaluation circuit. Each of the trained machine learning models can be assigned to at least one examination to be performed. Generating the plurality of trained machine learning models can comprise training with a training data set of training images.The training images can be annotated to indicate which area or areas of the plant or plants shown in the respective training image are suitable for the investigation to be carried out.

[0166] This allows data-driven image analysis steps to be learned, which are helpful in supporting plant examination.

[0167] The trained machine learning models can include deep learning (DL) models, such as CNNs (“convolutional neural networks”), VisionTransformers (ViTs), or other machine learning models.

[0168] Such machine learning models are particularly suitable for image analysis. Furthermore, well-established techniques exist for training these machine learning models.

[0169] The machine learning models can each provide an output indicating which areas of an image received as input are suitable for measurement for plant examination.

[0170] This allows the machine learning models to be used for the implementation of

[0171] Processing steps in devices, systems, and methods according to embodiments. The training can comprise training multiple machine learning models, each of which is trained to detect central leaf areas in the center of leaves, to detect leaf edges, to detect leaf areas in the vicinity of leaf stems, and to detect leaves arranged at the edges or in the center of a leaf group.

[0172] This allows the most relevant areas for different plant studies to be determined using multiple machine learning models.

[0173] The training may involve training multiple machine learning models for different plant species.

[0174] This allows the image analysis techniques required to support sample identification to be trained for different plant species, optionally also for different plant varieties.

[0175] Training may involve training multiple machine learning models for different growth stages.

[0176] This allows the image analysis techniques required to support sample identification to be trained for different growth stages (e.g., plant and / or leaf stages).

[0177] The method may include using at least one trained machine learning model by a method, apparatus, or system for assisting plant testing according to an aspect or embodiment.

[0178] According to a further aspect of the invention, a method for cultivating plants is provided, which comprises the method for assisting the plant examination, wherein at least one cultivation condition is set depending on a plant examination which is carried out based on the information determined by the method for assisting the plant examination.

[0179] Thus, the result of the support procedure can be used to improve the supply of crops or plants based on the condition assessment.

[0180] According to a further aspect of the invention, a method for cultivating plants is provided, which comprises determining areas suitable for measurement using the device or system according to an aspect or embodiment of the invention.

[0181] Thereby, the effects described with reference to the device according to the invention and the system of the invention are achieved.

[0182] In the method, at least one cultivation condition can be adjusted depending on a plant examination performed based on the information obtained with the device or system. Thus, the result of the provided examination support can be used to improve the care of crops or plants based on the condition determination.

[0183] According to a further aspect of the invention, machine-readable instruction code is provided which, when executed by a programmable computing unit, carries out the method according to one aspect or embodiment of the invention.

[0184] According to a further aspect of the invention, a storage medium is provided with machine-readable instruction code stored thereon, which, when executed by a programmable computing unit, carries out the method according to one aspect or embodiment of the invention.

[0185] The devices, methods, systems, and system components according to embodiments of the invention achieve various effects. In particular, a user can be image-based and assisted in selecting one or more samples for measurements from a multitude of possible samples. The device used for this purpose can be compact, for example, as a handheld device.

[0186] The devices, methods, systems, and system components can be used in various fields. This includes, but is not limited to, the condition determination of crops or plants in agricultural engineering.

[0187] BRIEF DESCRIPTION OF THE CHARACTERS

[0188] Embodiments of the invention are described with reference to the figures. In the figures, similar or identical reference numerals designate elements with similar or identical design and / or function.

[0189] Figure 1 is a block diagram of a device according to an embodiment.

[0190] Figure 2 is a schematic representation of a human-machine interface during operation of the device.

[0191] Figure 3 is a schematic representation of a system according to an embodiment.

[0192] Figure 4 is a flowchart of a method according to an embodiment.

[0193] Figure 5 is a flowchart of a method according to an embodiment.

[0194] Figure 6 is a schematic representation of a system according to an embodiment.

[0195] Figure 7 is a schematic representation of a human-machine interface during operation of the device.

[0196] Figure 8 is a schematic representation of a human-machine interface during operation of the device.

[0197] Figure 9 is a schematic representation of a human-machine interface during operation of the device. Figure 10 is a schematic representation explaining the functioning of the device according to one embodiment.

[0198] Figure 11 is a schematic diagram to explain the operation of the device according to an embodiment.

[0199] Figure 12 is a schematic diagram to explain the operation of the device according to an embodiment.

[0200] Figure 13 shows a schematic representation of a device designed as a handheld device according to an embodiment.

[0201] Figure 14 shows a schematic representation of an agricultural vehicle according to an embodiment.

[0202] Figure 15 is a schematic representation of a system according to an embodiment.

[0203] Figure 16 shows a schematic representation of a flying object according to an embodiment.

[0204] Figure 17 is a flowchart of a method according to an embodiment.

[0205] DETAILED DESCRIPTION OF EMBODIMENTS

[0206] Embodiments of the invention are described with reference to the figures. In the figures, similar or identical reference numerals designate elements with similar or identical design and / or function.

[0207] While embodiments are described in connection with a nutrient concentration determination using leaves of strawberry plants, the embodiments are not limited thereto.

[0208] The features of the embodiments can be combined with each other unless this is expressly excluded in the following description.

[0209] Devices, systems and methods according to embodiments of the invention are configured to automatically determine and visualize one or more areas in a captured image or in multiple images at which a measurement is taken for a plant examination.

[0210] The term "plant testing" or "plant examination," as used here, includes a measurement to record measurement results as well as the further evaluation of the measurement results, which is carried out with regard to a specific objective. The further evaluation can be carried out automatically by an electronic processing device. The term "plant testing" encompasses a quantitative or qualitative determination of nutrient concentrations for one or more nutrients (for example, for at least one macronutrient and / or for at least one micronutrient). The term "plant," as used here, also includes eukaryotes, in particular algae.

[0211] The term nutrient concentration determination, as used here, includes the determination of the nutrient concentration of one or more nutrients as a value that can be determined from a continuous numerical range (for example, in the units in which the concentration for the corresponding nutrient is determined in laboratory analyses). The term nutrient concentration determination, as used here, particularly also includes the determination of an estimate and / or evaluation of the respective nutrient concentration. However, the term nutrient concentration determination also includes the determination of values ​​that can, for example, only come from a set of discrete values ​​and can indicate whether the nutrient concentration is within a target range or outside the target range, or whether there is an undersupply, oversupply, or appropriate supply of the corresponding nutrient.

[0212] The term "image," as used here, refers to signals or data that represent at least one spatially resolved parameter of the imaged plant(s). The images can each represent a detected parameter pixel by pixel, particularly in a regular pixel arrangement. The images can have one, two, three, or more different image channels (e.g., different color channels).

[0213] The term "measurement" as used here also includes the acquisition of image data using a sensor, such as a camera chip, to capture the optical properties of a sample. The measurement can be performed in situ on the plant. This is especially true for measurements that capture a spectrum and / or an image. The measurement does not have to be performed in situ but can also be performed remotely from the plant using a sample taken. This is especially true for measurements for laboratory testing.

[0214] Systems according to embodiments may also comprise a measuring device that performs one or more measurements according to the automatically determined ranges suitable for measurement. The measuring device may comprise a spectral analytical measuring device. The term "spectral analytical measuring device" as used here encompasses a device that is capable of and configured to detect an optical measurement variable detected for several (two or more than two) wavelengths at the same measuring area of ​​the plant. The spectral analytical measuring device may comprise a spectrometer. However, this is not necessarily required.For example, as an alternative or in addition to using a spectrometer, the spectral analytical measuring device can be configured to record the optical measurement variables for the plurality of wavelengths sequentially in time, for example by actively irradiating with different wavelengths and determining the intensity of the respectively detected scattered or reflected light. Figure 1 shows a device 10 for determining the condition of a plant 2. The device 10 has a housing 20. The device 10 has a camera 21 arranged on or in the housing 20. The camera 21 is configured to capture at least one image of at least one plant or multiple plants 2. The at least one image does not have to show the plant(s) in its entirety. The device 10 is configured such that the image capture takes place while the device 10 is not in contact with the plant or the multiple plants 2.

[0215] The device 10 has a human-machine interface 24 which is configured to output information determined from the at least one image, which indicates at least one area suitable for a measurement for examining the plant or the plurality of plants 2.

[0216] The device 10 thus assists a user in selecting one or more samples from a multitude of possible samples. The examination of the plant(s) 2 is thus supported, for example, at least partially guided, by the device 10.

[0217] The information output via the human-machine interface 24 can be determined and output depending on the examination to be conducted. The information visualized via the human-machine interface 24 can, for example, indicate which area or areas of samples (e.g., which leaves and / or which leaf areas) are suitable for a spectral analysis, camera-based, or laboratory-based examination of the plant or multiple plants. This informs the user, visually or in another way, which samples (e.g., which leaves and / or which leaf areas) are suitable for the respective examination.

[0218] The device 10 can be configured such that the information output via the human-machine interface 24 selectively or cumulatively indicates which areas (e.g., which leaves and / or which leaf areas) are suitable for determining a nutrient concentration, determining a health status, determining biotic stress, and / or determining abiotic stress of the plant(s) 2.

[0219] The corresponding information can be output via the human-machine interface 24 as information graphically superimposed on the at least one image. For this purpose, the human-machine interface 24 can comprise a display device or another optical output device, for example, a projector for superimposing the information on the user's real-world view. In particular, the device can also be configured as a VR ("virtual reality") or AR ("augmented reality") device, for example, in the form of VR or AR glasses.

[0220] For image evaluation and controlling the human-machine interface 24 (and, if present, a data interface 23), the device 10 has an evaluation circuit 26. The evaluation circuit 26 is configured to perform an image evaluation 27 of the at least one image. In this process, pixel values ​​or values ​​derived from the pixel values ​​through preprocessing (e.g., features or feature vectors) can be processed. The image evaluation 27 can be used to identify the plants and plant regions (e.g., leaf regions) relevant to the respective plant species and the respective plant examination to be performed.The results of several image analysis steps (e.g., to identify leaves of a specific leaf stage, to identify leaves with a specific position in leaf groups, and / or to identify specific areas within the leaves) can be logically linked to identify areas that cumulatively meet several conditions to be met for the respective plant examination.

[0221] The device 10 has a storage system 25, which can be mounted in the housing 20 or provided outside the housing 20. The storage system 25 stores parameters for the various image analysis steps, for example, parameters of support vector models, filter parameters (e.g., filter kernels) for edge- or region-based segmentation, or trained machine learning models. The evaluation circuit 26 is configured to retrieve from the storage system 25 how the image analysis should be performed (e.g., using which machine learning model parameters), depending on the plant species and / or the examination to be performed.

[0222] The storage system 25 can also store assignment data that indicate which criteria areas must (abstractly) meet to be suitable for a specific study (e.g., a nutrient concentration determination of a specific nutrient or multiple nutrients in a specific plant species). This abstract suitability can, for example, indicate that first criteria (e.g., a specific leaf stage and a specific leaf area) are relevant for a first study, and that second criteria (e.g., a specific leaf location in a leaf group and a specific leaf area) are relevant for a second study.The evaluation circuit 26 can be configured to determine, depending on the assignment data, which image evaluation steps are to be carried out in each case and how the results of the image evaluation steps are to be linked to one another (for example, by pixel-by-pixel multiplication of segmentation results to ensure that the identified region or the plurality of identified regions cumulatively meet several criteria).

[0223] For example, the allocation data can define that for strawberry plants a measurement to check whether there is a calcium (Ca) deficiency should be taken at the leaf tips, a measurement to check whether there is a boron (B) deficiency should be taken at the leaf edges, a measurement to check whether there is an iron (Fe) deficiency should be taken at the leaf veins and a measurement to check whether there is a potassium (K) deficiency should be taken at the leaf centers.

[0224] Alternatively or additionally, the allocation data can define that for strawberry plants, a measurement to check whether there is a deficiency of potassium (K) or magnesium (Mg) should be carried out on old leaves, a measurement to check whether there is a deficiency of calcium (Ca) should be carried out on newly grown leaves and a measurement to check whether there is a deficiency of iron (Fe) or boron (B) should be carried out on young leaves.

[0225] Knowledge about which leaf areas and / or leaf stages and / or plant stages are most likely to detect certain phenomena at the earliest possible stage is known to the expert and is provided in the specialist literature. This knowledge can be translated into the corresponding classification data.

[0226] The evaluation circuit 26 can have an interface controller 28. The interface controller 28 can be configured to control the human-machine interface 24 to output the information. The interface controller 28 can optionally be configured to provide the image-based information concerning the area or areas suitable for measurement to a controller in order to automatically or semi-automatically adjust the position, orientation, and / or measurement data recording of a measuring device. The corresponding data communication can take place via the data interface 23. The interface controller 28 can optionally also be configured to output the at least one image and / or the determined information via the data interface 23 for storage or use by a computer system separate from the device 10.

[0227] The interface controller 28 can also be configured to control the human-machine interface 24 to enable inputs. The inputs can specify a user-defined plant species and / or a type of examination to be performed (e.g., nutrient concentration determination, biotic stress determination, abiotic stress determination, health status determination). The evaluation circuit can consider the input during image evaluation and information gathering, for example, to determine which image analysis steps need to be performed. The plant species can also be determined automatically by image analysis and does not necessarily have to be user-defined. To perform the various control and processing functions, the evaluation circuit 26 can comprise one or more integrated circuits.The one or more integrated circuits may, for example, comprise any one or any combination of the following circuits or circuit components: an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a processor, a controller, one or more quantum gates, a circuit for quantum information processing, and other integrated circuits. The evaluation circuit 26 may comprise one or more circuits specifically configured for image processing, such as one or more embedded devices and / or one or more graphics processing units (GPUs) and / or tensor processing units (TPUs). Such configurations enable particularly efficient execution of the image evaluation.

[0228] Figure 2 schematically shows the human-machine interface 24 of the device 10. The area or areas suitable for examination are shown as an overlay 30 or otherwise on the acquired image. The evaluation circuit 26 can be configured to generate an output image from the acquired image and the determined measurement recommendation (i.e., where the measurement is to be taken with respect to the image). The interface controller 28 can be configured to control the human-machine interface to output the output image.

[0229] The evaluation circuit 26 can be configured to repeatedly perform the image evaluation and control of the human-machine interface 24 for visualizing the measurement recommendation, for example continuously (with a repetition interval that can be fixed or variable) or in response to a trigger event (for example, a relative movement between the device 10 and the plant or plants 2). The output image can thus be updated.

[0230] When visualizing a measurement recommendation during repeated acquisition and evaluation of images (e.g., individual images of a video sequence, e.g., a video stream), tracking techniques can be used. The visualization of the measurement recommendation can be adapted and tracked. The evaluation circuit 26 can identify corresponding regions in several consecutive images (e.g., individual images of a video sequence, e.g., a video stream) and use them to track a measurement recommendation, once determined, from one image to another image acquired later. The tracking can also use output signals from position and / or orientation sensors, e.g., to obtain a first estimate for a positional banding of the device 10.The human-machine interface 24 can be controlled in such a way that the output is updated according to the movement of the device 10 and is enriched with the measurement recommendation.

[0231] The use of tracking methods has several advantages. It increases consistency between region visualizations in different images during repeated acquisitions. This reduces flicker in the output. Subsequent control for measurement is better and more efficient, since the area suitable for measurement does not always have to be determined image-based without prior knowledge, but rather utilizes the previous evaluation results.

[0232] The visualization may also include controlling the human-machine interface 24 based on previously provided measurement recommendations and / or positions of previously performed measurements. This allows new measurement recommendations to be visualized in relation to previously provided measurement recommendations and / or positions of previously performed measurements, further improving user support.

[0233] Figure 3 is a schematic representation of a system 1 according to an exemplary embodiment. The system 1 comprises the device 10 and a measuring device 40. The measuring device 40 is configured to perform the measurement according to the areas suitable for measurement determined by the device 10. If the device 10 and the measuring device 40 are implemented as separate and movable devices, the device 10, in particular, can be designed as a compact and easy-to-handle device. This facilitates the determination of the areas suitable for measurement.

[0234] The measuring device 40 may comprise a spectral analytical measuring device, for example, a spectrometer. The measuring device 40 may also comprise a camera-based device, for example, the device disclosed in the application entitled "Method, Device, and System for Determining Nutrient Concentration in a Plant and Method for Providing an Evaluation Device or an Evaluation System for Determining Nutrient Concentration" by Carl Zeiss AG.

[0235] Figure 4 is a flowchart of a method 50. The method 50 can be performed automatically by the device 10.

[0236] At 52, the camera 21 captures an image. During image capture, the device 10 can be spaced from the imaged plants. Imaging optics of the device 10 can be optimized for typical distances (e.g., at least 10 cm, or at least 50 cm, or at least 100 cm).

[0237] The spacing can depend on the plant species. The imaging optics can be adjustable to support different spacings. For example, the spacing for strawberry plants can range from 10 cm to 1.5 m. For tomato plants, the spacing can range from 30 cm to 4 m. For almond trees, the spacing can range from 30 cm to 10 m.

[0238] At 53, the image is evaluated. The image evaluation by the evaluation circuit 26 determines which areas depicted in the image are suitable for plant examination.

[0239] The result of the image analysis is output at 54. The human-machine interface 24 can display, superimposed on the captured image, which areas of one or more plants should be measured for the examination (e.g., determining nutrient concentration, determining biotic stress, determining abiotic stress, determining a health status).

[0240] Steps 52-54 may be repeated to update the output of the human-machine interface 24. This may be done continuously (e.g., with a repeat interval) or trigger-based.

[0241] Figure 5 is a flowchart of a method 50'. The method 50' can be performed automatically by the device 10.

[0242] At 51, the human-machine interface 24 is activated to enable an input. The input can define the investigation (e.g., the measurement method (image-based, spectrometric, or laboratory-based), the objective, and / or optionally also the type of evaluation of the measurement results) for which the appropriate measurement areas are to be identified. The input can optionally also define a plant species.

[0243] The input is taken into account during image analysis 53. Image analysis 53 is performed specifically with regard to the planned examination.

[0244] Figure 6 is a schematic representation of a system 1 according to an exemplary embodiment. The system 1 comprises the device 10 and the measuring device 40. The measuring device 40 is configured to perform the measurement according to the ranges determined by the device 10 as suitable for the measurement.

[0245] The system 1 can comprise a computing device 4. The evaluation of the image captured by the device 10 and / or the further evaluation of the measurement results captured by the measuring device 40 can be performed by the computing device 4. Alternatively or additionally, results determined by the device 10 and / or the measuring device 40 can be stored by the computing device 4. Communication between the computing device 4, the device 10, and / or the measuring device 40 can take place via wireless data transmission channels. Communication can also take place via a local area network (e.g., a WLAN), a wide area network 6, or a cellular network.

[0246] When the results of the device 10 are used automatically when the measurement is carried out by the measuring device 40, the device 10 can transmit the areas to be measured to the measuring device 40 either via a direct communication connection (which can be implemented, for example, by Bluetooth or near-field communication) or via a communication network 6 and / or the computing device 4.

[0247] The device 10, the system 1, and the method can thus be used as follows: Taking an image of the plant to be measured from a distance (i.e., no contact measurement). A camera (e.g., an RGB camera, optionally with near-field optics) can be used for this purpose.

[0248] Automatic localization of suitable samples and positions in the captured image for intended use (e.g., measuring nutrient concentrations for determining nutrient concentrations). Due to plant physiological processes, the effects of nutrient deficiencies are visible differently across the leaf area, for example, on plant leaves.

[0249] Converting localization results into measurement recommendations.

[0250] Visualizing measurement recommendations via the human-machine interface.

[0251] The device 10, the system 1, and the method can be configured such that location coordinates for each captured image are collected and stored, for example, as metadata. Such coordinates can be determined by the device 10 or a communication unit connected to it (for example, a smartphone or tablet). By determining and storing the location coordinates, it is easier to locate a measuring point in a cultivation area (for example, a field). Comparisons of evaluations performed at different times are easier because the same measuring point can be visited repeatedly and the measurement can be performed repeatedly.

[0252] With reference to Figures 7, 8, and 9, the functionality is explained in more detail in the context of strawberry plants. In each case, the result of the evaluation is shown as an example on the human-machine interface 24. However, it is not necessary to display these specific evaluation results.

[0253] The device 10, for example the evaluation circuit 26 of the device 10, can be configured to carry out the following image evaluation steps: o Detection of the middle of three strawberry leaves

[0254] ■ Objective: To avoid measuring a non-central leaf in strawberries

[0255] ■ Input data: Image

[0256] ■ Result: Border marking around one or more contained middle leaves

[0257] ■ An example is shown in Figure 7 o Segmentation of leaf tips ■ Goal: targeted measurement of leaf tips (if this is suitable for the nutrient to be investigated)

[0258] ■ Input data: Image

[0259] ■ Result: Segmented leaf tips

[0260] ■ An example is shown in Figure 8 o Detecting leaf edges

[0261] ■ Objective: targeted measurement of leaf margins (if this is suitable for the nutrient to be examined)

[0262] ■ Input data: Image

[0263] ■ Result: Segmented leaf margins

[0264] ■ An example is shown in Figure 9 o Detection of leaf edges / areas with central leaf veins

[0265] ■ Objective: targeted measurement of leaf centers (if this is suitable for the nutrient to be examined)

[0266] ■ Input data: Image

[0267] ■ Result: Segmented leaf centers

[0268] ■ An example is shown in Figure 2 o Detecting leaves by leaf age

[0269] ■ Aim: Targeted measurement of leaves of the corresponding age (if this is relevant for the nutrient to be examined)

[0270] ■ Input data: Image

[0271] ■ Result: Detected leaves of the respective age classes (e.g. young, newly grown, fully grown, old)

[0272] ■ An example is not shown

[0273] Some or all of the image analysis steps may involve determining a confidence or uncertainty for the analysis result. Confidence or uncertainty may be determined, for example, using one, several, or all of the following techniques:

[0274] Uncertainty can be derived directly from the output values ​​of a machine learning model. For example, in a classification procedure, the entropy over single-class probabilities can be used as an uncertainty measure. Alternatively, the difference between the most certain and second-most certain classes can be used as an uncertainty measure.

[0275] An uncertainty measure can be determined by estimating the similarity of at least one image to the training data. For this purpose, so-called "activation shaping" for CNNs ("convolutional neural networks") can be used, for example. Separate models can also be used to determine anomalies or deviations.

[0276] Uncertainties can be determined by mapping them to the uncertainty of a test dataset (during training). For example, the mean error on a test dataset can be determined and used as the expected uncertainty for each output value obtained with the machine learning model.

[0277] The above-mentioned techniques can be used in particular for evaluations that solve classification tasks.

[0278] For detection tasks, for example, confidence or uncertainty can be determined by having the machine learning model have an output that also outputs a confidence value. The machine learning model can therefore be trained and used in such a way that, in addition to a detection result, it also outputs the associated confidence or uncertainty.

[0279] For segmentation tasks, confidence or uncertainty can be determined using one or more of the following techniques:

[0280] Uncertainty can be derived directly from the output values ​​of a machine learning model. For example, the entropy can be averaged over individual class probabilities for all pixels in a region of the same class.

[0281] Uncertainty can be derived from a variation of the output values ​​upon repeated, random variation of the input image (augmentation during application or so-called "test time augmentation" techniques) or from intermediate calculations (for example, by so-called "test time drop-out" techniques).

[0282] Alternatively or additionally, in detection and / or segmentation tasks, an uncertainty measure can be determined by estimating the similarity of at least one image area (e.g., a region or box) to image areas in the training data. Separate models can also be used to determine anomalies or deviations.

[0283] The results of the image evaluation steps can be filtered or weighted based on the associated confidence or uncertainty. For example, only those results whose confidence lies above a threshold can be reused (e.g., for visualization). Only the k most certain results can be output. The threshold or the number k can be user-defined. For example, in the visualizations shown in Figures 7, 8, and 9, only those areas whose confidence is above a threshold or which are among the k most certain results can be visualized. The results of multiple image evaluation steps can be combined, as already explained. The human-machine interface 24 is controlled to visualize the evaluation result.

[0284] When logically combining multiple image analysis steps, the uncertainty of the results can also be taken into account. For example, a combined uncertainty can be determined from the uncertainties of the image analysis steps.

[0285] The corresponding image analysis steps can be implemented using analysis techniques designed for the respective objective. These can include, for example, support vector methods (optionally with fixed or learnable features), filtering, region- or edge-based segmentation methods, or trained machine learning models. Data-driven training is advantageous. This allows the system to learn how to analyze an image in an objective, data-based manner.

[0286] The device 10 can also receive updated and / or additional trained machine learning models via the data interface 23 during field operation. This allows the device 10 to be configured specifically for the application cases required by the respective user.

[0287] The images can be analyzed, in particular, using deep learning models for image segmentation and / or object detection. For example, CNNs ("Convolutional Neural Networks") or VisionTransformer (ViT), which are familiar to those skilled in the art from the specialist literature, can be used.

[0288] Segmentation can be performed, for example, using a trained SegFormer pre-trained on publicly available image datasets. The segmentation model can be fine-tuned depending on which of the above-mentioned segmentation tasks is to be performed. Training can be performed using images annotated, for example, with outlined regions of the respective leaf types or leaf regions.

[0289] Detection can be performed, for example, with a YoloV5, which is pre-trained on publicly available image datasets. The detection model can be fine-tuned depending on which of the above-mentioned detection tasks is to be performed. Training can be performed using images annotated, for example, with outlined regions of the respective sample areas.

[0290] One advantage of such techniques is that potentially relevant features (such as size, texture, color, etc.) do not need to be defined by experts. Instead, they are learned data-driven.

[0291] When training, it is desirable to use a sufficiently large dataset of, for example, at least 100, at least 500 or at least 1000 annotated images.

[0292] The devices, systems, and methods can be configured to use images acquired with device 10 to further improve the machine learning model(s). For example, the acquired images can be annotated with the subsequently user-selected measurement positions, used to expand the training dataset, and employed for retraining or new training. The machine learning model can thus be adapted and further improved during continued field use of the disclosed devices, systems, and methods.

[0293] Figure 10 schematically shows an implementation of the image evaluation 27 by the evaluation circuit using a trained machine learning model 70. Multiple trained machine learning models can be used to solve various detection and / or segmentation tasks. An input (e.g., an input layer) of the machine learning model 70 can receive pixel values ​​of the at least one image 60 or values ​​derived therefrom by preprocessing 75. An output of the machine learning model 70 (e.g., at an output layer or another output) can provide a result of the trained segmentation or detection task.

[0294] When using a camera with multiple color channels, the pixel values ​​of the different color channels can be fed to multiple input nodes of the machine learning model 70.

[0295] Figure 11 schematically shows further features of the at least one trained machine learning model 70.

[0296] The machine learning model 70 has an input (e.g., an input layer) 71, one or more hidden layers 73, and an output (e.g., an output layer) 72. The machine learning model 70 can be configured to receive pixel values ​​of the at least one image and / or data or features derived therefrom as input. The machine learning model has one or more hidden layers 73. The machine learning model 70 can be trained such that the output provides the solution to the segmentation or detection task for which the machine learning model 70 is trained.

[0297] Figure 12 schematically shows various image evaluation steps 81-87 that can be performed automatically by the evaluation circuit 26. Alternative or additional image evaluation steps can be used depending on the plant species and / or the examination to be performed. Not all of the image evaluation steps 81-87 need to be performed.

[0298] The evaluation circuit 26 can be configured to determine the plant species 81, the plant variety 82, the plant stage 83, the leaf stage 83 (e.g., young, young-mature, mature, old), and / or a leaf position 85 in a group of leaves (e.g., middle position or edge position in the group). The plant stage can be selected from a set of plant stages that, for example, can be selected from morphological developmental stages of the plant (e.g., according to the BBCH (Federal Biological Research Center for Agriculture and Forestry, Federal Plant Variety Office and Chemical Industry) scale or codes based thereon). The plant stage can be selected from a set of plant stages that includes at least the following stages: germination, flower development, flowering, fruit development, and harvest maturity.

[0299] These are detection and / or classification tasks that can be solved using detection techniques (such as trained machine learning models). The evaluation circuit 26 can be configured to identify different leaf regions 86. This is a segmentation task that can be solved using appropriate segmentation techniques (such as trained machine learning models).

[0300] The results can be combined in image evaluation step 87. For example, if a measurement is to be performed on leaves at a specific leaf stage and in each case in an edge region of a central leaf, the results of image evaluation steps 83, 84, 85, and 86 can be logically linked. This is schematically represented symbolically by an AND link. The logical link can be performed, for example, by pixel-by-pixel multiplication of binary values ​​(0 / 1) obtained by image evaluation steps 83, 84, 85, and 86. A logical "1" can indicate that the corresponding pixel is located on a leaf with the desired leaf stage, on a centrally located leaf, and on an edge region of the leaf.

[0301] Parameters that define evaluation steps to be carried out for the respective plant species or plant variety (in image evaluation steps 83-87) and / or the respective examination (in image evaluation steps 81-87) can be stored as image processing parameters 88 in the storage system 25 for use by the evaluation circuit 26.

[0302] Parameters that define for the respective plant species and / or the respective investigation which of the evaluation results are required and how these are to be logically linked can be stored as assignment data 89 in the storage system 25 for use by the evaluation circuit 26.

[0303] The device 10, the system 1, and the method thus provide support to the user in selecting and measuring a sample. By supporting the user, the risk of performing an improper measurement is reduced. Improper measurement would entail the risk of an invalid measurement and, consequently, a misinterpretation of a test objective (e.g., a nutrient supply, a health condition such as plant diseases, biotic stress, and / or abiotic stress). Various further modifications and features are described below that can optionally be used to support the technical effects already described.

[0304] Camera: The camera 21 can have multiple color channels. This is advantageous, for example, for assessing the plant leaf stage or condition using the evaluation circuit 26 or for distinguishing between color inhomogeneities, color contrast structures, and glare above the leaf surface.

[0305] With regard to camera settings and calibration, defined camera parameters (e.g., a defined exposure time) are advantageous. Calibrating the camera to a defined reflection standard is advantageous, ensuring consistent image quality.

[0306] For use in scenarios with rapidly changing ambient lighting (such as outdoor fields), automatic adjustment of lighting time, white balance, or other measures can be advantageous.

[0307] The device 10 may comprise an illumination source and / or illumination optics adapted for the image acquisition to be carried out (for example, by a specially adapted depth of field for intended working distances).

[0308] The illumination source may comprise a UV light source. The device may be configured to make the plant fluoresce (e.g., in the range of 400-630 nm or 630-800 nm). This provides additional information for the correct selection of samples.

[0309] Output of image analysis results: The result of the measurement recommendation can be determined and visualized in real time. For example, a real-time calculation of a "contact-analog" overlay of a suggested measurement spot can be performed in the live image mode of a camera image display. Alternatively, an image can be captured trigger-based and analyzed. The result is visualized (e.g., overlaid on the image, which may be displayed with a time delay).

[0310] Evaluation results can be output locally on the device 10 or via an output unit remote from the device 10. This provides direct feedback while the user is still on-site at the plant.

[0311] Regardless of where the evaluation results are output, tracking techniques can be used for repeated acquisition and evaluation of images (e.g., individual images of a video sequence, e.g., a video stream). The visualization of the measurement recommendation can be adapted and tracked. The evaluation circuit 26 can identify corresponding regions in several consecutive images (e.g., individual images of a video sequence, e.g., a video stream) and use them to track a measurement recommendation, once determined, from one image to another, subsequently acquired image. The tracking can also use output signals from position and / or orientation sensors, e.g., to obtain an initial estimate for a positional banding of the device 10.

[0312] The visually perceptible output can be updated according to the movement of the device 10 and enriched with the measurement recommendation.

[0313] User-defined restriction of plant species and / or test type: Users usually know in advance which plant species they want to measure. This can be entered, so that only the image analysis needs to be performed for that plant species. Alternatively, or additionally, the analysis can be specifically restricted to user-defined nutrients for which a nutrient concentration is to be determined based on the measurement yet to be performed. For example, if only nutrients for which leaf edges are to be measured are to be examined, the leaf center does not need to be additionally located.

[0314] Use of a measuring device 40 for spectral analysis: It may be advantageous to carry out the measurement and, advantageously, also the evaluation of the measurement on-site immediately after identifying suitable measurement ranges with the device. This can be done by the spectral analysis measuring device 40, optionally in conjunction with a remote computer 4. The spectral analysis acquisition device can be a measuring system that enables the recording of multiple wavelengths. This is possible, for example, by using a conventional spectrometer (in which light is split depending on the wavelength), filter-based systems (by blocking light components), single-wavelength diodes, or other techniques. For example, a detector with multi-sensor channels that are sensitive to different spectral ranges can be used.

[0315] Use of device 10 to assist in sample collection for laboratory analysis: Device 10 can also be used to determine where physical samples should be taken. The collected samples can then be analyzed in the laboratory (e.g., chemically, physically, or physicochemically).

[0316] Adaptation of the measurement recommendation depending on additional factors: The conversion of the localization results into a measurement recommendation can depend on additional factors, for example, environmental parameters such as ambient temperature, time, or lighting conditions. These factors can be determined (for example, using at least one sensor for measuring environmental parameters and / or using a timer) and taken into account when determining the areas where the measurement can be taken. Different applications of the device 10: The device 10 can be configured in different ways.

[0317] Figure 13 schematically shows an embodiment of the device 10 as a handheld device having a structure 39 for holding the device 10.

[0318] Figure 14 shows a further embodiment in which a vehicle 90 according to one exemplary embodiment has the device 10 mounted thereon. Accordingly, the device 10 can be configured with a support structure for attachment to an agricultural vehicle. The human-machine interface can be arranged in a driver's cab.

[0319] In yet another embodiment, the device 10 can be configured as a stationary system. The device 10 can be mounted with a stationary support near a conveyor on which plants are transported past the device 10.

[0320] Figure 15 shows a system 1 according to an embodiment in which a robot 101 (for example, a robot arm with at least two or three axes) automatically positions the measuring device 40. Positioning can be performed by a controller 102 and / or an actuator 102, which obtains information about the samples and measurement positions from the device 10. A relative position and relative orientation of the device 10 to the plants and the robot can be determined by a conventional position and orientation determination system (for example, using speckle patterns, laser-based distance measurements, etc.) and used in the control.

[0321] A base of the robot 101 may be movable, for example along a rail system.

[0322] Figure 16 shows a further embodiment in which a flying object 109 comprises the device 10. The flying object 109 can be remotely controllable and / or configured for autonomous flight operation.

[0323] Areas of application: The disclosed devices, systems, and methods can be used for various applications. In particular, the devices, systems, and methods can be used on crops or cultivated plants. The plants on which the techniques can be used according to the invention can, in particular, also include forestry plants (e.g., trees), hybrids, and / or genetically modified plants.

[0324] Examples of such plants can be:

[0325] • Crops

[0326] • Fodder plants

[0327] • Fiber plants

[0328] • Oil plants

[0329] • Ornamental plants

[0330] • Industrial crops, such as o Oilseeds o Tobacco o Hemp o Hops o Aromatic, culinary and medicinal plants o Seeds for herbaceous oil plants o Seeds for linseed (and consequently fibre flax) o Energy crops o Plants used for the production of feedstocks for renewable energy production

[0331] The study for which the measurement is to be conducted may include, but is not limited to, the qualitative and / or quantitative determination of nutrient concentrations. For example, the condition assessment may include analyses regarding one, several, or all of the following plant conditions:

[0332] Nutrient supply problems, for example o Nutrient deficiency o Toxicity biotic stress, for example due to o Nematodes o Insects o Arachnids o Fungi o Bacteria o Viruses

[0333] Abiotic stress, for example due to o drought o excessive water supply and / or waterlogging o salinity o temperature-related stress (cold, frost, heat) o UV light o metal toxicity o mechanical stress (e.g. pressure, squeezing, pressing).

[0334] The results of the study can be used to improve plant conditions. This can be achieved through automatic or semi-automatic adjustment of artificially applied substances (such as approved fertilizers or approved insecticides) and artificially applied water. Numerous other applications are possible, for example, for monitoring plant health with regard to environmental and regulatory issues.

[0335] Figure 17 is a flowchart of a method 110 that may be performed using the apparatus 10.

[0336] At 111, the device 10 generates a measurement recommendation.

[0337] At 112, the measuring device 40 performs a measurement in accordance with the measurement recommendation.

[0338] At 113, the result of the evaluation of the measurement result is used to determine measures to improve conditions for the plants. The measures can then be implemented to improve the conditions. The evaluation in step 113 can optionally also use the at least one image evaluated in step 111.

[0339] The disclosed devices, systems, and methods provide various technical effects. In particular, they offer improvements in supporting plant testing. The device 10 enables suitable samples and areas for measurement to be determined non-destructively, precisely, and quickly.

[0340] The devices, systems and methods can be used in combination with the features of the devices, systems and methods described in German patent application 10 2023 113 704.2 of Carl Zeiss AG, filed on May 25, 2023, entitled "Device, evaluation device, evaluation system and method for a condition analysis of a plant" and / or in German patent application 10 2023 113 704.2 of Carl Zeiss AG, filed on the same day, entitled "Method, device and system for determining the nutrient concentration of a plant and method for providing an evaluation device or an evaluation system for determining the nutrient concentration". In particular, the devices, systems and methods of the present application can be used to determine on which plants or plant regions a further evaluation should be carried out using the techniques that describe the features of the devices, systems and methods described in German patent application 10 2023 113 704.2 of Carl Zeiss AG, filed on May 25, 2023, entitled "Device, evaluation device, evaluation system and method for a condition analysis of a plant".May 2023, filed in German patent applications 10 2023 113 704.2 of Carl Zeiss AG entitled "Device, evaluation device, evaluation system and method for a condition analysis of a plant" and / or in the German patent applications of Carl Zeiss AG filed on the same day entitled "Method, device and system for determining the nutrient concentration of a plant and method for providing an evaluation device or an evaluation system for determining the nutrient concentration".

[0341] Embodiments of the invention can be used in particular in connection with plant cultivation in agriculture, which is a central economic and industrial sector of humanity. Global trends such as a growing population and a simultaneous decrease in the availability of agricultural land lead to a constantly growing demand for crop yields per area. At the same time, the unsustainable exploitation of soils leads to an increased need for targeted fertilization in order to maintain or even increase previous yields. Furthermore, a growing awareness of sustainable management and increasing regulatory pressure are creating new requirements for precisely tailored fertilization, in particular to avoid over-fertilization.

[0342] Of particular importance for the growth of any crop is the appropriate supply of nitrogen (N), as it is the most important element of chlorophyll and thus essential for optimal metabolism. In addition to N, other macronutrients are also relevant, such as phosphorus (P) and potassium (K), which can often be added to the soil together with N as so-called NPK fertilizers, as well as sulfur (S), calcium (Ca), and magnesium (Mg). The latter is the central element in the chlorophyll ring.

[0343] Additionally, depending on the plant type, other nutrients may also be relevant for optimal growth and yield, although these are often required in significantly smaller amounts and are therefore also referred to as micronutrients. These include, for example, boron (B), molybdenum (Mo), copper (Cu), manganese (Mn), zinc (Zn), iron (Fe), and chlorine (Cl).

[0344] Embodiments of the invention can contribute to creating optimal conditions with regard to nutrient supply throughout the entire growth phase of a plant. Only in this way can nutrient deficiencies or nutrient oversupply be appropriately responded to by adapted fertilizer applications and / or an adjustment of a water solution (for example, in the case of hydroponic systems) (through a specific fertigation program in the substrate or soil growth or as additional soil and / or foliar fertilization). Embodiments of the invention allow for the regular monitoring of plants. Compared to laboratory-based leaf analysis, the examination with a spectral analytical measuring device 40 can be carried out quickly. This is particularly desirable for plant species with rapid fruit ripening and fast-growing plants (for example, strawberries and lettuce).Embodiments of the invention thus also address the objective that there should not be a delay of several days between measurement and availability of the evaluation result, as otherwise the measurement result would already be invalid and would not allow for any meaningful adjustment of the nutrient application. Rapid evaluation is also desirable when environmental factors change rapidly (e.g., temperature, precipitation, etc.), as otherwise the condition would change significantly between the measurement and the time of analysis.

[0345] Embodiments of the invention enable the determination of samples and measurement ranges and the subsequent execution of a measurement for nutrient analysis at the intended application site without the need to transmit physical samples. While embodiments have been described with reference to the figures, modifications may be implemented in further embodiments. For example, the modifications already explained may be used cumulatively or alternatively.

[0346] While embodiments applicable to strawberry plants have been described, the disclosed techniques may also be used in other applications. Exemplary further applications include the following non-limiting examples:

[0347] - Rice: Locating Y-leaves

[0348] - Sugar beet: Finding the second fully grown leaf

[0349] - Black walnut and / or pecan: Finding pairs of leaves

[0350] While embodiments have been described in which measurement areas are determined on leaves of plants, the disclosed techniques are also applicable to other plant parts.

[0351] The present disclosure also encompasses embodiments with any combination of features mentioned or shown for different embodiments. It also encompasses individual features in the figures, even if they are shown there in connection with other features and / or are not mentioned above or below. Furthermore, the alternative embodiments described in the figures and the description and individual alternative features thereof may be excluded from the subject matter of the invention or from the disclosed subject matter.

[0352] The terms "comprising" and "having" and derivatives thereof indicate a non-exhaustive relationship and do not exclude the presence of other elements or steps. The indefinite article "a" or "an" and derivatives thereof do not exclude the presence of a plurality of the corresponding elements. The functions of several features listed in the claims may be fulfilled by one unit or step.

[0353] A machine-readable instruction code that can be executed by a programmable circuit to perform methods according to embodiments may be stored and / or distributed on a suitable medium, such as an optical storage medium or a solid-state medium provided together with or as part of other hardware. The instruction code may also be distributed in another form, such as a modulated data signal sequence.

[0354] Embodiments of the invention provide improved support for plant testing.

Claims

CLAIMS 1. Device (10) or system (1) for supporting an examination of a plant or of several plants (2), comprising: an image capture device (21) which is set up to capture at least one image (60) of at least a part of the plant or of the several plants (2) and a human-machine interface (24) which is set up to output information determined on the basis of the at least one image (60) which indicates at least one area which is suitable for a measurement for examining the plant or of the several plants (2).

2. Device (10) or system (1) according to claim 1, wherein the device (10) or the system (1) is configured such that the information indicates which areas are suitable for measurement for a spectral analytical, camera-based or laboratory-based examination of the plant or the plurality of plants (2).

3. Device (10) or system (1) according to claim 1 or claim 2, wherein the device (10) or the system (1) is arranged such that the information indicates which areas are suitable for a nutrient concentration determination, active ingredient concentration determination and / or determination of a health status of the plant or of the plurality of plants (2).

4. Device (10) or system (1) according to one of the preceding claims, wherein the device (10) or the system (1) is arranged to determine and output the information depending on the examination to be carried out.

5. The device (10) or system (1) of claim 4, wherein the human-machine interface (24) is configured to receive a user input specifying the examination to be performed.

6. Device (10) or system (1) according to one of the preceding claims, further comprising an evaluation circuit (26) which is configured to determine the information to be output based on the at least one image (60), wherein the evaluation circuit (26) is configured to determine the information depending on one, several or all of the following information: Plant stage, leaf stage, plant species, Plant variety, at least one nutrient to be analyzed, at least one type of biotic or abiotic stress, position of the leaf on the plant.

7. Device (10) or system (1) according to claim 6, wherein the evaluation circuit (26) is configured to determine the plant species, the plant variety, the plant stage, the leaf stage and / or the position of the leaf on the plant based on an evaluation of the at least one image (60) and / or based on a user input received via the human-machine interface (24).

8. Device (10) or system (1) according to claim 6 or claim 7, wherein the evaluation circuit (26) is configured to evaluate the at least one image (60) with at least one trained machine learning model (70) for determining the information, which is configured to receive as input pixel values ​​of the at least one image (60) or input values ​​determined from preprocessing (75) of the pixel values.

9. Device (10) or system (1) according to one of claims 6 to 8, wherein the evaluation circuit (26) is configured to carry out an automatic detection of leaves based on the at least one image (60) in order to determine the information.

10. Device (10) or system (1) according to one of the preceding claims, wherein the device (10) or the system (1) is arranged such that the information indicates both leaves suitable for the measurement and leaf regions of the leaves suitable for the measurement.

11. Device (10) or system (1) according to one of the preceding claims, wherein the device (10) or the system (1) is configured to output the information as information (30) superimposed on the at least one image (60) in a visually perceptible manner via the human-machine interface (24).

12. Device (10) or system (1) according to one of the preceding claims, further comprising a housing (20) in which the image capture device and the human-machine interface (24) are mounted.

13. A method for assisting plant testing, comprising: capturing at least one image (60) of at least part of a plant or several plants (2), Evaluating the at least one image (60) by an evaluation circuit (26) to determine information indicating at least one area suitable for a measurement for examining the plant or the plurality of plants (2), and Providing the information for output via a human-machine interface (24).

14. A method for providing an evaluation circuit (26) to support a plant examination, comprising: Generating a plurality of trained machine learning models (70), each of which is configured to receive pixel values ​​of at least one image of a plant or of several plants (2) or of input values ​​determined from the pixel values ​​by preprocessing, and Storing parameters of the plurality of trained machine learning models (70) for use by the evaluation circuit (26), wherein each of the trained machine learning models (70) is assigned to at least one examination to be carried out, and wherein the generation of the plurality of trained machine learning models (70) comprises training with a training data set of training images, each of which is provided with annotations that indicate which area or areas of the plant or multiple plants (2) represented in the respective training image are suitable for the at least one examination.

Images