Method and system for automatic plant image annotation

Abstract

This invention relates to a computer-implemented method comprising: - acquiring (406) first training images (108) using a first image acquisition technique (104), each first training image depicting a plant-related motivation; - acquiring (402) second training images (106) using a second image acquisition technique (102), each second training image depicting a motivation depicted in a corresponding first training image; - automatically assigning (404) at least one label (150, 152, 154) to each acquired second training image; - describing the motivation... The first and second training images with the same motivation are spatially aligned (408) into aligned training image pairs; - a machine learning model (132) is trained (410) based on the aligned training image pairs and the labels, wherein, during training, the machine learning model (132) learns to automatically assign one or more labels (250, 252, 254) to any test image (205) acquired using the first image acquisition technique that depicts plant-related motivations; and - the trained machine learning model (132) is provided (412).

Description

Technical Field

[0001] This disclosure relates to methods and systems for improving the labeling of images depicting plants and plant-related motivations. Background Technology

[0002] Spectral imaging uses specialized sensors to detect the light reflected from an object. The reflected light is measured within a spectral band. Several spectral imaging methods acquire light across many spectral bands and / or over a large area of ​​the visible or invisible spectrum. Generally, the more spectral bands and the wider the spectral range covered, the higher the accuracy, flexibility, and information content of the acquired spectral image.

[0003] Spectral imaging is now widely used in agriculture and precision farming. For example, satellites and cameras mounted on drones are using multispectral or hyperspectral imaging techniques to acquire crop images containing high information content, and thus enabling the execution of various image analysis techniques.

[0004] For example, patent application WO2017105177A1 describes a precision agriculture system that uses maps of multispectral and hyperspectral images captured by high-speed and high-resolution cameras mounted on the underside of an unmanned aerial vehicle that serves as a geographic coordinate reference.

[0005] In precision agriculture, several image classification algorithms exist based on hyperspectral image data. However, hyperspectral and multispectral cameras are expensive. Furthermore, hyperspectral and multispectral cameras are typically larger and / or heavier than RGB cameras, making it generally impossible, or at least difficult or expensive, to mount such cameras on existing but less stable scaffolding, buildings, or drones. Meanwhile, drones with good RGB cameras are available in a wide range of consumer markets. Their large quantity results in relatively lower prices, while drones with hyperspectral cameras are usually custom-made, serving only a small market, and are therefore relatively expensive.

[0006] Therefore, the use of image data in precision agriculture and related technologies is associated with the following issues: Existing image classification algorithms require hyperspectral image data as input, but acquiring this data is generally more expensive and technically complex compared to other relatively simple image acquisition techniques (such as using a standard camera to capture RGB images). However, RGB images generally cannot replace hyperspectral images because their information content differs from that of hyperspectral images, and therefore cannot be used as input for existing hyperspectral image-based image analysis methods.

[0007] For machine learning systems / processes to generate high-quality models capable of accurately solving prediction tasks, they must be trained on a sufficiently large training data corpus. If the data base is too small or biased, the models obtained during training will have poor generalization, low accuracy, and will be imbalanced. However, labeling a sufficiently large training data corpus requires a significant amount of labor. This is often the bottleneck in generating highly accurate ML models that can be used for various prediction tasks, such as semantic tagging of digital images.

[0008] Nguyen et al. described the automatic copying of hyperspectral images from RGB images in their presentation "Training-Based Spectral Reconstruction from a Single RGB Image" at ECCV 2014. For some tasks, such as scene lighting, a "computational" hyperspectral image can be used instead of a "real" hyperspectral image. However, since RGB images only cover a small portion of the spectrum of a hyperspectral image, a "computational" hyperspectral image may not accurately replace a "real" hyperspectral image in other applications. Summary of the Invention

[0009] This invention provides an improved method, computer program product, and system for analyzing digital images describing plant-related motivations as described in the independent patent claims. Embodiments of the invention are given in the dependent claims. These embodiments can be freely combined with each other without mutual exclusion.

[0010] In one aspect, the present invention relates to a computer-implemented method comprising:

[0011] - First training images are acquired using a first image acquisition technique, each first training image depicting a plant-related motivation, wherein the plant-related motivation is selected from the group including: indoor or outdoor agricultural areas, plants, plant products, a part of a plant, and a part of a plant product;

[0012] - Use a second image acquisition technique to acquire second training images, each second training image depicting the motivation depicted in a corresponding first training image;

[0013] - Automatically assign at least one label to each acquired second training image;

[0014] - Align the first and second training images depicting the same motivation in the motivation in space to form an aligned training image pair;

[0015] - A machine learning model is trained based on aligned training image pairs and labels, wherein, during training, the machine learning model learns to automatically assign one or more labels to any test image acquired using a first image acquisition technique depicting plant-related motivations; and

[0016] - Provides trained machine learning models.

[0017] This can be advantageous because embodiments of the invention allow for the provision of software (here: a trained machine learning model) suitable for automatically calculating and predicting tags based on readily available digital images (obtained using a first image acquisition technique), thereby automatically predicting the value and location of one or more tags in readily available digital images, even if high-quality digital images (e.g., hyperspectral images typically used for calculating tags) are unavailable.

[0018] Embodiments of the present invention provide a trained ML model suitable for automatically predicting and assigning labels to any digital input image acquired using a first image acquisition technique. Therefore, embodiments of the present invention accelerate and simplify the generation of labeled datasets that can be used for various purposes, such as assessing plant health or other phenotypic information. Manually creating annotated training data is a time-consuming, expensive, and error-prone process. Using information contained in information-rich digital images acquired using a second, often complex and / or extended, image acquisition technique for automatically predicting labels, and training the ML model on aligned pairs of first and second training images, can have the advantage of generating a trained ML model that is also suitable for automatically predicting and generating these labels based on the first image acquired using a relatively inexpensive / low-complexity first image acquisition technique. This allows for the avoidance of tedious and biased manual labeling, as a trained ML procedure is provided that is suitable for fully automated calculation and assignment of labels.

[0019] The provided trained ML model allows for the generation of accurate spatial labels for images acquired using a first image acquisition technique, based on the learned correlation between image features of a first training image and labels assigned to a second training image. The labels for the training images are derived from spectral and / or spatial information contained in the second training image, which may not be present in the first training image. Learning this correlation between image features acquired using the first image acquisition technique and true labels obtained from images acquired using the second image acquisition technique allows for the avoidance of time-consuming and potentially biased manual pixel-by-pixel labeling of the first (training and testing) images.

[0020] In precision agriculture and other related technical fields, generating labeled images as training data for various machine learning methods is extremely time-consuming. Embodiments of the present invention use a trained ML model to automatically create labels at test time. The trained ML model can be created very quickly, for example, automatically or semi-automatically. For example, a second training image is acquired, containing sufficient information to allow for rapid and automatic label prediction and assignment. These labels are mapped to a first training image depicting the same motivation. The first training image and the labels are used as input to train the ML model so that the model learns to predict labels based on image features from the first training image. In other words, a second training image, which can be easily and fully automatically labeled, can be used together with the first training image (which can be obtained more cheaply, faster, etc.) as input to generate a trained model capable of predicting these labels based on digital images obtained via a first image acquisition method.

[0021] According to an embodiment, the plant-related motivation is a macroscopic motivation with a size of at least 1 mm, particularly at least 1 cm.

[0022] According to an embodiment, the acquisition of a corresponding second training image in each of the first training images and the second training images is performed simultaneously or closely consecutively, such that the optical sensors used to acquire the first and second training images that subsequently form one of the aligned image pairs have substantially the same or highly similar sensor positions relative to the plant-related motivation. In other words, the first and second training images depicting the same or similar motivations that are spatially aligned with each other during training are images acquired simultaneously or closely consecutively. Therefore, the motivations depicted in the first and second training images of each aligned image pair are the same or at least approximately the same.

[0023] According to an embodiment, the first image acquisition technique may be a cheaper and / or less technically complex image acquisition technique compared to the second image acquisition technique. For example, the first image acquisition technique may be the acquisition of an RGB image, while the second image acquisition technique may be the acquisition of a hyperspectral image.

[0024] According to one embodiment, assigning at least one label to each of the second training images is implemented such that a corresponding label is assigned to each pixel of the second training image. According to other embodiments, at least some labels are assigned to groups of pixels, for example, to individual pixel blocks or image segments. According to a further embodiment, a single label is assigned (“globally”) to the second training image.

[0025] According to embodiments, tags may include or consist of one or more data values ​​of any type. For example, tags may include Boolean values, numeric values, or alphanumeric strings, or a combination thereof. These tags can be used to indicate the membership of a pixel, pixel block, or image in a predefined category (e.g., “soil,” “healthy plant,” or “infected plant”).

[0026] According to an embodiment, the trained ML model is a model suitable for performing prediction tasks, where the prediction task can be a regression problem or a classification problem.

[0027] According to an embodiment, the prediction task includes predicting the value and location of a tag and assigning the tag to a digital image.

[0028] For example, each label can indicate a category name, a numerical value indicating the probability of category membership, or another parameter value related to downstream image analysis operations. For example, category name labels can be used for image classification tasks. For example, numerical labels can be used for regression tasks, for the entire image, a portion of an image, or each individual pixel (e.g., semantic image segmentation, which considers adjacent pixel regions with the same or sufficiently similar labels as pixels belonging to the same segment).

[0029] For example, each label can be a number, such as an integer or a floating-point number. According to an embodiment, the trained ML model has learned and been configured to automatically assign these numerical values ​​as labels to any input image acquired via a first image acquisition technique during testing. Thus, the trained ML has learned to automatically label input images with numerical values, such as percentage values. The labeled numerical values ​​of one or more test images can be used to assess plant health or other phenotypic information. For example, this assessment can be implemented as a regression problem using automatically labeled test images. For example, a numerical value could be "68%", indicating the probability that a pixel, pixel patch, or image depicts a specific category of object (e.g., "soil").

[0030] According to another example, each tag represents the name of a class included in a finite set of predefined classes. These classes can indicate the category of a motif or object to which an image, pixel patch, or individual pixel belongs. For example, category names can be specified as Boolean values ​​(e.g., "true" for infected and "false" for healthy) or strings ("healthy plant", "infected plant", "soil", "50% soil - 50% healthy plant", "50% soil - 50% infected plant") or numerical values ​​(e.g., the 77% probability of belonging to the "healthy plant" category). According to an embodiment, the trained ML model has learned and is configured to automatically assign these predefined category names as tags to any input image acquired via a first image acquisition technique at test time. Thus, the trained ML has learned to automatically tag input images with the category names of one or more predefined classes. The tags assigned to one or more test images can be used as second-order training data, which is provided as input to another ML program that learns to solve a classification problem based on the automatically tagged test images.

[0031] According to one embodiment, the alignment of a first training image and a second training image depicting the same motivation is performed manually. According to other embodiments, automatic image alignment techniques such as image registration, photogrammetry, or georeferenced techniques are used to perform the alignment. According to one embodiment, alignment can be performed on a per-pixel basis.

[0032] According to an embodiment, the spatial alignment of the first and second training images in each pair includes: aligning the first and second images depicting the same motivation based on their respective geographical locations to provide a roughly aligned image pair; and then refining the alignment based on pixel intensity and / or color similarity to minimize the intensity and / or color differences between the first and second images to provide alignment of the first and second images in the pair. The plant-related motivation depicted in the test image is preferably similar to the plant-related motivation depicted in the first and second training images. For example, if training is performed on individual plants of a specific species, the test image should also be an image of the same or similar species of plant acquired at a similar relative location between the optical sensor and the plant. If training is performed in an agricultural area, the test image should also be an image of the agricultural area acquired at a similar relative location between the optical sensor and the agricultural area, and preferably under similar ambient light conditions.

[0033] According to an embodiment, the method includes: using a trained machine learning model to predict one or more labels for a test image; and outputting the labeled test image.

[0034] These steps can be repeated on multiple test images acquired using the first image acquisition technique. Therefore, a large number of automatically labeled test images can be computed. This set of labeled images can be used for various purposes. For example, to perform image segmentation of the test images, labels can be used to identify areas in agricultural regions that require the application of water, fertilizer, pesticides, and / or fungicides, or areas where a certain type of plant should be planted or harvested. One or more labeled test images output by a trained ML model can also be used as training images for various second-order machine learning tasks.

[0035] According to an embodiment, the method further includes extracting a first feature (also referred to as an "image feature") from each first training image. Training of the ML model is performed such that the ML model learns the spatial correlation of the first feature and label assigned to the second training image based on the spatial correlation of the first feature and label within each training image in the aligned first and second training image pair.

[0036] For example, features can be extracted from each first training image using a machine learning program used to train an ML model. Several neural network architectures available today for image-based machine learning tasks include various image feature extraction algorithms that are applied to any input image to use the extracted first image features as input during training. For example, various neural networks such as DeepLabv3+ include image feature extraction algorithms. During training, the ML model learns the relevance of at least some automatically extracted first image features or combinations of these first image features with the content and location of labels assigned to second training images (the labels of the second training images are already aligned with the labels of first training images describing the same or substantially the same plant-related motivations).

[0037] According to embodiments, the first feature includes one or more image features selected from the group consisting of: intensity value, intensity gradient, contrast, intensity gradient direction, color index and / or spectral index, and linear and nonlinear combinations of two or more of the above image features. According to some embodiments, the software program for training the ML model includes one or more algorithms for automatically extracting multiple distinct image features from an image, and during model training, the model learns a subset of the first features and / or combinations of two or more first features that are particularly predictive for a specific marker aligned with a first training image.

[0038] According to an embodiment, the method further includes extracting a second feature (also referred to as an "image feature") from each second training image. Automatically assigning at least one label to each second training image includes: analyzing the second feature extracted from the second training image to predict at least one label for the second training image based on the second feature extracted from the second training image.

[0039] According to an embodiment, the extraction of the second feature is also performed by an ML program for training an ML model. However, according to a preferred embodiment, the extraction of the second image features from the second training image is performed by a separate feature extraction software program or module. This may be advantageous because the content and location of the tags can vary greatly depending on the respective computational tasks, and therefore standard image feature extraction algorithms included in current machine learning frameworks may not be able to cover the types of second image features required for automatically predicting tags.

[0040] For example, the second training image can be a hyperspectral image, and the label to be assigned to the second training image can be selected from a finite set of predefined category names listed below, where “S” represents “soil,” “HP” represents “healthy plants,” and “IP” represents “infected plants”: “~100%S,” “~100%HP,” “~100%IP,” “~50%S&~50%HP,” “~50%S&~50%IP,” “~50%IP&~50%HP,” “~25%S&~25%HP~50%IP,” “~25%S&~50%HP~25%IP,” “~50%S&~25%HP~25%IP.” Therefore, the label “~50%S&~50%HP” indicates that the image area assigned this label depicts an agricultural area that is half covered by soil and half by healthy plants. From each second training image… The second feature extracted from the image can be, for example, a spectral signature of a pixel or pixel block. The extracted signature can be compared with a predefined set of known spectral reference signatures. Each spectral reference signature can be a characteristic of a corresponding category in one or more of the aforementioned categories. For example, there might be a reference spectral signature of "100% S," representing a specific intensity pattern over a broad spectral range characteristic of bare soil. Similarly, there might be a reference spectral signature of "100% HP," representing a specific intensity pattern over a broad spectral range characteristic of healthy plants of a particular species. By comparing the extracted feature ("second image feature") with the reference spectral features, one of the reference features most similar to the extracted second feature can be identified, and the category name of the class represented by the identified reference spectral signature can be used as a label assigned to the pixel or pixel block from which the spectral signature was extracted.

[0041] According to an embodiment, plant-related motivation is an indoor or outdoor agricultural area with a variety of plants, plants, plant products, parts of plants, and parts of plant products, whereby no plants or plant products are modified, chemically treated, and / or dyed to provide labeling or promote labeling.

[0042] It may not be necessary to stain or chemically treat the plant to visualize certain features, because in particular, hyperspectral reference features can allow for the detection of virtually any feature of the plant that affects how the plant absorbs or reflects light.

[0043] According to other embodiments, the second feature includes one or more image features selected from the group consisting of: spectral features, spectral indices, spatial spectral features (features that specify the relationship between intensity and space, such as spectral gradients), and intensity-based features.

[0044] The number and type of the second feature, and therefore the algorithm used to extract the second feature, largely depend on the label that should be predicted based on the second feature.

[0045] According to an embodiment, the first image acquisition technique and the second image acquisition technique are different image acquisition techniques selected from the group consisting of:

[0046] - Hyperspectral image acquisition;

[0047] - RGB image acquisition;

[0048] - Monochrome image acquisition; for example, monochrome image acquisition may include using a monochromator (an optical device that transmits mechanically selectable narrowband light wavelengths or other radiation selected from a wider range of wavelengths available at the input).

[0049] - Active image acquisition using excitation light source;

[0050] - Multispectral image acquisition; and

[0051] -IR image acquisition.

[0052] According to an embodiment, automatically assigning labels to pixels or regions of each second training image in the second training images includes:

[0053] - For each of the predefined sets of motivation categories, obtain spectral reference features from the physical reference motivation belonging to that motivation category;

[0054] - Compare the spectral reference features with the second training image to identify the spectral similarity between pixels or regions in the second training image and the spectral reference features; and

[0055] - Assign a label to each pixel or region in each of the second training images, the label indicating the motivation category that is most similar to that pixel or region in the spectrum.

[0056] Using spectral features extracted from the second training image and spectral reference features as properties of the object category of interest to predict labels can be advantageous because spectral features are informative and readily available, provided the image covers a sufficiently wide spectral range (especially for hyperspectral and multispectral images). Therefore, accurate label predictions based on a single feature type can be provided. Using spectral features as a second feature for label prediction leverages the fact that certain objects or motives leave unique “fingerprints” in the electromagnetic spectrum, which are highly characteristic of object categories.

[0057] Furthermore, this method is highly flexible and can be used to automatically identify virtually any type of object of interest without requiring significant adjustments to the feature extraction algorithm used to extract the second feature (here, spectral features). For example, to automatically detect a plant of a specific species or physiological state in multiple second training images, it may be sufficient to obtain one or more hyperspectral or multispectral reference images depicting the agricultural area covered by that specific plant. Reference features are then extracted from those portions of the one or more reference images depicting these plants. The feature extraction steps for extracting the second feature from the second training images include: extracting spectral features at each pixel in each second training image and using them to divide the second training images into similar pixel groups (fragments) using different methods. As a final step, a label (e.g., a category name) is assigned to each fragment (or each pixel in a fragment) for example by comparing the features of each pixel (or the average spectral features of the pixels in the fragment) with known spectral reference features of the specific plant of interest (or other object). Ultimately, a correct match between the spectral features in the pixels or fragments of the second training images and the reference spectral features leads to accurate predictions and the assignment of labels indicating the presence of the aforementioned plant of interest to the second training images. By adding additional reference spectral features to the reference spectral feature library and considering these additional reference spectral features during feature comparison and label prediction, this approach can be easily extended to other object categories.

[0058] Therefore, using spectral features for label prediction can be beneficial because it eliminates the need to define algorithms that explicitly search for color, shape, texture, or other features that programmers consider "characteristics" of a particular object. This can have the advantage that many "characteristics" may not be included in the "visible" portion of the spectrum and are therefore imperceptible to humans.

[0059] According to an embodiment, the second image acquisition technique is hyperspectral image acquisition using a hyperspectral sensor.

[0060] This can have the advantage of the second training image being rich in spectral information. The spectral features derived from each pixel of the second training image can therefore be used to predict labels that indicate the type of object depicted in the image (or the probability of depicting that type of object) with high accuracy.

[0061] According to an embodiment, the second image acquisition technique covers a larger portion of the entire electromagnetic spectrum than the first image acquisition technique (e.g., from below 1 Hz to above 10 Hz). 25 hertz).

[0062] For example, the second image acquisition technique could be hyperspectral imaging or multispectral imaging, while the first image acquisition technique could be RGB image acquisition or monochrome image acquisition. This can have the advantage of the second training image being rich in spectral information, and thus provides a good foundation for accurately predicting tags solely or primarily based on spectral information (e.g., spectral features of individual pixels). Image acquisition techniques like RGB or monochrome imaging have the advantages that the optical sensors used are typically inexpensive, mechanically robust, lightweight, and / or have high spatial resolution. The ML model is trained by spatially allocating the image features of the first training image to the tags predicted based on the spectral information in the second training image, thus providing a trained ML model capable of predicting tags based on features extracted from images acquired using relatively inexpensive image acquisition techniques, which may contain less spectral information than images acquired using the second image acquisition technique.

[0063] According to an embodiment, compared with the first image acquisition technique, the second image acquisition technique covers different parts of the entire electromagnetic spectrum (e.g., IR instead of UV, and vice versa).

[0064] According to an embodiment, compared with the first image acquisition technique, the second image acquisition technique is characterized by higher information content per spatial region, specifically, higher spectral information content per spatial region. For example, the second training image may include more "data layers" than the first training image, such as more spectral band-specific sub-images for each region.

[0065] According to an embodiment, the second image acquisition technique is characterized by information content for each spatial region that differs from that of the first image acquisition technique (e.g., spectral information related to different wavelength ranges in the electromagnetic spectrum).

[0066] According to an embodiment, the first image acquisition technique has a higher spatial resolution than the second image acquisition technique.

[0067] According to an embodiment of the present invention, the second image acquisition technique is a hyperspectral image acquisition technique, and the first image acquisition technique is an RGB image acquisition technique or a multispectral image acquisition technique.

[0068] For example, to acquire the first training and / or test images, a “standard” RGB camera on a smartphone, an RGB camera integrated into a drone used in precision agriculture, or an RGB camera integrated into a microscope for acquiring magnified images of plants, plant products, and their parts with high spatial resolution can be used. The spatial resolution of these RGB cameras is significantly greater (typically orders of magnitude) than that of many hyperspectral cameras used in precision agriculture (e.g., hyperspectral cameras for precision agriculture). This can be advantageous because the high information density of spatial feature information contained in the first training image allows the ML model to learn during training to associate spatial image features extracted from the first training image as first features with spatially aligned markers already predicted based on spectral information included in the second training image. Thus, the ML program “learns” during training to predict markers based on image features that exist only in the first training image but are absent (or have reduced presence) in the second training image. Therefore, according to an embodiment, a low-resolution hyperspectral image with abundant spectral information is used to automatically predict markers, which are then automatically aligned to a high-resolution image with lower spectral information. The applicant observed with surprise that high-resolution images with limited spectral information (e.g., RGB or monochrome images) often contain features that enable the successful training of accurate ML models based on labeled data from different image acquisition techniques (e.g., hyperspectral imaging). This requires less manual labor and allows for a (semi-)automated workflow compared to existing methods (e.g., manual labeling). For example, a trained ML model could infer disease progression states from spectral features acquired at low resolution using a hyperspectral sensor, based on variations in the shape and / or color of objects in a high-resolution RGB image, as a result of the training process.

[0069] According to an embodiment, the first training image is an RGB image. The second training image is a hyperspectral image. The spatial alignment of the first and second training images in each of these pairs includes:

[0070] - For each pixel of the second training image, the intensity values ​​of red, green and blue are calculated by averaging the spectral intensity values ​​of the visible red, green and blue spectral bands covered by the second training image.

[0071] - Generate an RGB representation of the second training image based on the calculated intensity values ​​of red, green and blue; for example, the RGB representation can be calculated by combining the three intensity values ​​of red, green and blue into pixel-based tuples, or preferably based on a more complex method, such as r = R / (R+G+B), g = G / (R+G+B), b = B / (R+G+B);

[0072] - Calculate a first greenness image, where the intensity of each pixel in the first greenness image is a greenness index calculated based on the intensity values ​​of red, green, and blue in the first training image;

[0073] - Calculate the second greenness image, where the intensity of each pixel in the second greenness image is a greenness index calculated based on the intensity values ​​of red, green, and blue in the RGB representation of the second training image;

[0074] - Automatically perform alignment of the first and second training images to minimize the difference in greenness index between the first and second images.

[0075] The greenness index highlights greening motivating components such as plants. Greenness can separate green tissue from the residue / soil background. For example, it can be calculated using the following formula: Greenness = 2G - RB or Greenness = 2G + B - 2R.

[0076] Using image alignment based on the greenness index can be beneficial because the alignment process inherently focuses on objects of particular interest, namely plants and plant parts that are typically green. Plants and plant parts are flexible objects whose precise outlines can vary slightly, for example, depending on wind and other parameters. Therefore, using the greenness index in the context of image alignment may be more suitable for aligning image regions that depict plants and plant parts than alignment methods based, for example, on GPS coordinates assigned to certain pixels in the training image.

[0077] According to an embodiment, the trained machine learning model is configured to: assign one or more tags to a test image such that at least one of the tags is assigned to each pixel of the test image; and to perform semantic segmentation of the test image by grouping pixels of test images that share the same tag or share sufficiently similar tags into the same segment.

[0078] For example, a "sufficiently similar" tag could be a tag whose similarity value is calculated relative to another tag, such that the similarity value is higher than a predefined threshold. According to another example, the tag could be a numerical value, and two "sufficiently similar" tags could be tags within the same predefined numerical range in a predefined set of numerical ranges, or tags whose numerical difference is lower than a predefined threshold.

[0079] These embodiments of the invention can be advantageous because they provide a trained model that can be applied to perform image segmentation tasks quickly and accurately without relying on image acquisition using a second image acquisition technique. In another advantageous aspect, the segmentation results can be compared with a reference image that includes the “known” correct segmentation boundaries, allowing for the identification of biases in the trained model. For example, the model may accurately identify “soil segments” and, with high sensitivity, “healthy plant segments” in images depicting agricultural areas. However, the trained model may sometimes incorrectly identify areas depicting infected plants as “healthy plant” segments. This “bias” regarding the model’s predictions can be identified by comparing segments identified by the trained model with known correct “reference segments.” This is a major advantage compared to classification systems based on the overall image that only allow identification of overall accuracy. Furthermore, these segments can be highly valuable for various downstream processing methods, such as identifying sub-regions within agricultural areas that require irrigation or application of fertilizers, pesticides, and / or herbicides.

[0080] According to an embodiment, the trained machine learning model is a neural network, specifically a neural network including at least one bottleneck layer. A bottleneck layer is a layer containing fewer nodes compared to previous layers. It can be used to obtain a dimensionality-reduced input representation. An example of this is using an autoencoder with a bottleneck layer for non-linear dimensionality reduction.

[0081] According to an embodiment, the motivations depicted in the plant-related training and testing images are indoor and / or outdoor agricultural areas. These markers are selected from a group of predefined motivation categories that include:

[0082] - Areas covered with healthy plants;

[0083] - Areas covered with plants infected with specific diseases and / or parasites;

[0084] - Areas covered with plants of specific species;

[0085] - Areas covered with specific varieties (i.e., subspecies) of plants;

[0086] - Cover areas of plants with a specific substance, specifically fungicide, insecticide, herbicide and / or fertilizer;

[0087] - Areas covered with plants treated according to a specific irrigation program;

[0088] - Areas not covered by any plants;

[0089] - Areas covered with specific types of soil;

[0090] -A region that covers a mixture of two or more predefined portions of the above-mentioned types of coverings.

[0091] These features can be particularly beneficial in the context of precision agriculture. For example, for each of the aforementioned tags and corresponding object categories, a corresponding spectral reference feature can be empirically acquired and stored in a storage medium of a computer operatively coupled to a processor that performs training on the ML model. Furthermore, spectral reference features can be empirically acquired, corresponding to mixed categories, comprising objects from two or more of the aforementioned categories in a given proportion. For example, one of these additional spectral reference features could be obtained from an agricultural area where approximately half is covered by soil and the other half by healthy vegetation. The trained ML model can be used to automatically assign tags indicating any of the aforementioned categories to a test image acquired using a first image acquisition technique. For example, this can predict whether a particular agricultural area or sub-area requires irrigation, treatment with fungicides, fertilizers, or herbicides, harvesting, or any other form of substance or treatment, such as to increase crop yield and / or combat infectious diseases or parasites.

[0092] According to other embodiments, the motivation depicted in plant-related training and testing images is a plant, a plant product, and / or a part of a plant or plant product. These labels are selected from a group of predefined motivation categories that include:

[0093] - A surface area of ​​a plant or a product or part of that plant, wherein the surface area is healthy;

[0094] - A surface area of ​​a plant or a product or part of that plant, wherein the surface area displays symptoms associated with infection with a specific disease in that area;

[0095] - A surface area of ​​a plant or a product or part of that plant, wherein the surface area shows that the area is infected by a specific parasite;

[0096] - A surface area of ​​a plant or a product or part of the plant, wherein the surface area displays cellular structures or organelles within a predetermined range;

[0097] - A surface area of ​​a plant or a product or part of the plant, wherein the surface area displays cellular structures or organelles in a predetermined state;

[0098] - A surface area of ​​a plant or a product or part of the plant, wherein the surface area shows morphological changes caused by the local application of a particular substance;

[0099] - A surface area covered by a mixture of two or more predefined portions of the above types of surface areas.

[0100] These methods described above can be used for diagnostic purposes, such as identifying diseases or parasites infecting specific plants or parts of plants. Alternatively, they can be used for quality control of seeds and other forms of plant products, for example, to ensure that seeds are free from infection and parasites and are likely to germinate. These methods can also be used to test the effectiveness of topically applied specific pesticides or fungicides.

[0101] According to an embodiment, automatically tagged test images are processed to identify plants that include one or more desired or undesirable traits, and plants with desired traits and / or lacking undesirable traits are selectively used in plant breeding projects.

[0102] In another aspect, the present invention relates to a computer-implemented method for automatically assigning one or more tags to a test image acquired using a first image acquisition technique. The test image depicts a plant-related motif. The method includes:

[0103] - Provides a trained machine learning model that is suitable for automatically predicting one or more labels to assign to any input image acquired using a first image acquisition technique and depicting motivations related to plants.

[0104] - Use a trained machine learning model to predict one or more labels for a test image; and

[0105] - Output the predicted labels for the test image.

[0106] This can be advantageous because even if a second image acquisition technique for generating the second training image is unavailable (e.g., because the corresponding camera is too expensive, too complex to use or maintain, or too heavy to be mounted on a drone), the trained ML model can still automatically predict the labels for the test image, even though the test image was acquired using the first image acquisition technique.

[0107] According to an embodiment, the trained machine learning model is adapted to automatically predict one or more labels based on the spatial relevance learned from a first feature extracted from a first training image obtained using a first image acquisition technique and labels assigned to a second training image, the second training image displaying the same motivation as the first training image and spatially aligned with the first training image.

[0108] According to embodiments of the invention, the expression "the same motivation" as used herein means "at least substantially the same," because in many real-world applications, the first and second training images with the same motivation are captured sequentially in time, for example, within a delay of less than one hour, preferably less than 20 minutes, more preferably less than 5 minutes, and more preferably less than 5 seconds. However, this brief time period may result in minute absolute changes in the position of plants and plant parts due to wind or other environmental factors, or may result in minute relative changes between the position of the depicted motivation and the camera used to acquire the first and second training images. However, since these differences are generally small, the motivation depicted in the first and second training images, which are very close in time and spatially aligned with a pair of training images, can be considered at least substantially the same.

[0109] According to embodiments, the method further includes generating a trained machine learning model according to any of the embodiments described herein.

[0110] In another aspect, the present invention relates to an image analysis system. The image analysis system includes at least one processor and a volatile or non-volatile storage medium. The storage medium includes computer-interpretable instructions that, when executed by the at least one processor, cause the processor to perform any of the computer-implemented methods described in the embodiments herein.

[0111] For example, an image analysis system can be a standard computer (e.g., a desktop computer) or a distributed computer system (e.g., a cloud computer system).

[0112] The image analysis system may include an interface for receiving a first training image and a second training image and / or for receiving one or more test images acquired using a first image acquisition technique. For example, the interface may be a network interface for receiving images from another computer system. Alternatively, the interface may be a USB interface for receiving images via a USB storage device. Furthermore or alternatively, the image acquisition system is operatively coupled to one or more optical sensors configured to acquire the first and / or second training images and / or configured to acquire one or more first test images. The image analysis system may be used both to train an ML model and to apply the trained ML model to one or more test images. Alternatively, different image analysis systems may be used to train an ML model and to apply the trained model to one or more test images. In this case, each image analysis system may have the features described above. The image analysis system for training the ML model preferably includes software applications for extracting second features from the second training images, software for predicting labels based on the second features, and software for spatially aligning pairs of first and second training images (including their labels) depicting the same plant-related motivations. Image analysis systems for training ML models and / or applying trained ML models include software for extracting a first feature from any image acquired using a first image acquisition technique that has been input into the machine learning model. In some embodiments, the software for extracting the first feature may be a component of machine learning software for generating the trained ML model.

[0113] "Plant products" can be, for example, one or more seeds, one or more fruits, cuttings, tubers, bulbs, onions, beans, etc.

[0114] The “training images” used in this paper are digital images used to train the ML model. Conversely, the “test images” used in this paper are digital images used as input to the trained model at the test time (“prediction time”). While the training images are provided to the model to be trained in association with labels deemed correct (“ground reality”), the test images are provided to the trained ML model without assigning any labels. Instead, the task of the trained ML program is to correctly compute and predict the labels and their locations.

[0115] As used herein, "spectral characteristics" refers to the change in a material's reflectance or emissivity relative to wavelength (i.e., reflectance / emissivity as a function of wavelength). For example, the spectral characteristics of a particular object (e.g., soil, a particular species of plant, a plant in a particular physiological state, etc.) can be a characteristic of that particular object type or state. The spectral characteristics of an object are a function of the incident electromagnetic wavelength and the interaction of the material with that portion of the electromagnetic spectrum. According to embodiments, the spectral characteristics of objects depicted in a digital image can be extracted in the form of image features per pixel, for example, for segmentation purposes or for groups of pixels, such as blocks.

[0116] As a final step, they are assigned a category (classification) to each group by comparing them with known spectral features. Depending on the pixel resolution, a pixel can represent many spectral features “mixed” together—which is why extensive remote sensing analysis is performed to “demix” them. Ultimately, correctly matching the spectral features recorded by the image pixels with the spectral features of existing elements results in accurate classification in remote sensing.

[0117] The “spectral reference features” used in this paper are spectral features taken from objects considered typical representatives of a class of objects. For example, software used to predict labels for a second training image may include a library of spectral reference signals. A first reference signal includes hyperspectral features derived from images depicting agricultural areas covered with bare soil. A second reference signal includes hyperspectral features derived from images depicting agricultural areas covered with healthy plants of a specific species. A third reference signal includes hyperspectral features derived from images depicting agricultural areas covered with plants of a specific species infected with a specific disease or parasite. Other reference signals include hyperspectral features derived from images depicting agricultural areas covered with a specific mixture of two or more of the aforementioned categories: “soil,” “healthy plants,” and “infected plants.”

[0118] As used in this paper, the term "machine learning (ML)" refers to the study, development, or use of computer algorithms that can be used to extract useful information from training datasets by automatically building probabilistic models (referred to as machine learning models or "predictive models"). Machine learning algorithms build mathematical models based on sample data (referred to as "training data") to make predictions or decisions without being explicitly programmed to perform a task. Machine learning can be performed using learning algorithms such as supervised or unsupervised learning, reinforcement algorithms, self-learning, etc. Machine learning can be based on a variety of techniques, such as clustering, classification, linear regression, support vector machines, neural networks, etc. A "model" or "predictive model" can be, for example, a data structure or program, such as a neural network, support vector machine, decision tree, Bayesian network, etc., or a portion thereof suitable for performing a prediction task. The model is suitable for predicting unknown values ​​(e.g., labels and / or label locations) from other, known values.

[0119] For example, an ML model can be a predictive model that has learned to perform prediction tasks (such as classification or regression). Classification is the problem of predicting discrete class labels for an input (such as a test image or a portion thereof). Regression is the problem of predicting a continuous number of outputs for an input.

[0120] The "hyperspectral image acquisition technology" used in this paper is an image acquisition technique that collects and processes information from the entire electromagnetic spectrum. The goal of hyperspectral imaging is to acquire the spectrum of each pixel in a scene image for the purpose of locating objects, identifying materials, or detecting processes. In hyperspectral imaging, the recorded spectrum has good wavelength resolution and covers a wide range of wavelengths. Unlike multispectral imaging, which measures intervals of spectral bands, hyperspectral imaging measures continuous spectral bands. According to embodiments, hyperspectral sensors are suitable for capturing electromagnetic signals within narrow spectral bands over a continuous spectral range, thereby generating the spectrum of all pixels in the scene. A sensor with only 20 bands covering the 500 to 700 nm range can also be hyperspectral, where each of the 20 bands is 10 nm wide. (While a sensor with 20 discrete bands covering visible, near-wave, short-wave, mid-wave, and long-wave infrared would be considered multispectral). Hyperspectral imaging (HSI) uses a continuous and continuous range of wavelengths (e.g., 400–1100 nm with a step size of 1 nm), while multispectral imaging (MSI) uses a subset of the target wavelengths at selected locations (e.g., 400–1100 nm with a step size of 20 nm).

[0121] Multispectral imaging techniques (acquiring 5-7 bands) have been observed to provide a good overview of crops, such as overall growth, but sometimes fall short in solving more complex problems, such as identifying weeds, certain diseases, or parasites. Hyperspectral techniques, due to their higher number of spectral bands, offer greater detection capabilities and can be used for virtually any problem encountered in precision agriculture and related fields.

[0122] The “RGB image acquisition technique” used in this article refers to any image acquisition technique that uses a camera or camcorder to provide digital images, where each pixel has been assigned a red (R), green (G), and blue (B) intensity value. For example, a digital camera using a CMOS or CCD image sensor can be used to acquire RGB images. This CMOS or CCD image sensor includes three different sensors for three spectral ranges corresponding to red, green, and blue light in the visible spectrum. Some RGB image acquisition systems can use a Bayer filter arrangement where the number of green detectors is twice that of red and blue (a ratio of 1:2:1) to achieve luminance resolution higher than chromaticity resolution. The sensor has a grid of red, green, and blue detectors arranged such that the first row is RGGRGRG, followed by GBGBGBGB, and this sequence is repeated in subsequent rows. For each channel, missing pixels are obtained by interpolation to construct a complete image. Furthermore, other processes can be applied to map the camera's RGB light intensity measurements to a standard RGB color space.

[0123] The "multispectral image acquisition technique" used in this article is an image acquisition technique suitable for capturing images in discrete and relatively narrow frequency bands. "Discrete and relatively narrow" is the difference between multispectral imaging at visible light wavelengths and color photography. Multispectral sensors can have many bands, covering the spectrum from visible light to long-wave infrared. Multispectral images do not produce the "spectrum" of an object.

[0124] The "monochrome image acquisition technique" used herein is an image acquisition technique suitable for providing digital images with a single "color" channel. According to some methods, a camera is used to capture a monochrome image suitable for selectively sensing light signals within a single, preferably narrow, spectral band. According to other embodiments, the camera is suitable for capturing electromagnetic signals over a wide spectral range, and the intensity information captured thereby is further processed to generate a monochrome image. For example, further processing may include applying one or more optical filters to filter out all spectral bands from the multispectral / broadspectral image except for the single spectral band. For example, the single spectral band may cover a relatively narrow spectral range whose wavelengths differ from the median wavelength of that range by less than 5%.

[0125] The “IR image acquisition technology” used in this article is an image acquisition technology that uses an instrument called an infrared spectrometer (or spectrophotometer) to capture the infrared spectrum of an object or scene. The infrared portion of the electromagnetic spectrum covered by the IR image acquisition technology according to embodiments of the present invention can cover the near-infrared (0.7-2.5 pm wavelength), mid-infrared (2.5-25 pm) and / or far-infrared (25-1000 points).

[0126] The “active image acquisition technique” used in this paper is any image acquisition technique, including any of the image acquisition techniques described above, that uses an excitation light source to illuminate the scene on which its image will be acquired. For example, the excitation light source can be a UV light source configured to emit UV pulses on plant-related motifs (e.g., the whole plant, a seed, or a part thereof). The UV pulses can induce a fluorescence signal, which is captured by an enhanced CCD camera. For example, active imaging can be used for fluorescence imaging of plants, specifically multispectral fluorescence imaging of leaf fluorescence emission band maxima, i.e., in the blue (440 nm), green (520 nm), red (690 nm), and far-red (740 nm) spectral regions. For example, blue-green fluorescence originates from ferulic acid covalently bound to the cell wall, while red and far-red fluorescence originate from chlorophyll a in the chloroplasts of green mesophyll cells. Fluorescence intensity is affected by (1) variations in the concentration of emitting substances, (2) the internal optical elements of the leaf that determine the penetration of excitation radiation and the partial reabsorption of emitted fluorescence, and (3) the energy distribution between photosynthesis, heat production, and chlorophyll fluorescence emission, thus providing valuable information about the plant’s health status. Active imaging using, for example, UV-excited light sources can be applied to close-range screening or remote sensing in the context of precision agriculture and related fields.

[0127] The operations described in the flowchart are based on the systems / apparatus shown in the block diagram. However, it should be understood that the operations of the flowchart can be performed by embodiments of systems and apparatus other than those discussed in the block diagram, and the embodiments discussed in the reference systems / apparatus can perform operations different from those discussed in the reference flowchart.

[0128] Given the various arrangements of the embodiments described herein, this specific implementation is intended to be illustrative only and should not be considered as limiting the scope of the invention. Therefore, all modifications that may fall within the scope of the appended claims and their equivalents are claimed. Consequently, this description and drawings should be considered illustrative rather than restrictive. Attached Figure Description

[0129] In the following, exemplary forms of the invention are explained in more detail with reference to the accompanying drawings, which illustrate these exemplary forms. They show:

[0130] Figure 1A This is a block diagram of a system for generating ML models that have learned to label RGB images;

[0131] Figure 1B These are automatically labeled hyperspectral training images;

[0132] Figure 2It is a block diagram of a computer system used to predict the labels of RGB test images using a trained ML model;

[0133] Figure 3 It consists of RGB test images automatically labeled by a trained model and hyperspectral images automatically labeled based on their spectral features;

[0134] Figure 4 This is a flowchart of a method for providing a trained ML model suitable for automatically labeling images acquired using a first image acquisition technique;

[0135] Figure 5 This is a flowchart of a method for automatically labeling images acquired using a first image acquisition technique using a trained ML model; and

[0136] Figure 6 It is a curve with two spectral characteristics. Detailed Implementation

[0137] Figure 1A A block diagram of a system 100 for generating an ML model 132, which has learned to label RGB images, is shown. The model to be generated should be able to automatically identify and label beet plants infected with a specific disease or parasite (e.g., Cercospora) based on RGB images that can be easily acquired using a standard camera. Cercospora is a fungus of the genus Ascomycota. Most species in this genus cause plant diseases and form leaf spots.

[0138] The system includes a computer system 120 (e.g., a standard desktop computer system); one or more RGB cameras 104 for acquiring digital RGB images of a test field of beet plants infected with Cercospora oryzae; and one or more hyperspectral cameras 102 for acquiring digital hyperspectral images of the test field. The use of RGB cameras 104 is referred to as a “first image acquisition technique”, while the use of hyperspectral cameras 102 is referred to as a “second image acquisition technique”.

[0139] The hyperspectral camera 102 and the RGB camera 104 are placed very close to each other in space, so that both depict the test field from substantially the same distance and angle. Alternatively, the two camera types 102 and 104 are placed in the same location and used sequentially to acquire images.

[0140] In one embodiment, the HySpex Mjolnir is used as the hyperspectral camera 102, while the Sony Alpha 7rlI is used as the RGB camera 104. Unmanned aerial vehicles (UAVs), such as drones, are equipped with two cameras 102 and 104.

[0141] The hyperspectral image 106 acquired by the hyperspectral camera 102 and the RGB image 205 acquired using the RGB camera are respectively referenced to geographic coordinates based on a high-precision GNSS-assisted IMU (where GNSS refers to Global Navigation Satellite System and IMU refers to Inertial Measurement Unit). An IMU is an electronic device that uses a combination of accelerometers, gyroscopes, and sometimes magnetometers to measure and report specific forces, angular velocities, and sometimes the orientation of an object. Using a GPS device with IMU functionality allows the GPS receiver to operate when GPS signals are unavailable, such as in the presence of electronic interference.

[0142] The acquired images 106 and 205 are transmitted to computer system 120 and stored in storage medium 121. Storage medium 121 is preferably a non-volatile storage medium, such as an electromagnetic or optical storage medium, like a hard disk drive, DVD, etc. Transmission can be performed via a mobile telecommunications connection while the UAV is flying over the area. Alternatively, transmission can be performed after the UAV has landed, for example, by manually transferring the image to computer system 120 via the UAV's SD card, USB storage device, or other type of portable storage device. During the training of the ML model, hyperspectral image 106 is used as a second training image, while the transmitted RGB image 205 is used as a first training image.

[0143] Computer system 120 includes one or more processors 112 configured to instantiate and run one or more software programs or modules 114, 118, 122, 126 involved in generating trained model 132.

[0144] For example, feature extraction module 114 is configured to extract image features, referred to herein as second feature 116, from each pixel of each second training image 106. The second feature preferably consists of or includes spectral features. For example, a spectral feature may be a curve indicating the light intensity observed at a wavelength continuum of the spectrum covered by a hyperspectral sensor.

[0145] The label prediction module 116 is configured to receive the extracted second features as input and calculate one or more labels for each second training image 106. For example, the label prediction module 118 may include a repository containing multiple reference spectral features. Each reference spectral feature describes the spectral characteristic properties of a specific type of object. For example, the repository may include hyperspectral reference feature properties of plain soil, hyperspectral reference feature properties of healthy beet plants, hyperspectral reference feature properties of beet plants infected with Cercospora, hyperspectral reference feature properties of a 50:50 mixture of healthy and Cercospora-infected beet plants, and so on. By comparing the spectral reference features stored in the repository of module 118 with the expected spectral features in each second training image, module 118 can identify one of the reference spectral features most similar to the spectral features of the corresponding pixel. The category name of this "most similar reference spectral feature" of the pixel in the second training image is assigned to that pixel. Alternatively, a numerical value indicating the probability that a pixel in the second training image depicts the object type represented by the "most similar reference spectral feature" is assigned as a label to the pixel in the second training image.

[0146] The label prediction module 118 outputs a labeled second training image 110 for each second training image 106.

[0147] Alignment module 122 is configured to spatially align a first training image and a second training image depicting the same or substantially the same motivation. For example, alignment may be performed based on GPS coordinates assigned to the images by cameras 102, 104, or it may be performed based on known camera parameters (e.g., known, fixed camera positioning relative to the motivation and / or relative to other types of cameras). Alternatively, a greenness index may be calculated and used as the basis for aligning the first and second training images. Label prediction module 118 may calculate and assign at least one label to each second training image (or a sub-region thereof, such as a pixel block or a single pixel) before or after the alignment module performs image alignment.

[0148] According to an embodiment in which a label is assigned to a single pixel or pixel block, the alignment module will essentially also spatially align the labels assigned to, or to be assigned to, the pixels or pixel regions of the second training image with the corresponding pixels or pixel regions of the first training image.

[0149] Aligned labels 124 (i.e., the content of the labels and indications of one or more pixels in the first training image to which the labels are aligned) are input together with the first training image 205 to which the labels have been aligned into software 126 configured to train a machine learning model. For example, software 126 may include module 128 comprising multiple algorithms for extracting features 130 from each first training image. Furthermore, software 126 may include additional algorithms and modules required during training. For example, software 126 may include a loss function configured to compare the labels predicted by ML module 132 based on the first features 130 extracted during training with the labels 124 provided as training data during training, and adjust model 132 such that the deviation of the predicted first labels from the provided "true" labels 124 is minimized. For example, software DeepLabv3 can be used as training software. DeepLabv3 is a state-of-the-art deep learning software that specifies a deep learning model for semantic image segmentation, the goal of which is to assign semantic labels that indicate, for example, the class membership of each pixel in the input image. DeepLabv3 includes several image feature extraction algorithms, as well as other modules suitable for training ML models based on extracted first features and additional training data provided by the user.

[0150] Preferably, the training data used to train the ML model 132 includes hundreds or preferably thousands of first training images and a corresponding number of second training images, which are aligned with each other to form hundreds or preferably thousands of pairs of aligned training images.

[0151] Figure 1B The automatically labeled hyperspectral training image generated by the system according to an embodiment of the invention is described in more detail. Initially, the label prediction module 118 assigns labels to each individual pixel of the second training image 106. In the depicted example, only three different labels are used: label 150 indicating soil, label 152 indicating healthy beet plants, and label 154 indicating beet plants infected with Cercospora. After the alignment module 122 has performed image alignment, the labeled second training image, or a combination of labels and label location information alone, can be provided as input to the machine learning training software 126.

[0152] Figure 1BA graphical representation of the labeled second training image is depicted, wherein different sub-regions of the image have been identified by applying a segmentation algorithm that groups pixels with the same or similar labels into the same segments. For example, the segmentation algorithm has identified image regions 150 depicting soil, large image regions 152 depicting healthy beet plants, and multiple image patches 154 depicting beet plants infected with Cercospora spp. Applying image segmentation algorithms and using different colors or shading to represent different segments can be advantageous because such graphical representation simplifies human interpretation of label predictions. For example, the labeled and segmented image 110 can be output to a user via screen or printout.

[0153] Figure 2 This is a block diagram of a computer system 120 for predicting labels 150, 152, and 154 of an RGB test image 108 using a trained ML model 132.

[0154] Computer system 120 can be the same computer system used to perform the training. Alternatively, the computer system can be any other computer system, such as a cloud computer system or desktop computer system on which the trained ML model has been transferred.

[0155] A computer system for applying a trained ML model during testing includes a storage medium 121 and one or more processors 112, which are associated with a previously referenced... Figure 1A The described storage media and processes are identical or functionally equivalent. Computer system 120 includes prediction software 202 having a feature extraction module 128 and a trained ML model 132. The feature extraction module is configured to extract the same type of image features extracted as "first features" during the training of the ML model. Feature extraction module 128 may be an integral part of prediction software 202, or it may be a separate software application or module configured to preprocess any received test image 108 to extract first features 204 and provide the first features as input to prediction software 202.

[0156] The storage medium includes one or more test images 108, each acquired using a first image acquisition technique. (See Figure 1 and...) Figure 2 In the example depicted, the first image acquisition technique is an RGB image acquisition technique. Test images can be received from different computers via a network, read from local or remote storage media (e.g., USB storage devices), and / or received directly from the RGB camera 104. The RGB camera 104 can be... Figure 1A The RGB cameras depicted are different from RGB cameras, and the same reference numerals only indicate functional equivalence.

[0157] Each RGB test image 108 is provided as input to the prediction software 202. The feature extraction module 128 extracts multiple first features 204 from each RGB test image. For example, the first image features 204 may include intensity gradients, textures and other patterns, intensity values, color values, color gradients, contrast values, etc. The extracted first features 204 are provided as input to a trained ML model 132. During training, the model learns the spatial correlation between the markers and the first image features extracted from the RGB training images. Therefore, based on the first features 204 provided by the feature extraction module 128, the trained ML model 132 is able to predict at least one marker and its corresponding location for each RGB test image 108. For example, in some embodiments, only a single marker is predicted for each image. Preferably, a marker is predicted for each pixel in the RGB test image.

[0158] The labeled test image 206 depicts image segments obtained by segmenting the test image 206 based on pixel-by-pixel markers predicted by software 202. The labeled and segmented test image 206 includes several sub-regions indicated by white, which are assigned markers 250 indicating soil. Image 206 also includes a large area indicated by a first shading, which is assigned marker 252 indicating healthy beet plants; and multiple small image patches indicated by a second dark shading, which are assigned marker 254 indicating beet plants infected with Cercospora.

[0159] Figure 3 The RGB test image 206, which is automatically labeled by a trained model, is depicted in more detail, where different image segments with different labels are represented by different colors rather than different shades.

[0160] To demonstrate the accuracy of the proposed label prediction method, Figure 3The lower half of the description illustrates the markers obtained for the same test field using a hyperspectral camera and marker prediction software that uses hyperspectral features to predict markers. Hyperspectral camera 102 is used to acquire a hyperspectral image 302 depicting the same agricultural area depicted in test image 108. A comparison of RGB test image 108 and hyperspectral test image 302 reveals that the two images depict the same agricultural area. Of course, spectral information beyond the visible spectrum range contained in hyperspectral image 302 cannot be described here. A second feature 116 is extracted in the form of spectral features by feature extraction module 1144, and a pixel-specific marker can be calculated by marker prediction module 118, as previously described, by comparing the extracted spectral features of each pixel with corresponding reference spectral features. A marked and segmented hyperspectral image 304 is generated by performing an image segmentation step based on the markers. A comparison of the two marked images 206 and 304 reveals that the trained ML model is able to predict the type and location of markers with substantially the same accuracy as the marker prediction module 118 using hyperspectral data as input. Therefore, although the RGB camera used to acquire the first test image only covers a small spectral range, the trained ML program is able to accurately predict the location and type of the markers. Thus, the method for automatically labeling test images acquired using the first image acquisition technique combines the advantages of RGB and hyperspectral imaging techniques: hyperspectral images are highly flexible and can automatically identify virtually any type of object based on information-rich spectral features. RGB images can be acquired using standard, inexpensive RGB cameras. By performing automatic labeling using hyperspectral images only during training but RGB images during testing, the cost and effort associated with using a hyperspectral camera occur only during the training phase, not the testing phase.

[0161] Figure 4 This is a flowchart of a method for providing a trained ML model suitable for automatically labeling images acquired using a first image acquisition technique (e.g., RGB imaging).

[0162] For example, this method can be derived from Figure 1A The system described in the text is used to execute it.

[0163] First, in step 402, the method includes acquiring multiple second training images 102 using a second image acquisition technique (e.g., a hyperspectral camera 102). Furthermore, the second image acquisition technique can be used to acquire reference spectral features of objects from one or more different categories of interest. For example, hyperspectral reference features of soil, healthy beet plants, and beet plants infected with Cercospora can be obtained.

[0164] Next, in step 404, at least one label is calculated for each second training image 106. For example, the feature extraction module 114 extracts spectral features for each pixel in each second training image and uses them as second features 116. The extracted spectral features are compared with spectral reference features to determine one of the reference spectral features most similar to the spectral features of the currently examined pixel. The object category represented by the identified most similar reference feature is assigned to each second training image in the form of at least one label. For example, one label may be assigned to each pixel of each second training image.

[0165] According to one embodiment, each hyperspectral second training image is compared with the aforementioned hyperspectral reference features to calculate a per-pixel similarity score for the spectra of soil, healthy plants, and plants infected with Cercospora globulus using a spectral angle mapper algorithm. According to the embodiment, the aforementioned three categories of spectral reference features are obtained empirically. Furthermore, by combining the aforementioned “pure” reference spectra, an additional 13 categories are calculated, representing a mixture of soil, healthy plants, and plants infected with Cercospora globulus, with a step size of 25%.

[0166] The Spectral Angle Mapper (SAM) algorithm is suitable for measuring the spectral similarity between two spectra. Spectral similarity can be obtained by treating each spectrum as a vector in a q-dimensional space, where q is the number of bands, and comparing the two vectors. According to one embodiment of the invention, the obtained similarity score (representing the similarity to a reference spectrum such as "soil" or "healthy beet plant") is used as a label to obtain a low-resolution score image, whereby the score represents a label assigned per pixel.

[0167] Embodiments of the present invention are particularly useful in the context of precision farming, quality control in breeding companies, and related technical fields. Almost every category of plant-related objects (a specific group or species of plant, a weed-infested field, a plant infected with a specific disease, a nutrient-deficient plant, etc.) can be characterized by specific physiological states or changes in state that affect the object's reflectance properties. Healthy crops and disease-affected crops reflect sunlight differently. Hyperspectral imaging can detect minute changes in plant physiology and correlate them with the reflectance spectrum, thereby automatically labeling a large number of hyperspectral training images.

[0168] Furthermore, in step 406, one or more first training images 205 are acquired using a first image acquisition technique, such as in RGB camera 104. Steps 402 and 406 may be performed simultaneously or subsequently. In any case, steps 402 and 406 must be performed such that the first and second training images substantially depict the same motivation and are therefore spatially aligned with each other. Substantially depicting the same motivation as used herein means that the relative positions (distance and angle) between the camera used to acquire the images and the motivation, as well as the preferred environmental conditions (light intensity, position of the light source, temperature, spectral composition of the light emitted by the light source), are the same or substantially the same.

[0169] Next, in step 408, each first training image is spatially aligned with one of the second training images depicting the same motivation.

[0170] According to an embodiment, image alignment is performed as follows: the RGB representation of the hyperspectral image used in the second training image 106 is calculated by averaging the corresponding spectral bands. A corresponding greenness index is calculated from the "real" RGB image 205 obtained as one of the first training images and the "calculated" RGB image obtained as a derivative of one of the second training images 106. The greenness indices are compared to each other to calculate and estimate the displacement field. For example, this estimate can be calculated using the MATLAB function "imregdemons".

[0171] The advantage of using a greenness index (or any other motif-derived feature that minimizes the difference between two aligned images during alignment) is that plants or plant parts located in slightly different positions in the first and second images can also be correctly aligned. For example, factors such as wind, a time delay of several hours between the first and second images that capture the same motif, using different drones to capture the first and second images, and / or using drones with slightly different trajectories can cause a shift in the position of the motif depicted in the first and second images.

[0172] According to an embodiment, a first image acquisition system for acquiring a first training image and a second image acquisition system for acquiring a second training image are mounted on the same carrier system (e.g., a drone). This ensures that the motivation depicted within the first and second training image pairs, which depict the same motivation, has only a small spatial offset of a few pixels.

[0173] According to other embodiments, the first training image is obtained by a first sensor mounted on a first carrier system, and the second training image is obtained by a second sensor mounted on a second carrier system. The second carrier system is different from the first carrier system, or it is the same first carrier system used multiple times to acquire the first and second training images in multiple different flights. For example, the first and second carrier systems can be different UAVs, or they can be the same UAV used to acquire the first and second training images in multiple different flights.

[0174] For example, the first training image may be acquired in one or more flights of the first carrier system, and the second training image may be acquired in one or more flights of the second carrier system. The flights of the first and second carrier systems are conducted at different times; specifically, the time interval between flights is at least 5 minutes, or even several hours. During this time interval, the position of the plant may change slightly, for example, due to wind, or due to the movement or reorientation of the plant or plant parts toward the light.

[0175] According to some embodiments, which are particularly useful for acquiring first and second images in multiple subsequent flights of the same or different carrier systems, the first and second images are geographic coordinate reference images, i.e., images with specified location information, specifically, coordinates in a geographic reference coordinate system. For example, the carrier system used to carry the first and / or second sensors may include IMU sensors, specifically, GNSS-assisted IMU sensors.

[0176] An inertial measurement unit (IMU) is a sensor device that includes, for example, motion sensors (accelerometers) and / or rotation sensors (gyroscopes) to continuously calculate the position, orientation, and velocity (direction and speed of motion) of a moving object without the need for external references. Inertial sensors are typically supplemented by barometric altimeters, and occasionally by magnetic sensors (magnetometers) and / or velocity measurement devices.

[0177] Specifically, the IMU sensor can be a GNSS-assisted IMU. The term "GNSS" (Global Navigation Satellite System) is a globally covering navigation system that uses satellites to provide autonomous geospatial positioning. It allows small electronic receivers to determine their position (longitude, latitude, and altitude / altitude) with high accuracy (within centimeters to meters) using time signals transmitted along the line of sight from satellites. This system can be used to provide location, navigation, or track the location of objects equipped with receivers (satellite tracking). As of September 2020, the US Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and the European Union's Galileo system were all fully operational. Japan's Quasi-Zenith Satellite System (QZSS) is a (US) GPS-based augmentation system designed to improve GPS accuracy, with GPS-independent satellite navigation planned for launch in 2023. India's Regional Navigation Satellite System (IRNSS) plans a long-term expansion to a global version. The geographic location of the carrier system at the time of acquiring the first or second training image is stored in association with the corresponding training image for later use during image alignment.

[0178] Using a GNSS-assisted IMU sensor to identify the location of the carrier system during the acquisition of the first and second training images allows the first and second image acquisition sensors to be placed on different carrier systems and / or the first and second training images to be acquired subsequently.

[0179] According to an embodiment, the alignment of the first and second images in each pair includes aligning the first and second images depicting the same motivation based on their respective geographic locations, thereby providing a roughly aligned image pair, and then refining the alignment based on pixel intensity and / or color similarity (e.g., based on the greenness index) to provide the alignment of the first and second images in the pair.

[0180] Next, at step 410, the labeled second training images (or just their labels) aligned with the RGB images are input into the machine learning program used to train the model. For example, a semantic segmentation deep neural network, DeepLabv3+, can be used, which already includes various feature extraction algorithms. Therefore, the aligned first image can be input into DeepLabv3+ before the first feature extraction process begins. DeepLabv3+ then automatically performs the extraction of the first features and the training of the ML model.

[0181] According to another embodiment, a first training image is processed to extract first image features, and the extracted first features and the first training image are provided to machine learning software.

[0182] Regardless of whether the machine learning program itself or the preprocessing module performs the extraction of the first image features, the spatial alignment of the labels and the first image features enables the machine learning model (e.g., a semantic segmentation deep neural network) to learn the spatial correlation between the labels and the first feature during training.

[0183] As a result of the training, a trained ML model is provided, which learns the correlation between first features extracted from RGB training images and labels spatially aligned with the first training image and its first image features. The trained ML model is able to predict image labels for any input image that has been acquired using the first image acquisition technique and depicts a plant-related motivation similar to that depicted in the training image.

[0184] At step 412, a trained ML model is provided. For example, training software 126 or a portion thereof may be transferred to another computer via a network connection or via a portable storage medium, and used on that other computer to automatically label RGB test images. The assigned labels indicate the type of object depicted by the labeled test image pixels. These categories are the same as those used to label the second training images during the training phase. Figure 3 As shown, the trained ML program accurately produces classification results that visually resemble ground reality. In other words, according to the embodiment, the trained ML program can be used as a classifier suitable for accurately predicting labels for high-resolution RGB images, although labels for hyperspectral images were used during training. The generation of training data 110 and 205 is performed fully or semi-automatically, without relying on manual annotation.

[0185] Figure 5 This is a flowchart of a method for automatically labeling images acquired using a first image acquisition technique using a trained ML model 132.

[0186] In the first step 502, a trained ML model 132 is provided. For example, the model and optional additional software modules, such as feature extraction module 128, are stored on a computer system 120, which includes or is configured to receive one or more test images 108.

[0187] Next, in step 503, a first image acquisition technique, specifically an RGB image acquisition technique, is used to acquire one or more test images. Each test image depicts a plant-related motif, such as an agricultural area, plant, plant product, or a portion thereof. The plant-related motifs in the test images are similar to those in the first and second training images used to train the ML model. Step 503 can be performed prior to step 502.

[0188] Next, in step 504, the trained ML model is applied to each test image. Therefore, a first feature is extracted from each test image. For example, prediction software 202 may be substantially the same as training software 126 used to train the model and may include a feature extraction module 128 comprising multiple algorithms for extracting different image features from the RGB image. The image features 204 extracted from the test images are used by the trained ML model to predict one or more labels and assign labels to the test images.

[0189] Next, in step 506, the predicted markers are output. For example, the predicted markers can be used in the segmentation step to calculate the segmented image displayed to the user via screen or printout.

[0190] Figure 6 A diagram is shown that includes a first spectral reference feature 602 as a soil property and a second spectral reference feature 604 as a water property. By comparing the spectral reference features 602, 604 with the spectral features of each pixel in the second training image, a marker indicating the type of object depicted in the image or image region, or indicating the likelihood of depicting such an object, can be calculated.

[0191] List of reference numerals

[0192] 100 System

[0193] 102 Hyperspectral Camera

[0194] 104 RGB Camera

[0195] 106 Second training images obtained using a second image acquisition technique

[0196] 108 First training images obtained using the first image acquisition technique

[0197] 110 Labeled second training images

[0198] 112 processor

[0199] 114 Feature Extraction Module

[0200] 116 Extracted second feature

[0201] 118 Label Prediction Module

[0202] 120 Computer System

[0203] 121 Storage Media

[0204] 122 Alignment Module

[0205] 124-aligned markers

[0206] 126 Machine Learning Training Software

[0207] 128 Feature Extraction Module

[0208] 130 The first feature extracted

[0209] 132 Machine Learning Models

[0210] 150 shades used as markers: soil

[0211] 152. Shade used as a marker: healthy plants.

[0212] 154 Shading used as a marker: Plants infected with Cercospora (CR)

[0213] 200 system

[0214] 202 Prediction Software

[0215] 204 First features extracted from the test image

[0216] The test image acquired by 205 is the first image acquisition technology.

[0217] 206 A labeled image generated by prediction software 202 from test image 205

[0218] 250 shades used as a marker: soil

[0219] 252 Shades used as markers: healthy plants

[0220] 254 Shading used as a marker: Plants infected with Cercospora.

[0221] 302 Hyperspectral Image

[0222] 304 Labeling based on hyperspectral image prediction

[0223] Steps 402-412

[0224] Steps 502-506

[0225] 602 Spectral Characteristics

[0226] 604 Spectral Characteristics

Claims

1. A computer-implemented method, comprising: First training images (108) are acquired (406) using a first image acquisition technique (104), each first training image depicting a plant-related motivation, wherein the plant-related motivation is selected from the group consisting of: indoor or outdoor agricultural areas, or plants; A second training image (106) is acquired (402) using a second image acquisition technique (102), wherein the second image acquisition technique is a hyperspectral image acquisition using a hyperspectral sensor, each second training image depicting the motivation depicted in a corresponding first training image in the first training image, wherein the second image acquisition technique covers a larger portion of the entire electromagnetic spectrum than the first image acquisition technique, and / or wherein the first image acquisition technique has a higher spatial resolution than the second image acquisition technique; At least one label (150, 152, 154) is automatically assigned (404) to each of the acquired second training images, wherein the automatic assignment of the label to a pixel or region of each of the second training images comprises: For each of the predefined sets of motivation categories, obtain spectral reference features from the physical reference motivation belonging to that motivation category; The spectral reference feature is compared with the second training image to identify the spectral similarity between pixels or regions of the second training image and the spectral reference feature; and A label is assigned to each pixel or region of each of the second training images, the label indicating the motivation category that is most spectrally similar to that pixel or region; The first and second training images depicting the same motivation are spatially aligned (408) into an aligned training image pair; A machine learning model (132) is trained (410) based on the aligned training image pairs and the labels, wherein, during training, the machine learning model (132) learns to automatically assign one or more labels (250, 252, 254) to any test image (205) acquired using the first image acquisition technique depicting plant-related motivations; and Provide (412) a trained machine learning model (132).

2. The computer-implemented method according to claim 1, wherein, The plant-related motivations are macroscopic motivations with a size of at least 1 mm.

3. The computer-implemented method according to claim 1 or 2 further includes: Extract a first feature from each of the first training images (130); The training is performed such that the machine learning model learns the spatial correlation between the first feature and the label based on the spatial correlation between the first feature and the label in each aligned first training image and second training image pair.

4. The computer-implemented method according to claim 1 or 2, further comprising: Extract (114) second features (116) from each of the second training images (106); The automatic assignment of the at least one label to each of the acquired second training images includes: analyzing the second features extracted from the second training images to predict (118) at least one label of the second training images based on the second features extracted from the second training images.

5. The computer-implemented method according to claim 1 or 2, wherein, The first image acquisition technique is an image acquisition technique selected from the group consisting of: RGB image acquisition (104); Monochrome image acquisition; Active image acquisition using excitation light source; Multispectral image acquisition; and IR image acquisition.

6. The computer-implemented method according to claim 1 or 2, wherein, Spatial alignment of the first and second training images in each pair includes: Align the first and second images depicting the same motivation based on their respective geographical locations, thereby providing roughly aligned image pairs, and then... The alignment is refined based on pixel intensity or color similarity to minimize the intensity or color difference between the first and second images, thereby providing alignment between the first and second images of the pair.

7. The computer-implemented method according to claim 1 or 2, wherein, The first training image is an RGB image, and the second training image is a hyperspectral image, wherein the spatial alignment of the first and second training images in each pair includes: For each pixel of the second training image, the intensity values ​​of red, green and blue are calculated by averaging the spectral intensity values ​​of the visible red, green and blue spectral bands covered by the second training image. The second training image is generated as an RGB representation based on the calculated intensity values ​​of red, green, and blue. Calculate a first greenness image, wherein the intensity of each pixel in the first greenness image is a greenness index calculated based on the intensity values ​​of red, green and blue in the first training image; Calculate a second greenness image, wherein the intensity of each pixel in the second greenness image is a greenness index calculated based on the intensity values ​​of red, green and blue in the RGB representation of the second training image; The first training image and the second training image are automatically aligned to minimize the difference in greenness index between the first image and the second image.

8. The computer-implemented method according to claim 1 or 2, wherein, The trained machine learning models (132, 202) are configured as follows: The one or more tags (250, 252, 254) are assigned to the test image (205) such that at least one of the tags is assigned to each pixel of the test image; as well as The test image (205) is semantically segmented by grouping pixels of test images that share the same tag or share sufficiently similar tags into the same segment.

9. The computer-implemented method according to claim 1 or 2, wherein, The motivations depicted in the plant-related training and testing images are for indoor and / or outdoor agricultural areas, and the markers are selected from a group of predefined motivation categories that include: Areas covered with healthy plants; Areas covered with plants infected with specific diseases and / or parasites; An area covered with plants of a specific species; An area covered with a specific variety of plants; Areas covered with specific substances, specifically fungicides, insecticides, herbicides, and / or fertilizers used to treat plants; Areas covered with plants treated according to a specific irrigation program; Areas not covered by any vegetation; Areas covered with a specific type of soil; A region covered with a mixture of two or more predefined portions of the above-mentioned types of coverings.

10. The computer-implemented method according to claim 1 or 2, wherein, The motivation depicted in the plant-related training and testing images is a plant, and the label is selected from a group of predefined motivation categories that include: The surface area of ​​the plant, according to which the surface area is healthy; A surface region of a plant, based on which symptoms are associated with infection with a specific disease in that region; A surface area of ​​a plant, which indicates that the area is infected by a specific parasite; A surface region of a plant, wherein the surface region displays cellular structures or organelles within a predetermined range; A surface region of a plant, wherein the surface region displays cellular structures or organelles in a predetermined state; A surface region of a plant, which shows morphological changes caused by the local application of a specific substance; A surface area covered by a mixture of two or more predefined portions of the above-mentioned types of surface areas.

11. The computer-implemented method according to claim 1 or 2, wherein, The plant-related motivation is an indoor or outdoor agricultural area having a variety of plants, plants, plant products, a portion of the plants, a portion of the plant products, where, according to the indoor or outdoor agricultural area, none of the plants or the plant products are modified, chemically treated, or dyed to provide labeling or promote labeling.

12. The computer-implemented method according to claim 1 or 2, in, The first training image is acquired by a first sensor mounted on a first carrier system, and the second training image is acquired by a second sensor mounted on a second carrier system. The second carrier system is different from the first carrier system.

13. The computer-implemented method according to claim 1 or 2, in, The first training image is acquired by a first sensor mounted on a first carrier system, and the second training image is acquired by a second sensor mounted on a second carrier system that is the same as or different from the first carrier system. The first training image is obtained in one or more flights of the first carrier system, and the second training image is obtained in one or more flights of the second carrier system. The flights of the first carrier system and the second carrier system are performed at different times, specifically, the time interval between flights is at least 5 minutes.

14. A computer-implemented method for automatically assigning one or more markers (250, 252, 254) to a test image (25) acquired using a first image acquisition technique (104), the test image depicting a plant-related motif, the method comprising: The trained machine learning model (132) is generated according to the method of any one of claims 1-13. Provide (502) a trained machine learning model adapted to automatically predict one or more labels (250, 252, 154) to be assigned to any input image acquired using the first image acquisition technique and depicting a plant-related motivation, wherein the plant-related motivation is selected from the group consisting of: indoor or outdoor agricultural areas, or plants. The trained machine learning model described in (504) is used to predict the one or more labels (250, 252, 254) of the test image; and Output (506) the predicted label of the test image (206).

15. An image analysis system (120), comprising: At least one processor (112); Storage medium (121) includes computer-interpretable instructions that, when executed by the at least one processor, cause the processor to perform the computer-implemented method according to any one of claims 1-14.

Images