Ontology based training of image classifiers

Abstract

The present subject matter relates to a method for training an image classifier using training data. The training comprises for a given image of the training dataset: using (301) the image classifier for providing a prediction of a primary object class of the given image. An estimate of the secondary object class of the one or more secondary object categories may be derived (303) from the prediction. One or more labels indicating a ground truth class of the one or more secondary object categories may be derived (305) from the label of the given image. The label of the given image may be compared (307) with the prediction. The one or more derived labels may be compared with the one or more estimates. The comparison result may be used for evaluating (309) a loss function; wherein the training is performed to optimize (3the loss function.

Description

[0001] DESCRIPTION

[0002] ONTOLOGY BASED TRAINING OF IMAGE CLASSIFIERS

[0003] Technical Field

[0004] Various example embodiments relate to image classification, and more particularly to an apparatus and method for training an image classifier.

[0005] Background

[0006] Herbicides research and development processes may require specific trials to monitor the effect of different herbicide formulations, quantities and protocols over the different plant species. These field trials may be conducted to evaluate their performance, which is measured by visual assessment. To develop a digital assessment tool, the identification of the weed species may be essential.

[0007] Summary

[0008] Example embodiments provide a method for training an image classifier for classifying images of objects, the objects having object categories which are organized in accordance with an ontology, the ontology being a biological ontology, the method comprising: determining a primary object category of the object categories, the primary object category having a set of primary object classes; determining one or more secondary object categories of the object categories, the one or more secondary object categories having specific relationships with the primary object category in accordance with the ontology, the secondary object category having one or more secondary object classes; providing training data comprising images and associated labels, the label of the image indicating a ground truth class of the image, the training data comprising multiple training datasets, each training dataset representing ground truth classes which are at least a subset of the set of primary object classes; training the image classifier using the training data, the training comprising for a given image of the training dataset: using the image classifier for providing a prediction of the primary object class of the given image; deriving from the prediction an estimate of the secondary object class of the one or more secondary object categories; deriving from the label of the given image one or more labels indicating a ground truth class of the one or more secondary object categories; comparing the label of the given image with the prediction and comparing the one or more derived labels with the one or more estimates, thereby evaluating a loss function; wherein the training is performed to optimize the loss function.

[0009] Example embodiments provide a computer program for training an image classifier for classifying images of objects, the objects having object categories which are organized in accordance with an ontology, the ontology being a biological ontology, the computer program comprising instructions which, when the program is executed by a computer system cause the computer system to perform at least the following: determining a primary object category of the object categories, the primary object category having a set of primary object classes; determining one or more secondary object categories of the object categories, the one or more secondary object categories having specific relationships with the primary object category in accordance with the ontology, the secondary object category having one or more secondary object classes; providing training data comprising images and associated labels, the label of the image indicating a ground truth class of the image, the training data comprising multiple training datasets, each training dataset representing ground truth classes which are at least a subset of the set of primary object classes; training the image classifier using the training data, the training comprising for a given image of the training dataset: using the image classifier for providing a prediction of the primary object class of the given image; deriving from the prediction an estimate of the secondary object class of the one or more secondary object categories; deriving from the label of the given image one or more labels indicating a ground truth class of the one or more secondary object categories; comparing the label of the given image with the prediction and comparing the one or more derived labels with the one or more estimates, thereby evaluating a loss function; wherein the training is performed to optimize the loss function.

[0010] Example embodiments provide a computer system for training an image classifier for classifying images of objects, the objects having object categories which are organized in accordance with an ontology, the ontology being a biological ontology, the computer system being configured for: determining a primary object category of the object categories, the primary object category having a set of primary object classes; determining one or more secondary object categories of the object categories, the one or more secondary object categories having specific relationships with the primary object category in accordance with the ontology, the secondary object category having one or more secondary object classes; providing training data comprising images and associated labels, the label of the image indicating a ground truth class of the image, the training data comprising multiple training datasets, each training dataset representing ground truth classes which are at least a subset of the set of primary object classes; training the image classifier using the training data, the training comprising for a given image of the training dataset: using the image classifier for providing a prediction of the primary object class of the given image; deriving from the prediction an estimate of the secondary object class of the one or more secondary object categories; deriving from the label of the given image one or more labels indicating a ground truth class of the one or more secondary object categories; comparing the label of the given image with the prediction and comparing the one or more derived labels with the one or more estimates, thereby evaluating a loss function; wherein the training is performed to optimize the loss function.

[0011] Example embodiments provide an image classifier for classifying images of objects the objects having object categories which are organized in accordance with an ontology, the ontology being a biological ontology, wherein the image classifier is trained according to a method for training an image classifier as described herein.

[0012] Brief Description of the Drawings

[0013] The accompanying figures are included to provide a further understanding of examples, and are incorporated in and constitute part of this specification. In the figures:

[0014] FIG.1 depicts a biological ontology in accordance with an example of the present subject matter; FIG. 2 is a flowchart of a method for providing training data in accordance with an example of the present subject matter;

[0015] FIG. 3 is a flowchart of a method for training an image classifier in accordance with an example of the present subject matter;

[0016] FIG. 4 is a flowchart of a method for defining a global loss function in accordance with an example of the present subject matter;

[0017] FIG. 5 is a flowchart of a method for classifying an image of an object in accordance with an example of the present subject matter;

[0018] FIG. 6 is a diagram representing a plant ontology in accordance with an example of the present subject matter;

[0019] FIG. 7A is a flowchart of a method for providing training data using a plant ontology in accordance with an example of the present subject matter;

[0020] FIG. 7B shows a table listing the content of training data in accordance with an example of the present subject matter;

[0021] FIG. 8 is a flowchart of a method for defining a global loss function in the context of a plant ontology in accordance with an example of the present subject matter;

[0022] FIG. 9 represents a general computerized system suited for implementing at least part of method steps as involved in the disclosure;

[0023] FIG. 10 is a diagram representing a plant ontology and, together with FIG. 11, shows the working principle of the trained image classifier in accordance with an example of the present subject matter;

[0024] FIG. 11 is a diagram representing a plant ontology, together with FIG. 10, shows the working principle of the trained image classifier in accordance with an example of the present subject matter.

[0025] Detailed Description

[0026] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, interfaces, techniques, etc., in order to provide a thorough understanding of the examples. However, it will be apparent to those skilled in the art that the disclosed subject matter may be practiced in other illustrative examples that depart from these specific details. In some instances, detailed descriptions of well-known devices and / or methods are omitted so as not to obscure the description with unnecessary detail. The present subject matter may enable to classify images of objects. The object may refer to a tangible, physical object capable of being rendered in an image. The object may be described in accordance with an ontology. The ontology may represent knowledge in which object categories (or concepts) are described both by their meaning and their relationship to each other. The ontology may, for example, be a biological ontology or bio-ontology. The biological ontology may, for example, be any one of: gene ontology, plant ontology, anatomical ontology, cell ontology, animal ontology or ontology of human anatomy. The biological ontology may, for example, represent knowledge about any one of: Organisms, Anatomy, Cell, Cellular anatomy, Gene Product, Gross Anatomy, Human, Human Developmental Anatomy, Molecule, Mouse Anatomy, Phenotype, Plant, Plant Anatomy, Plant Development, Protein, Subcellular, or Subcell ular anatomy. In case of a plant ontology, the image may include representations of at least one of: one or more plants or soil. This may enable a wide application of the present subject matter in different fields of biology.

[0027] The biological ontology may consist of a network of object categories. Each object category in the network may be defined as describing specific biological entities. The object categories may be related to each other based on rules that define the relationships between the object categories and between classes of the object categories. The object in the image may represent a class of each object category of the biological ontology. For example, one set of object categories in the plant ontology may comprise plant species, plant genus and plant Class. In this case, a plant object may, for example, have the palemeri class of plant species, Amaranthus class of plant genus, and Dicotyledoneae class of the plant Class. This set of object categories may further comprise one or more object categories that represent the plant damage types. Examples of damage types may be necrosis and leaf curling. However, symptoms of other damage types, such as bleaching, may be included, too. In another example, one set of object categories in the plant ontology may represent the plant damages e.g., a hierarchy of two or more levels may be used for the definition of plant damage types. For example, the imaged objects may have a number K of object categories which is higher than or equal to two, K > 2.

[0028] The present subject matter may use the biological ontology to form a semantic framework for creating training data and training models for image classification. The image classification of an image may comprise the assignment of a class to the image. The class may be predicted by an image classifier. The image classifier may be configured to receive the image and output the class of the image. The predicted class by the image classifier may be the class of a selected object category of the object categories of the biological ontology. The selected object category may be referred to as the primary object category. The term "primary” is used for naming purpose to distinguish the different object categories of the biological ontology which are used for training the image classifier. The primary object category may be used to provide training data to train the image classifier. The primary object category may have a set of classes, wherein the set of classes may be referred to as set of primary object classes. The set of primary object classes may be named PSET. For example, an object of the primary object category may be classified into one of the set of primary object classes. The selection or identification of the primary object category may be performed using existing training datasets e.g., by determining that the primary object classes have been used to label the images in the existing training datasets. This may enable to make use of existing training data to create further data. Alternatively, the primary object category may be selected from the object categories of the biological ontology using a selection criterion. The selection criterion may, for example, require the primary object category having specific types and number of relationships with secondary object categories or requiring the object category having the highest number of classes.

[0029] The training data may comprise images, wherein each image is associated with a label. The training data may, for example, comprise entries, wherein each entry comprises a pair of image and label. The label of the image indicates a ground truth class of the image. In case the image classifier is a whole image classifier, the label may comprise a class of the set of primary object classes PSET. In case the image classifier is a pixel-wise image classifier, the label of the image may comprise for each pixel of the image a class of the set of primary object classes PSET.

[0030] The training data may comprise homogeneous training datasets, meaning that the whole set of primary object classes PSET is used for labeling each individual training dataset. The homogeneous training data may provide homogeneous training samples, wherein the homogeneous training samples may have higher affinity among them enabling a fast convergence of the training process when applied on the training data. Alternatively, the training data may comprise heterogenous training datasets, meaning that each training dataset of the training data is labeled with at least a subset of the set of primary object classes PSET, e.g., a training dataset may be labeled with a subset of PSET, while another training dataset may be labeled with a different subset of PSET. In another example, the heterogeneity may allow to label a class of an object category as "unknown” if it is not known e.g., an imaged unknown Amaranthus may be assigned a specie class which is "unknown”. For example, the number of unknown classes assigned to the images of the training data may be smaller than a threshold. Alternatively, the fraction of unknown classes assigned to the images of the training data may be smaller than a threshold. The fraction of unknown classes may be the ration of the number of the unknown classes and the total number of classes used for classification of the images. For example, the threshold may be defined using the total number of classes used for classification of the images. For example, the threshold may be a percentage of the total number of classes used for classification of the images, wherein the fraction of the unknown classes is smaller than the percentage value e.g., the threshold may be 10% or 15%. The image or pixel which is labeled as an "unknown” species may be classified by the image classifier into a class which is different from "unknown” class. However, even if the specific species is not classified correctly, the classifier may still learn something from its parent classes using the global loss. The heterogeneous training data may provide a larger size of training data due to the availability of a higher number of training images. The heterogeneity may allow to build a robust learning system by leveraging the intrinsic knowledge among data. Providing this heterogenous content of the training datasets may enable to collect the training data at different point of times and from different sources. The training data may be used to training one or more image classifiers which in turn may enable accurate classification of the images.

[0031] The present subject matter may use the training data in order to efficiently train the image classifier. For that, one or more secondary object categories may be identified in the object categories. The secondary object category may be an object category of the object categories of the biological ontology. Each secondary object category may have a respective one or more classes, wherein the one or more classes may be referred to as secondary object classes. For example, an object of a specific secondary object category of the secondary object categories may belong to a class of the one or more secondary object classes of said specific the secondary object category. The term "secondary” is used for naming purpose to distinguish the object categories used for training the image classifier. The number of secondary object categories may, for example, be n2, where n2 is an integer higher than zero, n2>=1 . For example, the secondary object categories may be referred to as klt..., kn2respectively. The secondary object category k may have a set SSET 1 of one or more secondary object classes, secondary object category k2may have a set SSET2 of one or more secondary object classes, and so forth.

[0032] The image classifier may be trained using the training data. The training may be performed using at least part of the images (and associated labels) of the training data. In one example, the at least part of the images of the training data may be all images of the training data. Alternatively, the at least part of the images of the training data may be selected from the images of the training data. The selection may be randomly performed or may be performed such that the same number of images is obtained from each training dataset of the training data. The training may be performed for each current image of the at least part of the images of the training data as follows. The image classifier may be used for providing a prediction of the primary object class of the current image. For example, in case the image classifier is a whole image classifier, the prediction may comprise a class of the set of primary object classes PSET. The prediction may, for example, be provided as a vector of probability values, each probability value represents a probability of the whole image to be in a respective class of the classes PSET (the vector size may be equal to the number classes in PSET). In case the image classifier is a pixel-wise image classifier, the prediction may comprise for each pixel of the current image a class of the set of primary object classes PSET. The prediction may, for example, be provided as a matrix of probability values, each probability value represents a probability of a given pixel to be in a respective class of the classes PSET.

[0033] An estimate of the secondary object class of each secondary object category of the one or more secondary object categories may be derived from the prediction. For example, for each secondary object category, the prediction which is obtained for the primary object category may be used to derive an estimate of the corresponding class of said secondary object category. For example, for the secondary object category kltthe prediction may be used to derive an estimate of the secondary object class of the set of secondary object classes SSET 1 . In case the image classifier is a whole image classifier, the estimate may comprise a class of the set of secondary object classes SSET 1 . In case the image classifier is a pixel-wise image classifier, the estimate may comprise for each pixel of the current image a class of the set of secondary object classes SSET 1 . For the secondary object category k2, the prediction may be used to derive an estimate of the secondary object class of the set of secondary object classes SSET2. In case the image classifier is a whole image classifier, the estimate may comprise a class of the set of secondary object classes SSET2. In case the image classifier is a pixel-wise image classifier, the estimate may comprise for each pixel of the current image a class of the set of secondary object classes SSET2 and so forth.

[0034] Thus, the step of deriving the estimate from the prediction may result in a number n2 of estimates which is equal to the number of secondary object categories. One or more labels indicating a ground truth class of the one or more secondary object categories may be derived from the label of the current image. For example, for each secondary object category, the label of the current image may be used to derive the corresponding label of said secondary object category. For example, for the secondary object category kltthe label of the current image may be used to derive a label of the set of secondary object classes SSET 1 . In case the image classifier is a whole image classifier, the derived label of the current image may comprise a class of the set of secondary object classes SSET1. In case the image classifier is a pixel-wise image classifier, the derived label may comprise for each pixel of the current image a class of the set of secondary object classes SSET 1 . For the secondary object category k2, the label of the current image may be used to derive a label of the set of secondary object classes SSET2. In case the image classifier is a whole image classifier, the derived label of the current image may comprise a class of the set of secondary object classes SSET2. In case the image classifier is a pixel-wise image classifier, the derived label may comprise for each pixel of the current image a class of the set of secondary object classes SSET2 and so forth. Thus, the label derivation may result in a number of derived labels which is equal to the number of secondary object categories, e.g., n2 derived labels.

[0035] Hence, the current image may be associated with a set of predicted classes and corresponding set of labels. The set of predicted classes may be obtained using the image classifier. The set of predicted classes comprises the primary object class which is obtained by the image classifier, and the one or more secondary object classes which are derived from the primary object class using the relationships between the object category classes in the biological ontology. The set of predicted classes comprises n2 estimated secondary object classes and the one predicted primary object class (n2+1 predicted classes). The set of labels may comprise one label which is assigned to the image in the training data and the remaining labels of the set of labels may be derived from the one label using the relationships between the object category classes in the biological ontology. For example, the relationships between the secondary object classes and the primary object classes in the biological ontology may be used to identify the secondary object class that is related to the primary object class. The set of labels comprises n2 derived labels and the one label which is associated with the current image in the training data class (n2+1 labels).

[0036] The set of labels may be compared against the set of predicted classes respectively e.g., n2+1 comparisons may be performed between the n2+1 labels and the n2+1 predicted classes respectively. A loss function may be evaluated using the results of these comparisons. The loss function may provide a global loss based on all these comparisons. The loss function may thus be referred to as global loss function. The training may be performed for optimizing the global loss function. For example, the training may be performed for minimizing the global loss function.

[0037] The present subject matter may thus enable an efficient training of the image classifier by making use of the organization of the object categories in the biological ontology. The present subject matter may use the global loss function to cope with the dataset heterogeneity on largely growing and evolving datasets where the granularity of each class may vary or increase from season to season. For example, in case of a plant ontology, some of the training datasets may only be annotated with few species while other training datasets may be labelled with a higher number of species. The present image classifier may be configured to infer subtle common features of the super-class that could help describe the relationships between different classes and, at the same time, allow learning from any annotated sample, even if that specific species was not annotated in a given dataset.

[0038] The derivation of the secondary object classes or labels of the image may be implemented based on the representation of the biological ontology. In one example, the derivation of the secondary object classes from the primary object class may be performed by processing or analyzing the biological ontology and building one or more mappings that enable to map each primary object class to the corresponding secondary object classes in accordance with the biological ontology. The mappings may, for example, be provided for the secondary object categories respectively, so that each mapping may be used to map the primary object class to the corresponding secondary object class of the secondary object category of the mapping. The mapping may, for example, be a matrix, wherein each element of the matrix represents a distinct pair of primary object class and secondary object class. If the pair of primary object class and secondary object class are related according to the biological ontology, the corresponding element in the matrix may be set to one; otherwise, it may be set to zero. For example, the biological ontology may be a plant ontology, where the primary object category may be plant species and one of the secondary object categories may be the plant genus. The plant ontology may be represented by a data structure (e.g., graph). The data structure may indicate the relationships between the plant species and the plant genus. In case an image is classified as palmeri, using the data structure it may be automatically derived that the genus Amaranthus is the secondary object class which is associated with the palmeri.

[0039] The global loss function may be evaluated using results of the comparisons between the set of predicted classes and the set of labels respectively. Each compared pair of predicted class and label represents the same object category e.g., the predicted primary object class is compared against the label of the primary object classes. The set of comparisons may result in a set of results. For example, the comparison is failed if the predicted class does not match the label. The evaluation of the global loss function may use the whole set of comparison results. The global loss function may increase the penalty with increased number of failed comparisons. For example, the global loss function may provide a largest penalty in case the whole set of comparisons are failed. Thus, the global loss function may include contributions of all predicted classes of the image. This may be advantageous for the following reasons. Following the above example according to which the palmeri class is the predicted class and Amaranthus is derived / predicted class, in case the label (or ground truth class) of the image is blitolides, the first predicted class would be wrong; however, the derived genus class Amaranthus is a correct prediction because the label derived from the blitolides class is also Amaranthus. Thus, the second comparison is successful. If only the first comparison result is used to evaluate the loss function, the penalty may be the largest one; however, by using the two comparison results according to the present subject matter the penalty may be smaller. This value of the loss function indicates that although the image classifier failed to predict the primary object class but it is still on the "right direction”. This may enable a quick convergence of the training. In addition, the training may infer subtle common features of the superclass that could help describe the relationships between different classes and, at the same time, allow learning from any annotated sample, even if that specific species was not annotated in a given dataset. In one example, the global loss function may implement a given loss, wherein the given loss may be any one of: entropy loss, mean-squared error loss, likelihood loss, dice loss, generalized dice loss, focal loss, or Tversky loss.

[0040] The evaluation of the global loss function may be improved according to the present subject matter by identifying individual losses and then integrate them into the global loss. According to one example, the comparison of the label of the image with the prediction is performed to evaluate a first individual loss function. The comparison of the one or more derived labels with the one or more estimates is performed to evaluate respective second individual loss functions. For example, the n2+1 comparisons may be performed between the n2+1 labels and the n2+1 predicted classes of the image respectively in order to evaluate n2+1 individual loss functions. The global loss function may be evaluated for the image by evaluating the individual loss functions and combining the evaluated individual loss functions. For example, the global loss function may be a combination of the n2+1 individual loss functions.

[0041] According to one example, the global loss function may be a weighted sum of the individual loss functions. This may enable to control the contributions of the individual losses into the global loss. This in turn defines contributions of the different object category classes into the training optimization process. The weighted sum of the n2+1 individual loss functions may be performed by a set of n2+1 weights. The weights may be user defined weights. For example, a user input may be received, wherein the input indicates the weights. Alternatively, the weights may be determined by at least: ranking the one or more secondary object categories according to their relationships with the primary object category. The weights may be assigned to the one or more second object categories in accordance with the ranking. In addition, a weight may be assigned to the primary object category, wherein the weight of the primary object category is higher than the weights of the one or more second object categories. The weights may, for example, be positive real numbers which are smaller than or equal to one. This embodiment may in particular allow modifying the weight of each secondary category so that the importance of specific category is higher or lower. For example, it may be considered that the category Genus or Order is of lesser importance for the calculations and, thus, the corresponding weights may be decreased. It may be provided that all weights are set to the same value, e.g. set to one, except for classes that are unknown or defined as miscellanea at this taxonomy level, i.e. generic, objects (e.g. other_broad, other_amaranthus, other_dicotiledonea). In these cases of unknown or miscellanea information, weight values may be decreased or set to zero.

[0042] In one example, the individual loss functions may implement the same loss or different losses e.g., one individual loss function may implement the entropy loss while the other individual loss function may implement the likelihood loss.

[0043] According to one example, a global class balancing weight may be determined for each class of each object category of the primary and secondary object categories. In addition, a dataset class balancing weight may be determined per training dataset for each class of each object category of the primary and secondary object categories. Each individual loss function of the first and second individual loss functions is weighted with the respective global class balancing weight and / or the respective dataset class balancing weight. The global class balancing weights may correct for imbalance in different classes. The global class balancing weight may, for example, be determined using a frequency of the class in the training datasets. The application of the dataset class balancing weights may remove the influence of non-annotated classes in specific datasets of the training data. This may further improve the evaluation of the global loss function.

[0044] The global class balancing weights may, for example, be provided as vectors, wherein each vector (referred to Wcg) is associated with a respective object category of the primary and secondary object categories. Each element of the vector Wcg jmay represent a respective class of the / -th object category. The dataset class balancing weights may, for example, be provided as vectors, wherein each vector (referred to Wcdj) is associated with a respective object category of the primary and secondary object categories and a specific dataset of the training data. Each element of the vector Wcdgmay represent a respective class of the / -th object category.

[0045] According to one example, the image classifier is a pixel-wise classifier and the individual loss function is (further) weighted with a pixel weight. Optionally, the pixel weight may be applied in addition to other weights, in particular the weights in accordance with the ranking of the one or more second object categories and / or the global class balancing weight. It may be provided that the pixel weights of all pixels are set to the same value, e.g. set to one, except for pixels and / or classes of pixels that are unknown or defined as miscellanea objects (e.g. other_broad, other_amaranthus, other_dicotiledonea). Balancing the weight of the classes per pixel may allow to support classes that are known but only up to certain hierarchy level per pixel. Typically, when working at pixel level, each pixel has a certain level of annotation detail. Hence, during annotation of training data, the class of some pixels may be known up species level, but for other pixels the class may only be known up to a higher hierarchy level, such as up to the genus or order. There may be specific labels at each level to allow labelling certain elements of the hierarchy.

[0046] For example, a pixel may be labeled as "Amaranthus_other” if the specific species is not known but the genus is known (e.g. the genus being "amaranthus”). Typically, when applying a pixelwise annotation, in particular if there are multiple species present, in some cases it is almost impossible to annotate all species on the images accurately. A pixel weight for individual pixels may be used to mark some object categories (e.g. species) as unknown or miscellanea and giving this pixel a weight of zero for the individual pixel and / or the corresponding object category of the pixel. This may allow the system to ignore the outcome of the algorithm at the corresponding object category level (e.g. at species level) but taking it completely into account at another object category (e.g. at genus level). Without this embodiment, at least some training may be not usable or may only be usable with significantly lesser results. For example, if a species on an image is not known, the image may have to be discarded without the application of this embodiment, leading to fewer usable training images. In particular, a pixel-wise classification becomes a lot more feasible by applying the pixel weights.

[0047] The evaluation of the loss function may be performed in an optimal manner using available processing resources in case the image classifier is a pixel-wise image classifier. According to one example, the image classifier is a pixel-wise image classifier, wherein the evaluation of the first and second individual loss functions is performed using a matrix equation, wherein the element of the matrix represents a pixel of the image and a class. This example may enable to perform the evaluation of the global loss function in parallel for the pixels of the image. This may speed up the training process according to the present subject matter. The matrix notation may allow setting flexible number and type of secondary object categories that do not affect the architecture of the image classifier. The pixel-wise classification may enable "semantic segmentation", for semantically understanding the contribution of each pixel in the image.

[0048] According to one example, the training may comprise for the current image: determining whether the global loss function fulfills a convergence criterion. In response to determining that the global loss function does not fulfil the convergence criterion, learnable parameters or weights of the image classifier may be updated and the training step may be repeated for a further image of the training data. The update of the weights may, for example, be performed using gradient descent. In response to determining that the loss function fulfils the convergence criterion, this may indicate that the loss function is optimized and thus the training may be stopped and the trained image classifier may be provided.

[0049] In one example, the image classifier may be retrained in response to any one of: an expiry of a validity time period or the training data being updated. The training data may be updated by adding entries to and / or removing entries from the training data. For example, after expiry of the validity time period, the training data may be updated and the image classifier may be retrained using the updated training data. This may enable an up-to-date classifier which may improve the classification. In one example, the training data may be updated by augmenting the images of the training data. According to one example, the augmentation comprises for each image of at least part of the images of the training data, at least one of: performing rotation and / or height and / or width shifts of the image, performing a zoom of the image, performing vertical and / or horizontal flips of the image, performing a pixel-intensity shift of the image, or performing a Gaussian blur and artificial shadow simulation of the image.

[0050] The biological ontology may provide different types of relationships e.g., a pair of object categories of the biological ontology may be related with a given type of relationships. According to one example, the specific relationship, between a given second object category and the primary object category, may be a sequence of one or more relationships of the same type or of different types. In case the specific relationship is a sequence of more than one relationship, there may be intermediate one or more secondary object categories between the given secondary object category and the primary object category. The type of the relationship may, for example, include any one of: IS-A relationship, HAS-A relationship or PART-OF relationship.

[0051] According to one example, the biological ontology may be represented by a graph comprising nodes representing the object categories and edges representing relationships between the object categories. After identifying or selecting the primary object category, the secondary object categories may be determined such that a distance between the node representing each secondary object category and the node representing the primary object category is smaller than a maximum number of edges and / or the one or more edges between the node representing each secondary object category and the node representing the primary object category represent a specific relationship type. This example may enable an automatic control of the relationships between the primary and secondary object categories. According to one example, the weights used in the weighted sum of the individual loss function may be determined using the graph as follows. The weight assigned to the secondary object category is inversely proportional to the number of edges between the node representing the secondary object category and the node representing the primary object category. For example, the weight assigned to the secondary object category is equal to the inverse of the number of edges between the node representing the secondary object category and the node representing the primary object category.

[0052] According to one example, the biological ontology is a hierarchical ontology, wherein the primary object category belongs to a given level of the hierarchy, wherein the secondary object categories represent levels of the hierarchy which are higher than the given level. According to one example, the given level is the lowest level of the hierarchy, and the secondary object categories belong to consecutive levels of the hierarchy. According to one example, the primary object category is plant species, the secondary object categories comprise at least: a plant genus, plant class and vegetation. Including the object category "vegetation” may, for example, help the image classifier not to focus on learning differences between plants, e.g. plants that may be damaged, and soil as it learns this on the first hierarchy. In particular, including this object category can be advantageous when using a pixel-wise classifier, for example, partially-known pixels may be supported by taking advantage of information of whether a pixel is vegetation or something else, such as soil.

[0053] The present subject matter may improve the training process by improving access to the images of the training data. In one example, the images of the training data may be acquired using different imaging systems, wherein the imaging system comprises any one of: a camera, a mobile device or a drone. In case of a plant ontology, the image may include representations of at least one of: one or more plants or soil. For example, the image may be recorded by a camera mounted on a drone while the drone is flying over an agricultural field. In another example, the image may be taken with a camera of a mobile device (e.g., a smart phone) by a human user (e.g., a farmer). The image may, for example, be taken by a camera with a resolution that is high enough to visualize the damage symptoms on the imaged crop plants. In another example, the training data may be received from one source or from multiple sources.

[0054] According to one example, the current image in each iteration of the training may be fetched from the training datasets in accordance with a fetching rule, wherein the fetching rule requires any one of: the images are fetched from the different training datasets in an equal manner; or the images are randomly fetched from the difference training datasets. Fetching images in an equal manner from the different training datasets of the training data may be advantageous because it might prevent bias that may be introduced by using more images from specific dataset(s). Randomly fetching the images from the training datasets may be advantageous because it may be straightforward to implement and may save processing resources that would otherwise be required to implement a conditional fetching or a rule-based fetching of the images.

[0055] The images of the training data may have the same size in pixels. According to one example, the images may be pre- processed before the pre-processed images are used for performing the training. The image classifier may be configured to receive as input the resulting pre-processed image, and to process the pre-processed image for the training or inference of the image classifier. The pre-processing of the images may be performed by at least one of the following: resizing the images to a common size; correcting colour of the images; creating tiles from the image, or changing orientation of the images. The initial image size of the training image may be a large size e.g., between 3000x4000 pixels and 4000x6000 pixels. However, the image size which can be processed by the image classifier may be limited by the memory used for processing the image classifier. Therefore, the image pre-processing may adjust the image size (e.g., in pixels) of the training image to a rescaled image matching the size that can be used by the image classifier. Preferably, the resized image is reduced in size compared to the training image. The resolution of the rescaled image may still be high enough to perform classification.

[0056] According to one example, the image classifier is a deep neural network. The deep neural network may, for example, be a convolutional neural network (CNN) or a transformer.

[0057] The image classifier may, for example, be configured to perform regression and pixel wise regression.

[0058] According to one example, the images of the training data are field images which are acquired from agricultural fields after herbicide application. In this example, the biological ontology may be a plant ontology.

[0059] According to one example, the method further comprises: receiving an image of an object described by the biological ontology, and using the trained image classifier for segmenting the image, the image classifier being a pixel-wise image classifier. For example, the trained image classifier may be stored in one or more computer systems. The one or more computer systems may, for example, be provided as servers for client systems, wherein each client system may be configured to send the image to be segmented to the server in order to be segmented by the server using the trained image classifier, and the resulting segmented image may be sent by the server to the client. In one example, the inference of the trained image classifier may be provided as part of a mobile application. The mobile application may be installed on a mobile device. The mobile device may or may not comprise the trained image classifier. For example, upon launching the mobile application on the mobile device a user of the mobile device may upload an image to be segmented. The mobile device may be configured to connect to the server where the trained image classifier is stored and request a segmentation of the uploaded image. In response, the mobile device may receive from the server the segmented image.

[0060] According to one example, the biological ontology is the plant ontology. The method further comprises: receiving a field image of a field, and using the trained image classifier for segmenting the image, wherein the image classifier is a pixel-wise image classifier. The field image may, for example, be acquired from an agricultural field after herbicide application to the agricultural field. This example may refer to inference phase. The image used in the inference phase may have the same properties e.g., size in pixels, as the images used in the training of the image classifier. The training of the image classifier may be described using the following notation. The object category may be referred to as kj where the sub-script j is an integer that varies between zero and K - 1, where K is the total number of object categories in the biological ontology. The sub-script may be referred to as an index that represents the primary object category or one of the secondary object categories. For example, a sequence of indexes may, for example, be used to represent the primary object category and the secondary object categories respectively. In the following, the primary and secondary object categories may be indexed so that the first index j = 0, may represent the primary object category and the subsequent indexes (i.e., j >0) may be used to represent the secondary object categories respectively. However, as the person skilled in the art will appreciate, any other suitable indexing can be used as well. The set of predicted classes may be referred to as ykj, e.g., the predicted primary object class may be referred to as yko. The set of labels may be referred to as yk.. The individual loss functions may be referred to as Lind(yk. , yk]^. The global loss function may be referred to as Lglobal(y, y). The global loss function may be a weighted sum of the individual loss functions as follows: Lglobal(y, y) = !j=o W Lindwhere Wj are the weights assigned to the primary and secondary object categories respectively. The individual loss functions may be defined as follows:Lind x Wcd j, where Lj is a selected base loss for the / -th object category involved in the individual loss function, Wcggare global class balancing weights and Wcd jdataset class balancing weights. Alternatively, the individual loss functions may be defined as Lind(ykj, yk^ = Lyx Wcggx Wcd jx Wswhere Wsare the pixel weights in case the image classifier is a pixel-wise classifier.

[0061] In case the image classifier is a pixel-wise image classifier, the matrix notation may be used as follows. The predicted class ykmay be a s x nkmatrix, where nkis the number of classes in the object category kj, wherein for each pixel of the image, the matrix ykmay comprise prediction values or probability values for the pixel being in the nkclasses respectively. The label ykmay be a s x nkmatrix. The base loss Lj may be s x nkmatrix, where s is the number of pixels in the image and nkis the number of classes of the / -th object category. The global class balancing weights may be provided as a nkx 1 matrix and the dataset class balancing weights Wcdgmay be provided as a nkx 1 matrix per training dataset. The pixel weights may be provided as a s x 1 matrix. Thus, the individual loss function Lind(ykj,ykj) may be provided as a s x 1 matrix.

[0062] The global loss function may, for example, be defined using mappings as follows. where a mapping matrix Hkj,kj+1may be used to convert a label ykof the object category kj into a label yk]+1of a secondary object category kj+1. That is, the mapping matrix {k.k.+1may be used to derive the label yk]+1from the label matrix. The individual loss function Lindyk. , yk.j may, for example, be provided in matrix notation as follows: where operator “*” may refer to a Hadamard product and operator May refer to inner product, where the two indices between parenthesis (e.g., L (l,l)) represent a vector or matrix element.

[0063] The conversion of the label ykof the object category kj into a label yk of a secondary object category k] +1may, for example, be provided in matrix notation as follows:

[0064] According to one example, the trained image classifier is configured to provide a probability prediction of the primary object classes in the primary object category and select a primary object class to be the correct primary object class based on the probability prediction of the primary object classes and the relationship of the primary object classes with the secondary object categories. Selecting a primary object class may optionally be performed by deriving from the probability prediction of the primary object classes a probability prediction for the secondary object classes, in particular by adding the probabilities of the primary object classes related to the respective secondary object class, and, for each object category starting at the highest hierarchy level, selecting the object class with highest probability and propagate to the respective related next object category related to the selected object class until a primary object class is selected. The image classifier may optionally be configured to receive input, in particular user input, regarding the adaption of probabilities. Optionally, if a species is known not to be present, the probability for this species may be moved to a generic class or be set to zero. Optionally, if some specific class prediction is lower than a calculated threshold (e.g. calculated from the validation dataset to maximize a metric (such as accuracy, F1 Score, ...the probability for this species may be moved to the generic class (i.e. OTHER SPECIES) or be set to zero. The threshold may be different for different classes.

[0065] FIG. 1 illustrates a data structure representing a biological ontology in accordance with an example of the present subject matter.

[0066] The data structure may be implemented as a graph 100. The graph 100 comprises nodes kQthrough kK-±and edges T through In this example, the nodes k0through may represent the object categories of the biological ontology respectively and edges T through rK-1may represent relationships between the object categories. The relationship may, for example, be anyone of: IS-A relationship, HAS-A relationship, PART-OF relationship or any other type of relationships which are defined by rules of the biological ontology.

[0067] Each object category kj of the object categories may have a set of nk. classes, where nk. > 1, j = 0, ... , K - 1. This is indicated in FIG. 1 , where each of the nodes k0through kK- 1is associated with a list of classes of the associated object category. For example, the node kQis associated with a number nkgof classes, the node k is associated with a number nkiof classes, slC71...slC7nand so forth.

[0068] Each edge linking two nodes kj through / cj+1may be associated with an attribute which describes a mappingkj,kj+1of relationships between classes of the object categories of the nodes linked to the edge. This is indicated in FIG. 1 where, for example, the classes of the node k0are related to the classes of the node k3through the mapping Hfeo;fe3, and the classes of the node k0are related to the classes of the node k2through the mapping Hfeo;felHfel;fe2. For simplification of the drawings, not all mappings are shown.

[0069] In one alternative example implementation, each node of the nodes may represent an instance of the object category, meaning that the node represents a specific class of the object category. In this case, the edges linking the instances of object categories may represent the relationships between the object category classes. Each edge of the edges may be associated with an attribute indicating the type of relationship between the nodes connected to the edge.

[0070] FIG. 2 is a flowchart of a method for providing training data in accordance with an example of the present subject matter. The method of FIG. 2 may, for example, be implemented using a biological ontology as described with reference to FIG. 1.

[0071] A primary object category of the object categories may be determined in step 201 . The primary object category has a set of primary object classes. Secondary object categories may be determined in step 202. The secondary object categories have specific relationships with the primary object category in accordance with the biological ontology. Following the biological ontology of FIG. 1 , an example (named selected example) may be provided. According to the selected example, the primary object category may be represented by the node kQand the secondary object categories may, for example, be the two object categories represented by the nodes k2and k3respectively. In this case, the primary object category has the relationship r3with the secondary object category k3, and the primary object category has the relationships T and r2with the secondary object category k2, that is, the secondary object category k2is not linked directly with the primary object category k0.

[0072] Multiple training datasets may be provided in step 203, wherein each training dataset is labeled with ground truth classes which are at least a subset of the set of primary object classes. The training data comprising the training datasets may be provided e.g., for training image classifiers. Following the selected example, each training dataset of the multiple training datasets may be labeled by a subset of the set of classes or by the whole set of classes

[0073] FIG. 3 is a flowchart of a method for training an image classifier in accordance with an example of the present subject matter. The method of FIG. 3 may, for example, use the training data obtained by the method of FIG. 2.

[0074] Steps 301 to 311 may be repeated until the loss function has the optimal value, wherein in each repetition an entry of the training data may be used, wherein the entry comprises an image and associated label representing the class of the image.

[0075] The image classifier may be used to provide in step 301 a prediction of the primary object class of the current image. For example, the current image may be input to the image classifier for obtaining the prediction. An estimate of the secondary object class of the one or more secondary object categories may be derived in step 303 from the prediction. One or more labels indicating a ground truth class of the one or more secondary object categories may be derived in step 305 from the label of the current image. In step 307, the label of the current image may be compared with the prediction and the one or more derived labels may be compared with the one or more estimates. A loss function (named global loss function) may be evaluated in step 309 using the result of the comparisons. It may be determined in step 311 whether the loss function has an optimal value. In case, the loss function has the optimal value the trained image classifier may be provided in step 313; otherwise, the method goes back to step 301 for repeating the method using a next entry of the training data.

[0076] For example, if the training is performed using the primary and secondary object categories as defined in the selected example, the predicted class of step 301 may be a class of the classes of the primary object category k0. In step 303, two estimates may be derived from the prediction. The first estimate may be a class of the classes s3Cl ...s3Clnk3of the secondary object category / c3. This derivation may, for example, be performed by using the mapping Hk kto find the class of the secondary object category classes s3Cl ...s3Clnthat corresponds to the predicted class. The second estimate may be a class of the classes s2Cl ...s2Clnk^ of the secondary object category k2. This derivation may, for example, be performed by using the mapping Hfc0,fc1Hki k2to find the class of the secondary object category classes s2Cl1...s2Clnk^ that corresponds to the predicted class. In this case, the predicted class may be linked with the secondary object class through another secondary object class. Similarly, two labels may be derived in step 305 from the label of the image by using the mappings, one label that represents object category k3and another label that represents object category k2. In step 307, three comparisons may be performed between the label of the image and the predicted primary object class and between the two derived labels and respective derived secondary object classes.

[0077] FIG. 4 is a flowchart of a method for defining a global loss function in accordance with an example of the present subject matter. The method of FIG. 4 may provide an example implementation of the evaluation step 309 of FIG. 3. The comparison of the label of the current image with the prediction may be used to evaluate in step 401 a first individual loss function. The comparison of the one or more derived labels with the one or more estimates may be used to evaluate respective second individual loss functions in step 403. The first and second individual loss functions may be combined in step 405 to obtain the global loss function.

[0078] For example, if this method is performed using the primary and secondary object categories as defined in the selected example, three individual loss functions may be evaluated. The first individual loss function may be evaluated by comparing the predicted primary object class with the label associated with the image. The two second individual loss functions may be evaluated by comparing the two derived secondary object classes with the two derived labels respectively.

[0079] FIG. 5 is a flowchart of a method for classifying an image of an object in accordance with an example of the present subject matter. The method of FIG. 5 may, for example, use the trained image classifier obtained by the method of FIG. 3. The imaged object may be described by the biological ontology.

[0080] An image of the object described by the biological ontology may be received in step 501. The received image may be input in step 503 to the trained image classifier. An output may be obtained in step 505 from the trained image classifier, wherein the output may comprise a class or label of the image. For example, if the image classifier is a pixel-wise image classifier, the output may be a segmented image.

[0081] FIG. 6 is a diagram representing a biological ontology in accordance with an example of the present subject matter.

[0082] The biological ontology may be a plant ontology 600. The plant ontology 600 may be a hierarchical ontology. The plant ontology 600 comprises four object categories 601 , 602, 603 and 604, each representing a respective level of the hierarchy.

[0083] The object category 601 may be the plant species. The object category 602 may be the plant genus. The object category 603 may be the plant Class. The object category 604 may be the plant vegetation.

[0084] The object category 601 may have a set of object category classes 607 e.g., palemeri is an object category class of the object category 601. The object category 602 may have a set of object category classes 609 e.g., Setaria is an object category class of the object category 602. The object category 603 may have a set of object category classes 611 e.g., Dicotyledoneae is an object category class of the object category 603. The object category 604 may have a set of object category classes 613 e.g., vegetation is an object category class of the object category 604.

[0085] FIG. 6 further shows the relationships between classes of the different sets of classes 607, 609, 611 and 613. For example, the object category class helbola has an IS-A relationship with the object category class Setaria, the object category class Setaria has an IS-A relationship with the object category class Monocotyledoneae, and the object category class Monocotyledoneae has an IS-A relationship with the object category class vegetation.

[0086] The mapping matrix that enables to obtain the classes 613 from the classes 609 may, for example, be provided as follows: where the two columns represent the classes 613 of the vegeattion level while the four rows represnt classes 609 of the genus level. This matrix may be used to derive one of the classes 613 of the vegetation level that is associated with a given genus class.

[0087] FIG. 7A is a flowchart of a method for providing training data in accordance with an example of the present subject matter. The method of FIG. 7 may, for example, be implemented using the plant ontology 600 of FIG. 6.

[0088] The plant species 601 may be selected in step 701 as a primary object category of the object categories.

[0089] Multiple training datasets may be provided in step 703, wherein each training dataset is labeled with ground truth classes which are at least a subset of the set of species classes. The resulting training data are shown in FIG. 7B.

[0090] FIG. 7B shows a table listing the content of training data in accordance with an example of the present subject matter. The training data 630 comprises training datasets 631 through 638. The training datasets 631 through 637 are labeled with respective sets of species 641 through 647 respectively. The distinct species of all sets of species 641 through 647 may form the current set of species of the plan ontology shown in FIG. 6.

[0091] FIG. 8 is a flowchart of a method for defining a global loss function for training a pixel-wise image classifier in accordance with an example of the present subject matter. The training may be performed using training data obtained using the plant ontology 600.

[0092] The object categories 602, 603 and 604 may be selected in step 801 as secondary object categories. The secondary object categories 602, 603 and 604 represent a sequence of hierarchy levels of the plant ontology 600. The hierarchy level may be referred to as kj, where the index j = 0 represents the lowest level i.e. , species, j = 1 represents the hierarchy level genus, j = 2 represents the hierarchy level of plant Class and j = 3 is represents the highest level for plant vegetation.

[0093] The plant ontology may be used in step 803 to generate mappings that enable to map the classes of one level to the classes of a higher level. For that, the mappings may be provided as hierarchy matrices. A hierarchy matrix Hk kmay convert an output ykfrom hierarchy level kj into an output yk]+1of the upper level kj+1, where the matrix -kj,kJ+1is of a size nkx nkj+i, where nkand nkare the number of classes in the levels kj and kj+1respectively. Each element / cell of the matrix Hkj,kj+1may represent a distinct pair of classes (ykj, ykj+1) of the hierarchy levels kj and k]+1respectively. The element (ykof the matrix may be set to one if the class ykfrom hierarchy level kj belongs to super-class ykj+1under level / cj +1. Based on this matrix definition, the conversion of an output ykfrom hierarchy level kj into an upper hierarchy level kj+1to obtain the class yk may be performed by a matrix inner product multiplication: ykj+1= ykj* kj,kJ+1< where yke Rsx"fc / , where s is the number of pixels in the image a

[0094] The global loss function may be defined in step 805 using the mappings as follows: Lglobal(y, y) = WoLind The individual loss function Lind may be defined as follows: Lind(yfej. , yfej.) = Lyx Wcgj x Wcd jx Ws, where Lj is a selected base loss for the / -th level associated with the individual loss function, Wcg jare global class balancing weights for the / -th level, Wcdgare dataset class balancing weights the / -th level and Wsare the pixel weights. This loss function may allow setting flexible hierarchies that do not affect the architecture of the model.

[0095] The present method may prevent complexity increase on the architecture or parameters of the image classifier, by defining any classification hierarchy level in the plant ontology e.g., 600, as a sum of the components of the prediction or ground truth vector that belongs to the appropriate super-class. This is depicted in FIG. 6, where the output vector yspecies representing plant species may be used to generate the corresponding super-class vectors. For example, yspecies corresponds to the (pixel-wise) one-hot encoded classification output vector for each species class. The species that share some super-class on any of the upper super-classes in the taxonomy are added into their corresponding super-class outputs.

[0096] FIG. 9 represents a general computerized system suited for implementing at least part of method steps as involved in the disclosure. The client system 302 or each backend system of the set of backend systems may, for example, comprise the computer system 1200. The components of the computer system 1200 may include, but are not limited to, one or more processors or processing units 1203, a storage system 1211 , a memory system 1205, and a bus 1207 that couples various system components including memory system 1205 to processor 1203. Memory system 1205 may include any one or combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, etc.)) and nonvolatile memory elements (e.g., ROM, erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), programmable read only memory (PROM). Note that the memory system 1205 may have a distributed architecture, where various components are situated remote from one another, but can be accessed by the processor 1203.

[0097] The software in memory system 1205 may include one or more separate programs, each of which comprises an ordered listing of executable instructions for implementing logical functions, notably functions involved in embodiments of this disclosure. The software in memory system 1205 shall also typically include a suitable operating system (OS) 1222. The OS 1222 essentially controls the execution of other computer programs, such as possibly software 1227 for implementing methods as described herein. For example, the memory system 1205 may comprise an image classifier 1230 to be trained in accordance with the present subject matter. The training may be performed, for example, using training data 1231 stored in the storage system 1211.

[0098] Computer system 1200 may also communicate with one or more external devices such as a keyboard, a pointing device, a display, etc.; one or more devices that enable a user to interact with computer system; and / or any devices (e.g., network card, modem, etc.) that enable computer system 1200 to communicate with one or more other computing systems. Such communication can occur via I / O interface(s) 1219. Still yet, computer system 1200 may communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet) via network adapter 1209 that may comprise a Wireless and / or mobile network adapter. As depicted, network adapter 1209 communicates with the other components of computer system 1200 via bus 1207.

[0099] FIG. 10 and FIG. 11 show a diagram representing a plant ontology illustrating the working principle of the trained image classifier in accordance with an example of the present subject matter. The trained image classifier is configured to provide a probability prediction, y PRED, of the primary object classes in the primary object category 601 . In this example, the probability prediction is greater than zero for the primary object classes SPECIES 1, SPECIES 2, SPECIES 3, OTHER SPECIES, SPECIES 7 and OTHER SPECIES that are related to the secondary object classes GENUS 1 and GENUS 3. In this embodiment, optionally, if a species is known not to be present, the probability for this species may be moved to the generic class (i.e. OTHER SPECIES) or be set to zero. In this embodiment, optionally, if some specific class prediction is lower than a calculated threshold (e.g. calculated from the validation dataset to maximize a metric (such as accuracy, F1 Score, ...the probability for this species may be moved to the generic class (i.e. OTHER SPECIES) or be set to zero. The image classifier is configured to derive from the probability prediction of the primary object classes of the primary object category 601 a probability prediction for the secondary object classes of the secondary object categories 602-604. The probability prediction for the secondary object classes is determined by adding the probabilities of the primary object classes related to the respective secondary object class. Thus, GENUS 1 and CLASS 1 are assigned a probability prediction of 0,65 and GENUS 3 and CLASS 2 are assigned a probability prediction of 0,35, leading to a probability prediction for VEGETATION of 1 ,0. In a following step, shown in FIG. 11 , starting at the object category 604 of highest hierarchy the object class with assigned with the highest probability prediction is selected. From the related object classes of the next lower object category 603, again, the object class is selected (in this example, there is only one, namely CLASS 1 with a probability of 0,65). This concept is propagated up to the primary object category 601, such that SPECIES 2 is selected due to having the highest probability prediction (namely 0.25) of all the primary object classes that are related to GENUS 1 of the next higher object category 602.

[0100] As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as an apparatus, method, computer program or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit,” "module” or "system.” Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer executable code embodied thereon. A computer program comprises the computer executable code or "program instructions".

[0101] Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable storage medium. A ‘computer-readable storage medium' as used herein encompasses any tangible storage medium which may store instructions which are executable by a processor of a computing device. The computer-readable storage medium may be referred to as a computer-readable non-transitory storage medium. The computer-readable storage medium may also be referred to as a tangible computer readable medium. In some embodiments, a computer-readable storage medium may also be able to store data which is able to be accessed by the processor of the computing device.

[0102] ‘Computer memory' or 'memory' is an example of a computer-readable storage medium. Computer memory is any memory which is directly accessible to a processor. ‘Computer storage' or 'storage' is a further example of a computer- readable storage medium. Computer storage is any non-volatile computer-readable storage medium. In some embodiments computer storage may also be computer memory or vice versa.

[0103] A 'processor' as used herein encompasses an electronic component which is able to execute a program or machine executable instruction or computer executable code. References to the computing device comprising "a processor” should be interpreted as possibly containing more than one processor or processing core. The processor may for instance be a multi-core processor. A processor may also refer to a collection of processors within a single computer system or distributed amongst multiple computer systems. The term computing device should also be interpreted to possibly refer to a collection or network of computing devices each comprising a processor or processors. The computer executable code may be executed by multiple processors that may be within the same computing device or which may even be distributed across multiple computing devices.

[0104] Computer executable code may comprise machine executable instructions or a program which causes a processor to perform an aspect of the present invention. Computer executable code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages and compiled into machine executable instructions. In some instances, the computer executable code may be in the form of a high level language or in a pre-compiled form and be used in conjunction with an interpreter which generates the machine executable instructions on the fly.

[0105] Generally, the program instructions can be executed on one processor or on several processors. In the case of multiple processors, they can be distributed over several different entities. Each processor could execute a portion of the instructions intended for that entity. Thus, when referring to a system or process involving multiple entities, the computer program or program instructions are understood to be adapted to be executed by a processor associated or related to the respective entity.

Claims

CLAIMS1 . A method for training an image classifier for classifying images of objects, the objects having object categories which are organized in accordance with an ontology, the ontology being a biological ontology, the method comprising: determining a primary object category of the object categories, the primary object category having a set of primary object classes; determining one or more secondary object categories of the object categories, the one or more secondary object categories having specific relationships with the primary object category in accordance with the ontology, the secondary object category having one or more secondary object classes; providing training data comprising images and associated labels, the label of the image indicating a ground truth class of the image, the training data comprising multiple training datasets, each training dataset representing ground truth classes which are at least a subset of the set of primary object classes; training the image classifier using the training data, the training comprising for a given image of the training dataset: using the image classifier for providing a prediction of the primary object class of the given image; deriving from the prediction an estimate of the secondary object class of the one or more secondary object categories; deriving from the label of the given image one or more labels indicating a ground truth class of the one or more secondary object categories; comparing the label of the given image with the prediction and comparing the one or more derived labels with the one or more estimates, thereby evaluating a loss function; wherein the training is performed to optimize the loss function.

2. The method of claim 1 , wherein the comparison of the label of the given image with the prediction is performed to evaluate a first individual loss function, and the comparison of the one or more derived labels with the one or more estimates is performed to evaluate respective second individual loss functions, wherein the loss function is a combination of the first and second individual loss functions.

3. The method of claim 2, wherein the combination is a weighted sum, the method further comprising: ranking the one or more secondary object categories according to their relationships with the primary object category; assigning weights to the one or more second object categories in accordance with the ranking; assigning a weight to the primary object category, the weight of the primary object category being higher than the weights of the one or more second object categories; and using the weights for performing the weighted sum.

4. The method of any of the preceding claims 2 to 3, the method further comprising: for each class of each object category of the primary and secondary object categories: determining a global class balancing weight; determining per training dataset a dataset class balancing weight; wherein each individual loss function of the first and second individual loss functions is weighted with the respective global class balancing weight and / or the respective dataset class balancing weight.

5. The method of claim 4, the image classifier being a pixel-wise classifier wherein the individual loss function is further weighted with a pixel weight.

6. The method of any of the preceding claims, wherein the specific relationship of the second object category is a sequence of one or more IS-A relationships and / or a sequence of one or more HAS-A relationships.

7. The method of any of the preceding claims, wherein the ontology is a hierarchical ontology, wherein the primary object category belongs to a given level of the hierarchy, wherein the secondary object categories belong to levels of the hierarchy which are higher than the given level.

8. The method of claim 7, the given level being the lowest level of the hierarchy, and the secondary object categories belong to consecutive levels of the hierarchy.

9. The method of any of the preceding claims 7 to 8, the primary object category being plant species, the secondary object categories comprising at least: a plant genus, plant class and vegetation.

10. The method of any of the preceding claims 2 to 9, the image classifier being a pixel-wise image classifier, wherein the evaluation of the first and second individual loss functions is performed using a matrix equation, wherein the element of the matrix represents a pixel of the image and a class.11 . The method of any of the preceding claims, the image classifier being a pixel-wise image classifier or a whole image classifier.

12. The method of any of the preceding claims, further comprising pre-processing the images, the pre-processing comprising at least one of the following: resizing the images to a common size; correcting colour of the images; or changing orientation of the images.

13. The method of any of the preceding claims, the method further comprising: receiving an image of a field; and using the trained image classifier for segmenting the image, the image classifier being a pixel-wise image classifier.

14. A computer program for training an image classifier for classifying images of objects, the objects having object categories which are organized in accordance with an ontology, the ontology being a biological ontology, the computer program comprising instructions which, when the program is executed by a computer system cause the computer system to perform at least the following: determining a primary object category of the object categories, the primary object category having a set of primary object classes; determining one or more secondary object categories of the object categories, the one or more secondary object categories having specific relationships with the primary object category in accordance with the ontology, the secondary object category having one or more secondary object classes; providing training data comprising images and associated labels, the label of the image indicating a ground truth class of the image, the training data comprising multiple training datasets, each training dataset representing ground truth classes which are at least a subset of the set of primary object classes; training the image classifier using the training data, the training comprising for a given image of the training dataset: using the image classifier for providing a prediction of the primary object class of the given image; deriving from the prediction an estimate of the secondary object class of the one or more secondary object categories; deriving from the label of the given image one or more labels indicating a ground truth class of the one or more secondary object categories; comparing the label of the given image with the prediction and comparing the one or more derived labels with the one or more estimates, thereby evaluating a loss function; wherein the training is performed to optimize the loss function.

15. A computer system for training an image classifier for classifying images of objects, the objects having object categories which are organized in accordance with an ontology, the ontology being a biological ontology, the computer system being configured for: determining a primary object category of the object categories, the primary object category having a set of primary object classes; determining one or more secondary object categories of the object categories, the one or more secondary object categories having specific relationships with the primary object category in accordance with the ontology, the secondary object category having one or more secondary object classes; providing training data comprising images and associated labels, the label of the image indicating a ground truth class of the image, the training data comprising multiple training datasets, each training dataset representing ground truth classes which are at least a subset of the set of primary object classes; training the image classifier using the training data, the training comprising for a given image of the training dataset:using the image classifier for providing a prediction of the primary object class of the given image; deriving from the prediction an estimate of the secondary object class of the one or more secondary object categories; deriving from the label of the given image one or more labels indicating a ground truth class of the one or more secondary object categories; comparing the label of the given image with the prediction and comparing the one or more derived labels with the one or more estimates, thereby evaluating a loss function; wherein the training is performed to optimize the loss function.

16. An image classifier for classifying images of objects the objects having object categories which are organized in accordance with an ontology, the ontology being a biological ontology, wherein the image classifier is trained with a method of any of claims 1 to 13.

17. The image classifier according to claim 16, wherein the image classifier is configured to provide a probability prediction of the primary object classes in the primary object category; select a primary object class to be the correct primary object class based on the probability prediction of the primary object classes and the relationship of the primary object classes with the secondary object categories, optionally by deriving from the probability prediction of the primary object classes a probability prediction for the secondary object classes, in particular by adding the probabilities of the primary object classes related to the respective secondary object class, for each object category starting at the highest hierarchy level, selecting the object class with highest probability and propagate to the respective related next object category related to the selected object class until a primary object class is selected.

Images