Solution for assessing alterations such as stresses, resistances, damages, diseases and active ingredient concentrations of / in biological tissue using multispectral or hyperspectral imaging

Abstract

The present invention relates to a solution for assessing alterations in biological tissue in a non-invasive manner based on computer-implemented analysis of hyper and / or multispectral images of the biological tissue of interest. Alterations can be related to stresses, aging, pests, diseases, changes related to concentration of an active ingredient, and / or resistances against active ingredient, or the like as well as combination thereof. In a particular embodiment such alterations can be assigned to an alteration type and / or the degree of alteration can be quantified over time.

Description

[0001] Solution for assessing alterations such as stresses, resistances, damages, diseases and active ingredient concentrations of / in biological tissue using multispectral or hyperspectral imaging

[0002] CH D OF I M ’ A NTION

[0003] The present invention relates to a solution for assessing alterations in biological tissue in a non-invasive manner based on computer-implemented analysis of hyper and / or multispectral images of the biological tissue of interest. Alterations can be related to stresses, aging, pests, diseases, changes related to concentration of an active ingredient, and / or resistances against active ingredient, or the like as well as combination thereof. In a particular embodiment such alterations can be assigned to an alteration type and / or the degree of alteration can be quantified over time.

[0004] BACKGROUND OF THE INVENTION

[0005] The general background of this invention is the automated characterisation of biological tissue imaged non-invasively with hyper or multispectral cameras for assessment of tissue alterations such as stress and / or resistances and / or damages and or diseases and or active ingredient concentration over time.

[0006] For example, in herbicide / disease / pest and resistance screening assessment of plants for visible damages is essentially conducted visually by trained personal [Wang, H., Liu, W., Zhao, K., Yu, H., Zhang, J., & Wang, J. 2017. Evaluation of weed control efficacy and crop safety of the new HPPD-inhibiting herbicide-QYR301. Scientific reports. 2017.].

[0007] This is on one hand time intensive (lower throughput = higher cost) and prone to inaccuracies due to personal biases. Furthermore, the visual inspection does not allow to assess non- visible (the human eye is for one limited for the perceivable wavelength region (380-700nm) and cannot distinguish close wavelength) effects like elucidate the underlying mode of action (MoA) of chemical / disease induced damages, which is crucial for prioritization of chemical hits into field candidates. There is also a need for a solution, which allows identifying novel mode of actions, crucial to fight resistance in the field. Additionally, spotting damages as early as possible is advantageous, reducing screening time and thereby enhancing throughput. While multispectral and hyperspectral imaging are already widely used in e.g. material science, their application in screening of alterations of biological tissue is not industrialized. Prioritization of chemical hits is usually conducted in consideration of the following criteria or for one or more of the following purposes: a) Assigning the type of alteration to a known MoA or identifying new types of alteration potentially linked to new MoA’s - A new mode of actions not falling into a library of known modes of action can lead to new chemical hit classes, b) Optimizing chemical structure of a chemical hit class- Knowledge of the MoA (wanted / unwanted) in combination with efficacy of the chemical allows targeted chemical structure optimization, c) Identifying chemicals with multiple wanted MoAs can lead to broad band chemical hit, d) Checking if compound optimization is still on the desired target and / or e) Guiding structure optimization by unique biological phenotypes - Combine biological phenotypes with chemical structure information help guide structure optimization.

[0008] As a further example plant damages or plant stress may be caused by drought, temperature, nutrition deficiencies, diseases and plant resistances. Also, such stress or damages are commonly assessed visually by trained expert or with analytical techniques [Hadidi, A., Czosnek, H., Barba M. 2004. Dna microarrays and their potential applications. J. Plant Pathol. 2004.; Holl, G. 2009. Climate change and extreme weather. IOP Conf. Ser. Earth Environ. Sci. 2009; Heap, I. 2013. Global perspective of herbicide-resistantweeds. Society of Chemical Industry. 2013.].

[0009] Another example is assessing active ingredient concentrations in herbal plant, to identify the correct harvest time to fulfill quality requirements [WHO, (World Health Organisation),. 2007. WHO guidelines for assessing quality of herbal medicines with reference to contaminants and residues. 2007.].

[0010] In the pharmaceutical field diseases such as tumors are stained in histological slices of biopsies with antibodies. This has two major drawbacks. For one, this is a destructive method, since tissue must be sampled, and second antibody staining is time and cost intensive [Javaeed, A., Qamar, S., Ali, S., Mustafa, M. A., Nusrat, A., Ghauri, S. K. 2021. Histological Stains in the Past, Present, and Future. Cureus. 2021.]. In all fields it would be highly advantageous to have improved means for assessing alterations such as stress and / or resistance and / or disease and / or damages and / or diseases and / or active ingredient concentration of biological tissue. While technology for image acquisition is already available a dedicated software pipeline is required for its efficient use, and such a solution is described below.

[0011] SUMMAR Y OF THE INVENTION

[0012] The object underlying the invention is achieved by the combination of features according to the independent Claims. Exemplary embodiments of the invention can be gathered from dependent Claims thereto.

[0013] The invention is described in more detail without any division within the subject matter of the invention (method, system, etc). The explanations below are intended to be applicable analogously to all the subject matter of the invention, in either context.

[0014] DETA1LLED DESCRIPTION OF THE INVENTION

[0015] In the present invention the problem of identifying alterations such as damaged biological tissue was overcome by using time course multispectral or hyperspectral imaging (HSI) in combination with dedicated analysis software. The solution of the invention is particularly useful for the study of plant alterations in a non-invasive manner. While HSI is an established method to study plant physiology, so far, no commercial high throughput method (requiring analysis of 40TB and more of data / experiment) capable of screening herbicide / disease / pest damages or discriminate between resistant or no resistant plants are available.

[0016] The wording “alteration”, in the present application refers to any modification of a biological tissue between two images; this may be positive or negative such as but not limited to stress, damage, disease, resistance, recovery, and / or change in active ingredient concentration of biological tissue. In other words, alterations in the present invention are any physiological changes over time, which can be identified by comparing images at a time ti with a reference image showing an alteration, or an image a time ti-i or to .

[0017] Hyperspectral imaging technology is a combination of imaging techniques and spectroscopy. It has been known as a non-destructive method for studying physiological conditions of a plant (=plant phenotyping) [Lu, G., & Fei, B. 2014. Medical hyperspectral imaging: a review. J Biomed Opt. 2014.; Mishra, P., Asaari, M. S. M., Herrero-Langreo, A., Lohumi, S., Diezma, B., & Scheunders, P. 2017. Close Range hyperspectral imaging of plants: A review. Biosystems Engineering. 2017.; Mishra, P., Lohumi, S., Ahmad Khan, H., & Nordon, A. 2020. Close-range hyperspectral imaging of whole plants for digital phenotyping: Recent applications and illumination correction approaches. Computer and Electronics in Agriculture. 2020.; Mishra, P., Polder, G., Vilfan, N. 2020. Close Range Spectral Imaging for Disease Detection in Plants Using Autonomous Platforms: a Review on Recent Studies. Current Robotics Reports. 2020.; Asaari, M. S. M., Mishra, P., Mertens,

[0018] S., Dhondt, S., Inze, D., Wuyts, N., Scheunders, P. 2018. Close-range hyperspectral image analysis for the early detection of stress responses in individual plants in a high-throughput phenotyping platform. ISPRS Journal of Photogrammetry and Remote Sensing. 2018.; Liu, H., Bruning, B., Garnett, T., Berger, B. 2020. Hyperspectral imaging and 3D technologies for plant phenotyping: From satellite ti close-range sensing. Computers and Electronics in Agriculture. 2020.; Behmann, J., Steinriicken, J., Plumer, L. 2014. Detection of early plant stress responses in hyperspectral images. ISPRS Journal of Photogrammetry and Remote Sensing. 2014.; Li, L., Zhang, Q., Huang, D. 2014. A Review of Imaging Techniques for Plant Phenotyping. Sensors. 2014.]. A hyperspectral image is composed of up to several hundred monochrome images captured at different wavelengths, typically within narrow spectral bands with a spectral resolution of order Inm or narrower [Mishra et al., 2020 cited above], Hyperspectral Imaging (HSI) is known to be a valuable complementary measurement technology to existing RGB and fluorescence measurements used commonly for analysis of forest damages using satellite images [Nasi, R., Honkavaara, E., Lyytikainen- Saarenmaa, P., Blomqvist, M., Litkey, P., Hakala, T., Vijanen N., Kantola, T., Tanhuanpaa,

[0019] T., & Holopainen M. 2015. Using UAV-Based Photogrammetry and Hyperspectral Imaging for Mapping Bark Beetle Damage at Tree-Level. Remote Sens. 2015.] and waste separation [Bonifazi, G., Serranti, S. 2006. Imaging spectroscopy based strategies for ceramic glass contaminants removal in glass recycling. Waste Management. 2006.]. In this context HSI- measurements show higher accuracy by reducing false negative or false positive identification by verifying results from other sensors.

[0020] There is a need for solution allowing much earlier detection rates than e.g. RGB or human eye analysis methods, which would detect small and early alterations in biological tissues of interest in a non-invasive way. Such solution should be able to provide deeper insight into the mode of action of the screened chemicals, plant resistances, disease progressions / types a degree or changes in active ingredient concentrations. They should be able to quantify and compare alterations in a robust way. The problem was solved by a system and computer-implemented methods for analysis of at least one, preferably series of hyperspectral or multispectral images (generically referred to as multi wavelength images) at several wavelengths selected within a range from typically but not limited to a range from 250nm to 14000 nm of a scene within which at least part of the biological tissue is located.

[0021] The terms “multiwavelength image“, “multispectral image” or “hyperspectral image” used herein refer to a collection of several monochrome images of the same scene, wherein each monochrome image is an image at a single frequency or within a narrow spectral band (e. g. 0.3 nm). These terms within the scope of the present invention refers to a collection of images at frequency bands or within discrete wavelength ranges of interest within human visible and human invisible light spectrum, that is more than RGB. The generic term “multiwavelength” used herein refers to hyperspectral or multispectral collectively.

[0022] For the purpose of the present invention a hyperspectral image comprises images of contiguous ranges of wavelengths (e.g. 250 to 14000 nm in 0.3nm steps, in particular 400 to 1000 nm).

[0023] For the purpose of the present invention, a multispectral image is composed of images captured at different wavelengths of interest, typically within narrow spectral bands with a spectral resolution of order 0.3nm or narrower within a limited number of discrete wavelength ranges of interest.

[0024] It is clear to the person skilled in the art that multispectral or hyperspectral data can be analysed with the methods of the invention independently of the acquisition system provided the image resolution is good enough for acceptable accuracy.

[0025] In an embodiment, alterations can be categorized by their respective mode of action (MoA) or type, in particular, as environmental, physical or chemical alteration of the biological tissue, for example water stress, light stress, chemical binding to receptors of agrochemical or pharmaceutical agents, etc, or a combination thereof. When applying the solution of the application, it may be of interest to attribute an identified alteration to a mode of action, it can be useful to limit the investigations to a particular mode of action. Depending on such an application e.g., if only a known MoA or type - resistances to a disease, stress, concentrations of an active ingredient - are to be analyzed, a general procedure can be to identify characterizing wavelength(s) within the hyperspectral image and to image at those identified wavelengths with a multispectral camera. Selection of characterizing wavelength(s) improves acquisition speed and spatial resolution for routine analysis. When referring to a multi wavelength image, definition of wavelength(s) of interest is assumed.

[0026] In an embodiment, use of time series analysis of hyperspectral / multispectral data may allow identifying type and degree of alteration such as water or nutrient deficiency, even before visual assessment is possible, allowing correction of water and nutrient supply at an earlier stage. Potential use is keeping crop productivity at constant high level. In respect of plant / crop disease and resistance this method may spot infection, diseases as well as the degree of infection / resistance before visual effects occur. This allows a much earlier reaction time for e.g., field applications and / or other field management measures. Hyperspectral / multispectral imaging can also be used to monitor active ingredient concentration within herbal plant to determine the best harvest time to minimize substandard goods.

[0027] The skilled person will appreciate that multi wavelength imaging can be used to spot alterations in samples of other biological tissue and cells therein (so, it is not limited to plant tissue) - thus allowing earlier treatment with enhanced chances for rehabilitation, decreasing the time for analysis and reducing cost. Also, imaging does not damage tissue biopsies, which can later be used for other tests.

[0028] Finally, for effect screening purposes, the analysis time can be drastically reduced, saving costs, and allowing screening of larger quantities of chemicals or more dosages.

[0029] Objects of the present invention are a method and a system for assessing alterations of biological tissue using at least one multi wavelength image image as claimed. Exemplary embodiments of the method and system according to the invention can be gathered from the Claims dependent thereof.

[0030] The system for assessing alterations of biological tissue using multi wavelength imaging comprises one or more processors configured for:

[0031] Causing acquisition at least one multi wavelength image of a scene within which at least part of the biological tissue of interest is located at several wavelengths selected from a range from 250nm to 14000 nm;

[0032] Assigning, in the at least one multi wavelength image, pixels to the biological tissue(s) versus background using one or more deep learning algorithms for segmentation of biological tissue(s) from background trained with segmentation masks; Assigning each tissue assigned pixel to one alteration or non-alteration area(s) using one or more trained algorithms for pixel classification of region(s) showing an alteration and / or showing no alteration;

[0033] Causing output of the alteration and / or non-alteration area(s) to the user by way of a user interface.

[0034] To fully exploit the additional information contained in the multiple bands, the monochrome images are considered as one multi -spectral image rather than as a set of monochrome graylevel images. For an image with k bands, the brightness of each pixel as a point can be described in a k-dimensional space represented by a feature vector of length k or by a curve of intensity at wavelengths of interest. Such feature vector or corresponding curve of intensity is herein together referred to as “spectral signature”. To classify a pixel as belonging to a particular region, its intensities in the different bands describing its location in the k-dimensional feature space, means its spectral signature, are considered.

[0035] In an embodiment, the one or more processors are configured for determining at least one measure of alteration of the biological tissue in the one or more alteration areas based on the number of pixels in the one or more alteration areas. The one or more processors are preferably configured to cause output of the at least one measure of alteration of the biological tissue in the one or more alteration areas, most preferably of one or more representations thereof.

[0036] In an embodiment, the apparatus further comprises: at least one camera detector for multi wavelength imaging; preferred at least one light source having a continuous spectrum (e.g. halogen lamps) or a discrete spectrum (e.g. laserlines), a combination of multiple LEDs or the sun for field applications; wherein, the at least one camera detector is configured to acquire at least one multi wavelength image of a scene within which at least part of the biological tissue is located at several wavelengths selected from a range from 250nm to 14000 nm.

[0037] In an embodiment the at least one camera detector can be configured to acquire time series of multi wavelength images of a scene, wherein at least one multi wavelength image of the scene within which biological tissue is located can be acquired per instant ti (time point). In an embodiment, the system can be configured to acquired multi wavelength images at different angles, which can be aligned to 3-dimensional (3D)-measurements.

[0038] In an embodiment, multi wavelength images and related additional data can be acquired, such as time point of image acquisition, identifier for the biological tissue of interest, etc. Multi wavelength images and related additional data are also together referred as input.

[0039] In an embodiment, alterations can be identified by comparison of spectral signature of pixels within said image at instant ti with one or more reference spectral signature, predefined reference ranges for spectral signature or with one or more reference image on record.

[0040] In an embodiment, reference spectral signature may be available from a reference image, which may be any image or segments thereof with known spectral features e.g., a previous image of a time series at instant to or at instant ti-xalternatively a reference image for the biological tissue of record.

[0041] Comparison with said reference pixel values may show negative or positive alteration. In case effect of an active ingredient is screened, a reference image may be an image of the same tissue at different time points (e. g. maturation states of plants, before / after treatment).

[0042] Reference spectral signature and / or reference image of record can be selected by the user from a library over a user interface. In an example, the computer system can be configured for selecting a reference image, in particular for identification of type of alteration. This example is further described below. Also, a previous image can be used as reference image for comparison. In an embodiment, the user can select an image of a time series to be used as a reference image for analysis of alteration. The computer system can be configured to select an image of a time series to be used as a reference image for analysis of alteration. Per default image at to (first image of a time series) can be used.

[0043] In consideration of the multiplicity or complexity of possible alterations, there is a need to provide a solution allowing accurate identification of biological tissues in the shortest possible time.

[0044] For this purpose, the one or more processors are configured for analysis of the multi wavelength image, wherein first a deep learning algorithm for semantic segmenting the target biological tissue from the background in the image is used. In the present invention, analysis of an image is achieved by way of a computer implemented method comprising:

[0045] Causing acquisition of at least one multi wavelength image of a scene within which at least part of the biological tissue(s) of interest is located, said multi wavelength image comprising a plurality of monochrome images captured at several wavelengths selected from a range from 250nm to 14000 nm;

[0046] Assigning, in the at least one multi wavelength image, pixels to the biological tissue(s) versus background using one or more deep learning algorithms for segmentation of biological tissue(s) from background trained with segmentation masks;

[0047] Assigning each tissue assigned pixel to one alteration or non- alteration area(s) using one or more trained algorithms for pixel classification of region(s) showing an alteration;

[0048] Causing output of the alteration and / or non-alteration area(s) to the user by way of a user interface.

[0049] For the segmentation step, said one or more processors can be configured to conduct a deep learning approach for semantic image segmentation based on training masks for image segmentation. Said approach was chosen for better accuracy.

[0050] In the present application, the terms “segmentation masks” and “training masks” are equivalent. They relate to image data comprising labels and pixel-wise masks, wherein the masks are class-labels for each pixel. Each pixel is given one of three categories:

[0051] Class 1 : pixel belonging to the object,

[0052] Class 2: pixel bordering the object” or “edge / rim pixel”;

[0053] Class 3 : pixel belonging to none of the above also referred to as background pixels.

[0054] The deep learning approach was found to be improved by using a training data set obtained according to the method described below.

[0055] A computer-implemented method for generating a training segmentation data set comprising training masks was developed and used for training of one or more deep learning algorithms for segmentation of biological tissue(s) from background. With the present application, acquired images can be saved typically in pseudo RGB format. Training masks can be saved as binary masks.

[0056] Generated training data set can be stored in a repository and / or used in training one or more deep learning algorithm for segmentation of biological tissue versus background in a new multi wavelength image of a scene within which said biological tissue is located.

[0057] In an embodiment, said training segmentation data set can be obtained by way of:

[0058] Causing acquisition of a training data set of multi spectral / hyperspectral images of a scene within which the biological tissue is located from an image acquisition unit or from an repository of images;

[0059] Clustering each pixel of one multi spectral / hyperspectral image in an unsupervised manner using a clustering algorithm to obtain a set of k cluster, wherein k is the number of identified clusters and each cluster is defined by a cluster centroid and a total within-cluster variance TWCV;

[0060] - Using the elbow method computing for different k’s the explained TWCV and selecting the cluster set with the lowest k for which 90% of the TWCV is explained; Storing the selected clusters set as preliminary cluster(s) in a (transient) repository, wherein, for each preliminary cluster, the cluster centroid and the explained TWCV are stored;

[0061] Computing an image in which each pixel is assigned to one preliminary cluster by way of computing the Euclidean distance of the pixel spectral signature towards the cluster centroids and assigning a preliminary pixel labeling based thereon;

[0062] Optionally causing revising of preliminary cluster pixel labeling to the biological tissue versus background by way of displaying of the image with the preliminary cluster(s) to a user for cluster pixel labeling by user;

[0063] Saving pixels and assigned cluster pixel labeling to the biological tissue in form of a training segmentation mask together with respective training image in a repository (for later use in training one or more deep learning algorithm for segmentation of the biological tissue);

[0064] - Reiterating the method for the other multi spectral / hyperspectral images of the training data set.

[0065] The method can be used to generate generic cluster pixel labeling related to one tissue of interest for example “tissue”, yes (for tissue) / no (for background), or a more specific label for tissue type thereof, for example “leave”, “fruit”, “stem” or the like, if specific type of tissue is of interest.

[0066] In an embodiment, the one or more processors of the computer system are configured to conduct the steps for the method described above.

[0067] Obtained training segmentation data set can be used to train one or more deep learning algorithm for semantic image segmentation of an image by way of.

[0068] Causing feeding a deep learning algorithm for semantic image segmentation with a training segmentation data set comprising at least one multispectral image alongside one or more segmentation masks containing the whole or parts of the tissue;

[0069] Fitting the deep learning algorithm for semantic image segmentation by optimizing assignment of each pixel to tissue of interest or background in the training segmentation data set using an optimizer (optimizing algorithm);

[0070] Causing storage of the trained deep learning algorithm for semantic image segmentation for later use on new hyperspectral / multispectral images.

[0071] In an embodiment, the one or more processors are configured to conduct the training steps for the deep learning algorithm for semantic image segmentation mentioned above. In an embodiment, one or more deep learning algorithm for semantic image segmentation of pixels related to biological tissue versus background can be retrieved manually or automatically from a repository comprising a collection of trained deep learning algorithms.

[0072] In the training phase of the deep learning algorithm for semantic image segmentation, a training image refers to the original multispectral image or a selection of wavelength (such as a red, green, and blue channel) alongside one or more training masks (e. g. binary training masks) containing the whole or parts of the tissue.

[0073] Training masks or a refined training set of segmented images can also be used to train one or more deep learning / machine learning algorithms for more detailed segmentation of different types of biological tissue (for example plant organs, such as stem, leaf veins and leaf tissue).

[0074] For whole tissue or type-specific segmentation, neural networks can be used. Fig, 4 and 5 show examples of whole tissue or tissue type-specific segmentation according to the method described above. In an embodiment, segmentation of different tissue types can be derived automatically after segmentation of the tissue versus background by comparing spectral signature or pixel values for each pixel with stored tissue reference spectral signatures, since different tissue parts have unique spectral properties (cf. example below, Fig. 5) and corresponding pixel labeling.

[0075] In an embodiment, deep learning algorithm for semantic image segmentation can be trained with a training set of segmented images, which pixels are annotated with the classes for tissue types of interest. Such training set can be obtained by the method described above and further detailed below.

[0076] In an embodiment, the one or more processors of the computer system can be configured to conduct such comparison and labeling automatically. User may select segmentation per tissue type using the user interface.

[0077] The solution for assigning each pixel of a multi wavelength image to the tissue of interest versus background from a multi wavelength image is also referred to as segmenting method.

[0078] Once image is segmented as described above, only pixels assigned to a tissue / tissue type of interest are analyzed for recognition of alteration area(s) by way of one or more algorithm for recognition of alteration areas and / or non-alteration areas. Recognition of alteration areas is achieved using one or more algorithm assigning, preferred classifying, every pixel assigned to the tissue or / or tissue type to an alteration or a non-alteration area. This process is also referred to as classification and the related method is generically referred to as classification method; the corresponding algorithms are generically referred to as algorithms for pixel classification of region(s) showing an alteration and / or showing no alteration.

[0079] In an embodiment, a centroid-based clustering method can be used for the recognition of (non-)alteration areas (also referred to as (non-)alteration segments or (non-)alteration regions) in a segmented multi wavelength image. In the present application, alteration areas and / or non-alteration areas are characterized as a group of pixels with similar spectral signatures within the pixels assigned to the tissue of interest. Within the present application similarity of spectral signature is characterized by the Euclidean distance of the pixel spectral signature towards the cluster centroid. Assignment of a pixel to a cluster is “accurate” when, Euclidian distance is within a defined acceptable threshold. Pixel not fulfilling these criteria may be of particular interest for identifying new alteration types (based on novel / unknown spectral signatures). An algorithm for recognition of (non-)alteration areas can be unsupervised like k-means algorithm, k nearest neighbors, PC A, UMAP, t-SNE, DBSCAN methods, unsupervised trained classification methods / algorithms, such as convolutional neural networks (CNN), autoencoders, variational autoencoders, Generative Adversarial Networks (GAN), random forest approaches, or combination thereof.

[0080] In case classification networks such as convolutional neural networks (CNN), autoencoders, variational autoencoders, Generative Adversarial Networks (GAN) are used, the pixel spectra are encoded into a lower dimensional space, that is a latent space capturing the most important features or aspects representing the input data. This latent space can be two or three dimensional; it can be visualized as a scatter plot allowing to separate regions of alteration and non-alteration. Computing and presenting a 2D- or 3D- representation of the latent space to the user, allows him judging on how well alteration and non-alteration are separated.

[0081] In case unsupervised neural networks are used, representation of the computed latent space can be highly valuable in finding new / unusual spectral signatures (also referred to as novelty detection), based on the position of the pixel in the latent space.

[0082] In an embodiment new mode of actions can be identified using this method, if spectral signatures appear in completely different locations of the lower dimensional space.

[0083] In case a supervised classification algorithm is used for recognition of alteration areas, training of algorithm can be achieved using annotated images and / or user-refined annotated images in view of alteration of interest; label may be a generic class label such as “alteration yes / no” or more specific to an alteration type for the pixel.

[0084] Convolutional neural networks are preferred algorithm for recognition of (non-)alteration areas because they use not only the cluster centroid but also TWCV (the distance to the centroid) to assign each pixel to one cluster, making clustering more robust. Autoencoders or variational autoencoders are other unsupervised deep learning methods which learn a strong feature representation within the latent space for reconstruction of spectral signatures from pixels within the tissue. Such feature representation can be used for classification of the pixels of segmented image data and requires no user input at all. These methods are particularly useful for identifying new spectral signatures (novelty detection) based on the location in the latent space. The person skilled in the art will appreciate that one or more classification algorithms for pixel classification for recognition of alteration areas can be trained with training data comprising multi wavelength images of scenes comprising whole or part of the biological tissue of interest and corresponding level of pixelwise labeling, that is non-alteration or a specific class label relating to a specific alteration type. Alternatively unsupervised methods like clustering or deep learning modes like variational autoencoders, GANs or transformers can be used, to generate labels in an unsupervised manner. Trained classification algorithm(s) can be later used for classification of the tissue assigned pixels of a new segmented multi wavelength image of a new sample of biological tissue of interest.

[0085] In case prior knowledge on alteration is limited (e. g. no training data set for supervised training of an algorithm available), the method can be configurated to proceed to an unsupervised classification, in order to spot novel spectral signatures and / or to allow the user to add labels classification for training of supervised classification. This embodiment can be advantageous to identify alteration type for example fungal or pest infestation or stress and / or to identify new mode of actions.

[0086] It is noted that a limited number of classified images may be sufficient for supervised training of a classification algorithm for recognition of alteration or non-alteration area, provided every tissue assigned pixel in a training image is classified to a non-alteration vs alteration area and labeled accordingly either in a generic manner and / or by alteration type. Data volume for reliable training can be readily achieved for training considering that one single image comprises many classified pixels. In other words, for the purpose of supervised training of a classification algorithm, a training data set comprising one or more classified image, or a plurality of classified pixels can be acquired from an image data repository, wherein each pixel assigned to the tissue or tissue type of interest is additionally assigned to one of the labels related to alteration or non-alteration, that is a label selected from non- alteration label, a generic alteration label and / or a specific class label relating to a specific alteration type. For the purpose of unsupervised training of a classification algorithm, a training data set comprising one or more non-classified image, or a plurality of unclassified pixels can be acquired from an image data repository, wherein each pixel is assigned to the tissue or tissue type of interest and no label related to alteration is assigned.

[0087] The segmented multiwavelength image treated with the classification method is also referred to as a “classified (multi wavelength) image”. In a classified image, pixels assigned to the same class are grouped in a cluster; the cluster is represented by a mean vector also referred to a cluster centroid. The mean vector can be the average spectral signature of the cluster; For example, Fig. 5c shows averaged spectral signatures of alteration / non-alteration areas identified in segmented image 5a.

[0088] A group of pixels assigned to an alteration, or a non-alteration area is also alone or together referred to as cluster area(s), corresponding to the surface in the imaged scene belonging to a respective cluster. In the present invention a cluster area is defined by the spectral signature of its cluster centroid and its number of pixels alternatively by its pixel fraction, that is a ratio of the number of pixels for the cluster area to the whole number of pixels attributed to the biological tissue or to a tissue type of interest. Pixels assigned to a cluster area are automatically annotated with corresponding class label.

[0089] In an embodiment, the user can select one or more level of classification or analysis: identifying alteration vs non-alteration, identifying alteration type, quantification of alteration at a single time point, progress analysis, etc. The system is configured to automatically conduct corresponding instructions / algorithm and cause display of results.

[0090] In an embodiment, a spatial representation of the identified alteration or non-alteration area(s), assigning each area with a false color can be computed.

[0091] In an embodiment, at least one measure of alteration of the biological tissue in the one or more alteration areas can be determined based on the number of pixels in the one or more alteration areas. In an embodiment, pixel fractions can be used to computing a representation of the alteration as stacked bars. Pixels fractions can be used to compute alteration curve or value between different time points or between an image and a reference image. Use of robust segmentation and classification methods is of advantage for reliable quantification of a tissue alteration over time.

[0092] Fully classified images can be later used as reference image in some examples of the present invention. Time series of fully classified images can be later used as reference time series in some examples of the present invention; computed alteration curves for these time series can be used a reference alteration curves. Fully classified images can also be used within a training data set or a validation data for training of a classification algorithm.

[0093] In an embodiment, spectral signature of tissue assigned pixels can be compared with reference spectral signatures of record for the biological tissue at stake showing no alteration or a particular type of alteration, preferably at a several time point. In an embodiment, alteration type can be identified using a trained classification algorithm. Training can be achieved using a training data set comprising segmented and classified images of a scene comprising the tissue of interest, wherein alteration pixels have been assigned to an alteration type with sufficient validity, either by way of a computer implemented method and / or revision by user.

[0094] In an embodiment, average spectral signature(s)of (non-)alteration areas identified in new image data can be compared with reference spectral signature(s) of record for the biological tissue at stake showing no alteration or a particular type of alteration, eventually at a particular point in time.

[0095] Preferred embodiments of this invention are further described, including the best mode known to the inventors for carrying out the invention.

[0096] Variations of those preferred embodiments can become apparent to those of ordinary skilled artisans to employ such variations as appropriate, and the inventors intend for the inventions to be practiced otherwise than specifically described herein.

[0097] Accordingly, this invention includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the invention unless otherwise indicated herein or otherwise clearly contradicted by context.

[0098] In an example, acquisition of the at least one multi wavelength image of a scene can be achieved using a hyperspectral / multispectral camera 20, controlled light environments 60, for example lamps covering the full wavelength spectrum housed in a light shielded cabinet to exclude light from outside connected to a computer system 40 comprising one or more processors to process acquired data. The at least one camera detector can be configured to acquire at least one, or time series of multi wavelength image of a scene within which at least part of the object of interest, e.g. plant or tissue part, is located (Fig. 3).

[0099] In step a) multiwavelength image may be acquired in the one or more processors configured for analysis of multiwavelength images and subjected to normalization to reduce or eliminate the influence of illumination effects and / or detector sensitivity. For this purpose, said one or more processors can be configured to normalize the data of the image. Standard methods for normalization such as referencing to a white reference and / or measuring a dark current can be used.

[0100] In an alternative embodiment, normalization can be conducted separately, the computer system acquiring normalized multi wavelength images from an external source such as a repository for normalized images.

[0101] A wide variety of methods for normalization is available from literature (Mishra, 2020) ((Mishra, Lohumi, Ahmad Khan, & Nordon, 2020)). These methods span classical averaging, standard normal variate normalization as well as wavelength dependent normalization techniques such as SVN (Variable Sorting for Normalization). These approaches are usually carried out on normalized data to account for geometric and illumination influences.

[0102] Working on 2-dimensional images of 3D-objects (e.g. whole plant), it can be advantageous for better accuracy to consider geometric influences on pixel data: Geometrical influences are particularly important for accurate pixel classification to (non-)alteration areas. Solutions for consideration of geometric influence are detailed below. The person skilled in the art will appreciate that geometric influence may also impact pixel assignment to tissue (type) versus background.

[0103] Normalized multi wavelength images comprising a scene within biological tissue of interest can be used to build a training segmentation data set.

[0104] For this purpose, each normalized multi wavelength images of a data step is submitted to:

[0105] - Preliminary clustering image pixels in an unsupervised manner using a clustering algorithm to generate a set of k preliminary clusters, wherein k is the number of identified preliminary clusters, single pixel spectral signatures are grouped by similarity of spectral shapes, and each preliminary cluster is defined by its centroid and a total within-cluster variance TWCV;

[0106] - Using the elbow method computing for different k’s the explained TWCV and selecting the cluster set with the lowest k for which 90% of the TWCV is explained; Storing the selected clusters set as preliminary cluster(s) in a (transient) repository, wherein, for each preliminary cluster, the cluster centroid and the explained TWCV are stored; Storing the selected clusters set as preliminary cluster(s) in a (transient) repository, wherein, for each preliminary cluster, the cluster centroid and the explained TWCV are stored;

[0107] Computing an image in which each pixel is assigned to one preliminary cluster by way of computing the Euclidean distance of the pixel spectral signature towards the cluster centroids and assigning a preliminary pixel label based thereon;

[0108] Optionally causing revising of preliminary cluster pixel label to the biological tissue versus background by way of displaying of the image with the preliminary cluster(s) to a user for user labeling;

[0109] Saving pixels with the label to the biological tissue in form of a binary mask as a training segmentation mask together with respective training image in a repository.

[0110] The method is reiterated for the other multi wavelength images of the training data set.

[0111] In the approach mentioned above, pixels are first clustered in an unsupervised manner using a clustering algorithm (also called algorithm for unsupervised clustering), such as a k-means algorithm, k nearest neighbors, PCA,UMAP, t-SNE, DBSCAN methods and / or unsupervised classification neural networks (e.g. variational autoencoders, GANS, transformers); the person skilled in the art will appreciate that any method for unsupervised clustering may be used. Methods for unsupervised clustering is described for example by Madhulatha [Madhulatha, T., S. 2012. An Overview on Clustering Methods. IOSR Journal of Engineering. 2012; Cantorna D. et al, Oil spill segmentation in SAR images using convolutional neural networks. A comparative analysis with clustering and logistic regression algorithms, Applied Soft Computing, Volume 84, November 2019, 105716; https: / / www.sciencedirect.com / science / article / pii / S1568494619304971 ].

[0112] In an embodiment of the method of the invention, k-means++ for seeding clusters can be used. The clustering algorithm aims to partition n observations (in this case multispectral or hyperspectral reflection or absorption intensity values of pixels) into a fixed number k of clusters. This can be achieved in an iterative way in which a fixed number k of seeds, that is potential mean vectors / centroids for the k clusters, are placed randomly and the squared Euclidean distance of each pixel is calculated towards each centroid; the obtained pixel-to- centroid distances are summed up to obtain a total within-cluster variance (TWCV). By changing the centroid, the clustering algorithm minimizes this TWCV resulting in a separation of observations with the lowest possible TWCV. One drawback of many centroid-based clustering algorithms is the uncertainty of the number of clusters (k) to be used for accurate result.

[0113] In a preferred embodiment, this uncertainty can be addressed using the elbow method, which basically computes for different k’s explained TWCV for each cluster then choses the set of clusters with the lowest k for which 90% of the variance is explained. For each cluster, one cluster centroid and the explained TWCV, are saved and stored for example in a (transient) cluster repository. In the present invention, unsupervised clustering of a training set of multi wavelength images leads to a preliminary cluster assignment for each pixel, to either tissue of interest (or tissue type), or background.

[0114] In an embodiment, the system can be configured to prompt a user via a user interface, so preliminary cluster assignment to a tissue or tissue type of interest can be defmed / confirmed by the user via the user interface. In an embodiment, user may define user preferences, if each preliminary cluster assignment is to be displayed for user confirmation or if display shall occur based on a computed validity score for segmentation.

[0115] As a matter of example, in case a whole plant is considered, tissue of interest may be the whole plant; tissue type may be or one or more single organs of the plant. Pixels which are not assigned to the tissue of interest or to a tissue type thereof are assigned to the background. The skilled person will appreciate that centroid based clustering by tissue type can be obtained by way of introducing class(es) for tissue type(s) of interest.

[0116] In such method, assignment of each pixel to one cluster relies on the closest squared Euclidean distance to the cluster centroids, so each pixel is assigned to one single cluster. The accuracy of the pixel assignment for use in a training segmentation data set (images and training masks) for training of a deep learning algorithms for segmentation was found to have major influence on the reliability of the solution when applied to a new image data of a scene comprising the biological tissue. Quality of the segmentation has a particularly big impact on the accuracy of subsequent quantification of alteration.

[0117] Several methods can be used to enhance the quality of the training segmentation data set and subsequent training of a deep learning algorithm for semantic image segmentation.

[0118] In an embodiment, training of deep learning algorithms for segmentation of biological tissue can be enhanced by augmentation of single pixels or of the whole image, for example, for training the segmentation algorithm to geometric and / or illumination influences. Augmentation is of particular interest for analysis of three-dimensional tissue, such as whole plant or plant organs, for which consideration of geometric and / or illumination variations during imaging may strongly influence segmentation accuracy.

[0119] In an embodiment, spectral augmentation e. g. by adding one or more constant factors to the spectra intensities (mimicking noise, changes in contrast, illumination, orientation, rotations and translations) can be used to make the algorithms more robust towards geometric and illumination effects.

[0120] In an embodiment, the one or more processors are configured to proceed to augmentation of the acquired multi wavelength images, e. g. by adding constant factors to the spectra intensities to simulate geometric and / or illumination effects.

[0121] “Geometric influences / effects” as used in the present application refers to angle and height reflectance variations of an imaged 3 -dimensional object in a 2-dimensional image.

[0122] In an embodiment, it is preferred that a data set comprising user-revised training masks is used for training of the algorithm for segmentation of biological tissue. To obtain user- revised trainings masks, the set of preliminary clustered images obtained from the algorithm for unsupervised clustering can be caused to be displayed for revision by user input for a more reliable training data set. It is preferred that the system allows the user zooming in a displayed image or part of the image for better revision.

[0123] In an embodiment, user can assign clusters by way of revising labeling for each pixel, in particular edge / rim pixels (transition from target biological tissue to background), either to the biological tissue of interest or background.

[0124] In an embodiment, an image with labeling of preliminary assigned clusters can be generated, wherein each pixel is assigned to a preliminary cluster with a computed validity score, and wherein each pixel is displayed to the user for convenient confirmation or correction of preliminary cluster assignment in consideration of the computed validity score. In an embodiment, only pixel attributed to a preliminary cluster with a computed validity score below a predefined threshold are placed for user revision.

[0125] In an embodiment, a pixel group corresponding to one preliminary cluster is shown to a user via a user interface, asking him to select or confirm whether, all pixels of the displayed group belong to the target tissue (alternatively tissue of interest), to the background or comprises a mixture of both. In the first two scenarios, the clustering process is finished; the pixels are annotated accordingly. In case the cluster is annotated by the user as a mixture of pixels belonging to target tissue and background (mixed cluster), the clustering step with the algorithm for unsupervised clustering is repeated for the mixed cluster, until only labeling as target tissue or background is achieved. In contrast to conventional semantic segmentation algorithms or methods, this results in a much higher accuracy of the clusters since the user does not have to annotate pixelwise. This is most profitable for later unsupervised segmentation of new images since especially edge / rim pixels (transition from target biological tissue to background) will highly influence accuracy of quantification results. Each annotated image is saved as training image data in a refined training data set; the group of pixels attributed to a target tissue is also referred to as training masks. In an example only training masks (that is pixels related to a tissue of interest) are saved.

[0126] In an embodiment, the refined training data set can be used to train one or more semantic / deep learning segmentation algorithm, for example a neural network, for image segmentation of new image data.

[0127] In a preferred embodiment, deep learning algorithms, for example convolutional, transformer or dense neural networks or random forest approaches can be used for semantic image segmentation, that is the process of linking each pixel in an image to a class label related tissue (eventually tissue type) vs background.

[0128] Neural networks are preferred segmentation algorithm because they use not only the cluster centroid but also TWCV (the distance to the centroid) to assign a pixel, making segmentation more robust. Most preferred are convolutional neural network.

[0129] In an embodiment, one or more deep learning algorithms for segmentation of biological tissue(s) from background trained with segmentation masks can be used for segmentation of new multi wavelength image data in the real-time operating system.

[0130] The present method preferably uses one or more deep convolutional classification neural networks comprising at least three layers of processing elements: a first layer with input neurons (nodes), an Nth layer with at least one output neuron (node), and N-2 inner layers, where N is a natural number greater than 3.

[0131] In such a neural network, the output neuron(s) serve(s) to provide an information, the information indicating whether the pixel is assigned either to the tissue (or a tissue type) of interest or to the background. The input neurons serve to receive multi wavelength image data (and optionally additional data) as input values. Preferably normalized m multi wavelength image data are used. Usually there is one input neuron for each pixel of the at least one image of the tissue of interest. There can be additional input neurons for additional data. The processing elements of the layers are interconnected in a predetermined pattern with predetermined connection weights therebetween; said patterns and weights have been predetermined by way of training of the neural network with a training data set comprising multi wavelength images and respective segmentation masks obtained using the method described above and a training algorithm.

[0132] For the purpose of training one or more deep learning algorithm for semantic segmentation of a biological tissue in a multi wavelength (hyperspectral or multi spectral) image, a training data set obtained by the method mentioned above comprising at least one segmented image, is introduced in a neural network, in a backward propagation, patterns and weights are optimized to minimize loss using a training algorithm.

[0133] When trained, the connection weights between the processing elements contain information regarding the relationship between the multi wavelength image data (and optionally additional data) (input) and the information about the assignment to tissue or background(output). Each network node represents a calculation of the weighted sum of inputs from prior nodes and a non-linear output function. The combined calculation of the network nodes relates the inputs to the output(s). Trained neural network can be used to segment new image data (and optionally additional data).

[0134] In an embodiment, the biological tissue may be segmented by tissue type. For example, a plant may be segmented in leaves and stems or even roots if root system is also part of the image. It may be of interest to focus the analysis on one or more type of the biological tissue. For this purpose, the list of class labels relating to the biological tissue mentioned above may be complemented accordingly. Segmentation by tissue type may be achieved by splitting the training mask(s) into the tissue types of interest (e. g. leaves, stems, etc.) and train one or more deep learning algorithms - preferred convolutional neural networks - for segmentation of the biological tissue by tissue type.

[0135] In an embodiment, considering different biological tissue types show different spectral properties, the class labels related to segmentation of the pixels by tissue type can be obtained by comparison of the spectral signature of each pixel with the spectral signatures of reference pixels or reference clusters and labeling to a specific biological tissue type. Said comparison can be computer-implemented.

[0136] To identify tissue type, the corresponding clusters are best annotated with a unique class label.

[0137] The solution of the invention solves the following major pitfalls in hyperspectral / multispectral image analysis in comparison to the art:

[0138] - Generating accurate training masks according to the art is highly time intensive and therefore, expensive.

[0139] - Due to the low spatial resolution of the images the edge (=rim) pixels (pixel between target tissue and surrounding background) may comprise spectral properties of both plant / tissue and background and thus, are often removed for further analysis leading to a loss of quantification accuracy. A convenient user-supported solution for generating training masks is proposed. Described solution avoids drawing accurate masks not containing edge pixels with a significant increase in evaluation time and cost.

[0140] The solution described above provides computer-support for convenient pixel classification to tissue or background confirmed by expert user.

[0141] Typically, one or more processors are configured to run a computer program implementing the method(s) related to semantic segmentation described above.

[0142] In summary, computer-implemented semantic segmenting of a new multi wavelength image into defined areas (segments) with defined contours comprising pixels annotated as belonging to the biological tissue or a tissue type, can be achieved by automatically attributing each pixel to one segment of the image using one or more classical but preferably deep learning algorithm for segmentation of the biological tissue trained based on the training masks obtained by the method described above. The skilled person will appreciate that optionally the method can cause asking the for confirmation of segmentation of a new image.

[0143] Once an image is segmented using a trained deep learning algorithm for semantic image segmentation, only pixels assigned to a tissue / tissue type of interest are analyzed for recognition of tissue alteration area(s) by way of one or more algorithm for recognition of alteration areas. Recognition of alteration areas is achieved using one or more algorithm assigning, preferred classifying, every pixel assigned to the tissue or / or tissue type to an alteration or a non- alteration area. This process is also referred to as classification and the related method is generically referred to as classification method. Algorithms for recognition of tissue alteration areas are referred to as algorithms for pixel classification of alteration and / or non- alteration areas.

[0144] Identifying tissue alteration area(s) can be achieved by way of applying one or more trained algorithms for pixel classification of alteration and / or non-alteration areas to the pixels assigned to tissue (type).

[0145] In an embodiment, one or more trained algorithm for unsupervised recognition of such areas can be used, wherein said algorithm is clustering tissue pixels assigned to tissue into similar groups or clusters. Examples of unsupervised clustering of pixels into clusters is shown on Fig. 5 to 11.

[0146] Unsupervised pixel classification of alteration and / or non-alteration areas can be achieved by performing e. g. k-mean, k-nearest neighbours, DBSCAN methods, or using one or more unsupervised trained image classification networks, such as autoencoders, variational autoencoders or Generative Adversarial Networks (GAN) on the segmented images.

[0147] In an embodiment, one or more unsupervised trained image classification networks are used.

[0148] Unsupervised clustering approaches are usually carried out on normalized data. Normalization methods mentioned by Mishra et al [Mishra, P., Lohumi, S., Ahmad Khan, H., & Nordon, A. 2020. Close-range hyperspectral imaging of whole plants for digital phenotyping: Recent applications and illumination correction approaches. Computer and Electronics in Agriculture. 2020; Mishra, P., Polder, G., Vilfan, N. 2020. Close Range Spectral Imaging for Disease Detection in Plants Using Autonomous Platforms: a Review on Recent Studies. Current Robotics Reports. 2020] are widely used in the art to account for illumination and / or geometric influences in the image analysis of three-dimensional objects. These methods span classical averaging, standard normal variate normalization as well as wavelength dependent normalization techniques such as SVN (Variable Sorting for Normalization).

[0149] However, these methods were found to lack the ability to reliably distinguish spectral signature alterations due to 3 -dimensional geometry of the imaged object from those really related to biological alterations of the tissue(s) of interest. To solve this problem, it is proposed to consider prior knowledge on geometric effect. In an embodiment, prior knowledge on geometry alterations of acquired images can be introduced in a training data set for unsupervised learning of neural network to consider angle and height reflectance variations of an imaged 3-dimensional object in the acquired images. This can be achieved by normalizing the image data and / or by performing spectral data augmentation to mimic geometric influences in the data set during the training of the algorithm for semantic image -

[0150] To our knowledge, geometric influences correction using spectral data augmentation in classification network training instead of using fixed methods like standard normal variate has not been used in the art, although different heights and angles of biological tissue in the imaged scene can influence the different parts of the spectra, causing loss of discrimination of important segment of the spectra, therefore impacting identification of related alterations. It was even found that fixed methods such as standard normal variate normalize data but may also accidently worsen geometric effects.

[0151] Another object of the invention is therefore a computer-implemented method for training an algorithm for pixel classification of tissue region(s) in a multi wavelength image into alteration and non-alteration area, comprising:

[0152] Causing feeding the algorithm for pixel classification of tissue region(s) showing an alteration and / or showing no alteration with a training data set comprising at least one multi wavelength image, wherein each pixel assigned to tissue is either classified to nonalteration vs alteration area and labeled accordingly or not classified in view of alteration;

[0153] Submitting the training data set to spectral data augmentation by adding one or more constant factors to the spectra intensities for the training data set;

[0154] Fitting the deep learning algorithm for pixel classification by optimizing assignment of each pixel to an alteration or non-alteration area using an optimizer;

[0155] Causing storage of algorithm for pixel classification for later use on new multi wavelength images.

[0156] Note that the algorithm for pixel classification of tissue region(s) showing an alteration and / or showing no alteration can be trained with a training data set comprising at least one multi wavelength image, wherein each pixel assigned to tissue is either classified to non- alteration vs alteration area and labeled accordingly (supervised training) or not classified in view of alteration (training) depending on available prior knowledge of alteration of interest.

[0157] In an embodiment, prior knowledge can be achieved additionally, or alternatively to spectral data augmentation, by studying the tissue at stake in a flattened form to incorporate training data on spectral signatures caused by alterations without geometrical influences at image acquisition. In such an embodiment, a training data set comprises multi spectral / hyperspectral images of a scene within which flattened biological tissue is located.

[0158] Using images of the tissue in flattened form, in particular annotated images, in the training set for the training of the one or more deep learning algorithms for classification of alterations allows the networks to learn the difference between influences caused by tissue alterations and geometric influences. This solution relies on learning the “errors” instead of correcting them, making it more robust with limited effort. This is particularly valuable for reliable imaging analysis of three-dimensional tissue, for example a plant or plant organs.

[0159] For unsupervised trained classification networks, autoencoders, variational autoencoders or GAN networks it is advisable to use augmentation to learn the geometric influence rather than to use fixed methods such as SNV.

[0160] In an embodiment unsupervised clustering algorithms e. g. k-means, autoencoders, variational autoencoders, GAN networks can be used to assign pixels to a unique alteration / non-alteration area. In an embodiment, also clusters centroids from a spectral library can be used as reference and each pixel can be assigned via closest squared Euclidean distance. In an embodiment, trained classification networks can be used to assign pixels directly to a (non-)alteration class.

[0161] As a summary, using one of the methods described above, each pixel can be assigned one or more of the following descripting features and / or labels:

[0162] Spectral signature;

[0163] - Background versus tissue and / or tissue type label;

[0164] Generic or specific labels relating to alteration, non-alteration and / or alteration type; for example, alteration area yes / no and / or specific class label relating to a specific alteration type; TWCV (the distance to the centroid) in relation to adequate TWCV threshold for accurate assignment, alternatively probability for accuracy of pixel labeling in case neuronal networks are used;

[0165] - False colour for spatial representation of alteration vs non-alteration areas, as well as discrimination of tissue subparts (leaves, stems) if required. In an embodiment, user may define which tissue features shall be represented and / or which colour shall be used.

[0166] Typically, level of labeling may be defined by user in methods settings.

[0167] In an embodiment, user may define if segmentation by tissue type is needed bevor the method is run and / or define tissue type of interest.

[0168] In an embodiment, each cluster, that is a computed group of pixels with similar signatures, can be assigned one or more of the following descripting features:

[0169] Spectral signature of the cluster usually visualized as the cluster centroid;

[0170] - Number of pixels assigned to the cluster, i. e. tissue or tissue type of interest, alteration area and / or non-alteration area;

[0171] - Label for cluster type, e.g. tissue, tissue type, non-alteration and / or alteration type;

[0172] - False color for representation (color-code by clusters).

[0173] TWCV or probability for accuracy of pixel labeling are together referred to as “validity score”.

[0174] The person skilled in the art will appreciate that a validity score can be computed for each machine assigned labeling.

[0175] In an embodiment, computed validity score can be used to disregard pixels not fulfilling the minimum distance to the cluster centroid or which probability for assignment accuracy is below the defined threshold.

[0176] In an embodiment, user can set a threshold for acceptable accuracy / validity score for labeling.

[0177] In an embodiment, computed validity score can be used to select pixels for revision of labeling by user. The method can be set to allow user reviewing the computed labeling in case validity score is outside defined acceptable range. In an embodiment, computed validity score can be used for similarity metrics, to identify pixels with unusual signature and / or of unusual alteration type.

[0178] In an embodiment, wherein used classification models are neural networks or dimension reduction methods (PCA, UMAP, t-SNE), a latent space for the pixel labeling can be computed. In case supervised models are used, labels are already known, so representation can be computed showing latent phase areas in false colors. In cases unsupervised models are used, preliminary labels can be automatically assigned by density-based clustering methods (e.g. DBSCAN). A representation of a computed latent space showing latent phase areas in false colors can be computed based on preliminary labels. It is preferred that centroid is computed for each latent phase area (Fig. 6a, 6b).

[0179] In an embodiment, a representation of the latent space can be used for validity check of the pixel classification and / or similarity metrics to identify pixel(s) with unusual signature based on the position of pixel in the latent space.

[0180] In an embodiment, a pixel or a group of pixels with unusual signature and / or which validity score for the computed classification is outside the defined acceptable range can be automatically labeled as unknown alteration.

[0181] In an embodiment, the representation of the latent space grouped in latent phase areas based on alteration label displayed with respective false color can be used for validity check of the pixel classification based on the position of the pixel in the latent space. Additionally or alternatively, it can be highly valuable for identifying new / unusual spectral signatures corresponding to unknown alteration(s) (also referred to as novelty detection).

[0182] In an embodiment the method is configured to allow the user labeling pixels or group of pixels as unknown alteration based on their position in the latent space.

[0183] In an embodiment, pixels not fulfilling the validity score or pixels labeled as “unknow areas”, can be displayed to the user for expert analysis, for example for novelty detection, allowing to identify new spectral signatures for alteration identification.

[0184] In an embodiment, the method can be configured to allow the user, based on the position of pixel(s) or group(s) within the latent space one or more of:

[0185] Judging on how well alteration and non-alteration are separated;

[0186] Disregarding or selecting pixel(s) or pixels group(s) for further analysis; Labeling pixels as unknown alteration and / or amending labeling manually based on similarity of spectral signature.

[0187] In an embodiment, the method can further comprise comparing the computed latent space with computed latent space for reference data, to identify new / unknown alteration types by way of similarity metrics.

[0188] In an embodiment, one or more representations of the (non-)alteration areas can be generated based on one or more of the described features.

[0189] In an embodiment, for each image one or more of the following describing features can be computed:

[0190] - Ratio of the number of pixels for the alteration and / or non-alteration area to the whole number of pixels attributed to the biological tissue or a tissue type;

[0191] Average validity score of pixel labeling.

[0192] In an embodiment, an averaged validity score and / or a distribution curve of the validity score can be computed for all pixels assigned to an alteration or non-alteration area. In an embodiment, the system is implemented to allow the user based on averaged probability and / or a distribution curve to set a predefined threshold for accurate labeling. In case neuronal networks are used, the user may be proposed to retrain used neuronal networks.

[0193] In an embodiment, the computed ratio of the number of pixels for the alteration area to the whole number of pixels attributed to the biological tissue or a tissue type can be represented in form of a graph, e. g. stacked bars.

[0194] Alternatively, or additionally, a representation of the spectral signature of the cluster centroid for the alteration / non-alteration areas can be computed and provided.

[0195] The system can be configured for causing display of the one or more representation of the (non-)alteration areas.

[0196] In an embodiment, the user can customize display for example, by way of selecting between one or more of the false colors for selective display of corresponding annotated cluster areas, expand regions of selected false color for better view, switch between spatial representation of the tissue in false color, graph view, and / or spectral signature view or a combination thereof. In an embodiment, the method further comprises:

[0197] Computing for the pixel(s) or pixels group(s) selected for further analysis based on their position in the latent space, a spatial representation with tissue in false color, graph representation of the computed ratios, and / or the average spectral signature for the pixels assigned to the unknow area;

[0198] Presenting computed representation(s) to the user for labeling of pixels attributed to unknown alteration;

[0199] Confirming unsupervised classification and / or labeling single pixel(s) and / or pixels of a latent space area as unknown alteration.

[0200] Examples of phenotype alteration of a plant can be herbicide or more generally agrochemicals induced damage, degree of fungal infection or other plant disease, active ingredient changes (metabolism), water stress, pest induced damages or other physical plant stress such as sun bum or a combination thereof without being limited to it. Alterations can also be related to aging, changes related to concentration of an active ingredient, and / or resistances against active ingredient, or the like as well as combination thereof. Unknown alteration areas may relate to a new mode of action.

[0201] For example, the solution of the invention has been used to quantify herbicide damage, degree of fungal infection, resistances, active ingredient concentration measurements and water stress.

[0202] In an embodiment, the solution allows quantifying one or more of identified alterations.

[0203] In an embodiment, the ratios described above can be used for quantification of the tissue alterations in an image acquired at an instant ti. In an embodiment, the processing unit can be configured to utilize this determined measure of alteration in the biological tissue at an instant ti and a calibration for the biological tissue of record to determine at least one absolute quantitative value of alteration for the biological tissue. In an example, the calibration may comprise a plurality of quantitative values for several type of alterations as a function of plurality of pixel fractions.

[0204] In an embodiment quantification can be achieved by way of comparison with reference image data. In an embodiment, reference image data from a storage comprising reference image data can be used for comparison with a new image; said reference image data comprising, for each reference image, data on cluster areas, spectral signature and number of pixels or pixel fraction thereof.

[0205] Quantification of alteration can be achieved by way of:

[0206] Retrieving reference image data from a storage for reference images;

[0207] Assigning clusters to each pixel either using Euclidean distance with reference cluster centroids or classification networks;

[0208] Calculating a pixel fraction of each alteration and non-alteration area (together referred to as cluster area) as a ratio the number of pixels for the cluster area to the whole number of pixels attributed to the biological tissue or tissue segment, (also called cluster fraction);

[0209] In an embodiment, quantification may comprise:

[0210] For each cluster area, comparing the number of pixels or the pixel ratios / fractions mentioned above at an instant ti with reference data of the biological tissue showing similar spectral signature;

[0211] Calculating a measure of the variation of the number of pixel or of the same pixel fractions between the two images (over time or towards reference data of record);

[0212] Optionally, selecting one or more cluster area showing change of pixel ratio and / or change of spectral signature over time and annotating these cluster areas as alteration cluster area.

[0213] In an embodiment, the solution allows classifying of the alteration per alteration type as described below.

[0214] In an embodiment, comparing the number of pixels or the pixel fractions mentioned above at an instant ti with a biological tissue in reference image series data showing similar spectral signature can be supported by recurrent neuronal networks, to identify known or new temporal patterns in time series data.

[0215] The term “recurrent neural network (RNN)” refers to a class of artificial neural networks where connections between nodes can create a cycle, allowing output from some nodes to affect subsequent input to the same nodes. This allows it to exhibit temporal dynamic behavior.

[0216] In an embodiment the cluster areas can be represented as pixel fractions in graph form, for example as a stacked bar graph, wherein each cluster area (means alteration and nonalteration areas) is represented in percent, such that all cluster areas add up to 1 (100%) (Fig. 7). Such representation allows quick identification of ongoing alterations by variance analysis of time series data, in particular time series of pixel fractions.

[0217] In an embodiment, graph patterns were found to help distinguish experimental groups of alterations.

[0218] In an embodiment, significant alterations represented by respective pixel fractions can be analyzed by performing student t-tests for two groups or ANOVA for time-course experiments or TWO-WAY-ANOVA for treatment and time experiments, or any other method for statistical analysis of significancy of alteration.

[0219] Additionally, or alternatively, pixel fractions over time can be used to train a recurrent neural network, allowing to get the overall accuracy, based on computed probabilities for accuracy of pixel labeling for the pixels assigned to the alteration area at stake from the recurrent neural network, in separating alteration for each given day. Fig. 8 shows an example of variation of accuracy of an alteration classification over time represented in a line graph for two imaged plants using a recurrent neural network.

[0220] In an embodiment, the pixel fractions can be used as feature for recurrent neuronal networks to classify the images by alteration type based on their similarity in temporal changes in cluster percentages towards a reference time series. In other words, a neural network comprising output neuron(s) providing an information on experimental group of alteration (also referred to as alteration type) can be used, while the input neurons receive pixel fraction data (and optionally additional data) as input values. Usually, one input neuron is used for each pixel fraction of the cluster areas. There can be additional input neurons for additional data.

[0221] In an embodiment, alteration type can be identified by comparison with reference data retrieved from a repository for reference data. For example, average spectral signature for a cluster (centroid) can be compare with a reference centroid of record specific for the biological tissue or a reference centroid for known alteration of the biological tissue at stake. In an embodiment, similarity of cluster centroids towards reference cluster centroids can be quantified as a measure of Euclidean distance between the respective vectors.

[0222] In an embodiment, classification networks from a repository can be used to classify the type of alteration.

[0223] In an embodiment, latent space distributions within a neuronal network can be used to describe similarity between tissue / cluster at stake and reference data, based on their spatial location within the latent space. Pretrained classification models or autoencoders / variational autoencoders / GAN networks can be used which learn a strong representation by reconstruction spectral data. Comparing latent space distributions allows qualitative evaluation of the difference between an image towards a reference image or an image at instant ti.

[0224] Additionally, a density base clustering can be performed on the activations of the latent space of a complete dataset to cluster areas of non-altered and altered tissue.

[0225] Comparison with reference data can be used for identification of the mode of action (MoA) of an alteration (similarity metrics), for example attributing alteration to a fungi infection, and / or quantify how strong such an infection may be.

[0226] In case new spectral signatures are detected with unsupervised models, negative result of such comparison with reference data may point out new mode of actions.

[0227] In case unknown alterations are investigated, spatial representation of the tissue in false color, graph representation of the computed ratios, and / or the average spectral signature for the pixels assigned to the alteration area can be of particular interest. They may be stored in a repository for later investigation.

[0228] In an embodiment, labeling of such novel alterations can be achieved by experts looking at the spatial distributions of alterations or molecular assays to identify molecular targets and thereby identifying new mode of actions. Comparison can be conducted by the user via a user interface or be obtained by machine comparison with a library of existing data (spectral reference library). For such a machine comparison best-matching algorithm as established in the art can be used. In an embodiment, a machine-supported comparison may be implemented, and the method may be causing display of the result of the comparison for user interpretation. In an embodiment, the apparatus is configured to provide information on alteration over time. Information can be provided in form of a sequence of spatial representations, a sequence of (bar) graphs, at least one difference curve or an averaged difference curve determined from difference curves originating from different clusters.

[0229] Recognition of the type of alteration is particularly useful for herbicide, fungicide, and insecticide screening, for plant breeding, determining active ingredient concentration and / or resistance experiments, for disease / recovery detection also in pharmacological tissue sections.

[0230] In an embodiment, a RGB false-color spatial representation of the object of interest can be computed and displayed on a user interface, wherein such spatial representation visualizes spatial occurrence of the alterations within the tissue or tissue subparts. This spatial form of display (also called spatial display or false color image) supports interpretation on how a certain condition affects the physiology of the biological tissue at specific location. For example, in herbicide screening not only the degree (cluster percent) of a specific alteration is important but also the spatial occurrence of the alterations, since this latter often relates to different symptoms induced by chemicals. One example is a classical burning of a plant, which is usually widely distributed versus a chlorosis, wherein newly developed leaves are impacted first.

[0231] Alternatively or additionally, a representation in graph for, e.g. stack bar graph, optionally with markings of descriptive clusters for alteration (for example as shown in Fig. 6, 7) or a curve showing alterations of the pixel fractions over time can be computed and displayed.

[0232] While patterns of (bar) graphs help were found to help distinguish experimental groups (alteration type), also the spatial information in which these changes occur on the plant over time are valuable for the biologist to better understand the alteration for example the mode of action of a chemical or the propagation of a disease. In an example the user can expand the cluster areas in graph form into the spatial display as needed for convenient interpretation (Fig 6).

[0233] In a further example values or ratios of values of the spectral signature for the pixels of a cluster area can be used as indices for describing alterations in biological tissue. Time series of these can be plotted as curves, displaying degree of alteration, allowing to study kinetic behavior of treatments / alteration. This can be used in compound screening to identify only compound with a favorable kinetic. In an embodiment, fitting machine learning or deep learning models, such as recurrent neuronal networks, for these curves can be generated, to make predictions of alterations based on only the first days of measurements and can even be used to make forecasts for alteration progression. These models can also be saved in a reference model library for later use.

[0234] In an embodiment, the computed false color images can be fed into another deep learning model to identify spatial patterns of alteration in addition to cluster fractions and evolution thereof. Such deep learning model(s) can be:

[0235] - Pretrained classification networks (such as googlenet, alexnet etc.) for feature extraction from an image data set and using the extracted features for unsupervised clustering using an unsupervised clustering algorithm, e. g. k.means; and / or Autoencoders, variational autoencoders or GAN networks used to reconstruct images, thereby learning strong feature representation of false color images. These features can be extracted from the latent space and again clustered in an unsupervised manner using density -based clustering methods such as DBCSAN.

[0236] For this purpose, each computed false color image is compressed into a n-dimensional (n>l) space, in which features represent essential parts from the image containing information selected from spatial distributions, form, color, size and / or alteration.

[0237] In an embodiment, spatial progression of an alteration can be studied / computed over time based on the spatial patterns of alteration within time series of multi wavelength images.

[0238] In an embodiment, alterations over time can be described by at least one pixel fraction showing change in comparison to a reference (or over time) used as quantitative values, later referred to as “descriptive cluster for alteration”. In an embodiment, descriptive clusters for alteration can be stored in a repository for cluster alteration.

[0239] In an embodiment, the method may be used to detect resistant plants in fields as early as day two (with RGB usually it takes a minimum of ten days), so specialized treatment against resistant plants can be selected.

[0240] In an embodiment, the method can be used to detect alterations of active ingredients within the tissue, helping for example identifying the ideal harvesting time. In an embodiment, the method may cause user warning or activation of a device for plant treatment in case one or more descriptive clusters for alteration are identified. In an embodiment warning and / or activation of a plant treatment device may be related to quantification of alteration being over one or more predefined threshold.

[0241] In an embodiment, descriptive clusters for alteration, that is cluster showing change, can be selected and used for further analysis. Optionally, the method is causing display of the descriptive clusters for alteration with corresponding label.

[0242] In an embodiment, the method can be computing, for each condition (chemical treatment, resistance type, disease, damage) and for each instant ti, stacked bar graphs, wherein stacked bars can be grouped by similarity to identify unique time patterns in cluster distributions. These cluster distributions can be used for example to discriminate different alterations and / or alteration phases (as shown for example in Fig. 8).

[0243] In an embodiment, characteristic spectral signature for a cluster area can be computed and displayed. In an embodiment, characteristic spectral signature can be compared with a catalog of characteristic spectral signatures for example for weeds species and biotypes (resistant and non-resistant mutants). In the field this method is expected to significantly enhance weed biotype identification (resistant or non-resistant strain) after initial chemical treatment by utilizing a combinatory approach in which data from RGB and fluorescence measurements are taken into account as each weed species and biotype is expected to show specific hyperspectral adsorption fingerprint upon chemical treatment.

[0244] In an embodiment, the method can be conducted on at least one multi spectral / hyperspectral new image data at instant ti (time point) of a scene within which biological tissue is located.

[0245] In an embodiment, the method can be used on time series of new multi spectral / hyperspectral image data. It may be of interest to annotate said time point for example to assess the progress of the alteration and / or define alteration phases or gradual changes.

[0246] In an embodiment, the apparatus can be configured to establish a recording period (TA) such that, within said recording period (TA), an asymptotic limit AW is reached for the number of difference pixels for a scene.

[0247] In an embodiment, the apparatus is configured to acquire successive images at predefined constant time interval, for example every day. In an embodiment, properties and / or identifier of the biological tissue can be defined by user input, stored as an additional label / data to the images. The method may be implemented for using these additional labels for a search or for storage of new image data set in a spectral library for the biological tissue at stake.

[0248] As a matter of example, the identifier and / or identification of the plant may be provided as user input.

[0249] In an embodiment, the type of alteration can be defined by user input. Alternatively, the type of alteration may be computed according to the method(s) described above. In other words, user may select the type of alteration of interest, so the corresponding algorithm(s) can be selected from a repository and run. Alternatively, the user can instruct running the method, so one or more alteration areas are identified and label(s) for alteration type can be selected by user based on expertise.

[0250] In an example, a whole series of calibration curves or look up libraries / repositories can be stored and the correct one selected for the specific biological tissue.

[0251] In an embodiment, the processing unit can be configured to utilize an identification of the plant / tissue, or generally speaking information on plant / tissue properties, to select the relevant calibration(s) / reference data from a plurality of calibrations / reference data for a plurality of different plants / tissues. Plant / tissue identification and / or type of alteration can be introduced by user in order to select the relevant reference data for calculation of the quantitative values and / or relevant reference experiment(s) of record for supported interpretation of new results. Alternatively, identification of relevant reference data can be computer-implemented to proceed automatically by comparison of spectral information and / or cluster information within all or some reference directories selected by the user by way of one of the methods described above.

[0252] In summary, the computed spatial representation of the tissue in false color, the computed graph representation of the computed ratios, and / or the computed average spectral signature for the pixels assigned to the alteration area can be used for comparison with corresponding classified reference data from a repository, said classified reference data being representative of tissue type, tissue biotype, tissue phenotype, type of alteration, and / or alteration status at an instant ti. User input is typically conducted using standard peripherals.

[0253] Output typically occurs by causing display, print or storage in peripherals.

[0254] In computer technology, “peripherals” refer to all devices which are connected to the computer and serve for the control of the computer and / or as input and output devices. Examples thereof are monitor (screen), printer, scanner, mouse, keyboard, drives, camera, microphone, loudspeaker, etc. Internal ports and expansion cards are, too, considered to be peripherals in computer technology.

[0255] In an embodiment, the system can be implemented to allow user revising classification, for example before storing classified image in a repository. In an example, the system may allow qualifying a classified image or time series as reference image or reference time series for later use. Also, qualification for training or validation set for training of algorithms for the method described herein may be allowed. Such qualification can be stored as additional data for the classified image.

[0256] In an example multispectral camera(s) can be used instead of hyperspectral camera(s). For image acquisition the at least one camera detector is configured to acquire a multi wavelength image of the scene within which the at least part of the target tissue is located. The processing unit is configured to utilize the multi wavelength image for the method described above.

[0257] Use of a multispectral camera provides images with higher spatial resolution and thus better accuracy since more pixels are available for analysis in a shorter time, due a much lower image acquisition time. However, for use of multispectral cameras it is preferred to already have extensive knowledge of relevant wavelength(s). It is therefore most preferred to use information from the hyperspectral data to identify wavelength(s) best revealing alterations of the biological tissue at stake. Hyperspectral images can be analyzed for relevant wavelength and wavelength filters for multispectral camera(s) can be chosen accordingly. In an example the processing unit is configured to support selection of wavelengths of interest. This can be achieved by analyzing a pool of hyperspectral images of record for a target tissue, in particular images showing one or more type of alteration, extracting the descriptive clusters for the selected alterations, and outputting the characteristic wavelength(s) for these clusters. In an example, the apparatus is configured to acquire images with the same camera. An example for such an apparatus is shown in figure 3. At this point, it is pointed out that hyperspectral or multispectral images of a scene may be acquired in a device differing from the design of figure 3.

[0258] The apparatus may also be configured to acquire images with different camera detectors for example using a beam splitter approach, etc... The skilled person will understand that any apparatus able to acquire at least one hyperspectral image or at least one multispectral image at wavelengths of interest, containing at least part of the target tissue, with adequate resolution is usable for the purpose of the present invention. In case several detectors are used, the processing unit may be configured to combine several images into one image for analysis.

[0259] Owing to the multiplicity of measurements in a screening process, there was a need to provide a solution making possible to investigate the measurements on biological tissues in the shortest possible time.

[0260] The proposed solution solves the pitfalls mentioned above and allow massive speed, accuracy and robustness enhancement of model training time. The method of the invention was found to allow high accuracy segmentation of time series of image data. It was also found that high accuracy segmentation and robust correction for geometric influences of time series of image data increases data robustness for studying the physiological changes in target tissue over time by eliminating false positive and / or negative hits.

[0261] In an embodiment, the method can be used for pesticide, herbicide, and resistance trials. In conducted experiments the obtained time patterns uniquely allowed distinguishing between different biotypes (resistant and non-resistant strains) and assessing type and degree of resistance at time points where physical inspection did not reveal any difference.

[0262] In the field of screening active compounds comparison with alteration patterns of chemicals with known MoA can be used to cluster new compounds either into a group with a known mode of action or into an unknown group with potential new mode of action. Identifying known and new temporal alteration patterns are of special interest in chemical kinetics study. For pest and disease control, HSI can be used to map infection distribution and degree as well as classification of disease. Additionally, herbicide or disease or pest induced damages can be assessed much more accurately, since also changes not visible in RGB imaging can be quantified The methods described above can be conducted for a given biological tissue allowing reliable unsupervised computer-implemented clustering of alterations in tissue phenotype and related physiology over time.

[0263] A system comprising one or more processors is typically configured to process the computer-implemented methods.

[0264] A computer program or computer program element can be provided that is characterized by being configured to execute the method steps of the method according to one of the preceding embodiments, on an appropriate system. The computer program element might therefore be stored on a computer unit, which might also be part of an embodiment. This computing unit may be configured to perform or induce performing of the method described above. Moreover, it may be configured to operate the components of the above-described apparatus and / or system. The computing unit can be configured to operate automatically and / or to execute the orders of a user. A computer program may be loaded into a working memory of a data processor. The data processor may thus be equipped to carry out the method according to one of the preceding embodiments.

[0265] This exemplary embodiment of the invention covers both, a computer program that right from the beginning uses the invention and computer program that by means of an update turns an existing program into a program that uses invention.

[0266] Further on, the computer program element might be able to provide all necessary steps to fulfill the procedure of an exemplary embodiment of the method as described above.

[0267] A non-transient computer readable medium, such as a CD-ROM, USB stick or the like, is also presented wherein the computer readable medium has a computer program element stored on it which computer program element is described by the preceding section.

[0268] A computer program may be stored and / or distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the internet or other wired or wireless telecommunication systems. In other words, the computer program may also be presented over a network like the World Wide Web and can be downloaded into the working memory of a data processor from such a network. A further object of the invention is, therefore, a non-transitory computer readable medium having stored thereon instructions that, when executed by a processor of a computer system, cause the computer system to execute the method(s) mentioned above.

[0269] Another object of the invention is a computer system comprising:

[0270] A memory storing one or more instructions;

[0271] One or more repository for storing image data and / or trained models; and

[0272] One or more processors configured carrying out any one of the methods described above.

[0273] In an embodiment, the one or more processors can be configured for causing transmission of a notification message with or display or the determined result dataset to one or more peripherals.

[0274] A “computer system” is a system for electronic data processing that processes data by means of programmable calculation rules. Such a system usually comprises a “computer”, that unit which comprises a processor for carrying out logical operations, and also peripherals.

[0275] Computer systems of today are frequently divided into desktop PCs, portable PCs, laptops, notebooks, netbooks and tablet PCs and so-called handhelds (e.g. smartphone); all these systems can be utilized for carrying out the invention.

[0276] The system according to the invention can, for example, be configured as a computer (e.g. desktop computer, tablet computer, smartphone, server) or a combination of computers.

[0277] Generally, a computer system of exemplary implementations of the present disclosure may be referred to as a computer and may comprise, include, or be embodied in one or more fixed or portable electronic devices. The computer may include one or more of each of several components such as, for example, processing unit connected to a memory (e.g., storage device).

[0278] The processing unit may comprise one or more processors alone or in combination with one or more memories. The processing unit is generally any piece of computer hardware that is capable of processing information such as, for example, data, computer programs and / or other suitable electronic information. The processing unit is composed of a collection of electronic circuits some of which may be packaged as an integrated circuit or multiple interconnected integrated circuits (an integrated circuit at times more commonly referred to as a “chip”). The processing unit may be configured to execute computer programs, which may be stored onboard the processing unit or otherwise stored in the memory of the same or another computer.

[0279] In an embodiment, one or more repositories comprise calibration information, look up tables e. g. for reference pixel value ranges and / or reference data, trained models, and the like.

[0280] The processing unit may comprise one or more processors, a multi -core processor and / or some other type of processor, depending on the specific implementation. Further, the processing unit may be implemented using several heterogeneous processor systems in which a main processor is present with one or more secondary processors on a single chip. As another illustrative example, the processing unit may be a symmetric multi-processor system containing multiple processors of the same type. In yet another example, the processing unit may be embodied as or otherwise include one or more ASICs, FPGAs or the like. Thus, although the processing unit may be capable of executing a computer program to perform one or more functions, the processing unit of various examples may be capable of performing one or more functions without the aid of a computer program. In either instance, the processing unit may be appropriately programmed to perform functions or operations according to example implementations of the present disclosure.

[0281] The memory is generally any piece of computer hardware that can store information such as, for example, data, computer programs (e.g., computer-readable program code) and / or other suitable information either on a temporary basis and / or a permanent basis. The memory may include volatile and / or non-volatile memory and may be fixed or removable. Examples of suitable memory include random access memory (RAM), read-only memory (ROM), a hard drive, a flash memory, a thumb drive, a removable computer diskette, an optical disk, a magnetic tape or some combination of the above. Optical disks may include compact disk - read only memory (CD-ROM), compact disk - read / write (CD-R / W), DVD, Blu-ray disk or the like. In various instances, the memory may be referred to as a computer-readable storage medium. The computer-readable storage medium is a non-transitory device capable of storing information and is distinguishable from computer-readable transmission media such as electronic transitory signals capable of carrying information from one location to another. Computer-readable medium as described herein may generally refer to a computer- readable storage medium or computer-readable transmission medium.

[0282] In addition to the memory, the processing unit may also be connected to one or more interfaces for displaying, transmitting and / or receiving information. The interfaces may include one or more communications interfaces and / or one or more user interfaces. The communications interface(s) may be configured to transmit and / or receive information, such as to and / or from other computer(s), network(s), database(s) or the like. The communications interface may be configured to transmit and / or receive information by physical (wired) and / or wireless communications links. The communications interface(s) may include interface(s) to connect to a network, such as using technologies such as cellular telephone, Wi-Fi, satellite, cable, digital subscriber line (DSL), fiber optics and the like. In some examples, the communications interface(s) may include one or more short-range communications interfaces configured to connect devices using short-range communications technologies such as NFC, RFID, Bluetooth, Bluetooth LE, ZigBee, infrared (e.g., IrDA) or the like.

[0283] The user interfaces may include a display. The display may be configured to present or otherwise display information to a user, suitable examples of which include a liquid crystal display (LCD), light-emitting diode display (LED), plasma display panel (PDP) or the like. The user input interface(s) may be wired or wireless and may be configured to receive information from a user into the computer system, such as for processing, storage and / or display. Suitable examples of user input interfaces include a microphone, image or video capture device, keyboard or keypad, joystick, touch-sensitive surface (separate from or integrated into a touchscreen) or the like. In some examples, the user interfaces may include automatic identification and data capture (AIDC) technology for machine-readable information. This may include barcode, radio frequency identification (RFID), magnetic stripes, optical character recognition (OCR), integrated circuit card (ICC), and the like. The user interfaces may further include one or more interfaces for communicating with peripherals such as printers and the like.

[0284] As indicated above, program code instructions may be stored in memory, and executed by processing unit that is thereby programmed, to implement functions of the systems, subsystems, tools, and their respective elements described herein. As will be appreciated, any suitable program code instructions may be loaded onto a computer or other programmable apparatus from a computer-readable storage medium to produce a particular machine, such that the machine becomes a means for implementing the functions specified herein. These program code instructions may also be stored in a computer-readable storage medium that can direct a computer, processing unit or other programmable apparatus to function in a particular manner to thereby generate a particular machine or article of manufacture. The instructions stored in the computer-readable storage medium may produce an article of manufacture, where the article of manufacture becomes a means for implementing functions described herein. The program code instructions may be retrieved from a computer-readable storage medium and loaded into a computer, processing unit or other programmable apparatus to configure the computer, processing unit or other programmable apparatus to execute operations to be performed on or by the computer, processing unit or other programmable apparatus.

[0285] Retrieval, loading and execution of the program code instructions may be performed sequentially such that one instruction is retrieved, loaded, and executed at a time. In some example implementations, retrieval, loading and / or execution may be performed in parallel such that multiple instructions are retrieved, loaded, and / or executed together. Execution of the program code instructions may produce a computer-implemented process such that the instructions executed by the computer, processing circuitry or other programmable apparatus provide operations for implementing functions described herein.

[0286] Execution of instructions by processing unit, or storage of instructions in a computer- readable storage medium, supports combinations of operations for performing the specified functions. In this manner, a computer system may include processing unit and a computer- readable storage medium, or memory coupled to the processing circuitry, where the processing circuitry is configured to execute computer-readable program code stored in the memory. It will also be understood that one or more functions, and combinations of functions, may be implemented by special purpose hardware-based computer systems and / or processing circuitry which perform the specified functions, or combinations of special purpose hardware and program code instructions.

[0287] A unique advantage of the solution of the invention over the standard approach is the detection of alterations, for example adverse effects of herbicides / resistances / diseases / pests, at a much earlier stage, even before effects are visible with the naked eye and beyond effects visible with the naked eye (plant necrosis). Eye-visible damages occur much later than the underlying physiological changes in a plant. Thus, detecting the early changes in physiology ultimately leading to visible effects, substantially reduces the required screening time, the required space and ultimately, the costs of screening activities. In resistance trials, a mobile setup can be used to check weed (herbicides) or crop plants (fungicides / insecticides) in realtime in the field right after treatment to check whether plants or diseases / pests show resistance against the applied product. This enables farmers to respond much earlier as today to treatment failures and substantially reduce yield losses and prevent crop failures, respectively, thus significantly enhance productivity and, finally, income of farming. In disease detections, this approach can spot abnormal alterations in tissue, without the need of antibody staining, thus reducing time and cost and increase the time window for corrective measures.

[0288] The person skilled in the art will appreciate, that the solution of the invention may be implemented for pharmacological purposes: for example, healthy vs. abnormal tissue can be distinguished in a computer-supported manner.

[0289] In a further example, the method of the present invention may be used for image-based assessment of coating thickness on bulk material such as pills or seeds.

[0290] In a further example, the method of the presented invention may be used for image-based assessment of active ingredient concentrations within in tissue.

[0291] As already mentioned, the solution was developed for identifying alterations of plant tissue using at least one hyperspectral or multispectral image of plants. Such a particular embodiment is explained in somewhat greater detail below, but without any intention of restricting the invention to this embodiment.

[0292] BRIEF DESCRIPTION OF THE DRAWINGS

[0293] Exemplary embodiments will be described in the following with reference to the following drawings:

[0294] Fig. 1 shows a schematic representation of the principle of HSI-acquisition method as known in the art [Mishra, P., Asaari, M. S. M., Herrero-Langreo, A., Lohumi, S., Diezma, B., & Scheunders, P. 2017. Close Range hyperspectral imaging of plants: A review. Biosystems Engineering. 2017.] ;

[0295] Fig. 2 shows a vegetation HSI-spectrum as described in Elowitz, Mark R. “What is Imaging Spectroscopy (Hyperspectral Imaging)?”, School of Physical Sciences Open University.

[0296] Fig. 3 shows a schematic example of an apparatus for determining alterations of biological tissue in plants using multispectral imaging;

[0297] Fig. 4a to 4e show the steps related to assigning image pixels, in particular rim pixels to tissue or background for generating training masks Fig. 5a to 5d: shows clustering of pixel spectra and representation as false color heat maps of alteration area (treated) and non-alteration areas (untreated) of the plant tissue.

[0298] Fig. 6a, 6b shows a representation of a latent space obtained by unsupervised classification of tissue pixels using a variational autoencoder network.

[0299] Fig. 7 : shows condensation of cluster areas in a stacked bar graph for a given tissue at a given time, as well as spatial distribution of the corresponding clusters. For each cluster also the corresponding RGB image is shown for user information

[0300] Fig. 8 shows patterns of stack bar graphs obtained from image analysis of plants after chemical treatment over time.

[0301] Fig. 9 shows accuracy of categorization per type of resistance over time (metabolic, target, or no resistance) wherein ALOMY / LOLRI are different types of grass.

[0302] Fig. 10 shows clustering quality for different approaches on flattened and 3D tissue applied to ivy leaves.

[0303] Figs. I la, 11b show an example of use of flattened and 3D tissue for fungi detection on wheat leaves.

[0304] Fig 12a shows an example of funghi detection on soy leaves over time. Percentage areas of mild and strong infection are plotted in percentage in Fig 12b and Figl2c. Figl2d shows the average distance of infected areas from the leaf rim, indicating that the fungus moves from leaf rims towards the center of the leaf.

[0305] Fig 13 shows a schematic diagram of the system of the invention.

[0306] Fig. 14 show a flow chart of exemplary embodiments of the methods according to the invention for the analysis of new images or image series.

[0307] Fig. 15 shows a flow chart of exemplary embodiments of the method for generating segmentation masks and subsequent training one or more deep learning algorithms for segmentation of biological tissue(s) from background with the segmentation masks.

[0308] Fig. 16 shows a flow chart of exemplary embodiments of the method for training a deep learning network for semantic image classification of tissue region(s) to an alteration and / or non-alteration area. Fig. 17 shows a flow chart of exemplary embodiments of the method for training a deep learning network for pixel classification of tissue region(s) to an alteration and / or nonalteration area

[0309] References:

[0310] System 10

[0311] Camera detector 20

[0312] Wavelength filters 30

[0313] Processing unit 40

[0314] Memory storing one or more instructions for:

[0315] - controlling acquisition of images 41

[0316] - storing of images 42

[0317] - segmenting images 43

[0318] - normalizing of images 44

[0319] - clustering of images 45

[0320] - computing the spectral signature of the biological tissue 46

[0321] - identifying alteration segments 50

[0322] - for identification of plant 52

[0323] - memory for reference information 51 one or more processors configured to execute the one or more instructions (not shown)

[0324] User interface 55 (not shown)

[0325] Light source 60

[0326] Splitter 61 (not shown)

[0327] Background pixel 90

[0328] Plant pixel 91

[0329] Rim / edge pixel 92

[0330] PARTICULAR EMBODIMENT OF THE INVENTION - SOLUTION FOR PLANT STUDY

[0331] Fig. 1 shows the principle of HSI acquisition as described in the art HSI is a combination of spectroscopy and conventional imaging. HIS or multispectral image techniques acquire one spatial image for each wavelength. These images are stacked analog to a RGB image and each pixel than has its own spectra. Fig. 2: shows an example of spectra of a plant from 400 up to 2500nm. The selected range allows studying of plant physiology, since pigments and water content can be correlated to the spectra.

[0332] Fig. 3 shows an example of an apparatus 10 for determining alterations of biological tissue. The apparatus comprises at least one hyperspectral or multispectral camera detector 20 (using spatial scanning, spectral scanning, snapshot or spatio-spectral scanning as described by Lu et al. for hyperspectral / multispectral image acquisition [Lu, G., & Fei, B. 2014. Medical hyperspectral imaging: a review. J Biomed Opt. 2014.], at least one light source 60, at least one wavelength filter 30, and a processing unit 40. The at least one camera detector 20 is configured to acquire at least one hyperspectral / multispectral image of a scene within which at least part of a plant is located. The at least one camera detector 20 is configured to acquire images at wavelengths of interest in a push broom, spectral scanning, snapshot or spatio-spectral scanning approach. At least one hyperspectral or at least one multispectral image is acquired.

[0333] In this example, the apparatus is configured to capture images as top view images.

[0334] In an example, the camera detector 20 is a hyperspectral camera for the acquisition of hyperspectral images with the following features: Line scanning or snapshot to acquire the entire scene with the wavelength spectrum which can range from UV (250nm) to far red (4000nm), with a bandwidth of Inm. For example, a headwall camera with the spectral range from 380nm to lOOOnm and a spatial resolution of 1600 pixel, was used.

[0335] In a further example the camera detector 20 is a multispectral camera for the acquisition of hyperspectral images comprising multiple wavelength filters 30 for relevant wavelength (this information is obtained usually for hyperspectral images) to study relevant alterations within tissue.

[0336] The light sources 60 can be an arrangement of halogen lamps arranged in arrays or LEDS with a continues wavelength spectrum. An arrangement of 3x3 halogens lamps was used. The light sources can illuminate the plant in a continuous manner.

[0337] The system may comprise one or more beam splitter 61.

[0338] Acquired images are transmitted to the processing unit 40 configured for the performance of the method for analysis of new images. The same or a separate processing unit may be configured for the performance of the method for generating training masks from a training image set, and / or of the trainings of the one or more algorithms for image analyses mentioned above.

[0339] For analysis of three-dimensional plant images, it was shown to be most preferred to account for illumination and geometry of the acquired images:

[0340] In an example, images were acquired, the processing unit 40 was configured to normalize acquired multi wavelength images utilizing a white reference and a measurement of the dark current. Normalized images were used for further procession. Such normalization improves reproducibility of image analysis by cleaning environmental imaging variations related to illumination effects and / or detector sensitivity. Standard methods for normalization can be used.

[0341] In an embodiment, the image was normalized using the following formula (I) according to the method of Mishra et al. [Mishra, P., Schmuck, M., Roth, S., Nicol, A., Nordon, A., 2019. HOMOGENISING AND SEGMENTING HYPERSPECTRAL IMAGES OF PLANTS AND TESTING CHEMICALS IN A HIGH-THROUGHPUT PLANT PHENOTYPING SETUP . IEEE. 2019.]:

[0342] Wherein Icis Intensity of calibrated image

[0343] Irawis intensity of acquired image lWRis intensity of white reference

[0344] IDCis intensity of dark current

[0345] Normalization of illumination resulted in double precision values leading to an increase in data size to 4.7GB

[0346] Normalized images were used for further processing.

[0347] Reflectance measurements, such as HSI, were found to be highly dependent on the distance and object geometry.

[0348] For example leaf / tissue reflectance p is given by : (II)

[0349] Wherein pmeas is the measured leaf / tissue reflectance pref is the leaf reflectance at a reference

[0350] P is a multiplicative term that models the illumination effects due to leaf distance from the sensor and inclination towards the incident light. a is a scalar constant is the wavelength as described by Mohad et al. [Mohad Asaari, M. S., Mishra, P., Mertens, S., Dhondt, S., Inze, D., Wuyts, N., & Scheunders, P. 2018. Close-range hyperspectral image analysis for the early detection of stress responses in individual plants in a high-thoughput phenotyping plarform. ISPRS Journal of Photogrammetry and Remote Sensing. 2018.].

[0351] First, computer-implemented segmentation of the at least part of the object (e.g. plant) from background in the hyperspectral / multispectral image was considered. For this purpose, a deep learning algorithm for segmentation of biological tissue(s) from background trained with a training data set comprising segmentation masks was selected.

[0352] Fig. 4a-e shows the process steps relating to assigning image pixels, in particular rim pixels to tissue or background for generating training masks. The method performs unsupervised clustering of pixels. The different clusters are shown to the user as RGB representations, wherein Fig. 4a shows pixel clearly attributed to the plant (91) or to the background (90), when generating new masks the user is asked to choose to which category the cluster belongs: biological tissue, background or mixed cluster. Identified tissue pixels can be used to build a training mask for the biological tissue (Fig. 4 e) for later used in training. Fig. 4b shows a mixed cluster in rim pixels 92, means a group identified to comprise a mixture of tissue and background pixels; assignment of rim pixels 92 is most challenging. Mixed clusters were resubmitted to unsupervised clustering and caused to be classified by the user until a clear separation was achieved (Fig. 4c and 4d), that is all pixels were assigned either to tissue of background. Each pixel attributed to the biological tissue was added to the training mask for the image and related to the target tissue. This procedure allowed a much higher accuracy of the training masks, since edge / rim pixels (pixels at the crossing from target tissue from background), can be much more reliable eliminated compared to manual drawing of masks.

[0353] Images and respective trainings masks were used to build a training data set for training of a deep learning algorithm for segmentation of biological tissue(s) from background with segmentation masks. In the example, a neural network for semantic image segmentation was used. Neural network was trained using a method of the art using a training algorithm and training data set obtained as described above.

[0354] Only pixels attributed to the target tissue (e.g. plant) in multiple images covering all experimental conditions and timepoints were used for assigning to alteration or nonalteration areas.

[0355] For this purpose, classification deep learning networks were used as algorithms for pixel classification tissue region(s) showing an alteration and / or showing no alteration; they were trained with a training data set of classified segmented images enhanced using:

[0356] - spectral data augmentation to mimic geometric influences, and / or

[0357] - multi wavelength images of the tissue at stake in a flattened form wherein the regions of the tissue showing spectral change due to induced alterations are annotated.

[0358] In an example hyperspectral images of a complete ivy rank, the separated leaves and leaves flattened with nylon threads were acquired (Fig. 10, column A).

[0359] Unsupervised clustering of flattened leave tissue was performed, using classic k-means clustering on spectra without geometrically correction (column B, fig. 10). Column B shows the result of the same correction method on natural leaves, with or without vein, showing k means clustering on raw spectra only captures variance in illumination and height. Column C shows SNV corrected images alongside t-SNE (t-Distributed Stochastic Neighbor Embedding) in combination with DBSCAN on spectra. Identified clusters were further condensed by combining clusters with similar shape but different intensities. Resulting clusters were used to train a classification network. Column D shows the clustering of an image using a CNN for cluster assignment trained without augmentation. Column D and E shows the clustering of an image using a CNN for cluster assignment trained with a training set of images acquired with augmentation or strong augmentation respectively to simulate geometric influences such as height and angles. For augmentation, spectra were altered by adding additive term to the acquired spectra of a training data set to simulate geometric influences such as height and angles. Upon visual inspection of the clusters identified by the classification network the false color images of Fig.10, column E and F show that the stems, leaf veins and intermediate leaf tissue were nicely separated, and that clustering makes biological sense, while k means clustering on spectra only captured variance in illumination and height. While SNV was found to be capable of reducing illumination effects, still no clear separation of sub-tissue parts was possible. The new approach allowed a reliable separation of leaf and stem for flattened up to full vein and using augmentation also allows to segment leaf veins. Although identifying veins in leaves is not an “alteration” in the sense of the present application, this experiment nicely shows the scope of use for the used solution for correction of geometric influences in image analysis.

[0360] To further validate this approach Septoria fungi infected wheat plants Fig. l la-b were compared in flattened and native form for the cluster area percentages and location.

[0361] Fig. I la shows an example of strong fungi detection on wheat leaves. Plant spatial representations in false color are shown, wherein fungal infection is overlayed in white in both flattened (left plant image) and non-flattened (right plant image). The spectra shown on the curves are cluster centroids of uninfected plant tissue (dotted) and infected tissue (solid line). Fig 1 lb shows the results for weak infection.

[0362] In both cases, it was possible to identify the same cluster areas within non flattened plants, showing that the geometric correction improves identification of tissue alterations in-non flattened objects. This reliable identification of cluster areas in intact non-flattened plant is important for not damaging plants though flattening and for outdoor field applications in which flattening is not feasible.

[0363] In summary, classical normalization techniques did not allow an accurate separation of leave organs (stem, veins etc.). The training method of the invention provided a trained model highly robust against geometric influences. It was shown able to identify alterations at the same precision on flat tissue as based on 2-dimensional images of 3-dimensional full plants.

[0364] Fig. 5a to 5d show clustering of pixel spectra and representation as false color heat maps of alteration area (treated) and non-alteration areas (untreated) of the plant tissue. In this experiment, 45 plants were used for unsupervised spectral clustering of images. Fig. 5a shows the training mask for the segmentation of the biological tissue. . Fig. 5b and 5c show how each spectral signature of the 2.5 Mio. target tissue pixels are further clustered by similarity of spectral signature and assigned to alteration / non-alteration areas (together referred to as clusters). Clusters are visualized as a false colour representation in Fig. 5c and in the spatial representation Fig. 5d of the target tissue next to RGB representation for the user to correlate cluster patterns to biological responses of the target tissue, wherein “treated” refers to alteration areas of plant tissue resulting from agrochemical treatment, “untreated” refers to non-alteration areas of plant tissue.

[0365] Fig. 6a shows a representation of a latent space obtained by classification of tissues pixels using a classification network, the representation of the latent space shows pixels grouped in latent phase areas based on (preliminary) alteration label, displayed with respective false color (left). It is preferred that centroid is computed for each pixel group, so user can easily evaluate similarity of respective centroids visually (right). The system is configured to allow the user:

[0366] Selecting, comparing in the visualization and combining displayed latent space areas or splitting latent space areas to improve classification based on similarity of spectral signature;

[0367] Disregarding or selecting pixel(s) or pixels group(s) for further analysis based on their position in the latent space; and / or

[0368] Classifying pixels as unknown alteration based on the position of pixel within the latent space.

[0369] Fig. 6b shows an excerpt of the representation of Fig. 6a, wherein two latent phase areas (left) are selected for comparison of centroid spectral signature (right).

[0370] In an example, areas attributed to known alterations can be combined (large area) to isolate latent space area attributed to unknown alteration for further analysis.

[0371] Fig. 7: shows pixels fractions in stacked bar graph for a given tissue at a given time after chemical treatment, as well as spatial distribution of the corresponding clusters. For each cluster also the corresponding RGB image is shown for user information. For spatial interpretation of the alterations the cluster areas can be output as spatial RGB representation to get hints on how a certain condition affects the physiology of the biological tissue at specific location. User may choose a false color representation alongside the corresponding RGB view (also called spatial view), in which spatial distribution of all cluster areas are shown within the false colour representation (upper images), or a split spatial view, wherein only spatial distribution of a particular cluster (cluster 1 to 11) of the bar graph is shown. The computation of the number of pixels for each cluster (alteration and non- alteration areas) and corresponding pixel fractions are shown as a stacked bar graph (adding up 1 or 100%). The processing unit is configured to combine stacked bar representations for all conditions and time points; such combination allows study of kinetic patterns. Such kinetic patterns are useful to discriminate biological tissue under different conditions. In an example these patterns can be used to discriminate resistant from non-resistant plants as well as degree and types of resistances.

[0372] Fig. 8 shows patterns of stack bar graphs obtained from image analysis of plants which were resistent or non-resistent after chemical treatment over time (tO to tl 3).

[0373] Fig. 9 shows an example of variation of accuracy of an alteration classification per type of resistance (metabolic, target, or no resistance) over time represented in a line graph for two imaged plants computed using a recurrent neural network - ALOMY / LOLRI are different types of grass.

[0374] The overall accuracy for discrimination between non-resistant and different types of resistances was computed for each time step, allowing to reduce experimental time to a specific accuracy threshold. These results further reveal that resistances can be detected between day 2 and 3 with an accuracy of categorization pursuant to resistance / no resistance of about 80% which is only possible by eye after more than 10 days. The solution further allows categorization of resistance per type (metabolic or target). This gives the farmer a shorter response time for revising application of herbicide on the field, either by applying another herbicide (target resistance) or using higher dosage (metabolic resistance). Another option for the field applications is to only treat some plants on the field and check for resistances after two or three days and then spray the entire field depending on the results. The method of the invention was used to condense 48TB (experiment) of data for biological interpretation.

[0375] Fig. 12a, 12b, 12c and 12d show an output of the method of the invention, wherein time series images of rust infected soy plants were recorded, and percentage of infected leaf tissue were determined. The infection was further discriminated between strong infection (bright) and mild infection (dimmer gray, 12a). Percentage areas of mild and strong infection are plotted in percentage in Fig 12b and Figl2c. Both weak and strong infection areas increased over time (Fig. 12b, 12c). Beside percentage analysis, also spatial progression was studied over time (Fig 12d). Fig. 12d shows the distance of the infection front towards the leaf rim, clearly revealing a progression from the leaf rims towards the centre. Spread is also clearly visible on the spatial representation of Fig. 12a.

[0376] Fig 13 shows a schematic diagram of a system 10 of the invention comprising a camera detector 20 and processing unit 40 comprising memory storing one or more instructions for controlling acquisition of images 41, storing and retrieving of images 42, normalizing images 43, segmenting images by assigning pixels to tissue or background 44, computing the average spectral signature of the biological tissue 45, identifying altered / non-altered regions of the tissue 46, computing representation(s) of alteration / non-alteration areas 47, generating segmentation masks 50, training models deep learning algorithms for segmentation of biological tissue(s) from background with the segmentation masks 51, training algorithms for pixel classification of tissue region(s) showing an alteration and / or no alteration 52, one or more processors configured to execute the one or more instructions (not shown), one or more memories for image data / reference image data 60 comprising classified image data, one or more memories for storing segmentation masks 61, one or more memory for storing trained models 62. Processing unit 40 is connected to camera detector 40 and a user interface 80.

[0377] Fig. 14 shows a flow chart of exemplary embodiments of the methods according to the invention for the analysis of new images or image series, wherein:

[0378] 501 Causing acquisition of at least one multi wavelength image of a scene within which at least part of the target tissue is located, said multi wavelength image comprising a plurality of monochrome images captured at several wavelengths selected from a range from 250nm to 14000 nm and optionally additional data.

[0379] 502 Saving the image to a binary (or container format).

[0380] 503 Normalizing image data with white reference and dark current.

[0381] 504 Assigning each pixel to biological tissue or background, by way of at least one neural network for semantic image segmentation trained with segmentation masks, wherein said neural network is retrieved from a repository comprising trained algorithms for semantic image segmentation trained with segmentation masks; alternatively, a classical algorithm such as k-Means can be used. Use of neural network is preferred, in particular in case tissue of interest is more difficult to distinguish from background. 505 Optionally prompting user for confirmation of attribution, if confidence of attribution is below a predefined threshold.

[0382] 506 Saving images with tissue vs background label.

[0383] 507 Optionally correcting geometry with SNV approach or by using a 3D point cloud mapped on top of the RGB image.

[0384] 508 Selecting areas attributed to biological tissue.

[0385] 509 Assigning each tissue assigned pixel to one alteration or non-alteration area(s) using one or more trained one or more trained classification networks for pixel classification of regions showing an alteration and / or showing no alteration, wherein the one or more trained classification networks are retrieved from a repository of trained algorithms for pixel classification of regions showing an alteration and / or showing no alteration.

[0386] 510 Optionally computing an averaged probability and / or a distribution curve of probability for accuracy for all pixels assigned to an alteration or non-alteration area.

[0387] Optionally, the system can be configured to prompt user for confirmation of attribution, if accuracy (A) for a pixel is below a predefined threshold or outside an acceptable range, prompting comprises the following steps:

[0388] 511 Computing a pixel group comprising all pixels for which accuracy (A) is below the acceptable threshold.

[0389] 512 Computing a spatial representation of the tissue with marking of the pixel or group of pixels assigned with low accuracy for display to user and / or a signature representation comprising pixel signatures.

[0390] 513 Selecting manually pixel from the group of pixels to amend machine labeling, wherein user can magnify or switch between spatial and signature representation of S12.

[0391] 514 Am ending / confirming labeling for manually selected pixel or group of pixels and saving labeling.

[0392] S15 Computing representations of alteration / non-alteration areas based on labeling, wherein representations may be selected from: SI 5a A spatial representation of tissue and alteration / non-alteration areas using RGB false color attributed to corresponding labeling in (spatial representation).

[0393] SI 5b An average spectral signature for the pixels assigned to the alteration and / or non-alteration areas.

[0394] SI 5c A graph representation of pixel fractions wherein pixel fractions are obtained by way of:

[0395] Computing a number of pixels assigned to the biological tissue or tissue type, a number of pixels assigned to each of the alteration and / or non-alteration areas.

[0396] Computing a ratio of the number of pixels assigned to each alteration and / or non- alteration areas to the whole number of pixels attributed to the biological tissue or tissue type (also referred to as pixel fractions); and

[0397] Computing a graph representation of the pixel fractions e.g. as stacked bar diagrams.

[0398] S16 Causing output of one or more of the presentations of alteration / non-alteration; storing.

[0399] In an embodiment time series are treated, for this purpose the method further comprises the steps:

[0400] Optionally iterating SOI to SI 5 to provide a time series of classified images and corresponding results of image analysis, in particular the spatial representation, the one or more pixel-fractions and / or the average spectral signature for the pixels assigned to the alteration and / or non-alteration areas.

[0401] 520 storing said time series of classified images at time ti from tO to tn.

[0402] 521 Comparing pixel fractions and / or spatial representation of cluster areas to either a reference image (control conditions) to spot alterations due to treatment / diseases effects, or with an earlier time point tO or ti-x of the time series to study a time effect or a combination of both.

[0403] S23 Causing output of results.

[0404] The method may further comprise further steps as shown in the flow chart of Fig. 15. For examples images can be acquired continuously until a particular event is achieved. For this purpose, the method may comprise:

[0405] S30 Comparing the percentage of one or more pixel fraction at time ti with a predefined asymptotic limit AW serving as an indicator:

[0406] - if percentage of pixel fractions at time ti < asymptotic limit AW, causing acquisition and analysis of further images of the scene SOI; or

[0407] - S20 if percentage of pixel fractions at time ti is NOT < asymptotic limit AW, stop causing acquisition of further images of the scene; storing time series of classified images.

[0408] In an embodiment a classified image at ti and / or time series of classified images can be compared with one or more reference image or times series of reference images from a repository. The method may comprise the further steps of:

[0409] S40 Statistically determining difference between percentages of pixel fractions / spectral signatures between different time series (control group and treated / disease group) using e.g. parametrized methods such as One-Way-ANOVA or in case multiple conditions are compared e.g. Two-Way-ANOVA.

[0410] S50 Comparing the pixel fractions graph of SI 5c at time ti with reference data and / or reference alteration curves for the biological tissue stored in a data repository, and determining at least one quantitative value of alteration for the biological tissue.

[0411] 560 Utilizing one or more trained neural networks to predict one or more quantitative values for alteration and / or alteration type, for example potential mode of actions of chemicals for different tissue types at different time points based one or more unique spectral signature of SI 5b or unique pixel fractions graph of SI 5c for the purpose of extracting corresponding further knowledge from the image.

[0412] 561 Predicting further alteration over time based on the comparison of S60.

[0413] S70 Causing display of results on the user interface.

[0414] In an embodiment the method may be implemented for causing acquisition of images of a next scene and / or of one or more images of the scene at different angle(s). Fig. 16 shows a flow chart of exemplary embodiments of the method for generating segmentation masks and subsequent training one or more deep learning algorithms for segmentation of biological tissue(s) from background with the segmentation masks, said method comprising

[0415] 5101 Causing acquisition of n multi wavelength image(s) of a scene within which at least part of the target tissue is located, said multi wavelength image comprising a plurality of monochrome images captured at several wavelengths selected from a range from 250nm to 14000 nm and optionally additional data.

[0416] 5102 Saving the image to a binary (or container format).

[0417] 5103 Normalizing image data with white reference and dark current.

[0418] 5104 Clustering each pixel of one of the images in an unsupervi sed manner using a clustering algorithm to obtain a set of k cluster, wherein k is the number of identified clusters and each cluster is defined by a cluster centroid and a total within-cluster variance TWCV.

[0419] 5105 Using the elbow method computing for different k’s the explained TWCV and selecting the cluster set with the lowest k for which 90% of the TWCV is explained, assigning each pixel in the image to one preliminary cluster by way of pixel labeling based on closest Euclidean distance towards the cluster centroids.

[0420] 5107 Storing the selected clusters set as preliminary cluster(s) in a repository, wherein, for each preliminary cluster, the cluster centroid is saved.

[0421] 5108 In view of user preferences, computing an image in which each pixel is assigned to one preliminary cluster by way of pixel labeling.

[0422] 5109 Optionally, causing revising of preliminary cluster pixel labeling to the biological tissue versus background by way of displaying of the image with the preliminary cluster(s) to a user for user labeling; causing revision may be triggered by considering the validity score (here closest Euclidean distance towards the cluster centroids) being outside a set threshold.

[0423] For revision of the preliminary cluster pixel labeling by user, the method may further comprise: SI 10 Displaying a pixel group corresponding to the one preliminary cluster to the user, prompting him to select or confirm whether, all pixels of the displayed group belong to the biological tissue, to the background or to a mixed cluster comprising a mixture of pixels belonging to tissue and background.

[0424] Si l l Repeating the clustering step with the algorithm for unsupervised clustering SI 04 to SI 08 for the mixed cluster, until only labeling to biological tissue or background is achieved.

[0425] SI 12 Saving pixels with the labeling to the biological tissue in form of a training segmentation masks together with respective training image in a repository;

[0426] SI 13 Reiterating the method for the other images to build a training data set.

[0427] SI 14 Causing feeding a deep learning algorithm for semantic image segmentation with a training segmentation data set comprising at least one (multiwavelength) image alongside one or more segmentation masks containing the whole or parts of the tissue.

[0428] SI 15 Fitting the deep learning algorithm for semantic image segmentation by optimizing assignment of each pixel to tissue of interest or background in the training segmentation data set using an optimizer;

[0429] SI 16 Causing storage of the trained deep learning algorithm for semantic image segmentation for later use on new (multi wavelength) images as described above.

[0430] Fig. 16 shows a flow chart of exemplary embodiments of the method for training a deep learning network for pixel classification of tissue region(s) to an alteration and / or nonalteration area, comprising:

[0431] S200 Causing acquisition of a training data set of n classified 200a or non-classified S200b images or a plurality of classified pixels from an image data repository, wherein each pixel assigned to the tissue or tissue type is additionally assigned to one of the labels selected from non-alteration or a specific class label relating to a specific alteration type; and wherein the training data set comprises classified images of the tissue in flattened form;

[0432] S201 Submitting one or more images of the training data set to spectral data augmentation, wherein spectral data in particular images of the tissue in flattened form are augmented; 5202 Causing feeding algorithm for pixel classification of tissue region(s) showing an alteration and / or showing no alteration with a training data set comprising at least one classified [multi wavelength] image, wherein each pixel assigned to tissue is classified to a non-alteration vs alteration area and labeled accordingly (S202a); Alternatively, algorithm for pixel classification of tissue region(s) showing an alteration and / or showing no alteration with a training data set comprising at least one unclassified [multi wavelength] image, in case classical clustering approaches or generative neural networks such as variational autoencoder, GAN or diffusion networks are used.

[0433] 5203 Fitting the deep learning algorithm for pixel classification by optimizing assignment of each pixel to an alteration or non-alteration area using an optimizer;;

[0434] 5204 Causing storage of algorithm for pixel classification for later use on new [multi wavelength] images;

[0435] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive. The invention is not limited to the disclosed embodiments. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing a claimed invention, from a study of the drawings, the disclosure, and the dependent claims.

[0436] In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. A single processor or other unit may fulfill the functions of several items re-cited in the claims. The mere fact that certain measures are re-cited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.

Claims

Claims:

1. Computer-implemented method comprising:Causing acquisition of at least one multi wavelength image of a scene within which at least part of a biological tissue(s) of interest is located, said multiwavelength image comprising a plurality of monochrome images captured at several wavelengths selected from a range from 250nm to 14000 nm;Assigning, in the at least one multi wavelength image, pixels to the biological tissue(s) versus background using one or more deep learning algorithms for semantic segmentation of biological tissue(s) from background trained with segmentation masks;Assigning each tissue assigned pixel one label for alteration or non-alteration area(s) using one or more trained algorithms for pixel classification of tissue region(s) showing an alteration and / or no alteration;Computing a representation of the pixels assigned to one alteration or non-alteration area(s);Causing output of the representation of the alteration and / or non-alteration area(s) to the user by way of a user interface.

2. The computer-implemented method according to 1, wherein computing a representation of the pixels assigned to one alteration or non-alteration area(s) comprises one or more of:Computing a number of pixels assigned to the alteration and / or non-alteration areas;Computing a ratio of the number of pixels assigned to each alteration and / or non- alteration areas to the whole number of pixels attributed to the biological tissue or a tissue type;Computing an average spectral signature for the pixels assigned to the alteration and / or non-alteration areas.

3. The computer-implemented method according to any one of the claims 1 or 2, wherein each alteration and / or non-alteration area is assigned a predefined false color; and a spatial representation of the tissue in false color, a graph representation of the computed ratios is computed, and / or an average spectral signature view is computed.

4. The computer-implemented method of to any of the claims 1 to 3, further comprisingComputing a validity score for pixel classification to alteration or non-alteration, wherein said validity score is an averaged probability or a distribution curve of probability for accurate classification of all pixels assigned to an alteration or non-alteration area;Classifying one or more of pixel spectral signatures not fulfilling a preset validity score range as unknown alteration; andCausing output of the pixels classified as unknown alteration to the user by way of a user interface.

5. The computer implemented method according to any of the claims 1 to 4, wherein the one or more trained algorithms for pixel classification of region(s) showing an alteration and / or no alteration is one or more classification networks trained in a supervised or in unsupervised manner.

6. The computer-implemented method of claim 5, further comprising:Computing a latent space for the classification network;Classifying pixels as unknown alteration based on the position of pixel in the latent space; and / orComputing and causing presenting the user a 2D- or 3D- representation of the latent space forjudging on how well alteration and non-alteration are separated and / or labeling one or more pixels as unknown alteration based on the position of pixel within the latent space.

7. The computer-implemented method according to any one of the claims 1 to6, further comprising: computing a spatial representation with tissue in false color, graph representation of the computed ratios, and / or the average spectral signature for the pixels assigned to the alteration; presenting computed representation(s) to the user for labeling of pixels attributed to unknown alteration.

8. The computer-implemented method according to any one of the claims 1 to7, wherein the spatial representation of the tissue in false color, the graph representation of the computed ratios, and / or the average spectral signature for the pixels assigned to the alteration area are used for comparison with corresponding classified reference data from a repository, said classified reference data being representative of tissue type, tissue biotype, type of alteration, and / or alteration status at an instant ti.

9. The computer-implemented method according to any one of the claims 1 to 8, further comprising: comparing the number of pixels or the pixel ratios and / or the false colour representation at an instant ti with reference image time series data of the biological tissue showing similar spectral signature using one or more recurrent neuronal networks, identifying known or new temporal alteration patterns in time series data.

10. The computer-implemented method according to any one of the claims 1 to 9, wherein acquisition of time series of multi wavelength images is caused and alteration over time is computed and / or predicted.

11. The computer-implemented method according to claim 10, wherein one or more spatial pattern(s) of alteration are identified and / or spatial progression of an alteration is computed based on the spatial patterns of alteration within time series of multi wavelength images.

12. The computer-implemented method according to any one of the claims 1 to 11, wherein the biological tissue is plant tissue or a part of or a whole plant.

13. The computer-implemented method according to claim 12, wherein alterations are agrochemicals induced damage, degree of fungal infection or other plant disease, active ingredient changes, water stress, other physical plant stress or a combination thereof.

14. Computer-implemented method for training one or more deep learning algorithms for segmentation of biological tissue(s) from background with segmentation masks in an image of a scene within which at least part of the biological tissue(s) of interest is located, comprising:Causing feeding a deep learning algorithm for semantic image segmentation with a training segmentation data set comprising at least one multi wavelength image alongside one or more segmentation masks containing the whole or parts of the tissue;Fitting the deep learning algorithm for semantic image segmentation by optimizing assignment of each pixel to tissue of interest or background in the training segmentation data set using an optimizer;Causing storage of the trained deep learning algorithm for semantic image segmentation for later use on new multi wavelength images; wherein the training segmentation data set is obtained by way of:Causing acquisition of a training data set of (multi wavelength) images of a scene within which the biological tissue is located from an image acquisition unit or from an external source of images;Clustering each pixel of one of the images in an unsupervised manner using a clustering algorithm to obtain a set of k cluster, wherein k is the number of identified clusters and each cluster is defined by a cluster centroid and a total within-cluster variance TWCV;Using the elbow method computing for different k’s the explained TWCV and selecting the cluster set with the lowest k for which 90% of the TWCV is explained;Storing the selected clusters set as preliminary cluster(s) in a repository, wherein, for each preliminary cluster, the cluster centroid and the explained TWCV are stored, and wherein each pixel is assigned to one preliminary cluster by way of pixel labeling to the biological tissue, to a background or to a mixed cluster comprising a mixture of pixels assigned to tissue or background based on the closest Euclidean distance to the cluster centroids towards the pixel;Optionally computing an image in which each pixel is assigned to its preliminary cluster; and Causing revising of preliminary cluster pixel labeling by way of displaying of the image with the preliminary cluster(s) to a user for user labeling;Saving pixels with the labeling to the biological tissue in form of a training segmentation masks together with respective training image in a repository;Reiterating the method for the other multi spectral / hyperspectral images of the training data set.

15. The computer-implemented method of claim 14, wherein: a pixel group corresponding to each preliminary cluster is displayed to the user, prompting him to select or confirm whether, all pixels of the displayed group belong to the biological tissue, to the background or to a mixed cluster comprising a mixture of both; and wherein clustering with the algorithm for unsupervised clustering is repeated for the mixed cluster, until only labeling as target tissue or background is achieved.

16. The computer-implemented method of any one of the claims 14 or 15, wherein, in the image in which each pixel is assigned to one preliminary cluster by way of pixel labeling, each pixel is assigned to a preliminary cluster, and wherein, each pixel is displayed, to the user for confirmation or correction of preliminary cluster assignment.

17. Computer-implemented method for training an algorithm for pixel classification of tissue region(s) in a multi wavelength image into alteration and nonalteration area, comprising:Causing feeding the algorithm for pixel classification of tissue region(s) showing an alteration and / or showing no alteration with a training data set comprising at least one multi wavelength image, wherein each pixel assigned to tissue is either classified to a nonalteration vs alteration area and labeled accordingly or not classified in view of alteration;Submitting the training data set to spectral data augmentation by adding one or more constant factors to the spectra intensities for the training data set;Fitting the deep learning algorithm for pixel classification by optimizing assignment of each pixel to an alteration or non-alteration area using an optimizer;Causing storage of algorithm for pixel classification for later use on new multi wavelength images.

18. Computer-implemented method according to claim 17, wherein the training data set comprises training data set submitted to spectral data augmentation and / or classified images of the tissue in flattened form.

19. System comprising:One or more repository for storing image data, labels and / or trained algorithms;One or more processors configured carrying out one or more methods selected from the method according to any one of the claims 1 to 13, the method according to any one of the claims 14 to 16 and / or the method according to any of the claims 17 to 18.

20. The system according to claim 19, further comprising:- at least one camera detector for multi wavelength imaging;- at least one light source having a continuous spectrum or a discrete spectrum;- wherein, the at least one camera detector is configured to acquire the at least one multi wavelength image of the scene within which at least part of the biological tissue(s) of interest is located.

21. A non-transient computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out one or more methods selected from the method according to any one of the claims 1 to 13, the method accordingto any one of the claims 14 to 16 and / or the method according to any of the claims 17 to 18.

Images