Methods, systems, computer programs and computer-readable data carrier for image generation and classification

Abstract

A method for classifying a target image (1) into a first image type or a second image type is suggested. The method comprises generating (C1) a segmentation result by applying a trained image segmentation module (2.1) to the target image (1); generating (C2) an input feature vector based on the segmentation result, wherein the input feature vector comprises higher-level statistics obtained from the segmentation result; generating (C3) a classification result based on the input feature vector by applying a trained classifier (2.3) to the input feature vector; and classifying (C4) the target image (1) based on the classification result.

Description

[0001] Methods, systems, computer programs and computer-readable data carrier for image generation and classification

[0002] Technical field

[0003] The invention relates to methods, systems, computer programs and computer readable data carriers for image generation and classification.

[0004] Background

[0005] Image classification is widely applicable in many industrial sectors ranging from autonomous driving and robotics to security applications and product inspection. Various artificial intelligence (Al) guided machine learning (ML) and deep learning (DL) models have been employed to facilitate automatic image analysis and pattern recognition in these areas.

[0006] Applications in the field of product inspection pose additional challenges to these models. For example, when attempting to recognize small foreign objects such as contaminants or certain defects in sensible goods such as food products, demands with respect to accuracy, processing speed and degree of automation are particular high.

[0007] In particular, foreign objects or defects not visible from the surface of a product as well as lightweight contaminants, notably plastics, that are difficult to detect due to their similarity to background materials and the limited dynamic range of detectors often evade reliable detection in state-of-the-art inspection systems. Moreover, contaminants or defects can be very small yet extremely hazardous e.g., in food products where small foreign objects inside the food or defects in the package seal area can render the product unsafe to consume. It is difficult to separate these objects from the complex patterns that can be formed by two- dimensional planar projections of common products in images, e.g., taken with a line-scan camera. Moreover, natural and innocent image features that occur due to product packaging or contents may be mistaken for contaminants, further deteriorating detection efficiencies.

[0008] In the prior art, Al-guided models for image recognition and classification typically require very large training sets in particular when small objects need to be classified, especially inside complex patterns. When such training is performed at the inspection site and rather rare events such as the occurrence of foreign objects in food products shall be classified, the respective training can take a very long time, ranging up to several months per product on a production line and typically requiring dedicated labelling of contaminants. Those models are also not very flexible when the performance deteriorates and a retraining is required during deployment, e.g., due to changing system parameters or external conditions. Therefore, a substantial amount of maintenance and external human-guided training and supervision is often still required not only during model development but also after deployment. In particular, when a large variety of food and beverage products need to be classified state of the art models are either limited in their applicability or require extensive use of engineering resources.

[0009] Thus, an object of the invention is to overcome such limitations and provide methods and systems for image generation and classification that improve upon the available methods and devices.

[0010] Summary

[0011] This object of the invention is achieved by the methods, systems, computer programs and computer readable data carriers as described in the appended independent claims. Advantageous developments and embodiments are described in the dependent claims.

[0012] In a first aspect, the invention relates to a classification method for classifying a target image into a first image type or a second image type. The method comprises generating a segmentation result by applying a trained image segmentation module to the target image; generating an input feature vector based on the segmentation result, wherein the input feature vector comprises higher-level statistics obtained from the segmentation result; and generating a classification result based on the input feature vector by applying a trained classifier to the input feature vector. The trained classifier has been trained to provide a classification result that is indicative of whether the input feature vector has been generated based on a segmentation result obtained from an image of the first image type or from an image of the second image type. The method further comprises classifying the target image based on the classification result.

[0013] With the proposed method, a two-step model is proposed, wherein image segmentation is used as a basis for classification of an image as a whole into a first and second image type. In a first step, the whole target image is segmented by a trained image segmentation module applied directly to the target image as the input to the image segmentation model. Then, in a second step, higher-level statistics are obtained from the segmentation result to efficiently form a small-sized input feature vector as a direct input to the (lightweight) classifier. An input feature vector can thus also be called a feature generated vector, said vector being used for input into the classifier. The lightweight classifier thus that does not directly classify the target image as an input, but rather the small-sized input feature vector that comprises higher-level statistics obtained from the target image. However, since the input feature vector is based on the target image, a classification of the input feature vector then also yields the classification of the target image.

[0014] It is noted that a classification into a first and second image type in the spirit of this application does not exclude that multi-class classification into more than two image types is being performed. In other words, a multi-class classification into more than two image types may comprise a classification into a first and second image type.

[0015] The proposed method is particular flexible and adaptable, because training and / or performance monitoring and calibration of the image segmentation module and the classifier can be performed largely independent from each other (since segmentation module and classifier are treated as two distinct models). Moreover, the access to high-level statistics (based on the output of the segmentation module / model) provides valuable and intuitive insight into the performance and accuracy of the method when internal and external conditions change, thereby providing means for fast and efficient continuous adaptation, self-monitoring and auto-calibration of the proposed method and system (e.g., of the classifier only) also during the deployment phase.

[0016] In the following, an (i.e., every) image of the second image type may show an object of at least one object type. An image of the first image type may not show an object of the at least one object type. In other words, no image of the first image type may show an object of the at least one object type. In this case, the classification method may be used to classify (target) images into those that show an object of the at least one object type and images that do not show an object of the at least one object type.

[0017] Optionally, an image of the first image type shows a product of a product type that does not comprise an object of at least one object type. An image of the second image type may show a product of the product type with (or comprising) an object of the at least one object type.

[0018] A product type may be a consumer product, a food product, a beverage product, and / or a factory good which may be packaged or loose. An object type may be a foreign object, a contaminant and / or a defect. An object type may also classify an object more narrowly. For example, an object type may be a food product contaminant, a beverage product contaminant and / or a consumer good defect etc. An object type may also define the shape, geometry and / or material composition of an object.

[0019] For example, an image of the first image type may be an uncontaminated product image that shows a product free of contaminants of at least one contaminant type. An image of the second image type may be a contaminated product image that shows a product of the product type with a contaminant of the at least one contaminant type. In this case, the object may be a contaminant, the object type is a contaminant type and the classification method may be used for product inspection, e.g., to detect contaminants in products.

[0020] Optionally, the image segmentation module is trained for providing a segmentation result for locating and / or detecting an object of the at least one object type in a (target) image, e.g., in an image of the second image type, and / or is trained to generate a segmentation result from which at least the location of an object of the at least one object type in a (target) image (of the to-be-determined second image type) can be obtained. The image segmentation module may be realised in software or hardware and may be configured for binary or multi-class segmentation.

[0021] The image segmentation module may comprise or be a learning model, e.g., a deep learning model, for image segmentation. Optionally, the image segmentation model is convolutional neural network model.

[0022] Optionally, the segmentation result comprises pixel-by-pixel probabilities as the output of the image segmentation module. A segmentation result may correspond or be identical with the pixel-by-pixel probabilities. The pixel-by-pixel probabilities may comprise for each pixel of the target image a probability that said pixel shows an object of the at least one object type or at least a fraction / part of an object of the at least one object type. Optionally, the higher-level statistics are obtained from (or determined by) the pixel-by-pixel probabilities. Additionally, or alternatively, the higher-level statistics can also be obtained from (or determined by) the pixel-by-pixel intensity values of the target image and / or from pixel-by-pixel attenuation values of (or obtained from) the target image. A pixel attenuation value can be obtained from a pixel intensity value of the target image. Each pixel attenuation value of a given pixel may correspond to a different energy channel or energy bin of the electromagnetic radiation used for imaging. A pixel attenuation value for a given energy channel or energy bin may be obtained from a number of photons detected at the respective pixel in said energy channel or energy bin during imaging (i.e., during a predetermined time span). The pixel attenuation values may be taken from a discrete set of pixel attenuation values. Optionally, the discrete set may comprise three attenuation values each being obtained respectively from a number of detected photons in discrete energy bins / channels, e.g., a low, high and total energy channel. For example, a first attenuation value of / for a given pixel may be obtained from a first number of detected photons at the given pixel in (or corresponding to) a first energy bin / channel. A second attenuation value of a given pixel may be obtained from a second number of detected photons at the given pixel in a second energy bin / channel. A third attenuation value of a given pixel may be obtained from a third number of detected photons at the given pixel in a third energy bin / channel. Optionally, the first and second energy bin / channel are different and / or do not overlap. The third energy bin / channel may correspond to a sum of the first and second energy bin / channel. For each pixel, an attenuation vector may be formed comprising the first attenuation value as a first vector component, the second attenuation value as a second vector component and / or the third attenuation value as a third vector component. The pixel-by-pixel attenuation values of a given image may comprise the attenuation vectors for each pixel of said image.

[0023] Advantageously, the higher-level statistics may comprise at least one of a statistical measure from a list of statistical measures. At least one element of the input feature vector, preferably each element of the input feature vector, corresponds to a statistical measure from the list of statistical measures. The list of statistical measures may comprise at least one of a sum over pixel-by-pixel probabilities of the segmentation result, a maximum pixel- by-pixel probability of the segmentation result, the number of pixels with a pixel-by-pixel probability of the segmentation result larger than a predetermined threshold, a (weighted) sum over pixel-by-pixel attenuation values (or components of pixel-by-pixel attenuation vectors) of the target image, a sum over pixel-by-pixel attenuation values (or components of pixel-by-pixel attenuation vectors) of the target image weighted by pixel-by-pixel probabilities of the segmentation result. The classifier may be configured to classify an input feature vector into a first vector type or a second vector type. There may be a one-to-one correspondence between the first vector type and the first image type and between the second vector type and the second image type. The first vector type may classify the input feature vector to be obtained from an image of the first image type. The second vector type may classify the input feature vector to be obtained from an image of the second image type. A type may correspond to a class. The length of an input feature vector may be smaller than 100, preferably smaller than 20.

[0024] The classifier may be a learning model. The classifier may be a supervised learning model, e.g., a regression model. Optionally, the classifier is a logistic regression model, decision tree model or k-nearest neighbor model. Alternatively, the classifier may also be an unsupervised learning model, e.g., an outlier detection model.

[0025] In a second aspect, the invention relates to a classification system for classifying a target image into a first image type or a second image type. The classification system comprises a trained image segmentation module, a feature extraction module and a trained classifier. The trained image segmentation module is configured to generate a segmentation result for the target image. The feature extraction module is configured to generate an input feature vector based on the segmentation result. The input feature vector comprises higher-level statistics obtained from the segmentation result. The trained classifier is configured to generate a classification result based on the input feature vector, wherein the trained classifier has been trained to provide a classification result that is indicative of whether the input feature vector has been generated based on a segmentation result obtained from an image of the first image type or from an image of the second image type. The trained classifier is further configured to classify the target image based on the classification result.

[0026] The classification system may further comprise an imaging apparatus configured to take the target image for classification. The imaging apparatus may also be configured to take the training images used to train the image segmentation module and / or used for training the classifier. The image segmentation module, feature extraction module and the classifier can also be integrated in the imaging apparatus.

[0027] The imaging apparatus may be or may comprise a camera. Optionally, the imaging apparatus is an X-ray imaging apparatus. The target image may be an X-ray image. The camera may be a photon-counting camera. Preferably, the camera is a dual-energy or multi-energy camera. The dual-energy or multi-energy camera may be configured to detect photons in at least two different energy bins / channels. If the camera is also a photon-counting camera it may be configured to count the number of (transmitted or reflected) photons in at least two different energy bins / channels. This may allow to obtain pixel-by-pixel attenuation values or reflected radiation values (or vectors) for images taken by the camera as explained further above, wherein at least two attenuation values or reflected radiation values for each pixel can be obtained from the number of detected photons in two respective energy bins / channels.

[0028] Pixel-by-pixel attenuation values may be obtained from the target image. Each pixel of the target image may comprise or correspond to a plurality of pixel-by-pixel attenuation values. Each pixel attenuation value of a given pixel of the target image may correspond to a different energy or frequency channel / bin for the photons and / or the electromagnetic radiation impinging at the imaging apparatus at a position corresponding to said pixel. In other words: each pixel-by-pixel attenuation value of a given pixel may quantify the number of photons used to illuminate said pixel with an energy in a given energy channel or bin.

[0029] The imaging apparatus may be configured to provide pixel-by-pixel attenuation value for an image taken by the imaging apparatus, e.g., for the target image. The imaging apparatus may be configured to generate pixel attenuation values of the target image from a discrete set of pixel attenuation values. Optionally, the imaging apparatus may be configured to determine for each pixel of an image taken by the imaging apparatus a plurality of pixel-by- pixel attenuation values as described further above. This description of an attenuation value may be applied not only to the target image, but any image taken by the imaging apparatus.

[0030] In a third aspect, the invention relates to a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the classification method according to the first aspect or any variant thereof.

[0031] In a fourth aspect, the invention relates to a computer-readable data carrier having stored thereon the computer program of the third aspect.

[0032] A computer program (or a sequence of instructions) in the spirit of the present application may use software means for performing the method when the computer program runs in a computing unit. The computer program can be stored directly in an internal memory, a memory unit, or the computer. A computer program can be stored in machine-readable data carrier(s), preferably digital storage media. A computer or computer unit in the spirit of the present application may be or comprise a processor, a central processing unit (CPU), a graphical processing unit (GPU), a neural processor or a microcontroller. The computing resources required to perform the mentioned segmentation and classification analysis on images of a typical dimension, resolution and single or multiple channel depth are sufficiently modest that the computer, computing unit and / or computer-readable data carrier may be integrated or embedded in the imaging apparatus.

[0033] In a fifth aspect, the invention relates to an image generation method. The image generation method comprises obtaining a first set of images of a first image type, wherein each image of the first set of images is taken by an imaging apparatus; obtaining object images, wherein each object image shows an object of at least one object type; and generating a second set of images of a second image type, wherein each image of the second set of images is generated synthetically by combining an image of the first set of images with at least one of the object images. At least the synthetically generated images of the second set are used for training a classification system and / or an image segmentation module and / or a classifier for classifying a target image into a first image type or a second image type.

[0034] With the proposed image generation method, it becomes possible to generate quickly and efficiently a very large training data set for the image segmentation module. Further, images of the second image type used to generate input feature vectors that are comprised in a classification training set for training a classifier can be generated synthetically without the necessity for manual labelling.

[0035] An object image is an image of an isolated object of the at least one object type. The objects used to take the object images can be selected such that the training data can be adapted to a specific segmentation and / or classification task, e.g., for at least one specific object type. The object type may be a contaminant, defect or foreign object. The object type may alternatively or additionally characterise a size range, a shape and / or a material composition of an object.

[0036] At the same time, since the first set of images used to generate the second set of images are taken by the same imaging apparatus that is used to take the target image to be segmented by the trained image segmentation module and / or classified by using the trained classifier, the generated second set of images and thus the training of the image segmentation module and / or the classifier are accurately adapted to the specifics of the imaging apparatus, e.g., image size, number of pixels, pixel intensity values, noise properties etc. Preferably, the object images are also taken by the (same) imaging apparatus. Alternatively, the object images may also be pre-stored in a database or library.

[0037] Preferably, the synthetically generated images of the second set are used for training the image segmentation module and / or the classifier of the classification method and system of the first and second aspect. Since the output of the trained (or pre-trained) image segmentation module can be used to train the classifier, training images used to train the image segmentation module can also be used for training the classifier using input feature vectors obtained from higher-level statistics of segmentation results, wherein the segmentation results are obtained from the training images. However, training images used to train directly the image segmentation module and training images used to generate input feature vectors from the trained (pre-trained) image segmentation module for training the classifier can also be different from each other. Preferably, the training images for both, for training the image segmentation module and for training the classifier, are generated synthetically according to the image generation method described here. This training procedure allows labelling of contaminants or defects to be avoided completely if desired.

[0038] Optionally, each image of the second set shows at least one object of at least one object type and no image of the first set shows an object of the at least one object type. The absence or presence of an object of the at least one object type may correspond to two different (training) label values used to label the synthetically generated images of the second set and / or the training data set(s) used to train the classifier. For example, and as explained further above, an (every) image of the first image type may be an uncontaminated product image corresponding to a first label or first label value and an (every) image of the second image type may be a contaminated product image corresponding to a second label or second label value.

[0039] Optionally, the imaging apparatus is an X-ray imaging apparatus. Optionally, the first set of images and / or the object images are taken by the imaging apparatus in a transmission configuration. For example, a radiation device or source may be configured to emit electromagnetic radiation (photons) that transmits through the product(s) of the product type and / or the object(s) of the at least one object type, wherein the transmitted part of the electromagnetic radiation is detected by the imaging apparatus for taking the corresponding images. In this case, the products and objects are positioned between the radiation device and the imaging apparatus.

[0040] Optionally, pixel-by-pixel attenuation values may be obtained from the images of the first image type, the images of the second image type and / or the object images. It may also be the case that the images of the first image type, the images of the second image type and / or the object images may comprise pixel-by-pixel attenuation values. An attenuation value may be obtained from the transmitted part of the electromagnetic radiation in a transmission configuration impinging at an imaging apparatus as explained above. Since the images of the first set and the object images are taken by the same imaging apparatus that takes the target image, their pixel-by-pixel attenuation values may be configured in a same manner as described with respect to the target image further above. An attenuation value may correspond to a measure of the reduction of intensity of electromagnetic radiation between radiation emitted by the radiation device and the transmitted radiation impinging at the imaging apparatus. An attenuation value may be characterised by luminosity or electrical charge converted into a grey-scale image / pixel value or into a number of detected photons. Specifically, a pixel attenuation value for a given pixel and energy channel / bin may correspond (e.g., up to normalisation factors) to a logarithm In (Nem / of a number of photons Nememitted by the radiation device for imaging in the given energy channel / bin divided by a fraction of said emitted photons Ntthat is being transmitted and detected by the given pixel in the given energy channel / bin. Since said number / fraction of transmitted photons depends on the number of photons being absorbed along a transmission path between the radiation device and the imaging apparatus, an attenuation value may be a measure for the number of absorbed photons along said transmission path.

[0041] Optionally, the combining an image of the first set of images with at least one of the object images comprises adding according to a weighted sum model a pixel attenuation value of a pixel in the image of the first set of images and a pixel attenuation value of a pixel in the at least one object image, wherein the pixel in the at least one object image corresponds to the object of the at least one object type shown in the at least one object image to obtain a pixel attenuation value of a pixel in the image of the second set of images.

[0042] For example, the pixel attenuation values corresponding to the object of the at least one object type in an object image can be added pixel-by-pixel to the pixel attenuation values of the pixel region in the image of the first set at which the object is to be placed. The resulting image of the second image type then corresponds to a superposition of the corresponding image of the first image type and the object image. The adding of the pixel intensity values may generate images of the second image type that are particular realistic when the imaging apparatus operates in a transmission configuration and / or the imaging apparatus is an X-ray imaging apparatus. Optionally, the weighted sum model may be a simple sum (with weights=l corresponding to an identity operation).

[0043] Optionally, the image of the first set is combined randomly with the at least one of the object images such that the at least one object of the at least one object type is located in the image of the second set generated from the image of the first set and the object image at a random pixel position. Optionally, an object of the at least one object type shown in the object image is positioned at a random pixel position in the image of the second set. Optionally, the image of the second set is a random superposition of the image of the first set and the object image. The randomization of the object location may help generating a large training dataset with reduced bias and modest computational efforts.

[0044] Optionally, for each image of the first set of images a plurality of images of the second set are generated by varying the position and / or orientation of the object of the at least one object type from the at least one object image among the plurality of images of the second set generated from the image of the first set of images. Optionally, the number of images in the second set is larger than the number of images in the first set and / or larger than the number of object images used to generate the second set. Optionally, for each single image of the first set of images a plurality of images of the second set are generated by varying the position and / or orientation of the at least one object of the at least one object type obtained from the at least one object image randomly.

[0045] Optionally, a plurality of images of the second set are generated by combining each image of the first set of images with a plurality of objects of the at least one object type and / or a plurality of object images such that for each image of the first set of images a plurality of images of the second set is generated.

[0046] In a sixth aspect, the invention relates to an image generation system comprising an imaging apparatus and a hybrid synthetic generator. The imaging apparatus is configured to take a first set of images of a first image type; take object images, wherein each object image shows an object of at least one object type; and take a target image for classification. The hybrid synthetic generator is configured to generate a second set of images of a second image type, wherein each image of the second set is generated synthetically by combining an image of the first set with at least one of the object images. At least the synthetically generated images of the second set are used for training a classification system and / or an image segmentation module and / or a classifier for classifying the target image into a first image type or a second image type.

[0047] The imaging apparatus may be configured for polychromatic or multi-energy imaging. Optionally, the imaging apparatus comprises a camera and the first set of images, the object images and the target images are taken by the camera with the same camera settings. For example, the same camera settings may comprise at least one of a same image size, a same number of pixels, a same discrete set of pixel attenuation values for each pixel, the same energy thresholds for binning of detected photons (corresponding to the discrete set of pixel attenuation values for each pixel), a same focal parameter, a same aperture, a same zoom factor etc. Optionally, the image generation system additionally comprises a conveyer, e.g., a conveyer belt. The conveyer may be configured to provide a (target) product for taking the target image of the target product by the imaging apparatus; provide (training) products for taking the first set of images of the (training) products by the imaging apparatus; and / or provide (training) objects of at least one object type for taking the object images of the (training) objects by the imaging apparatus. The conveyer may further be configured to move the (target) product, the (training) products and / or the (training) objects one after the other into an illumination region for imaging by the imaging apparatus.

[0048] Optionally, the image generation system comprises a radiation device. The radiation device may be configured to emit electromagnetic radiation towards the illumination region, the conveyer and / or the imaging apparatus. More specifically, the radiation device may be configured to emit the electromagnetic radiation towards the (target) product for taking the target image by the imaging apparatus, towards the (training) products for taking the first set of images by the imaging apparatus and towards the (training) objects for taking the object images by the imaging apparatus.

[0049] The radiation device may be a polychromatic radiation device. The radiation device may be an X-ray source, e.g., an X-ray tube. The spectrum of electromagnetic radiation emitted by the radiation source may be kept the same during the taking of the target image, the first set of images and the object images by the imaging apparatus.

[0050] Preferably, the (target) product, the (training) products and the (training) objects are positioned by the conveyer between the radiation device and the imaging apparatus for imaging in a transmission configuration.

[0051] Advantageously, the (target) product, the (training) products and the (training) objects are transported by the conveyer through the illumination region illuminated by the electromagnetic radiation emitted by the radiation device at the same conveyer speed so that the (target) product, the (training) products and the (training) objects are illuminated substantially by the same amount of electromagnetic radiation during imaging by the imaging apparatus.

[0052] In a seventh aspect, the invention relates to a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the image generation method according to the fifth aspect or any variant thereof.

[0053] In an eight aspect, the invention relates to a computer-readable data carrier having stored thereon the computer program of the fifth aspect. In a ninth aspect, the invention relates to a training method. The training method comprises obtaining a first set of images of a first image type; obtaining a second set of images of a second image type; generating first segmentation results for the first set of images, wherein each first segmentation result is generated by applying a trained image segmentation module to another image of the first set; and generating second segmentation results for the second set of images, wherein each second segmentation result is generated by applying the trained image segmentation module to a different image of the first portion of the second set.

[0054] The method further comprises generating first input feature vectors, wherein each first input feature vector comprises higher-level statistics obtained from another first segmentation result of the generated first segmentation results; and generating second input feature vectors, wherein each second input feature vector comprises higher-level statistics obtained from another second segmentation result of the second segmentation results.

[0055] The method further comprises training with a classification training set a classifier for classifying a target image into a first image type or a second image type, wherein the classification training set comprises the first input feature vectors and the second input feature vectors.

[0056] Optionally, the method may additionally comprise training the image segmentation module with a segmentation training set before generating the first and second segmentation results and / or training the classifier. The segmentation training set may comprise images of the second image type. For example, the segmentation training set may comprise the second set of images of the second image type.

[0057] The images of the second image type in the segmentation training set and / or the segmentation training set may be generated synthetically from images of the first image type and object images by using the image generation method according to the fifth aspect or the image generation system according to the sixth aspect or any variant thereof.

[0058] The images of the second image type in the segmentation training set may be obtained from the first set of images also used to train the classifier or from another set of images of the first image type (and object images). The set of images of the first image type and the object images used to generate synthetically the segmentation training set may be taken by the same imaging apparatus used to classify the target image and used to take the first set.

[0059] The segmentation training set used to train the image segmentation module may be substantially larger than the classification training set used to train the classifier. The number of images of the second image type in the segmentation training set may be larger than the number of images in the second set of images used for training the classifier.

[0060] Specifically, the number of images of the second image type used to train directly the image segmentation module may be substantially larger than the number of images of the second image type from which the second input feature vectors of the classification training set are obtained. In other words, the number of images of the second image type in the segmentation training set may be substantially larger than the number of second input feature vectors in the classification training.

[0061] Preferably, the first set of images used to generate the first segmentation results for training are taken by the same imaging apparatus used to take the target image for classification.

[0062] The second set of images of the second image type used for generating the second input feature vectors for training the classifier may also be taken (directly) by the imaging apparatus (and may thus not be generated synthetically). In this case the classification training set would have to be labelled quasi-manually (e.g., contaminated product images and uncontaminated product images would have to be identified and labelled accordingly). However, since the classifier does not directly take whole images as an input, but rather input feature vectors with a substantially smaller dimension / length (higher-level statistics), far less training samples are required for training the classifier and such labelling would be less resource-intense compared to a classification training set that would comprises whole images as a direct input to a (not claimed) classifier.

[0063] In a tenth aspect, the invention related to a training system. The training system comprises a training module configured to carry out the steps of the training method according to the ninth aspect or any variant thereof.

[0064] The training system may also comprise the image segmentation module. The image segmentation module may be trained (or pre-trained). The image segmentation module may be configured to generate a segmentation result for an image as a direct input to the image segmentation module, wherein the segmentation result is configured and / or can be used to locate (automatically) an object of at least one object type in the image. If an image as an input to the image segmentation module is of the first image type the segmentation result may indicate that no object of the at least one object type is shown at any pixel location in the image (low pixel-by-pixel probabilities). If an image as an input to the image segmentation module is of the second image type the segmentation result may indicate a pixel region in the image where an object of the at least one object type is shown (high pixel- by-pixel probabilities in said pixel region).

[0065] The training system may also comprise the classifier. The classifier may be configured to generate a classification result based on an input feature vector. The input feature vector may comprise higher-level statistics obtained from the segmentation result generated by the image segmentation module. The length of an input feature vector as an input to the classifier may be smaller than 100, preferably smaller than 20.

[0066] The training system may also comprise an imaging apparatus. The imaging apparatus may be configured to take the first set of images of the first image type. The imaging apparatus may also be configured to take object images, wherein each object image shows an (training) object of at least one object type. The imaging apparatus may also be configured to take the target images.

[0067] The training system may also comprise a hybrid synthetic generator. The hybrid synthetic generator may be configured to generate the second set of images of the second image type, wherein each image of the second set may be generated synthetically by combining an image of the first set with at least one object image as explained in relation to Figures 1 to 3.

[0068] In an eleventh aspect, the invention related to a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the training method according to the ninth aspect or any variant thereof.

[0069] In a twelfth aspect, the invention relates to a computer-readable data carrier having stored thereon the computer program according to the eleventh aspect.

[0070] In a thirteenth aspect, the invention relates to a self-monitoring classification method for self-monitoring the performance of classifying target images into a first image type or a second image type. The self-monitoring classification method comprises classifying each target image of a plurality of target images into the first image type or the second image type according to the classification method according to the first aspect or any variant thereof with the same trained image segmentation module and the same trained classifier during a deployment phase. The self-monitoring classification method further comprises carrying out the self-monitoring steps during the deployment phase of determining a performance metric for the trained classifier after at least one target image of the plurality of target images is classified and if the performance metric does not meet a predetermined threshold criterion retraining the trained classifier. The deployment phase may be a time duration during which the plurality of target images or at least a portion thereof is being classified. The deployment phase may also be a time duration during which the conveyer maintains a constant speed for imaging the target products and / or for taking the target images.

[0071] Optionally, the self-monitoring classification method may comprise retraining only the trained classifier without retraining the trained image segmentation module during the deployment phase. After the retraining of the classifier, the self-monitoring classification method may comprise classifying during the deployment phase target images of the plurality of target images into the first image type or the second image type with the classification method according to the first aspect or any variant thereof with the same trained image segmentation module, but the retrained classifier.

[0072] Optionally, the determining a performance metric for the trained classifier during the deployment phase comprises: generating at least one additional test image of the second image type based on the at least one already classified target image and at least one additional object image, wherein the at least one additional object image shows at least one (training or re-training) object of an object type; generating at least one additional classification result based on the at least one additional test image by using the trained classifier; and determining the performance metric for the trained classifier based on the at least one additional classification result.

[0073] In case the target image is being classified as being of the first image type, the generating at least one additional test image of the second image type based on the at least one already classified target image and at least one additional object image may comprise combining the at least one already classified target image with the at least one additional object image.

[0074] Preferably, the at least one additional test image of the second image type is generated synthetically by the image generation method according to the fifth aspect or any variant thereof.

[0075] It may be determined that the at least one additional test image is generated only if a target image has been classified as being of the first image type. At least one additional test image may be generated for each target image being classified as being of the first image type or in regular intervals, e.g., for every Nth target image of the first image type, where N is an integer larger than 1. Additionally, or alternatively, in case the target image is being classified as being of the second image type, the generating at least one additional test image of the second image type based on the at least one already classified target image and at least one additional object image may comprise masking out based on the segmentation result for the target image the pixel area corresponding to an (or all) object(s) of the at least one object type in the classified target image of the second image type, thereby synthetically generating a modified target image of the first image type that is free of objects of the at least one object type. The synthetically generated target image of the first image type can then be used to generate the additional test image of the second image type in a similar manner as described further above, i.e., in a case where the target image has been classified as being of the first image type.

[0076] The masking out of a pixel area may comprise synthetically assigning zero or small pixel-by- pixel probabilities (e.g., in a range from 0.001 to 0.01) to pixels corresponding to the masked-out pixel area and / or (synthetically) labeling the corresponding target image as being of the first image type. Masking out a pixel area may also correspond to assigning zero weight to the pixels of said pixel area either for the purpose of segmentation or when forming the corresponding input feature vector. This may correspond to effectively ignoring said pixel area in the segmentation and / or classification.

[0077] Being able to use target images of both image types for generating the at least one additional test image and the determining of the performance metric enables more continuous monitoring and re-training. It may also help re-training in a case where a shift in the data may occur such that the production model finds spurious contaminants in all target images and would thus not be able to evaluate its performance and retrain itself based on target images classified as being of the first image type.

[0078] Optionally, the generating at least one additional classification result by the trained classifier based on the at least one additional test image comprises generating at least one additional second segmentation result by applying the trained image segmentation module to the at least one additional test image; generating at least one additional second input feature vector based on the at least one additional second segmentation result, wherein the at least one additional second input feature vector comprises higher-level statistics obtained from the at least one additional second segmentation result; and generating the at least one additional classification result by applying the trained classifier to the at least one additional input feature vector.

[0079] Optionally, the self-monitoring method additionally comprises storing the at least one additional second input feature vector; retraining with a classification retraining set the trained classifier, wherein the classification retraining set may comprise the at least one additional second input feature vector.

[0080] Optionally, the retraining of the classifier may be performed with the training method according to the ninth aspect or any variant thereof. In this case the retraining set here replaces the training set of the ninth aspect.

[0081] Optionally, the at least one target image classified as being of the first image type during the deployment phase comprises a plurality of target images classified as being of the first image type, the at least one additional test image comprises a plurality of additional test images, the at least one additional classification result comprises a plurality of additional classification results, the at least one additional segmentation result comprises a plurality of additional segmentation results, the at least one additional classification result comprises a plurality of additional classification results, and / or the at least one additional second input feature vector comprises a plurality of additional second input feature vectors.

[0082] Optionally, the performance metric is a precision and recall metric. Optionally, the predetermined threshold criterion comprises a threshold for the number of false positives (or the false positive rate) among the at least one additional classification result. The number of false positives may be derived from the additional classification results. In particular, since it is known that the at least one additional test image is an image of the second image type (since it is purposefully generated this way, e.g., synthetically), said known information can be compared to the at least one additional classification result in order to arrive at a number of false positives.

[0083] In a fourteenth aspect, the invention relates to a self-monitoring classification system comprising a self-monitoring module. The self-monitoring module is configured to carry out the steps of the self-monitoring classification method according to the thirteenth aspect or any variant thereof.

[0084] Optionally, the self-monitoring classification system may also comprise a training system according to the tenth aspect or any variant thereof. The training module may be additionally configured to carry out the retraining of the classifier.

[0085] In a fifteenth aspect, the invention relates to a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the self-monitoring classification method according to the thirteenth aspect or any variant thereof. In a sixteenth aspect, the invention relates to a computer-readable data carrier having stored thereon the computer program of the fifteenth aspect.

[0086] In a seventeenth aspect, the invention relates to an image generation method. The image generation method according to the seventeenth aspect may be considered as an alternative to the image generation method according to the fifth aspect.

[0087] The image generation method according to the seventeenth aspect comprises obtaining a first set of images of a first image type, wherein each image of the first image type shows a product of a first product type and obtaining a second set of images of a second image type, wherein each image of the second image type shows a product of a second product type.

[0088] The first and / or second set of images are used for training an image segmentation module and / or a classifier for classifying a target image taken by an imaging apparatus into the first image type or the second image type.

[0089] The obtaining a first set of images comprises generating each image of the first set synthetically by simulating computationally the propagation of electromagnetic radiation from a radiation device through at least a part of a product of the first product type to the imaging apparatus; and / or the obtaining a second set of images comprises generating each image of the second set synthetically by simulating computationally the propagation of electromagnetic radiation from a radiation device through at least a part of a product of the second product type to the imaging apparatus.

[0090] With the proposed (purely) synthetic image generation method images of the first and / or second image type can be generated on-demand, fast and efficiently in a large number, e.g., for the purpose of training the image segmentation module and / or the classifier. Here, a product of a first product type can be a product lacking an object of at least one object type, e.g., a contaminant of a contaminant type and a product of a second product type can be a product comprising at least at least one object of the at least one object type, e.g., at least one contaminant of the contaminant type. The images, the products, the contaminants, the image segmentation module, the classifier, the electromagnetic radiation, the radiation device and the imaging apparatus may be configured as described with respect to other aspects of the invention and variants thereof. Preferably, the synthetic image generation method is applied to simulate X-ray electromagnetic radiation in transmission imaging. The propagation of electromagnetic radiation through at least a part of a product of the first and / or second product type may also include a case in which a part of the electromagnetic radiation only partially penetrates the product and a part of the electromagnetic radiation is reflected from a surface of the product as it would be the case in reflection imaging.

[0091] Optionally, the simulating computationally the propagation / transmission of electromagnetic radiation through the product comprises obtaining an initial spectral distribution of the electromagnetic radiation as emitted by a radiation device; and obtaining a three- dimensional voxel model of the product of the first product type and / or second product type. The voxel model comprises a (value of a) voxel thickness and a (specification of a) voxel material for each voxel.

[0092] The three-dimensional voxel model for a product of the second product type can be a combination of a 3D voxel model of a product of the first product type and a 3D voxel model of a contaminant of the contaminant type. Both types of 3D voxel models may be pre-stored for a large variety of products and contaminants in a voxel model database.

[0093] The image generation method may further comprise obtaining from a material database attenuation data for each voxel of the voxel model based on a voxel thickness and the voxel material of each voxel, obtaining based on the initial spectral distribution and the attenuation data for each voxel an estimate of pixel-by-pixel attenuation values corresponding to the part of the electromagnetic radiation transmitted through the product and detected at the imaging apparatus. From the estimated pixel-by-pixel attenuation values a fully synthetically generated image can then be obtained that shares high similarity with an image that would have been taken by the (real) imaging apparatus when a (real) product is illuminated by (real) electromagnetic radiation emitted by a (real) radiation device.

[0094] The obtaining a first set of images may comprise generating each image of the first set synthetically by additionally simulating computationally the interaction of electromagnetic radiation with matter, e.g., with the product of the first product type. The obtaining a second set of images may also comprise generating each image of the second set synthetically by additionally simulating computationally the interaction of electromagnetic radiation with matter, e.g. with the product of the second product type. By simulating computationally, the interaction of electromagnetic radiation with matter, attenuation data and / or reflection data for each voxel or a number of voxels (e.g. corresponding to a part or surface of the product) may be generated and obtained. The attenuation data and / or reflection data may account for and / or comprise (the effects of) the interaction of electromagnetic radiation with matter, e.g., the interaction of X-rays with matter.

[0095] The attenuation data for each voxel may be a transmission matrix. The elements of the transmission matrix for each voxel can be defined as follows: We may assume N discrete energy channels / bins and we may consider a voxel of a given size and thickness and of a given material. We may further consider a monochromatic beam of energies in energy bin i and that Nin iof these photons are incident at the voxel. Further, it may be assumed that Nout,i photons in an energy bin j are being transmitted through and emerge from the voxel. In this case an element of the transmission matrix may be given by Ti j=N0Utj / Nin i. The diagonal matrix elements of the transmission matrix are then given by the Beer-Lambert law as discussed further below. The off-diagonal elements may include physical effects due to Compton scattering, Rayleigh scattering and / or the photoelectric effect. Here, the individual values / elements of the transmission matrix are computed in terms of monochromatic beams. The transmission matrix as a whole can then be multiplied with an (arbitrary) spectral distribution of incoming radiation.

[0096] The obtaining based on the initial spectral distribution and the attenuation data for each voxel an estimate of the pixel-by-pixel attenuation values corresponding to the part of the electromagnetic radiation transmitted through the product and detected at the imaging apparatus may comprise multiplying the initial spectral distribution with a matrix product of transmission matrices, wherein each transmission matrix in the matrix product corresponds to a different voxel along a transmission path of electromagnetic radiation through the voxel model and towards the imaging apparatus. Repeating this step for a plurality of transmission paths ending at a plurality of pixels of the imaging apparatus may yield a plurality of attenuation values from which an image can be generated. The image may be an image of the first image type, e.g., when the product is of the first product type. The image may be an image of the second image type, e.g., when the product is of the second product type.

[0097] Multiplying the transmission matrices across a transmission path corresponding to a series of stacked voxels yields a fast and scalable method of computing multi-energy-bin photon counts also for non-monochromatic (polychromatic) electromagnetic radiation passing through a complex product corresponding to the voxel model. From the final pixel-by-pixel photon counts, the corresponding pixel-by-pixel attenuation values can be obtained by computing the logarithmic ratios defining the pixel attenuation values as explained further above. The attenuation data and / or transmission matrices may be obtained from particle physics simulation, e.g., by using open source platforms / toolkits for the simulation of the passage of particles through matter (e.g., the Geant4 platform).

[0098] In a eighteenth aspect, the invention relates to an image generation system configured to carry out the steps of the image generation method according to the seventeenth aspect or any variant thereof. In an nineteenth aspect, the invention relates to a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the image generation method according to the seventeenth aspect or any variant thereof.

[0099] In a twentieth aspect, the invention relates to a computer-readable data carrier having stored thereon the computer program of the seventeenth aspect.

[0100] According to any aspect of the present invention, the length or number of components of an input feature vector may be less than 100, less than 50, or even less than 20. The number of pixels of any image of the first and / or second image type or of any object image according to any aspect of the present invention may be larger than 10000, larger than 100000, or even larger than 1000000.

[0101] It is emphasized that the features of the various aspects of the invention can be combined with each other in embodiments of the invention. For example, features of images of the first and second image type, object images, target images, input feature vectors, segmentation results, classification results, image segmentation module(s), classifier(s) etc. as well as steps of a method and components of a system according to any aspect of the invention as described further above can be combined with each other and can be combined with features of any other aspect of the invention.

[0102] In summary, efficient methods and systems are provided to enable the training of powerful models (custom Al-type) that can be used, e.g., for the inspection of very diverse products, particularly in food and beverage. The methods and systems relate to synthetic generators, image classification and real-time self-monitoring for auto-calibration of the models and systems. This is an important capability to maintain the lifecycle performance of an Al monitoring system in a dynamic production environment. In particular, the methods and systems can be used to detect foreign object contaminants of a specific type from the inspected products, avoiding confusion with innocent features shown in an image.

[0103] The proposed methods and systems require only limited amount of uncontaminated data for training. Because of the proposed generators, it is possible to train models using a limited (e.g., < 500) number of uncontaminated product images. Large volumes / sizes of training data can still be generated without the use of manual labelling, which has proven to be time consuming and error prone. The proposed solutions allow training more quickly to a high level of recall and precision with minimal labelling requirements, improving detection efficiency especially in the difficult inspection tasks and ensuring lifelong reliability after deployment, even in the face of changing parameters. More specifically, a recall of larger or equal to 90% can be achieved at a false positive rate of 0. Also, a recall larger or equal to 98% can be achieved at a false positive rate smaller than 10%. In particular, such favorable performance can be achieved for contaminants commonly regarded difficult or impossible to detect such as rubber, ceramics and lightweight glass.

[0104] The training can be achieved with modest computing resources and without necessarily depending on generative adversarial networks. Customizable learning models are trained, rather than using pre-trained general-purpose solutions. The generators enable training of a potentially complex and powerful deep learning segmentation model. Pre-trained solutions (like VGG16 or 19) are less reliable and accurate, especially with non-standard image domains like dual-energy or multi-spectral / energy attenuation images.

[0105] The methods and systems may also take advantage of all information in dual-energy or multi-spectral / energy photon counting images. Pre-trained models are typically trained on large datasets of standard images in the visible spectrum, not attenuation values coming from photon-counting cameras or detectors. Moreover, prior known models and systems often assume or even require either one color channel (grayscale) or three color channels (RGB) in the visible range, but are not flexible to admit to larger set of possible channels.

[0106] According to the proposed methods and systems the output of a segmentation model is turned into higher-level statistics, which are then used to make classification predictions. This two-step approach is very flexible and allows for the incorporation of prior physics knowledge.

[0107] The proposed solutions can also be used to limit the number of false positives of the classification. False positives severely limit the applicability of anomaly detection models in a high-throughput production environment. By having final classification decisions being made by results obtained from a simple and tunable classifier using higher-level statistics, it becomes possible to self-monitor and calibrate the proposed methods and systems in order to limit the probability of a false positive event, something which would be very hard, if not impossible, to do with a deep-learning-only classification model, where images are classified directly in one step.

[0108] Training and monitoring can be automated. In particular, it is not necessary to store large amounts of data when performing model calibration. Tuning, adaptation and retraining can be accomplished by generating and saving small, higher-level statistics or small input feature vectors, not whole images.

[0109] Furthermore, only little Al expertise and / or human intervention is needed to build or tune the proposed models. Floor technicians that manually segment or label data as would be the case if the pipeline was based solely on uncontaminated data become obsolete.

[0110] Moreover, a major limitation of an anomaly detection algorithm is that it is very hard, if not impossible, to determine if such a model is working correctly. It may be possible to set up an alarm that is triggered by too many positives (which either indicates multiple contaminated products or else a very high false positive rate), but there is no equivalent test that determines whether the model is losing sensitivity, nor real-time metrics for quantifying model sensitivity to different kinds of contaminants. The proposed method and systems overcome this limitation of the known prior art.

[0111] According to the proposed methods and systems continuously higher-level statistics can be collected for uncontaminated product images on the belt during deployment. Synthetically contaminated product images can be generated from those scans, the images themselves however do not need to be stored in a long-term memory, thereby reducing computational resource requirements.

[0112] Detailed Description

[0113] Exemplary embodiments of the invention are illustrated in the drawings and will now be described with reference to figures 1 to 9.

[0114] In the figures:

[0115] Fig. 1 shows steps of an image generation method,

[0116] Fig. 2 shows images of the first and second image type and object images,

[0117] Fig. 3 shows an image generation and training system,

[0118] Fig. 4 shows steps of a training method,

[0119] Fig. 5 shows an image segmentation module, feature extraction module and classifier,

[0120] Fig. 6 shows steps of a classification method,

[0121] Fig. 7 shows a classification apparatus,

[0122] Fig. 8 shows steps of a self-monitoring classification method,

[0123] Fig. 9 shows a self-monitoring classification system. In the following embodiments, we discuss methods and systems applicable in the field of product inspection, in particular methods and systems for detecting contaminants or defects in products. Thus, in the following an image of the first image type is also called an uncontaminated product image that shows a product without a contaminant of a contaminant type. An image of the second image type is a contaminated product image that shows a product with a contaminant of the contaminant type.

[0124] Specifically, a product is a food or beverage product and a contaminant is a food or beverage contaminant. Example products include sliced ham, bags of pistachio or pots of yogurt. Example contaminants comprise aluminium particles, Teflon particles, bone particles etc.

[0125] However, it is emphasised that the discussed methods and systems are not limited to this application and can also be applied in other fields.

[0126] Figure 1 shows an embodiment of the image generation method. The image generation method is used to generate a second set of images of a second image type SET2 from a first set of images of a first image type SET1 and object images OBJ as shown in Figure 2. For that purpose, an image generation system as shown in Figure 3 is employed.

[0127] Specifically, the image generation method as shown in Figure 1 comprises obtaining G1 the first set of images of a first image type SET1. For that purpose, each image of the first set SET1 of images is taken by an imaging apparatus 3. The image generation method also comprises obtaining G2 object images OBJ. Each object image OBJ shows an object of at least one object type and is also taken by the imaging apparatus 3.

[0128] The second set of images of the second image type SET2 is generated synthetically. Each image of the second set SET2 is generated synthetically by combining an image of the first set of images SET1 with at least one of the object images OBJ. The synthetically generated images of the second set SET2 are then used for training an image segmentation module and a classifier for classifying a target image 1 taken by the imaging apparatus 3 into a first image type or a second image type as discussed further below.

[0129] Recurring features are provided in the following figures with identical reference signs as in Figure 1.

[0130] Figure 2 shows examples of images of the first set SET1, object images OBJ and images of the second set SET2. Each image of the first set SET1 is an uncontaminated product image and shows a (training) product without any contaminants. Each object image OBJ shows an isolated object (training object) that is a contaminant. In Figure 2, contaminants with triangular, pentagon and circular shape are shown. Each image of the second set SET2 is a contaminated product image and shows a (training) product with a (training) object, i.e., a product with a contaminant.

[0131] For each image of the first set of images SET1 a plurality of images of the second set SET2 are generated by varying in the plurality of images of the second set SET2 randomly the position of one contaminant taken from one object image OBJ (in Figure 2, for each image of the first image type two images of the second image type are shown, but more are actually generated). Specifically, the image of the first set SET1 is combined randomly with the object image OBJ such that the contaminant is located in each image of the second set SET2 at a random pixel position.

[0132] In particular, the combining an image of the first set SET1 with one of the object images OBJ comprises adding according to a weighted sum model pixel attenuation values of the image of the first set SET1 and pixel attenuation values of the contaminant of the object image OBJ with which it is combined. The adding of the pixel attenuation values is explained in more detail further below.

[0133] It should be realized that in an alternative embodiment, also different object images OBJ may be combined with one or a plurality of different images of the first set SET1 to randomize the images of the second set of images SET2.

[0134] Recurring features are provided in the following figures with identical reference signs as in Figure 2.

[0135] Figure 3 shows an image generation and training system.

[0136] Target products 6.1, training products 6.2 and training objects 6.3, i.e., training contaminants, are provided. The target products 6.1 are (prior unknown) products 6.1 for which it is a priori unknown whether these products comprise a contaminant or not and that shall thus be classified. It is thus a-priori unknown whether the target images corresponding to the target products 6.1 are contaminated product images or uncontaminated product images.

[0137] The training products 6.2 and the training objects 6.3 (training contaminants) are known, selected and prepared for imaging and training purposes. The training products 6.2 have been selected to not comprise a contaminant. Examples of training products 6.2 include different products of sliced ham, bags of pistachio or pots of yogurt. Typical thickness of a training product 6.2 is between 2 to 5 centimeters, but the thickness may also range up to 10 cm or 15 cm. In the case of inspecting loose, non-packaged, multi packaged or bulk packaged products, the bed depth for inspection may also be much thicker. Training products may be packaged in many materials including glass jars, aluminium or steel cans and may include liquid beverages.

[0138] The training objects 6.2 have been prepared and selected to correspond to isolated contaminants (without any product). Example training contaminants 6.3 include stainless steel balls, stain less steel wire, aluminium particles, PTFE Teflon particles, Bone particles, lightweight glass, ceramic and / or plastics, e.g., rubber, nylon, acetal or HDPE etc. Contaminants of various shapes and sizes are being prepared as training objects 6.3. A typical size (e.g., thickness or diameter) of a contaminant ranges from 0.1 to 5 mm. The proposed methods and systems are particularly suited for plastics as a contaminant type.

[0139] The image generation system comprises an imaging apparatus 3 configured to take the first set of images of the first image type SET1 of the training products 6.1 for training, to take the object images OBJ of the training objects 6.3 for training and to take (prior unknown) target images 1 of the target products 6.1 for classification. Each image of the first set SET1 shows another training product 6.2. Each image of the object images OBJ shows another training object 6.3.

[0140] The imaging apparatus 3 is an X-ray imaging apparatus. Specifically, the imaging apparatus 3 is an X-ray camera. The X-ray camera is a dual-energy and photon-counting X-ray camera. One advantage of using a photon-counting camera is the ability to maintain a high efficiency of photon detection across a wide range of energies. Moreover, high spatial and spectral resolution (corresponding to the true energy distribution / spectrum of the photons) can be obtained from very low soft x-ray energies, e.g., 6 keV which provides high contrast between lightweight and similar density materials, thus enhancing probability of detection. The X-ray camera has a line-scan configuration with a width of 600 pixels, where the length can vary between 300 and 800 pixels depending on the product type. Each pixel is about one third of a millimeter, so an 800 times 600 image has a size of about 266 mm times 200 mm. Sometimes a contaminant (e.g. a training object 6.3) occupies only a single pixel. Therefore, a classification of uncontaminated product images and contaminated product images can become very challenging, in particular when the background comprises a complex-shaped product like a pistachio.

[0141] The X-ray camera 3 has camera settings related to the energy threshold(s) used for binning of the received / detected photons. The dual-energy X-ray camera has two energy channels corresponding to a low energy channel and a high energy channel. The two energy channels are converted to three pixel attenuation values for each pixel corresponding to the number of photons detected at that pixel in the low energy channel, the high energy channel and the total energy channel, wherein the total energy channel comprises the first and second energy channel. Consequently, for each 800 pixel times 600 pixel image, the image comprises about 1.44 million pixel attenuation values (800*600*3). Storing a few thousand of those images, even in a compressed format, would thus take about 5GB of memory.

[0142] The X-ray camera 3 is configured such that each pixel (i,j) of an image taken by the imaging apparatus 3 comprises (or corresponds to) the pixel attenuation vector atj formed from a discrete set of three possible pixel attenuation values corresponding to the low energy channel with attenuation value (1), high energy channel with attenuation value (2) and total-energy channel with attenuation value cti7(3) for the transmitted electromagnetic radiation 4.1 that arrives at the imaging apparatus 3.

[0143] The first set of images SET1, the object images OBJ and the target image 1 are taken by the X-ray camera 3 with the same camera settings to ensure that the images of the first set SET1 and the object images OBJ from which the images of the second set SET2 are generated and used for training during a training phase are configured like the (prior unknown) target images 1 taken by the X-ray camera 3 for classification during a deployment phase.

[0144] The image generation system additionally comprises a radiation device 4 configured to emit electromagnetic radiation 4.1 towards the target products 6.1 for taking the target images 1, towards the training products 6.2 for taking the first set of images SET1 and towards the training objects 6.3 for taking the object images OBJ.

[0145] The radiation device 4 is a polychromatic X-ray source configured to emit X-ray radiation 4.1. The target images 1, first set of images SET1 and the object images OBJ are X-ray images. Since the second set of images SET2 is generated by adding X-ray pixel attenuation values, the images of the second set SET2 can also be considered synthetically generated X-ray images. Specifically, the pixel attenuation values of the images of the first set SET1 and of the object images OBJ correspond to X-ray attenuation values obtained from absorption of the electromagnetic radiation 4.1 of the X-ray source in a transmission configuration as shown in Figure 3.

[0146] The X-ray source is an X-ray tube. The X-ray tube 4 is arranged in a fixed positional relationship with respect to the imaging apparatus 3. The X-ray tube 4 emits the electromagnetic radiation 4.1 in a continuous polychromatic frequency spectrum typically in the range of 5keV up to 160keV and sometimes up to 300keV. The energy of the photons and the photon flux is controlled by the electrical control parameters of the X-ray tube 4, i.e., electrical voltage and current.

[0147] The image generation system additionally comprises a conveyer 6 configured to provide the target products 6.1 for taking target images 1 of the target products 6.1 by the imaging apparatus 3. The conveyer 6 is also configured to provide the training products 6.2 for taking the first set of images SET1 of the training products 6.2 by the imaging apparatus 3 and to provide the training objects 6.3 of at least one object type for taking the object images OBJ of the training objects 6.3 by the imaging apparatus 3.

[0148] The conveyer 6 comprises a conveyer belt that is configured to transport products and objects towards an illumination region. Specifically, the target products 6.1, the training products 6.2 and the training objects 6.3 are moved and positioned by the conveyer 6 between the radiation device 4 and the imaging apparatus 3 for imaging in transmission configuration. For that purpose, the conveyer 6, the target products 6.1, the training products 6.2 and the training objects 6.3 are positioned between the X-ray source 4 and the X-ray camera 3.

[0149] Moreover, the target products 6.1, the training products 6.2 and the training objects 6.3 are transported by the conveyer 6 through the illumination region illuminated by the electromagnetic radiation 4.1 emitted by the radiation device 4 at the same conveyer speed so that said target products 6.1, training products 6.2 and training objects 6.3 are illuminated substantially by the same amount of electromagnetic radiation 4.1 during imaging by the imaging apparatus 3.

[0150] In particular, it is noted that the faster the products and objects move through the illumination region the less photons can pass through any given region of said products and objects, thereby affecting the quality of the (higher-level) attenuation statistics. It has been found that keeping the same camera settings and the same conveyer speed during training and deployment can considerably improve the classification. In particular, if there are inconsistencies in the X-ray camera setup between imaging training products 6.2 and training objects 6.3, or between training and deployment, there can also be significant differences between the training data and the deployment data, i.e., between training and target images. These differences can negatively impact the performance of the image segmentation module and the classifier.

[0151] The image generation system also comprises a hybrid synthetic generator 2.4 configured to generate the second set of images of the second image type SET2, wherein each image of the second set SET2 is generated synthetically by combining an image of the first set SET1 with at least one of the object images OBJ as explained further above.

[0152] Specifically, the pixel attenuation value of the image of the first set SET1 and a pixel attenuation value of the object image OBJ corresponding to the object of the at least one object type are added according to a weighted sum model to obtain a pixel intensity value of an image of the second set SET2. The weighted sum model amounts to a good approximation based on the following arguments:

[0153] Assuming for demonstration purposes only that electromagnetic radiation emitted from an X-ray tube would be monochromatic and a number of N initial photons is released from an X-ray tube to arrive at a target pixel of an imaging apparatus. Assuming further that a product without a contaminant is arranged between an X-ray tube and an imaging apparatus so that a smaller number of photons n actually arrives and is detected at the target pixel, because N-n photons have been absorbed along the way by an uncontaminated product, then the attenuation value of the target pixel in the final image would be log N / n.

[0154] Assuming further that an isolated contaminant instead of the uncontaminated product is arranged in between the X-ray tube and the imaging apparatus and would absorb N-k photons corresponding to an attenuation value log N / k. If said isolated contaminant would be combined with the product, e.g., such that the contaminant is placed on top of the product, then n of the original N photons would be transmitted through the product and (n k / N) photons of those would be transmitted through the contaminant as well and finally arrive at the target pixel. Thus, of the original N photons only nk / N photons arrive at the target pixel corresponding to an attenuation value log N / (^) = logN2 / nk = log N / n + log N / k, i.e., the sum of the attenuation values of the uncontaminated product and the (isolated) contaminant.

[0155] This observation is related to the Beer-Lambert law, which strictly speaking is valid for monochromatic radiation. To be more precise, a simple sum, i.e., a weighted sum with weights=l, for adding the attenuation values would count all photons equally regardless of energy levels / bins / channels. However, the X-ray source as part of the image generation, training and classification method is a polychromatic (or multi-spectral) source and photon energies can also change and even shift when passing through a material (e.g., due to beam hardening). The simple sum of attenuation values is thus not exact for the low and high energy channels of the images considered here, but it is observed that it still can result in a good and efficient approximations even in the low and high energy ranges. Moreover, with a photon-counting camera 3 comprising two or more energy channels, the energy range of interest can be selected to optimize the approximation (e.g., independent of other spectral effects).

[0156] It should also be clear from these considerations that the image generation method and the hybrid synthetic generator described in this embodiment with an addition of attenuation values is based on the laws of attenuation and transmission physics and hence does not apply to RGB images in reflection geometry and / or the visible spectrum.

[0157] Finally, it is pointed out that a further improvement of the approximation can be achieved when the simple sum X+Y of attenuation values X,Y is replaced by a weighted sum X*wl + Y*w2 of attenuation values X, Y with weights wl,w2, wherein the weights wl, w2 can be learned by imaging contaminated products to obtain contaminated product images with the prescribed imaging apparatus 3, radiation device 4 and belt 6 (rather than generating those synthetically) and comparing the resulting attenuation values with the attenuation values of uncontaminated product images and object images taken in the same manner. This can be done for each product and / or object type and / or energy bin / channel separately and different weights can be obtained for each combination of these types and / or bins / channels.

[0158] Figure 3 also comprises a smart module 2 and a training module 7. The smart module comprises the image segmentation module 2.1, the feature extraction module 2.2 and the classifier 2.3 (shown in Figure 5). The training module 7 is configured to train the image segmentation module 2.1 and the classifier 2.3. In alternative embodiments, the smart module and the training module 7 as well as the hybrid synthetic generator 2.4 can also be part of the imaging apparatus 3.

[0159] Recurring features are provided in the following figures with identical reference signs as in Figure 3.

[0160] Figure 4 shows an embodiment of a training method. Figure 5 shows a smart module 2 that is trained with the training method.

[0161] The training method comprises training (or pre-training) the image segmentation model 2.1 with a segmentation training set comprising images of the second image type. The images of the second image type in the segmentation training set are generated synthetically according to the image generation method described in relation to Figures 1 to 3. For examples, the segmentation training set may comprise the second set SET2 or a portion thereof as described in relation to Figures 1 to 3.

[0162] The image segmentation module 2.1 is trained and thus configured to provide a segmentation result that can be used to locate a contaminant in a product. Specifically, the segmentation result comprises pixel-by-pixel probabilities Py, wherein a pixel-by-pixel probability Ptj is a probability that a pixel with row index i and column index j of an image shows a contaminant (assuming a two-dimensional image, where an image pixel row index i and an image pixel column index j identify a specific pixel).

[0163] The image segmentation model 2.1 is a deep learning model for performing binary segmentation (here, binary segmentation is considered part of multiclass segmentation, i.e., classifying into more than two image types comprises a classification into a first and a second image type). In an alternative embodiment, the image segmentation model 2.1 can additionally be configured to identify the contaminant type. This is not discussed here any further for the brevity of the presentation.

[0164] In the following, the image segmentation model 2.1 is a convolutional neural network model. Specifically, the convolutional neural network model comprises the U-Net architecture. The U-Net model has depth 2 with 38207 parameters, 8 initial features, downsampling and up-sampling resolution of 2, an attendant doubling or halving of the number of features by a factor of 2 and a filter size of (3,3). This architecture incorporates dropout (with rate 0.05), attention mechanisms, and residual connections. The learning rate is set to 0.01 initially, using the Adam optimizer, binary cross entropy loss, with learning rate annealing on plateaus and automated saving of best models based on validation loss.

[0165] The first set of images of the first image type SET1 for training the image segmentation module 2.1 comprises 1200 uncontaminated product images. Between 50 and 100 epochs are performed. Each time, between 1 and 20 more images of the second image type of the second set SET2 are generated synthetically for each first set of images SET1 of the first image type by combining each image of the first set SET1 randomly with an object image OBJ, i.e., an object of the object type shown in the object image OBJ. Each epoch corresponds to another randomization seed of the random number generator used to randomize the pixel location and the taken object images for the image synthesis. In total, the image segmentation training set comprises about 100.000 distinct images of the second set SET2 of the second image type.

[0166] Once the image segmentation model 2.1 is trained, it will give a segmentation result that corresponds to a segmentation or pixel-by-pixel classification. While this is valuable, it is not the ultimate goal, which is to classify (target) images as a whole. Therefore, here a two-step model is proposed in which the segmentation is used as a basis for the classification. Training a (pure) classification model directly with training images of the second image type, e.g., the second set SET2, is very challenging because of the small size of the contaminants. For example, a 1 mm contaminant takes up about 9 pixels, which is less than 0.002% of a 600 x 800 image, and it is not reasonable to expect a deep learning model to make a correct decision (especially with zero false positives) for a (target) image based on such subtle differences. Secondly, deep learning models are often black boxes, whereas the proposed classifier is lightweight, interpretable and easily tunable. Thirdly, if one attempts to train a deep learning model to classify images directly, it may only identify contaminants of the contaminant types, e.g., contaminant geometry and / or shapes, produced by the hybrid synthetic generator for the segmentation training set.

[0167] In order to overcome these limitations, we suggest a combination of an image segmentation model 2.1 and a lightweight classifier 2.3, wherein the image segmentation model 2.1 finds all the contaminated pixels, and then a very flexible classifier 2.3 performs classification without being bothered at least directly by contaminant geometry.

[0168] The training method thus also comprises training the classifier 2.3 using the (pre-) trained image segmentation module 2.1.

[0169] Specifically, the training method comprises obtaining T1 images of the first image type taken by the imaging apparatus 3 for training the classifier 2.3, each image of the first image type showing a training product 6.2. For example, the images of the first image type used to train the classifier 2.3 may correspond to at least a portion of the first set SET1 shown in Figure 2.

[0170] The training method also comprises obtaining T2 images of the second image type for training the classifier 2.3. The images of the second image type can also be taken by the imaging apparatus 3. For that purpose, additional training products that show pre-selected products contaminated with an object of the at least one object type, i.e., with a contaminant, would be provided for taking the images of the second image type by the imaging apparatus 3. However, here we will consider the case where the images of the second image type used to train the classifier 2.3 are also generated synthetically according to the image generation method described in relation to Figures 1 to 3. For example, the images of the second image type used to train the classifier 2.3 may be at least a portion of the second set SET2 shown in Figure 2.

[0171] Step T3 comprises generating first segmentation results for the images of the first image type obtained in step Tl. Each first segmentation result is generated by applying the trained image segmentation module 2.1 to an image of the first image type obtained in step Tl. Step T4 comprises generating second segmentation results for the images of the second image type obtained in step T2. Each second segmentation result is generated by applying the trained image segmentation module 2.1 to an image of the second image type obtained in step T2.

[0172] The training method further comprises generating T5 first input feature vectors Fk, wherein each first input feature vector Fkcomprises higher-level statistics obtained from a (different) first segmentation result of the generated first segmentation results and generating T6 second input feature vectors Fk, wherein each second input feature vector comprises higher-level statistics obtained from another second segmentation result of the second segmentation results.

[0173] The higher-level statistics are obtained from the pixel-by-pixel probabilities of the respective segmentation result and the pixel intensity values (here attenuation values) of the respective image from which the respective segmentation result has been obtained.

[0174] The higher-level statistics comprise a sum over weighted pixel-by-pixel probabilities of the segmentation result, a maximum pixel-by-pixel probability of the segmentation result, the number of pixels with a pixel-by-pixel probability of the segmentation result larger than a predetermined threshold, and sums over pixel-by-pixel attenuation values weighted by pixel-by-pixel probabilities of the segmentation result.

[0175] Specifically, the first element (or component) of an input feature vector corresponds to the sum over pixel-by-pixel probabilities of the segmentation result:

[0176] F = ,ij Pij (i-e., the total sum of pixel-by-pixel probabilities).

[0177] The second element corresponds to the maximum pixel-by-pixel probability of the segmentation result:

[0178] F2= max Ptj (i.e., a maximum pixel-by-pixel probability). tj

[0179] The third element corresponds to number of pixels with a pixel-by-pixel probability of the segmentation result larger than a predetermined threshold: i.e., the third element corresponds to the number of pixels with a probability higher than Pth (corresponding to the number of contaminated pixels). Here, a (pre-determined) threshold probability Pth has been introduced for deciding what pixel-by-pixel probability is high enough to label a pixel as contaminated. For example, Pth=0.5.

[0180] The fourth to sixth elements (or components) correspond a sum over pixel-by-pixel attenuation values of the target image 1 weighted by pixel-by-pixel probabilities of the segmentation result i.e., the sum over all the pixel attenuation values and vectors in the image weighted by their probability of showing a contaminant (i.e., a probability-weighted sum of image statistics).

[0181] The seventh to ninth element corresponds to a sum over weighted pixel-by-pixel attenuation values weighted by certain pixel-by-pixel probabilities: 7:9 = ijCPij > Pth) l: 3), i.e., the sum over all the pixel attenuation values / vectors in the image for those pixels whose probability of showing a contaminant is larger than the threshold 0.5 (i.e., a sum of image statistics among contaminated pixels)

[0182] The tenth to twelfth element corresponds to: i.e., a probability-weighted maximum of image statistics.

[0183] The thirteenth to fifteenth element corresponds to: i.e., the maximum of image statistics among contaminated pixels.

[0184] Consequently, the length of an input feature vector, i.e., the first and second input feature vectors, is 15. As shown in Figure 4, the first and second input feature vectors are generated by the feature extraction module 2.2 configured to generate an input feature vector from a segmentation result as described further above.

[0185] The training method finally comprises training T7 with a classification training set the classifier 2.3 for classifying a target image 1 into a first image type or a second image type, wherein the classification training set comprises the first input feature vectors and the second input feature vectors.

[0186] In the classification training set, the first input feature vectors are associated with a first label indicating the first image type. The second input feature vectors are associated with a second label indicating the second image type. Specifically, the classifier 2.3 generates a classification result for an input feature vector, i.e., whether the input feature vector to-be-classified has been generated from a first segmentation result obtained from an image of the first image type (first input feature vectors with first label) or from an image of the second image type (second input feature vectors with the second label).

[0187] However, since the classifier 2.3 takes as an input an input feature vector of length 15 (comprising high level statistics of a segmentation result of an original image) rather than the entire original image (whose feature vector would have a length corresponding to the number of pixel attenuation values of said image), the classifier 2.3 can be considered a lightweight classifier.

[0188] Here, the classifier 2.3 is a logistic regression model. As an example, 1200 images of the first image type are taken by the imaging apparatus 3 as uncontaminated product images for generating 1200 first input feature vectors in the classification training set for training the classifier 2.3. For each uncontaminated product image 10 contaminated product images are generated synthetically as described further above giving 12000 images of the second image type for generating 12000 second input feature vectors in the classification training set for training the classifier 2.3.

[0189] The training system used to carry out the training method comprises the training module 7. The training module is configured to carry out the steps of the training method as explained further above. The training system also comprises the imaging apparatus 3, the conveyer 6, the radiation source 4, the smart module 2 and the hybrid synthetic generator 2.4 as described in relation to Figures 3 and 4. The smart module 2 comprises the image segmentation module 2.1, the feature extraction module 2.2 and the classifier 2.3 shown in Figure 5. In alternative embodiments, the smart module 2 may be part of the imaging apparatus 3 or may be a software module running on an external device with established data links to the imaging apparatus 3.

[0190] Recurring features are provided in the following figures with identical reference signs as in Figures 4 and 5.

[0191] Figure 6 shows steps of a classification method for classifying target images 1 into the first or second image type using the trained smart module 2, i.e., the trained image segmentation module 2.1 and the trained classifier 2.3.

[0192] The classification method comprises obtaining CO target images 1 by taking an image of a target product 6.1 with the imaging apparatus 3. It is prior unknown whether the target product 6.1 comprise a contaminant or not, i.e., whether the corresponding target image 1 is of the first or second image type.

[0193] The classification method further comprises generating Cl a segmentation result by applying the trained image segmentation module 2.1 to the target image 1. The classification method further comprises generating C2 an input feature vector based on the segmentation result, wherein the input feature vector comprises higher-level statistics obtained from the segmentation result; and generating C3 a classification result based on the input feature vector by applying the trained classifier 2.3 to the input feature vector. The input feature vector has been generated by the feature extraction module 2.2.

[0194] Since the trained classifier 2.3 has been trained to provide a classification result that is indicative of whether the input feature vector obtained from the target image 1 has been generated based on a segmentation result obtained from an image of the first image type or from an image of the second image type, the target image 1 can be classified into the first or second image type based on the classification result. The classification method thus further comprises classifying the target image based on the classification result.

[0195] Figure 7 shows the classification system classifying the target image 1. The classification system comprises the imaging apparatus 3, the radiation device 4, the conveyer 6 and the trained smart module 2 shown in Figures 3 and 5.

[0196] The trained image segmentation module 2.1 is configured to generate the segmentation result for the target image 1. The feature extraction module 2.2 is configured to generate an input feature vector based on the segmentation result for the target image 1, wherein the input feature vector comprises the higher-level statistics obtained from the segmentation result as described further above. The trained classifier 2.3 is then configured to generate a classification result for the input feature vector based on the segmentation result for the target image 1. The trained classifier 2.3 is further configured to classify the target image 1 based on the classification result.

[0197] In general, once a learning model is deployed, it can become very difficult to evaluate its performance, especially when there is no natural process producing the true labels for the model's predictions, as it is the case, e.g., in industrial inspection. Since the performance of a learning model is expected to vary, and often deteriorate, with time, such a monitoring is very important.

[0198] In the absence of direct performance metrics, proxy metrics are often used. For example, if a model predicts that 1 in every 10 products is contaminated, giving a positive rate (false or otherwise) of 0.1, this might trigger an alarm indicating suspicious model behavior. Indeed, either the model can be correct, and there are many, many contaminants on the conveyer belt, or else it is producing a high rate of false positives. Other proxy metrics detect shifts in the input data or the distribution of the model's predictions, and when these metrics pass an established threshold an alarm is triggered, so that an Al expert is brought in to try and determine if model performance is indeed compromised.

[0199] The challenge with using proxy metrics is that they can be indirect and crude, so that they can only detect certain kinds of changes in the data or the model, that they are often very abstract and non-intuitive (how do you get a good measure of whether or not a new image looks unusual relative to older images).

[0200] Figure 8 shows an extension of the classification method discussed in relation to Figure 6 which overcomes the aforementioned limitations. The extension comprises self-monitoring steps that can be used to retrain the classifier 2.1 and thus improve the performance of the classification.

[0201] The self-monitoring classification method shown in Figure 8 is for self-monitoring the performance of classifying target images 1 into a first image type or a second image type.

[0202] It comprises classifying each target image 1 of a plurality of target images 1 into the first image type or the second image type according to the classification method as discussed further above in relation to Figure 6 with the same trained image segmentation module 2.1 and the same trained classifier 2.3 during a deployment phase.

[0203] Based on a classification result it is first determined whether the target image 1 corresponds to an image of the first image type, i.e., an uncontaminated product image (see "1sttype?" in Figure 8).

[0204] If this is not the case, the classification method continues with classifying the next target image 1 during the deployment phase, i.e., the classification method is repeatedly carried out for the next target image 1 on the belt conveyer 6 and it is again determined whether said next target image 1 is of the first image type.

[0205] If it is the case that the (original) target image 1 is of the first image type, the following selfmonitoring steps are being carried out. Note, that it would also be possible in an alternative embodiment to carry out the following self-monitoring steps only for every second or third or Nth target image 1 that is being classified as being of the first image type (where N is an integer N>0). Self-monitoring steps:

[0206] For the target image 1 classified as being of the first image type a plurality of test images TEST of the second image type are generated synthetically based on said target image of the first image type 1 by using the image generation method and the hybrid synthetic generator 2.4 as discussed in relation to Figures 1 to 3 (see "Generate 2ndtype" in Figure 8).

[0207] Specifically, the target image 1 classified as being of the first image type is combined synthetically with additional object images OBJ to obtain the plurality of test images TEST, wherein each additional object image OBJ shows one training object 6.3 of the object type, i.e., a contaminant. The additional object images OBJ can be taken by the imaging apparatus 3 in advance, stored in a data repository and be obtained during the deployment phase and self-monitoring from the data repository.

[0208] For each test image TEST of the second image type an additional (second) segmentation result is obtained by applying the trained image segmentation module 2.1 to the test image TEST. An additional (second) input feature vector based on the additional segmentation result is obtained by applying the feature extraction module 2.2, wherein the additional input feature vector comprises higher-level statistics obtained from the additional segmentation result as explained further above.

[0209] The additional input feature vectors obtained from the test images TEST are stored with the second label, i.e., as belonging to or being indicative of the second image type. For the target image 1 that has been classified as being of the first image type, the corresponding input feature vector is stored with the first label, i.e., as corresponding to or being indicative of the first image type (see " Store Fk's " in Figure 8).

[0210] It is emphasized that only modest storage space is required for storing the input feature vectors of the first and second label because of their small length and thus a large number of input feature vectors can be stored.

[0211] As mentioned further above, the self-monitoring steps discussed so far can be carried out for each target image 1 that is being classified as being of the first image type or only after predetermined intervals. Once enough additional input feature vectors are being generated and stored according to a predetermined criterion, the performance of the trained classifier 2.3 can be checked based on this stored data.

[0212] In particular, for each stored input feature vector a classification result is obtained by applying the trained classifier 2.3 to the stored input feature vector. Note, that it would also be possible to apply the classifier 2.3 immediately to each additional input feature vector once said additional input feature vector is generated and the corresponding classification result could be stored as well in an alternative embodiment before checking the performance metric.

[0213] The classification results generated for the stored input feature vectors are then compared to the stored labels of the stored input feature vectors and a performance metric is determined based on that comparison (see "Check Performance" in Figure 8).

[0214] The performance metric is a precision and recall score. Specifically, it is determined how many false positives have been determined by the trained classifier 2.1 for the stored additional input feature vectors.

[0215] If the performance metric meets a predetermined threshold criterion ("Good" in Figure 8) the trained classifier 2.3 is kept and additional target images 1 are classified using the (unmodified) trained classifier 2.3 during the deployment phase and the above-mentioned self-monitoring steps are repeatedly carried out. Additionally, the precision and recall scores are stored in a report.

[0216] If the performance metric does not meet the predetermined threshold criterion ("Bad" in Figure 8) a retraining of the trained classifier 2.3 is performed and additional target images 1 are classified using the (improved) re-trained classifier 2.3.

[0217] More specifically, the classification of the classifier 2.3 is carried out by training the classifier 2.3 with a retraining set comprising the stored input feature vectors and their labels. This can be achieved by using the training method as described in relation to Figure 4.

[0218] Alternatively, another independent new training set can be used for retraining the classifier 2.3 using the afore-mentioned training method. The retrained classifier 2.3 can then be tested again based on the stored input feature vectors and retraining can be continued with additional (re-)training data until the target false positive rate is achieved, i.e., until the performance of the retrained classifier 2.3 meets the predetermined threshold criterion.

[0219] The predetermined threshold criterion comprises a maximum percentage of false positives. For example, according to the threshold criterion it may be insisted that on all contaminated target products, one should have a recall about 90% with a false positive rate no higher than 0.5%.

[0220] Figure 9 shows a self-monitoring classification system configured to carry out the steps of the self-monitoring classification method according to Figure 7. The self-monitoring classification system is substantially identical to the classification system shown in Figure 7, but additionally comprises the hybrid synthetic generator 2.4 and the training module 7 from Figure 3. The hybrid synthetic generator 2.4 is used to generate the additional test images TEST and the training module 7 is used to retrain the classifier 2.3 in the smart module 2.

[0221] It is emphasized that according to the self-monitoring classification method and system described above, it is not necessary to retrain the image segmentation module 2.1 to achieve a performance improvement, but rather only the classifier 2.3 can be retrained which is much more efficient due to the small length of the input feature vectors and the associated small size of the retraining classification set. However, it is of course also possible to additionally retrain the image segmentation module 2.1 as well.

[0222] In summary, in the usual operation of the classification model / system during the deployment on the belt, one can scan / image target products 6.1 and obtain their segmentation results and input feature vectors that are then used to decide if the target product 6.1 is contaminated or not. In the self-monitoring classification method / system the hybrid synthetic generator 2.4 is incorporated into the pipeline, so that the classification method / system declares that a product is uncontaminated, its input feature vector is stored, and a number of synthetically contaminated product image versions of this product are generated, for which the input feature vectors are also collected. Since each input feature vector is quite small (e.g., has a length less than 20) relative to the size of a full image (e.g., over IM values), the memory footprint or storage requirements are very small. In this way, large datasets of labelled input feature vectors are generated from (presumably) uncontaminated product images and (synthetically) contaminated product images, so that for each input feature vector we know if the original image comprised a contaminated product or not, and if so, with what contaminant type and geometry.

[0223] Here, the method / system parameters that can be predetermined comprise a frequency of generating synthetic contaminated product images (e.g., for every (target) image classified as uncontaminated product image, for every ten such target images, etc.), the number of contaminated product images to be generated synthetically per original uncontaminated product (target) image, and / or a duration for the deployment phase during which the selfmonitoring is performed and input feature vectors are collected (e.g., 12 hours, 24 hours etc). These parameters can be determined by the conveyer belt speed and the computational limits of the classification system. Ideally, at least 10,000 input feature vectors are stored before proceeding to the performance check. Once the labelled input feature vectors have been collected, it is determined whether the current classifier 2.3 as a logistic regression model and the predetermined threshold used to decide on the type of input feature vector (i.e., whether it is based on an image of the first or second image type) provides good precision and recall metrics. Good here means that the precision and recall scores exceed a metric threshold decided in advance, e.g., as depending on the needs of the customer (or related risk measure such as severity times frequency). For example, it may be decided that one should have a recall of about 90% with a false positive rate no higher than 0.5%. If the results are good, the precision and recall scores are stored in a report and the current monitoring cycle is terminated. If the results are no longer good, the classifier 2.3 is retrained the logistic regression model is tested on new data and set the threshold correspondingly to obtain the target false positive rate. The precision and recall scores are stored in a report. The retraining of the logistic regression model can also be called recalibration, since the underlying segmentation model 2.1 remains fixed.

[0224] In an alternative embodiment, e.g. in case for the retrained logistic regression model the target recall scores can still not be obtained, the underlying segmentation model 2.1 may additionally be improved and retrained. Retraining the segmentation model 2.1 comprises generating and saving new, contaminated product images, and so in a fully automated pipeline the failure of the recalibrated model to achieve good recall scores triggers the collection of new contaminated product image data which is then transferred to the cloud and used to retrain the new segmentation model 2.1.

[0225] Features of the different embodiments which are merely disclosed in the exemplary embodiments as a matter of course can be combined with one another and can also be claimed individually.

Claims

Claims1. A classification method for classifying a target image (1) into a first image type or a second image type, the method comprising: generating (Cl) a segmentation result by applying a trained image segmentation module (2.1) to the target image (1), generating (C2) an input feature vector based on the segmentation result, wherein the input feature vector comprises higher-level statistics obtained from the segmentation result; generating (C3) a classification result based on the input feature vector by applying a trained classifier (2.3) to the input feature vector, wherein the trained classifier (2.3) has been trained to provide a classification result that is indicative of whether the input feature vector has been generated based on a segmentation result obtained from an image of the first image type or from an image of the second image type; classifying (C4) the target image (1) based on the classification result.

2. The classification method of claim 1, wherein an image of the first image type shows a product of a product type that does not comprise an object of at least one object type and an image of the second image type shows a product of the product type with an object of the at least one object type.

3. The classification method of claim 2, wherein the object is a contaminant and the at least one object type is a contaminant type and the method is used for detecting contaminants in products.

4. The classification method of claim 2 or 3, wherein the image segmentation module (2.1) has been trained to provide a segmentation result for locating an object of the at least one object type in an image of the second image type.

5. The classification method according to any of the preceding claims, wherein the length of the input feature vector is smaller than 100.

6. The classification method of any of claims 2 to 5, wherein the segmentation result comprises pixel-by-pixel probabilities, wherein a pixel-by-pixel probability is a probability that a pixel of the target image (1) shows at least a part of an object of the at least one object type.

7. The classification method of claim 6, wherein the higher-level statistics are obtained from the pixel-by-pixel probabilities of the segmentation result and / or from pixel-by- pixel attenuation values obtained from the target image (1).

8. The classification method according to claims 6 or 7, wherein the input feature vector and / or the higher-level statistics comprise at least one of a sum over pixel-by-pixel probabilities of the segmentation result, a maximum pixel-by-pixel probability of the segmentation result, the number of pixels with a pixel-by-pixel probability of the segmentation result larger than a predetermined threshold, a sum over pixel-by-pixel attenuation values obtained from the target image (1) weighted by pixel-by-pixel probabilities of the segmentation result.

9. A classification system for classifying a target image (1) into a first image type or a second image type, the classification system comprising: a trained image segmentation module (2.1) configured to generate a segmentation result for the target image (1); a feature extraction module (2.2) configured to generate an input feature vector based on the segmentation result, wherein the input feature vector comprises higher-level statistics obtained from the segmentation result; a trained classifier (2.3) configured to generate a classification result based on the input feature vector, wherein the trained classifier (2.3) has been trained to provide a classification result that is indicative of whether the input feature vector has been generated based on a segmentation result obtained from an image of the first image type or from an image of the second image type; and the trained classifier (2.3) is further configured to classify the target image (1) based on the classification result.

10. The classification system of claim 9, wherein the classification system further comprises an X-ray imaging apparatus (3) configured to take the target image (1) for classification and the target image (1) is an X-ray image.

11. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the classification method according to claim 1.

12. A computer-readable data carrier having stored thereon the computer program of claim 11.

13. An image generation method comprising: obtaining (Gl) a first set of images of a first image type (SET1), wherein each image of the first set (SET1) of images is taken by an imaging apparatus (3); obtaining (G2) object images (OBJ), wherein each object image (OBJ) shows an object of at least one object type; generating (G3) a second set of images of a second image type (SET2), wherein each image of the second set of images (SET2) is generated synthetically by combining an image of the first set of images (SET1) with at least one of the object images (OBJ); and wherein at least the synthetically generated images of the second set (SET2) are used for training a classification system for classifying a target image (1) taken by the imaging apparatus (3) into a first image type or a second image type.

14. The image generation method of claim 13, wherein each object image (OBJ) is taken by the imaging apparatus (3).

15. The image generation method of claim 13 or 14, wherein each image of the second set (SET2) shows at least one object of at least one object type and no image of the first set (SET1) shows an object of the at least one object type.

16. The image generation method of claim 13 or 14 or 15, wherein the combining an image of the first set of images (SET1) with at least one of the object images (OBJ) comprises adding according to a weighted sum model a pixel attenuation value of a pixel in the image of the first set of images (SET1) and a pixel attenuation value of a pixel in the at least one object image (OBJ) to obtain a pixel attenuation value of a pixel in the image of the second set of images (SET2), wherein the pixel in the at least one object image (OBJ) belongs to the object of the at least one object type shown in the at least one object image (OBJ).

17. The image generation method of any of claims 13 to 16, wherein the combining an image of the first set of images (SET1) with at least one of the object images (OBJ) comprisesgenerating a random superposition of the image of the first set (SET1) and the object image (OBJ) to generate an image of the second set of images (SET2), wherein an object of the at least one object type shown in the object image (OBJ) is superposed with the image of the first set (SET1) at a random pixel location.

18. The image generation method of any of claims 13 to 17, wherein for each image of the first set of images (SET1) a plurality of images of the second set (SET2) are generated by varying the position and / or orientation of at least one object of the at least one object type from the at least one object image (OBJ) among the plurality of images of the second set (SET2) generated from the image of the first set of images (SET1); and / or a plurality of images of the second set (SET2) are generated by combining each image of the first set of images (SET1) with a plurality of objects of the at least one object type and / or a plurality of object images (OBJ) such that for each image of the first set (SET1) of images a plurality of images of the second set (SET2) is generated.

19. An image generation system comprising an imaging apparatus (3) configured to take a first set of images of a first image type (SET1); take object images (OBJ), wherein each object image shows an object of at least one object type; take a target image (1) for classification; a hybrid synthetic generator (2.4) configured to generate a second set of images of a second image type (SET2), wherein each image of the second set (SET2) is generated synthetically by combining an image of the first set (SET1) with at least one of the object images (OBJ); and wherein at least the synthetically generated images of the second set (SET2) are used for training a classification system for classifying the target image (1) into a first image type or a second image type.

20. The image generation system according to claim 19, wherein the imaging apparatus(3) comprises a camera and the first set of images (SET1), the object images (OBJ) and the target image (1) are taken by the camera with the same camera settings.

21. The image generation system of claim 19 or 20, wherein the system additionally comprises a conveyer (6) configured to provide a target product (6.1) for taking the target image (1) of the target product (6.1) by the imaging apparatus (3); provide training products (6.2) for taking the first set of images (SET1) of the training products (6.2) by the imaging apparatus (3); provide training objects (6.3) of at least one object type for taking the object images (OBJ) of the training objects (6.3) by the imaging apparatus (3), a radiation device (4) configured to emit electromagnetic radiation (4.1) towards the target product (6.1) for taking the target image (1) by the imaging apparatus (3), towards the training products (6.2) for taking the first set of images (SET1) by the imaging apparatus (3), and towards the training objects (6.3) for taking the object images (OBJ) by the imaging apparatus (3).

22. The image generation system of claim 21, wherein the first set of images (SET1) and the object images (OBJ) are taken by the imaging apparatus (3) in a transmission configuration and / or the target product (6.1), the training products (6.2) and the training objects (6.3) are positioned by the conveyer (6) between the radiation device(4) and the imaging apparatus (3) for imaging in transmission configuration.

23. The image generation system of claim 21 or 22, wherein the target product (6.1), the training products (6.2) and the training objects (6.3) are transported by the conveyer (6) through an illumination region illuminated by the electromagnetic radiation (4.1) emitted by the radiation device (4) at the same conveyer speed so that the target product (6.1), the training products (6.2) and the training objects (6.3) are illuminated substantially by the same amount of electromagnetic radiation (4.1) during imaging by the imaging apparatus (3).

24. The image generation system of any of claims 19 to 23, wherein the imaging apparatus (3) is an X-ray imaging apparatus; and / or the electromagnetic radiation (4.1) is X-ray radiation; and / or the imaging apparatus (3) comprises a photoncounting camera; and / or each pixel of an image of the first set (SET1) and an object image (OBJ) comprises a plurality of pixel attenuation values corresponding to different energy channels of the electromagnetic radiation (4.1).

25. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the image generation method according to claim 13.

26. A computer-readable data carrier having stored thereon the computer program of claim 25.

27. A training method comprising: obtaining (Tl) a first set of images of a first image type (SET1); obtaining (T2) a second set of images of a second image type (SET2); generating (T3) first segmentation results for the first set of images (SET1), wherein each first segmentation result is generated by applying a trained image segmentation module (2.1) to another image of the first set (SET1); generating (T4) second segmentation results for the second set of images (SET2), wherein each second segmentation result is generated by applying the trained image segmentation module (2.1) to another image of the second set (SET2); generating (T5) first input feature vectors, wherein each first input feature vector comprises higher-level statistics obtained from another first segmentation result of the generated first segmentation results; generating (T6) second input feature vectors, wherein each second input feature vector comprises higher-level statistics obtained from another second segmentation result of the second segmentation results; training (T7) with a classification training set a classifier (2.3) for classifying a target image (1) into a first image type or a second image type, wherein the classification training set comprises the first input feature vectors and the second input feature vectors.

28. The training method of claim 27, wherein the first set of images (SET1) used to generate (T3) the first segmentation results for training are taken by the same imaging apparatus (3) used to take the target image (1) for classification.

29. The training method of claims 27 or 28, wherein the length of each of the first and second input feature vectors is smaller than 100.

30. The training method of any of claims 27 to 29 further comprising training with a segmentation training set the image segmentation module (2.1) before generating the first and second segmentation results, wherein the segmentation training set comprises at least images of the second image type.

31. A training system, wherein the training system comprises a training module (7) configured to carry out the steps of the training method according to claim 27.

32. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the training method according to claim 27.

33. A computer-readable data carrier having stored thereon the computer program of claim 32.

34. A self-monitoring classification method for self-monitoring the performance of classifying target images (1) into a first image type or a second image type, the selfmonitoring classification method comprising: classifying each target image (1) of a plurality of target images (1) into the first image type or the second image type according to the classification method of claim 1 with the same trained image segmentation module (2.1) and the same trained classifier (2.3) during a deployment phase; and carrying out the following self-monitoring steps during the deployment phase: determining a performance metric for the trained classifier (2.3) after at least one target image (1) of the plurality of target images (1) is classified; and if the performance metric does not meet a predetermined threshold criterion retraining the trained classifier (2.3).

35. The self-monitoring classification method of claim 34, wherein the determining a performance metric for the trained classifier (2.3) during the deployment phase comprises: generating at least one additional test image (TEST) of the second image type based on the at least one target image (1) and at least one additional object image (OBJ), wherein the at least one additional object image (OBJ) shows at least one object(6.3) of an object type; generating based on the at least one additional test image (TEST) at least one additional classification result by using the trained classifier (2.3); determining the performance metric for the trained classifier (3) based on the at least one additional classification result.

36. A self-monitoring method according to claim 35, wherein the generating at least one additional classification result by the trained classifier(2.3) based on the at least one additional test image (TEST) comprises: generating at least one additional second segmentation result by applying the trained image segmentation module (2.1) to the at least one additional test image (TEST); generating at least one additional second input feature vector based on the at least one additional second segmentation result, wherein the at least one additional second input feature vector comprises higher-level statistics obtained from the at least one additional second segmentation result; generating the at least one additional classification result by applying the trained classifier (2.3) to the at least one additional input feature vector.

37. A self-monitoring classification system configured to carry out the steps of the selfmonitoring classification method of claim 34.

38. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the self-monitoring classification method according to claim 34.

39. A computer-readable data carrier having stored thereon the computer program of claim 38.

40. An image generation method comprising: obtaining a first set of images (SET1) of a first image type, wherein each image of the first image type shows a product of a first product type; obtaining a second set (SET2) of images of a second image type, wherein each image of the second image type shows a product of a second product type; and wherein the first (SET1) and / or second (SET2) set of images are used for training an image segmentation module (2.1) and / or a classifier (2.3) for classifying a target image (1) taken by an imaging apparatus (3) into the first image type or the second image type; and wherein the obtaining a first set of images (SET1) comprises generating each image of the first set (SET1) synthetically by simulating computationally the propagation of electromagnetic radiation (4.1) from a radiation device (4) through a product of the first product type (6.1, 6.2) to the imaging apparatus (3); and / or the obtaining a second set of images (SET2) comprises generating each image of the second set (SET2) synthetically by simulating computationally the propagation of electromagnetic radiation (4.1) from a radiation device (4) through a product of the second product type (6.1, 6.2) to the imaging apparatus (3).

41. An image generation method according to claim 40, wherein simulating computationally the propagation of electromagnetic radiation (4.1) through the product (6.1, 6.2) comprises: obtaining an initial spectral distribution of the electromagnetic radiation (4.1) as emitted by the radiation device (4); obtaining a three-dimensional voxel model of the product (6.1, 6.2), wherein the voxel model comprises a voxel thickness and a voxel material for each voxel; obtaining from a material database attenuation data for each voxel of the voxel model based on the voxel thickness and the voxel material of each voxel; obtaining based on the initial spectral distribution of the electromagnetic radiation (4.1) and the attenuation data for each voxel an estimate of pixel-by-pixelattenuation values corresponding to the part of the electromagnetic radiation (4.1) transmitted through the product (6.1, 6.2) and detected at the imaging apparatus (3).

42. An image generation method according to claim 41, wherein the attenuation data for each voxel is a transmission matrix.

43. An image generation method according to claim 42, wherein the obtaining based on the initial spectral distribution of the electromagnetic radiation (4.1) and the attenuation data for each voxel an estimate of pixel-by-pixel attenuation values corresponding to the part of the electromagnetic radiation (4.1) transmitted through the product (6.1, 6.2) and detected at the imaging apparatus (3) comprises: multiplying the initial spectral distribution with a matrix product of transmission matrices, wherein each transmission matrix in the matrix product corresponds to a different voxel along a transmission path of electromagnetic radiation (4.1) through the three-dimensional voxel model and towards the imaging apparatus (3).

44. An image generation method according to any of claims 40 to 43, wherein a product of a first product type is a product lacking a contaminant of a contaminant type and a product of a second product type is a product comprising at least one contaminant of the contaminant type.

45. An image generation system configured to carry out the steps of the image generation method of claim 40.

46. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the image generation method according to claim 40.

47. A computer-readable data carrier having stored thereon the computer program of claim 46.

Images