Method and system for detecting anomalies in images

Abstract

This disclosure relates to a method for detecting anomalies in images. The method includes: a step of obtaining a set of negative samples, each negative sample comprising at least a portion of an image (71) that does not contain anomalies; a step (S12) of training a processing chain (21) using the set of negative samples; and a step of processing an unknown image (S14) using the trained processing chain (21) to determine whether the unknown image contains anomalies. In the disclosed method, the step of training the processing chain (21) (S12) includes: a step of obtaining an array of one-dimensional feature sets using a pre-trained deep encoder, each one-dimensional feature set in the array representing the features of the corresponding negative sample from the set of negative samples; a step (S64) of performing a reduction analysis, particularly principal component analysis, on the array of one-dimensional feature sets to obtain a corresponding array of reduced feature sets; and a step (S65) of training a one-class model based on the array of reduced feature sets. This disclosure further relates to a corresponding anomaly detection system.

Description

Technical Field

[0001] The present disclosure relates to a method for detecting anomalies in images and a corresponding anomaly detection system.

Background Art

[0002] Generally, computer-based automated image analysis, and particularly the detection of anomalies in images, is an important aspect in many fields of technology. For example, in autonomous driving, the detection of anomalies, i.e., unknown objects or foreign matter on the driving surface, is important to trigger warnings and / or initiate emergency measures. Other examples that rely on automated anomaly detection are medical diagnostic systems, such as the detection of anomalies in tissue samples, and the detection of defects in manufactured articles, such as non-destructive testing.

[0003] As a specific example, it is desirable to detect defects in laminated wood, i.e., the decorative strip of laminated wood applied to the upper surface of a wood-based workpiece or the edge of the workpiece. Such laminates or other decorative surfaces can have frequently changing designs. Furthermore, due to the large surface area generated and the frequently changing desired patterns, it may not be practical to manually inspect each manufactured surface.

[0004] So far, the automatic detection of anomalies has typically been based on traditional image processing methods or highly specialized algorithms. These approaches often require manual adaptation and depend on the quality and quantity of available template data. Although less common, supervised AI-based learning models have been adopted for anomaly detection. However, in this case, a large dataset of anomalies has been required to effectively train the model.

[0005] One conventional approach is to use image analysis algorithms that allow objects to be separated from their backgrounds. One example is the so-called Otsu thresholding, which is included by default in many image processing tools and libraries. This method is described in detail in Non-Patent Document 1. [Non-Patent Document 1] "A Threshold Selection Method from Gray-Level Histograms", IEEE Transactions on Systems, Man, and Cybernetics

[0006] One concrete example of an AI-based learning model is the use of so-called support vector machines (SVMs) and supervised learning for anomaly detection. SVMs are trained using both positive and negative sample images to detect anomalies in normal operation.

[0007] Neither of the above methods is suitable for use in manufacturing environments where the desired surface texture changes frequently, and at the same time, the exact nature and type of anomalies detected within the desired surface are partially or completely unknown. [Overview of the project] [Problems that the invention aims to solve]

[0008] This disclosure aims to solve the problem of detecting surface anomalies more efficiently and / or more precisely. Preferably, it should be suitable for scenarios where only limited training data is available, and / or where the exact nature and type of anomalies to be detected on the desired surface are partially or completely unknown.

[0009] Therefore, the challenge is to provide improved methods and systems for detecting anomalies in images that can be rapidly deployed in manufacturing environments, and that do not require the provision of extensive training sets and / or the adaptation of algorithms under human supervision. [Means for solving the problem]

[0010] In this context, an improved method for detecting anomalies is disclosed. The method includes the steps of: obtaining a set of negative samples, each negative sample containing at least a portion of an image that does not contain anomalies; training a processing chain using the set of negative samples; and processing an unknown image using the trained processing chain to determine whether the unknown image contains anomalies. The step of training the processing chain includes: obtaining an array of one-dimensional feature sets using a pre-trained deep encoder, each one-dimensional feature set in the array representing the features of the corresponding negative sample from the set of negative samples; performing a reduction analysis, in particular principal component analysis, on the array of one-dimensional feature sets to obtain a corresponding array of reduced feature sets; and training a one-class model based on the array of reduced feature sets.

[0011] The disclosed method employs and combines several different techniques to significantly simplify the amount of training required to train an AI-based anomaly detection system that has a high detection rate and a low false-positive rate. In particular, the disclosed method enables the implementation of a fully automated, rapid training phase based on the provision of negative samples only, i.e., images that do not contain anomalies.

[0012] In particular, the above advantages are achieved by using a pre-trained deep encoder to obtain a one-dimensional feature array from a given sample. This essentially reduces the input value domain from a two-dimensional pixel domain to a one-dimensional feature domain. Furthermore, in subsequent reduction analysis, the one-dimensional feature domain can be further reduced to a reduced set of features particularly relevant to a given target domain, i.e., the pattern contained in the image being monitored. Such reduction analysis, especially principal component analysis, can be performed based on a relatively small training set and greatly increases the significance of the extracted reduced set of features describing each image. Based on this reduced set of features, it is possible to successfully train one or more one-class models in a relatively short time, which can be used to distinguish between images containing anomalies and images without anomalies.

[0013] It is worth noting that the processing chain used during processing unknown images is essentially the same as the processing chain used during training. Therefore, the key difference between the two operating phases is that the samples provided to the processing chain, as well as the weights and other initialization parameters calculated during the training phase, are fixed during subsequent image classification.

[0014] Optionally, each negative sample corresponds to a training patch having a predefined target size, where the predefined target size corresponds to the size of the training image used during the pre-training of the deep encoder. The step of acquiring a set of negative samples includes: acquiring a sample image that does not contain abnormalities; dividing the sample image into multiple training patches of a predefined target size; and selecting at least a portion of the training patches as negative samples. The step of processing an unknown image includes: dividing the unknown image into multiple target patches of a predefined target size; processing each target patch using a trained processing chain to determine whether that target patch contains abnormalities; and determining that the unknown image does not contain abnormalities only if all of the target patches are determined to be non-abnormal.

[0015] By separating images into patches with predefined target sizes used during the pre-training of deep encoders, the processing power required to implement deep encoders is significantly reduced. Furthermore, a relatively large number of negative samples can be generated from a limited number of training images.

[0016] Optionally, the step of obtaining a set of negative samples includes: providing a pool of negative samples, in particular the multiple training patches detailed above; selecting an initial subset, in particular a random subset, from the pool of negative samples; pre-training a treatment chain using the initial subset to obtain a pre-trained treatment chain; processing the remaining samples from the pool of negative samples not included in the initial subset using the pre-trained treatment chain to determine whether each of the remaining samples is classified as abnormal; and selecting all samples classified as containing abnormalities by the pre-trained treatment chain as part of a set of negative samples for training the treatment chain.

[0017] The two-stage selection of negative samples described above significantly improves the detection rate of the trained treatment chain. In particular, the first training stage allows for a rapid acquisition of a pre-trained treatment chain using a relatively small number of randomly selected negative samples. This pre-trained treatment chain can then be used to quickly validate its accuracy for a specific application domain by classifying the remaining samples. Any false positives—samples that are derived from images without abnormalities but are considered to contain abnormalities—can be used to retrain the treatment chain, thereby significantly improving its accuracy for a specific problem domain. It should be noted that no human intervention is required in either training stage, as it is clear from the outset that each patch provided to the treatment chain during training represents a negative sample.

[0018] Optionally, the selected set of negative samples for training the treatment chain may include an additional initial subset. Including an additional random set of negative samples avoids excessive bias in the trained treatment chain towards problematic image fragments.

[0019] Optionally, training the processing chain further includes the steps of: generating a set of false-positive samples based on a set of negative samples, each false-positive sample being generated by modifying at least one of the negative samples; and fine-tuning a pre-trained deep encoder using a set of negative samples labeled as negative and a set of false-positive samples labeled as positive.

[0020] While the use of pre-trained deep encoders significantly speeds up the extraction of potentially relevant features from generic images acting as input to the processing chain, they may not be tailored to a specific area of ​​application or problem domain. Therefore, by automatically generating false-positive samples using intentionally modified, i.e., distorted versions of samples such as anomaly-free images or image patches, the deep encoder can be fine-tuned to select a feature set specific to a given problem domain.

[0021] For example, noise can be added to each negative sample to obtain the corresponding false positive sample. This can be easily done, for example, using the corresponding filter in an image processing library.

[0022] Optionally, during the processing chain training stage, multiple one-class models are trained. Each one-class model is trained independently based on a reduced feature set. The stage of processing an unknown image includes: obtaining a reduced feature set corresponding to the unknown image; processing the reduced feature set separately using at least a subset of the multiple one-class models; and determining whether the unknown image contains anomalies based on the outputs provided by the one-class models used.

[0023] The use of multiple one-class models further improves the detection accuracy of the disclosed processing chain. In particular, not all types of one-class models are suitable for detecting all types of possible anomalies in the problem domain. Combining the outputs of multiple one-class models may further reduce the overall detection accuracy.

[0024] Optionally, each one-class model of a plurality of one-class models is configured to output a binary value indicating whether an unknown image contains an anomaly. The output binary values of the one-class models used are combined using, in particular, majority voting, minority voting, or consensus decision-making to detect whether the unknown image contains an anomaly.

[0025] Depending on the specific application domain, the outputs of the plurality of one-class models can be combined in an appropriate way using one of the above decision criteria. For example, in the most restrictive consensus or hard voting decision, the output is considered clear only if all the one-class models used agree on a specific detection result. This could mean that a sample is considered to contain no anomaly only if all the one-class models used indicate the absence of an anomaly. Conversely, if at least one one-class model indicates the presence of an anomaly, the sample is considered to contain an anomaly regardless of the classification results of all the other one-class models. In less critical applications, less restrictive majority or minority voting decisions can be used to sort samples that may contain an anomaly and / or samples that are likely to contain no anomaly. For example, a single model indicating an anomaly may be overridden by several other models indicating the absence of an anomaly.

[0026] Optionally, if a one-class model of a plurality of one-class models indicates that an unknown image contains an anomaly, the decision that the unknown image contains an anomaly is made, and processing the reduced feature set using any of the remaining one-class models is stopped or omitted after that decision has been made.

[0027] The above implementation may be particularly suitable in high-speed manufacturing or other detection environments where samples are checked based on, for example, a live video stream of production output being filmed. In this situation, it may be easier to discard or sort samples that are potentially abnormal, such as products, rather than performing extensive and potentially long analysis on each sample.

[0028] Alternatively, the reduced feature set is processed separately using each one-class model of a plurality of one-class models. Each one-class model of the plurality of one-class models outputs the probability (also called likelihood) that an unknown image contains an anomaly. To determine whether an unknown image contains an anomaly, the output probabilities of all one-class models are combined, particularly by calculating the average likelihood.

[0029] The above alternative makes it possible to quantitatively determine the probability that a processed image contains an anomaly by considering the output of each one-class model used. In this way, one or more application domain-specific cut-off points or thresholds can be configured for detecting samples that are positive, negative, and potentially between clear positives and clear negatives.

[0030] Optionally, the plurality of one-class models includes at least one of an one-class vector support machine (OC-SVM) and an isolation forest. The OC-SVM estimates the support of a high-dimensional distribution for unsupervised outlier detection. The isolation forest is an ensemble of "isolation trees" that "isolate" observations by recursive random partitioning and can be represented by a tree structure. The number of partitions required to isolate a sample is lower for outliers and higher for normal values. The above AI-based one-class (classification) model is particularly suitable for detecting image anomalies by identifying outliers with respect to a known set of negative sample images.

[0031] This disclosure further includes an anomaly detection system comprising a deep encoding subsystem, a reduction subsystem, a classification subsystem, and a training subsystem. The deep encoding subsystem is configured to extract a one-dimensional feature set from at least a portion of an image. The reduction system is configured to obtain a reduced feature set based on reduction analysis of multiple one-dimensional feature sets, particularly principal component analysis. The classification system is configured to process the reduced feature set and implement at least one one-class model that outputs a value indicating whether an unknown image processed by the deep encoding subsystem and the reduction subsystem contains anomalies. The training subsystem is configured to initialize the reduction subsystem and train at least one one-class model based on a set of negative samples encoded by the deep encoding subsystem. Each negative sample comprises at least a portion of an image that does not contain anomalies.

[0032] The anomaly detection system described above is suitable for implementing previously used methods for detecting anomalies in images. In particular, the deep encoding subsystem, reduction subsystem, and classification subsystem can be used to implement the processing chain described above. Note that the training subsystem uses the other subsystems to initialize and train them during the training phase.

[0033] Optionally, the deep encoding subsystem includes a neural network pre-trained using a training set containing multiple different training images selectively labeled with multiple different classification terms. Such a pre-trained neural network is suitable for extracting relevant features from any image input to the deep encoding subsystem. Furthermore, such a pre-trained neural network is readily available, for example, based on a network trained using a pre-tagged selection of images used for performance benchmarking in automated object detection.

[0034] Optionally, the training subsystem is specifically configured to perform the steps of: extracting a one-dimensional feature set from each negative sample in a set of negative samples to obtain an array of one-dimensional feature sets, where each one-dimensional feature set in this array corresponds to one of the negative samples in the set of negative samples; and performing a reduction analysis, in particular principal component analysis, on the array of one-dimensional feature sets to construct a transformer of the reduction subsystem, in particular a transformation matrix [translation matrix], for converting the one-dimensional feature sets into corresponding reduced feature sets.

[0035] It should be noted that principal component analysis (PCA) and similar statistical methods for reduction analysis essentially perform the computation of a transformation matrix, for example, based on eigenvector analysis of the input matrix. The computed transformation matrix, which translates the (input) array of one-dimensional feature sets to the corresponding (output) array of the reduced feature sets, can later be used during the normal operation of the anomaly detection system to transform the output of a deep encoder to the expected input for the classification subsystem.

[0036] Optionally, the training subsystem is configured to further compute a reduced feature set for each of the one-dimensional feature sets of the array using a reduction subsystem configured with a transformation matrix, thereby obtaining multiple reduced feature sets; and to train the at least one one-class model based on the multiple reduced feature sets, which act as negative samples for training.

[0037] It should be noted that negative samples used during training can be used twice. Firstly, they are used to compute the transformation matrix as described above to select the most relevant features from a potentially large feature set extracted by the deep encoding subsystem. Secondly, the reduced feature set corresponding to each sample used during training is also used to train each one-class model. Thus, the disclosed anomaly detection system limits both the number of training samples required and the processing effort during training. Furthermore, positive samples are not required for training.

[0038] Another aspect of this disclosure provides an alternative method for training a one-class model for detecting anomalies in images. This method includes: providing a pool of negative samples, each negative sample comprising pixel data of an image that does not contain anomalies; performing an initial training phase; and performing at least one further training phase. Performing the initial training phase includes: selecting a first subset of negative samples from the pool of negative samples, the first subset of negative samples being smaller than the pool of negative samples; converting the pixel data of each negative sample in the first subset into a corresponding first subset of a first one-dimensional data structure that forms a first training set; and pre-training a one-class model using only the first training set as negative samples. Performing the aforementioned at least one further training phase includes: selecting a second subset of negative samples from a pool of negative samples, wherein the second subset of negative samples includes at least one negative sample not included in the first subset; converting the pixel data of each negative sample in the second subset into a corresponding second one-dimensional data structure; processing the resulting second one-dimensional data structure using a pre-trained one-class model to determine whether each sample is classified as abnormal; including each sample classified as abnormal and / or the corresponding second one-dimensional data structure in a second training set; and training the one-class model using the second training set as false-positive samples.

[0039] The above steps implement a two-stage training method or algorithm. In the first stage, a one-class model is trained on a relatively small set of negative samples, for example, randomly selected. In the second stage, the one-class model is retrained using a more sophisticated or challenging training set of negative samples, including false positives that were misidentified based only on the initial training in the first stage. Thus, hard mining of highly relevant training samples can be implemented.

[0040] Optionally, converting pixel data of a negative sample into a corresponding one-dimensional data structure includes: a step of using an encoder on the pixel data to obtain a one-dimensional feature set, where each one-dimensional feature set represents the features of the corresponding negative sample; and a step of using a reduction subsystem or transformer, in particular a reduction or transformation matrix, to reduce the one-dimensional feature set to obtain a one-dimensional data structure smaller than the one-dimensional feature set.

[0041] Optionally, the initial training phase includes: a step of obtaining an array of first one-dimensional feature sets using an encoder, where each first one-dimensional feature set in the array represents the features of the corresponding negative samples from a first subset of negative samples; and a step of performing a reduction analysis, in particular principal component analysis, on the array of one-dimensional feature sets to initialize a reduction subsystem or transformer.

[0042] Optionally, the at least one further training phase includes: a step of obtaining a second sequence of second one-dimensional feature sets using an encoder, wherein each second one-dimensional feature set in the second sequence exhibits features of the corresponding negative samples from a second subset of negative samples; and a step of performing a reduction analysis, in particular principal component analysis, on the second sequence of one-dimensional feature sets to reinitialize a reduction subsystem or transformer.

[0043] The above method for training a one-class model for detecting anomalies in an image may be used in combination with, or implemented within, a method for detecting anomalies in an image and a corresponding anomaly detection system.

[0044] The following describes various aspects and implementations of the anomaly detection method and system described above with reference to an attached set of figures that will help to better understand the present invention. While several specific embodiments of the disclosed method and system are presented, it should be noted that this disclosure is not limited to the embodiments described. Instead, the present invention is defined by the attached claims. [Brief explanation of the drawing]

[0045] [Figure 1] This document describes a method for detecting anomalies in images. [Figure 2] This shows an anomaly detection system with a processing chain and a training subsystem. [Figure 3] This illustrates the various phases of training an anomaly detection system. [Figure 4] This document outlines the method and procedure for obtaining a pre-trained deep encoder. [Figure 5A] This shows the method steps and training phases for mining relevant training samples. [Figure 5B] This shows the method steps and training phases for mining relevant training samples. [Figure 6A] This document describes the methodology, steps, and resulting dataset for training the processing chain of an anomaly detection system. [Figure 6B] This document describes the methodology, steps, and resulting dataset for training the processing chain of an anomaly detection system. [Figure 7]This document presents a system solution architecture for detecting anomalies in a live stream of images acquired in a manufacturing environment. [Modes for carrying out the invention]

[0046] Figure 1 schematically shows a flowchart of a method for detecting anomalies in an image. This method is based on deep learning and therefore includes a learning stage including steps S11 and S12, and a normal operation stage including steps S13 to S15.

[0047] As shown in Figure 1, during the learning or training phase, several training images are acquired at stage S11.

[0048] As explained earlier, in a manufacturing environment, typically only a limited set of images that do not show anomalies are available, or there are no images showing anomalies, or only a very small number are available. Therefore, according to this disclosure, the training images acquired in step S11 do not contain anomalies. For example, in a manufacturing environment, only pictures of the desired product surface that do not have defects or deviations from the expected design may be provided.

[0049] In general machine learning, and especially in relation to anomaly detection, it should be noted that training samples that do not contain the features to be identified, i.e., anomalies, are described as "negative samples." This is because the trained anomaly detection system is not intended to issue a warning when such images are processed. Conversely, training samples that exhibit anomalies are described as "positive samples."

[0050] It is worth noting that most known AI-based classification systems require the provision of both positive and negative samples. The disclosed approach addresses this challenge based on modern deep learning methods, particularly the so-called one-class model (OCM) combined with more traditional machine learning methods.

[0051] This approach does not require extensive training data and, in particular, does not require the provision of positive samples of anomalies, enabling the automatic and reliable detection of surface anomalies or inconsistencies. At the same time, the disclosed method is highly reliable and operates very efficiently, as it leverages the efficiency of a pre-trained encoder and the flexibility of a so-called one-classifier method. Above all, it allows the classification system to be trained in a very short time, i.e., minutes or at most a few hours, and is therefore suitable for applications in manufacturing environments where designs change frequently.

[0052] Therefore, in step S12, a processing chain including a combination of deep learning methods and traditional machine learning methods is trained using the negative samples obtained in step S11. That is, the processing chain is trained using only images that do not contain abnormalities, as will be explained in more detail below with respect to Figures 2 to 6.

[0053] In certain manufacturing environments, a machine may simply be trained based on a sample of the product to be manufactured, or the output of the manufacturing process in the early stages of its operation, where deviations from the intended design are unlikely to be included in the production output. Thus, the training performed in step S12 is unsupervised in the sense of the terminology used in machine learning and requires no additional effort in the intended application domain.

[0054] Once the anomaly detection system and its processing chain are trained in step S12, the anomaly detection system can switch to a normal operating mode that continuously monitors the output of the image stream. For example, it can capture and monitor a stream of digital images acquired by a camera in the same manufacturing environment where the training was performed.

[0055] Therefore, in step S13, one or more target images are acquired. The acquired target images are processed in step S14 using a previously trained processing chain. Processing in this context means that one or more target images or a part thereof are supplied to a previously trained processing chain, and the corresponding outputs of the processing chain are combined if necessary. As a result, for each image supplied to the anomaly detection system, a classification result indicating whether or not the image acquired in step S13 is considered to contain an anomaly is output in step S15.

[0056] Figure 2 provides a more detailed schematic representation of a detection system 20 according to an example of the present disclosure. The detection system 20 includes a processing chain 21 and a training subsystem 22.

[0057] The processing chain 21 comprises a deep encoding subsystem 23, a reduction subsystem 24, and a classification subsystem 25. It should be noted that the processing chain 21 incorporates different approaches adopted from the fields of artificial intelligence and statistics. In particular, the deep encoding subsystem 23 is based on a well-established and highly efficient encoder-decoder network known from the field of object detection. The reduction subsystem 24 can be implemented as a system for performing principal component analysis (PCA), a well-known statistical method for data reduction in multidimensional datasets. Finally, the classification subsystem 25 may be based on one or more so-called one-class classifiers, which are highly flexible and can be trained based only on negative training samples.

[0058] All components of the processing chain 21 can be controlled and coordinated by the training subsystem 22. In particular, the training subsystem 22 uses the deep encoding subsystem 23 to provide sufficient data for performing reduction analysis, and the results are used to configure the reduction subsystem 24 to perform reduction of features corresponding to the target image during the normal operation of the detection system 20. Furthermore, the output of the reduction subsystem 24 during training is further used to train the classification subsystem 24. Thus, the training subsystem 22 does not require any additional processing components in addition to the processing components of the processing chain 21 used during the normal operation of the anomaly detection system 20.

[0059] Between training and normal operation, the processing chain 21 receives image samples at its input 26 and provides classification results at its output 27. As detailed below, the image samples provided at input 26 may be, for example, a complete high-resolution image from a camera, or smaller patches of one or more larger images. Furthermore, the classification results provided at output 27 may be a binary value indicating whether the image sample is likely to contain anomalies, or another value such as a probability value indicating how likely the received image sample is to contain anomalies.

[0060] Figure 3 shows the various phases of the training stage. In the embodiment described, the training stage may be further subdivided into three stages 31 to 33. However, it should be noted that at least the first two parts of the training are optional.

[0061] In the first optional training stage 31, the deep encoding subsystem 23 is pre-trained. Details of this stage are provided below with reference to Figure 4. Pre-training of the deep encoding subsystem 23 ensures that the deep encoding subsystem 23 extracts relevant features from the input samples provided in input 26 for subsequent stages of the processing chain 21.

[0062] In the second optional training stage 32, a suitable set of training samples is selected. This stage is optional if, at least, only a very limited number of samples are available and the entire set could be used for training. However, as will be explained in more detail below with respect to Figures 5A and 5B, a special mining process can be used to select a particularly relevant subset of a larger set of available training samples, which improves both the efficiency of the training phase and the accuracy of the trained anomaly detection system 20.

[0063] In the final stage 33 of the training phase, at least some of the components of the processing chain 21 are trained using previously selected training samples. In particular, one or more OCMs of the classification subsystem 25 may be trained based on encoded image samples. Furthermore, in at least some embodiments, the reduction system 24 may be configured based on reduction analysis performed on the set of training samples selected in stage 32. This will be described in more detail below with reference to Figures 6A and 6B.

[0064] Figure 4 schematically illustrates a method for obtaining a pre-trained deep encoder. In particular, Figure 4 discloses in detail a possible implementation of the first training stage 31 shown in Figure 3.

[0065] For efficiency reasons, a pre-trained encoder network is obtained in step S41. Such an encoder-based network is based on a computer-implemented neural network. This type of pre-trained network is available or can be automatically trained using a tagged image library containing pictures from numerous domains, each tagged with a different keyword. This ensures that the corresponding deep encoder network is adapted to extract features from the input image sufficient to reproduce most of the informational content contained in the image.

[0066] In mathematical terms, such an encoder network reduces a two-dimensional array or grid of pixels to a one-dimensional vector of features that describe the image. Such encoder networks are well-known from the field of automated object detection and will not be discussed in detail here.

[0067] However, it should be noted that such pre-trained deep encoder networks are not specific to a given application domain. Therefore, very reasonable results can be obtained for images acquired from diverse domains using such general pre-trained deep encoder networks.

[0068] In the embodiments described, in an optional step S42, the deep encoder network obtained in step S41 is fine-tuned using several positive and negative sample images. Note that the terms “positive” and “negative” here refer to samples obtained or derived in a specific application, i.e., a given application domain. This ensures that, after fine-tuning, the encoder network extracts features that are particularly relevant to the given application domain.

[0069] As far as negative image samples are concerned, these can be obtained directly from input 26 during the training stage of the anomaly detection system 20 in step S43.

[0070] However, as described above, there are no positive samples available for the anomaly detection system 20 in the intended application domain. Therefore, in step S44, one or more of the available negative samples obtained in step S43 are modified to obtain a corresponding artificial positive sample in step S44. Such modifications may include, in particular, the addition of noise, or the intentional manipulation, exclusion, or replacement of segments within a given negative sample image.

[0071] Figure 5A schematically illustrates a method for performing training sample mining. In particular, Figure 5A discloses in detail a possible implementation of the second training stage 32 in Figure 3. Furthermore, Figure 5B shows a corresponding example of training sample mining.

[0072] Figures 5A and 5B disclose a mining technique for selecting a set of training samples to rapidly and efficiently train the classification subsystem 25. While the described mining technique is particularly useful for the anomaly detection system 20 shown in Figure 2, it should be noted that the described mining technique may also be applicable to different classification systems and methods.

[0073] In the first optional step S51, the high-resolution training image obtained from input 26 may be divided into several smaller image patches. In this disclosure, the terms patch or image patch refer to a subsection of a larger image, which in terms of resolution essentially corresponds to the resolution of the training image used during the initial training of the deep encoder network of the deep encoding subsystem.

[0074] For example, a publicly available encoder network may be trained using a sample picture library containing images with resolutions of 256x256 pixels or 224x224 pixels. However, the live camera data obtained at input 26 may have a considerably higher resolution, such as 4K resolution of 3840x2160 pixels, i.e., QHD resolution of 2560x1440 pixels, or HD resolution of 1920x1080 pixels.

[0075] Therefore, in order to simplify the calculations and at the same time increase the number of available training samples, a high-resolution input image of the first resolution may be segmented into several patches having lower resolutions in the first step S51. Preferably, the lower resolutions correspond to the resolutions used during the initial training or fine-tuning of the deep encoder network.

[0076] Therefore, a relatively large set of Patch 50 to serve as negative training samples, for example, 100,000 negative samples as shown in Figure 5B, can be obtained from a relatively small number of input images.

[0077] In step S52, this initial set of training samples may be divided into two or more subsets. Specifically, a relatively small first subset 53 of negative samples may be randomly selected. For example, an image sequence containing k=200 frames may be selected to provide a set of negative training samples. Each frame may be divided into, for example, n=138 patches, from which l=10 are randomly selected. Thus, m=k*l=200*10=2000 negative training samples are selected as the first subset 53 of negative samples. These may be stored in a single location for processing, for example, a folder labeled as negative samples. The remaining negative samples, i.e., the complement of the first subset 53, may be selected as a second subset 54 and may be stored in a different location.

[0078] A first subset of negative samples, 53, is used in step S55 for pre-training of the treatment chain 21. Pre-training of the treatment chain is performed in essentially the same manner as the final training of the treatment chain 21, and is described in more detail below with respect to Figures 6A and 6B.

[0079] In short, in step S55, the reduction system 24 and the classification subsystem 25 are initialized to provide detection results in the output 27 of the processing chain 21 for the image patches provided to the encoding system 23. For this purpose, a first subset 53 of negative samples is encoded using the encoding system 23, and then PCA analysis is performed on the sequence containing the corresponding feature vectors to construct a transformation matrix for the reduction subsystem 24, and the encoded and reduced features of each negative sample from the first subset 53 are further used to train the classification system 25. At this stage, the processing chain 21 is ready for operation.

[0080] Therefore, in step S56, the entire set of samples or patches 50 from the second subset 54 of negative samples can be classified using the pre-trained processing chain 21. For each sample in the second subset 54, the processing chain 21 outputs a value indicating whether the sample is considered to contain abnormalities.

[0081] It should be noted that at this stage, all patches are considered to be negative samples. Therefore, if any sample is shown to contain an abnormality, it represents a so-called false positive in the classification subsystem 25. Thus, a set of false positives 57 is collected, which includes all samples from the second subset 54 of negative samples that were classified in stage S56 as likely to contain an abnormality.

[0082] Furthermore, it should be noted that the set of false positives 57 identified in step S56 corresponds to a patch from the aforementioned relatively large set of training samples that are particularly difficult to identify. Therefore, in order to improve the overall performance of the anomaly detection system 20, in step S58, a first subset 53 containing randomly selected negative samples is combined with the set of false positive samples 57.

[0083] Based on the merged set in step S58, training of the processing chain 21 is repeated in step S59, and the classification subsystem 25 and optionally the reduction subsystem 24 are retrained using a particularly difficult and domain-specific subset of the initial set of training samples. As a result, the anomaly detection system 20 should correctly identify all available training samples as negative after training in step S59.

[0084] As shown in Figure 5A, it should be noted that the above processing steps are performed only once. That is, the only iterative step of the algorithm is performed to obtain a primary set of false positives 57. However, as further indicated by the dashed arrow in Figure 5B, the above negative hard mining process can be iterated sequentially to obtain a secondary set or a higher-order set of false positives, which can then be reused for further training phases to further improve the OCM used.

[0085] Figure 6A schematically illustrates a method for training a processing chain, such as the processing chain 21 in Figure 2, which includes a pre-trained deep encoding subsystem 23, a reduction subsystem 24, and a classification subsystem 25. In particular, Figure 6A provides a more detailed example of an implementation of the third training stage 33 in Figure 3. Note that the steps shown in Figure 6A can be used both during the pre-training of the processing chain 21 in stage S55 and during the final training of the processing chain in stage S59 in Figure 5. Figure 6B shows the resulting datasets during training and subsequent use of the processing chain 21 during anomaly prediction.

[0086] In the first stage S61, the training sample is obtained from each set of training samples, for example, a first set 53 of negative samples, or a set containing false-positive samples 57, or a combination of both.

[0087] Next, the acquired negative training samples are encoded in step S62 by the pre-trained deep encoding subsystem 23 to obtain a complete set of features describing each training sample. Hereafter, the output of the deep encoding subsystem 23 will be described as a first or complete feature vector. Here, the terms "feature vector" and "feature set" are used interchangeably. Thus, in step S62, a two-dimensional pixel bitmap is converted into a one-dimensional first feature vector.

[0088] Steps S61 and S62 are repeated as long as further training samples are available. Thus, in step S63, the method checks for the availability of further samples. For example, in the pre-training of the processing chain 21 in step S55, steps S61 and S62 may be repeated for all 2,000 samples of a first subset 53 of randomly selected negative samples. Thus, the sequence can be satisfied, where each entry in the sequence contains a first feature vector corresponding to one of the training samples from each set of training data.

[0089] It should be noted that this array essentially represents a matrix structure to which conventional statistical methods for data reduction can be applied. Therefore, in step S64, the principal component analysis (PCA) algorithm can be applied to the array of feature vectors to analyze the principal components corresponding to the eigenvectors of the matrix structure. Alternatively, other known algorithms for data reduction can be applied to reduce the relatively large number of features of the first vector to a reduced set of features representing the second feature vector.

[0090] This leads to a reduction in the amount of data considered in later stages of the processing chain 21, but experimental results based on real-world data show that including the reduction stage S64 leads to an improvement in the accuracy of the detection output. This can be caused, among other things, by performing clustering of relevant features and / or avoiding overfitting on a relatively large set of first feature vectors. In other words, the second or reduced feature vectors appear to be particularly relevant for anomaly detection. At the same time, the reduction in the amount of data considered by the OCM of the classification subsystem 25 during training and classification results in a considerable improvement in the performance of the anomaly detection system 20. Nevertheless, stage S64 should be considered optional and can be omitted if the OCM used can process the entire set of features directly.

[0091] As shown in Figure 6B, the deep encoding subsystem 23 can output a first feature vector containing 1280 features for each of m = 2,000 input samples. Thus, the first matrix 68 supplied to the PCA algorithm in step S64 can contain 2000 × 1280 values. These can be analyzed, for example, using PCA to generate a second matrix 69 containing 2000 entries, i.e., a 2000 × 256 matrix, with only 256 features. Note that the remaining 256 features are mathematically nearly independent of each other and should describe features that are particularly relevant for the application domain of the selected training samples.

[0092] Furthermore, depending on the implementation, step S64 may include two distinct sub-steps: namely, the calculation or fitting of a transformation matrix based on a statistical analysis of the first matrix 68, and the calculation of a second matrix 69 based on the first matrix 68 and the transformation matrix, namely the actual transformation of the first matrix 68 to the second matrix 69. However, in practice, both steps may be performed by a single algorithm or module, storing the transformation internally for later use and directly outputting the second matrix 69.

[0093] In the subsequent step S65, one or more one-class models (OCMs) can be trained using the resulting set of features for each training sample, i.e., the complete set of features obtained in step S62 or a reduced set of features obtained in step S64. Note that during this training, each sample is shown as a negative sample, since each training sample is known not to contain abnormalities. Thus, the corresponding classification model or algorithm, such as a one-class support vector machine (OC-SVM) or an isolated forest, may be trained based on the reduced sets of features of the set used for the negative samples.

[0094] Therefore, each OCM should be able to precisely identify samples from the desired application domain that do not contain anomalies. Conversely, anything that falls outside this data range is identified by the OCM as anomaly-containing sample. In Figure 6B, this situation is depicted as a larger input space 90 for the OCM, which includes a smaller subspace 91 representing the area to which the second training vector is mapped by the OCM.

[0095] As shown in step S66, step S65 above may be repeated for multiple models. This has the advantage that weaknesses of a single OCM can be compensated for by comparing its output with the output of another OCM. In the specific embodiment shown in Figure 6B and the architecture described later with respect to Figure 7, the output of the OC-SVM is combined with the output of an OCM based on an isolated forest to obtain a final decision on whether or not the input samples are considered to contain anomalies.

[0096] As further shown on the right side of Figure 6B, the processing chain 21 is operated in a corresponding manner during normal operation, i.e., to predict or detect anomalies in the target image. As illustrated, the target image or frame is decomposed into n0138 target samples 92, which are encoded into a corresponding set of first vectors. The first vectors are then transformed using the transformation module or transformation matrix obtained during the PCA in step S64 to obtain a corresponding second vector. The second vectors are then fed one by one to each of the previously trained OCMs to obtain a corresponding classification result 93 for each target sample 92 and / or OCM.

[0097] Figure 7 shows a specific architecture 70 of an anomaly detection system used in a production environment, for example, the anomaly detection system 20 in Figure 2.

[0098] As shown on the left side of Figure 7, several normal images 71 are first provided as input to the training module 72. The training module 72 then modifies the set of normal images 71 into a corresponding set of modified images 73. Each modified image 73 corresponds to one of the normal images 71 and is generated by adding noise to each normal image 71. For training purposes, the normal images 71 are thought to represent negative samples, while the modified images 73 are thought to represent false positives, as shown. Thus, each image is labeled as negative and positive, respectively, and then fed into the pre-trained encoder network 74.

[0099] In certain solutions, a so-called ResNet-18 encoder network is used. For each of the normal image and the corrected image 73, the pre-trained encoder network 74 generates corresponding feature vectors 75 that enable the distinction between the normal image 71 and the corrected image 73 by a first tag 76 ("bad") corresponding to the positive sample and an alternative second tag 77 ("good") corresponding to the normal image 71.

[0100] At this stage, the pre-trained encoder network 74 is fine-tuned to distinguish between positive and negative samples provided at its input and can be used in the processing chain 21 of the solution architecture 70. For better understanding, the fine-tuned encoder network is identified below by reference number 78.

[0101] The finely tuned encoder network 78 is used to encode normal images 71, which are used as negative samples to generate corresponding complete feature vectors 79. For each normal image 71, a corresponding complete feature vector 79 can be generated. The complete feature vectors for all normal images 71 can be combined to form a matrix or array (not shown in Figure 7) of complete feature vectors 79, as previously described. Thus, PCA can be applied to the array or matrix of complete feature vectors 79 to generate a corresponding matrix or array of reduced feature vectors 80.

[0102] The reduced feature vector 80 can be fed into an ensemble of OCMs 81. In a given example, the ensemble 81 includes an OC-SVM 82 and an isolated forest 83. Each OCM outputs a classification result 93 as detailed above with respect to Figure 6B.

[0103] In the specific embodiment shown in Figure 7, a so-called hard-vote consensus 84 is applied to the ensemble 81 to generate the final detection output 85. This means that as long as any of the OCMs from the ensemble 81 indicate the presence of an anomaly, the image sample supplied to the processing chain 21 is considered to contain an anomaly and therefore indicates a bad sample. Of course, other methods of combining the outputs of the OCMs from the ensemble 81 are also possible.

[0104] It should be noted that Figure 7 essentially shows the processing chain 21 that is operated during the pre-training stage described above with respect to stage S55 or during (re)training with respect to stage S59. The difference between the two stages is the set of training data provided as normal images 71.

[0105] The difference between running the processing chain 21 during training and normal operation of the anomaly detection system 20 lies in the fact that PCA is no longer performed. Instead, the transformation matrix computed during the PCA computation is used to transform the full feature vector 79 corresponding to the target sample into the corresponding reduced feature vector 80.

[0106] Furthermore, the samples supplied to the ensemble 81 of OCM 82 and 83 are no longer labeled. Therefore, OC-SVM 82 and the isolated forest 83 simply output detection output 85 without modifying the internal configuration such as the weights of OC-SVM 82.

[0107] The corresponding data structures and algorithms for implementing the individual parts of the anomaly detection system 20, in particular the encoder networks 74 and / or 78 of the deep encoding subsystem 23, the reduction subsystem 24, and the OC-SVM 82, isolated forest 83, and ensemble 81 of the classification subsystem 25, are provided, for example, by scikit-learn (https: / / scikit-learn.org / ), which provides Python libraries for various machine learning algorithms.

[0108] The described solution architecture 70 combines the advantages of several approaches known from artificial intelligence and statistical processing in a novel way to achieve high accuracy in detecting anomalies on images, such as the surface of a finished product, without requiring extensive or human-supervised learning phases.

[0109] In particular, the flexibility of the solution architecture 70 stems from the fact that the OCM of Ensemble 71 can be trained using only negative samples, i.e., normal patterns, and does not require information or training samples regarding potential anomalies. This is especially useful in situations where it is difficult to obtain extensive data on anomalies or where anomalies are inherently prone to change. Furthermore, because only normal, i.e., good data is processed, the effort required to collect and label negative samples is greatly limited. In fact, during the training phase, all input data entered into the OCM is simply shown or labeled as negative samples.

[0110] OCM is particularly robust to novel, previously unknown anomalies because it is trained to detect the presence of normal, or good, patterns, rather than to detect specific anomalies. Therefore, OCM can also detect novel or previously unknown anomalies that were not available or conceivable during the training phase.

[0111] Furthermore, the use of pre-trained encoder networks leverages the power of established object recognition techniques known in other fields of artificial intelligence. Such encoders are already trained using large and diverse datasets such as ImageNet (https: / / www.image-net.org). Using these enables transfer learning by applying general features learned based on image collection from publicly available datasets to specific application domains such as laminate surfaces. This significantly accelerates and stabilizes the learning phase of the disclosed architecture 70 and enables the efficient extraction of feature vectors 75 or 79 that allow the OCM to detect anomalies.

[0112] Further fine-tuning the general encoder network 74 to generate a fine-tuned encoder network 78 further improves the relevance of the extracted feature vectors 79. Modifying the normal image 71 to generate a modified image 73 is sufficient to train the encoder network 74 to select particularly relevant features for determining the ensemble 81 to detect anomalies in the modified image. For this purpose, it is sufficient to simply generate artificial noise in the normal image 71, thus eliminating the need to provide true positive samples.

[0113] Finally, a specific processing chain 21, including one or more OCMs (Over-Censored Modulators) with an encoder network 78 (deep encoding subsystem 23), a reduction subsystem 24, and a classification subsystem 24, successively reduces the amount of data being processed. In particular, the first data is reduced from 2D pixel data to 1D feature vectors (or sets). The 1D feature vectors are then further reduced to feature vectors (or sets) using PCA. Thus, relatively complex OCMs (one or more) only need to act on limited datasets, which enables anomaly detection to be performed in real time in a production environment. [Explanation of symbols]

[0114] 20 Anomaly Detection System 21 Processing Chain 22 Training Subsystems 23 Deep Encoding Subsystem 24. Shrinking subsystems 25 Classification subsystems 26 inputs 27 Output 30. First Training Phase 32. Second Training Phase 33. Third Training Phase Set of 50 patches 53 First subset of negative samples 54 Second subset of negative samples 57 Set of false positive samples 68 The first matrix (of the complete feature vectors) 69 The second matrix (of the reduced feature vectors) 70 Architecture 71 Normal image 72 Training Modules 73 Modified images 74 Pre-trained encoder networks 75 Feature Vectors 76 Tag "Delinquent" 77 Tag "Good" 78 Fine-tuned encoder network 79. Complete feature vector 80 Reduced feature vectors 81 OCM Ensemble 82 OC-SVM 83 Isolated Forest 84 Hard voting consensus 85 Detection output 90 input spaces 91 Subspace 92 Target Samples 93 Classification results

Claims

1. A method for detecting anomalies in an image, the method being: - A step of obtaining a set of negative samples, wherein each negative sample includes at least a portion of an image (71) that does not contain abnormalities; - A step (S12) in which the treatment chain (21) is trained using the set of negative samples; - The process includes the step of processing an unknown image using a trained processing chain (21) (S14) to determine whether the unknown image contains anomalies, The training step (S12) of the aforementioned processing chain (21) is: - A step of obtaining an array of one-dimensional feature sets using a pre-trained deep encoder, wherein each one-dimensional feature set in the array represents the features of the corresponding negative sample from the set of negative samples; - The step of performing reduction analysis, particularly principal component analysis, on the aforementioned sequence of the one-dimensional feature set to obtain the corresponding sequence of the reduced feature set (S64); The step of training a one-class model based on the array of the reduced set of features (S65) A method characterized by including

2. Each negative sample corresponds to a training patch having a predefined target size, the predefined target size corresponding to the size of the training image used during the pre-training of the deep encoder (74); The steps for obtaining the set of negative samples are: Obtain a sample image that does not contain any abnormalities (S11); The sample image is divided into multiple training patches of the predefined target size (S51); This includes selecting at least some of the aforementioned training patches as negative samples; Processing the aforementioned unknown image involves: The unknown image is divided into a plurality of target patches of the predefined target size; Each target patch is processed using the aforementioned trained processing chain (21), and it is determined whether or not the target patch contains anomalies; This includes determining that the unknown image does not contain abnormalities only if it is determined that none of the target patches contain abnormalities. The method according to claim 1.

3. The preceding step of obtaining a set of negative samples is: - A step of providing a pool of negative samples, particularly the plurality of training patches according to claim 2; - A step of selecting an initial subset (53), particularly a random subset, from the pool of negative samples; - The step (S55) of pre-training the processing chain (21) using the initial subset (53) to obtain a pre-trained processing chain (21); - A step (S56) in which the remaining samples (54) from the pool of negative samples not included in the initial subset (53) are processed using the pre-trained processing chain (21), and it is determined whether each of the remaining samples is classified as abnormal; - A step of selecting all samples (57) that have been classified as containing abnormalities by the pre-trained treatment chain (21) as part of the set of negative samples for training the treatment chain (21). The method according to claim 1 or 2, including the method according to claim 1 or 2.

4. The method according to claim 3, wherein the selected set of negative samples for training the processing chain (21) further includes the initial subset (53).

5. Training the aforementioned processing chain (21) further involves: - A step (S44) of generating a set of false positive samples based on the set of negative samples, wherein each false positive sample is generated by modifying at least one of the negative samples; - Step (S42) of fine-tuning the pre-trained deep encoder (74) using the set of negative samples labeled as negative and the set of false-positive samples labeled as positive. The method according to any one of claims 1 to 4, including

6. The method according to claim 5, wherein noise is added to each negative sample to obtain a corresponding false positive sample.

7. - In the step of training the processing chain (21), a plurality of one-class models are trained, each one-class model being trained independently based on the reduced feature set; Processing an unknown image (S14) is: Obtain a reduced feature set corresponding to the aforementioned unknown image; The reduced feature set is processed separately using at least a subset of the aforementioned one-class models; This includes determining whether the unknown image contains anomalies based on the output provided by the one-class model used. The method according to any one of claims 1 to 6.

8. Each of the multiple one-class models is configured to output a binary value indicating whether or not the unknown image contains an anomaly; The method according to claim 7, wherein the output binary values ​​of the one-class model used are combined, in particular using majority, minority, or consensus decision, to detect whether the unknown image contains anomalies.

9. - If any of the one-class models among the multiple one-class models indicates that the unknown image contains an anomaly, then a determination is made that the unknown image contains an anomaly; Processing the reduced feature set using any of the remaining one-class models is stopped or omitted after that decision has been made. The method according to claim 8.

10. The reduced feature set is processed separately using each of the multiple one-class models; Each of the aforementioned one-class models outputs the likelihood that the unknown image contains anomalies; - In order to determine whether the unknown image contains anomalies, the output probabilities of all one-class models are combined, in particular by calculating the mean probabilities. The method according to claim 7.

11. The method according to any one of claims 7 to 10, wherein the plurality of one-class models include at least one of a one-class vector support machine (82) and an isolated forest (83).

12. - A deep encoding subsystem (23) configured to extract a one-dimensional feature set (79) from at least a portion of the images (71, 73); - A reduction subsystem (24) configured to obtain a reduced feature set (80) based on reduction analysis, particularly principal component analysis, of multiple one-dimensional feature sets (79); A classification subsystem configured to implement at least one one-class model, which processes the reduced feature set (80) and outputs a value indicating whether the unknown image processed by the deep encoding subsystem (23) and the reduction subsystem (24) contains anomalies; - A training subsystem (22) configured to initialize the reduction subsystem (24) and train the at least one one-class model based on a set of negative samples (53, 54) encoded by the deep encoding subsystem (23), wherein each negative sample includes at least a portion of an image that does not contain abnormalities, and An anomaly detection system (20) having the following features.

13. The anomaly detection system (20) according to claim 12, wherein the deep encoding subsystem (23) includes a neural network (74) pre-trained using a training set comprising a plurality of different training images selectively labeled with a plurality of different classification terms.

14. The aforementioned training subsystem (22) specifically includes: - A step of extracting a one-dimensional feature set (79) from each negative sample in the set of negative samples to obtain an array of one-dimensional feature sets, wherein each one-dimensional feature set (79) in the array corresponds to one of the negative samples in the set of negative samples; - The steps of performing the reduction analysis, in particular principal component analysis, on the array of the one-dimensional feature set (79) to construct a transformer, in particular a transformation matrix, of the reduction subsystem (24) for converting the one-dimensional feature set into a corresponding reduced feature set (80), and An anomaly detection system (20) according to claim 12 or 13, configured to perform the following:

15. The aforementioned training subsystem further: - Using the transformation matrix configured in the reduction subsystem (24), a reduced feature set (80) is calculated for each of the one-dimensional feature sets (79) of the array, thereby obtaining a plurality of reduced feature sets (80); - Train the at least one one-class model based on the multiple reduced feature sets (80) that act as negative samples for training. An anomaly detection system (20) according to claim 14, configured as described above.

Images