System for determining a quality of a coating surface

Abstract

A system (100) for determining a coating surface quality is presented, comprising a height data providing unit (101) configured to provide height data indicative of heights of the coating surface relative to a substrate, and a defect detecting unit (102) configured to detect defects in the surface by a) determining, for any candidate defect position, height difference data, the height difference data for a candidate defect position indicating differences between the height at the candidate defect position and heights at a plurality of reference positions, and b) comparing the height difference data to predetermined reference height difference data indicating a defect. The system further comprises a quality determining unit (103) configured to determine the quality of the surface based on the detected defects. This allows for improved coating quality assessments, based on efficiently and yet accurately detected defects.

Description

[0001] System for determining a quality of a coating surface

[0002] FIELD OF THE INVENTION

[0003] The invention relates to a system, a method and a computer program for determining a quality of a surface of a coating applied on a substrate.

[0004] BACKGROUND OF THE INVENTION The article “Rapid surface defects detection in wire and arc additive manufacturing based on laser profilometer” by C. Huang et al., Measurement, volume 189 (2022), relates to a laser inspection system for monitoring surface defects arising while manufacturing metal components by use of wire arc additive manufacturing, wherein, upon detection of surface defects, appropriate repair actions are to be taken during the manufacturing in order to avoid that surface defects are covered and thereby internal defects are generated in the manufactured metal component.

[0005] The article “Online Convolutional Neural Network-based anomaly detection and quality control for Fused Filament Fabrication process” by J. Lyu et al., Virtual and Physical Prototyping, volume 16 (2021), refers to detecting anomalies in laser scan data of a respective layer generated in a fused filament fabrication process for manufacturing a metal component, wherein a thickness of a respective subsequent layer is adjusted based on the detected anomalies.

[0006] The article “A Deep-Learning-based 3D Defect Quantitative Inspection System in CC Products Surface” by L. Zhao et al., Sensors, volume 20 (2020), relates to inspection strategies for evaluating surfaces in continuous casting production lines for manufacturing metal products.

[0007] Quality assessments for processes in which substrates like automobile parts, for instance, are being coated by paints or other coatings require suitable quality measures. Among the features that can be used for such quality measures are a number and type of defects in the coating surface. However, detecting and classifying defects in coating surfaces is prone to errors, particularly when carried out manually. On the other hand, already implementing just parts of the detection and classification on a computer can quickly require relatively large computational resources in order to arrive at sufficiently accurate results. There is therefore a need for improved quality assessments of coatings applied on substrates.

[0008] SUMMARY OF THE INVENTION

[0009] It is an object of the invention to allow for an improved quality assessment of coatings applied on substrates.

[0010] In a first aspect, the invention relates to a system for determining a quality of a surface of a coating applied on a substrate. The system comprises i) a height data providing unit configured to provide height data, wherein the height data are indicative of heights of the surface of the coating relative to the substrate, and ii) a defect detecting unit. The defect detecting unit is configured to detect defects in the surface of the coating by a) determining, for any candidate defect position of the surface of the coating, height difference data based on the height data, wherein the height difference data for a candidate defect position are indicative of differences between the height at the candidate defect position and heights at a plurality of reference positions, the reference positions being determined based on the candidate defect position according to a predefined relation, and b) comparing the height difference data to predetermined reference height difference data indicating a defect. The system further comprises iii) a quality determining unit configured to determine the quality of the surface based on the detected defects.

[0011] Since the quality of the surface of the coating applied on the substrate is determined based on defects detected in the surface of the coating by determining, for any candidate defect position of the surface of the coating, height difference data based on height data indicative of heights of the surface of the coating relative to the substrate, wherein the height difference data for a candidate defect position are indicative of differences between the height at the candidate defect position and heights at a plurality of reference positions, the reference positions being determined based on the candidate defect position according to a predefined relation, the quality of the surface can be determined based on efficiently and yet accurately detected defects. In this way, an improved quality assessment of coatings applied on substrates can be achieved.

[0012] The height data could be regarded as a three-dimensional set of data in which two dimensions are indicative of a position on, or of, the surface of the coating, and the third dimension is indicative of the height of the surface of the coating relative to the substrate at the respective position. The height data providing unit may be configured to provide the height data based on a three-dimensional scan of the surface. The scan may be carried out using a chromatic confocal measurement system such as, for instance, the FocalSpec by LMI Technologies Inc.

[0013] The height data can be discretized. Discretized height data are understood as height data taking non-continuously distributed values, particularly in the two dimensions indicative of the position on, or of, the surface of the coating. Hence, for instance, the height data can correspond to a set of a) discrete coordinates of the surface of the coating and b) associated heights. The discrete coordinates of the surface of the coating can arise from a coordinate grid virtually laid upon the surface of the coating. The surface of the coating may then be thought of as being partitioned into individual tiles corresponding to the coordinate grid.

[0014] In fact, the height data may be provided in terms of a monochromatic digital image. The image may be partitioned into pixels whose coordinates correspond to coordinates of the surface of the coating, wherein a pixel value may be associated to each pixel that is indicative of the height of the surface at the respective coordinate position. For instance, the pixel values may be gray values, in which case lighter gray values may indicate greater surface heights than darker gray values or vice versa.

[0015] The height data, particularly if provided in terms of a monochromatic digital image, may be determined via a projection of the three-dimensional scan data of the surface onto a plane, which may be referred to as a projection or image plane. This plane may particularly correspond to a surface of the substrate. The height data may further be determined based on an interpolation of the projected three-dimensional scan data. This allows to carry out the three-dimensional scan of the surface with a relatively low resolution even in case a relatively high resolution is desired for the height data. In an example, the height data may be obtained from the scan data using the software MountainsMap by the company Dig i- talSurf.

[0016] In order to detect the defects based on the height data accurately, it can be advantageous that the substrate, i.e. the surface of the substrate, is sufficiently smooth and / or flat. For instance, variations in height and / or a curvature of the surface of the substrate may be below a predetermined threshold. However, even for relatively irregular substrates, the defect detecting unit may be able to detect defects by distinguishing them from irregularities in the coating stemming from the substrate.

[0017] The defect detecting unit is preferably configured to detect each defect individually. Hence, the defects are preferably distinguished from each other. This allows to count the defects, wherein a number or density of detected defects provides a good measure of the quality of the surface of the coating.

[0018] In particular, the defect detecting unit can be configured to determine a position of, i.e. localize, each defect on the surface of the coating based on the height data. Knowing the position and of the defects allows for particularly rich measures of the quality of the surface of the coating.

[0019] The height difference data are preferably indicative of height differences along the surface of the coating. The defect detecting unit is therefore preferably configured to determine the height difference data for a respective candidate defect positon of the surface of the coating based on the height data such that the determined height difference data are indicative of differences between the height of the surface of the coating at the candidate defect position and heights of the surface of the coating at the plurality of reference positions. In particular, the height difference data may be determined by determining, based on the height data, differences in the height of the surface between the candidate defect position and the plurality of reference positions. The predefined relation according to which the reference positions are determined based on the respective candidate defect position may thus particularly be predefined such that the reference positions are, like the candidate defect positions, positions of the surface of the coating.

[0020] The defect detecting unit may be configured to detect the defects based on a transformation of the height data into a feature map, wherein the feature map associates features to positions on the surface of the coating, the feature associated to any given position being indicative of a difference between a respective height, particularly coating surface height, indicated by the height data for this position and heights, particularly coating surface heights, indicated by the height data for reference positions. Hence, the feature map, which is a preferred type of height difference data, treats all positions as candidate defect positions. A candidate defect position is preferably understood herein to refer to a position potentially belonging to a defect, particularly a positon for which it is still to be determined whether it belongs to a defects or not.

[0021] The feature map can include all positions on the surface of the coating, i.e. features can be determined for all, i.e. all known and / or available, positions on the surface of the coating. For instance, if the height data are represented in terms of a digital image such that positions on the surface of the coating correspond to pixels, the feature map can include all pixels of the digital image. That is to say, the feature map may be constructed by determining a feature for each of the pixels of the digital image, namely based on the height indicated by the height data for the respective pixel and the height indicated by the height data for reference pixels. In principle, however, the feature map may also include only a subset of all positions, such as only a subset of pixels. This could save computational resources.

[0022] Preferably, the reference positions lie in a neighborhood of the position for which the feature is to be determined. The neighborhood can be defined, for instance, in terms of a maximum distance between the reference positions and the position for which the feature is to be determined. The maximum distance may be defined with respect to and / or depend on a resolution of a respective digital image in which the defect detection is carried out. As already indicated, positions may correspond to pixels, i.e. pixel positions. The neighborhood in which the reference position may lie may particularly be or represent a region of the surface of the coating.

[0023] The feature map may comprise more than one feature per coating surface position. For each of the features associated to a same position, a different set of reference positions, and hence different height difference data, may be considered as a basis. In other words, separate relations to be satisfied by the reference positions may be predefined for each of the features associated to a candidate defect position. It has been found that local neighborhoods around surface positions, even if the positions all belong to a same defect, can vary significantly in height data, wherein these variations can lead to different height difference data. For instance, image pixels corresponding to an inner region of a defect will typically be surrounded by pixels belonging to the defect as well, whereas pixels belonging to boundary regions of the defect will typically be partially surrounded by pixels not belonging to the defect. By considering several features per surface positon and, for each of the features, different reference positions, possibly lying in neighborhoods of different sizes around the respective candidate defect position, defect positions may be more accurately detected.

[0024] The height difference data can, in particular, correspond to a local binary patterning of the height data. A local binary patterning of the height data can be understood as a particular transformation of the height data into a feature map as indicated above. In other words, a local binary pattern, i.e. the result of the local binary patterning, can be understood as a particular type of feature map, i.e. of the above indicated specific type of height difference data.

[0025] A local binary pattern as understood herein can be represented by a map b mapping positions on the surface of the coating, which may correspond to pixels of an image of the surface of the coating, particularly of a grayscale image in which the gray values are indicative of surface heights, to a respective binary vector, i.e. a vector each of whose entries can only take one of two predefined values. For instance, in case of discretized height data

[0026] IR,j f(J) associating heights f f) to discretely distributed positions j e N on the surface P of the coating, the map b can be mathematically defined as wherein the s components {fe^(j)}^=1 sof the vector b f) satisfy o'o) = !Ci v t = 1..s.

[0027] J(2)(c2else with { / ^ =i..s indicating the reference positions, i.e. the reference positions satisfying the predefined relation with respect to the position j, and a corresponding to a predefined margin.

[0028] The predefined values c and c2can be understood as indicating height difference classes for the position for which the binary vector is determined with respect to the respective reference position. For instance, the predefined values can be one and zero, respectively, i.e. c = 1 and c2= 0. The margin a can be a predefined constant. By choosing different values for a, a sensitivity of the defect detecting unit can be controlled. Optionally, a may be chosen based on variations in height and / or a curvature of the surface of the substrate on which the coating is applied. However, this is only one of many possible factors based on which a may be chosen. In particular, as indicated below, a can be chosen depending on a distance between the position j for which the respective local binary vector b(j) is calculated and the reference positions { / »#=i..s considered forthe calculation. In fact, a may also be “learned”, i.e. chosen based on an assessment of a detection quality with several different values of a.

[0029] Moreover, the sign of a may be chosen depending on a type of defect to be detected. For instance, in order to detect defects expected to correspond to depressions in a coating surface, such as valley- or crater-like defects, a may be chosen to be positive. On the other hand, in order to detect defects expected to correspond to elevations in a coating surface, such as ridge- or particle-like defects, a may be chosen to be negative. In the latter case, i.e. for detecting defects expected to correspond to elevations in a coating surface, the inequality sign in above equation (2) may additionally be inverted. Alternatively, just the function f in above equation (2) may be inverted for detecting defects expected to correspond to elevations in a coating surface. If, for instance, the function f indicates pixel values of a grayscale image representing coating surface heights, this would correspond to an inversion of the image.

[0030] Since the vector 6(7) also depends on the height data at the reference positions, wherein the reference positions may lie in a region surrounding the position j for which the vector is computed, in an alternative terminology the local binary patterns may also be understood in terms of maps from regions, or neighborhoods, to vectors, i.e. instead of in terms of maps from positions to vectors. In fact, since the vector 6(7) actually depends on the height data, i.e. (j) and the { / (7»}^=1„s, in yet a further alternative terminology the local binary patterns may be understood in terms of maps from images of regions, or neighborhoods, particularly the sets {7, {7 =i..s}, under the map , i.e. from heights, to vectors.

[0031] The reference positions can be chosen to all have a same distance Q to the position j for which the binary vector is determined, and particularly also a same distance o to their closest neighbors in the set {jt}t=,sof reference positions. The first of these optional requirements could be written d(jf,j) = Q f, with a predefined distance function d. For example, the distance function d could yield the Euclidean distance between two points it receives as an argument. The further optional requirement, i.e. that the reference positions can be chosen to all lie equidistantly distributed on a circle around the position forwhich the binary vector is determined, can, if enumerating the reference positions in the order of their angular position on the circle, for instance, be written as - 1.

[0032] The parameter a is preferably chosen depending on Q and s. Hence, the predefined margin between the height / (J) at the respective position j for which the local binary vector is determined and the heights £ (A’) =i..satits associated reference positions { / =i..sis preferably chosen depending on a) a size of the neighborhood that is considered for calculating the local binary vector and b) a number of reference positions considered for this neighborhood. Moreover, a may be chosen differently for each type of defect to be detected. Hence, particularly the dependence of a on Q and s may be chosen depending on a type of defect that is to be detected.

[0033] It is also possible that more than one binary vector is determined per candidate defect position, wherein for each of the binary vectors associated to a given candidate defect position a different set of reference positions is considered as a basis. For instance, several distances Q and a may be considered. As already indicated above, this can allow for a more accurate detection of defects, particularly their shapes. The several binary vectors could also be combined to a larger single binary vector.

[0034] In particular, for each type of defect to be detected, several local binary patterns may be defined, wherein the parameters of the respective local binary pattern are chosen depending on the respective type of defect. Hence, for each type of defect to be detected, several binary vectors may be computed per candidate defect position based on different sets of reference positions, wherein the definition of the several binary vectors per candidate defect position may be chosen in dependence on the respective type of defect to be detected. In order to detect defects of any type, the local binary patterns defined for all types of defects may be used collectively as a basis.

[0035] The defect detecting unit can be configured to detect the defects in the surface of the coating based on the binary vectors resulting from the local binary patterning. That is to say, the defect detecting unit can be configured to determine, for a given position on the surface of the coating, whether this position belongs to a defect based on the binary vector determined for this position, and possibly based further on binary vectors determined for neighboring positions.

[0036] For example, a distance between the binary vector of each position to each of a set of defect indicating binary vectors can be computed. The computed distance can refer, for instance, to a Hamming distance between the binary vector of the respective position and the respective defect indicating binary vector. The Hamming distance between two vectors corresponds to the number of entries in the two vectors whose position in the respective vectors is the same but whose value is not equal to each other. However, many other distance measures can be used. In fact, instead of determining a distance, the binary vectors at the respective positions may also be compared with defect indicating binary vectors differently, i.e. based on non-distance measures. If the comparison indicates a degree of similarity between the respective binary vectors that lies beyond a predefined threshold, it may be concluded by the defect detecting unit that the respective position belongs to a defect. In other words, the presence of a defect at the respective position may then be concluded.

[0037] The defect indicating binary vectors may indicate the presence of defects of particular types, such as holes, hills and / or valleys, for instance. Hence, several defect indicating binary vectors may be used, each of them indicating the presence of a different type of defect. Connected regions of positions having binary vectors associated, i.e. mapped, to them that lie within a predefined distance of a defect indicating binary vector, or are similar thereto beyond a predefined degree as measured by a predefined similarity measure, may be identified as corresponding to defects in the surface of the coating. Preferably, only connected regions of positions having binary vectors associated to them that all lie within a predefined distance of a same defect indicating binary vector, or are similar thereto beyond the predefined degree as measured by the predefined similarity measure, are identified as corresponding to defects in the surface of the region, namely to a respective same defect.

[0038] A set of defect indicating binary vectors may, for example, be established and thereafter provided for access by the defect detecting unit based on manual indications of defects in height data, such as projections of three-dimensional coating surface scans into monochromatic images in which gray values indicate surface heights, wherein binary vectors have been determined for this height data, such that, by assuming the manual indications of defects as ground truths, defect indicating binary vectors can be derived from the binary vectors determined for the positions in the height data belonging to the manually indicated defects.

[0039] The defect indicating binary vectors may also be predefined heuristically. In particular, the defect indicating binary vectors may be predefined in consideration of a definition of the respective local binary patterns and a typical binary vector expected for a respective defect. The system may further comprise a classifying unit configured to classify the detected defects into one of a plurality of defect classes, wherein the quality determining unit may be configured to determine the quality of the surface based on the classified detected defects.

[0040] The quality of the surface may be determined based on a number and / or class of detected defects and / or an overall area covered by the detected defects. The number and the overall area covered may be determined in a class-specific manner. Objective quality levels may be defined based on threshold values depending on the number and / or class and / or overall area covered by the detected defects.

[0041] Detected defects may be classified into one of the plurality of defect classes based on the defect detecting binary vectors. The defect detecting binary vectors are then preferably each indicative of a respective defect class, and hence become class-defining binary vectors. For instance, a connected region of positions having binary vectors associated to them that all lie within a predefined distance to a same class-defining binary vector, or are similar thereto beyond the predefined degree as measured by the predefined similarity measure, may be identified as corresponding to a defect of the class defined by the class-defining binary vector. In this way, connected regions of the surface of the coating may be associated with defects of particular classes. However, the detected defects could also be classified without reference to the defect indicating and / or class-defining binary vectors. That is to say, it is also an option that, after having detected the defects based on the defect indicating binary vectors, the defect indicating binary vectors are not used additionally for defect classification.

[0042] The classification of defects by the classifying unit may be carried out along with an estimation of a probability indicating a level of confidence into the correctness of the classification. Hence, for instance, the classifying unit may be configured to provide, for each of a plurality of detected defects to be classified, one or more classes of the defect and, for each of the one or more classes, a likelihood that the defects is a defect of the respective class.

[0043] Irrespective of the manner of classification, the defect classes can be indicative of any of the following: cracks, craters, particles, flakes, bubbles. Hence, the system allows to detect, by means of the detecting unit, whether and particularly where the coating comprises any of a crack, a crater, a particle, a flake and a bubble, and, by means of the classifying unit, allows to distinguish between them. These classes of defects may accidentally originate from the coating material itself and / or the coating process, i.e. from how the coating is applied. For instance, particles, flakes or air bubbles may be accidentally enclosed while applying the coating to a substrate, and flakes can have flaked off from other substrate samples with which the substrate whose coating is being assessed was transported together in a same transport box or rack. It will be understood that, by providing corresponding further class-defining binary vectors or adapting the classifying unit in other of its subsequently described aspects accordingly, also other defect classes may be recognized.

[0044] The classifying unit may be configured to classify the detected defects based on heights indicated by the height data for the detected defect. For instance, the classifying unit may receive a part of the height data in which a defect has been detected as input, and provide the class of the detected defect as output. Classifying the defects based on the height data can be efficient, since no further data has to be provided.

[0045] It may be preferred, however, that the system comprises a rendered image providing unit configured to provide a rendered image corresponding to a view onto the surface of the coating, wherein the classifying unit is configured to classify the detected defects based on rendered image data corresponding to the detected defect. Rendered image data have been found to allow for a particularly reliable defect classification.

[0046] The height data and the rendered image may be registered to each other such that a defect detected based on the height data may be localized in the rendered image. The rendered image providing unit may be configured to determine the rendered image based on the three-dimensional scan data based on which also the height data may be determined.

[0047] It should be understood that, even though the height data or the rendered image data could be used for determining shapes of the detected defects, wherein a classification of defects could in principle also be carried out based on the defect shapes, it is preferred to not take this additional step. Instead, the classifying unit can directly receive the height data or the rendered image data corresponding to a detected defect for classifying it. The shape of the defect may still be used implicitly by the classification unit forthe classification, but also this is not necessary.

[0048] The defects may also be classified based on other features. For instance, the defects may be classified based on a respective local binary pattern, i.e. particularly based on their associated binary vectors b(j). However, it is emphasized that the defects can already be detected, particularly localized, using local binary patterns. Local binary patterns are therefore preferably used already before the defects are classified, wherein their use for defect classification is regarded optional. The defect detecting unit may be configured to determine a bounding box for each detected defect in the height and / or rendered image data and / or the local binary pattern, wherein the classifying unit may be configured to receive the height and / or rendered image data and / or binary vectors corresponding to this bounding box, particularly its interior, as input, and to provide a class for the defect in the bounding box based on the received data.

[0049] The classifying unit can be configured to classify the detected defects using a trained machine learning model. This has been found to lead to good classification results. However, also other artificial intelligence techniques may be employed. For instance, instead of a trained machine learning model, an unsupervised image processing algorithm may be used for classifying the detected defects.

[0050] If using a machine learning model, the machine learning model can be trained based on training data comprising pairs of training input data and training output data, wherein the training input data in each pair comprise height data and / or rendered image data corresponding to a single defect in a coating of a surface, and the training output data correspond to a class of the respective defect. The classes in the training output data may be provided manually. For instance, for generating the training data, a trained person may have classified images of defects according to whether the respective defect would be viewed as a crack, a crater, a particle, a flake or a bubble defect.

[0051] The classifying unit may be configured to choose the machine learning model to be used based on the bounding box, such as based on a shape and / or size of the bounding box. In particular, an input layer of the machine learning model may be chosen accordingly. This allows for the use of bounding boxes of different shapes and / or sizes, such that the bounding boxes can be adapted to the respective detected defects. A more efficient classification can hence be achieved, since less height data may need to be processed per defect.

[0052] In particular, the machine learning model can comprise a convolutional neural network. The convolutional neural network may be configured to receive, as its input, any combination of the following features: the binary vectors, which correspond to local binary patterns, determined for the positions corresponding to the detected defects, the height data corresponding to the detected defects, the rendered image data corresponding to the detected defects.

[0053] The convolutional neural network preferentially comprises a plurality of convolutional layers for providing respective feature maps. In particular, the number of convolutional layers in the convolutional neural network can be equal to 5. The feature maps provided by the convolutional layers of the convolutional neural network, which is optionally used for classifying detected defects, are in general to be distinguished from the feature map based on which the defects can be detected, particularly from the local binary pattern that is preferably used for defect detection. Nevertheless, additionally or alternatively, also a feature map determined based on a local binary pattern can be used for defect classification and hence as input for the convolutional neural network.

[0054] A particular example of a feature map determined based on a local binary pattern, which can be used for the defect classification, is a digital image whose pixel values are determined based on the binary vectors of the local binary pattern. Such image-type feature maps could also be regarded as visual representations of the respective local binary pattern. The pixel values can be determined, for instance, by representing the binary vectors of the local binary pattern as binary strings, wherein the integer number associated to a binary vector can be defined to be the binary number corresponding to the binary string, optionally converted into the decimal system.

[0055] The convolutional layers may correspond to a convolution of the respectively received input data by one or more kernels, each of the kernels giving rise to a respective feature map. The kernels may be determined during the training of the convolutional neural network.

[0056] Furthermore, the convolutional neural network may comprise at least one pooling layer for pooling a respective one of the feature maps into a corresponding pooled feature map. The at least one pooling layer may be adapted to receive a feature map provided by a previous convolutional layer as input and provide an associated pooled feature map based thereon. In an example, max pooling may be applied.

[0057] Pooling is understood herein as a process of transforming a feature map by applying a function to the features of the feature map in one of a plurality of predefined regions of the feature map and repeating this application of the function for each of the predefined regions. In this way, a pooled feature map is generated, which corresponds to a map from the predefined regions to the results arising from the application of the function to features in the respective region. If the predefined regions cover a whole region for which the feature map is defined, i.e. a whole coating surface region, for instance, then the pooled feature map may also be defined in this whole region. The predefined regions, which may cover the whole region for which the non-pooled features are defined, may correspond to a square of adjacent pixels in a digital image, for instance. The pooled feature map is typically smaller than the original, non-pooled, feature map from which it arises. For instance, if the predefined regions, which could also be referred to as “pooling regions”, correspond to squares of m x n adjacent pixels, the pooled feature map may have a size of l / (m • n) as compared to the original feature map. In case of max pooling, the function applied to the features in the predefined regions corresponds to singling out a maximum value among the features.

[0058] In a variant, min pooling may be applied. Max pooling and min pooling may be understood as special cases of extremal pooling. This can be understood such that the function applied to the features in the predefined regions corresponds to singling out an extremal value among the features. In case of min pooling, the feature corresponds to a minimum in the predefined region.

[0059] The convolutional neural network may also apply spatial pyramid pooling. In particular, spatial pyramid pooling using the three sub-pooling structures 5 x 5, 3 x 3 and l x l can be applied. However, in general, many other sub-pooling structures could be used for the spatial pyramid pooling. Irrespective of the sub-pooling structure, spatial pyramid pooling allows to classify previously detected defects of arbitrary size, since it leads to data objects to be classified having the same size independently on the size of the input provided to the convolutional neural network.

[0060] Alternatively, already the initial input provided to the convolutional neural network could be resized to a uniform size, i.e. before it is passed to the network. Concretely, for instance, the bounding boxes arising from the defect detection could all be resized to a uniform shape, and only then provided as input to the convolutional neural network. However, since the sizes of defects have been observed to vary significantly, not resizing the input to the convolutional neural network and instead applying spatial pyramid pooling has been found to be advantageous for the classification results.

[0061] Besides the convolutional and the pooling layers, the convolutional neural network can comprise a fully connected layer, particularly a single fully connected layer. The spatial pyramid pooling layer may then correspond to the pooling layer arranged between the last convolutional layer and the fully connected layer. Hence, the spatial pyramid pooling layer is preferably the last pooling layer in the convolutional neural network.

[0062] Furthermore, the machine learning model may comprise a transformer architecture. This has been found to allow for good classification results as well. The material of the substrate can be selected from any of the following: wood, wood veneer, paper, paperboard, cardboard, textile, film, leather, nonwoven, plastics, glass, ceramic, mineral building materials, and metals. Moreover, the substrate may be precoated. Hence, the coating, i.e. the coating whose surface quality is to be determined, may have been applied to the precoated substrate. For instance, the coating may comprise, particularly consist of or correspond to, a clearcoat, a basecoat and / or a topcoat. Furthermore, the coating may have been applied by any of the following: drawdown application with a doctor blade, airmix and / or airless spray application, roller application, brush application, electrocoating, a combination of the foregoing.

[0063] A further aspect of the invention relates to a method for determining a quality of a surface of a coating applied on a substrate, wherein the method includes, in a first step, i) providing height data, wherein the height data are indicative of heights of the surface of the coating relative to the substrate. In a second step, the method includes ii) detecting defects in the surface of the coating by a) determining, for any candidate defect position of the surface of the coating, height difference data based on the height data, wherein the height difference data for a candidate defect position are indicative of differences between the height at the candidate defect position and heights at a plurality of reference positions, the reference positions being determined based on the candidate defect position according to a predefined relation, and b) comparing the height difference data to predetermined reference height difference data indicating a defect. In a third step, the method includes iii) determining the quality of the surface based on the detected defects. The method according to this aspect can particularly be a computer-implemented method.

[0064] A further aspect relates to a computer program for determining a quality of a surface of a coating applied on a substrate, wherein the computer program comprises instructions causing the above system, or generally a data processing apparatus, to execute the above method.

[0065] The invention also relates to a use of local binary patterns to detect and / or classify defects in a coating applied on a substrate based on height data indicative of heights of the surface of the coating relative to the substrate. As outlined in more detail above, the height data may correspond to a digital grayscale image of the coating surface in which pixel values indicate the height of the coating surface at a respective position of the coating surface.

[0066] Furthermore, a system for training a machine learning model to classify defects in a coating applied on a substrate, i.e. a training system, is presented. The training system comprises i) a model providing unit configured to provide a machine learning model to be trained, ii) a training data providing unit configured to provide training data comprising pairs of training input data and training output data, wherein the training input data in each pair comprise any of a) height data, b) data of a features map derived from the height data via a local binary patterning and / or c) rendered image data corresponding to a single defect in a coating of a surface, and the training output data correspond to a class of the respective defect, and iii) a training unit configured to train the machine learning model based on the training data.

[0067] Correspondingly, a method for training a machine learning model to classify defects in a coating applied on a substrate, i.e. a training method, is presented. The training method includes i) providing a machine learning model to be trained, ii) providing training data comprising pairs of training input data and training output data, wherein the training input data in each pair comprise height data and / or rendered image data corresponding to a single defect in a coating of a surface, and the training output data correspond to a class of the respective defect, and iii) training the machine learning model based on the training data.

[0068] In a further aspect, a computer program for training a machine learning model to classify defects in a coating applied on a substrate is provided, wherein the computer program comprises instructions causing the training system, or generally a data processing apparatus, to execute the training method.

[0069] It shall be understood that the system of claim 1 , the method of claim 14 and the computer program of claim 15, have similar and / or identical preferred embodiments, in particular as defined in the dependent claims.

[0070] It shall be understood that a preferred embodiment of the present invention can also be any combination of the dependent claims or above embodiments with the respective independent claim.

[0071] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter.

[0072] BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In the following drawings: Fig. 1 shows schematically and exemplarily a system for determining a quality of a surface of a coating applied on a substrate,

[0074] Fig. 2 shows a first example of height data,

[0075] Fig. 3A shows a monochromatic image of a coating surface corresponding to a second example of height data,

[0076] Fig. 3B shows a digital image of coating surface according to a further example,

[0077] Fig. 4A shows pixel values of a monochromatic image along a line across a coating surface corresponding to a further example of height data,

[0078] Fig. 4B shows pixel values of a rendered image along the line across the coating surface of Fig. 4A,

[0079] Fig. 4C shows the line across the coating surface in the rendered image,

[0080] Fig. 5A shows a rendered image of the coating surface according to the example of Fig. 3A,

[0081] Fig. 5B shows a rendered image of a coating surface according to a further example,

[0082] Fig. 6A shows schematically a neighborhood of pixels of a digital image and a corresponding definition of a local binary pattern according to a first example,

[0083] Fig. 6B shows schematically a neighborhood of pixels of a digital image and a corresponding definition of a local binary pattern according to a second example,

[0084] Fig. 7 shows schematically a neighborhood of pixels of a digital image and values of a corresponding binary vector arising from a local binary pattern according to a third example, Figs. 8A and 8B show schematically further examples of neighborhoods of pixels of digital images and values of corresponding binary vectors arising from respective local binary patterns,

[0085] Fig. 9A shows an exemplary rendered image of a surface of a coating applied on a substrate including bounding boxes around defects detected in the coating,

[0086] Fig. 9B shows a further example of a surface of a coating applied on a substrate including bounding boxes around defects detected in the coating,

[0087] Fig. 10 shows several examples of height data images and corresponding rendered images of coating defects,

[0088] Fig. 11 shows schematically and exemplarily a convolution of an image by a kernel,

[0089] Fig. 12 shows schematically and exemplarily a structure of a convolutional neural network used for classifying detected defects, and

[0090] Fig. 13 shows schematically and exemplarily a method for determining a quality of a surface of a coating applied on a substrate.

[0091] DETAILED DESCRIPTION OF EMBODIMENTS

[0092] Fig. 1 shows schematically and exemplarily a system 100 for determining a quality of a surface of a coating applied on a substrate. The system 100 comprises a height data providing unit 101 that is configured to provide height data. The height data are indicative of heights of the surface of the coating relative to the substrate. Furthermore, the system 100 comprises a defect detecting unit 102 that is configured to detect defects in the surface of the coating based on the height data provided by the height data providing unit 101 . For detecting the defects, the defect detecting unit 102 determines, for any candidate defect position of the surface of the coating, height difference data based on the height data, wherein the height difference data for a candidate defect position are indicative of differences between the height at the candidate defect position and heights at a plurality of reference positions, wherein the reference positions are being determined by the defect detecting unit 102 based on the candidate defect position according to a predefined relation. Then, the defect detecting unit 102 compares the height difference data to predetermined reference height difference data that indicate a defect. Preferentially, the system 100 further comprises a quality determining unit 103 configured to determine the quality of the surface based on the defect detection carried out by the defect detecting unit 102.

[0093] Fig. 2 shows, in its left part, a black-and-white digital image representing a region of a coating surface comprising a defect. The defect is represented by the white part of the digital image while its neighborhood is shown in black in Fig. 2. The image shown on the left in Fig. 2 is partitioned into pixels whose coordinates correspond to coordinates of the surface of the coating, wherein the pixel values associated to each of the pixels are indicative of the heights of the surface at the respective positions. The black-and-white nature of the image serves to illustrate the relation between heights and pixel values particularly clearly, although in a rather schematic manner. In general, monochromatic images representing height data will comprise more shades of gray. Nevertheless, a black-and-white image could be generated from any monochromatic image by thresholding the gray levels, i.e. the heights.

[0094] The pixel values corresponding to the black-and-white image shown in the left of Fig. 2 are shown in terms of an array on the right of Fig. 2. Since the image is a black-and-white image, its pixels take only one of two values, in this case either 0 or 1 , wherein a “0” indicates black and a “1 ” indicates white. In the example of Fig. 2, pixel values of 1 , i.e. white pixels, indicate a greater height of the surface of the coating at the position corresponding to the respective pixel, as compared to pixel values of 0, i.e. black pixels. Hence, the defect schematically and exemplarily shown in Fig. 2 is a region of the surface of the coating that is elevated relative to its neighborhood.

[0095] Fig. 3A shows height data in the form of a grayscale monochromatic digital image. Hence, the pixel values of the image of Fig. 3A do not only take one of two values as in Fig. 2, but can take any gray value on the grayscale. Generalizing the scheme of Fig. 2, lighter gray values indicate greater surface heights than darkergray values. The coating surface region covered by the image of Fig. 3A is larger than the one covered by the image of Fig. 2 and covers more than one defect region. However, the defects are not easily identifiable in the image of Fig. 3A due to the relatively uniform gray values of the image.

[0096] Height data in the form of monochromatic digital images as exemplarily shown in Fig. 3A may be determined based on a three-dimensional scan of the surface of a coating that has been applied to a substrate, i.e. of the coated substrate. Such a three-dimensional scan of a coating surface can be carried out using known technology. For instance, line confocal imaging sensors can be used, as employed in the FocalSpec by LMI Technologies Inc. A result of the three-dimensional scan typically consists of a three-dimensional data set in which two dimensions indicate a surface coating position and the third dimension indicates a height of the coating surface at the respective position. An arbitrary coordinate in the third dimension can be used to indicate the height. Thus, the height can be measured, for example, relative to the substrate on which the coating is applied. The height of the surface of the coating relative to the substrate could also be referred to as a depth of the coating. The height of the surface of the coating could, alternatively, also be measured relative to an imaginary plane above the coating surface, in which case smaller instead of greater heights would indicate elevations in the coating.

[0097] The three-dimensional scan data can be projected into two dimensions, such as onto a plane parallel to the substrate on which the coating is applied, in order to acquire two- dimensional image data. The plane into which the scan data are projected can also be referred to as a projection plane, or an image plane, and could correspond, for instance, to the surface of the substrate, if the substrate is substantially planar. The projection may be carried out in the above-mentioned third direction of the three-dimensional scan data, in which case the first two dimensions, which indicate the coating surface positions, would translate to positions in the projection image and the image values, i.e. the pixel values, could correspond to the heights indicated by the three-dimensional scan data.

[0098] The resolution of the projection image does not necessarily correspond to the resolution of the three-dimensional scan data in the first two dimensions. Instead, for instance, scan data with a relatively high spatial resolution could be projected into a lower resolution two-dimensional image, or scan data with a relatively low spatial resolution can be projected into a two-dimensional image having a higher resolution. In the former case, pixel values in the projection image may be determined based on more than one scan point by procedures like averaging, for instance, and in the latter case, the pixel values may be determined based on an interpolation of the three-dimensional scan data, for instance. These and other processing steps of the scan data may be carried out using tools like the software Moun- tainsMap by the company Dig italSurf, for instance.

[0099] While Fig. 3A shows a relatively uniform coating surface region in which the defects are hardly visible, Fig. 3B shows a monochromatic digital image of a coating surface region which is less uniform, and in which defects are visible more easily. Moreover, the types of defects visible in the image of Fig. 3B are of a crack-type and insofar different than those of Fig. 3A, which are of a particle-type.

[0100] Digital images representing height data of a coating surface, like the ones shown in Figs. 2, 3A and 3B, do not convey the visual impression the coating surface would have on an observer in reality. In order to visualize a more realistic impression of the coating surface, lighting has to be taken into account. For this purpose, the system 100 may comprise a rendered image providing unit configured to provide, based on given height data and / or based on a three-dimensional scan of the coating surface, rendered image data corresponding to a view onto the coating surface. The rendered image data may be determined by the rendered image providing unit based on the three-dimensional scan data based on which also the height data may be determined.

[0101] The relation between height data, as representable in terms of gray values of a grayscale image, for instance, and rendered image data is illustrated in Figs. 4A to 4C. Fig. 4A shows an elevation, i.e. height, profile determined based on pixel values along a line through a grayscale image representing height data of a surface of a coating applied on a substrate. The image itself, i.e. as a whole, is not shown. It can be seen in Fig. 4A that the height of the exemplary coating surface is relatively uniform along the chosen line except for a region 41 , where the height data indicate a relatively large elevation, which hints at the presence of a defect in the coating. Fig. 4B shows a rendering profile corresponding to the elevation profile shown in Fig. 4A, wherein the rendering profile has been determined based on a rendered image and along a line corresponding, respectively, to the grayscale image and the line through it, i.e. the image and the line to which Fig. 4A relates. Also the rendering profile is relatively uniform to a large part, i.e. except of the region corresponding to the region 41 . Moreover, the rendering profile, which, like the elevation profile, corresponds to gray values along the respective line, comprises, in the region corresponding to the elevated region 41 , a bright region 41 a and a dark region 41 b, indicated by relatively high and relatively low image values, respectively. The split of the elevated region 41 into the bright region 41 a and the dark region 41 b is a lighting effect generated from the rendering. Fig. 4C shows a larger portion of the full rendered image, in which the line along which the values of the rendering profile are taken is indicated.

[0102] Fig. 5A shows a rendered image corresponding to the height data representing the digital image shown in Fig. 3A, and Fig. 5B shows an exemplary further rendered image of a coating surface on a substrate. It is emphasized that, even though in principle a detection of defects could also be carried out in rendered image data, the defect detecting unit 102 is preferably configured to detect defects in height data like grayscale image data of images as shown in Figs. 2, 3A and 3B. Moreover, it should be emphasized that the detection of defects by the defect detecting unit 102 does not yet include a classification of the detected effects. In contrast, as further outlined below, the classification can be carried out as a subsequent step, after detecting the defects. Moreover, the classification can be carried out based on any of a) height data, b) data of a feature map derived from the height data by means of a local binary patterning and / or c) rendered image data.

[0103] The defect detecting unit 102 is preferably configured to treat all positions of the coating surface, i.e. all positions provided to the defect detecting unit 102, as candidate defect positions. Hence, for instance, all pixels in an image corresponding to the height data determined by the height data providing unit 101 may be treated as candidate defect positions. In order to determine whether a candidate defect position is an actual defect position, i.e. corresponds to an actual part of a defect, height difference data are determined by the defect detecting unit 102 based on the height data. In particular, the defect detecting unit 102 can be configured to determine, based on the gray value of a pixel in a two-dimensional image representing the heights of the coating surface in terms of a grayscale, and based on the gray value of reference pixels, i.e. based on the height indicated for a pixel and heights indicated for reference pixels, height difference data. The height difference data, which are associated to a given candidate defect position corresponding to a given image pixel, indicate differences between the height at the candidate defect position and the heights at the reference positions. Hence, several height differences can be associated to a given candidate defect position, as represented by a corresponding pixel.

[0104] The reference positions, also represented by pixels, can be determined by the defect detecting unit 102 for a given candidate defect position according to any of a set of predefined relations. For instance, the reference positions can be chosen from the neighborhood of a candidate defect position. Preferably, the predefined relation considered by the defect detecting unit 102 for choosing the reference positions is the same for all candidate defect positions, and hence possibly for all positions on the coating surface, i.e. possibly all pixels in the height image representing the coating surface. By comparing the height difference data determined for a given candidate defect position to predetermined reference height difference data indicating the presence of a defect, the defect detecting unit 102 is capable of determining whether the candidate defect position belongs to a defect or not. The comparison of the height difference data determined for a given candidate defect position to the predetermined reference height difference data can involve a comparison of the height difference data determined for the given candidate defect position with each of a set of predetermined reference height difference data elements indicating different types of defects. In order to conclude that the candidate defect position belongs to a defect, it may be sufficient that the height difference data determined for the candidate defect position has a sufficiently high degree of similarity with any one of the set of predefined reference height difference data elements. The reference height difference data elements could also be referred to as defect indicating height difference data elements, or just as defect indicating data elements.

[0105] For the comparison of the height difference data to the predetermined reference height difference data, any suitable comparison measure may be used. The measure may be chosen depending on how the height difference data are represented. For instance, if representing the height difference data in terms of vectors associated to the respective candidate defect position, wherein the entries of the vectors correspond to height differences between the candidate defect position and the respective reference position or quantities derived therefrom, in which case the reference height difference data would preferably also be represented in terms of vectors, the comparison may correspond to forming difference vectors between the vectors corresponding to the height difference data determined for the respective candidate defect position and the vectors corresponding to the reference height difference data, and computing a vector norm of the respective difference vectors. However, as indicated further below, other comparison measures may be preferred, which are not necessarily based on differences.

[0106] Preferentially, the defect detecting unit 102 is configured to detect each defect individually. That is to say, the defect detecting unit 102 is preferentially able to distinguish between detected defects. For instance, for a set of positions on the surface of the coating that have been determined to be actual defect positions, i.e. belong to defects in the coating, the defect detecting unit 102 can be configured to decide whether the positions belong to the same defect and, if not, associate the positions with the respective different defects. In other words, the set of detected defect positions can be segmented into subsets corresponding to separate defects. The separate defects can be localized in an image representing the height data of the coating surface. Optionally, the defect detecting unit 102 is configured to determine bounding boxes surrounding one or more of the detected and localized defects. These bounding boxes, i.e. the corresponding image segments, which could also be referred to as defect image segments, can be used as a basis for classifying the defects inside them. However, a classification of the detected defects may not be necessary for determining a quality of the surface, i.e. for assessing a quality with which the coating has been applied. For instance, reliable quality measures can already be defined based on a number, particularly a density, of detected defects, and / or a coating surface area covered by defects.

[0107] A particularly accurate and efficient detection of defects has been found to be achievable by using local binary patterns. Local binary patterns are considered a particularly useful type of height difference data. Hence, the defect detecting unit 102 is preferentially configured to carry out a local binary patterning of the height data to determine the height difference data. A local binary patterning of the height data can be understood as a particular type of transformation of the height data into a feature map, wherein the feature map associates features to positions on the surface of the coating, the feature associated to any given position being indicative of differences between a respective height indicated by the height data for this position and heights indicated by the height data for the reference positions.

[0108] As illustrated schematically and exemplarily by Figs. 6A and 6B, a local binary pattern on a digital grayscale image can be represented by a map b mapping pixels of the image, which can correspond to positions on the surface of a coating applied to a substrate, to a respective binary vector, whose entries are either 1 or 0. Hence, the map b can be written as b: ? -> {0, 1}S(3a) j &( / ) , (3b) with the image space T comprising pixels indexed by j and s corresponding to the size of the vector b(J). It is understood that j,s e N, and that the image space P could also be represented, i.e. indexed or coordinated, by two natural number instead of just one, since it is spatially two-dimensional. The number s, and hence the size of the vector b(J), indicates the number of reference pixels considered for each pixel j. Preferably, the binary vector b(J), whose components are written herein as { ( / )}#=i..s, is chosen to satisfy f Q) ■= f1if f( + a < f(jf) V = l.. s ,

[0109] <-0 else (4) with {jf f=1..s indicating the s reference positions for the pixel j, f(j) and / ( / » indicating the heights, i.e. the gray values, for the pixels j and respectively, and a corresponding to a predefined defect detection margin.

[0110] The parameter a may be chosen depending on Q and s, wherein this dependence may be chosen differently for each type of defect to be detected. For a given candidate defect position j, several binary vectors b(J) may then be determined, each of them corresponding to a different type of defect to be detected and hence being determined using a different value of a. As already indicated further above, also the sign of a, the inequality sign in above equation (4) and / or the identification of pixel gray values with heights via the function f may be inverted depending on a type of defect to be detected. In particular, a > 0 may be chosen for detecting depression-type defects like valley-type or crater-type defects, and a < 0 may be chosen for detecting elevation-type defects like ridge-type or particle-type defects. According to another example, equation (4) with a > 0 and values of increasing with increasing coating surface heights as represented by increasingly light pixel gray values, could be used for detecting valley-like or crater- 1 ike defects, whereas equation (4) with a > 0 and inverted / could be used for detecting ridge-like or particle-like defects. With inverted / , increasing coating surface heights would be represented by increasingly dark pixel gray values. Thus, in other words, a gray level image used for detecting the defects may be inverted depending on whether depression-type or elevation-type defects are to be detected.

[0111] The reference positions can, as indicated in Figs. 6A and 6B, be chosen to all lie on a common circle around the positions j for which the binary vector is determined, particularly distributed equidistantly along the circumference of the circle. In Figs. 6A and 6B, which show circles with two different radii, respectively, the reference pixels are indicated by instead of = l.. s = 8. It will be understood from Figs. 6A and 6B that the reference positions themselves may be defined with respect to a continuous geometry, whereas the reference pixels corresponding to the reference positions can be determined by determining to which pixel a given reference position corresponds, i.e. into which pixel region the reference position falls. Hence, depending on, for instance, the radius of the circle on which the reference positions are chosen to lie and the number of reference positions s, the pixel set corresponding to the geometry with respect to which the reference positions are chosen may differ more or less from the reference geometry. In Fig. 6A, for instance, the radius of the reference circle has been chosen to be Q = 1 dpx, wherein dpxis the pixel width, and the number s of reference positions is equal to 8, which results in a square-shaped pixel set having a side length of three pixels. This square-shaped pixel set could be regarded as the reference pixel region for the candidate defect pixel indicated by j in Fig. 6A. Meanwhile, in Fig. 6B, the circle has been chosen to have a radius of two pixel widths, i.e. Q = 2 dpx, while the number s of reference positions is still 8, which results in a pixel set having a diamond shape with equal side lengths, which could also be regarded as a rotated square, the side lengths being larger than those of the square-shaped reference pixel region of Fig. 6A. From the pixel sets corresponding to the reference geometry, i.e. the reference pixel regions, the gray values indicating the heights of the respective coating surface positions are taken as reference height data for the height at the candidate defect position j.

[0112] Fig. 7 shows schematically and exemplarily a pixel region of a grayscale image comprising pixels with different gray values. A local binary patterning as schematically shown in Fig. 6A has been applied to the pixel region, associating the binary vector b(J) = (l,0,0,l,0,l,0,l)Tto the center pixel. In this case, the detection margin parameter a can be considered to be zero, i.e. a = 0. In line with the above formula forthe vector b(j), its entries are equal to 1 forthose reference pixels which have a higher gray value, which in this case corresponds to a lighter shade of gray, as compared to the center pixel, and equal to 0 otherwise. The pixel region shown in Fig. 7 does not indicate the presence of a defect. In contrast, the pixel regions shown schematically and exemplarily in Fig. 8A and 8B indicate the presence of a valley-type defect.

[0113] The local binary patterning applied according to Fig. 8A corresponds to the one applied according to Fig. 7. However, the pixel values in the shown pixel regions are different. The two different pixel configurations shown in Fig. 8A have in common that the middle row of pixels are darker than the top and the bottom row of pixels, which indicates the presence of a valley-type defect in both cases. However, in the left of the pixel configurations shown in Fig. 8A, the center pixel value is higher than the pixel values to its left and to its right, whereas in the right of the pixel configurations shown in Fig. 8A, the pixel value in the center is lower than the pixel values to its left and to its right. This difference between the two pixel configurations leads to different binary vectors associated to the respective center pixels, even though both pixel configurations are similar in that they indicate a valley-type defect. In fact, the two pixel configurations shown in Fig. 8A could be pixel configurations arranged at two different, possibly even close, axial positions along one and the same valley-type defect, the difference between the two pixel configurations corresponding to a random fluctuation of the gray values along the valley, which is however negligible in comparison to the respective difference with respect to the much higher, and in this case even maximal, gray values, i.e. white values, outside of the valley. While for the local binary patterns illustrated in Fig. 8A the parameter a was chosen to be equal to zero, forthe local binary patterns illustrated by Fig. 8B the parameter a was chosen to be greater than zero, i.e. a > 0. The pixel configurations shown in Fig. 8B themselves are, however, the same as the pixel configurations shown in Fig. 8A. The non-zero parameter a has the effect that, despite the difference in pixel configuration between the left and the right, the binary vector associated to the center pixel is the same. The local binary patterning illustrated by Fig. 8B can therefore be regarded as being more desirable than the one applied according to Fig. 8A, since it is to some degree insensitive against irrelevant variations in pixel values and hence reflects defect features more accurately.

[0114] In order to decide whether a pixel j belongs to a defect, i.e. in order to decide whether a candidate defect position is an actual defect position, the binary vector b(J) associated to the pixel j by the local binary pattern is compared to one or more reference binary vectors indicating the presence of a defect of a respective type. For instance, for a local binary pattern of the type as shown in Figs. 6A to 8B, and in consideration of the possible inversions of a, the inequality sign in equation (4) and / or the underlying image, a reference binary vector associated with ridge-type defects could be brefridge(j) = (1, 0,0,0, 1,0, 0,0)T, and a reference binary vector associated with valley-type defects could be brefvadey(j) = (0,l,l,l,0,l,l,l)T. Both ridge-type and valley-type defects can arise from cracks in the coating and / or the substrate and could therefore be classified accordingly. For a particle-type defect, a reference binary vector could be, for instance, 6ref,particie( / ) = (0,0,0,0,0,0,0,0)T, and a reference binary vector associated with crater-type defects could be brefcrater(j) = (1,1,1,1,1,1,1,1)T.

[0115] Depending on a size of the respective defect and the chosen parameters of the local binary pattern, particularly Q, the reference binary vectors may be chosen differently. For instance, ridge-type defects may also be represented by 6ref, ridge 0) = (l,0,l,0)Tor 6ref,ridge ( / ) = (l,l,0,0,0,0,0,l,l,0,0,0,0,0,0)T, and valley-type defects by 6ref,vaiiey( / ) = (0,l,0,l)Tor ^ref, valley ( / ) = (0,0,l,l,l,l,l,0,0,l,l,l,l,l,l,l)T.

[0116] Generally, a reference binary vector for each defect type to be detected may be predefined heuristically. In particular, for any given defect type to be detected, a reference binary vector may be predefined in consideration of a) the parameters of a respective local binary pattern used for detection and b) a typical binary vector expected to result from the respective local binary pattern for the given defect type. In order to determine whether a given candidate defect position belongs to a given type of defect, the defect detecting unit 102 may be configured to compare a binary vector determined for the given candidate defect position to the reference binary vector defined for the given type of defect by determining a Hamming distance between the two vectors. Instead of a Hamming distance, also other distance measured may be used, wherein it may also be selected based on a type of defect to be detected which distance measure to use.

[0117] An exemplary alternative distance measure could be defined based on an assignment, to a given binary vector, of a number of segments in the binary vector, particularly a number of segments of consecutive entries equal to cxand hence in the above case equal to 1. A distance between a) a binary vector associated to a candidate defect position and b) a reference vector defined for a type of defect to be detected could then be defined such that it corresponds to a difference between the respective numbers of segments in the two vectors. For instance, for detecting valley-type defects with s = 8, the number of segments with consecutive “1 ”s in a binary vector associated to a candidate defect position may be determined and then compared to the number of segments with consecutive “1 ”s in the aboveindicated reference binary vector brefvaney( ) = (0,1, 1,1,0, 1,1, 1)T, which is equal to 2.

[0118] Figs. 9A and 9B show exemplarily rendered images of coating surfaces on which, using height data and local binary patterns as outlined above, defects have been detected, wherein bounding boxes around the detected defects have been transferred to the rendered images.

[0119] Fig. 10 shows on its left height data image segments corresponding to different defect classes on a coating surface, and on its right respectively corresponding rendered image segments. Going from the top to the bottom of the five image pairs, the first two image pairs show particle-type defects, the third image pair shows crack-type defects, and the fourth and the fifth image pair show crater-type defects. Along the further defect types that can be detected by the defect detecting unit 102 are, for instance, flakes and bubbles.

[0120] After the defects have been detected, they can be classified. Hence, the system 100 may additionally comprise a classifying unit configured to classify the detected defects based on heights indicated by the height data for the detected defects. For instance, the classifying unit may receive as its input the image segments of the grayscale images showing the detected defects, which may have been cropped from complete images along respective bounding boxes. In other words, the classifying unit may be configured to classify the detected defects in grayscale image segments showing the detected defects, based on the pixel values of the respective image segments. Additionally or alternatively, the classifying unit may be configured to classify the detected defects based on rendered image data corresponding to respectively detected defects.

[0121] In particular, the local binary patterns, which may already have been determined for the image segments during defect detection as described above, may also be used for classifying the detected defects. Concretely, the classifying unit may, for instance, receive the binary vectors b(j) for all pixels j in a height image segment comprising a detected defect as input for classifying the defect.

[0122] Optionally, the classifying unit is configured to classify the detected defects using a trained machine learning model. The machine learning model may be trained based on training data comprising pairs of training input data and training output data, wherein the training input data are prepared like the input data to be received by the trained machine learning model, i.e. after deployment. For instance, if the trained machine learning model is supposed to receive as its input height image segments of single defects, it is preferably also trained with height image segments comprising single defects. Moreover, the machine learning model may be trained using training data corresponding to as many defect types as possible. In this way, the trained machine learning model may be able to classify various different defects accurately. The training output data may correspond to assignments of a defect class to the respective training input data. Hence, the training output data may be provided as a form of ground truth.

[0123] While using a trained machine learning model has been observed to allow for reliable classification results, also unsupervised classification algorithms or yet other artificial intelligences may be used for classifying the detected defects.

[0124] In an example, the machine learning model used can comprise a convolutional neural network configured to receive, as its input, the respective image segments. Hence, in this case no particular features derived from the image segments are received as input by the network, even though, in other embodiments, this would be possible. The features may rather be extracted by the convolutional neural network itself.

[0125] The convolutional neural network may particularly comprise a plurality of convolutional layers for providing respective feature maps. In other words, the features may be determined by the convolutional neural network, and not beforehand. The number of convolutional lay- ers in the convolutional neural network can be equal to five, for instance. However, in principal, other numbers of convolutional layers can be used as well. Fig. 1 1 schematically and exemplarily illustrates the processing carried out by a convolutional layer of the convolutional neural network for an exemplary image I consisting of pixel values 0 and 1 , using a kernel K that is specific for the respective convolutional layer.

[0126] Moreover, the convolutional neural network may comprise one or more pooling layers for pooling a respective one of the feature maps into a corresponding pooled feature map. For instance, max pooling may be applied. Furthermore, the convolutional newer network may carry out spatial pyramid pooling. In particular, after the last convolutional layer, spatial pyramid pooling using the three sub-pooling structures 5 x 5, 3 x 3 and l x l may be applied.

[0127] A data processing effected according to a preferred convolutional neural network structure is illustrated schematically and exemplarily in Fig. 12. As indicated therein, the input data I, which could be a grayscale image of a coating surface whose gray levels indicate coating surface heights, is convoluted, in the first convolutional layer, by one or more kernels of substantially arbitrary size into an equal number of feature maps (in the illustrated, nonlimiting, case, two kernels are used, leading to two feature maps). The features maps, whose sizes can be equal to the input image I or not, are subsequently pooled by a first pooling layer, leading to first pooled features maps of smaller size. This procedure is repeated further four times, wherein, however, in the last, i.e. fifth, pooling layer, spatial pyramid pooling is applied. In contrast to the previous poolings, the spatial pyramid pooling increases the number of features maps. Assuming, as illustrated, the fifth convolutional layer results in Fsfeature maps, the spatial pyramid pooling results in 3 blocks of Fsfeature maps, wherein the size of the feature maps in the three different blocks is in this case 5 x 5, 3 x 3 and l x l, respectively, i.e. independently on the size of the previous feature maps or the initial input I. The feature maps resulting from the spatial pyramid pooling are subsequently flattened, such as to a vector of length (5 x 5 + 3 x 3 + 1 x 1) x JVF, wherein the flattened output, particularly the vector, is provided as an input to a single fully connected layer which maps it to respective defect classes C C2, et cetera. It will be understood that Fig. 12 just illustrates the basic structure of the convolutional neural network, wherein it will be determined by the training of the network how exactly the network layers map their respective inputs to their respective outputs, as indicated by the arrows in Fig. 12. Fig. 13 shows schematically and exemplarily a method 200 for determining a quality of a surface of a coating applied on a substrate. The method 200 includes, in a step 201 , providing height data, wherein the height data are indicative of heights of the surface of the coating relative to the substrate. In a second step 202, defects in the surface of coating are detected by a) determining, for any candidate defect position of the surface of coating, height difference data based on the height data, wherein the height difference data for a candidate defect position are indicative of differences between the height at the candidate defect position and heights at a plurality of reference positions, the reference positions being determined based on the candidate defect position according to a predefined relation, and b) comparing the height difference data to predetermined reference height difference data indicating a defect. In a following step 203 of the method 200, the quality of the surface of the coating applied on the substrate may be determined based on the detected defects.

[0128] Insofar as reference was made above to a determination of a quality of surface of a coating applied on a substrate, the determined quality could also be used as an indicator for a quality of the coating in general, particularly not only of its surface when applied to a substrate, but also of its bulk and / or independently of whether it is applied on a substrate or not. Moreover, the determined quality can also serve as an indicator for a quality of the respective coating application process, i.e. for how well the coating has been applied to the respective substrate. Based on the quality, the coating and / or the application procedure can then be adjusted, for instance.

[0129] Moreover, although the above embodiments were mainly described with reference to assessments of coatings, the same or similar embodiments could be used for assessing any surfaces. For instance, surfaces may in general be assessed with respect to the presence and type of particular irregularities. Depending on the type of surfaces, the height data may then be acquired with suitable other data acquisition devices. The surfaces, and hence the height data, may be microscopic or macroscopic.

[0130] Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.

[0131] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single unit or device may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0132] Procedures like the providing of height data, the detecting of defects, the determining of a quality, the classifying of detected defects, the providing of an image, the use of a machine learning model and / orthe training thereof by, inter alia, providing a machine learning model and training data, et cetera, performed by one or several units or devices can be performed by any other number of units or devices. These procedures can be implemented as program code means of a computer program and / or as dedicated hardware.

[0133] A computer program may be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state medium, supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

[0134] Any reference signs in the claims should not be construed as limiting the scope.

[0135] A system for determining a coating surface quality is presented, comprising a height data providing unit configured to provide height data indicative of heights of the coating surface relative to a substrate, and a defect detecting unit configured to detect defects in the surface by a) determining, for any candidate defect position, height difference data, the height difference data for a candidate defect position indicating differences between the height at the candidate defect position and heights at a plurality of reference positions, and b) comparing the height difference data to predetermined reference height difference data indicating a defect. The system further comprises a quality determining unit configured to determine the quality of the surface based on the detected defects. This allows for improved coating quality assessments, based on efficiently and yet accurately detected defects.

Claims

CLAIMS1 . A system (100) for determining a quality of a surface of a coating applied on a substrate, wherein the system comprises: a height data providing unit (101) configured to provide height data, wherein the height data are indicative of heights of the surface of the coating relative to the substrate, a defect detecting unit (102) configured to detect defects in the surface of the coating by a) determining, for any candidate defect position of the surface of the coating, height difference data based on the height data, wherein the height difference data for a candidate defect position are indicative of differences between the height at the candidate defect position and heights at a plurality of reference positions, the reference positions being determined based on the candidate defect position according to a predefined relation, and b) comparing the height difference data to predetermined reference height difference data indicating a defect, and a quality determining unit (103) configured to determine the quality of the surface based on the detected defects.

2. The system as defined in claim 1 , wherein the defect detecting unit (102) is configured to detect each defect individually.

3. The system as defined in any of the preceding claims, wherein the height difference data correspond to a local binary patterning of the height data.

4. The system as defined in any of the preceding claims, further comprising a classifying unit configured to classify the detected defects into one of a plurality of defect classes, wherein the quality determining unit (103) is configured to determine the quality of the surface based on the classified detected defects.

5. The system as defined in claim 4, wherein the defect classes are indicative of any of the following: cracks, craters, particles, flakes, bubbles.

6. The system as defined in any of claims 4 and 5, wherein the classifying unit is configured to classify the detected defects based on heights indicated by the height data for the detected defect.

7. The system as defined in any of claims 4 to 6, further comprising a rendered image providing unit configured to provide a rendered image corresponding to a view onto the surface of the coating, wherein the classifying unit is configured to classify the detected defects based on rendered image data corresponding to the detected defect.

8. The system as defined in any of claims 4 to 7, wherein the classifying unit is configured to classify the detected defects using a trained machine learning model.

9. The system as defined in claim 8, wherein the machine learning model comprises a convolutional neural network.

10. The system as defined in claim 9, wherein the convolutional neural network comprises a plurality of convolutional layers for providing respective feature maps.1 1. The system as defined in claim 10, wherein the convolutional neural network comprises at least one pooling layer for pooling a respective one of the feature maps into a corresponding pooled feature map.

12. The system as defined in claim 11 , wherein the convolutional neural network applies spatial pyramid pooling.

13. The system as defined in any of claims 8 to 12, wherein the machine learning model comprises a transformer architecture.

14. A method (200) for determining a quality of a surface of a coating applied on a substrate, wherein the method includes: providing (201) height data, wherein the height data are indicative of heights of the surface of the coating relative to the substrate, detecting (202) defects in the surface of the coating by a) determining, for any candidate defect position of the surface of the coating, height difference data based onthe height data, wherein the height difference data for a candidate defect position are indicative of differences between the height at the candidate defect position and heights at a plurality of reference positions, the reference positions being determined based on the candidate defect position according to a predefined relation, and b) comparing the height difference data to predetermined reference height difference data indicating a defect, and determining (203) the quality of the surface based on the detected defects.

15. A computer program for determining a quality of a surface of a coating applied on a substrate, wherein the computer program comprises instructions causing the system as defined in claim 1 to execute the method as defined in claim 14.

Images