System for determining quality of coated surface

Abstract

It is proposed a system (100) for determining the quality of a coating surface, the system comprising: a height data providing unit (101) configured to provide height data indicative of a height of the coating surface relative to a substrate; and a defect detection unit (102) configured to detect a defect in the surface by: a) determining height difference data for any candidate defect position, the height difference data for the candidate defect position indicating a difference between a height at the candidate defect position and a height at a plurality of reference positions, and b) comparing the height difference data with predetermined reference height difference data indicative of a defect. The system further comprises a quality determination unit (103) configured to determine a quality of the surface based on the detected defect. This allows for improved coating quality assessment based on efficiently and accurately detected defects.

Description

Technical Field

[0001] This invention relates to a system, method, and computer program for determining the quality of a surface coating applied to a substrate. Background Technology

[0002] The article “Rapid surface defects detection in wire and arc Additional Manufacturing based on laser profilometer” by C. Huang et al., published in Measurement, Volume 189 (2022), describes a laser inspection system for monitoring surface defects that occur when manufacturing metal parts using wire arc additive manufacturing. Upon detection of surface defects, appropriate remedial actions should be taken during manufacturing to prevent the formation of internal defects in the manufactured metal parts due to the surface defects being covered up.

[0003] The article "Online Convolutional Neural Network-based anomaly detection and quality control for Fused Filament Fabrication process" published by J. Lyu et al. in Volume 16 (2021) of Virtual and Physical Prototyping involves detecting anomalies in laser scanning data of corresponding layers generated during the fabrication of filaments used to manufacture metal parts, and adjusting the thickness of subsequent layers based on the detected anomalies.

[0004] The article "A DeepLearning-based 3D Defect Quantitative Inspection System in CC Products Surface" published by L. Zhao et al. in Volume 20 (2020) of Sensors relates to an inspection strategy for evaluating surfaces in continuous casting production lines used to manufacture metal products.

[0005] For processes involving the application of paint or other coatings to substrates such as automotive parts, quality assessment requires appropriate quality metrics. Characteristics that can be used for such metrics include the quantity and type of defects on the coated surface. However, detecting and classifying defects on the coated surface is prone to error, especially when performed manually. On the other hand, even when only partial detection and classification are performed on a computer, obtaining sufficiently accurate results can quickly require relatively large computational resources. Therefore, there is a need to improve the quality assessment of coatings applied to substrates. Summary of the Invention

[0006] The purpose of this invention is to allow for improved quality assessment of coatings applied to substrates.

[0007] In a first aspect, the present invention relates to a system for determining the quality of a surface of a coating applied to a substrate. The system includes: i) a height data providing unit configured to provide height data, wherein the height data indicates the height of the surface of the coating relative to the substrate; and ii) a defect detection unit. The defect detection unit is configured to detect defects in the surface of the coating by: a) determining height difference data based on the height data for any candidate defect location on the surface of the coating, wherein the height difference data for the candidate defect location indicates the difference between the height at the candidate defect location and the height at a plurality of reference locations determined based on the candidate defect location according to a predefined relationship; and b) comparing the height difference data with predetermined reference height difference data indicating a defect. The system further includes iii) a quality determination unit configured to determine the quality of the surface based on the detected defects.

[0008] Since the surface quality of the coating applied to the substrate is determined by measuring height difference data for any candidate defect locations on the coating surface based on height data indicating the height of the coating surface relative to the substrate, and based on defects detected on the coating surface, where the height difference data for candidate defect locations indicates the difference between the height at the candidate defect location and the height at multiple reference locations determined according to predefined relationships based on the candidate defect locations, the surface quality can be determined based on efficiently and accurately detected defects. In this way, improved quality assessment of coatings applied to the substrate can be achieved.

[0009] Height data can be viewed as a three-dimensional dataset, where two dimensions indicate the position of the coating surface or the location of the coating surface itself, and the third dimension indicates the height of the coating surface relative to the substrate at that position. The height data providing unit can be configured to provide height data based on a surface-based three-dimensional scan. The scan can be performed using a color confocal measurement system, such as FocalSpec from LMI Technologies.

[0010] Height data can be discretized. Discretized height data is understood as height data taking discontinuously distributed values, specifically in two dimensions indicating the surface of the coating or the location of the coating surface. Thus, for example, height data can correspond to a) the discrete coordinates of the surface of the coating and b) a set of associated heights. The discrete coordinates of the surface of the coating can be generated by a coordinate grid that is actually laid on the surface of the coating. The surface of the coating can then be considered as being divided into individual tiles corresponding to the coordinate grid.

[0011] In fact, height data can be provided in the form of monochrome digital images. The image can be divided into pixels whose coordinates correspond to the coordinates of the surface of the coating, where each pixel can be associated with a pixel value indicating the height of the surface at the corresponding coordinate location. For example, the pixel value can be a grayscale value, in which case a lighter grayscale value can indicate a greater surface height than a darker grayscale value, and vice versa.

[0012] Height data (especially if provided in the form of a monochrome digital image) can be determined by projecting a 3D scan of the surface onto a plane, which may be referred to as the projection plane or image plane. This plane may specifically correspond to the surface of the substrate. The height data can be further determined based on interpolation of the projected 3D scan data. This allows for 3D scanning of the surface at a relatively low resolution, even when a relatively high resolution is desired for the height data. In the example, height data can be obtained from the scan data using MountainsMap software from DigitalSurf.

[0013] For accurate defect detection based on height data, a sufficiently smooth and / or flat substrate (i.e., the surface of the substrate) can be advantageous. For example, variations in the height and / or curvature of the substrate surface can be below a predetermined threshold. However, even for relatively irregular substrates, the defect detection unit may be able to detect defects by distinguishing them from irregularities in the coating originating from the substrate.

[0014] The defect detection unit is preferably configured to detect each defect individually. Therefore, defects are preferably distinguished from one another. This allows for the counting of defects, where the number or density of detected defects provides a good measure of the surface quality of the coating.

[0015] Specifically, the defect detection unit can be configured to determine the location of each defect on the surface of the coating based on height data, i.e., to locate each defect. Knowing the location of the defects allows for a particularly rich measurement of the surface quality of the coating.

[0016] The height difference data preferably indicates the height difference along the surface of the coating. Therefore, the defect detection unit is preferably configured to determine the height difference data of corresponding candidate defect locations on the surface of the coating based on the height data, such that the determined height difference data indicates the difference between the coating surface height at the candidate defect location and the coating surface height at a plurality of reference locations. Specifically, the height difference data can be determined by determining the surface height difference between the candidate defect location and the plurality of reference locations based on the height data. Therefore, the predefined relationship upon which the reference locations are determined based on the corresponding candidate defect locations can be specifically predefined such that the reference locations, like the candidate defect locations, are locations on the coating surface.

[0017] The defect detection unit can be configured to detect defects based on a transformation from height data to a feature map, wherein the feature map associates features with positions on the surface of the coating, and the feature associated with any given position indicates the difference between the corresponding height (in particular, the coating surface height) indicated by the height data at that position and the height (in particular, the coating surface height) indicated by the height data at a reference position. Therefore, the feature map, as a preferred type of height difference data, treats all positions as candidate defect locations. Candidate defect locations are preferably understood herein as locations that potentially belong to defects, particularly locations where it is still necessary to determine whether they belong to defects.

[0018] A feature map can include all locations on the surface of the coating; that is, features can be determined for all (i.e., all known and / or available) locations on the surface of the coating. For example, if the height data is represented as a digital image such that locations on the surface of the coating correspond to pixels, then the feature map can include all pixels of the digital image. In other words, a feature map can be constructed by determining the features of each pixel in the digital image (i.e., based on the height indicated by the height data of the corresponding pixel and the height indicated by the height data of a reference pixel). However, in principle, a feature map can also include only a subset of all locations, such as only a subset of pixels. This can save computational resources.

[0019] Preferably, the reference location is located in the neighborhood of the location where the feature is to be determined. The neighborhood can be defined, for example, as the maximum distance between the reference location and the location where the feature is to be determined. The maximum distance can be defined relative to and / or depending on the resolution of the corresponding digital image on which the defect detection is performed. As already indicated, the location can correspond to a pixel, i.e., a pixel position. The neighborhood where the reference location can be located can, in particular, represent a region of the surface of the coating.

[0020] The feature map can include more than one feature for each coating surface location. For each feature associated with the same location, different sets of reference locations and therefore different height difference data can be considered as the basis. In other words, a separate relation to be satisfied by the reference locations can be predefined for each feature associated with a candidate defect location. It has been found that even when all these locations belong to the same defect, the local neighborhood around the surface location can vary significantly in height data, where these variations can lead to different height difference data. For example, image pixels corresponding to the interior region of a defect are often surrounded by pixels belonging to the defect, while pixels belonging to the boundary region of the defect are often partially surrounded by pixels not belonging to the defect. By considering several features for each surface location, and for each feature, considering different reference locations in neighborhoods of different sizes that may be located around the corresponding candidate defect location, defect locations can be detected more accurately.

[0021] Elevation difference data can specifically correspond to local binary patterning of elevation data. Local binary patterning of elevation data can be understood as a specific transformation of elevation data into a feature map as indicated above. In other words, a local binary pattern (i.e., the result of local binary patterning) can be understood as a specific type of feature map, namely, the feature map of the specific type of elevation difference data indicated above.

[0022] As understood in this paper, a local binary pattern can be represented by mapping the positions on the surface of the coating (which may correspond to pixels in an image of the surface of the coating, particularly pixels in a grayscale image (where grayscale values ​​indicate surface height)) to corresponding binary vectors (i.e., vectors whose entries can only take one of two predefined values). To represent. For example, in discretized high-resolution data. height With the surface of the coating Location of upper discrete distribution In the case of association, mapping It can be defined mathematically as

[0023] (1a)

[0024] (1b)

[0025] Where, vector of Quantity satisfy

[0026] (2)

[0027] in, Indicates a reference position, that is, satisfies the condition relative to the position. The reference position of the predefined relationship, and This corresponds to a predefined margin.

[0028] Predefined value and This can be understood as indicating the category of height difference that determines the position of a binary vector relative to a corresponding reference position. For example, the predefined values ​​can be one and zero, i.e. and .margin It can be a predefined constant. (By specifying...) By selecting different values, the sensitivity of the defect detection unit can be controlled. Optionally, The selection can be based on variations in the height and / or curvature of the surface of the substrate on which the coating is applied. However, this is only based on the availability of options. One of many possible factors. In particular, as indicated below, it can depend on the computation of the corresponding local binary vector. Location Compared with the reference position considered for this calculation Choose based on the distance between them In fact, It can also be "learning," that is, based on utilizing several different The value is selected based on the evaluation of the detection quality.

[0029] In addition, the choice can be made depending on the type of defect to be detected. The symbol. For example, to detect defects that are expected to correspond to depressions in the coating surface, such as valley defects or pit defects, the symbol can be... Select positive. On the other hand, to detect defects that are expected to correspond to protrusions in the coating surface, such as ridge-like or granular defects, one can... The choice is negative. In the latter case, that is, in order to detect defects that are expected to correspond to protrusions in the coating surface, the inequality sign in equation (2) above can be additionally inverted. Alternatively, only the function in equation (2) above can be... Invert the value to detect defects that are expected to correspond to bumps in the coating surface. For example, if the function... If the pixel values ​​of the grayscale image representing the height of the coating surface are indicated, this will correspond to inverting the image.

[0030] Due to vectors It also depends on the height data at the reference location (where the reference location can be located at the position of the calculated vector). (within the surrounding region), therefore, in alternative terminology, the local binary pattern can also be understood as a mapping from the region or neighborhood to a vector (i.e., instead of a mapping from location to a vector). In fact, due to vectors It actually depends on the height data, i.e. and In other alternative terms, local binary patterns can be derived from a region or neighborhood of an image (in a mapping). Below, especially sets (That is, to understand it in the form of a mapping from height to vector.)

[0031] Reference position It can be selected as all reference positions having the same position as the determined binary vector. same distance And specifically, with the set of reference positions. Its nearest neighbor has the same distance The first of these optional requirements can be written as The predefined distance function is For example, distance function It can generate the Euclidean distance between two points that it receives as a parameter. For example, if the reference positions are enumerated in order of their angular positions on the circle, an additional optional requirement (i.e., the reference positions can be chosen to be all equally spaced on the circle around the position of the binary vector) can be written as .

[0032] This parameter Preferably depending on and The selection is made based on a) the size of the neighborhood considered for computing the local binary vector and b) the number of reference locations considered for that neighborhood. Height Its associated reference position Height A predefined margin between them. Furthermore... Different options can be selected for each type of defect to be detected. Therefore, in particular, right and The dependency can be selected based on the type of defect to be detected.

[0033] Each candidate defect location can also determine more than one binary vector, where, for each of the binary vectors associated with a given candidate defect location, a distinct set of reference locations is considered as the basis. For example, several distances can be considered. and As mentioned above, this allows for more accurate detection of defects, especially their shape. Several binary vectors can also be combined into a larger single binary vector.

[0034] Specifically, for each type of defect to be detected, several local binary patterns can be defined, where the parameters of the corresponding local binary patterns are selected based on the corresponding type of defect. Therefore, for each type of defect to be detected, several binary vectors for each candidate defect location can be computed based on different sets of reference locations, where the definition of the several binary vectors for each candidate defect location can be selected according to the corresponding defect type to be detected. To detect any type of defect, the local binary patterns defined for all types of defects can be used together as a basis.

[0035] The defect detection unit can be configured to detect defects on the surface of the coating based on binary vectors generated by local binary patterning. That is, the defect detection unit can be configured to determine whether a given location on the surface of the coating is a defect based on a binary vector determined for that location and possibly further based on binary vectors determined for adjacent locations.

[0036] For example, the distance between the binary vector at each location and each defect indicator binary vector in the set of defect indicator binary vectors can be calculated. The calculated distance can, for example, refer to the Hamming distance between the binary vector at the corresponding location and the corresponding defect indicator binary vector. The Hamming distance between two vectors corresponds to the number of entries in the two vectors whose positions in the corresponding vectors are the same but whose values ​​are not equal. However, many other distance metrics can be used. In fact, instead of determining the distance, the binary vector at the corresponding location can be compared differently with the defect indicator binary vector, i.e., based on a non-distance metric. If the similarity between the compared corresponding binary vectors exceeds a predefined threshold, the defect detection unit can conclude that the corresponding location belongs to a defect. In other words, it can then be concluded that a defect exists at the corresponding location.

[0037] For example, a defect indicator binary vector can indicate the presence of a specific type of defect, such as holes, mounds, and / or valleys. Therefore, several defect indicator binary vectors can be used, each indicating the presence of a different type of defect. Connected regions with locations associated with (i.e., mapped to) a binary vector can be identified as corresponding defects in the surface of the coating, these locations being within a predefined distance of the defect indicator binary vector, or similar to the defect indicator binary vector to a predefined degree, such as by a predefined similarity metric. Preferably, only connected regions with locations associated with a binary vector are identified as corresponding defects in the surface of that region, i.e., corresponding to the corresponding identical defects, these locations all being within a predefined distance of the same defect indicator binary vector, or similar to the defect indicator binary vector to a predefined degree, such as by a predefined similarity metric.

[0038] For example, a set of defect indication binary vectors can be established, and then access to the defect detection unit can be provided based on the manual indication of the defect in the height data (e.g., the projection of a three-dimensional coated surface scan onto a monochrome image of grayscale indicating the surface height), wherein a binary vector has been determined for the height data, such that by assuming the manual indication of the defect is the ground truth, the defect indication binary vector can be obtained from the binary vector determined for the location of the manually indicated defect in the height data.

[0039] Alternatively, defect indicator binary vectors can be predefined heuristically. In particular, defect indicator binary vectors can be predefined taking into account the definition of the corresponding local binary pattern and the typical binary vectors expected for the corresponding defects.

[0040] The system may further include a classification unit configured to classify detected defects into one of a variety of defect categories, wherein a quality determination unit may be configured to determine the quality of the surface based on the classified detected defects.

[0041] Surface quality can be determined based on the number and / or category of detected defects and / or the total area covered by the detected defects. The number and total area covered can be determined in a categorized manner. The objective quality level can be defined based on a threshold that depends on the number and / or category and / or total area covered by the detected defects.

[0042] Detected defects can be classified into one of several defect categories based on defect detection binary vectors. These defect detection binary vectors preferably each indicate a corresponding defect category and thus become category-defining binary vectors. For example, connected regions with locations associated with their binary vectors can be identified as defects corresponding to a category defined by the category-defining binary vectors, where all locations are within a predefined distance of the same category-defining binary vector, or similar to the category-defining binary vector to a predefined degree, such as that measured by a predefined similarity metric. In this way, connected regions on the surface of the coating can be associated with defects of a specific category. However, detected defects can also be classified without reference to the defect indicator binary vectors and / or the category-defining binary vectors. That is, it is also an option to use the defect indicator binary vectors for defect classification without additional reference after defects have been detected based on them.

[0043] The classification unit can classify defects together with an estimate of the probability of a confidence level indicating the correctness of the classification. Thus, for example, the classification unit can be configured to provide one or more categories for each of a plurality of detected defects to be classified, and to provide the probability that the defect belongs to the corresponding category for each of the one or more categories.

[0044] Regardless of the classification method, the defect category can indicate any of the following: cracks, pits, particles, spalling, and bubbles. Therefore, the system allows the detection unit to determine whether a coating includes any of these defects, and particularly allows for differentiation using a classification unit when the coating does include any of these defects. These categories of defects may inadvertently originate from the coating material itself and / or the coating process, i.e., from the manner in which the coating is applied. For example, particles, spalling, or bubbles may be inadvertently encapsulated during coating application to a substrate, and spalling may have already peeled off from other substrate samples, where the substrate being evaluated for its coating was transported in the same shipping box or rack as other substrate samples. It should be understood that other defect categories can also be identified by providing a corresponding additional binary vector of category definitions or by adjusting the classification unit accordingly in other aspects described below.

[0045] The classification unit can be configured to classify detected defects based on their height, indicated by the height data of the detected defects. For example, the classification unit can receive a portion of the height data containing detected defects as input and provide the category of the detected defects as output. Classifying defects based on height data can be efficient because additional data is not required.

[0046] However, it may be preferable that the system includes a rendered image providing unit configured to provide a rendered image corresponding to a view on the surface of the coating, wherein the classification unit is configured to classify detected defects based on rendered image data corresponding to the detected defects. Rendered image data has been found to allow for particularly reliable defect classification.

[0047] Height data and rendered images can be registered together to enable the location of defects detected based on the height data within the rendered image. The rendered image providing unit can be configured to determine the rendered image based on 3D scan data, and can also determine the height data based on the 3D scan data.

[0048] It should be understood that while height data or rendered image data can be used to determine the shape of a detected defect, and defect classification could theoretically be performed based on the defect shape, it is preferable not to take this additional step. Instead, the classification unit can directly receive height data or rendered image data corresponding to the detected defect for classification. The shape of the defect can still be implicitly used by the classification unit for classification, but this is not necessary.

[0049] Defects can also be classified based on other features. For example, they can be classified based on corresponding local binary patterns (i.e., particularly based on their associated binary vectors). Defects are classified. However, it should be emphasized that defects, especially localized defects, can already be detected using local binary patterns. Therefore, it is preferable that local binary patterns have been used before defects are classified, where the use of local binary patterns for defect classification is considered optional.

[0050] The defect detection unit can be configured to determine the bounding box of each detected defect in height and / or rendered image data and / or local binary pattern, wherein the classification unit can be configured to receive height and / or rendered image data and / or binary vectors corresponding to the bounding box (particularly its interior) as input, and to provide classification of defects in the bounding box based on the received data.

[0051] The classification unit can be configured to classify detected defects using a trained machine learning model. This has been found to lead to good classification results. However, other artificial intelligence techniques can also be employed. For example, instead of a trained machine learning model, unsupervised image processing algorithms can be used to classify detected defects.

[0052] If a machine learning model is used, it can be trained based on training data consisting of pairs of training input and training output data. In each pair, the training input data includes height data and / or rendered image data corresponding to a single defect in the coating on the surface, and the training output data corresponds to the category of the corresponding defect. The categories in the training output data can be provided manually. For example, to generate training data, a trained person could classify images of defects based on whether the corresponding defect is considered a crack, pit, particle, spalling, or bubble defect.

[0053] The classification unit can be configured to select the machine learning model to use based on bounding boxes (e.g., based on the shape and / or size of the bounding boxes). Specifically, the input layer of the machine learning model can be selected accordingly. This allows for the use of bounding boxes of different shapes and / or sizes, making the bounding boxes suitable for the corresponding detected defects. Therefore, more efficient classification can be achieved because each defect may require processing less height data.

[0054] Specifically, the machine learning model may include a convolutional neural network. A convolutional neural network may be configured to receive any combination of the following features as its input: a binary vector corresponding to a local binary pattern determined for the location corresponding to the detected defect, height data corresponding to the detected defect, and rendered image data corresponding to the detected defect.

[0055] Convolutional neural networks preferably include multiple convolutional layers to provide corresponding feature maps. In particular, the number of convolutional layers in a convolutional neural network can be equal to 5.

[0056] Feature maps provided by the convolutional layers of a convolutional neural network, optionally used for classifying detected defects, should generally be distinguished from feature maps on which defects can be detected (especially from local binary patterns, which are preferably used for defect detection). However, additionally or alternatively, feature maps determined based on local binary patterns can also be used for defect classification and thus as input to the convolutional neural network.

[0057] A specific example of a feature map determined based on local binary patterns that can be used for defect classification is a digital image whose pixel values ​​are determined by binary vectors of local binary patterns. Such image-type feature maps can also be viewed as visual representations of the corresponding local binary patterns. Pixel values ​​can be determined, for example, by representing the binary vectors of the local binary patterns as binary strings, where the integers associated with the binary vectors can be defined as binary numbers corresponding to the binary strings, optionally converted to a decimal system.

[0058] A convolutional layer can correspond to the convolution of separately received input data by one or more kernels, with each kernel producing a corresponding feature map. The kernels can be determined during the training of the convolutional neural network.

[0059] Furthermore, the convolutional neural network may include at least one pooling layer for pooling corresponding feature maps in the feature maps into corresponding pooled feature maps. The at least one pooling layer may be adapted to receive a feature map provided by a previous convolutional layer as input and to provide an associated pooled feature map based on that feature map. In the example, max pooling may be applied.

[0060] Pooling is understood in this paper as a process of transforming a feature map by applying a function to the features of a feature map in one kernel of multiple predefined regions of the feature map and repeating this application of the function for each predefined region. In this way, a pooled feature map is generated, which corresponds to a mapping from the predefined regions to the features produced by applying the function to the corresponding regions. If the predefined regions cover the entire area where the feature map is defined, i.e., the entire coating surface area, then the pooled feature map can also be defined over that entire area. For example, a predefined region that can cover the entire area where the non-pooled feature is defined can correspond to a square of adjacent pixels in a digital image. The pooled feature map is typically smaller than the original non-pooled feature map that produced it. For example, if the predefined region (which may also be referred to as the "pooling region") corresponds to... If adjacent pixels form a square, then the pooled feature map can have a difference compared to the original feature map. Size. In the case of max pooling, the function applied to features within a predefined region corresponds to picking out the maximum value among the features.

[0061] In the variant, min-pooling can be applied. Max-pooling and min-pooling can be understood as special cases of extreme pooling. This can be understood as making the function applied to the features in the predefined region correspond to picking out the extreme values ​​in the features. In the case of min-pooling, the features correspond to the minimum values ​​in the predefined region.

[0062] Convolutional neural networks can also apply spatial pyramid pooling. Specifically, a three-sub-pooling structure can be used. , and Spatial pyramid pooling is a common sub-pooling method. However, many other sub-pooling structures can typically be used for spatial pyramid pooling. Regardless of the sub-pooling structure, spatial pyramid pooling allows for the classification of defects of arbitrary size that have been previously detected, because it results in the data objects to be classified having the same size, regardless of the size of the input provided to the convolutional neural network.

[0063] Alternatively, the initial input provided to the convolutional neural network can be resized to a uniform size, i.e., before being passed to the network. Specifically, for example, bounding boxes generated by defect detection can all be resized to a uniform shape and then provided to the convolutional neural network only as input. However, since significant variations in defect size have been observed, it has been found that applying spatial pyramid pooling instead of resizing the input to the convolutional neural network is advantageous for classification results.

[0064] In addition to convolutional and pooling layers, a convolutional neural network may include fully connected layers, specifically a single fully connected layer. Then, a spatial pyramid pooling layer may correspond to a pooling layer positioned between the last convolutional layer and the fully connected layer. Therefore, the spatial pyramid pooling layer is preferably the last pooling layer in the convolutional neural network.

[0065] Furthermore, machine learning models can include transformer architectures. This has been found to allow for good classification results.

[0066] The substrate material can be selected from any of the following: wood, wood veneer, paper, cardboard, hardboard, textiles, films, leather, nonwovens, plastics, glass, ceramics, mineral building materials, and metals. Furthermore, the substrate can be pre-coated. Therefore, a coating (i.e., a coating whose surface quality is to be determined) may have already been applied to the pre-coated substrate. For example, the coating may include, particularly consist of, or correspond to the following: varnish, primer, and / or topcoat. Furthermore, the coating can be applied by any of the following: scraping with a scraper, air mixing and / or airless spraying, roller coating, brush coating, electroplating, or a combination of the foregoing.

[0067] Another aspect of the invention relates to a method for determining the quality of the surface of a coating applied to a substrate, wherein the method includes, in a first step, i) providing height data, wherein the height data indicates the height of the surface of the coating relative to the substrate. In a second step, the method includes: ii) detecting defects in the surface of the coating by: a) determining height difference data based on the height data for any candidate defect location on the surface of the coating, wherein the height difference data for the candidate defect location indicates the difference between the height at the candidate defect location and the heights at a plurality of reference locations determined based on the candidate defect location according to a predefined relationship; and b) comparing the height difference data with predetermined reference height difference data indicating defects. In a third step, the method includes iii) determining the quality of the surface based on the detected defects. The method according to this aspect can be particularly a computer-implemented method.

[0068] Another aspect relates to a computer program for determining the quality of the surface of a coating applied to a substrate, wherein the computer program includes instructions to cause the aforementioned system or, typically, a data processing apparatus, to perform the aforementioned method.

[0069] The present invention also relates to the use of local binary modes for detecting and / or classifying defects in a coating applied to a substrate based on height data indicating the height of the coating surface relative to the substrate. As summarized in more detail above, the height data may correspond to a digital grayscale image of the coating surface, wherein pixel values ​​indicate the height of the coating surface at a corresponding location on the coating surface.

[0070] Furthermore, a system for training a machine learning model to classify defects in a coating applied to a substrate is proposed, namely, a training system. The training system includes: i) a model providing unit configured to provide a machine learning model to be trained; ii) a training data providing unit configured to provide training data including pairs of training input data and training output data, wherein the training input data in each pair includes any one of the following: a) height data, b) data of a feature map obtained from the height data via local binary patterning, and / or c) rendered image data corresponding to a single defect in the coating on the surface, and the training output data corresponds to the category of the corresponding defect; and iii) a training unit configured to train the machine learning model based on the training data.

[0071] Accordingly, a method for training a machine learning model to classify defects in a coating applied to a substrate is proposed, namely, a training method. This training method includes: i) providing a machine learning model to be trained; ii) providing training data comprising pairs of training input data and training output data, wherein the training input data in each pair includes height data and / or rendered image data corresponding to a single defect in the coating on the surface, and the training output data corresponds to the category of the corresponding defect; and iii) training the machine learning model based on the training data.

[0072] In another aspect, a computer program is provided for training a machine learning model to classify defects in a coating applied to a substrate, wherein the computer program includes instructions for causing a training system or, typically a data processing apparatus, to perform a training method.

[0073] It should be understood that the system according to claim 1, the method according to claim 14, and the computer program according to claim 15 have similar and / or identical preferred embodiments, particularly as defined in the dependent claims.

[0074] It should be understood that the preferred embodiments of the present invention may also be any combination of the dependent claims or the above embodiments and the corresponding independent claims.

[0075] These and other aspects of the invention will become apparent and will be illustrated with reference to the embodiments described below. Attached Figure Description

[0076] In the following figures:

[0077] Figure 1 A system for determining the quality of a surface coating applied to a substrate is illustrated schematically and exemplary.

[0078] Figure 2 The first example of height data is shown.

[0079] Figure 3A A monochrome image of the coated surface corresponding to the second example of height data is shown.

[0080] Figure 3B A digital image of a coated surface, based on another example, is shown.

[0081] Figure 4A The image shows pixel values ​​of a monochrome image along a line on a coated surface corresponding to another example of height data.

[0082] Figure 4B It shows the process along the cross Figure 4A The pixel values ​​of the rendered image of the lines on the coated surface.

[0083] Figure 4C The image shows lines across the surface of the coating.

[0084] Figure 5A It shows according to Figure 3A A rendered image of an example coated surface.

[0085] Figure 5B A rendered image of a coated surface based on another example is shown.

[0086] Figure 6A The diagram schematically illustrates the corresponding definitions of the neighborhood and local binary patterns of pixels in a digital image according to the first example.

[0087] Figure 6B The diagram schematically illustrates the corresponding definitions of the neighborhood and local binary patterns of pixels in a digital image according to the second example.

[0088] Figure 7 The diagram schematically illustrates the neighborhood of a pixel in a digital image according to the third example and the values ​​of the corresponding binary vectors generated by the local binary pattern.

[0089] Figure 8A and Figure 8B The diagram illustrates another example of the neighborhood of a pixel in a digital image and the values ​​of the corresponding binary vectors generated by the respective local binary patterns.

[0090] Figure 9A An exemplary rendered image of a coating applied to a substrate is shown, including bounding boxes around defects detected in the coating.

[0091] Figure 9B Another example of a coating applied to a substrate surface is shown, including a bounding box around defects detected in the coating.

[0092] Figure 10 Several examples of corresponding rendered images of height data and coating defects are shown.

[0093] Figure 11 This schematically and exemplary illustrates the convolution of an image using a kernel.

[0094] Figure 12 The structure of a convolutional neural network for classifying detected defects is illustrated schematically and exemplary.

[0095] Figure 13 A method for determining the quality of a surface coating applied to a substrate is illustrated schematically and exemplary. Detailed Implementation

[0096] Figure 1 A system 100 for determining the quality of a surface of a coating applied to a substrate is illustrated schematically and exemplary. System 100 includes a height data providing unit 101 configured to provide height data. The height data indicates the height of the coated surface relative to the substrate. Furthermore, system 100 includes a defect detection unit 102 configured to detect defects in the surface of the coated layer based on the height data provided by the height data providing unit 101. To detect defects, defect detection unit 102 determines height difference data for any candidate defect location on the surface of the coated layer based on the height data, wherein the height difference data for the candidate defect location indicates the difference between the height at the candidate defect location and the height at a plurality of reference locations, wherein these reference locations are determined by defect detection unit 102 based on the candidate defect locations according to predefined relationships. Defect detection unit 102 then compares the height difference data with predetermined reference height difference data indicating defects. Preferably, system 100 further includes a quality determination unit 103 configured to determine the quality of the surface based on the defect detection performed by defect detection unit 102.

[0097] Figure 2 On its left is a black-and-white digital image representing an area, including defects, on the coated surface. Defects are represented by white portions of the digital image, while their neighborhoods are shown in white. Figure 2 It is shown in black. Figure 2 The image shown on the left is divided into pixels, whose coordinates correspond to the coordinates of the surface of the coating, where the pixel value associated with each pixel indicates the height of the surface at the corresponding location. The black-and-white nature of the image is used to illustrate the relationship between height and pixel values ​​particularly clearly, albeit in a rather illustrative way. Typically, a monochrome image representing height data would include more grayscale. However, a black-and-white image can be generated from any monochrome image by thresholding the grayscale levels (i.e., height).

[0098] and Figure 2 The corresponding pixel values ​​of the black and white image shown on the left are... Figure 2 The array on the right is shown in this form. Since the image is black and white, its pixels take only one of two values, in this case 0 or 1, where "0" indicates black and "1" indicates white. Figure 2 In the example, pixel value 1 (i.e., white pixel) indicates a greater height of the coating surface at the location corresponding to the respective pixel compared to pixel value 0 (i.e., black pixel). Therefore, in Figure 2 The defect illustrated schematically and exemplary is a region on the surface of the coating that protrudes relative to its neighborhood.

[0099] Figure 3A The height data is displayed as a grayscale monochrome digital image. Therefore, Figure 3A The pixel values ​​of the image are not only taken from... Figure 2 It can be one of two values, and can take any grayscale value. (Summary) Figure 2 In this scheme, lighter gray values ​​indicate a greater surface height than darker gray values. Figure 3A The area of ​​the coating surface covered by the image is larger than that of the image. Figure 2 The image covers the coated surface area and more than one defect area. However, due to the relatively uniform grayscale values ​​of the image, in Figure 3A Defects are not easily identified in images.

[0100] like Figure 3AThe height data, exemplarily shown in the monochrome digital image, can be determined based on a three-dimensional scan of the surface of a coating applied to a substrate (i.e., the coated substrate). This three-dimensional scan of the coating surface can be performed using known techniques. For example, a line confocal imaging sensor, such as the one used by LMI Technologies in FocalSpec, can be used. The result of the three-dimensional scan typically consists of a three-dimensional dataset where two dimensions indicate the location of the surface coating, and a third dimension indicates the height of the coating surface at the corresponding location. Any coordinate in the third dimension can be used to indicate height. Thus, height can be measured, for example, relative to the substrate on which the coating is applied. The height of the coating surface relative to the substrate can also be referred to as the depth of the coating. Alternatively, the height of the coating surface can also be measured relative to an imaginary plane above the coating surface, in which case a smaller, rather than a larger, height will indicate a protrusion in the coating.

[0101] Three-dimensional scan data can be projected into two dimensions, for example, onto a plane parallel to the substrate on which the coating is applied, to obtain two-dimensional image data. The plane onto which the scan data is projected can also be called the projection plane or image plane, and if the substrate is substantially planar, it can correspond to, for example, the surface of the substrate. Projection can be performed upwards from the aforementioned third dimension of the three-dimensional scan data, in which case the first two dimensions indicating the position of the coating surface will be translated to the position in the projected image, and the image values ​​(i.e., pixel values) can correspond to the height indicated by the three-dimensional scan data.

[0102] The resolution of a projected image does not necessarily correspond to the resolution of the 3D scan data in the first two dimensions. Instead, for example, scan data with relatively high spatial resolution can be projected onto a lower-resolution 2D image, or scan data with relatively low spatial resolution can be projected onto a higher-resolution 2D image. In the former case, the pixel values ​​in the projected image can be determined based on more than one scan point using, for example, an averaging procedure, and in the latter case, the pixel values ​​can be determined, for example, based on interpolation of the 3D scan data. These and other processing steps of the scan data can be performed using tools such as MountainsMap software from DigitalSurf.

[0103] Figure 3A A relatively uniform coating surface area is shown, where defects are almost invisible, while Figure 3B A monochrome digital image of a coated surface area is shown, where the coating surface is somewhat uneven and defects are more easily visible. Furthermore, in... Figure 3B The defect type visible in the image is a crack, and in some ways different from... Figure 3A The particle-type defect type.

[0104] Digital images representing the height data of the coated surface (such as...) Figure 2 , Figure 3A and Figure 3B The digital image shown does not convey the visual impression that the coated surface would have on the observer in reality. To visualize a more realistic impression of the coated surface, lighting must be considered. For this purpose, system 100 may include a rendering image providing unit configured to provide rendered image data corresponding to a view on the coated surface based on given height data and / or based on a three-dimensional scan of the coated surface. The rendered image data may be determined by the rendering image providing unit based on the three-dimensional scan data, and the height data may also be determined based on the three-dimensional scan data.

[0105] For example, in Figures 4A to 4C The diagram illustrates the relationship between height data (e.g., represented as grayscale values ​​of a grayscale image) and rendered image data. Figure 4A This illustrates a protrusion (i.e., height) profile determined based on pixel values ​​along a line passing through a grayscale image representing the height data of a coating surface applied to a substrate. The image itself (i.e., as a whole) is not shown. [The image can be viewed...] Figure 4A As can be seen, the height of the exemplary coating surface is relatively uniform along the selected line, except for region 41, where the height data indicates relatively large protrusions, which suggests the presence of defects in the coating. Figure 4B It shows the relationship with Figure 4A The rendered contour corresponding to the raised contour shown is based on the rendered image and along lines respectively corresponding to the grayscale image and passing through it (i.e., with...). Figure 4A The rendering contour is determined by lines corresponding to the relevant images and lines. Furthermore, the rendering contour is largely relatively uniform, i.e., except for the area corresponding to region 41. Additionally, similar to the raised contour, the rendering contour corresponding to the grayscale values ​​along the corresponding lines includes bright regions 41a and dark regions 41b, indicated by relatively high and relatively low image values, respectively, in the region corresponding to the raised region 41. Dividing the raised region 41 into bright regions 41a and dark regions 41b is a lighting effect produced by the rendering. Figure 4C A large portion of the complete rendered image is shown, indicating the line along which the values ​​of the rendered contours were obtained.

[0106] Figure 5A It shows the representation Figure 3A The rendered image corresponds to the height data of the digital image shown, and Figure 5B An exemplary additional rendered image of the coated surface on a substrate is shown.

[0107] It should be emphasized that although defect detection can theoretically be performed on rendered image data, the defect detection unit 102 is preferably configured to detect height data (such as...). Figure 2 , Figure 3A and Figure 3B Defects in the grayscale image data of the image shown. Furthermore, it should be emphasized that the defect detection unit 102's detection of defects does not include the classification of detected defects. In contrast, as further outlined below, classification can be performed as a subsequent step after defect detection. Furthermore, classification can be performed based on any of a) height data, b) feature map data obtained from the height data using local binary patterning, and / or c) rendered image data.

[0108] The defect detection unit 102 is preferably configured to process all locations on the coating surface (i.e., all locations provided to the defect detection unit 102) as candidate defect locations. Therefore, for example, all pixels in an image corresponding to the height data determined by the height data providing unit 101 can be considered candidate defect locations. To determine whether a candidate defect location is an actual defect location (i.e., corresponding to the actual portion of a defect), height difference data is determined by the defect detection unit 102 based on the height data. Specifically, the defect detection unit 102 can be configured to determine the height difference data based on the grayscale values ​​of pixels in a two-dimensional image that represent the height of the coating surface in grayscale form and based on the grayscale values ​​of reference pixels (i.e., based on the height indicated for the pixel and the height indicated for the reference pixel). The height difference data associated with a given candidate defect location corresponding to a given image pixel indicates the difference between the height at the candidate defect location and the height at the reference location. Therefore, several height differences can be associated with a given candidate defect location as represented by the corresponding pixel.

[0109] A reference location (also represented by pixels) can be determined by the defect detection unit 102 for a given candidate defect location based on any predefined relation in a predefined relation set. For example, a reference location can be selected from the neighborhood of the candidate defect location. Preferably, the predefined relation considered by the defect detection unit 102 for selecting the reference location is the same for all candidate defect locations, and therefore the same for all locations on the coating surface (i.e., possibly for all pixels in an image representing the height of the coating surface). The defect detection unit 102 is able to determine whether a candidate defect location is a defect by comparing the height difference data determined for a given candidate defect location with predetermined reference height difference data indicating the presence of a defect. Comparing the height difference data determined for a given candidate defect location with the predetermined reference height difference data may involve comparing the height difference data determined for a given candidate defect location with each predetermined reference height difference data element in a set of predetermined reference height difference data elements indicating different types of defects. For the conclusion that a candidate defect location is a defect, it may be sufficient for the height difference data determined for the candidate defect location to have a sufficiently high similarity to any one of the predefined reference height difference data elements in the set of predefined reference height difference data elements. The reference height difference data element can also be called the defect indication height difference data element, or simply the defect indication data element.

[0110] To compare height difference data with predetermined reference height difference data, any suitable comparison metric can be used. The metric can be chosen depending on how the height difference data is represented. For example, if the height difference data is represented as a vector associated with the corresponding candidate defect location, where the entries of the vector correspond to the height difference between the candidate defect location and the corresponding reference location, or a quantity derived therefrom, then the reference height difference data would preferably also be represented as a vector. The comparison could correspond to forming a difference vector between the vector corresponding to the height difference data determined for the corresponding candidate defect location and the vector corresponding to the reference height difference data, and calculating the vector norm of the respective difference vector. However, as further indicated below, other comparison metrics may be preferred, and these comparison metrics are not necessarily based on differences.

[0111] Preferably, the defect detection unit 102 is configured to detect each defect individually. That is, the defect detection unit 102 is preferably able to distinguish the detected defects. For example, for a set of locations on the surface of the coating that have been identified as actual defect locations (i.e., defects belonging to the coating), the defect detection unit 102 can be configured to determine whether these locations belong to the same defect, and if not, to associate these locations with the corresponding different defects. In other words, the set of detected defect locations can be segmented into subsets corresponding to individual defects. Individual defects can be located in an image representing height data of the coating surface. Optionally, the defect detection unit 102 is configured to determine bounding boxes around one or more of the detected defects and local defects. These bounding boxes (i.e., the corresponding image segments, which may also be referred to as defect image segments) can be used as a basis for classifying defects within them. However, the classification of detected defects may not be necessary for determining the quality of the surface (i.e., for assessing the quality of the applied coating). For example, reliable quality metrics can already be defined based on the number (especially density) of detected defects and / or the surface area of ​​the coating covered by defects.

[0112] It has been found that particularly accurate and efficient defect detection can be achieved by using local binary patterns. Local binary patterns are considered a particularly useful type of height difference data. Therefore, the defect detection unit 102 is preferentially configured to perform local binary patterning of height data to determine height difference data. Local binary patterning of height data can be understood as a specific type of transformation from height data to a feature map, where the feature map associates a feature with a position on the surface of the coating, and the feature associated with any given position indicates the difference between the corresponding height indicated by the height data at that position and the height indicated by the height data at a reference position.

[0113] like Figure 6A and Figure 6B As illustrated and exemplarily shown, local binary patterns on a digital grayscale image can be... This is represented by a mapping that maps the pixels of an image to corresponding binary vectors. Each pixel in the image can correspond to a position on the surface of a coating applied to a substrate. The entries in these binary vectors are either 1 or 0. Therefore, the mapping... It can be written as

[0114] (3a)

[0115] (3b)

[0116] Image space Including by The indexed pixel, and Corresponding to vector The size. It should be understood that... And image space It can also be represented by two natural numbers instead of just one (i.e., indexing or coordination) because it is spatially two-dimensional. Quantity And therefore vector Size indicator for each pixel The number of reference pixels considered. Preferably, its components are selected as described herein. binary vector to satisfy

[0117] (4)

[0118] in, Indicator Pixel of Reference location, and Indicate pixels respectively and The height (i.e., grayscale value), and This corresponds to the predefined defect detection margin.

[0119] parameter It can depend on and This dependency can be selected differently for each type of defect to be detected. For a given candidate defect location... Thus, several binary vectors can be determined. Each binary vector in the binary vector corresponds to a different type of defect to be detected, and therefore different types are used. The value is used to determine this. As further indicated above, The symbols, the inequality symbols in the above equation (4), and / or via functions Recognition of pixels with high grayscale values ​​can also be inverted depending on the type of defect to be detected. Specifically, one can choose... Used to detect depression-type defects (such as valley-type defects or pit-type defects), and can be selected Used to detect protrusion-type defects (such as ridge-type defects or particle-type defects). According to another example, equation (4) (where and The value increases with the increase of the coating surface height (as represented by increasingly lighter pixel gray values) and can be used to detect valley defects or pit defects, while equation (4) (where and Inversion (or negation) can be used to detect ridge-like or granular defects. Using inversion... The increasing surface height of the coating is represented by increasingly darker pixel grayscale values. Therefore, in other words, the grayscale image used for defect detection can be inverted depending on whether the defect to be detected is a depression or a protrusion.

[0120] Reference position It is possible Figure 6A and Figure 6B Selected as all located in The positions of the binary vectors are determined on a common circle, specifically equidistant along the circumference of the circle. Figure 6A and Figure 6B The diagram shows two circles with different radii, with the reference pixel being... It means, rather than by ( (This is an expression.) Figure 6A and Figure 6B It will be understood that the reference position itself can be defined relative to a continuous geometry, and the reference pixel corresponding to the reference position can be determined by determining which pixel a given reference position corresponds to (i.e., which pixel region the reference position falls into). Therefore, it depends, for example, on the radius of the circle in which the reference position is located and the reference position itself. The number of pixels, and the set of pixels corresponding to the geometry for which the reference location is chosen, may differ more or less from the reference geometry. Figure 6A In, for example, the radius of the reference circle has been chosen as... ,in, It is the pixel width and the number of reference positions. An integer equals 8, which produces a square pixel set with sides of three pixels. This square pixel set can be considered as... Figure 6A Zhongyou The reference pixel region for the candidate defective pixels is shown. Meanwhile, in Figure 6B In the diagram, the circle has been selected as having a radius of two pixels wide, i.e. When the number of reference positions When the value is still 8, this results in a set of pixels having a rhombus shape with equal side lengths, which can also be viewed as a rotated square with side lengths greater than 8. Figure 6A The side length of the square reference pixel region. From the pixel set (i.e., the reference pixel region) corresponding to the reference geometry, the grayscale value indicating the height of the corresponding coating surface location is considered as reference height data for the height of the candidate defect location. .

[0121] Figure 7 The illustration schematically and exemplary shows a pixel region of a grayscale image comprising pixels with different grayscale values. For example... Figure 6AThe schematic diagram illustrates how local binary patterning has been applied to pixel regions, transforming binary vectors... Associated with the center pixel. In this case, the detection margin parameter... It can be considered as zero, that is . With the vectors mentioned above Consistent with the formula, for reference pixels with higher gray values ​​(in this case, higher gray values ​​correspond to lighter gray values ​​compared to the center pixel), their entries are equal to 1, and in other cases equal to 0. Figure 7 The pixel areas shown do not indicate the presence of defects. In contrast, Figure 8A and Figure 8B The pixel regions shown schematically and exemplarily in the diagram indicate the presence of valley-type defects.

[0122] according to Figure 8A The local binary patterning of the application corresponds to the following: Figure 7 The application uses local binary patterning. However, the pixel values ​​in the shown pixel regions are different. Figure 8A The two different pixel configurations shown share the characteristic that the middle row pixels are darker than the top and bottom row pixels, indicating a valley-type defect in both cases. However, in Figure 8A In the pixel configuration shown on the left, the center pixel value is higher than the pixel values ​​on its left and right sides, while... Figure 8A In the pixel configuration shown on the right, the center pixel value is lower than the pixel values ​​to its left and right. This difference between the two pixel configurations results in different binary vectors associated with the corresponding center pixel, even though the two pixel configurations are similar in that they indicate valley-type defects. In fact, Figure 8A The two pixel configurations shown can be pixel configurations arranged at two different, possibly even closer, axial locations along the same valley-type defect. The difference between the two pixel configurations corresponds to random fluctuations in grayscale values ​​along the valley, however, which is negligible compared to the corresponding difference in grayscale values ​​(i.e., white values) that are higher and in this case even the highest, relative to the outside of the valley.

[0123] for Figure 8A The local binary pattern shown in the figure has the following parameters. It was chosen to be equal to zero, while for Figure 8B The local binary pattern shown in the figure has the following parameters. It was selected as being greater than zero, that is .However, Figure 8B The pixel configuration shown is itself with Figure 8A The pixel configuration shown is the same. Non-zero parameters. The effect is that, despite differences in pixel configuration between the left and right sides, the binary vector associated with the center pixel is the same. Therefore, by Figure 8B The displayed local binary pattern can be considered better than based on Figure 8A The local binary patterning of the application is more ideal because it is somewhat insensitive to uncorrelated changes in pixel values ​​and therefore reflects defect features more accurately.

[0124] To determine pixels To determine whether a candidate defect location is an actual defect location, a local binary model will be used to compare the pixel values. Associated binary vectors Compare with one or more reference binary vectors that indicate the presence of a defect of the corresponding type. For example, for a... Figures 6A to 8B The local binary pattern of the type shown, and taking into account The inequality signs in equation (4) and / or the possible inversion of the underlying image, the reference binary vector associated with the ridge-type defect can be Furthermore, the reference binary vector associated with the valley-type defect can be Ridge-type defects and valley-type defects can both be caused by cracks in the coating and / or substrate, and therefore can be classified accordingly. For particle-type defects, the reference binary vector can be, for example... Furthermore, the reference binary vector associated with the pit-type defect can be .

[0125] Depending on the size of the corresponding defect and the selected parameters of the local binary mode, specifically Different reference binary vectors can be chosen. For example, ridge-type defects can also be represented using... or This indicates that valley-type defects can be represented, and valley-type defects can be used... or express.

[0126] Typically, a reference binary vector can be predefined heuristically for each type of defect to be detected. In particular, for any given defect type to be detected, a reference binary vector can be predefined by considering a) the parameters of the corresponding local binary pattern used for detection and b) the typical binary vector expected to be generated by the corresponding local binary pattern of the given defect type.

[0127] To determine whether a given candidate defect location belongs to a given type of defect, the defect detection unit 102 can be configured to compare the two vectors by determining a Hamming distance between a binary vector determined for the given candidate defect location and a reference binary vector defined for a defect of the given type. Instead of the Hamming distance, other measured distances can be used, wherein the choice of distance metric can be based on the type of defect to be detected.

[0128] It can be based on multiple segments in a binary vector (especially equal to) And therefore, in the above case, multiple segments of consecutive entries equal to 1 are assigned to a given binary vector to define an exemplary alternative distance metric. The distance between a) the binary vector associated with the candidate defect location and b) a reference vector defined for the type of defect to be detected can then be defined as the difference between the number of corresponding segments in the two vectors. For example, to detect valley-type defects, where This allows determining the number of segments with consecutive "1"s in the binary vector associated with the candidate defect location, and then comparing them with the aforementioned reference binary vector. The number of segments with consecutive "1"s (which equals 2) is compared.

[0129] Figure 9A and Figure 9B An illustrative rendered image of a coated surface is shown, on which defects have been detected using the height data and local binary mode as described above, wherein the bounding boxes around the detected defects have been transferred to the rendered image.

[0130] Figure 10 On its left are height data image segments corresponding to different defect categories on the coating surface, and on its right are corresponding rendered image segments. Looking from top to bottom of the five image pairs, the first two image pairs show particle-type defects, the third image pair shows crack-type defects, and the fourth and fifth image pairs show pit-type defects. Other defect types that can be detected by the defect detection unit 102 include, for example, spalling and bubbles.

[0131] After defects have been detected, these defects can be classified. Therefore, system 100 may additionally include a classification unit configured to classify detected defects based on heights indicated by height data of the detected defects. For example, the classification unit may receive as input image segments of grayscale images showing the detected defects, which may have been cropped from the complete image along corresponding bounding boxes. In other words, the classification unit may be configured to classify detected defects in grayscale image segments showing the detected defects based on pixel values ​​of the corresponding image segments. Additionally or alternatively, the classification unit may be configured to classify detected defects based on rendered image data corresponding to the respective detected defects.

[0132] In particular, the local binary pattern that may have been determined for the image segment during defect detection as described above can also be used to classify the detected defects. Specifically, the classification unit may, for example, receive all pixels in the image segment including the height of the detected defect. binary vector As input for classifying defects.

[0133] Optionally, the classification unit is configured to classify detected defects using a trained machine learning model. The machine learning model can be trained on training data comprising pairs of training input data and training output data, wherein the training input data is prepared as if it were the input data to be received by the trained machine learning model, i.e., prepared after deployment. For example, if the trained machine learning model is to receive height image fragments of a single defect as its input, it preferably also utilizes height image fragments of the single defect for training. Furthermore, the machine learning model can be trained using training data corresponding to as many defect types as possible. In this way, the trained machine learning model can be able to accurately classify a wide variety of defects. The training output data can correspond to assigning defect categories to the corresponding training input data. Therefore, the training output data can be provided in the form of ground truth values.

[0134] While trained machine learning models have been observed to allow for reliable classification results, unsupervised classification algorithms or other artificial intelligence can also be used to classify detected defects.

[0135] In this example, the machine learning model used may include a convolutional neural network configured to receive a corresponding image segment as its input. Therefore, in this case, specific features derived from the image segment are not received as input by the network, which would be possible even in other embodiments. These features could instead be extracted by the convolutional neural network itself.

[0136] Convolutional neural networks (CNNs) can specifically include multiple convolutional layers to provide corresponding feature maps. In other words, features can be determined by the CNN, rather than predetermined. For example, the number of convolutional layers in a CNN can be five. However, other numbers of convolutional layers can also be used in principle. Figure 11 The diagram illustrates, and exemplarily demonstrates, the processing performed by a convolutional layer of a convolutional neural network on an exemplary image I consisting of pixel values ​​0 and 1 using a kernel K specific to the respective convolutional layer.

[0137] Furthermore, convolutional neural networks can include one or more pooling layers that pool corresponding feature maps in the feature maps into corresponding pooled feature maps. For example, max pooling can be applied. Additionally, newer convolutional networks can perform spatial pyramid pooling. Specifically, after the last convolutional layer, a three-sub-pooling structure can be applied. , and Spatial pyramid pooling.

[0138] exist Figure 12 The diagram schematically and exemplaryly illustrates data processing implemented according to a preferred convolutional neural network architecture. As indicated therein, input data I (which may be a grayscale image of the coating surface indicating the grayscale level of the coating surface) is convolved in a first convolutional layer by one or more kernels of substantially arbitrary size into an equal number of feature maps (in the non-limiting case shown, two kernels are used, resulting in two feature maps). The feature map, whose size may be equal to or not equal to that of the input image I, is then pooled by a first pooling layer, resulting in a smaller first pooled feature map. This process is further repeated four times, however, in which spatial pyramid pooling is applied in the last (i.e., fifth) pooling layer. Spatial pyramid pooling increases the number of feature maps compared to the previous pooling. As shown, it is assumed that the fifth convolutional layer produces... Feature maps are generated by spatial pyramid pooling. The feature map is divided into three blocks, where, in this case, the sizes of the feature maps in the three different blocks are respectively... , and That is, it is independent of the size of the previous feature map or the initial input I. The feature map generated by spatial pyramid pooling is then flattened to, for example, a length of ( + + ) The vector, where the flattened output (specifically the vector) is provided as input to a single fully connected layer that maps the input to the corresponding defect category. , Wait a minute. It should be understood that... Figure 12 This only demonstrates the basic structure of a convolutional neural network, where training the network determines how accurately each layer maps its corresponding input to its corresponding output. Figure 12 As shown by the arrow in the image.

[0139] Figure 13 A method 200 for determining the quality of the surface of a coating applied to a substrate is illustrated schematically and exemplary. Method 200 includes providing height data in step 201, wherein the height data indicates the height of the coated surface relative to the substrate. In a second step 202, defects in the surface of the coating are detected by: a) determining height difference data based on the height data for any candidate defect location on the surface of the coating, wherein the height difference data for the candidate defect location indicates the difference between the height at that candidate defect location and the height at a plurality of reference locations determined based on the candidate defect location according to a predefined relationship; and b) comparing the height difference data with predetermined reference height difference data indicating a defect. In a subsequent step 203 of method 200, the quality of the surface of the coating applied to the substrate can be determined based on the detected defects.

[0140] In the case of determining the surface quality of the coating applied to the substrate, as mentioned above, the determined quality can also be used as an indicator of the general coating quality (especially not only the surface quality when applied to the substrate, but also the volumetric quality and / or its quality regardless of whether it is applied to the substrate). Furthermore, the determined quality can also be used as an indicator of the quality of the corresponding coating application process, i.e., an indicator of the extent to which the coating has been applied to the corresponding substrate. For example, based on the quality, the coating and / or application procedures can be adjusted.

[0141] Furthermore, although the above embodiments are described primarily with reference to coating evaluation, the same or similar embodiments can be used to evaluate any surface. For example, a surface can typically be evaluated with regard to the presence and type of specific irregularities. Depending on the type of surface, appropriate additional data acquisition devices can then be used to acquire height data. The surface and therefore the height data can be microscopic or macroscopic.

[0142] By studying the accompanying drawings, this disclosure, and the appended claims, those skilled in the art can understand and implement other variations of the disclosed embodiments when practicing the claimed invention.

[0143] In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a / an” does not exclude multiple / types.

[0144] A single unit or device can perform the functions of several items listed in the claims. The fact that certain measures are listed in different dependent claims does not indicate that combinations of these measures cannot be used advantageously.

[0145] The following procedures can be performed by any other number of units or devices, such as providing height data; detecting defects; determining quality; classifying detected defects; providing images; and training machine learning models using machine learning models and / or by providing machine learning models and training data, etc. These processes can be implemented as program code devices and / or dedicated hardware for computer programs.

[0146] Computer programs can be stored / distributed on suitable media, such as optical or solid-state storage media provided with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems.

[0147] Any reference numerals in the claims should not be construed as limiting the scope.

[0148] A system for determining the quality of a coated surface is proposed. The system includes: a height data providing unit configured to provide height data indicating the height of the coated surface relative to a substrate; and a defect detection unit configured to detect defects in the surface by: a) determining height difference data for any candidate defect location, the height difference data indicating the difference between the height at the candidate defect location and the heights at a plurality of reference locations; and b) comparing the height difference data with predetermined reference height difference data indicating the defect. The system further includes a quality determination unit configured to determine the quality of the surface based on the detected defects. This allows for improved coating quality assessment based on efficient and accurate defect detection.

Claims

1. A system (100) for determining the quality of the surface of a coating applied to a substrate, wherein, The system includes: A height data providing unit (101) is configured to provide height data, wherein the height data indicates the height of the surface of the coating relative to the substrate. A defect detection unit (102) is configured to detect defects in the surface of the coating by: a) determining height difference data based on the height data for any candidate defect location on the surface of the coating, wherein the height difference data for the candidate defect location indicates the difference between the height at the candidate defect location and the height at a plurality of reference locations determined based on the candidate defect location according to a predefined relationship; and b) comparing the height difference data with predetermined reference height difference data indicating a defect, and A quality determination unit (103) is configured to determine the quality of the surface based on detected defects.

2. The system as claimed in claim 1, wherein, The defect detection unit (102) is configured to detect each defect individually.

3. The system as described in any one of the preceding claims, wherein, The height difference data corresponds to the local binary patterning of the height data.

4. The system as described in any of the preceding claims, further comprising a classification unit configured to classify the detected defects into one of a plurality of defect categories, wherein, The quality determination unit (103) is configured to determine the quality of the surface based on the detected defects that have been classified.

5. The system as described in claim 4, wherein, The defect category indicates any of the following: cracks, pits, particles, spalling, bubbles.

6. The system as described in any one of claims 4 and 5, wherein, The classification unit is configured to classify the detected defects based on the height indicated by the height data of the detected defects.

7. The system of any one of claims 4 to 6, further comprising a rendering image providing unit configured to provide a rendering image corresponding to a view on the surface of the coating, wherein, The classification unit is configured to classify the detected defects based on rendered image data corresponding to the detected defects.

8. The system as claimed in any one of claims 4 to 7, wherein, The classification unit is configured to classify the detected defects using a trained machine learning model.

9. The system of claim 8, wherein, The machine learning model includes a convolutional neural network.

10. The system of claim 9, wherein, The convolutional neural network includes multiple convolutional layers for providing corresponding feature maps.

11. The system of claim 10, wherein, The convolutional neural network includes at least one pooling layer, which is used to pool the corresponding feature maps in the feature maps into corresponding pooled feature maps.

12. The system of claim 11, wherein, The convolutional neural network uses spatial pyramid pooling.

13. The system as claimed in any one of claims 8 to 12, wherein, The machine learning model includes a transformer architecture.

14. A method (200) for determining the surface quality of a coating applied to a substrate, wherein, The method includes: Provide (201) height data, wherein the height data indicates the height of the surface of the coating relative to the substrate; Defects in the surface of the coating (202) are detected by: a) determining height difference data based on the height data for any candidate defect location on the surface of the coating, wherein the height difference data for the candidate defect location indicates the difference between the height at the candidate defect location and the height at a plurality of reference locations determined based on the candidate defect location according to a predefined relationship; and b) comparing the height difference data with predetermined reference height difference data indicating a defect, and The quality of the surface is determined (203) based on the detected defects.

15. A computer program for determining the quality of the surface of a coating applied to a substrate, wherein, The computer program includes instructions that cause the system of claim 1 to perform the method of claim 14.

Images