Inspection device for a surface coating

Abstract

The invention relates to a method for inspecting a coating on a surface (e.g. paintwork) of an object and a corresponding device which is used to classify defects on the surface quickly and easily. A plurality of first images of the object surface are generated in relation to a first optical inspection channel (K1), a plurality of second images of the object surface are generated in relation to a second optical inspection channel (K3) and optionally a plurality of further images of the object surface are generated in relation to at least one further optical inspection channel (K3 to K5), wherein the plurality of images of each inspection channel are automatically analysed for the presence of a defect, wherein, if a defect is detected, for each inspection channel for which at least one image section (K1.1 to K1.5, K2.1 to K2.2, K3.1 to K3.4, K5.1 to K5.3) of the defect can be assigned, and for each defect detected, by means of an NN algorithm, an inspection-channel-specific defect type (K1.8, K2.8, K3.8, K5.8) and an inspection-channel-specific evaluation variable (K1.9, K2.9, K3.9, K5.9) is determined and a defect type is then automatically selected from the inspection channel-specific defect types, which is then output for the detected defect at an interface of a data processing device.

Description

[0001] Inspection device for a surface coating

[0002] DESCRIPTION

[0003] The present invention relates to a device for inspecting a coating of an object surface, for example a paint coating, and a corresponding method.

[0004] Automated production, for example, in the automotive industry, places high demands on quality assurance. Especially when inspecting surface coatings on workpieces, such as paintwork, it is necessary to quickly and easily detect, classify, and locate potential defects. Various such devices and methods are already known and in practical use.

[0005] From the document WO 2017 / 081029 A2, a method for the automated detection and robot-assisted processing of defects in a workpiece surface is known, in which an optical inspection of the surface is carried out to detect defects, then a three-dimensional measurement of the workpiece surface is carried out using optical sensors in the area of ​​detected defects, subsequently a topography of the workpiece surface in the area of ​​at least one defect is determined, then a parameter set which characterises the at least one defect is determined, the at least one defect is categorised based on the determined parameter set, wherein the defect is assigned to a defect category, in the next step a processing process is selected depending on the defect category of the at least one defect, including a processing path,along which the defect is to be processed, and finally, a computer-aided robot program for robot-assisted processing of at least one defect is created. In the known method, the processing of the defects is adapted to the type of defect.

[0006] Document WO 2021 / 151412 A describes a device for automatically detecting damage to vehicles, comprising a light tunnel. The light tunnel has a plurality of strip-shaped light sources, at least one camera, at least one first master camera, at least one second master camera, and at least one local computer with one or more powerful graphics processors. The computer receives the images from the at least one camera, the first master camera, and the second master camera, evaluates them, and calculates an output signal. The output signal generates the class of the defect object and its coordinates in the image. Deep learning-based object detectors are available for detecting damage in the form of defect objects. This known device is comparatively complex in design.

[0007] Document WO 2022162417 A1 describes a method for detecting defects in the paint layers of objects using machine learning / artificial intelligence (AI or AI) methods. A plurality of patterns are projected onto the surface of the object, and images of the surface with the patterns are generated. These images are then fed to a machine learning module to determine whether the object surface has defects. The images with the detected paint defects are then displayed. Only very limited types of paint defects can be detected using such a method.

[0008] To eliminate defects in a surface coating, it is necessary to know the type of defect. Once the defect type and its location on the surface of the object are known, further measures can be taken quickly and easily to correct the defect.

[0009] The above object is achieved by a method having the features of claim 1 and a device having the features of claim 7.

[0010] In particular, the above object is achieved by a method for inspecting a coating of a surface of an object, wherein a plurality of first images of the object surface are generated with respect to a first optical inspection channel, a plurality of second images of the object surface with respect to a second optical inspection channel and optionally a plurality of further images of the object surface with respect to at least one further optical inspection channel, wherein the plurality of images of each inspection channel are automatically analyzed with respect to the presence of a defect, wherein the following steps are carried out for each inspection channel and each detected defect:

[0011] • Assigning those image sections of the multitude of images of the respective inspection channel that contain the location of the detected defect,

[0012] • Determining an inspection channel-specific defect type and an associated inspection channel-specific evaluation value of the determined inspection channel-specific defect type for those inspection channels to which at least one image section of the location of the detected defect could be assigned, wherein the determination is carried out by means of a first NN algorithm trained for the respective inspection channel (iean algorithm which contains an artificial neural network trained for the respective inspection channel) on the basis of the image sections of the respective inspection channel assigned to the respective defect, whereby for each detected defect of the object surface a defect type is then selected by means of soft voting from the inspection channel-specific defect types of the respective inspection channels determined for the respective defect and the respectively associated inspection channel-specific evaluation variables, whereby for each detected defect of the object surface the respectively selected defect type is output at an interface of a data processing device.

[0013] As explained above, at least two different optical inspection channels exist, but in many cases there are considerably more than two inspection channels (e.g. four to fifteen different inspection channels). A plurality of images (recordings) of the object surface are generated for each of these inspection channels. The images for each optical inspection channel are generated by means of an illumination unit and an associated image recording unit and, if applicable, a data processing device. Each inspection channel is characterized by a predetermined inspection-channel-specific illumination by the illumination unit, an inspection-channel-specific recording by the image recording unit and / or by an inspection-channel-specific processing of the recorded data. With the inspection-channel-specific illumination, a plurality of possibly digitized recordings (i.e.Image data) of the light reflected from the coated object surface is generated by the image recording unit, comprising a predetermined area of ​​the object surface to be inspected (for example, a plurality of images from a matrix camera). These images are transmitted to the data processing unit for further analysis, but can already be pre-processed in the image recording unit (e.g., digitized or processed with regard to image parameters such as brightness or contrast, or processed by subtracting a low-pass filtered image from each image). Depending on the respective inspection channel, the image data transmitted by the image recording unit is made directly available for one or more inspection channels without prior further processing by the data processing unit. The images of these inspection channels are therefore generated directly from the transmitted image data (e.g.,Images from a matrix camera) that correspond to image data are generated. For example, the images of the first and second inspection channels may differ in that they were generated using different illumination patterns. Alternatively, the recorded data is processed in the data processing unit such that predetermined areas of the recorded data are used for defect detection, for example the areas along a stripe of a stripe pattern or the areas between two stripes of a stripe pattern. Recorded data processed in this way is assigned to one inspection channel, which focuses on the areas along the stripes, and to another inspection channel, which examines the areas between the stripes for possible defects.In a further alternative, the image data generated by the illumination unit under different lighting conditions is further processed based on a calculation rule (see examples below) and made available as an additional inspection channel. This means that each inspection channel has inspection-channel-specific illumination and / or inspection-channel-specific image acquisition by the image acquisition unit and / or inspection-channel-specific processing of the image data. This means that each inspection channel differs from another inspection channel by at least one of these influencing factors (lighting, acquisition, image data processing).Examples of this are shown below and can be based on the use of different lighting patterns with regard to the influencing factor of lighting, for example, with regard to the influencing factor of recording, for example, on the use of different cameras and / or different camera modes (e.g. image data from different camera sections or camera pixel groups and / or image data from different wavelength ranges and / or image data from different time intervals (sequence) can be read out) and with regard to the influencing factor of image data, on the use of different calculation rules (e.g. to generate a summed image, contrast image, phase image or curvature image, subtraction of a low-pass filtered image). When comparing all the influencing factors, the influencing factors of lighting and processing of the image data are particularly important because they are the easiest to vary.

[0014] The illumination unit is configured to illuminate the object in an area to be inspected with a predetermined, inspection-channel-specific illumination pattern. The illumination pattern can differ for different inspection channels, or for an inspection channel, recorded data generated with different, inspection-channel-specific illumination patterns can be used. The illumination pattern is generated either by the light-emitting part of the illumination unit itself or by illuminating a grating that generates the illumination pattern. The illumination occurs in the visible wavelength range. The illumination unit can be formed as a single piece or from several spaced-apart elements.

[0015] The image recording unit, e.g. one or more cameras (e.g. matrix camera(s)), records a plurality of images of the illuminated area of ​​the object surface, i.e. it generates an image recording sequence of the images of the illumination pattern which the image recording unit views in reflection (incident light arrangement) in relation to the object. The image recording unit and the object can move relative to one another so that the entire area of ​​the object surface to be inspected can be recorded by the image recording unit. Accordingly, the illumination unit can also move relative to the object, e.g. together with the image recording unit. The digitized recording data generated by the image recording unit are sent by the image recording unit to the data processing device (computer, processor or the like).) and made available for this purpose at a corresponding interface of the image recording unit connected to the data processing device.

[0016] The image recording unit can be composed of one or more matrix cameras. The matrix camera is also known as an area scan camera and records a larger surface area of ​​the object. It can be designed as a CCD camera, for example. The matrix camera records the light intensity of a large number of pixels in the illuminated area, which are arranged in rows and columns, i.e. in a matrix. For this purpose, the matrix camera has a light-sensitive element (e.g. a CCD sensor) for each pixel. The size of the area recorded by each light-sensitive element (pixel) determines the resolution of the matrix camera. For example, several matrix cameras can be arranged next to one another in a direction transverse to a relative movement of the image recording unit and the object in order to inspect a particularly wide object.

[0017] As explained above, the object and the image recording unit (and optionally also the lighting unit) can move relative to one another at a predetermined speed in a feed direction. The transverse direction runs transversely, preferably perpendicular to the feed direction. For example, the object is guided past the image recording unit (and optionally also the lighting unit) on a corresponding belt or as a web at a predetermined speed, and the image recording unit (and optionally also the lighting unit) does not move. Alternatively, only the image recording unit (and optionally also the lighting unit) moves, or both the image recording unit (and optionally also the lighting unit) and the object move simultaneously.It is advantageous for the evaluation of image data if the relative movement of the object and the image acquisition unit (and, if applicable, the illumination unit) represents a uniform movement, i.e., a movement at a constant speed. This simplifies the evaluation of the image acquisition data for the inspection of the object.

[0018] The lighting unit or each element of the lighting unit can have a plurality of individually switchable lighting elements, for example light-emitting diodes (LEDs) or groups of jointly switchable light-emitting diodes (LEDs). In this case, the lighting pattern is generated by the entirety of the lighting elements themselves. An individually switchable lighting element can therefore be constructed from a single lighting fixture, for example an LED, or from a plurality of lighting fixtures combined in a group and switchable together, for example a group of several LEDs. In one embodiment, the data processing device can be configured to control the individual lighting elements differently, i.e., to switch them on and off. For example, bright fields / stripes are realized by switched-on LEDs and dark fields / stripes are realized by switched-off LEDs.This allows for flexible implementation of various inspection channel-specific lighting configurations with different patterns and / or with light of different wavelengths.

[0019] Alternatively or additionally, the lighting unit can include a light fixture (e.g., one or more fluorescent tubes) that completely illuminates the area to be illuminated. A grid is arranged between the light fixture and the object, creating the desired lighting pattern. The bright fields / lines are created by corresponding slits in the grid, and the dark fields / lines are created by corresponding bars.

[0020] The illumination unit can provide illumination patterns that implement bright-field illumination and / or dark-field illumination. Alternatively or additionally, the illumination device can emit light of different wavelengths.

[0021] Since dark-field illumination does not occur immediately at the transition from an illuminated to an unilluminated illumination element, it may be advantageous in one embodiment of an illumination pattern to combine a total of several switched-off, i.e., dark, illumination elements next to one another. In a further embodiment, an optical light-shaping unit can be arranged between the illumination unit and the object and / or between the object and the image recording unit. The light-shaping unit can be constructed, in particular, from lenses, diffusers, and / or microlenses.

[0022] The illumination pattern can, for example, represent a stripe pattern of light (luminous) and dark (non-luminous) stripes, which runs in the feed direction explained below or transversely to this feed direction. The stripe pattern can be designed in the same shape for different inspection channels (e.g., same stripe shape, same stripe width, same stripe direction), but offset from one another. The stripe patterns of different inspection channels can also differ in the stripe width or the signal shape of the stripes in the transverse direction to the stripes (e.g., rectangular shape, sinusoidal shape). In addition, an inspection channel can comprise a pattern that represents full illumination (i.e., all illumination elements are illuminated). The inspection method can be a computer-implemented inspection of the object based on the image data, i.e.,as a method carried out using a data processing device (computer). The method can also include controlling the lighting unit and the image recording unit in such a way that images for the first inspection channel, the second inspection channel and, if applicable, the further inspection channel are generated for all sections of the area to be inspected, for example one after the other and / or simultaneously. For example, on the basis of the image recordings, the defects are determined using defect detection methods as local deviations from an image recording signal that would arise from a defect-free object. In the context of the present invention, recordings, recording data or images and image data are understood to mean data that assigns an intensity value (e.g. between 0 and 255) to each point of a predetermined two-dimensional matrix (image). This can be a pixel image or a vector graphic.

[0023] The lighting unit and the image acquisition unit are optionally electrically connected to each other via a control unit, so that the illumination of the object and the generation of images of the object's surface can be controlled. The data processing unit can also be connected to the lighting unit and the image acquisition unit and form the control unit. Alternatively, the data processing unit can be arranged remotely from the lighting unit and the image acquisition unit, which includes, in particular, the use of a remote server or cloud computing.

[0024] According to the method according to the invention, after the images have been generated, they are analyzed for the presence of defects on / in the surface coating of the object. The type of defect is not important at this point. It is simply determined that there is a deviation of the surface of the object from the desired quality, which is generally assessed as a defect, and the location of the respective defect is identified. The method according to the invention then carries out a more precise analysis of the detected defect in relation to the defect type, as explained below. For example, the following defect types can be distinguished: inclusions, craters (dents), protrusions (bumps), contamination (dust), pseudo-edges, orange peel, pores, cracking, grinding marks, specks, surface defects, blistering, scratches, wet prints.A pseudo-edge is a type of defect that occurs when the edge of a car body is detected. The presence of defects on / in the surface coating of the object is determined using image processing methods. In one embodiment, the detection of a defect in the images of the respective inspection channel is carried out depending on the respective inspection channel. In one embodiment, a low-pass filtered image and / or a gradient image is used for defect detection. For example, for some of the inspection channels, the respective low-pass filtered image is subtracted from each image and then examined to determine whether the gradients of neighboring pixels present in the image exceed or fall below a predetermined first threshold or whether the absolute values ​​of the pixels exceed or fall below a predetermined second threshold.In this case, the respective threshold value can be individually set for predefined areas of the image (so-called tiles). At points where the first or second threshold value is exceeded or undershot, the presence of an error is assumed. In a further embodiment, the criterion of the magnitude of the deviation can be used as a further criterion. This means, for example, that an error is only recognized as such if the area of ​​the image where the threshold value is exceeded or undershot has a predefined number of pixels (for example, at least 5 pixels). This prevents artifacts from image processing from being recognized as errors. In a further embodiment, before examining whether a gradient or absolute value present in the image exceeds or falls below a first or second threshold value, a mask (overlay) is applied to the image (i.e.This overlay can be calculated based on the image data or specified based on the location of the detected object surface. This is particularly useful when a predefined illumination pattern is used for an inspection channel or when an object edge is located within the image area.

[0025] If the presence of a defect is detected (and also its position / location, e.g. in relation to the object), then for each existing inspection channel (i.e. the first inspection channel, the second inspection channel and each further inspection channel) those image sections are determined which contain the location of the respectively detected defect. In one embodiment, several image sections per inspection channel can be assigned to the location of the detected defect, since for each inspection channel several image sections are generated from each section of the area of ​​the object surface to be inspected, for example in such a way that the image recording unit has several cameras, each of which is directed at the same section and records it from different directions, or in such a way that the same section is recorded multiple times during the relative movement of the object and the respective image recording unit.Here, an image section is a partial area / section / section of an image (of a recording) of the respective inspection channel, wherein the location of the defect is arranged approximately centrally in relation to the image section and wherein, under certain circumstances, the image section can also comprise the entire image. Furthermore, in one exemplary embodiment, the image section is designed such that it has the same predetermined size (number of pixels or size of the pixel matrix in both directions) for all inspection channels (e.g., 80 pixels x 80 pixels). The location of the defect in the respective image can be determined based on the known relative movement of the object and the respective image recording unit of the respective inspection channel. Accordingly, the image section assigned to the respective defect can be determined from this on the basis of the predetermined size of the image section.

[0026] Subsequently, for each inspection channel and each detected defect, an inspection channel-specific defect type and an associated inspection channel-specific evaluation value of the determined inspection channel-specific defect type are determined by means of a first NN algorithm trained for the respective inspection channel on the basis of the image sections of the respective inspection channel assigned to the respective defect, wherein one image section or several image sections can be assigned to the respective inspection channel.

[0027] The first NN algorithm used represents an assignment based on a neural network. Using the neural network, the first NN algorithm assigns an inspection-channel-specific defect type and a corresponding inspection-channel-specific evaluation variable for this defect type to the image sections of an inspection channel. For example, a confidence value derived from the NN algorithm is used. The evaluation or confidence value indicates the probability of this defect type being present for each defect type and is derived from the previous training of the NN algorithm. The evaluation variable can, for example, have a value that is greater than zero but less than or equal to 1. Additionally, defect types that cannot be determined using the respective inspection channel can be assigned a rating of zero.In one embodiment, the neural network represents a convolutional neural network (CNN for short), the structure of which is known, which performs the assignment specified above. A CNN is a NN algorithm that is particularly suitable for image processing problems. In another embodiment, the NN algorithm used is a residual neural network (ResNet for short), the structure of which is known, in which the feature vector (inspection channel-specific error type and associated inspection channel-specific evaluation variable) is "amplified" in the form of feedback. The ResNet has a large number of layers, across which so-called "skip connections" or "shortcuts" are created during training (e.g. double or triple layer skip connections), which skip layers if they do not improve the algorithm (model).This accelerates training and also accelerates the matching process when using the learned algorithm. In one embodiment, a ResNet with at least 18 layers is used. For example, a ResNet18 is used as the NN algorithm, which consists of 18 layers and uses a residual block architecture. Residual blocks enable the training of deep networks by adding a "shortcut" or "skip connection," which allows the network to send information back from previous layers during training. This prevents the "disappearing gradient" problem that occurs with deep networks. In one embodiment, the training of the ResNet18 model is performed using the Adam optimizer and the negative log-likelihood loss function.The Adam optimizer is an adaptive learning rate optimizer that calculates an adaptive learning rate for each model parameter and updates the parameters accordingly. This optimizer allows the model to converge quickly and improve accuracy. The negative log-likelihood loss function is well suited for the classification task at hand. This function measures the error between the model's prediction of the defect and the actual defect type. The larger the error, the higher the loss. The goal of training is to minimize the loss by adjusting the model parameters. Furthermore, one embodiment uses a "stepwise learning policy" method to adjust the learning rate during training.With the stepwise learning policy, the learning rate is reduced after a certain number of epochs to improve the model's learning behavior and increase convergence speed. The initial learning rate, step size, and learning rate schedule are determined for the task at hand (defect classification with specific error types) based on prior experimental evaluations.

[0028] If an inspection channel-specific defect type and an associated evaluation parameter have been determined for the respective defect for all or some of the inspection channels (depending on whether at least one image section of the location of the respective defect could be assigned to this inspection channel for the respective inspection channel), a defect type is then selected for this identified defect of the object surface using soft voting from the inspection channel-specific defect types of the respective inspection channels and the associated inspection channel-specific evaluation parameters. Soft voting is characterized by the fact that the evaluation parameter (confidence) is taken into account when selecting the most probable defect type. In soft voting, for example, the frequency and an average value (e.g.The arithmetic mean (or arithmetic mean) of the respective assessment criteria determined for the respective defect type is calculated. The frequency of each defect type is then multiplied by its average assessment criterion value (also related to the number of inspection channels) in relation to the number of inspection channels examined, and a product is calculated in this way. The defect type selected is the one with the largest product. The defect type selected in this way is the one that is most likely present in the defect based on the results from the individual channels. If the product of two defect types is the same, then the defect type that was determined more frequently (more often) is selected. If the frequency is also the same, then both defect types can, for example, be specified as the result.

[0029] The defect type determined for each defect on the object surface is then output to an interface of a data processing device. The method according to the invention provides a simple, fast, and cost-effective inspection of coated object surfaces, which is also characterized by very good results.

[0030] The first NN algorithm is trained, as described in more detail below. It can thus be adapted to the specific inspection task / object to be inspected, as data specific to the respective object / inspection task can be used for training. This achieves high inspection accuracy.

[0031] Based on the defect types output at the interface of the data processing device (e.g. as a list of all defects with the corresponding defect type for the respective object), an automatic assessment of the quality of the object or the object's surface coating can be made. The data can also be output on a corresponding display device so that a quality assurance employee can visually inspect it. Based on the data made available at the interface, the object can also be automatically removed from the production process if, based on this data, the object does not meet the quality requirements. This can be the case, for example, if the number of defects of one or more specific defect types is exceeded.

[0032] In one embodiment, the location (position) of each detected defect on the object surface is additionally determined in a predefined coordinate system. This can be done, for example, by means of coordinate transformation based on a CAD model of the object, the calibration of the image recording unit and / or, if applicable, the speed of the relative movement of the object and image recording device or the respective positions. The predefined coordinate system can, for example, represent the coordinate system of the object. In this way, and if applicable with the help of the comparison with the CAD model of the object, the defect can then be localized very precisely on the object surface. Based on this information, for example, a marking device downstream of the inspection device can mark the defect, for example by applying (e.g. spraying) a water-soluble paint by circling the surface of the object.Alternatively or additionally, knowledge of the fault location can facilitate the control of a fault removal device.

[0033] In one embodiment, the image sections of an inspection channel are generated based on a plurality of images of different sections of the object surface when the object surface is illuminated with a predefined, inspection-channel-specific pattern. The images of the respective inspection channel are generated directly from the possibly preprocessed images of the object surface transmitted by the image acquisition unit, with all images of an inspection channel being generated when illuminated with the same predefined pattern. Different images can contain different sections of the object surface, for example, due to a relative movement of the object and the image acquisition unit.If necessary, the images are processed by means of the image recording unit and / or data processing device, for example, digitization, contrast modification and / or pre-processing are carried out or image data generated by low-pass filtering of the respective image is extracted from each image.

[0034] Images of an inspection channel are generated from transmitted image data, with either only the areas on stripes or only the transition areas between two adjacent stripes being considered for further analysis. Despite identical lighting and acquisition conditions, the corresponding images are assigned to two different inspection channels because the image data is processed differently.

[0035] In one embodiment, the images of an inspection channel are generated on the basis of a first plurality of images of different sections of the object surface when the object surface is illuminated with a first predetermined pattern and at least a second plurality of images of different sections of the object surface when the object surface is illuminated with at least a second predetermined pattern, as well as the pixel-by-pixel calculation of a sum image, a contrast image or a phase image from the images of the section at the same location.

[0036] For example, images of the object surface are generated using two or four different patterns, each with a sinusoidal, striped intensity distribution. The two patterns are shifted, for example, by half a sine period, and the four patterns by a quarter of a sine period. The intensities of the pixels in the images for the two patterns are designated IA and IB, while the intensities of the pixels in the images for the four patterns are designated h, I2, I3, and I4. A sum, contrast, or phase image is generated using the following calculation rules, with each pixel of the sum image (Io), contrast image (K), phase image ( / ®), or curvature image (C) being calculated pixel by pixel from the images at the same location as follows:

[0037] A curvature image (C) can be calculated from the phase image by continuousizing the phase image and forming a mathematical derivative perpendicular to the direction of the fringes of the illumination pattern used (transferred to the recordings or the resulting image). The multitude of calculated sum, contrast, phase, or curvature images for an object each represent an inspection channel (i.e., the inspection channel of the sum images, the inspection channel of the contrast images, and so on). The images of these inspection channels can also be used, analogous to the inspection channel images described above, to determine an inspection channel-specific defect type and an associated inspection channel-specific evaluation variable using an appropriately trained first NN algorithm.

[0038] In one embodiment, for a specific defect, a second NN algorithm is used to determine a binarized defect representation from the image sections assigned to the respective defect of at least one predetermined inspection channel of the first inspection channel, the second inspection channel and, if applicable, the further inspection channel, which contains the shape and size of the respective defect and the shape and size can be determined accordingly.

[0039] The second NN algorithm used represents an assignment based on a neural network. The second NN algorithm uses the neural network to assign a binarized representation of the defect to all image sections of all inspection channels assigned to the respective defect. In one embodiment, the second neural network is a convolutional neural network (CNN for short), which generates a binarized image from the image sections, in which the pixels belonging to the defect appear in a first color (e.g. white) and pixels that do not belong to the defect appear in a second color different from the first color (e.g. black). Using the second NN algorithm, a probability is determined for each pixel of the image sections of all inspection channels of the respective defect as to whether this pixel belongs to the defect or not.For example, a white color (or a 1) is assigned to pixels for which the probability that the pixel belongs to the error is greater than or equal to 0.5, and a black color (or a 0) is assigned to all other pixels. For example, a neural network based on the U-Net architecture is used for this purpose, which has proven to be very suitable for such image-to-image transformation problems. The UNet model consists of two main parts: the so-called "encoder" and the "decoder." The encoder consists of several convolutional layer blocks, which gradually compress the image sections into a compact representation (feature map). The decoder is a symmetric structure to the encoder. It consists of a sequence of up-convolutional layer blocks that convert the compressed feature map back to the original resolution of the input image.Additionally, there is a connection between the encoder and decoder, often referred to as a "skip connection." At the end of the UNet model, a softmax layer is used to calculate the probability that a given pixel belongs to a given error type.

[0040] In one embodiment, the inspection channel-specific NN algorithm is trained using ground-truth images of defects that have previously been assessed by experts with regard to the various defect types and assigned to a specific defect type. With regard to the second NN algorithm, for example, the defects are outlined by experts in training image sections with a polygonal line. From this, a mask is generated that has pixels within the polygonal line that belong to the defect and therefore have, for example, the value 1, and that outside the polygonal line have, for example, the value 0. A plurality of such masks is used to train the NN algorithm, for example in the form described above with regard to ResNet18.

[0041] In one embodiment, as described above or below, the training of the first NN algorithm is carried out by means of an adaptive learning rate optimizer and / or the training of the second NN algorithm is carried out by means of a Dice-Coefficient-Loss loss function.

[0042] The above object is further achieved by a device for inspecting a coating of a surface of an object, wherein the device has an image recording unit, for example a camera, and a data processing device, wherein the image recording unit and optionally the data processing device is / are set up such that a plurality of first images of the object surface are generated with respect to a first optical inspection channel, a plurality of second images of the object surface with respect to a second optical inspection channel and optionally a plurality of further images of the object surface with respect to at least one further optical inspection channel, wherein the data processing device is set up such thatthat the plurality of images of each inspection channel are automatically analyzed for the presence of a defect and the following steps are performed for each inspection channel and each detected defect:

[0043] • Assigning those image sections of the plurality of images of the respective inspection channel which contain the location of the respectively detected defect, • Determining an inspection channel-specific defect type and an associated inspection channel-specific evaluation value of the determined inspection channel-specific defect type for those inspection channels to which at least one image section of the location of the detected defect could be assigned, wherein the determination is carried out by means of a first NN algorithm trained for the respective inspection channel on the basis of the image sections of the respective inspection channel assigned to the respective defect, wherein the data processing device is set up in such a way,that for each detected defect of the object surface, a defect type is then selected by means of soft voting from the inspection channel-specific defect types of the respective inspection channels and the respective associated inspection channel-specific evaluation variables, and that for each detected defect of the object surface, the respectively selected defect type is output at an interface of the data processing device.

[0044] The above device has the advantages and corresponding embodiments described above with regard to the method. Therefore, with regard to the device, reference is made to the above and below explanations of the method and the device, respectively.

[0045] In one embodiment of the device, the data processing device is configured such that the location of each detected defect of the object surface is additionally determined in a predetermined coordinate system.

[0046] In one embodiment of the device, the data processing device is configured such that the images of an inspection channel are generated on the basis of a plurality of images taken by the image recording unit of different sections of the object surface when the object surface is illuminated by means of an illumination device with a predetermined, inspection channel-specific pattern.

[0047] In one embodiment of the device, the data processing device is configured such that the images of an inspection channel are generated on the basis of a first plurality of images taken by the image recording unit of different sections of the object surface when the object surface is illuminated with a first predetermined pattern and at least a second plurality of images taken by the image recording unit of different sections of the object surface when the object surface is illuminated with at least a second predetermined pattern, as well as the pixel-by-pixel calculation of a sum image, a contrast image, a phase image, or a curvature image from the images of the section at the same location.In one embodiment of the device, the data processing device is configured such that, for a specific defect, a binarized defect representation containing the shape and size of the respective defect is determined by means of a second NN algorithm from the image sections assigned to the respective defect of at least one predetermined inspection channel of the first inspection channel, the second inspection channel and, if applicable, the further inspection channel.

[0048] In one embodiment, analogous to the above procedure, ground truth image sections are used, for which a binarized representation is provided as corresponding annotation. During training, the annotated images are first provided. The data is divided into training, validation, and test sets, for example, in order to train the second NN algorithm (e.g., UNet model) and evaluate its performance. The UNet model is then configured and created, for example, to perform the training. The model consists of an encoder and a decoder architecture with a softmax output at the end, which calculates the probability of assigning the respective pixel to the error. For example, the Dice coefficient loss is used as a loss function to minimize the difference between the predicted segmentation and the actual segmentation of the training data.The model is then trained using the training sets. The model is fed the images and corresponding annotation data and adjusted to improve performance. Training can occur over multiple epochs, with the model weights being updated after each epoch. A validation curve is then created that monitors the loss function on the training and validation sets for each epoch. If the validation loss decreases, the trained weights are saved for that epoch. The validation sets are image data that were not included in the training sets. The same applies to the test sets. At the end of training, the model is evaluated on a test set to assess its performance on new data.

[0049] Further advantages, features, and possible applications of the present invention will become apparent from the following description of exemplary embodiments and the drawings. All described and / or illustrated features, individually or in any combination, constitute the subject matter of the invention, regardless of their summary in the claims or their references.

[0050] They show schematically

[0051] Fig. 1 shows an embodiment of the device according to the invention in a side view, Fig. 2 shows an exemplary representation of the inspection channels and images assigned to each inspection channel and

[0052] Fig. 3 is a flow chart of part of the method according to the invention.

[0053] Fig. 1 shows an embodiment of a device according to the invention, with which an object in the form of a car body 3 provided with a paint coating is inspected. The body 3 is moved past the device on a conveyor belt (not shown), specifically in the feed direction, which is illustrated in Fig. 1 by the arrow 4.

[0054] The inspection device comprises a lighting unit 5 which is attached to a robot arm 6, wherein the robot arm 6 enables precise adjustment of the position of the lighting unit relative to the body 3 or other objects to be inspected.

[0055] The lighting unit 5 has a plurality of LEDs which generate an illumination pattern by switching these LEDs on and off. For example, as shown in Fig. 1, a stripe pattern 7 can be generated that extends parallel to the feed direction on the surface of the body 3. Other patterns, e.g., staggered stripe patterns parallel to the feed direction, stripe patterns transverse to the feed direction, or full illumination (all LEDs of the lighting unit 5 are switched on), can also be realized. The light reflected from the surface of the body 3 is recorded from various angles by matrix cameras 10, 11, whereby a plurality of images of the body 3 are generated for all of the lighting configurations specified below and during the relative movement of the body 3 and cameras 10, 11, digitized, and forwarded to the data processing device (computer) 13.Each matrix camera 10, 11 captures images of different sections of the body. The data processing device 13 is electrically connected to the matrix cameras 10, 11. It also controls the times at which the matrix cameras 10, 11 are read out. Furthermore, the data processing device 13 is connected to the lighting unit 5 to control the lighting patterns implemented by the latter. Naturally, the readout of the matrix cameras 10, 11 and the setting of the respective lighting pattern are synchronized.

[0056] An example of illumination by means of the illumination unit 5 and the corresponding generation of images by means of the matrix cameras 10, 11 includes the following configurations: Illumination 1: all LEDs of the illumination unit 5 switched on,

[0057] Illumination 2: Illumination pattern includes rectangular stripes perpendicular to the feed direction, Illumination 3: Illumination pattern includes rectangular stripes parallel to the feed direction (see Illumination pattern 7),

[0058] Lighting 4: Lighting pattern like Lighting 3, but offset by period, Lighting 5: like Lighting 2,

[0059] Illumination 6: Illumination pattern as illumination 3, but offset by 1 / 2 period, Illumination 7: Illumination pattern as illumination 3, but offset by % period, Illumination 8: Illumination pattern contains sine stripes parallel to the feed direction, Illumination 9: Illumination pattern as illumination 8, but offset by 1 / 2 period offset.

[0060] The lighting configurations 1 to 9 are set sequentially, and the images are captured using the matrix cameras 10 and 11. The body 3 moves / rests during the lighting and capture of the images.

[0061] The multitude of images for the various sections of the body 3 are forwarded to the data processing device 13. There, images of a plurality of inspection channels are generated from these images. The following images of the inspection channels (abbreviated to: K) are generated in this exemplary embodiment for further inspection:

[0062] K1 : Images of illumination 1 , where the respective low-pass filtered image is subtracted from all images,

[0063] K2: Images of illumination 2, where only the signals along the stripes are included for the evaluation,

[0064] K3: Images of illumination 3, where only the signals along the stripes are included for the evaluation,

[0065] K4: Images of illumination 4, where only the signals along the stripes are included for the evaluation,

[0066] K5: Images of illumination 5, where only the signals along the stripes are included for evaluation,

[0067] K6: Images of illumination 6, where only the signals along the stripes are included for evaluation,

[0068] K7: Images of illumination 7, where only the signals along the stripes are included for evaluation,

[0069] K8: Images of illumination 2, where only the signals of the transition between two stripes are included for the evaluation,

[0070] K9: Images of illumination 3, where only the signals of the transition between two stripes are included for the evaluation,

[0071] K10: Images of illumination 4, where only the signals of the transition between two stripes are included for the evaluation,

[0072] K11 : Images of illumination 5, where only the signals of the transition between two stripes are included for the evaluation,

[0073] K12: Images of illumination 6, where only the signals of the transition between two stripes are included for the evaluation, K13: Images of illumination 7, where only the signals of the transition between two stripes are included for the evaluation,

[0074] K14: Sum images from the recordings of illuminations 8 and 9 using equation (1) shown above.

[0075] All images are then examined to determine whether they contain errors. Errors are detected in different ways for the various inspection channels: For K1, for example, the previously calculated difference to the low-pass filtered image is binarized using an adaptive tile-based thresholding operation. This is used to determine whether errors are present in the respective image and, if so, where they are located. For all other inspection channels, an overlay image is first calculated. This overlay image contains information about the location of the dark stripes, the light stripes, and the transition areas between the dark and light stripes. Gradients are then calculated based on the acquired images. The overlay images (masks) are used to clean up the gradient images (e.g., by addition).For inspection channels K2 to K7, the cleanup is performed in the transition area between the stripes; for inspection channels K8 to K13, the cleanup is performed in the area of ​​the light or dark stripes themselves. After the cleanup, a threshold-based binarization is also performed, and the binarized image is used to determine whether an error is present in the respective image (e.g., by examining the size of the highlighted areas in the binarized image).

[0076] If a defect is detected, its location is determined, and the image sections that also contain the defect are identified for all channels K1 to K14. These sections can, for example, have a size of 80 pixels x 80 pixels. This determination is based on the spatial assignment of the image sections to the location of the defect. The location of the defect and the determination of the image sections corresponding to that location are performed, for example, in the object's coordinate system.

[0077] The result is a series of channels for which at least a section of the detected defect could be determined. Fig. 2 illustrates this for channels K1 to K5. The following explanation of the method applies accordingly to the other channels K6 to K14.

[0078] Fig. 2 shows the channels K1 to K5, whereby five image sections K1.1 to K1.5, which contain the detected defect, can be assigned to channel K1. Correspondingly, channel K2 contains two image sections K2.1 and K2.2, channel K3 four image sections K3.1 to K3.4, channel K4 no image section and channel K5 three image sections K5.1 to K5.3. Since the fourth channel does not contain an image section that can be assigned to the detected defect, it is not taken into account in the further process, which is illustrated with reference to Fig. 3. For each inspection channel to which image sections related to the detected defect could be assigned (here K1, K2, K3 and K5), a defect type and an evaluation variable are now determined using a first NN algorithm trained for the respective channel in the form of a ResNet (e.g. ResNet18 as described above). For channel K1, the error type is K1.8 and the evaluation value is K1.9.Accordingly, the error type K2.8 and the evaluation size K2.9 are assigned for channel K2, the error type K3.8 and the evaluation size K3.9 for channel K3, and the error type K5.8 and the evaluation size K5.9 for channel K5. For example, the values ​​contained in the following table can be determined for two sample values, whereby three error types A, B, and C (e.g., inclusion, crater, or protrusion (bulge)) are generally possible. In both examples, error type A is selected for this error and output at the interface of the data processing device.

[0079] The inspection channel-specific error types K1.8, K2.8, ​​K3.8, and K5.8, along with the associated inspection channel-specific evaluation variables K1.9, K2.9, K3.9, and K5.9, are then fed to a soft voting module 28 of the method. During the soft voting performed by this module, the method described above is applied (the average value is determined as the arithmetic mean), and for both examples of the error, error type A is selected from the inspection channel-specific error types K1.8, K2.8, ​​K3.8, and K5.8, since the calculated soft voting value for this error type is the largest (see the tables above). The error type thus determined (error type A) is subsequently made available for the respective error at the interface of the data processing device 13 (see step 30 in Fig. 3).

[0080] If another error is detected, the steps of channel-wise assignment of the image sections belonging to the error and the subsequent steps explained above are repeated.

[0081] Furthermore, using coordinate transformation, the location (position) of the respective defect can be converted from the coordinate system of the inspection device into the position in the coordinate system of the body 3. Using a second NN algorithm, a binarized defect representation can be determined from all image sections K1.1 to K5.3 of all channels K1 to K5 that have been assigned to the defect, which is shown as an example in Fig. 4. There, the pixels in area F1 are white – they contain the shape and size of the respective defect. The areas around area F1 are black (shown striped here). The striped areas are the areas other than the defect.

[0082] Based on the determined area F1, the shape and size of the defect can be easily calculated.

[0083] The binarized error representation can be determined analogously for all detected errors by applying the second NN algorithm.

[0084] The location of all detected errors and their binarized error representation can also be made available at the interface of the data processing device 13.

Claims

Patent claims 1. A method for inspecting a coating on a surface of an object, wherein a plurality of first images of the object surface are generated with respect to a first optical inspection channel (K1), a plurality of second images of the object surface with respect to a second optical inspection channel (K2) and optionally a plurality of further images of the object surface with respect to at least one further optical inspection channel (K3 to K5), wherein the plurality of images of each inspection channel are automatically analyzed with respect to the presence of a defect, wherein the following steps are carried out for each inspection channel and each detected defect: • Assigning those image sections (K1.1 to K1.5, K2.1 to K2.2, K3.1 to K3.4, K5.1 to K5.3) of the multitude of images of the respective inspection channel that contain the location of the respective detected defect, • Determination of an inspection channel-specific defect type (K1.8, K2.8, ​​K3.8, K5.8) and an associated inspection channel-specific evaluation variable (K1.9, K2.9, K3.9, K5.9) the determined inspection channel-specific defect type for those inspection channels to which at least one image section of the location of the detected defect could be assigned, wherein the determination is carried out by means of a first NN algorithm trained for the respective inspection channel on the basis of the image sections of the respective inspection channel assigned to the respective defect, wherein for each determined defect of the object surface a defect type is then selected by means of soft voting from the inspection channel-specific defect types of the respective inspection channels determined for the respective defect and the associated inspection channel-specific evaluation variables, wherein for each determined defect of the object surface the respectively selected defect type is output at an interface of a data processing device.

2. Method according to claim 1, characterized in that the location of each detected defect of the object surface is additionally determined in a predetermined coordinate system.

3. Method according to one of the preceding claims, characterized in that the images of an inspection channel are generated on the basis of a plurality of images of different sections of the object surface when the object surface is illuminated with a predetermined, inspection channel-specific pattern.

4. Method according to one of the preceding claims, characterized in that the images of an inspection channel are generated on the basis of a first plurality of images of different sections of the object surface when the object surface is illuminated with a first predetermined pattern and at least a second plurality of images of different sections of the object surface when the object surface is illuminated with at least a second predetermined pattern and the pixel-by-pixel calculation of a sum image, a contrast image, a phase image or a curvature image from the images of the section at the same location.

5. Method according to one of the preceding claims, characterized in that for a specific defect, a binarized defect representation is determined by means of a second NN algorithm from the image sections assigned to the respective defect of at least one predetermined inspection channel of the first inspection channel, the second inspection channel and optionally the further inspection channel, which contains the shape and size of the respective defect.

6. Method according to one of the preceding claims, characterized in that the training of the first NN algorithm is carried out by means of an adaptive learning rate optimizer and / or the training of the second NN algorithm is carried out by means of a Dice Coefficient Loss function.

7. A device for inspecting a coating on the surface of an object, the device comprising an image recording unit and a data processing device, the image recording unit and optionally the data processing device being configured to generate a plurality of first images of the object surface with respect to a first optical inspection channel (K1), a plurality of second images of the object surface with respect to a second optical inspection channel (K2), and optionally a plurality of further images of the object surface with respect to at least one further optical inspection channel (K3 to K5), the data processing device being configured to automatically analyze the plurality of images of each inspection channel with respect to the presence of a defect, and to perform the following steps for each inspection channel and each detected defect: • Assigning those image sections (K1.1 to K1.5, K2.1 to K2.2, K3.1 to K3.4, K5.1 to K5.3) of the multitude of images of the respective inspection channel that contain the location of the respective detected defect, • Determination of an inspection channel-specific defect type (K1.8, K2.8, ​​K3.8, K5.8) and an associated inspection channel-specific evaluation variable (K1.9, K2.9, K3.9, K5.9) of the determined inspection channel-specific defect type for those inspection channels to which at least one image section of the location of the detected defect could be assigned, wherein the determination is carried out by means of a first NN algorithm trained for the respective inspection channel on the basis of the image sections of the respective inspection channel assigned to the respective defect, wherein the data processing device is set up in such a way that for each determined defect of the object surface, a defect type is then selected by means of soft voting from the inspection channel-specific defect types of the respective inspection channels determined for the respective defect and the respectively associated inspection channel-specific evaluation variables, and that for each determined defect of the object surface, the respectively selected defect type is output at an interface of the data processing device.

8. Device according to claim 7, characterized in that the data processing device is arranged such that the location of each detected defect of the object surface is additionally determined in a predetermined coordinate system.

9. Device according to one of claims 7 to 8, characterized in that the data processing device is set up such that the images of an inspection channel are generated on the basis of a plurality of images of the image recording unit of different sections of the object surface when the object surface is illuminated by means of an illumination device with a predetermined, inspection channel-specific pattern.

10. Device according to one of claims 7 to 9, characterized in that the data processing device is set up such that the images of an inspection channel are generated on the basis of a first plurality of images taken by the image recording unit of different sections of the object surface when the object surface is illuminated with a first predetermined pattern and at least a second plurality of images taken by the image recording unit of different sections of the object surface when the object surface is illuminated with at least a second predetermined pattern as well as the pixel-by-pixel calculation of a sum image, a contrast image, a phase image or a curvature image from the images of the section at the same location.

11. Device according to one of claims 7 to 10, characterized in that the data processing device is set up in such a way that for a specific defect, a binarized defect representation containing the shape and size of the respective defect is determined by means of a second NN algorithm from the image sections assigned to the respective defect of at least one predetermined inspection channel of the first inspection channel, the second inspection channel and optionally the further inspection channel.

12. Device according to one of claims 7 to 11, characterized in that the training of the first NN algorithm is carried out by means of an adaptive learning rate optimizer and / or the training of the second NN algorithm is carried out by means of a Dice Coefficient Loss function.

Images