Inspection device for surface coating

Abstract

The invention relates to a method for inspecting a coating on a surface of an object (e.g., a painted surface) and to a corresponding device for quickly and easily classifying defects on the surface. This involves generating a plurality of first images of the object surface with respect to the first optical inspection channel (K1), a plurality of second images of the object surface with respect to the second optical inspection channel (K3), and optionally a plurality of further images of the object surface with respect to at least one further optical inspection channel (K3 to K5), wherein a plurality of images of each inspection channel are automatically evaluated in order to determine whether a defect is present, and wherein, if a defect is detected, for each inspection channel which can assign at least one image section (K1.1 to K1.5, K2.1 to K2.2, K3.1 to K3.4, K5.1 to K5.3) to the defect and to each detected defect, the image section (K1.1 to K1.5, K2.1 to K2.2, K3.1 to K3.4, K5.1 to K5.3) is assigned to the defect and to each detected defect by means of an NN algorithm. According to the invention, inspection channel-specific defect types (K1.8, K2.8, K3.8, K5.8) and inspection channel-specific evaluation parameters (K1.9, K2.9, K3.9, K5.9) are determined, and then a defect type is automatically selected from the inspection channel-specific defect types, the defect type with respect to the detected defect being output at the interface of the data processing unit.

Description

describe

[0001] The present invention relates to an apparatus and a corresponding method for inspecting coatings on the surface of an object (e.g., a painted surface).

[0002] In automated manufacturing, such as in the automotive industry, quality assurance requirements are very high. In particular, when inspecting the surface coating of workpieces (e.g., painted products), there is a need to quickly and effortlessly detect, classify, and locate potential defects. Various such devices and methods are known and exist in practical use.

[0003] According to document WO 2017 / 081029 A2, a method for automatic detection and robot-assisted processing of defects on a workpiece surface is known. This method involves performing an optical inspection of the surface to detect defects, then using an optical sensor to perform a three-dimensional measurement of the workpiece surface in the region of the detected defect. Subsequently, the morphology of the workpiece surface in the region of at least one defect is determined, and then a set of parameters characterizing the at least one defect is determined. The at least one defect is classified based on the determined parameter set, wherein the defect is assigned to a defect category. In the next step, a processing procedure is selected according to the defect category of the at least one defect, including a processing path along which the defect will be processed. Finally, a robot program for robot-assisted processing of the at least one defect is generated by a computer. In this known method, the processing of defects is adapted to the type of defect.

[0004] Document WO 2021 / 151412A describes an apparatus for automatically detecting damage to vehicles. This apparatus includes a light tunnel having several light sources in a strip-like shape, at least one camera, at least one first main camera, at least one second main camera, and at least one local computer including one or more powerful graphics processors that receive and process images from the at least one camera, the first main camera, and the second main camera, and calculate output signals, wherein the output signals generate a classification of the defective object and its coordinates in the image. Deep learning-based object detectors exist for detecting damage in the form of defective objects. Such known apparatuses are relatively complex in design.

[0005] Document WO 2022162417 A1 describes a method for detecting defects in the color layer of an object using machine learning / artificial intelligence (AI) methods. Here, a large number of patterns are projected onto the surface of the object, generating surface images with the patterns. These images are then fed into a machine learning module to determine whether the object surface includes defects. Images with color defects detected in this way are then displayed. Using this method, only a very limited range of types of color defects can be detected.

[0006] To eliminate defects in a surface coating, it is necessary to know the type of defect in each case. If the type of defect is known and its location on the object's surface is also known, then quick and easy further measures can be taken to remedy the defect in question.

[0007] The above objective is achieved by a method having the features of claim 1 and an apparatus having the features of claim 7.

[0008] Specifically, the above objective is achieved by a method for inspecting a coating on the surface of an object, wherein a plurality of first images of the object surface are generated relative to a first optical inspection channel, a plurality of second images of the object surface are generated relative to a second optical inspection channel, and optionally, a plurality of additional images of the object surface are generated relative to at least one additional optical inspection channel. Specifically, to determine whether defects exist, multiple images from each inspection channel are automatically analyzed. For each inspection channel and each detected defect, the following steps are performed: - The image regions containing the location of the corresponding detected defects are assigned to multiple images of the corresponding inspection channels. - For the location of at least one image region of a detected defect, those inspection channels can be assigned to, determining an inspection channel-specific defect type and associated inspection channel-specific evaluation parameters for the determined inspection channel-specific defect type, wherein the determination is made based on the image region of the corresponding inspection channel assigned to the corresponding defect by means of a first NN algorithm trained for the corresponding inspection channel (i.e., an algorithm including an artificial neural network trained for the corresponding inspection channel). Subsequently, for each detected defect on the object surface, a defect type is selected by means of soft voting based on the inspection channel-specific defect type of the corresponding inspection channel and the associated inspection channel-specific evaluation parameters determined for the corresponding defect. The defect type selected in this way for each detected defect on the object surface is output at the interface of the data processing device.

[0009] As described above, there are at least two different optical inspection channels, but in many cases, there are significantly more than two (e.g., four to fifteen different inspection channels). For each of these inspection channels, multiple images (records) of the object surface are generated. The images associated with each optical inspection channel are generated by means of an illumination unit and an associated image recording unit, and optionally a data processing unit. Each inspection channel is characterized by predetermined inspection channel-specific illumination by the illumination unit, inspection channel-specific recording by the image recording unit, and / or inspection channel-specific processing of the captured data. Using the inspection channel-specific illumination, the image recording unit generates a large number of optionally digitized images (i.e., image data) of light reflected from the coated object surface, the images containing predetermined areas of the object surface to be inspected (e.g., a large number of records from a matrix camera). These records are transferred to the data processing unit for further analysis, but may have already been preprocessed in the image recording unit (e.g., digitized or processed with respect to image parameters such as brightness or contrast, or processed by subtracting a low-pass filtered record from each record). According to the corresponding inspection channel, the recorded data transmitted by the image recording unit for one or more inspection channels can be directly obtained without further processing by the data processing unit. Therefore, the images of these inspection channels are generated directly from the transmitted recorded data (e.g., recordings from a matrix camera) corresponding to the image data. For example, the images of the first and second inspection channels can differ because they are generated using different lighting patterns. Alternatively, the recorded data is processed in the data processing unit such that a designated area of ​​the recorded data is used for defect detection, for example, an area along a stripe in a stripe pattern or an area between two stripes in a stripe pattern. The recorded data processed in this way is assigned to one inspection channel focusing on the area along the stripe and another inspection channel examining whether there are possible defects in the area between the stripes. In another alternative, image data generated by the illumination unit under different lighting conditions is further processed based on calculation rules (see example below) and can be used as another inspection channel. This means that each inspection channel includes inspection channel-specific illumination and / or inspection channel-specific recording and / or inspection channel-specific processing of the recorded data performed by the image recording unit; that is, each inspection channel differs from another inspection channel in at least one of these characteristic factors (illumination, recording, processing of image data).Examples are shown below, and can be based on, for example, the use of different lighting patterns for the feature factor "illumination," the use of different cameras and / or different camera modes for the feature factor "recording" (e.g., recording data from different camera regions or camera pixel groups and / or recording data from different wavelength ranges and / or recording data from different time intervals (sequences) can be read out), and the use of different calculation rules for the feature factor "image data processing" (e.g., subtraction for generating summed images, contrast images, phase images or curvature images, low-pass filtered images). When comparing all feature factors, the feature factors "illumination" and "processing of image data" are particularly noteworthy because they are the most easily varied.

[0010] The illumination unit is configured to illuminate an object in an area to be inspected using a predetermined inspection channel-specific illumination pattern. The illumination pattern may differ for different inspection channels, or recorded data generated using different inspection channel-specific illumination patterns may be used for a single inspection channel. The illumination pattern is generated by the light-emitting portion of the illumination unit itself, or by illumination light generated through a grid of light that forms the illumination pattern. The illumination is within the visible light wavelength range. The illumination unit may be made as a single piece or as a plurality of spaced-apart elements.

[0011] An image recording unit (e.g., one or more cameras (e.g., a matrix camera)) captures multiple records of an illuminated area of ​​an object's surface, i.e., a sequence of image records that generate an illumination pattern, wherein the image recording unit sees the illumination pattern in reflection (incident light arrangement) relative to the object. In this case, the image recording unit and the object can perform relative movement relative to each other, such that the entire surface area of ​​the object to be inspected can be captured by the image recording unit. Therefore, the illumination unit can also perform relative movement relative to the object, for example, together with the image recording unit. The digitized recording data generated by the image recording unit is transmitted from the image recording unit to a data processing unit (computer, processor, etc.), and is available for this purpose at the corresponding interface of the image recording unit connected to the data processing unit.

[0012] An image recording unit can consist of one or more matrix cameras. Matrix cameras, also known as area cameras, record a large area of ​​an object. For example, they can be designed as CCD cameras. A matrix camera captures light intensity from an illuminated area through a large number of pixels arranged in rows and columns, i.e., in a matrix form. For this purpose, a matrix camera includes a photosensitive element (e.g., a CCD sensor) for each pixel. The size (in pixels) of the area captured by each photosensitive element determines the resolution of the matrix camera. For example, several matrix cameras can be arranged adjacent to each other in a direction perpendicular to the relative movement of the image recording unit and the object to examine particularly wide objects.

[0013] As described above, the object and the image recording unit (and optionally, the illumination unit) can perform relative movement with respect to each other in the feed direction at a predetermined speed. The vertical direction extends perpendicularly, preferably perpendicular to the feed direction. For example, the object passes over the image recording unit (and optionally the illumination unit) at a predetermined speed on a corresponding strip or as web material, and the image recording unit (and optionally the illumination unit) does not move. Alternatively, only the image recording unit (and optionally the illumination unit) moves, or both the image recording unit (and optionally the illumination unit) and the object move simultaneously. If the relative movement of the object and the image recording unit (and optionally the illumination unit) is uniform, i.e., at a constant speed, it is advantageous for processing the image capture data. This simplifies the processing of the image capture data to examine the object.

[0014] Each element of a lighting unit may include several individually switchable lighting elements, such as light-emitting diodes (LEDs) or groups of jointly switchable light-emitting diodes (LEDs). In this case, the lighting pattern is generated as a whole by the lighting elements themselves. Therefore, individually switchable lighting elements may consist of a single lighting device (e.g., an LED) or several lighting devices grouped together and switchable together (e.g., a group of several LEDs). In one embodiment, the data processing unit may be configured to control the individual lighting elements individually, i.e., to turn them on and off. For example, bright fields / stripes are achieved by LEDs that are turned on, while dark fields / stripes are achieved by LEDs that are turned off. This allows for flexible inspection channel-specific lighting configurations using different patterns and / or using light of different wavelengths.

[0015] Alternatively or additionally, the lighting unit may include a lighting device (e.g., one or more fluorescent tubes) that fully illuminates the area to be illuminated. In this case, a grid is arranged between the lighting device and the object, generating the desired lighting pattern. Bright fields / lines are generated by corresponding slits in the grid, while dark fields / lines are generated by corresponding stripes.

[0016] The lighting unit can provide lighting patterns to achieve bright field illumination and / or dark field illumination. The lighting device can alternatively or additionally emit light of different wavelengths.

[0017] Since dark-field illumination does not appear immediately in the transition from the illuminating light-emitting element to the non-illuminating light-emitting element, it may be advantageous to combine several adjacently cut-off, i.e., dark-emitting elements in one embodiment of the illumination pattern. In another embodiment, an optical light-shaping unit may be arranged between the illumination unit and the object and / or between the object and the image recording unit. In particular, the light-shaping unit may consist of lenses, diffusers, and / or microlenses.

[0018] The illumination pattern can be, for example, a stripe pattern of light (illumination) and dark (non-illumination) stripes extending in or perpendicular to the feed direction as explained below. For different inspection channels, the stripe patterns can have the same shape (e.g., the same stripe shape, the same stripe width, the same stripe direction), but can be formed offset relative to each other. The stripe patterns for different inspection channels can also differ in stripe width or the signal shape of the stripes in the direction perpendicular to the stripes (e.g., rectangular shape, sine shape). Additionally, the inspection channel can include a pattern representing full illumination (i.e., all illumination elements are illuminated).

[0019] The inspection method can be implemented as a computer-based method, i.e., a method executed by a data processing unit (computer) based on image capture data from the inspection of an object. The method may also include controlling a lighting unit and an image recording unit such that, for example, images of all regions of the area to be inspected are generated sequentially and / or simultaneously, regarding a first inspection channel, a second inspection channel, and optionally additional inspection channels. For example, based on image capture, a defect detection method is used to determine defects according to local deviations from image capture signals generated by defect-free objects. In the context of this invention, recording, recording data, or images and image data are understood as data that assigns intensity values ​​(e.g., between 0 and 255) to each point of a given two-dimensional matrix (image). This can be a pixel image or a vector graphic.

[0020] The lighting unit and image recording unit are optionally electrically connected to each other via a control unit, allowing control over the illumination of the object and the capture of recordings of the object's surface. A data processing unit can also be connected to the lighting unit and image recording unit and form a control unit. Alternatively, the data processing unit can be located remotely from the lighting unit and image recording unit, including, in particular, the use of a remote server or cloud computing.

[0021] According to the method of the present invention, after an image has been generated, the image is analyzed to determine whether defects exist on / in the surface coating of the object. At this point in time, the type of defect is irrelevant. Only deviations from the expected surface conditions typically considered defects are identified on the surface of the object, and the location of the corresponding defects is determined. The method of the present invention then performs a more detailed analysis of the detected defects according to their types, as described below. For example, the following defect types can be distinguished: inclusions, pits (dents), bumps (raises), contaminants (dust), pseudo-edges, orange peel, pores, cracks, grinding marks, spots, surface defects, blistering, scratches, and wet marks. Pseudo-edges are a type of defect that occurs when capturing the edges of the chassis. The presence of defects on / in the surface coating of the object is determined using image processing methods. In one embodiment, defect detection in the image of the corresponding inspection channel is performed according to the corresponding inspection channel. In one embodiment, low-pass filtered images and / or gradient images are used for defect detection. For example, for a subset of inspection channels, the corresponding low-pass filtered image is subtracted from each image, and then examined to determine whether the gradient of adjacent image points in the image exceeds or falls below a predetermined first threshold, or whether the absolute value of an image point (pixel) exceeds or falls below a predetermined second threshold. Here, the corresponding threshold can be set individually for specific regions of the image (so-called patches). At points where the first or second threshold is exceeded or not reached, the presence of a defect is assumed. As another criterion, in another embodiment, a standard of deviation magnitude can be used. This means, for example, that a defect is only identified if, in another embodiment, the region in the image that exceeds or falls below the threshold consists of a predetermined number of pixels (e.g., at least 5 pixels). This prevents artifacts from image processing from being identified as defects. In another embodiment, before examining whether the gradient or absolute value in the image exceeds or falls below the first or second threshold, a mask (cover) is applied to the image (i.e., added to, subtracted from, or partially replaced as measured recorded data). This cover can be calculated based on image data or preset based on the location of the captured object surface. In particular, use this application when the preset lighting pattern is used to inspect channels or when the edges of objects are positioned within a region of an image.

[0022] If the presence of a defect (and its location / position, e.g., relative to the object) is detected, for each existing inspection channel (i.e., the first inspection channel, the second inspection channel, and each additional inspection channel), those image regions containing the location of the corresponding detected defect are determined. In one embodiment, several image regions of each inspection channel may be assigned to the location of the detected defect because, for each inspection channel, several image regions are generated based on each region of the surface of the object to be inspected, etc., for example, such that the image recording unit includes several cameras, each pointing at the same region and capturing it from different directions, or such that the same region is captured multiple times during relative movement of the object and the corresponding image recording unit. Here, an image region is a sub-region / section / part of the image (picture) of the corresponding inspection channel, wherein the location of the defect is arranged approximately centrally relative to the image region, and wherein, in certain cases, the image region may also include the entire image. In another embodiment, the image regions are arranged such that they have the same predetermined size (number or size of the pixel matrix in both directions) for all inspection channels (e.g., 80 pixels × 80 pixels). The location of defects in a corresponding image can be determined based on the known relative movement of the object and the corresponding image recording unit of the corresponding inspection channel. Therefore, the image region assigned to the corresponding defect can be determined based on the specified size of the image region.

[0023] Subsequently, for each inspection channel and each detected defect, using a first NN algorithm trained for each inspection channel, based on the image region of the corresponding inspection channel assigned to the corresponding defect, for each inspection channel and each detected defect, an inspection channel-specific defect type and an associated inspection channel-specific evaluation parameter for the determined inspection channel-specific defect type are determined, wherein one or more image regions can be assigned to the corresponding inspection channel.

[0024] The first NN algorithm used represents a neural network-based assignment. The first NN algorithm uses a neural network to assign a channel-specific defect type and an associated channel-specific evaluation parameter, such as a confidence variable generated by the NN algorithm, to a region of the image being inspected. The evaluation or confidence variable indicates the probability of the presence of that defect type and is derived from previous training of the NN algorithm. The evaluation parameter can, for example, have a value greater than zero but less than or equal to 1. Additionally, an evaluation parameter of 0 can be assigned to a defect type for which the corresponding inspection channel is indeterminate. In one embodiment, the neural network represents a convolutional neural network (CNN) with a known structure, and the above assignment is performed through it. CNN is a type of NN algorithm particularly well-suited for image processing problems. In another embodiment, a residual neural network (ResNet) with a known structure is used as the NN algorithm, where the feature vectors (channel-specific defect types and associated channel-specific evaluation parameters) are, to some extent, "amplified" in the form of feedback. ResNet has a large number of layers through which so-called "skipped connections" or "shortcuts" (e.g., two- or three-layer skipped connections) are generated during training; if a layer does not improve the algorithm (model), it is skipped. This accelerates training and also speeds up assignment when using the trained algorithm. In one embodiment, a ResNet with at least 18 layers is used. For example, ResNet18, consisting of 18 layers and using a residual block architecture, is used as the NN algorithm. Residual blocks train deep networks by adding "shortcuts" or "skipped connections," allowing the network to send back information from the first few layers during training. This prevents the "vanishing gradient" problem that occurs in deep networks. In one embodiment, the ResNet18 model is trained using the Adam optimizer and a negative log-likelihood loss function. The Adam optimizer is an adaptive learning rate optimizer that computes an adaptive learning rate for each parameter of the model and updates the parameters accordingly. This optimizer allows the model to converge quickly and improves accuracy. The negative log-likelihood loss function is well-suited for the current classification task. This function measures the error between the model's prediction of a defect and the actual defect type. The larger the error, the higher the loss value. The goal of training is to minimize the loss value by adjusting the model's parameters. Furthermore, in one embodiment, a "stepwise learning strategy" approach is used to adjust the learning rate during training. A stepwise learning strategy is employed, reducing the learning rate to a specific number of epochs to improve the model's learning behavior and increase convergence speed. Based on previous experimental evaluations, an initial learning rate, step size, and learning rate schedule are determined for the current task (defect classification for a specific defect type).

[0025] If channel-specific defect types and corresponding evaluation parameters have been determined for all or a subset of inspection channels (depending on whether at least one image region locating the corresponding defect can be assigned to that inspection channel), then, using soft voting, a defect type is selected for the detected defect on the object surface based on the channel-specific defect type determined for the corresponding defect of that inspection channel and the corresponding associated channel-specific evaluation parameters. Soft voting is characterized by considering evaluation parameters (confidence levels) when selecting the most likely defect type. In soft voting, for example, for the corresponding defect types of all determined channel-specific defect types, the frequency and average (e.g., arithmetic mean) of the corresponding evaluation parameters determined for each defect type are calculated. In each case, the number of inspection channels examined is referenced, and then the frequency of the corresponding defect type is multiplied by its average evaluation parameter (also based on the number of inspection channels) to calculate the product. The defect type with the largest product is selected as the defect type. The defect type selected in this way is the defect type most likely to exist in the defect based on the results from each channel. If the products of two defect types are the same, the defect type that is determined to be more frequent (more common) is selected. If the frequencies are also the same, then for example, two defect types can be specified as the result.

[0026] Then, the defect type determined for each defect on the object surface is output at the interface of the data processing unit.

[0027] The method according to the invention provides a simple, rapid, and cost-effective inspection of the surface of a coated object, and is characterized by very good results.

[0028] The first neural network algorithm is trained as described in detail below. Therefore, it can be adapted to the specific inspection task / object to be inspected, since data specific to the corresponding object / inspection task can be used for training. This ensures high accuracy during inspection.

[0029] Based on the defect types output at the data processing unit's interface (e.g., a list of all defects of the corresponding defect type for a given object), an automatic assessment of the quality of an object or its surface coating can be performed. The data can also be output on a corresponding display device, allowing quality control personnel to visually evaluate it. Based on this data provided at the interface, if an object does not meet quality requirements, it can also be automatically removed from the production process. This might be the case, for example, if the number of defects exceeds one or more specific defect types.

[0030] In one embodiment, the location (position) of each detected defect on the object surface in a predefined coordinate system is also determined. This can be accomplished, for example, by means of coordinate transformation based on the object's CAD model, calibration of the image recording unit, and / or optionally based on the relative movement speed or corresponding position of the object and the image recording unit. The predefined coordinate system can, for example, represent the object's coordinate system. In this way, and optionally by means of comparison with the object's CAD model, defects can then be located very precisely on the object surface. Based on this information, a marking device downstream of the inspection apparatus can mark the defect, for example, by applying (e.g., spraying) water-soluble pigment around the defect on the object's surface. Alternatively or additionally, knowing the location of the defect can help control the apparatus used to remove it.

[0031] In one embodiment, when the object surface is illuminated with a predetermined pattern specific to the inspection channel, image regions of the inspection channel are generated based on multiple records of different areas of the object surface. The images of the corresponding inspection channels are generated directly from records of the object surface transmitted by the image recording unit, which can be preprocessed, wherein all records of the inspection channel are generated when illuminated with the same predetermined pattern. For example, due to relative movement between the object and the image recording unit, different images may contain different areas of the object surface. Optionally, the records are processed by the image recording unit and / or the data processing unit, for example, performing digitization, contrast modification, and / or preprocessing, or subtracting image data generated by low-pass filtering of the corresponding record from each record.

[0032] Images of the inspection channels are generated based on the transmitted recorded data. For further defect analysis, only areas on the stripes or only the transition area between two adjacent stripes are used. Despite identical lighting and recording conditions, corresponding images are assigned to two different inspection channels because the image data is processed differently.

[0033] In one embodiment, an image of the inspection channel is generated based on a first plurality of records of different regions of the object surface when the object surface is illuminated with a first predetermined pattern and at least a second plurality of records of different regions of the object surface when the object surface is illuminated with at least a second predetermined pattern, and based on pixel-by-pixel calculations of a summed image, a contrast image, or a phase image based on records of regions at the same location.

[0034] For example, a record of an object surface can be generated using two or four different patterns, where each pattern includes a sinusoidal fringe intensity distribution. For instance, two patterns might be shifted by half a sine cycle, and four patterns by a quarter sine cycle. The intensity of the pixels in the record for the two patterns is specified. I A and I BSimultaneously specify the intensity of the recorded pixels for the four patterns. I 1 、I 2 、I 3 and I 4 Generate a summation image, contrast image, or phase image using the following calculation rules, where the summation image ( I 0 Each pixel of the image, compared to the image ( K ), phase image ( ), or curvature image ( C The calculation is performed pixel by pixel based on records taken at the same location:

[0035]

[0036]

[0037] The curvature image (C) can be calculated from the phase image by stabilizing the phase image and calculating the mathematical derivatives of the directions of the stripes perpendicular to the illumination pattern used (transferred to the recorded and resulting images, respectively). The summation image, contrast image, phase image, or curvature image of the multiple calculations for the object each represent an inspection channel (i.e., an inspection channel of the summation image, an inspection channel of the contrast image, etc.). These inspection channel images can also be used in a manner similar to the inspection channel images described above to determine the inspection channel-specific defect type and associated inspection channel-specific evaluation parameters, respectively, by means of a correspondingly trained first NN algorithm.

[0038] In one embodiment, by means of a second NN algorithm, a binary defect representation for a particular defect is determined from an image region of a corresponding defect assigned to at least one designated inspection channel among a first inspection channel, a second inspection channel, and optional additional inspection channels. This binary defect representation contains the shape and size of the corresponding defect and can be used to determine the shape and size accordingly.

[0039] The second NN algorithm used represents a neural network-based assignment. The second NN algorithm uses a neural network to assign the binary representation of a defect to all image regions across all inspection channels associated with the corresponding defect. In one embodiment, the second neural network forms a Convolutional Neural Network (CNN) that generates a binary image from the image regions, where pixels belonging to a defect appear in a first color (e.g., white), and pixels not belonging to a defect appear in a second color (e.g., black). Using the second NN algorithm, for each pixel in the image region across all inspection channels of the corresponding defect, the probability of whether that pixel belongs to a defect is determined. For example, for pixels with a probability greater than or equal to 0.5 belonging to a defect, white (or 1) is assigned, while for all other pixels, black (or 0) is assigned. For example, a neural network based on the U-Net architecture is used for this purpose, and it has proven well-suited for this image-to-image transformation problem. The U-Net model consists of two main parts: a so-called "encoder" and a "decoder." The encoder consists of several convolutional layers that progressively compress the image region into a compact representation (feature map). The decoder is a structure symmetrical to the encoder. It consists of a series of upward convolutional blocks that cause the compressed feature map to return to the original resolution of the input image. Furthermore, there are connections between the encoder and decoder, often referred to as "skipped connections." At the end of the UNet model, a softmax layer is used to calculate the probability of whether a corresponding pixel belongs to the corresponding defect type.

[0040] In one embodiment, a channel-specific neural network (NN) algorithm is trained using a real image of the defect, which has previously been evaluated by an expert for various defect types and assigned to a specific defect type. For a second NN algorithm, for example, the defect is delineated as a polygon in a region of the training image by an expert. This generates a mask comprising pixels inside the polygon that belong to the defect and therefore have a value of, for example, 1, and pixels outside the polygon that have a value of, for example, 0. Multiple such masks are used to train the NN algorithm, for example, in the manner described above with respect to ResNet18.

[0041] In one embodiment, as described above or below, an adaptive learning rate optimizer is used to perform training of the first NN algorithm and / or the Dice coefficient loss function is used to perform training of the second NN algorithm.

[0042] The aforementioned objective is also achieved by an apparatus for inspecting a coating on a surface of an object, wherein the apparatus includes an image recording unit (e.g., a camera) and a data processing unit, wherein the image recording unit and optionally the data processing unit are configured to generate a plurality of first images of the object surface associated with a first optical inspection channel, a plurality of second images of the object surface associated with a second optical inspection channel, and optionally generate a plurality of additional images of the object surface associated with at least one additional optical inspection channel. The data processing unit is configured to automatically analyze multiple images of each inspection channel to determine the presence of defects, and to perform the following steps for each inspection channel and each detected defect: - The image regions containing the location of the corresponding detected defects are assigned to multiple images of the corresponding inspection channels. - For the location of at least one image region of the detected defect, those inspection channels can be assigned to, determining an inspection channel-specific defect type and associated inspection channel-specific evaluation parameters for the determined inspection channel-specific defect type, wherein this determination is made by means of a first NN algorithm trained for the corresponding inspection channel based on the image region of the corresponding inspection channel assigned to the corresponding defect. The data processing unit is configured such that for each detected defect on the object surface, a defect type is selected by means of soft voting, based on the inspection channel-specific defect type determined for the corresponding defect and the associated inspection channel-specific evaluation parameters, and the selected defect type is output at the interface of the data processing unit for each detected defect on the object surface.

[0043] The above apparatus has the advantages described above with respect to the method and corresponding embodiments. Therefore, refer to the above and below explanations regarding the method and apparatus.

[0044] In one embodiment of the apparatus, the data processing unit is configured to further determine the location of each detected defect on the object surface in a predetermined coordinate system.

[0045] In one embodiment of the apparatus, the data processing unit is configured to generate an image of the inspection channel based on multiple records made by the image recording unit of different areas of the object surface when the object surface is illuminated by the lighting device in a pattern specific to the predetermined inspection channel.

[0046] In one embodiment of the apparatus, the data processing unit is configured to generate an image of the inspection channel based on a first plurality of records made by the image recording unit of different regions of the object surface when the object surface is illuminated with a first predetermined pattern and at least a second plurality of records made by the image recording unit of different regions of the object surface when the object surface is illuminated with at least a second predetermined pattern, and pixel-by-pixel calculations based on summed images, contrast images, phase images or curvature images of records from regions at the same location.

[0047] In one embodiment of the apparatus, the data processing unit is configured such that, for a specific defect, a binary defect representation is determined from an image region of the corresponding defect assigned to at least one predetermined inspection channel among a first inspection channel, a second inspection channel, and optional additional inspection channels, by means of a second NN algorithm. The binary defect representation includes the shape and size of the corresponding defect.

[0048] In one embodiment, similar to the process described above, real image regions are used, and their binary representations are provided as corresponding annotations. During training, annotated images are first provided. The data is divided into training, validation, and test sets, for example, to train a second neural network algorithm (e.g., a UNet model) and evaluate its performance. The UNet model is then configured and created, for example, to perform training. This model consists of an encoder and decoder architecture with a softmax output at the end, which calculates the probability of assigning a corresponding pixel to a defect. For example, the Dice coefficient loss is used as the loss function to minimize the difference between the predicted and actual segmentations of the training data. The model is then trained with the training set. For this purpose, images and corresponding annotated data are provided to the model, and adjustments are made to improve performance. Training can be performed over several epochs, with the model weights updated after each epoch. Subsequently, a validation curve is created that monitors the loss function with respect to the training and validation sets relative to the corresponding epoch, and if the validation loss value decreases, the training weights associated with that epoch are preserved. Here, the validation set is image region data not included in the training set. The same applies to the test set. At the end of training, the model is evaluated against a test set to assess its performance on new data.

[0049] Further advantages, features, and possible applications of the invention will also be apparent from the following description of embodiments and accompanying drawings. All features described and / or illustrated individually or in any combination form the subject matter of the invention, even independently of their summary in the claims or their references.

[0050] This is illustrated schematically: Figure 1 This is a side view of an embodiment of the device according to the present invention.

[0051] Figure 2 This is an exemplary representation of the inspection channels and the image assigned to each inspection channel, and Figure 3 This is a flowchart that is part of the method according to the present invention.

[0052] Figure 1 An embodiment of the device according to the invention is shown, which is used to inspect an object, namely a car chassis 3 with a painted coating. The chassis 3 moves across the device on a conveyor belt (not shown), i.e., on... Figure 1 The feed direction is indicated by the middle arrow 4.

[0053] The inspection device includes an illumination unit 5 attached to a robotic arm 6, wherein the robotic arm 6 enables precise adjustment of the position of the illumination unit relative to the chassis 3 or relative to other objects to be inspected.

[0054] The lighting unit 5 includes multiple LEDs, and lighting patterns are generated by switching these LEDs on and off. For example, as... Figure 1 As shown, a stripe pattern 7 extending parallel to the feed direction can be generated on the surface of the chassis 3. Other patterns can also be achieved, such as offset stripe patterns parallel to the feed direction, stripe patterns perpendicular to the feed direction, or full illumination (all LEDs of the illumination unit 5 are turned on). Light reflected from the surface of the chassis 3 is recorded from different perspectives by matrix cameras 10 and 11. Multiple records of the chassis 3 are generated for all illumination configurations specified below during the relative movement of the chassis 3 and cameras 10 and 11. These multiple records are digitized and transmitted to the data processing unit (computer) 13. Each matrix camera 10 and 11 generates a record of a different area of ​​the chassis. The data processing unit 13 is electrically connected to the matrix cameras 10 and 11. It also controls the readout time of the matrix cameras 10 and 11. Furthermore, the data processing unit 13 is connected to the illumination unit 5 to control the illumination pattern generated by the illumination unit 5. Of course, the readout of the matrix cameras 10 and 11 and the setting of the corresponding illumination pattern are synchronized in time.

[0055] Examples of configurations generated using illumination from illumination unit 5 and corresponding recordings from matrix cameras 10 and 11 include the following: Lighting 1: All LEDs in lighting unit 5 are turned on; Illumination 2: The illumination pattern includes rectangular stripes perpendicular to the feed direction; Illumination 3: The illumination pattern includes rectangular stripes parallel to the feed direction (see illumination pattern 7); Lighting 4: The same lighting pattern as in Lighting 3, but offset by 1 / 4 of the cycle; Lighting 5: As in Lighting 2; Lighting 6: The lighting pattern is the same as in Lighting 3, but offset by 1 / 2 cycle; Lighting 7: The lighting pattern is the same as in Lighting 3, but offset by 3 / 4 of the cycle; Illumination 8: The illumination pattern includes sinusoidal stripes parallel to the feed direction; Lighting 9: The lighting pattern is the same as in Lighting 8, but offset by 1 / 2 cycle.

[0056] Lighting configurations 1 through 9 are set up one after another, and recordings are generated using matrix cameras 10 and 11. Chassis 3 moves / remains stationary during lighting and recording generation.

[0057] Multiple records from various sections of chassis 3 are transmitted to data processing unit 13. At data processing unit 13, images of multiple inspection channels are generated based on these records. The images of the following inspection channels (abbreviated as K) are generated in this embodiment for further inspection: K1: Record of illumination 1, where the corresponding low-pass filtered image is subtracted from all records; K2: Recording of illumination 2, which includes only the signal along the stripes for processing; K3: Recording of illumination 3, which includes only the signal along the stripes for processing; K4: Recording of illumination 4, which includes only the signal along the stripes for processing; K5: Recording of illumination 5, which includes only the signal along the stripes for processing; K6: Recording of illumination 6, which includes only the signal along the stripes for processing; K7: Recording of illumination 7, in which, during processing, only the signal along the stripes is included; K8: Recording of illumination 2, in which, during processing, only the signal of the transition area between two stripes is included; K9: Recording of illumination 3, in which, during processing, only the signal of the transition area between two stripes is included; K10: Recording of illumination 4, in which, during processing, only the signal of the transition area between two stripes is included; K11: Recording of illumination 5, in which, during processing, only the signal of the transition area between two stripes is included; K12: Recording of illumination 6, in which, during processing, only the signal of the transition area between two stripes is included; K13: Recording of illumination 7, in which, during processing, only the signal of the transition area between two stripes is included; K14: Summing the images recorded from illuminations 8 and 9 using equation (1) shown above.

[0058] All images are then examined to determine if they contain defects. For each inspection channel, defects are detected in different ways: for example, for K1, the difference calculated for the low-pass filtered image is binarized using an adaptive block-based thresholding operation, thereby determining whether a defect exists in the corresponding image and, if so, where it is located. For all other inspection channels, an overlay image is first calculated, containing information about the location of dark stripes, bright stripes, and the transition regions between them. Then, gradients associated with the captured images are calculated. The overlay image (mask) is used to correct the gradient image (e.g., by addition). For inspection channels K2 through K7, correction is performed in the transition regions between stripes; for inspection channels K8 through K13, correction is performed in the regions of the bright or dark stripes themselves. After correction, threshold-based binarization is also performed, and the binarized image is used to examine whether defects exist in the corresponding image (e.g., by considering the size of the highlighted regions in the binarized image).

[0059] If a defect is detected, its local location is determined, and for all channels K1 to K14, image regions that also contain the defect are identified. For example, these could have a size of 80 pixels × 80 pixels. This determination is based on a local allocation of the image region to the defect location. For example, in the object's coordinate system, the defect is located, and the associated image region is determined.

[0060] The result is a series of channels for which at least one image region of the detected defect can be identified. Figure 2 The diagram illustrates this point regarding channels K1 through K5. Further explanation of the method applies accordingly to the other channels K6 through K14.

[0061] Figure 2 Channels K1 to K5 are shown, wherein five image regions K1.1 to K1.5 containing detected defects can be assigned to channel K1. Correspondingly, channel K2 contains two image regions K2.1 and K2.2, channel K3 contains four image regions K3.1 to K3.4, channel K4 contains no image regions, and channel K5 contains three image regions K5.1 to K5.3.

[0062] Since the fourth channel does not contain any image region that can be assigned to detect defects, therefore... Figure 3 The fourth channel is not considered in the further process shown.

[0063] For each inspection channel (in this case K1, K2, K3, and K5) that an image region can be assigned to relative to a detected defect, the defect type and evaluation parameters are now determined using a first NN algorithm trained for the corresponding channel in the form of ResNet (e.g., ResNet18 as described above). For channel K1, the defect type is K1.8, and the evaluation parameter is K1.9. Similarly, for channel K2, the defect type is K2.8, ​​and the evaluation parameter is K2.9; for channel K3, the defect type is K3.8, and the evaluation parameter is K3.9; and for channel K5, the defect type is K5.8, and the evaluation parameter is K5.9. For example, the values ​​included in the table below can be determined for two examples of values, where there are essentially three defect types A, B, and C (e.g., inclusions, pits, or protrusions (dents)). In both examples, defect type A is selected for the defect and output at the interface of the data processing unit.

[0064]

[0065] Then, the channel-specific defect types K1.8, K2.8, ​​K3.8, and K5.8, along with their associated channel-specific evaluation parameters K1.9, K2.9, K3.9, and K5.9, are fed into the soft voting module 28 of the method. The soft voting performed by this module applies the method described above (the average value is determined as the arithmetic mean), and in both examples, defect type A is selected from the channel-specific defect types K1.8, K2.8, ​​K3.8, and K5.8 because the calculated value of the soft vote is the largest for this defect type (see the table above). Subsequently, the defect type (defect type A) determined in this way for the corresponding defect is provided at the interface of the data processing unit 13 (see...). Figure 3 Step 30 in the process.

[0066] If another defect is detected, repeat the steps for the image region belonging to that defect and the subsequent steps described above.

[0067] In addition, coordinate transformation can be used to convert the location (position) of the corresponding defect from the coordinate system of the inspection device to the position in the coordinate system of the chassis 3.

[0068] Using the second neural network algorithm, a binary defect representation can be determined from all image regions K1.1 to K5.3 that have been assigned to all channels K1 to K5 of the defect, such as... Figure 4 As shown. Here, the pixels in region F1 are white—they contain the shape and size of the corresponding defect. The area surrounding region F1 is black (shown here as stripes). The striped area is a different area from the defect.

[0069] Based on the determined region F1, the shape and size of the defect can be easily calculated.

[0070] By applying the second NN algorithm, binary defect representations can be determined for all detected defects in a similar manner.

[0071] The location of all detected defects and their binary defect representations can also be obtained at the interface of the data processing unit 13.

Claims

1. A method for inspecting a coating on the surface of an object, wherein, Multiple first images of the object surface are generated with respect to the first optical inspection channel (K1), multiple second images of the object surface are generated with respect to the second optical inspection channel (K2), and optionally, multiple additional images of the object surface are generated with respect to at least one additional optical inspection channel (K3 to K5). Specifically, to determine whether defects exist, multiple images from each inspection channel are automatically analyzed. For each inspection channel and each detected defect, the following steps are performed: --The image regions (K1.1 to K1.5, K2.1 to K2.2, K3.1 to K3.4, K5.1 to K5.3) that contain the location of the corresponding detected defects in the multiple images of the corresponding inspection channels. --For at least one image region that can be assigned to the location of the detected defect, determine the inspection channels specific to the inspection channel (K1.8, K2.8, ​​K3.8, K5.8) and the associated inspection channel specific evaluation parameters (K1.9, K2.9, K3.9, K5.9) for the determined inspection channel specific defect type, wherein the determination is based on the image region assigned to the corresponding inspection channel for the corresponding defect, made by means of a first NN algorithm trained for the corresponding inspection channel. Subsequently, for each detected defect on the object surface, a defect type is selected by means of soft voting, based on the inspection channel-specific defect type of the corresponding inspection channel determined for the corresponding defect and the associated inspection channel-specific evaluation parameters, wherein the defect type thus selected for each detected defect on the object surface is output at the interface of the data processing device.

2. The method according to claim 1, wherein, In addition, the location of each detected defect on the surface of the object is determined in a predetermined coordinate system.

3. The method according to any one of the preceding claims, wherein, An image of the inspection channel is generated based on multiple records of different areas of the object surface when the object surface is illuminated with a pattern specific to the predetermined inspection channel.

4. The method according to any one of the preceding claims, wherein, An image of the inspection channel is generated based on a first plurality of records of different regions of the object surface when the object surface is illuminated with a first predetermined pattern and at least a second plurality of records of different regions of the object surface when the object surface is illuminated with at least a second predetermined pattern, and pixel-wise calculations based on summed images, contrast images, phase images or curvature images of records from regions at the same location.

5. The method according to any one of the preceding claims, wherein, For a specific defect, a binary defect representation is determined by means of a second NN algorithm from the image region of the corresponding defect assigned to at least one predetermined inspection channel among the first inspection channel, the second inspection channel, and the optional additional inspection channels. The binary defect representation includes the shape and size of the corresponding defect.

6. The method according to any one of the preceding claims, wherein, The training of the first NN algorithm is performed using an adaptive learning rate optimizer and / or the training of the second NN algorithm is performed using the Dice coefficient loss function.

7. An apparatus for inspecting a coating on the surface of an object, wherein, The apparatus includes an image recording unit and a data processing unit, wherein the image recording unit and optionally the data processing unit are configured to generate a plurality of first images of the object surface with respect to a first optical inspection channel (K1), a plurality of second images of the object surface with respect to a second optical inspection channel (K2), and optionally generate a plurality of additional images of the object surface with respect to at least one additional optical inspection channel (K3 to K5). The data processing unit is configured to automatically analyze multiple images of each inspection channel to determine if a defect exists, and to perform the following steps for each inspection channel and each detected defect: --The image regions (K1.1 to K1.5, K2.1 to K2.2, K3.1 to K3.4, K5.1 to K5.3) that contain the location of the corresponding detected defects in the multiple images of the corresponding inspection channels. --For at least one image region that can be assigned to the location of the detected defect, determine the inspection channels specific to the inspection channel (K1.8, K2.8, ​​K3.8, K5.8) and the associated inspection channel specific evaluation parameters (K1.9, K2.9, K3.9, K5.9) for the determined inspection channel specific defect type, wherein the determination is based on the image region assigned to the corresponding inspection channel for the corresponding defect, made by means of a first NN algorithm trained for the corresponding inspection channel. The data processing unit is configured such that, subsequently, for each detected defect on the object surface, a defect type is selected by means of soft voting, based on the inspection channel-specific defect type determined for the corresponding defect and the associated inspection channel-specific evaluation parameters, and the selected defect type is output at the interface of the data processing unit for each detected defect on the object surface.

8. The apparatus according to claim 7, wherein, The data processing unit is configured to further determine the location of each detected defect on the surface of the object in a predetermined coordinate system.

9. The apparatus according to any one of claims 7 to 8, wherein, The data processing unit is configured to generate an image of the inspection channel based on multiple records made by the image recording unit of different areas of the object surface when the object surface is illuminated with a predetermined inspection channel-specific pattern by means of an illumination device.

10. The apparatus according to any one of claims 7 to 9, wherein, The data processing unit is configured to generate an image of the inspection channel based on a first plurality of records made by the image recording unit of different regions of the object surface when the object surface is illuminated with a first predetermined pattern, and at least a second plurality of records made by the image recording unit of different regions of the object surface when the object surface is illuminated with at least a second predetermined pattern, and pixel-by-pixel calculations based on summed images, contrast images, phase images, or curvature images of records from regions at the same location.

11. The apparatus according to any one of claims 7 to 10, wherein, The data processing unit is configured to, for a specific defect, determine a binary defect representation from an image region of the corresponding defect assigned to at least one of the first inspection channel, the second inspection channel, and optionally the other inspection channel, using a second NN algorithm. The binary defect representation includes the shape and size of the corresponding defect.

12. The apparatus according to any one of claims 7 to 11, wherein, The training of the first NN algorithm is performed using an adaptive learning rate optimizer and / or the training of the second NN algorithm is performed using the Dice coefficient loss function.

Images