Deep learning-supporting automated detection and measurement system for corrosion resistance properties of coatings

Abstract

A system for assessing corrosion resistance properties of a coating when applied on a substrate susceptible to corrosion, the system comprising: (i) an imaging unit configured to capture a plurality of grayscale images of the coating; (ii) a computational image processing unit configured to receive and reconstruct the captured plurality of grayscale images, and output a reconstructed topography image and a reconstructed color image; (iii) a data pre-processing unit configured to receive and combine the reconstructed topography image and the reconstructed color image, and output an image comprising high-dimensional data, the high-dimensional data comprising one-dimensional height data and at least three-dimensional color data; (iv) a corrosion detection unit configured to receive the high dimensional data and identify a corrosion feature, and output at least a location, classification, and region of the corrosion feature; wherein the corrosion detection unit comprises a corrosion detection neural network having an input layer and an output layer, the input data of the input layer comprising the high-dimensional data and the output data from the output layer comprising at least the location, classification and region of the corrosion feature; and (v) a computer-aided data analysis unit configured to receive and analyze the data containing at least the location, classification, and region of the corrosion feature, and output a predicted rating of corrosion severity; a computer-implemented method of assessing the corrosion resistance properties of a coating when applied on a substrate that is susceptible to corrosion; a computing device having a computed image processing unit, a data preprocessing unit, a corrosion detection unit, and a computer-aided data analysis unit disposed thereon; and a method for training a neural network for detecting corrosion.

Description

Technical Field

[0001] The present invention relates to a deep learning enabled automated detection and measurement system for corrosion failures, particularly suitable for automated detection and quantitative assessment of the anti-corrosion properties of protective coatings. Background Art

[0002] Steel corrosion causes huge losses to the global economy and is a basic infrastructure challenge. Protective coatings are widely used to prevent corrosion. Many protective coatings are provided to cover different anti-corrosion requirements from light industrial use to heavy-duty applications. This poses a challenge on how to quantify the anti-corrosion properties of protective coatings. The anti-corrosion properties of coatings are usually evaluated by first exposing the coated panels to salt spray to simulate the corrosive environment according to the ASTM B117-11 salt spray test, and then visually inspecting and rating the corrosion failures by operators. For example, rust and blister defects are rated by comparison with the standard patterns defined in ASTM D610-08 and ASTMD714-02, respectively, and creep defects are measured in millimeters according to ASTMD1654-08. In addition, such visual inspection and rating work performed by humans is laborious and time-consuming, which is also often subjective. The disturbance of the flow of the salt solution used in the salt spray test (also known as "rust bleed" or staining) and the insensitivity of the human eye to capture tiny bubbles and identify their distribution also make the test results of corrosion resistance properties prone to error.

[0003] Some imaging and machine vision systems and methods have been proposed for detecting corrosion defects using color digital camera imaging. However, no system or method relates to an autonomous and standardized process for detecting corrosion signatures of a coating when applied on a metal substrate subjected to a corrosive environment (such as salt spray) and rating the anti-corrosion properties of the coating. Therefore, it is desirable to provide a system and method for automated detection and quantitative assessment of the anti-corrosion properties of a coating when applied on a corrosion-susceptible substrate using deep learning. Summary of the invention

[0004] The present invention provides a novel system and method for quantifying the corrosion failure of a coating when applied on a corrosion-prone substrate (hereinafter "coated metal panel" or "coated panel") in a standardized ASTM B117-11 salt spray test by means of intelligent detection and automatic imaging analysis using a novel combination of computational imaging techniques, deep learning neural networks, and computer-assisted data analysis. The system of the present invention also implements a standardized method for detecting and quantifying the anti-corrosion properties of a coating on a coated metal panel. The predictive rating of the corrosion severity of the present invention has been verified by correlation with the results of human ratings, which implements an autonomous process for predicting an accurate and reliable rating of the corrosion severity compared to the actual results obtained by visual inspection and rating by humans. The method of the present invention can effectively mitigate the interference of rust bleeding caused by the flow of salt solution on the surface. The system of the present invention can also distinguish between subtle differences between different corrosion features (such as rust and blistering) that humans cannot distinguish through visual inspection. The method of the present invention can also greatly improve the consistency and accuracy of the test and reduce labor costs.

[0005] In a first aspect, the present invention is a system for evaluating the anti-corrosion properties of a coating when applied to a corrosion-susceptible substrate, the system comprising:

[0006] (i) an imaging unit configured to capture a plurality of grayscale images of the coating;

[0007] (ii) a computational image processing unit configured to receive and reconstruct the captured plurality of grayscale images, and output a reconstructed topographic image and a reconstructed color image;

[0008] (iii) a data preprocessing unit configured to receive and combine the reconstructed topographic image and the reconstructed color image, and output an image containing high-dimensional data, the high-dimensional data including one-dimensional height data and at least three-dimensional color data;

[0009] (iv) a corrosion detection unit configured to receive the high-dimensional data and identify corrosion features, and output at least the location, classification, and area of ​​the corrosion features; wherein the corrosion detection unit comprises a corrosion detection neural network having an input layer and an output layer, wherein input data of the input layer comprises the high-dimensional data, and output data from the output layer comprises at least the location, classification, and area of ​​the corrosion features; and

[0010] (v) a computer-assisted data analysis unit configured to receive and analyze the data containing at least the location, classification and area of ​​the corrosion feature and output a predicted rating of corrosion severity.

[0011] In a second aspect, the present invention is a computer-implemented method for evaluating the anti-corrosion properties of a coating when applied on a corrosion-susceptible substrate. The method comprises:

[0012] receiving a plurality of grayscale images of the coating;

[0013] reconstructing the plurality of grayscale images by computational image processing, and outputting a reconstructed topographic image and a reconstructed color image;

[0014] combining the reconstructed topographic image and the reconstructed color image into an image comprising high-dimensional data, the high-dimensional data including one-dimensional height data and at least three-dimensional color data;

[0015] inputting the high-dimensional data into a corrosion detection unit configured to identify corrosion features and output at least a location, a classification, and an area of ​​the corrosion features, wherein the corrosion detection unit comprises a corrosion detection neural network having an input layer for receiving input data and an output layer for outputting output data, wherein the input data comprises the high-dimensional data, and the output data comprises at least a location, a classification, and an area of ​​the corrosion features; and

[0016] The data including at least the location, classification and area of ​​the corrosion feature is received and analyzed by computer-assisted data analysis, and a predicted rating of corrosion severity is output.

[0017] In a third aspect, the present invention is a computing device having a computing image processing unit, a data pre-processing unit, a corrosion detection unit, and a computer-aided data analysis unit deployed thereon;

[0018] wherein the computational image processing unit is configured to receive and reconstruct a plurality of captured grayscale images, and output a reconstructed topographic image and a reconstructed color image;

[0019] wherein the data preprocessing unit is configured to receive and combine the reconstructed topographic image and the reconstructed color image, and output an image containing high-dimensional data, the high-dimensional data including one-dimensional height data and at least three-dimensional color data;

[0020] wherein the corrosion detection unit is configured to receive the high-dimensional data and identify corrosion features, and output at least the location, classification, and region of the corrosion features; wherein the corrosion detection unit comprises a corrosion detection neural network having an input layer and an output layer, wherein input data of the input layer comprises the high-dimensional data, and output data from the output layer comprises at least the location, classification, and region of the corrosion features; and

[0021] Wherein the computer-aided data analysis unit is configured to receive and analyze the data containing at least the location, classification and area of ​​the corrosion feature and output a predicted rating of corrosion severity.

[0022] In a fourth aspect, the present invention is a method for training a neural network for detecting corrosion. The method comprises:

[0023] collecting a plurality of grayscale images of a collection of the coatings when applied on a corrosion-susceptible substrate;

[0024] reconstructing the captured grayscale image for each coating layer by computational processing, and outputting a reconstructed topographic image and a reconstructed color image for each coating layer;

[0025] combining the reconstructed topographic image and the reconstructed color image for each coating layer, and outputting an image containing high-dimensional data for each coating layer, the high-dimensional data including one-dimensional height data and at least three-dimensional color data;

[0026] obtaining at least the actual location, classification and area of ​​corrosion features identified by quantified rusting extent, blistering extent and maximum creep for each coating layer;

[0027] creating a training data set comprising the set of high-dimensional data and a set of data comprising at least the actual locations, classifications, and regions of corrosion signatures of the set of coatings; and

[0028] The neural network is trained using the training data set; thereby obtaining a trained neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of a system for evaluating the anti-corrosion properties of a coating when applied to a corrosion-susceptible substrate is illustrated.

[0030] Figure 2 A flow chart illustrating a method for evaluating the anti-corrosion properties of a coating on a corrosion-susceptible substrate according to one example of the present invention.

[0031] Figure 3 A schematic diagram of a multi-angle illumination imaging device according to an example of the present invention is illustrated.

[0032] Figure 4 A schematic diagram of a multi-spectral illumination imaging device according to an example of the present invention is illustrated.

[0033] Figure 5 is a schematic illustration of the principle of the shape from shading algorithm.

[0034] Figure 6A schematic flow chart for obtaining a topographic image from a multi-angle illumination image of a coated metal panel according to an example of the present invention is illustrated.

[0035] Figure 7 Schematic diagram illustrating color image reconstruction from multiple single-wavelength illumination images.

[0036] Figure 8 A computational color image reconstructed from an image of a coated metal panel captured by a multi-spectral illumination imaging device according to one example of the present invention is illustrated.

[0037] Fig. 9 A schematic diagram illustrating high-dimensional data combined from the outputs of a computational image processing unit according to an example of the present invention is illustrated.

[0038] Fig.10 A schematic diagram illustrating a typical U-Net neural network.

[0039] Fig.11 A model training process for a deep learning neural network according to an example of the present invention is illustrated.

[0040] Fig.12 A schematic flow chart of a one-step deep learning neural network for corrosion identification and classification according to an example of the present invention is illustrated.

[0041] Fig.13 A schematic flow chart of a two-step deep learning neural network for corrosion identification and classification according to an example of the present invention is illustrated.

[0042] Fig.14 An example of creep calculation according to one example of the present invention is illustrated.

[0043] Fig.15 An exemplary process for rating rust on a coated metal panel according to one example of the present invention is illustrated.

[0044] Fig.16 An exemplary process for rating rust on a coated metal panel according to one example of the present invention is illustrated.

[0045] Fig.17 The final output image of the system of the present invention according to one example of the present invention is illustrated.

[0046] Fig.18 A schematic diagram of a cloud-based server cluster according to an example of the present invention is illustrated.

[0047] Fig.19 Correlation results for white painted metal panels evaluated by manual evaluation and by the system of the present invention according to Example 1 are illustrated.

[0048] Fig. 20 Correlation results for grey painted metal panels evaluated by manual evaluation and by the system of the present invention according to Example 2 are illustrated. DETAILED DESCRIPTION

[0049] When a test method number is not used to indicate a date, the test method refers to the most current test method as of the priority date of this document. Reference to a test method includes reference to both the testing association and the test method number. The following test method abbreviations and designations apply herein: ASTM refers to ASTM International method.

[0050] "Neural network" refers to an artificial neural network that is made of artificial neurons or nodes and is used to solve artificial intelligence (AI) problems. The connections of biological neurons are modeled in artificial neural networks as weights between nodes. Positive weights reflect excitatory connections, while negative values ​​mean inhibitory connections. All inputs are modified by the weights and summed. This activity is called a linear combination. Finally, the activation function controls the amplitude of the output.

[0051] "Machine learning" refers to a collection of methods that 'learn' from data to improve performance on a specific task. Machine learning algorithms build models based on historical data (also known as training data) to make predictions as model outputs.

[0052] "Deep learning" is a type of machine learning in which a model learns to perform classification tasks directly from images, text, or sound. Deep learning is often implemented using a neural network architecture. The term "depth" in deep learning refers to the number of layers in the network. The more layers, the deeper the network. Deep learning can include three or more layers or even hundreds of layers in a neural network.

[0053] "Grayscale" in digital images means that the value of each pixel only represents the intensity information of light. Grayscale images usually only show the darkest black to the brightest white. In other words, the image contains only black, white and gray, where gray has multiple levels. In a grayscale image, each pixel has a value between 0 and 255, where 0 corresponds to "black" and 255 corresponds to "white". The values ​​between 0 and 255 are varying shades of gray, where the values ​​closer to 0 are darker, and the values ​​closer to 255 are brighter.

[0054] "Image segmentation" refers to techniques used in digital image processing and analysis to separate an image into parts or regions, usually based on the characteristics of the pixels in the image.

[0055] "Coating" (interchangeable with "coating film") herein means a film or coating film formed by applying a coating composition to a substrate and drying or allowing the coating composition to dry.

[0056] "Coated metal panel" (interchangeable with "coated panel") refers herein to a corrosion-susceptible substrate having a coating applied thereon. "Corrosion-susceptible substrate" refers to a substrate susceptible to corrosion, such as a metal substrate, desirably a steel substrate.

[0057] An image of a coating, coating sample, or coated panel refers to an image of the surface of a coating, coating sample, or coated panel to which the coating has been applied.

[0058] The "anti-corrosion properties" of a coating are often characterized by one or more corrosion characteristics.

[0059] "Corrosion signatures" (interchangeable with "corrosion defects") of a coating refer to the characteristics of defects in the coating caused by corrosion of the corrosion-susceptible substrate to which the coating is applied. Corrosion signatures may include characteristics that can be used to rate the severity of corrosion, including the location, type (i.e., classification), size (e.g., length and / or width), number, density (e.g., distribution) of the defects. Classifications of corrosion signatures or corrosion defects herein include rust defects, blister defects, creep, or combinations thereof.

[0060] "Corrosion severity ratings" may include severity of rust, severity of blistering, maximum creep, or a combination thereof; in particular, ratings defined in the following ASTM standards for evaluating the severity of corrosion of a coating when applied to a substrate susceptible to corrosion. For example, the severity of rust is based on a quantified degree of rust, such as according to ASTM D610-08 (Standard Practice for Evaluating the Degree of Rust on Painted Steel Surfaces), which may include a rust grade identified by the size of the rusted area (e.g., as a percentage of the surface area rusted) and the type of rust distribution on the coating. The severity of blistering is based on a quantified degree of blistering, such as according to ASTM D714-02 (Standard Test Method for Evaluating the Degree of Blistering of Paints), which may include the size and frequency (e.g., density) of blisters on the coating. Maximum creep refers to, for example, the maximum corrosion width in millimeters (mm) from a scribe line on a coated metal panel according to ASTM D1654-08 (Standard Test Method for Evaluating Painted or Coated Test Specimens Subjected to Corrosive Environments).

[0061] Through a combination of multiple hardware and algorithms, the present invention can achieve a fully automated evaluation of the anti-corrosion properties of a coating when applied to a corrosion-prone substrate (hereinafter referred to as a "coated metal panel") according to ASTM standards. The present invention can greatly improve the shortcomings associated with manual evaluation, such as accuracy and time-consuming issues. After exposure to a corrosive environment such as salt spray, the surface of the coated metal panel has a three-dimensional surface structure but almost no contrast and is visually noisy. In particular, the initiation of corrosion on the coated metal panel is difficult to detect by visual inspection. Compared with conventional machine vision systems for corrosion detection that directly use images acquired by conventional digital color cameras as input for machine learning, the present invention accurately reconstructs the surface defects of the coating caused by corrosion by using computational imaging processing of grayscale images, and can obtain more accurate surface height maps and color information with higher discrimination during the data collection stage, which is necessary for the subsequent identification and quantitative analysis of corrosion features. The resulting reconstructed morphology and color images are combined into an image containing high-dimensional data. Using such high-dimensional data as input for training a corrosion detection neural network can improve the accuracy of detection and identification of corrosion features, enabling the present invention to distinguish actual rust on the coating surface from rust bleeding (thereby reducing the interference of rust bleeding on corrosion ratings), and provide data related to corrosion features sufficient for quantitative rating of corrosion severity in subsequent computer-assisted data analysis. Thus, the system and method of the present invention implements an autonomous process for detecting corrosion defects (even in the early stages of corrosion failure) and quantitatively rating corrosion severity, which has been verified by correlation with human rating results.

[0062] Figure 1 A schematic diagram of a system for evaluating the anti-corrosion properties of a coating on a corrosion-susceptible substrate ("coated panel" or "coating sample") according to one example of the present invention is illustrated. The system includes an imaging unit 101, a computational image processing unit 102, a data preprocessing unit 103, a corrosion detection unit 104, and a computer-assisted data analysis unit 105. The imaging unit 101 acquires a plurality of grayscale images of the coating surface of the coated panel as input, and the computer-assisted data analysis unit 105 outputs a corrosion severity rating.

[0063] Figure 2A flow chart of a method for evaluating the anti-corrosion properties of a coating when applied on a corrosion-prone substrate according to an example of the present invention is illustrated. The method includes image acquisition 201, computational image processing 202, data preprocessing 203, a corrosion detection neural network 204, and computer-assisted data analysis 205. Imaging acquisition 201 includes multi-angle illumination imaging and multi-spectral illumination imaging, and the images obtained therefrom are processed in computational image processing 202 by morphological image reconstruction and color image reconstruction, respectively. The images obtained after computational image processing 202 are combined into images containing high-dimensional data by data preprocessing 203. Then, the high-dimensional data is input to the corrosion detection neural network 204. The corrosion detection neural network 204 has been trained by model training to output segmentation results of corrosion defects, including the location, classification, and region of corrosion features. These corrosion features are quantitatively analyzed in computer-assisted data analysis 205, and the quantitative analysis then outputs a predicted rating result of corrosion severity.

[0064] Imaging unit and image acquisition

[0065] The system of the present invention includes an imaging unit that can be used for image acquisition. The imaging unit is configured to capture multiple images of a coating applied on a substrate susceptible to corrosion ("coated metal panel" or "coated panel"). The imaging unit generally enables the use of computational imaging, including, for example, multi-angle imaging such as multi-angle illumination imaging, multi-spectral imaging such as multi-spectral illumination imaging, or a combination thereof. The imaging unit includes an imaging device. The imaging device generally includes a programmed lighting device and a camera. For example, the imaging device may include a multi-angle illumination imaging device and a multi-spectral illumination imaging device. Alternatively, the imaging device may include a device having both multi-angle illumination imaging and multi-spectral illumination imaging functions. The camera in the imaging device may be any grayscale camera, such as a grayscale industrial camera with more than 10 million pixels (e.g., 500 million pixels or more pixels) and a data transmission and computer-controlled interface. Desirably, multiple grayscale images of the coating include images acquired by both multi-angle imaging and multi-spectral imaging, and more desirably, images acquired by multi-spectral illumination imaging and images acquired by multi-angle illumination imaging.

[0066] Multi-angle lighting imaging

[0067] Multi-angle or multi-direction refers to four or more directions, and can be 5 or more, 6 or more, or even 8 or more directions, and ideally 4 to 6 directions. "Multi-angle illumination imaging" refers to imaging performed with the aid of a multi-angle illumination device. A multi-angle illumination device is a device that can illuminate a sample with light in multiple directions to cast directional shadows around raised or depressed features on the sample.

[0068] An imaging device including a multi-angle illumination device may also be referred to as a multi-angle illumination imaging device. An exemplary multi-angle illumination imaging device typically includes a single camera to capture multiple images of a sample (i.e., a coating surface) illuminated by multiple light sources. The illumination source may be a ring light with four 90-degree quadrants, an array of four strip lights, or any other arrangement that produces multi-directional illumination.

[0069] The multi-angle illumination imaging device can be used to capture multiple images by emitting segmented light arrays from multiple angles. The multiple images can then be used in a computational image processing unit described later to generate a reconstructed topographic image, which is configured to obtain a shadow image in a process called "shadow recovery shape". Multi-angle illumination imaging can emphasize the three-dimensional surface structure of the sample, which is particularly suitable for detecting slight corrosion defects on the surface of a coating sample and performing three-dimensional (3D) surface reconstruction on the surface. Suitable examples of multi-angle illumination imaging devices may include the LSS-2404 available from CCS Inc. of the United States and the CV-X series available from Keyence Corporation.

[0070] Use of the multi-angle illumination imaging device in conjunction with the computational image processing unit described later can make visually noisy or highly reflective surfaces (such as glass) easy to inspect, which is particularly effective for coating surfaces that have 3D structure but little contrast. Figure 3 A schematic example of a multi-angle illumination imaging device 300 according to an example of the present invention is illustrated. The multi-angle illumination imaging device 300 with different illumination angles includes a camera and one or more illumination devices (e.g., lamp 1, lamp 2, lamp 3, and lamp 4) that emit light in multiple different directions. The surface plane of the coating sample is illuminated by light in different directions one by one, so that multi-angle images of the coating sample with the same sequence are captured by the camera. Therefore, multiple images of the surface of the coating sample (or coated panel) are obtained by the multi-angle illumination imaging device, and the multiple images can also be referred to as "multi-angle illumination images".

[0071] Multispectral illumination imaging

[0072] “Multispectral” refers to four or more wavelengths, such as 4 to 16 wavelengths or 4 to 8 wavelengths. “Multispectral illumination imaging” refers to imaging performed with the aid of a multispectral illumination device. A multispectral illumination device is an illumination device that includes multiple light sources having different specified wavelengths. Desirably, a multispectral illumination device having eight or more spectral channels of light can be used to obtain more accurate color information than a conventional digital color camera, and to distinguish between different classifications of corrosion features on a coating surface based on such color information. “Digital color camera” refers to a common color camera for industrial and home use with a Bayer filter. A Bayer filter is a color filter array (CFA) that arranges RGB color filters on a square grid of a photosensor.

[0073] An imaging device including a multi-spectral illumination device is also referred to as a multi-spectral illumination imaging device. Compared with a hyperspectral imaging device, such a device can provide a simpler and more practical solution for industrial imaging applications. The type of wavelength of the light source (such as an LED source) can be selected. For example, an LED light source with eight spectral channels and a monochrome grayscale camera can be used. The image captured by the multi-spectral illumination imaging device contains four or more spectral image planes, which represent four or more spectral channels, such as ultraviolet (405 nanometers (nm)), blue (457nm), green (527nm), orange (600nm), red (660nm), far infrared (730nm), infrared (860nm) or white (600nm), and all wavelength values ​​are approximate peak wavelength values. These images acquired by the multi-spectral illumination imaging device can be combined in a computational image processing unit described later to produce a reconstructed color image.

[0074] Under the illumination of the light source of each spectral channel, the corresponding signal intensity image of the coating sample in the spectral region can be obtained. For example, when imaging is completed in sequence under the illumination of eight spectral channels, eight response intensity images of the coating sample under the eight spectral channels are captured. Then, the intensity of the eight spectral positions converted into corresponding RGB or Lab color values ​​can be reconstructed into a color image according to the calculation image processing unit described later. Therefore, a plurality of images captured by the multi-spectral illumination imaging device may include response intensity images under spectral channels. Alternatively, a grayscale camera can be used together with a light source containing a plurality of spectral illumination channels that are independently controlled. When the light source cycles through each individual spectral channel, subsequent images are captured. Each of these images corresponds to the reflectivity of the coating for individual spectral illumination. These images can be combined and reconstructed into a color image later.

[0075] Using multispectral illumination imaging, specified corrosion features can also be distinguished from other corrosion features on the image based on how they respond to various spectral illuminations. The image resolution of multispectral illumination imaging matches the full pixel resolution of the grayscale camera in the multispectral illumination imaging device. In contrast, when imaging with a conventional digital color camera, multiple adjacent pixels are combined to resolve colors, giving an image with relatively low resolution. Therefore, compared to digital color camera imaging, multispectral illumination imaging provides higher resolution at a lower cost and with less system complexity. Suitable examples of multispectral illumination imaging devices may include the HPR2 series available from CCS Inc. and the CA-DRM10X available from Keyence Corporation.

[0076] Figure 4 A schematic example of a multi-spectral illumination imaging device according to an example of the present invention is illustrated. The multi-spectral illumination imaging device 400 includes a camera and one or more illumination devices (e.g., lamp 1, lamp 2, lamp 3, lamp 4, lamp 5, lamp 6, lamp 7, and lamp 8) that emit light of multiple different wavelengths. Therefore, multiple images of the coating sample are acquired by the multi-spectral illumination imaging device, and the multiple images may also be referred to as "multi-spectral illumination images".

[0077] Desirably, the plurality of grayscale images of the coating for input to a computational image processing unit (for computational imaging processing) described later include multi-angle illumination images and multi-spectral illumination images.

[0078] Computational image processing unit and computational image processing

[0079] The system of the present invention also includes a computational image processing unit that can be used for computational image processing. The computational image processing unit is configured to reconstruct the captured multiple grayscale images of the surface of the coating sample and output the reconstructed topographic image and the reconstructed color image. The computational image processing can use computer vision technology to reconstruct the image.

[0080] (A) Reconstruction of topographic images

[0081] A plurality of grayscale images ("raw images") of a coating sample may include images captured from different illumination angles, such as images acquired by a multi-angle illumination imaging device, which may be reconstructed into a topographic image using surface height map information by a shape-from-shadow algorithm. The shape-from-shadow algorithm may employ a height-driven process in which surface colors or features without height are removed, and the calculated surface image is output based on the shadow information. Reconstructing a topographic image using a surface height map Z(x, y) may be performed in two steps by a shape-from-shadow algorithm. In a first step, gradients in the x and y directions, represented as "p" and "q", respectively, may be calculated from the captured grayscale image. In a second step, a topographic image is obtained by integrating the gradients.

[0082] Figure 5 is a schematic illustration of the principle of the "Shape from Shading" algorithm. The intensity recorded by the camera depends primarily on the angle between the direction vector of the incident light ("s") and the normal vector ("n") of the observed surface element.

[0083] In general, the z-axis of the real-world coordinate system can be chosen to coincide with the optical axis of the camera. Thus, the image plane is parallel to the xy plane of the coordinate system. The intensity I(x,y) recorded by the camera depends only on the following parameters: the sensitivity of the camera sensor ("c"), the direction vector ("s") and intensity ("Q") of the incoming telecentric light, and the fraction of the light reflected towards the camera ("R(n,s)"):

[0084] I(x,y)=c·Q(x,y,z)·R(n,s)

[0085] The reflectivity map R(n,s) describes all the details about the reflection of light. R depends on the material-dependent and position-dependent reflection coefficients ("r") for diffuse and specular reflection, and on the angle ("θ") between the direction vector of the incoming light ("s") and the unknown normal ("n") of the considered surface element. Considering only diffuse reflection (Lambert's law), this yields:

[0086] R(n,s)=r·cosθ, where cosθ=n·s

[0087] This result does not depend on the viewing direction of the camera.To suppress specular reflections, the camera and the light fixture should be positioned in such a way that, according to the law "angle of incidence = angle of reflection", the specularly reflected light is deflected as much as possible away from the camera lens.

[0088] The intensity recorded in each pixel of the camera corresponding to a volume element of the surface is therefore described by:

[0089] I(x,y)=c·Q·r·n·s=ρ·n·s(where albedo ρ=c·Q·r)

[0090] Under these assumptions, it is impossible to separate the effects of c, Q, and r. Therefore, they are combined in a single variable called albedo.

[0091] The normal vector "n" of a surface element is defined by the gradient of the surface Z(x,y) in the x and y directions, that is, by the partial derivatives of Z(x,y) with respect to x and y:

[0092]

[0093] In order to eliminate ρ and calculate the gradients p and q, at least three independent equations for different light directions and therefore at least three pictures are required. If more than three pictures are captured and thus an overdetermined system of linear equations is resulted, improved results and estimates of the measurement error can be achieved. After solving the system of equations using standard methods, the topographic image is calculated by integrating over p and q (further details of the shading recovery shape algorithm can be found in BKP Horn and MJ Brooks (eds.), "Shadow Recovery Shape" (Shape from Shading), MIT Press, 1989).

[0094] In corrosion resistance testing of coatings when applied on corrosion-susceptible substrates (e.g., metals), corrosion often results in changes in the surface morphology of the coating, such as the formation of non-uniform defects on the originally flat surface of the coating. Computational image processing using shape-from-shading can rapidly characterize non-uniform corrosion defects of large-area samples and provide resulting topographic images of similar quality to conventional three-dimensional surface scanning tests (e.g., 3D laser scanning measurements that can progressively map a 3D map of a sample surface), while taking less time than 3D surface scanning tests.

[0095] Figure 6 A schematic flow chart for obtaining a topographic image from a multi-angle illumination image of a coated metal panel according to an example of the present invention is illustrated. A coated steel panel 601 (i.e., a coating applied to a steel panel) after exposure to an ASTM B117-11 salt spray test is provided. A plurality of grayscale images 602 are taken under illumination from different angles using a multi-angle illumination imaging device, wherein "normal" means that all lights are on, "upper" means that the upper position lights are on, "left" means that the left position lights are on, "lower" means that the lower position lights are on, and "right" means that the right position lights are on. The obtained multi-angle images are then processed by a shading recovery shape algorithm; thereby obtaining a reconstructed topographic image 603.

[0096] (B) Reconstruction of color image

[0097] The plurality of grayscale images ("raw images") of the coating may include images at different wavelengths, such as images acquired by the multi-spectral illumination imaging device described above. For example, eight images may be collected for eight different wavelengths. A spectral reconstruction algorithm may be used to reconstruct a color image from the multi-spectral illumination images. The spectral reconstruction algorithm may be based on the following process: for each image at a specified wavelength, the grayscale level represents the intensity of the reflected light; based on these data, a reflectance spectrum curve may be generated for each pixel, and the color value of each pixel may be calculated from the reflectance spectrum curve; then, a color image may be reconstructed from the color value of each pixel.

[0098] Image reconstruction can be performed using the CIE 1931 color space created by the International Commission on Illumination (CIE) in 1931. For each pixel, XYZ tristimulus values ​​can be calculated from the spectral data. According to the definition of tristimulus values, assuming that the spectral distribution function of the reflected light of an object is φ(λ), and the spectral tristimulus function is the decomposition of φ(λ) according to the spectral tristimulus values, the tristimulus values ​​corresponding to each wavelength can be obtained, and then the tristimulus values ​​are integrated from the entire visible light band to obtain the color tristimulus values, as shown in the following formula:

[0099]

[0100] where k is the adjustment factor, and is the CIE standard observer function (10 degrees). Usually, the value of k is 100. The tristimulus values ​​can also be changed by adjusting the value of k.

[0101] The resulting X, Y, and Z values ​​are then substituted into the following formula to obtain the R, G, and B stimulus values ​​for the target reflectance spectrum entering the human eye:

[0102]

[0103] By calculating the chromaticity coordinates, the position of the color light in the CIE color space can be obtained (for further details, please refer to: http: / / www.brucelindbloom.com / index.html?Eqn_Spect_to_XYZ.html).

[0104] Figure 7 A schematic diagram of color image reconstruction from multiple single-wavelength illumination images is illustrated. Eight grayscale images are generated from a multi-spectral illumination imaging device 702 with eight spectral channels 7011 to 7018 having wavelengths of 450nm, 475nm, 495nm, 525nm, 545nm, 580nm, 620nm and 670nm, respectively. For each pixel (x, y) (x and y mean the position of the pixel on the image), such as pixel (0, 0), a reflection spectrum curve 702 with eight data points can be generated. Then, the color value of such a pixel can be calculated from the reflection spectrum curve of such a pixel. After the color value of each pixel is generated, together with the position of each pixel, a color image 703 with 24 colors is reconstructed, wherein each of the 24 grids represents a different color.

[0105] Figure 8A computational color image reconstructed from an image of a coated panel captured by a multi-spectral illumination imaging device according to an example of the present invention is illustrated. Eight grayscale images 801 from eight spectral illuminations are processed using a spectral reconstruction algorithm in a computational image processing unit to obtain a reconstructed color image 802.

[0106] The reconstructed color image enables the distinction of smaller color differences present on the surface of the coated panel compared to images captured by conventional digital color cameras.

[0107] Desirably, a multi-angle illumination imaging device and a shading recovery shape algorithm are used to obtain a surface height map of the coated panel, thereby giving a reconstructed topographic image; then, a multi-spectral illumination imaging device and a spectral reconstruction algorithm are used to obtain color information of the sample surface; thereby giving a reconstructed color image.

[0108] Data preprocessing unit and data preprocessing

[0109] The system of the present invention further includes a data preprocessing unit that can be used to preprocess the data output from the computational image processing unit. The data preprocessing unit is configured to combine the reconstructed topographic image and the reconstructed color image into an image containing high-dimensional data. That is, the high-dimensional data is generated from the output of the above-mentioned computational image processing unit. Then, the high-dimensional data is used as an input of the corrosion detection unit described later.

[0110] "High-dimensional data" herein means data having at least four dimensions. The high-dimensional data in the present invention includes one-dimensional surface height data and at least three-dimensional color data (e.g., three or more color dimensions for each pixel on the sample image), and may have 4 or more, 5 or more, 9 or more, or even 10 or more color dimensions. The high-dimensional data, i.e., the combined image data, will be used as a combined input to the corrosion detection unit described later. For each pixel on the sample image, the deep learning neural network in the corrosion detection unit described later will use the color data and the surface height data together.

[0111] The reconstructed topographic image obtained from the above-mentioned computational processing unit is processed by grayscale values ​​representing surface height values ​​and converted into one-dimensional surface height data. The reconstructed color image obtained from the above-mentioned computational processing unit can be processed by the reflection intensity of the sample when illuminated by different spectral wavelengths and converted into at least three-dimensional color data. Then, the at least three-dimensional color data and the one-dimensional surface height data are combined into high-dimensional data by matrix addition operations.

[0112] Fig. 9Schematic diagram of high-dimensional data combined from the output of a computational image processing unit according to an example of the present invention. A reconstructed color image 9011 is processed by the reflection intensity of the red, green and blue (RGB) spectral channels and converted into three-dimensional color data 9021 (a number representing the reflection intensity). A reconstructed topographic image 9012 is processed and converted into one-dimensional surface height data 9022 (a number representing the reflection intensity). The three-dimensional color data and the one-dimensional surface height data are combined into four-dimensional data 903 (a number representing the reflection intensity) by a matrix addition operation.

[0113] The combination of the data in the reconstructed topographic image and the reconstructed color image thus forming high-dimensional data enables the present invention to distinguish actual rust on the coating surface from rust bleeding. "Rust bleeding" refers to color contamination on the coating surface caused by the flow of salt solution in, for example, the ASTM B117-11 salt spray test.

[0114] Corrosion detection units and corrosion detection and identification

[0115] The system of the present invention also includes a corrosion detection unit that can be used to detect corrosion features. The corrosion detection unit is configured to receive high-dimensional data and identify corrosion features and output at least the location, classification and area of ​​the corrosion features. The corrosion detection unit can be a one-step deep learning unit including a corrosion detection neural network. "Corrosion detection neural network" refers to a neural network used to identify corrosion defects in this article. The corrosion detection neural network has an input layer and an output layer. The input data of the input layer contains the high-dimensional data obtained above. The output data from the output layer contains data on at least the location, classification and area of ​​the corrosion features.

[0116] Corrosion detection neural networks are used to implement image segmentation, specifically semantic segmentation, for different corrosion defects on images containing high-dimensional data. Image segmentation is an image processing method that divides an image into different parts based on the characteristics and properties of each part. Semantic segmentation, a type of image segmentation, assigns a class to each pixel in a given image. Compared to other types of image segmentation that aim to group similar areas of an image, semantic segmentation can be used to quantify corrosion features (such as location, classification, or size) using deep learning.

[0117] The U-Net neural network is used to implement semantic segmentation functionality for defined corrosion features in the corrosion detection neural network (further details of the U-Net neural network can be found in Ronneberger, Olaf, Philipp Fischer, and Thomas Brox, "U-net: Convolutional networks for biomedical image segmentation", International Conference on Medical image computing and computer-assisted intervention, Springer, Cham, 2015). The U-Net neural network is used to supplement the usual contraction network through consecutive layers, where the pooling operation is replaced by an upsampling operator. Therefore, these layers increase the resolution of the output. Subsequently, the consecutive convolutional layers can learn to assemble an accurate output based on the results of the upsampling operation. One modification in the U-Net neural network compared to the fully convolutional network is that there are a large number of feature channels in the upsampling part, which allows the network to propagate contextual information to higher resolution layers. Therefore, the expansion path is symmetrical with the contraction part and a U-shaped architecture is produced. The U-Net neural network only uses the effective part of each convolution without any fully connected layers. Fig.10 A schematic diagram illustrating a typical U-Net neural network used in a corrosion detection neural network.

[0118] In order to further improve the prediction accuracy of the present invention, the corrosion detection unit may further include a region of interest (ROI) neural network before the corrosion detection neural network. "ROI neural network" refers to a neural network used to identify a region of interest. The ROI neural network is configured to receive high-dimensional data as input data and output a ROI recognition result. Based on the ROI recognition result, the boundary of the region of interest can be extracted from the high-dimensional input data to obtain ROI extracted data (i.e., high-dimensional data after ROI extraction). Then, the ROI extracted data is input into the corrosion detection neural network, which can predict at least the location, classification and region of the corrosion defect as output.

[0119] Model training process

[0120] The corrosion detection neural network may include a model training unit that can be used in a model training process. The model training process useful in the present invention is for training a deep learning neural network using a training data set using training coated panels (also referred to as "training coatings").

[0121] The corrosion detection training data set is used to train the corrosion detection model deployed on the corrosion detection neural network, so as to form a trained corrosion detection model deployed on the trained corrosion detection neural network that can predict at least the location, classification and area of ​​corrosion defects. The corrosion detection training data set includes a set of high-dimensional data of the training coated panel and a set of data containing corresponding actual corrosion features. The high-dimensional data of the training coated panel is obtained by: (i) collecting multiple grayscale images of each training coated panel; (ii) reconstructing the captured grayscale image for each training coated panel, and outputting the reconstructed topographic image and the reconstructed color image by computational imaging processing for each training coated panel; (iii) combining the reconstructed topographic image and the reconstructed color image for the training coated panel; thereby forming an image containing high-dimensional data for each training coated panel, the high-dimensional data containing one-dimensional height data and at least three-dimensional color data. Steps (i), (ii) and (iii) can be performed according to the description in the above-mentioned imaging unit, computational imaging processing unit and data preprocessing unit. The actual corrosion features of such training coated panels, including actual locations, classifications, and regions, can be identified based on the quantified degree of rust, blistering, and maximum creep; or based on ASTM D610-08, ASTM D714-02, and ASTM D1654-08. For example, these corrosion features can be marked by humans through visual inspection of the training coated panels. An experienced laboratory operator can use the LabelMe tool to manually mark the areas of corrosion features through visual inspection of the training coated panels, for example, using different labeling categories for different classifications of corrosion defects (such as rust, blistering, and creep) according to ASTM D610-08, ASTM D714-02, and ASTM D1654-08, respectively. LabelMe is a graphical image annotation tool developed by the Computer Science and Artificial Intelligence Laboratory of the Massachusetts Institute of Technology (MIT), with a website address of http: / / labelme.csail.mit.edu. In one embodiment, creep is marked as one category, while blistering and rust are marked as another category for training coated panels.

[0122] The ROI model deployed on the ROI neural network is trained using the ROI training data set to form a trained ROI model deployed on the trained ROI neural network that can predict regions of interest (ROIs). The ROI training data set includes a collection of high-dimensional data of the training coated panels and a collection of data containing corresponding ROIs ("actual ROIs") of such training coated panels labeled by humans. The high-dimensional data of the training coated panels are obtained as described above. Humans use the LabelMe tool, for example, to mark the regions of interest of the training coated panels by visual inspection of the training coated panels.

[0123] The size of the training dataset is important for the predictive accuracy of a neural network model. Typically, a large-scale training dataset (more than 10,000 samples) can greatly improve the accuracy of the model's predictions. Fig.11 The model training process for a deep learning neural network according to an example of the present invention is illustrated. The training process uses a training data set using multiple training samples (e.g., 20 samples), which is called the "original training data set", i.e., a small-scale training data set 1101. The small-scale training data set 1101 includes a small-scale image data set (i.e., high-dimensional data) and a small-scale labeled data set (i.e., human-labeled erosion features or human-labeled ROIs). The small-scale training data set 1101 can be further expanded to 1000-2000 times the number of data sets using a data augmentation algorithm 1102, and then a training data set of tens of thousands of pictures can be obtained, thereby forming an expanded training data set 1103 including a corresponding expanded image data set and an expanded labeled data set. The expanded training data set 1103 is used to train a neural network 1104 to obtain a trained neural network model 1105. Data augmentation in data analysis is a technique for increasing the amount of data by adding slightly modified copies of already existing data or synthetic data newly created from existing data. When training a deep learning model, data augmentation can act as a regularizer and help reduce overfitting. Data augmentation can increase the diversity of data available for training models without actually collecting new data. The training data set used in the present invention can first be expanded using a data augmentation algorithm before training the neural network to further improve the accuracy of the model. Common data augmentation techniques such as de-texturing, decolorization, cropping, padding, and horizontal flipping can be used in training neural networks.

[0124] During the model training process, among the original training data set, 10%-30% of the original training data set can be randomly selected as a cross-validation data set, and the rest is used as a new training data set. Use this new training data set to train the U-Net neural network. At the end of each iteration, the cross-validation data set will be used to verify the model. If the accuracy is not high enough (e.g., greater than 95%), the model training unit will continue to optimize the next iteration. After a certain number of iterations (e.g., 300-500 iterations), if the prediction accuracy of the model for the cross-validation data set reaches a relatively high accuracy, for example, greater than 95%, the training iteration can be stopped and a trained model can be generated.

[0125] Prediction Process

[0126] The corrosion detection unit of the present invention may also include a prediction unit, which includes one or more neural networks deployed together with a trained model and has the ability to predict or output at least the location, classification and size of corrosion defects and / or areas of interest when receiving input high-dimensional data.

[0127] The trained corrosion detection model and the trained ROI model obtained from the training process can be deployed in the corrosion detection neural network and the trained ROI neural network, respectively. The neural network deployed with the trained corrosion detection model is also referred to as a "trained corrosion detection neural network". The neural network deployed with the trained ROI model is also referred to as a "trained ROI neural network". When high-dimensional data of a coated panel is received, the trained corrosion detection neural network can identify corrosion defects and output segmentation results of the corrosion defects (also as identification results of corrosion defects), which segmentation results include data of at least the location, classification and region of the corrosion defects of such coated panels. When high-dimensional data of a coated panel is received, the trained ROI neural network can identify the region of interest and output the segmentation results of the ROI. The corrosion detection neural network and the region of interest neural network can each independently use a U-Net neural network.

[0128] When the system acquires new grayscale images (e.g., images of test coatings), these new images are processed into high-dimensional data (as input data) including surface height map data and color data associated with these new images, and the high-dimensional data is input into a trained neural network, and such a neural network can then generate predictions within seconds and label the predictions on the image. The prediction process can be performed by a deep learning neural network such as a U-Net neural network.

[0129] Fig.12 A schematic flow chart of a one-step deep learning neural network for corrosion identification and classification according to an example of the present invention is shown. The one-step deep learning neural network in this article means using only one trained corrosion detection neural network. The high-dimensional data 1202 of the new grayscale image is directly input into the trained corrosion detection neural network 1204 (i.e., a neural network deployed with a trained corrosion detection model), which then outputs a corrosion defect recognition result 1206, which includes the location and area data 1206A of creep and the location and area data 1206B of rust and blistering (output data). Therefore, the location, classification and area of ​​corrosion defects can be predicted by the trained corrosion detection neural network.

[0130] The corrosion detection unit may or may not further include a trained ROI neural network. If there is a clear background around the coated panel when the grayscale image is taken, the system can use a one-step deep learning neural network without the need for a ROI neural network for identifying regions of interest. Fig.13 A schematic flow chart of a two-step deep learning neural network for corrosion identification and classification according to an example of the present invention is illustrated. For example, the corrosion detection unit 1300 includes a trained ROI neural network in the ROI identification step 1301, followed by a trained corrosion detection neural network in the corrosion identification step 1302. The trained ROI neural network receives input data and outputs an ROI identification result. Based on the ROI identification result, the boundary of the region of interest can be extracted from the high-dimensional input data, thereby obtaining ROI extracted data, for example, high-dimensional data after ROI extraction. Then, following the same procedure as the above-mentioned one-step deep learning neural network, as a second step, the ROI extracted data is input into the trained corrosion detection neural network. The trained corrosion detection neural network then outputs a recognition result of the corrosion defect, which includes data (output data) containing at least the location, classification, and region of the corrosion defect. That is, the output data from the output layer of the trained corrosion detection neural network contains at least the predicted location, classification, and region of the corrosion feature.

[0131] Computer Assisted Data Analysis Unit

[0132] The system of the present invention further includes a computer-assisted data analysis unit that can be used to receive and analyze the output data from the corrosion detection unit. The computer-assisted data analysis unit may also have the functionality to distinguish between blister defects and rust defects in the output data. The computer-assisted data analysis unit is configured to provide a predictive rating of corrosion severity by analyzing the corrosion signature data output from the corrosion detection neural network. The computer-assisted data analysis may include image analysis and data statistics methods known in the art. The computer-assisted data analysis may be performed using an automated measurement algorithm.

[0133] After the trained corrosion detection neural network identifies the corrosion features and the trained ROI neural network (if any) identifies the boundaries of the region of interest, these output results related to the corrosion features are further evaluated by the computer-aided data analysis unit to predict the rating of corrosion severity. The output data of the corrosion severity rating may include the location of the defect, the classification in different colors, the single-sided creep width, the double-sided creep width, the degree of rust and the degree of blistering, or a combination thereof. "Creep" means the width of the corrosion at the scribe line on the coated panel. There are two main types of scratches in the industry for creep assessment: single straight lines and cross lines. Creep starts to grow from the scribe line, so "single-sided creep" means the width of the corrosion area on one single side of the scribe line, and "double-sided creep" means the width of the corrosion area on both sides of the scribe line, which is calculated as the entire creep width.

[0134] Fig.14 An example of creep calculation according to one example of the present invention is illustrated. The output from the corrosion detection neural network includes scratches "X" 1401 of the coated panel, including scribe line 1 and scribe line 2. The contours of the corrosion area at each scribe line are extracted and given in Figures 1402A and 1402B, respectively, where the x direction is the direction of each scribe line, and the y direction is the direction of corrosion growth. And the maximum distance between the two contours of each scribe line on the y axis is calculated.

[0135] In manual visual inspection, rust defects and blister defects are difficult to distinguish from each other. The difference between rust defects and blister defects is that rust defects cause color changes on the coating surface of the coated panel, while blister defects do not. Therefore, the reconstructed color image obtained above can be used to help distinguish rust defects from blister defects based on the average color value of the defects. For each corrosion defect output from the corrosion detection neural network, the average color value within each corrosion defect is checked to determine whether it is rust. Since there is an overlay effect between the color of the coating and the color of the rust on the surface of such coating, the criteria for determining the average color value of rust on coatings with different colors can be different. For corrosion defects on gray coatings and black coatings, the average RGB (red, green and blue) color index used to determine whether the corrosion defect is rust is as follows:

[0136] If a corrosion defect on a grey coating has R < 82 and G < 51 and B < 51, the corrosion defect is classified as rust; and

[0137] If the corrosion defect on the black coating has R>7 and G>7 and B>7, the corrosion defect is classified as rust.

[0138] For the white coating, first, the reconstructed color image is converted into its grayscale version, which has [0,1] as the grayscale value range. If the average grayscale value of the corrosion defect on the white coating <0.5, the corrosion defect is classified as rust.

[0139] To automatically determine the color of the coating, the average RGB color index of all background regions within the region of interest is used. The background region means the region without creep, rust or blistering. The following color index ranges are used in the determination of the colored coating:

[0140] Black coating: R < 20 and G < 13 and B < 10;

[0141] Grey coating: 62 < R < 133 and 59 < G < 122 and 48 < B < 112; and

[0142] White coating: R > 195 and G > 174 and B > 126.

[0143] The degree of blistering and the degree of rust can be rated respectively based on the analysis and evaluation criteria according to ASTM D714-02 and ASTM D610-08. For the characteristics of blistering, first, the severity level is evaluated based on the size of the largest blisters (e.g., the top ten largest blisters in terms of area), and then the distribution of blisters is evaluated based on the density of blisters (i.e., the number of blisters in a certain area). The blister evaluation can be carried out according to the ASTM D714-02 standard. For the characteristics of rust, first, the severity level is mainly calculated based on the area ratio of the rusted area (i.e., the percentage of the rusted surface area relative to the total area), and then combined with various rust sizes, the distribution of rust is calculated mainly through the positions of different rusts. The rust evaluation can be carried out according to ASTM D610-08. The rating criteria for blister evaluation and rust evaluation can be quantified respectively based on the analysis of the pictures and statistical values in the reference standards given in ASTM D714-02 and ASTM D610-08.

[0144] Fig.15 An exemplary process for rating rust on a coated panel according to an example of the present invention is illustrated. First, the rust defect 1503 is extracted from the output data from the corrosion detection neural network, where the rust defect is determined and identified based on the above average color value criterion. Then, the quantity and size distribution of the rust defect are calculated and compared with the rust criterion 1502. The final rust rating result 1504 is automatically generated according to the calculation. The rust criterion 1502 is obtained as follows:

[0145] Analysis in Figures 1 to 3The reference standard 1501 with visual examples given in , and quantifies the rust distribution type (i.e., spot, general, and pinpoint) by the number of rusted areas corresponding to each rust grade. "Area Ratio %" is the percentage of the rusted surface area relative to the total area as given in Table 1 of ASTM D610-08.

[0146] Fig.16 A process for rating blistering on a coated panel according to one example of the present invention is illustrated. First, blistering defects are extracted from the output data from the corrosion detection neural network after excluding rust defects 1603. Then, the amount and size distribution of blistering defects are calculated and compared with the blistering criteria 1602. The final blistering rating result 1604 is automatically generated based on the calculation.

[0147] The bubbling criterion 1602 is obtained as follows:

[0148] The blistering conditions in ASTM D714-02 are analyzed and quantified by the number of blistering areas corresponding to size and density (few, medium, medium dense, and dense). Figures 1 to 4 The photographic reference standard for blisters given in 1601. The "size" of the blister is counted in pixels (1 pixel = 0.035 mm 2 ) and included in the bubble criteria 1602.

[0149] Fig.17 A type of final output image from the system of the present invention according to an example of the present invention is given. The final output image is obtained by superimposing the output data from the corrosion detection neural network and the computer-aided data analysis results on the reconstructed color image obtained in the computing processing unit. Image 1700 includes the boundary 1701 of the region of interest (the region of interest is within the boundary frame 1701, and the region of no interest is outside the boundary frame), blistering 1702 (shown in red), rust 1703 (shown in blue), creep area 1704 in the "X" line, rust penetration 1705, and the predicted rating result of corrosion severity 1706. In order to highlight the blistering and rusting on the image, some rust is marked in the solid line box, and some blistering is marked in the dotted line circle. The predicted rating of corrosion severity includes the maximum unilateral creep and the maximum bilateral creep in millimeters (mm), the degree of rust, and the degree of blistering.

[0150] The present invention also relates to a computer-implemented method for evaluating the anti-corrosion properties of a coating when applied to a corrosion-susceptible substrate (i.e., the coated panel described above). The method may include: receiving a plurality of grayscale images of the coating; reconstructing the plurality of grayscale images by computational image processing, and outputting a reconstructed topographic image and a reconstructed color image; combining the reconstructed topographic image and the reconstructed color image into an image comprising high-dimensional data, the high-dimensional data comprising one-dimensional height data and at least three-dimensional color data; inputting the high-dimensional data into a corrosion detection unit, the corrosion detection unit being configured to identify corrosion features and output at least the location, classification, and region of the corrosion features, wherein the corrosion detection unit comprises a corrosion detection neural network having an input layer for receiving input data and an output layer for outputting output data, wherein the input data comprises the high-dimensional data, and the output data comprises at least the location, classification, and region of the corrosion features; and analyzing at least the location, classification, and region of the corrosion defects obtained from the corrosion detection unit by computer-aided data analysis, and outputting a predicted rating of corrosion severity. The resulting predicted rating of corrosion severity can be used to validate a coating composition (which can be marked as "qualified" or "unqualified" for anti-corrosion properties), modify the coating composition, or adjust the time interval for coating maintenance when the coating composition is used on a corrosion-susceptible substrate (previous version of the draft). Each step in the method is as described above in the corresponding unit of the system of the present invention. Desirably, the multiple grayscale images include images acquired from both multi-angle imaging and multi-spectral imaging, and more desirably, images acquired by both multi-spectral illumination imaging and multi-angle illumination imaging. The present invention can give a high predictive accuracy of the rating of corrosion severity of a defined corrosion feature, as indicated by a regression coefficient of >80% compared to human assessment results. The regression coefficient R 2 The range of is usually 0 to 1 and can be calculated according to the following equation (I):

[0151]

[0152] where y i is the actual value of the corrosion characteristic rated by a human panelist by visual inspection, i is a sample data point, and is the model prediction for this corrosion characteristic, and is the mean of the actual values ​​of the corrosion characteristics rated by human panelists through visual inspection. 2 The higher it is, the better the model fits the dataset.

[0153] The present invention also relates to a method for training a neural network for detecting corrosion on a coating when applied on a corrosion-prone substrate. The method includes: collecting a plurality of grayscale images of a set of coatings applied on a corrosion-prone substrate (i.e., the above-mentioned training coated panels or training coatings); reconstructing the captured grayscale images for each coating by computational processing, and outputting a reconstructed topographic image and a reconstructed color image for each coating; combining the reconstructed topographic image and the reconstructed color image for each coating, and outputting an image containing high-dimensional data for each coating, the high-dimensional data including one-dimensional height data and at least three-dimensional color data; obtaining at least the actual position, classification, and area of ​​corrosion features of each coating that can be identified according to the quantified degree of rust, degree of blistering, and maximum creep; creating a training data set, the training data set including a set of high-dimensional data and a set of data containing at least the actual position, classification, and area of ​​corrosion features of the set of coatings; and using the training data set to train a neural network; thereby obtaining a trained neural network, i.e., the above-mentioned corrosion detection neural network. Alternatively, the actual location, classification, and area of ​​corrosion features for each coating layer may be identified in accordance with ASTM D610-08, ASTM D714-02, and ASTM D1654-08. For example, these corrosion features may be marked by a human through visual inspection.

[0154] The present invention also relates to a computing device. The computing device useful in the present invention may include a processor and a data storage device, wherein the data storage device has stored thereon computer executable instructions that, when executed by the processor, cause the computing device to perform the functions of a computer-implemented method for evaluating the anti-corrosion properties of a coating on a coated panel.

[0155] The present invention also relates to a computing device having a computing image processing unit, a data preprocessing unit, a corrosion detection unit, and a computer-aided data analysis unit deployed thereon; the computing device may be a client device (e.g., a device actively operated by a user), a server device (e.g., a device providing computing services to client devices), or some other type of computing platform. Some server devices may operate as client devices from time to time to perform specific operations, and some client devices may incorporate server functionality.

[0156] Processors useful in the present invention may be in the form of one or more of any type of computer processing element, such as a central processing unit (CPU), a coprocessor (e.g., a math, graphics, neural network, or cryptographic coprocessor), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a network processor, and / or an integrated circuit or controller that performs processor operations.

[0157] The data storage device may include one or more data storage arrays including one or more drive array controllers configured to manage read and write access to a group of hard disk drives and / or solid state drives.

[0158] In some embodiments, computing devices can be deployed to support cluster architectures. The exact physical location, connectivity, and configuration of these computing devices may be unknown and / or unimportant to client devices. Therefore, the computing devices can be referred to as "cloud-based" devices, which can be housed at various remote data center locations, such as cloud-based server clusters. Desirably, the computing devices are cloud-based server clusters, and inputting concentration data into the decision tree ensemble is performed via a web-based user interface that a user can access.

[0159] Fig.18A schematic diagram of a cloud-based server cluster 1800 according to one example of the present invention is depicted. Desirably, the operation of the computing device can be distributed between the server device 1802, the data storage device 1804, and the router 1806, all of which can be connected by the local cluster network 308. The amount of server devices 1802, data storage devices 1804, and routers 1806 in the server cluster 1800 can depend on the computing tasks and / or applications assigned to the server cluster 1800. For example, the server device 1802 can be configured to perform various computing tasks of the computing device. Therefore, the computing tasks can be distributed between one or more server devices in the server device 1802. As an example, the data storage device 1804 can store any form of database, such as a structured query language (SQL) database or a trained model checkpoint. In addition, any database in the data storage device 304 can be integral or distributed across multiple physical devices. The router 1806 can include a network device configured to provide internal and external communications to the server cluster 300. For example, router 1806 may include one or more packet switching and / or routing devices (including switches and / or gateways) configured to (i) provide network communication between server device 1802 and data storage device 1804 through cluster network 1808, and / or (ii) provide network communication between server cluster 1800 and other devices through communication link 1810 to network 1812. Server device 1802 may be configured to transmit data to cluster data storage device 1804 and receive data from the cluster data storage device. In addition, server device 1802 may organize the received data into a web page representation. Such a representation may take the form of a markup language, such as hypertext markup language (HTML), extensible markup language (XML), or some other standardized or proprietary format. In addition, server device 1802 may have the ability to execute various types of computerized scripting languages, such as Perl, Python, PHP hypertext preprocessor (PHP), active server pages (ASP), JavaScript, etc. Computer program code written in these languages ​​can facilitate providing web pages to client devices and interaction of client devices with web pages.

[0160] Example

[0161] Some embodiments of the present invention will now be described in the following examples. Unless otherwise indicated, all parts and percentages are weight percentages. The following standard analytical equipment and methods are used in the examples and to determine the properties and characteristics stated below. OROTAN, KUAI YI and ACRYSOL are trademarks of The Dow Chemical Company.

[0162]

[0163] Salt spray test

[0164] Preparation of coated panels: The paint formulation was applied to Q panels (cold rolled steel) by using a 150 micrometer (μm) applicator and first dried at 23 degrees Celsius (°C) and 50% relative humidity (RH) for 5 minutes (min), then dried at 60°C for 30 minutes, and finally dried at 23°C and 50% RH for 7 days. An "X" shaped scratch was made by cutting through the dry film on the obtained coated panel using a razor blade. The edges of the coated panel were sealed with 3M vinyl electrical tape so that all uncoated areas and 5 mm wide coating film from each edge of the panel were covered by the tape. The areas covered by the tape are collectively referred to as areas of no interest.

[0165] Then, these coated panels were placed in a salt spray chamber Q-FOG SSP-600 from Q-Lab Corporation and exposed to a salt spray environment (5% sodium chloride mist) according to ASTM B117-11. The salt spray chamber simulates a corrosive environment. After a predetermined number of hours, the panels were taken out of the salt spray chamber to be evaluated by manual evaluation and the following automated evaluation according to the present invention, respectively.

[0166] (A) Manual Evaluation: The coated panels were manually evaluated by visual inspection by three laboratory test operators. The extent of rust, reported as the grade and distribution of rust, was evaluated according to ASTM D610-08. The extent of blistering, reported as the size and distribution of blistering, was evaluated according to ASTM D714-02 (Reapproved 2009), respectively. The maximum single-sided creep, reported in millimeters as "Creep Width", was measured according to ASTM D1654-08.

[0167] (B) Automated evaluation: The coated panels are automatically evaluated using the system of the present invention. An imaging unit including a Keyence CV-X, a CA-DRM10X ring light, and a CA-HX500M camera, all of which are available from Keyence Corporation, is mounted on top of the coated panel support to obtain multi-angle and multi-spectral images of each coated panel. An Ethernet cable is used to connect the computer of the present invention, which is deployed with a computational image processing unit, a corrosion detection unit, a data preprocessing unit, and a data analysis unit, to a Keyence controller in the imaging unit. The computer is used to control the imaging unit and process the obtained grayscale image according to the system of the present invention to automatically generate a corrosion resistance evaluation result.

[0168] Example 1

[0169] In order to verify the effectiveness and accuracy of the system of the present invention, white paint samples were used to compare the results from manual evaluation and automated evaluation according to the system of the present invention. Table 2 shows the white paint formulation containing binder 1 for forming a white coated panel 1. OROTAN KUAI YI 731A dispersant, SurfynolTG and TEGO Airex902W were mixed with water under low-speed stirring to ensure that all ingredients were fully dispersed. Ti-Pure R-706 was slowly added, and the rotation speed was adjusted in time to keep the grind in a "donut" shape. After the fineness of the grind was less than 30μm, further water was added and evenly mixed into the mixture. In the dilution stage, the binder emulsion, water and ammonia water were premixed, and then the grind was added to the premix and gently added to the premix. Thereafter, sodium nitrite (15%), Texanol and ACRYSOL RM-8W thickeners were added to give a paint formulation.

[0170] The other white paint formulations used to prepare white coated panels 2 to 22, respectively containing binders 2 to 22, were the same as the formulation used to prepare Example 1 for preparing white coated panel 1, except that the binder type, amount of binder, and amount of Texanol were as described below. The amount of each binder was adjusted based on the solid content (wt%) of the binder to ensure that the total solids of each paint formulation were equal. The amount of Texanol was adjusted based on the minimum film forming temperature (MFFT) of the binder used in ° C. and can be calculated based on the following equation:

[0171] Amount of Texanol=Weight of binder×solid content×MFFT / 200

[0172] Table 2 White paint preparation of Example 1

[0173] Material Amount, grams Grinding water 42.00 <![CDATA[OROTAN TM KUAI YI TM 731A Dispersant]]> 7.80 <![CDATA[SURFYNOL TM TG]]> 1.00 <![CDATA[TEGO TM Airex 902W]]> 0.46 Ti-Pure R-706 209.24 water 42.00 Thinning Binder 1 (solids: 42%, MFFT: 27°C) 606.86 water 198.43 Ammonia (28%) 4.00 Sodium nitrite (15%) 8.97 Texanol 34.41 <![CDATA[ACRYSOL TM RM-8W Thickener]]> 1.50

[0174] The resulting white paint formulations were used to prepare white coated panels 1 to 22, which were further characterized according to the salt spray test described above.

[0175] After the coated panel is taken out from the salt spray chamber, the coated panel is manually evaluated and automatically evaluated respectively. Three operators spend two minutes to observe and evaluate each sample, and the whole evaluation process of ten samples from visual inspection to recording rating results takes about 20min. In contrast, the system of the present invention only takes about 10 seconds from image acquisition to automatic analysis to evaluate each sample, and the whole evaluation process of ten samples from image acquisition to result presentation using the system of the present invention takes at most two minutes. Compared with the manual evaluation process, the system of the present invention shows a 10-fold improvement in evaluation speed. The rating results of the corrosion severity of these coated panels obtained by the manual evaluation performed by the operator and the automated evaluation performed using the system of the present invention are given in Table 3. Fig.19 A comparison of these ratings for some corrosion characteristics is shown in Fig.19 As shown, the correlation results indicate that for creep width (19A), the regression coefficient is 94.45%, for rust grade (19B), the regression coefficient is 89%, and for blister size (19C), the regression coefficient is 81.5%. The results show that the novel automated system of the present invention can greatly improve the evaluation efficiency while providing evaluation results close to those of a skilled laboratory operator, and also reduce the potential bias and error associated with manual inspection and evaluation by a laboratory operator.

[0176] Table 3 Corrosion severity ratings of white paint obtained by manual and automated evaluation

[0177]

[0178] NA – Not tested

[0179] Example 2

[0180] Gray paint formulations 1 to 15 containing binder 1 to binder 15, respectively, were prepared in Example 2 to verify the robustness of the system of the present invention for evaluating coatings of colors other than white paint. Example 2 was carried out following the same protocol as Example 1 based on the formulations given in Table 4. The obtained gray coated panels 1 to 15 were characterized according to the above-mentioned salt spray test. The rating results of the corrosion severity of these coated panels obtained by manual evaluation by an operator and automated evaluation using the system of the present invention are given in Table 5. Fig. 20 A comparison of these ratings for some corrosion characteristics is shown in Fig. 20As shown, the correlation results indicate that for creep width (20A), the regression coefficient is 90.07%, for rust grade (20B), the regression coefficient is 87.62%, and for blister size (20C), the regression coefficient is 83.46%. The results show that the novel automated system of the present invention has a skill close to the ability of a skilled laboratory operator in the rating of corrosion characteristics of gray paint.

[0181] Table 4 Gray paint formulation of Example 2

[0182] Material Amount, grams Grinding water 40.00 <![CDATA[OROTAN TM KUAI YI TM 731A Dispersant]]> 7.25 <![CDATA[SURFYNOL TM TG]]> 1.76 <![CDATA[TEGO TM Airex 901W]]> 1.76 BENTONE LT 0.80 <![CDATA[PRINTEX TM Powder-4]]> 1.54 Nubirox 106 52.78 Barium sulfate 92.36 Ti-Pure R-706 80.04 water 48.00 Thinning Binder 2 (solids: 41.5%, MFFT: 38°C) 615.71 Sodium nitrite (15%) 8.97 Texanol 48.55 <![CDATA[ACRYSOL TM RM-8W Thickener]]> 1.50

[0183] Table 5 Corrosion severity ratings of grey paint obtained by manual and automated evaluation

[0184]

[0185]

Claims

1. A system for evaluating the anti-corrosion properties of a coating when applied to a corrosion-susceptible substrate, the system comprising: (i) an imaging unit configured to capture a plurality of grayscale images of the coating; (ii) a computational image processing unit configured to receive and reconstruct the captured plurality of grayscale images, and output a reconstructed topographic image and a reconstructed color image; (iii) a data preprocessing unit configured to receive and combine the reconstructed topographic image and the reconstructed color image, and output an image containing high-dimensional data, the high-dimensional data including one-dimensional height data and at least three-dimensional color data; (iv) a corrosion detection unit configured to receive the high-dimensional data and identify corrosion features, and output at least a location, a classification, and an area of ​​the corrosion features; wherein the corrosion detection unit comprises a corrosion detection neural network having an input layer and an output layer, wherein input data of the input layer comprises the high-dimensional data, and output data from the output layer comprises at least the location, classification and region of the corrosion feature; and (v) a computer-assisted data analysis unit configured to receive and analyze said data containing at least location, classification and area of ​​said corrosion features and output a predicted rating of corrosion severity. 2 . The system according to claim 1 , wherein the plurality of grayscale images of the coating include images acquired by multi-spectral illumination imaging and images acquired by multi-angle illumination imaging.

3. The system according to claim 2, wherein the grayscale image acquired by multi-angle illumination imaging is reconstructed into the topographic image by a shape-from-shadow algorithm using surface height map information. 4 . The system according to claim 2 , wherein the grayscale image acquired by multi-spectral illumination imaging is reconstructed into the color image using a spectral reconstruction algorithm.

5. The system of claim 1 , wherein the corrosion detection neural network is trained using a set of training coatings applied to corrosion susceptible substrates by: Collect multiple grayscale images of each training coating; reconstructing the captured grayscale image for each training coating by computational processing, and outputting a reconstructed topographic image and a reconstructed color image for each training coating; combining the reconstructed topographic image and the reconstructed color image for each training coating, and outputting an image containing high-dimensional data for each training coating, the high-dimensional data including one-dimensional height data and at least three-dimensional color data; obtaining at least actual locations, classifications, and regions of corrosion features identified by quantified levels of rusting, blistering, and maximum creep for each training coating; creating a training data set comprising the set of high-dimensional data and a set of data comprising at least the actual locations, classifications, and regions of corrosion signatures of the set of coatings; as well as The corrosion detection neural network is trained using the training data set.

6. The system according to claim 1, wherein before the corrosion detection neural network, the corrosion detection unit further includes a region of interest neural network, and the region of interest neural network is configured to identify the boundary of a region of interest and output the high-dimensional data after extracting the region of interest.

7. The system of claim 6, wherein the corrosion detection neural network and the region of interest neural network each independently use a U-Net neural network.

8. A system according to claim 1, wherein in the data preprocessing unit, the reconstructed color image is processed using the reflection intensities of red, green and blue spectral channels and converted into three-dimensional color data, and the topography image is processed using grayscale values ​​representing surface height values ​​and converted into one-dimensional height data.

9. The system of claim 1, wherein the at least three-dimensional color data and the one-dimensional height data are combined into the high-dimensional data by a matrix addition operation.

10. The system of claim 1, wherein the computer-aided data analysis unit is further configured to quantify the rating criteria of the corrosion characteristics and output a predicted quantitative rating of corrosion severity.

11. The system of claim 1, wherein the predicted rating of corrosion severity includes blistering level, rusting level, creep, and combinations thereof.

12. A computer-implemented method for evaluating the anti-corrosion properties of a coating when applied to a corrosion-susceptible substrate, the computer-implemented method comprising: receiving a plurality of grayscale images of the coating; reconstructing the plurality of grayscale images by computational image processing, and outputting a reconstructed topographic image and a reconstructed color image; combining the reconstructed topographic image and the reconstructed color image into an image comprising high-dimensional data, the high-dimensional data comprising one-dimensional height data and at least three-dimensional color data; Inputting the high-dimensional data into a corrosion detection unit, the corrosion detection unit being configured to identify corrosion features and output at least a location, a classification, and an area of ​​the corrosion features, wherein the corrosion detection unit comprises a corrosion detection neural network having an input layer for receiving input data and an output layer for outputting output data, wherein the input data comprises the high-dimensional data, and the output data comprises at least a location, a classification, and an area of ​​the corrosion features; as well as The data including at least location, classification and area of ​​the corrosion features is received and analyzed by computer-assisted data analysis and a predicted rating of corrosion severity is output.

13. The computer-implemented method of claim 1, wherein the plurality of grayscale images are acquired through multi-spectral illumination imaging and multi-angle illumination imaging.

14. A computing device having a computing image processing unit, a data preprocessing unit, a corrosion detection unit and a computer-aided data analysis unit deployed thereon; wherein the computational image processing unit is configured to receive and reconstruct the captured plurality of grayscale images, and output a reconstructed topographic image and a reconstructed color image; wherein the data preprocessing unit is configured to receive and combine the reconstructed topographic image and the reconstructed color image, and output an image containing high-dimensional data, the high-dimensional data including one-dimensional height data and at least three-dimensional color data; wherein the corrosion detection unit is configured to receive the high-dimensional data and identify corrosion features, and output at least the location, classification and area of ​​the corrosion features; wherein the corrosion detection unit comprises a corrosion detection neural network having an input layer and an output layer, wherein input data of the input layer comprises the high-dimensional data, and output data from the output layer comprises at least the location, classification and region of the corrosion feature; and Wherein the computer-aided data analysis unit is configured to receive and analyze the data including at least the location, classification and area of ​​the corrosion feature and output a predicted rating of corrosion severity.

15. A method for training a neural network for detecting corrosion, the method comprising: collecting a plurality of grayscale images of a collection of the coatings when applied on a corrosion-susceptible substrate; reconstructing the captured grayscale image for each coating layer by computational processing, and outputting a reconstructed topographic image and a reconstructed color image for each coating layer; combining the reconstructed topographic image and the reconstructed color image for each coating layer, and outputting an image containing high-dimensional data for each coating layer, the high-dimensional data including one-dimensional height data and at least three-dimensional color data; obtaining at least the actual location, classification and area of ​​corrosion features identified by quantified rusting extent, blistering extent and maximum creep for each coating layer; creating a training data set comprising the set of high-dimensional data and a set of data comprising at least the actual locations, classifications, and regions of corrosion signatures of the set of coatings; as well as Using the training data set to train the neural network; Thereby obtaining the trained neural network.

Images