Method and apparatus for identifying effect pigments in a target coating

Abstract

The present invention relates to a computer-implemented method comprising at least the following steps: obtaining color values, texture values, and a digital image of a target coating using at least one measurement device; retrieving one or more preliminary matching formulas from a database comprising formulas for coating compositions and associated color values, associated texture values, and associated digital images based on the color values ​​and / or texture values ​​obtained for the target coating; determining sparkle points within the corresponding obtained image and within the corresponding image associated with the one or more preliminary matching formulas; creating a sub-image of each sparkle point from the corresponding image; providing the created sub-images to a convolutional neural network trained to associate the corresponding sub-images of the corresponding sparkle points with pigments and / or pigment classes; and determining, based on outputs of the neural network, at least one of the one or more preliminary matching formulas as a formula that best matches the target coating. The present invention further provides corresponding devices and non-transitory computer-readable media.

Description

Technical Field

[0001] The present invention relates to a computer-implemented method and apparatus for identifying effect pigments in a target coating. Background Art

[0002] Today, additional features for the color search and acquisition process, such as roughness, glitter area, glitter intensity, glitter level, and / or glitter color variation / distribution, in addition to color information, are used as auxiliary conditions to find the optimal solution for a given target color / target coating. These additional features are measures of different visual properties of the color texture appearance.

[0003] These additional features are often obtained from today's spectrometer instruments (such as the Xrite MA- MA- or Byk mac ) captures the target coating. The raw image data is processed by image processing algorithms. These algorithms output texture features, i.e., texture values ​​representing the optical properties of the target coating's texture. These texture values ​​are classified according to known industry standards.

[0004] Due to the nature of complex coating mixtures, it can sometimes be difficult to formulate, identify, and / or search for acceptable matching formulas and / or pigmentation. Ideally, a human could look at a complex coating mixture and determine the appropriate pigment within the coating mixture. However, in reality, the pigments in the coating mixture may not be readily available in the set of colorants used in the coating system to produce a matching coating. Therefore, the color matching technician must determine whether the coating system contains the appropriate offset and, if so, what additional changes are needed to accommodate the offset, as they do not exactly match the original pigmentation.

[0005] It would be desirable to have a method and apparatus that can measure an unknown target coating and, based on the measured data of the target coating, search a database for the best matching coating formulation within the database and / or create a new coating formulation based on the measured data of the target coating. However, currently, known systems are only able to determine the color or overall effect pigment type, but are generally unable to help determine, for example, the specific pearlescence required to match the unknown target coating.

[0006] Known techniques that utilize cameras and / or spectrometers, optionally in conjunction with microscopic evaluation of target coatings, are generally not well-defined to effectively address new effect pigment deposits or complex mixtures and primarily focus on individual evaluation of target coatings, i.e., on a case-by-case basis. This is a very time-consuming process, as each new, unknown target coating must pass through all analytical steps. Consequently, this time-consuming process may not satisfactorily address applications requiring efficient analysis of target coatings in conjunction with the provision of matching formulations.

[0007] Further strategies exist that use painted or virtual samples representing various textures and then compare them to an unknown target coating. However, such techniques typically require significant user intervention and are therefore subjective, which can produce inconsistent results.

[0008] Therefore, there is a need for methods and apparatus suitable for the efficient analysis of unknown target coatings comprising effect pigments. Summary of the Invention

[0009] The above objects are solved by a method and an apparatus having the features of the respective independent claims. Further embodiments are given by the following description and the respective dependent claims.

[0010] The present disclosure relates to a computer-implemented method comprising at least the following steps:

[0011] - obtaining color values, texture values ​​and digital images of the target coating using at least one measuring device,

[0012] - based on the color values ​​and / or texture values ​​obtained for the target coating, fetching one or more preliminary matching formulas from a database comprising formulas for coating compositions and associated color values, associated texture values ​​and associated digital images,

[0013] - performing image analysis for each of the obtained images of the target coating and the images related to / associated with the one or more preliminary matching formulations to find and determine / identify at least one flash point, i.e., determine / identify flash points that may exist within the corresponding image, using a computer processor operatively combined with at least one filtering technique,

[0014] - creating a sub-image of each flash point from the corresponding acquired image and from the corresponding image associated with one or more preliminary matching recipes,

[0015] - providing the created sub-images to a convolutional neural network that is trained to associate the respective sub-images of the respective glitter spots with pigments and / or pigment classes and to recognize the pigments and / or pigment classes based on the respective sub-images of the respective glitter spots,

[0016] - determination and output of statistical data of identified pigments and / or pigment classes, separately for the target coating and for each preliminary matching recipe,

[0017] - using a computer processor to compare the statistical data determined for the target coating with the statistical data determined for one or more preliminary matching formulations, and

[0018] - determining at least one of the one or more preliminary matching formulas as a formula that best matches the target coating.

[0019] The terms "formulation" and "formulated product" are used synonymously herein. The phrase "operably associated" means that the corresponding components (i.e., the computer processor and the at least one filtering technology) communicate with each other in such a manner that the computer processor can control / operate the at least one filtering technology and / or the at least one filtering technology can send corresponding filtering results to the computer processor. The terms "associated with" and "related to" are used synonymously. Both terms indicate the integrity of the components that are associated / related to each other.

[0020] Color values ​​are obtained using at least one measurement device by analyzing a spectral curve of the target coating, measured at different measurement geometries relative to the target coating surface. Typically, the spectral measurement geometry is defined by an illumination direction / angle and an observation direction / angle. A typical spectral measurement geometry is a fixed illumination angle of 45° measured relative to the surface normal of the coating and observation angles of -15°, 15°, 25°, 45°, 75°, and 110°. Each angle is measured relative to the specular reflection angle, i.e., the specular reflection direction, which is defined as the direction of emission at the same angle to the coating surface normal as the incident direction of the corresponding light ray.

[0021] Image values ​​and texture values ​​are obtained by capturing multiple digital images, particularly HDR images, using an image capture device, each image being obtained at a different measurement geometry with respect to the surface of the target coating. A typical image-based texture measurement geometry is a fixed position of the image capture device (i.e., camera) at an angle of 15° to the normal of the target coating surface. The illumination angles selected are r15as-15, r15as15, r15as-45, r15as45, r15as80, and based on X-Rite MA- A semi-diffuse angle is defined. "Semi-diffuse" here means "as diffuse as possible" with respect to the measuring device and its spatial dimensions. With respect to the geometric designation, the positions of at least one measuring device (e.g. camera) and the illumination are opposite. This means that the specular angle is defined here by the fixed position of the camera. Specifically, this means, for example, that the designation "r15 as-15" uses "r" for "reverse", "15" for the fixed position of the camera, i.e. an angle of 15° to the normal of the surface of the target coating, "as" for "anti-specular", and "-15" as the illumination angle measured relative to the specular angle.

[0022] The texture values / parameters are in particular the sparkle grade SG, the sparkle color variation CV and the roughness C or granularity G, the sparkle intensity S_i and the sparkle area S_a of the target coating.

[0023] The at least one measuring device may be selected as, for example, an Xrite Xrite MA- or Bykmac Such a spectrophotometer can also be combined with other suitable equipment (such as a microscope) in order to obtain more image data, such as, for example, microscope images.

[0024] The database is a recipe database, which includes recipes of coating compositions and associated colorimetric data. For each recipe, the associated colorimetric data includes spectral data, i.e., color values, texture values, and digital images of a sample coating based on the corresponding recipe. A preliminary matching recipe is selected from a plurality of recipes in the database based on a first matching metric. The first matching metric is defined / calculated by a color difference metric (e.g., CIE dE*) between the target coating and the corresponding sample coating for all or at least some of the above-mentioned spectral measurement geometries, and is optionally supplemented with at least one texture difference metric, such as at least one of the texture differences dSi, dSa, dG defined by Byk-Gardner [“Beurteilung von Effektlackierungen, Den Gesamtfarbeindruck objektiv messen”, Byk-Gardner GmbH]. The color difference metric and the at least one texture difference metric can be added, optionally by weighted addition. The color difference metric can be described by three color values: L* represents lightness from black (0) to white (100), a* from green (-) to red (+), and b* from blue (-) to yellow (+).

[0025] After obtaining digital images of the target coating, it may be useful to first pre-analyze the digital images to identify defects (such as scratches). Therefore, using an electronic computer processor operatively connected to at least one filtering unit, a first image analysis is performed on the obtained digital images to identify at least one bright region within each digital image by isolating image foreground data from image background data. A connected component analysis is then performed on each digital image to identify at least one damaged region within the at least one bright region. If at least one damaged region is found, the corresponding digital image is masked from further analysis, the corresponding digital image is rejected, and / or image capture is repeated.

[0026] This pre-analysis allows the detection of defects in the image of the target coating. The basic strategy of the proposed pre-analysis is to (1) find defects in the image by searching for typical structural features such as fingerprints and scratches, and (2) decide to reject the image or ignore the detected damaged / defective areas in the image for further image processing.

[0027] This means that measurements of images containing defects can be rejected, or defects / damaged areas in the images can be masked so that further texture analysis of the corresponding images can be performed. The pre-analysis can also be configured to notify the user of the image capture device that the measurement (the at least one digital image obtained) is invalid, for example by throwing / outputting a warning message / signal on an output device (such as a display and / or sound output device) via a computer processor, the output device being part of or in communication with the computer processor. It can also be configured to ask the user to remeasure the coating until the measurement data (i.e., the digital image obtained) is valid. Image capture can also be automatically repeated by the image capture device until the digital image obtained is valid, i.e., without detectable defects. Thus, via the communication connection between the image capture device and the electronic computer processor, the image capture device is automatically notified of at least one damaged area / defect detected within the corresponding one of the images.

[0028] As a result, more accurate results are achieved and errors are reduced during color search and acquisition. Furthermore, the laboratory workload for color development and customer service matching is reduced. The color matching process becomes more reliable and faster, while operating unit costs are reduced.

[0029] The wording "in communication with" indicates that a communication connection exists between the respective components.

[0030] After retrieving one or more preliminary matching recipes from the recipe database, basic statistics of the pigments and / or pigment classes are calculated from the image of the target coating and from the image associated with the preliminary matching recipes.

[0031] Then, based on the statistical data of the pigments and / or pigment classes of the target coating and the one or more preliminary matching formulas, at least one is selected from the one or more preliminary matching formulas as the optimal matching formula to minimize the first matching metric and the new sparkle difference. This means that based on the statistical data of the pigments and / or pigment classes of the target coating and the one or more preliminary matching formulas, i.e., based on a comparison of these statistical data, at least one preliminary matching formula can be identified whose sparkle difference with respect to the corresponding sparkle point of the target coating is minimized (whose sparkle point has the smallest sparkle distance from the corresponding sparkle point of the target coating).

[0032] The statistical data determined for the target coating and the statistical data determined for the one or more preliminary matching recipes may each be presented as a corresponding histogram and / or a corresponding vector.

[0033] Furthermore, a sub-image of each flash point from the acquired image and from the image associated with one or more preliminary matching recipes can be created with and / or without a background, i.e. with the real environment of the flash point and / or with a uniform background, in particular with a black background (corresponding to “without background”).

[0034] According to one embodiment of the proposed method, the method further comprises deriving, using a neural network, from each sub-image that accurately depicts a sparkle point, a correlation of the sparkle point with at least one pigment and / or pigment class, wherein the correlation indicates the contribution of the at least one pigment and / or pigment class to the distribution of sparkle points within the corresponding image from which the sub-image was cut. Advantageously, the neural network is used to derive, from each sub-image, a correlation of the depicted sparkle point with exactly one pigment. For each image, sub-images are created in such a way that the image is composed of sub-images. Typically, the number of sub-images is implicitly determined by the number of sparkle points within the image and lies in the interval from 100 to 1000 (e.g., from 200 to 500).

[0035] In the event that there are n sparkles in a digital image that has captured the target coating or is associated with a formulation of a coating composition and retrieved from a database, n sub-images are created and may result in a number S1 of sub-images being associated with pigment 1, a number S2 of sub-images being associated with pigment 2, and so on, until a number Sk of sub-images being associated with pigment k, where k is greater than 2 and less than or equal to n, where both k and n are integers.

[0036] The quantities S1, S2, ..., Sk together with the corresponding pigments 1, 2, ..., k allow summarizing statistics about the respective fractions of different pigments 1, 2, ..., k within the target coating (i.e. within the formulation associated with the target coating and / or within the formulation associated with the corresponding digital image).

[0037] The proposed method can be additionally performed before or after other pigment identification methods, in particular using glitter color distribution and / or glitter size distribution. Such methods are described, for example, in US 2017 / 0200288 A1 and in European application number 19154898.1, the contents of which are hereby incorporated by reference in their entirety.

[0038] Finally, the best matching formula is identified and forwarded to a mixing unit, which is configured to produce / mix a paint / coating composition based on the identified best matching formula. The mixing unit produces this paint / coating composition, which can then be used to replace the target coating. The mixing unit can be a component of the proposed device.

[0039] The neural networks used with the proposed method are based on a learning process called backpropagation. The neurons of the neural network are arranged in layers. These layers include a layer with input neurons (input layer), a layer with output neurons (output layer), and one or more inner layers. The output neurons are the pigments (i.e., toners) or pigment classes for the target coating (paint) formula to be determined / predicted.

[0040] During the training phase, the neural network's input neurons, as training data, consisted of sub-images of sample coating images, each based on a recipe containing exactly one previously known pigment / toner. Each of these sub-images accurately depicted a single shimmering point in the corresponding image from which it was cut.

[0041] The inner layers of a convolutional neural network consist of all or a subset of convolutional layers, max pooling layers, and fully connected dense layers. Convolutional + Relu (Rectified Linear Unit) layers apply filters to the input neurons (i.e., the input image) to extract features from the input image / incoming image. Pooling layers are responsible for reducing the dimensionality of the features from the convolution. Dense layers are the standard set of fully connected neurons in a neural network that map the high-level features from the convolutional + Relu and max pooling layers to the desired pigments and / or pigment classes.

[0042] Typically, a precise correlation of sparkle points with pigments requires a large amount of training data. Images of sample coatings based on a recipe comprising only exactly one previously known pigment / toner typically show a large number of sparkle points, and for each such sparkle point a sub-image is created, resulting in a correspondingly high number of sub-images. Thus, a sufficient amount of training data can be created despite the countable, i.e. finite, number of available pigments. The amount of training data (i.e. the number of available sub-images) can be further increased by using, for each pigment, both sub-images with a black background and sub-images with any other suitable background.

[0043] The neural network only has to be redefined, retrained, and retested if there are any changes in the number / size of available pigments.

[0044] "Previously known pigments / tintants" refer to pigments that are known and can be used as color components of color formulations.

[0045] The phrase "a formulation for a coating composition and an associated image" refers to a formulation for a coating composition and an image of the corresponding coating that has been captured. The phrase "an image associated with one or more preliminary matching formulations" refers to an image of the corresponding coating that has been captured for one or more preliminary matching formulations, respectively.

[0046] The proposed method is particularly used to provide statistics on different pigments associated with the sparkles identified in the target coating image, so as to determine which pigments and in which quantities form part of the recipe of the target coating. The neural network used is based on a learning process called backpropagation. Backpropagation should be understood here as a general term for supervised learning processes via error feedback. There are various backpropagation algorithms: for example Quickprop, Resilient Propagation (RPROP). The process uses a neural network comprising at least three layers: a first layer with input neurons, an nth layer with output neurons, and (n-2) inner layers, where n is a natural number greater than 2. In such a network, the output neurons are used to identify the pigment class and / or pigment included by the target coating (i.e. by the corresponding recipe).

[0047] "Identifying a / the pigment" means directly determining the specific pigment and / or determining the pigment class to which the pigment belongs. For example, one pigment class may consist of metallic effect pigments, while another pigment class may consist of pearlescent effect pigments. Other suitable categories, in particular further refined categories, are possible. It may, for example, be possible to split the pigment class "metal" into "coarse metal" and "fine metal" or "small coarse / fine metal" or "large coarse / fine metal". The pigment class "aluminum pigments" and the further class "interference pigments" may be provided. The class "aluminum pigments" can be further subdivided into subclasses, such as the subclass "corn flakes" and the subclass "silver dollars". The class "interference pigments" can be further subdivided into the subclasses "white mica", "golden mica", "blue mica", and further into subclasses “Glass”, “Natural Mica”, etc. After comparison of statistical data, some categories or subcategories can also be appropriately regrouped.

[0048] According to one possible embodiment of the proposed method, image analysis uses image segmentation techniques to identify the locations of the flash points in each image. An image mask is created that identifies the locations of the flash points based on color, texture and their gradients. In the mask, each pixel is marked with a "0", indicating that the pixel is not part of the flash point, or a "1", indicating that the pixel is part of the flash point. Contour detection of the mask image identifies the boundaries of the connected pixels of each individual flash point location. The identified flash point contours are overlaid on the original HDR (high dynamic range) image. Sub-images are created for all flash points identified in the mask by extracting the RGB (derived from the red, green and blue color space) pixel data of the associated pixel location from the original HDR image and placing the RGB pixel data in the center of a standard image frame, in which the RGB pixel data of the standard image frame was previously initialized with "0" (black) to provide a defined background.

[0049] Alternative or additional segmentation techniques include thresholding methods, edge-based methods, clustering methods, histogram-based methods, neural network-based methods, hybrid methods, etc.

[0050] According to another aspect of the proposed method, the correlation of each sub-image with at least one pigment and / or pigment class is derived with the aid of a convolutional neural network, which is configured to classify each sub-image of the corresponding flash point of each measurement geometry to a specific pigment and / or a specific pigment class with a predetermined probability.

[0051] Each such derived correlation for each measurement geometry at which the corresponding partial image was acquired is used to adjust the contribution of the at least one pigment when determining the optimal matching recipe.

[0052] According to another aspect of the proposed method, the step of determining the best matching formula comprises providing a list of pigments with corresponding quantities and / or concentrations of corresponding pigments.

[0053] In the case where the glitter points are associated with a pigment class, the determination of the specific pigment within the identified pigment class can be performed by using any of the above methods or a combination thereof using the glitter color distribution and / or glitter size distribution within the corresponding image. Alternatively, the specific pigment can be selected by manual input / decision.

[0054] Typically, the image area of ​​a sub-image is defined by the flash size of the flash points depicted in the corresponding sub-image. All sub-images of the corresponding image can be created using the same image area. In this case, the image area is defined by the flash size of the largest flash point in the corresponding image (i.e., by the maximum flash size). A typical image area might be a 10x10 pixel image area on a black background.

[0055] The present disclosure further relates to a device. The device comprises at least:

[0056] a database comprising formulations of coating compositions and associated color values, associated texture values ​​and associated digital images,

[0057] - at least one processor communicatively coupled to at least one measurement device, a database, at least one filtering technique, and a convolutional neural network, and programmed to perform at least the following steps:

[0058] a. Receive the color value, texture value and digital image of the target coating from the measuring device,

[0059] b. obtaining one or more preliminary matching formulas from a database based on the color value and / or texture value received for the target coating,

[0060] c. performing image analysis on each of the received images of the target coating and the images associated with the one or more preliminary matching formulations to find and determine at least one sparkle point within the respective images by using filtering techniques,

[0061] d. creating a sub-image of each flash point from the received image and from images related / associated with one or more preliminary matching recipes,

[0062] e. providing the created sub-images to a convolutional neural network that is trained to associate the corresponding sub-images of the corresponding glitter spots with pigments and / or pigment classes, and identifying the pigments and / or pigment classes based on the corresponding sub-images of the corresponding glitter spots,

[0063] f. Determine and output statistical data of the identified pigments and / or pigment classes, respectively for the target coating and for each preliminary matching formula,

[0064] g. comparing the statistics determined for the target coating with the statistics determined for one or more preliminary matching formulations, and

[0065] h. Determine at least one of the one or more preliminary matching formulas as the formula that best matches the target coating.

[0066] According to another aspect, the device further comprises at least one measuring device, a filtering technique and / or a convolutional neural network.

[0067] According to another embodiment of the proposed device, the processor is further configured to perform the step of deriving a correlation with at least one pigment and / or pigment class from each sub-image, wherein the correlation indicates a contribution of at least one pigment / pigment class to the distribution of sparkle points within the corresponding image from which the sub-image has been cut out.

[0068] The processor can be further configured to derive a correlation of each sub-image of each measurement geometry with at least one pigment and / or pigment class by means of a convolutional neural network, which convolutional neural network is configured to associate (with a predetermined probability) each sub-image of the corresponding flash point of each measurement geometry with a specific pigment and / or a specific pigment class.

[0069] The processor may be further configured to use each derived correlation for each measurement geometry at which the respective sub-image was acquired to adjust / estimate / determine a contribution of at least one pigment and / or pigment class in determining the best matching recipe.

[0070] The proposed apparatus may comprise an output unit configured to output the determined optimal matching formula.

[0071] The proposed device is particularly configured to perform embodiments of the method described above.

[0072] Typically, at least the database (also referred to as the recipe database) and the at least one processor are networked with each other via corresponding communication connections. In the event that at least one of the measurement device, the filtering technique, and the convolutional neural network is a separate component (i.e., not implemented on the at least one processor), whether internal or external to the device, the database and the at least one processor are also networked with those components via corresponding communication connections, i.e., they communicate with each other. Each of the communication connections between the different components can be a direct connection or an indirect connection. Each communication connection can be a wired or wireless connection. Any suitable communication technology can be used. The recipe database and the at least one processor can each include one or more communication interfaces for communicating with each other. Such communication can be performed using a wired data transmission protocol, such as Fiber Distributed Data Interface (FDDI), Digital Subscriber Line (DSL), Ethernet, Asynchronous Transfer Mode (ATM), or any other wired transmission protocol. Alternatively, the communication can be performed wirelessly via a wireless communication network using any of a variety of protocols, such as General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), Code Division Multiple Access (CDMA), Long Term Evolution (LTE), Wireless Universal Serial Bus (USB), and / or any other wireless protocol. The corresponding communication may be a combination of wireless and wired communication.

[0073] The processor may include or be in communication with one or more input devices, such as a touch screen, audio input, motion input, mouse, keypad input, etc. In addition, the processor may include or be in communication with one or more output devices, such as audio output, video output, screen / display output, etc.

[0074] Embodiments of the present invention may be used with or incorporated into a computer system that may be a standalone unit or include one or more remote terminals or devices that communicate with a central computer located, for example, in the cloud, via a network (such as the Internet or an intranet). Thus, the processor and related components described herein may be part of a local computer system or a remote computer or online system, or a combination thereof. The recipe database and software described herein may be stored in a computer's internal memory or in a non-transitory computer-readable medium.

[0075] Within the scope of this disclosure, a database may be a part of a data storage unit or may represent the data storage unit itself.The terms "database" and "data storage unit" are used synonymously.

[0076] The present disclosure further relates to a non-transitory computer-readable medium having a computer program having program code configured to perform at least the following steps when the computer program is loaded and executed by at least one processor communicatively coupled to at least one measurement device, a database, a filtering technique, and a convolutional neural network:

[0077] A. Receive the color value, texture value and digital image of the target coating from the measurement device,

[0078] B. obtaining one or more preliminary matching formulas from a database including formulas for coating compositions and associated color values, associated texture values, and associated digital images based on the color values ​​and / or texture values ​​obtained for the target coating,

[0079] C. performing image analysis on each of the obtained images of the target coating and the images associated with the one or more preliminary matching formulations to determine at least one sparkle point within the respective images by using filtering techniques,

[0080] D. creating a sub-image of each flash point from the received image and from images associated / correlated with one or more preliminary matching recipes,

[0081] E. providing the created sub-images to a convolutional neural network, the convolutional neural network being trained to associate the corresponding sub-images of the corresponding glitter spots with pigments and / or pigment classes, and identifying the pigments and / or pigment classes based on the corresponding sub-images of the corresponding glitter spots,

[0082] F. determining and outputting statistical data of the identified pigments and / or pigment classes, respectively for the target coating and for each preliminary matching formula,

[0083] G. comparing the statistics determined for the target coating with the statistics determined for one or more preliminary matching formulations, and

[0084] H. Determine at least one of the one or more preliminary matching formulas as the formula that best matches the target coating.

[0085] The present invention is further defined in the following examples. It should be understood that these examples are given by way of illustration only, indicating preferred embodiments of the present invention. From the above discussion and examples, one skilled in the art will be able to ascertain the essential characteristics of the present invention and, without departing from its spirit and scope, may make various changes and modifications to adapt the present invention to various uses and circumstances. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 Possible image-based measurement geometries are shown with angles labeled according to standard multi-angle spectrometer and / or color camera terminology.

[0087] Figure 2 A flow chart of an embodiment of the proposed method is schematically shown.

[0088] Figure 3 Figure 3a An image of a target coating captured at a given image-based measurement geometry is shown in FIG. Figure 3b The filtering during image analysis is shown in Figure 3a The image, in Figure 3c Shown in Figure 3a The detected flash points within the image.

[0089] Figure 4 An embodiment of the proposed device is shown.

[0090] Figure 5 Schematically illustrates how a neural network training set is generated according to a possible embodiment of the proposed method.

[0091] Figure 6 It is schematically shown how a neural network training set is generated according to another possible embodiment of the proposed method.

[0092] Figure 7 A neural network used in another embodiment of the proposed method is schematically shown. DETAILED DESCRIPTION

[0093] Conventional spectrometers and image capture devices are considered Figure 1 A possible image-based measurement geometry is shown in FIG. 1 , with light sources 111 to 115 and a camera 120 . Figure 1Industry-recognized terminology is used to describe the angles of the light sources 111 to 115 relative to the specular reflection angle 100. Conventional mathematical standards are used herein. In various embodiments, conventional light sources 111 to 115 using diffuse or collimated color-corrected light can be used, and an image capture device (e.g., a color camera with appropriate resolution) 120 can be used to collect images of the target coating 130 by illuminating at one, some, or all of the identified or similar angles of the light sources 111 to 115.

[0094] After obtaining digital images of the target coating, it may be useful to first perform a preliminary analysis of the digital images to identify defects (such as scratches). Therefore, using an electronic computer processor operably connected to at least one filtering unit, a first image analysis is performed on the obtained digital images to locate and identify at least one bright region within each digital image by isolating image foreground data from image background data. Thereafter, for each digital image, a connected component analysis is performed to locate and identify at least one damaged region within the at least one bright region. If at least one damaged region is found, the at least one damaged region is masked from further analysis of the corresponding digital image, the corresponding digital image is rejected, and / or image capture is repeated.

[0095] In the subsequent image analysis process, a high-pass filter can be applied to each image of the target coating that has been obtained from the image capture device to determine the brightest spot in each pixel in the image. The resulting data / image can only include information about bright locations. The high-pass filter can convolve a value matrix with high-value center points and low-value edge points with the intensity information matrix of the image. This isolates high-intensity pixels that can be identified as flash points. To further refine the flash points, an edge detection filtering method can be applied in combination with intensity filtering. The same process is applied to each image in the image associated with one or more preliminary matching formulas obtained from the database.

[0096] Figure 2 An embodiment of the proposed method is schematically shown. Figure 2As shown in FIG, after measuring / obtaining color values, texture values, and a digital image of a target coating using at least one measuring device at step 10, one or more preliminary matching formulas are retrieved from a database at step 12. The database includes coating composition formulas and associated color values, associated texture values, and associated digital images. One or more preliminary matching formulas are retrieved based on the color values ​​and / or texture values ​​obtained for the target coating. Preliminary matching formulas are selected from a plurality of formulas in the database based on a first matching metric defined by a color difference metric between the target coating and a sample coating of a corresponding formula from the plurality of formulas. Formulas whose corresponding first matching metric with respect to the target coating is less than or equal to a predefined / pregiven threshold are selected as preliminary matching formulas. Only one formula may be selected as a preliminary matching formula. As a further criterion for selecting a preliminary matching formula, at least one texture difference metric, such as at least one of texture differences dSi, dSa, and dG, may be determined for a sample coating of a corresponding database formula associated with the target coating. The color difference and the at least one texture difference may be added, particularly by a weighted sum. At step 14, and as previously described, the obtained image of the target coating may be pre-analyzed to detect and mask damaged areas, such as scratches. Following this preliminary analysis, at step 16, image analysis as described above is used to determine the sparkle points for each of the acquired images of the target coating and each of the images associated with one or more preliminary matching formulas and retrieved from the database. This image analysis is performed using a computer processor operatively coupled with at least one filtering technique. Once the sparkle points have been identified and isolated, at step 18, at least one sub-image is created for each sparkle point in the acquired image and the images associated with the one or more preliminary matching formulas, respectively. At step 20, the created sub-images are provided to a convolutional neural network (CNN). The neural network is trained to associate the corresponding sub-images of the corresponding sparkle points with pigments and / or pigment classes and to identify (i.e., output) the pigments and / or pigment classes based on the corresponding sub-images of the corresponding sparkle points.

[0097] At step 22, corresponding statistics of the identified pigments and / or pigment classes are determined and output for the target coating and for each preliminary matching formula. This output can be performed using a display device such as a screen. At step 24, the statistics determined for the target coating are compared with the statistics determined for the one or more preliminary matching formulas. At step 26, at least one of the one or more preliminary matching formulas is determined to be the best match for the target coating.

[0098] Figure 3aAn original HDR (High Dynamic Range) image of the target coating is shown. In the image analysis, at a first step, the intensity values ​​of the original image of the target coating are analyzed and adjusted as needed in order to depict the structure of the image as well as possible. Thus, interfering factors that may be caused by uneven lighting conditions can be eliminated. Then, according to one possible embodiment of the proposed method, the image analysis uses image segmentation techniques to identify the positions of the flash points in the image (different algorithms can be used to identify the flash points and obtain information about the brightness and position of the different flash points). Creating a Figure 3b The image mask shown in , which identifies the locations of the flash points based on color, texture and / or their gradient. In doing so, the image can be converted into a binary image via thresholding, the binary image can be further segmented using continuous regions, and connected component detection can be performed on these parts. The flash points described by their corresponding brightness and position are copied and pasted into a blank image with a black background, thereby creating a Figure 3b The image mask shown in . Within the created image mask, each pixel of the image is marked with a "0" indicating that the pixel is not part of the flash point, or a "1" indicating that the pixel is part of the flash point. The flash point is clearly visible against the black background. This means that due to filtering / thresholding, the original image ( Figure 3a ) all the shining points appear here as Figure 3b White or light gray dots in the image mask shown in .

[0099] Contour detection of the image mask identifies the boundaries of connected pixels at each individual flash point location. For final review, the identified flash point contours are overlaid on the original image of the target coating, as shown in Figure 2. Figure 3c . For better illustration, the flash points are presented in a different color than the main color of the original image of the target coating. The main color of the original image of the target coating is red (gray in the black and white image), so the flash points are represented in green (light gray in the black and white image).

[0100] like Figure 5 and Figure 6 As shown in , a sub-image is created for all flash points identified in the image (i.e., in the image mask) by extracting the RGB data of each flash point for the associated pixel position from the original HDR image and placing these extracted RGB data in the center of a standard image frame, in which the RGB pixel data of the standard image frame is previously initialized with "0" (black) to provide a defined background.

[0101] Figure 4An embodiment of a device 400 is shown that can be used to identify the pigment and / or pigment class of a coating mixture of a target coating. A user 40 can utilize a user interface 41, such as a graphical user interface, to operate at least one measuring device 42 to measure the characteristics of a target coating 43, i.e., capture digital images of the target coating with the aid of a camera, each image being obtained at a different image-based texture measurement geometry (e.g., at a different angle), and determining the color and texture values ​​of the different spectral measurement geometries using, for example, a spectrophotometer. Data from at least one measuring device (e.g., camera 42) can be transferred to a computer 44, such as a personal computer, a mobile device, or any type of processor. The computer 44 can communicate with a server 46 via a network 45, i.e., be connected to a communication channel. The network 45 can be any type of network, such as the Internet, a local area network, an intranet, or a wireless network. The server 46 communicates with a database 47, which can store data and information used by the method of the embodiment of the present invention for comparison purposes. In various embodiments, the database 47 can be used in, for example, a client-server environment or, for example, a network-based environment, such as a cloud computing environment. The various steps of the method of the embodiment of the present invention can be performed by the computer 44 and / or the server 46. In another aspect, the present invention can be implemented as a non-transitory computer readable medium containing software for causing a computer or computer system to perform the above method. The software may include various modules for causing a processor and a user interface to perform the method described herein.

[0102] Figure 5A schematic diagram illustrates one possible method for generating a neural network training set that can be used to train a neural network to identify pigments and / or pigment classes of pigments in a coating mixture of a target coating. Typically, a limited number of toners / effect pigments are provided. For each effect pigment, a digital image, particularly an HDR image, is captured from a coating that includes only the corresponding effect pigment as a pigment. These digital images are associated with the corresponding effect pigment and stored in a database (i.e., directory 501). When training the neural network, each such digital image is segmented as described above to isolate the glitter points at step 502. At step 503, the identified glitter points of this digital image are overlaid onto the original image. At step 504, a sub-image for each glitter point is created from the digital image. To do this, for each glitter point, the RGB data for the associated pixel location is extracted from the original HDR image and placed at the center of a standard image frame whose RGB pixel data has been previously initialized with "0" (black) to provide a defined background. In the case shown here, sub-images 504-1, 504-2, and 504-3 are created from the digital image against a black background. As it is known which pigment the coating comprises, all created sub-images can be explicitly associated with the respective pigment and stored in the respective pigment folder 506 at step 505. The folders for all pigments are stored in a directory 507. Thus, the input neurons (i.e., input images), in particular the respective sub-images of the coating each comprising only one pigment, and the output neurons (i.e., the respective pigments comprised by the coating) are known and can be used to train the neural network.

[0103] Figure 6Schematically illustrates another possibility of how to generate a neural network training set that can be used to train a neural network to identify the pigments and / or pigment classes of the coating mixture of a target coating. Typically, a limited number of toners / effect pigments are provided. For each effect pigment, a digital image is captured from a coating that only includes the corresponding effect pigment as a pigment. These digital images are associated / correlated with the corresponding effect pigment and / or pigment class and stored in a database (i.e., directory 601). At step 602, when training the neural network, each such digital image is overlaid with a frame that moves across the image, i.e., from left to right and from top to bottom, from one pixel to another, to find pixels to isolate the flash point. At step 603, a sub-image of each flash point is created from the digital image. In the case shown here, sub-images 603-1, 603-2, and 603-3 are created from the digital image by extracting motion frames from the digital image. At step 604, if it is known which pigments the coating comprises, all created sub-images can be explicitly associated with the corresponding pigments and / or pigment classes and stored in folders 605 of the corresponding pigments and / or pigment classes. The folders of all pigments and / or pigment classes are stored in a directory 606. Thus, the input neurons as well as the output neurons are known and can be used to train the neural network.

[0104] Figure 7 A convolutional neural network 700 is shown that can be used to identify the pigments and / or pigment classes of a coating mixture of a target coating. A neural network used in this context is based on a learning process called backpropagation. The neurons of the neural network are arranged in layers. These include a layer with input neurons (input layer), a layer with output neurons (output layer), and one or more inner layers. The output neurons are the pigments or pigment classes of the coating formula to be determined. The input neurons for the neural network are sub-images created from previously determined flash points in a digital image of the target coating and / or a digital image associated with one or more preliminary formulas matching the target coating. Typically, a convolutional neural network (CNN) is a neural network that uses convolution instead of general matrix multiplication in at least one of its layers. A convolutional neural network consists of an input layer 701 and an output softmax layer 706, as well as feature learning and classification inner layers. The feature learning portion of a CNN typically consists of a series of convolutional layers 703, which perform convolution using multiplication or other dot products. The activation function is typically a RELU (rectified linear unit) layer, followed by a pooling layer 704 that reduces the dimensionality of the convolution. The classification portion of the CNN consists of a fully connected layer 702 and an output softmax layer 706 for calculating multi-class probabilities. The neural network 700 is trained using a backpropagation algorithm that minimizes the error between the actual output and the predicted output by adjusting the weights in the convolutional and dense layers when training examples are presented to the neural network.

[0105] The input neurons are given by sub-images 705 extracted from a digital image of the target coating and / or from an image taken from a database of images associated with one or more preliminary matching formulations. The neural network 700 has been previously described by Figure 5 and Figure 6 Those sub-images 705 are assigned to paints and / or paint classes 706 via the neural network 700 .

[0106] It will be appreciated that embodiments of the present invention can be used in conjunction with other methods of pigment identification using texture parameters (e.g., hue, intensity, size, and / or reflectance data). In various embodiments, to correctly identify the type of toner used in an unknown or target coating, or its offset, observation from the correct angle and comparison with existing known toners in a previously created database are required. Binary mixtures of toners can be generated to evaluate the effect of various concentrations of the toners on their sparkle color properties.

[0107] Reference Symbols List

[0108] 100 Mirror Angle

[0109] 111 to 115 light source

[0110] 120 Camera

[0111] 130 Target Coating

[0112] 10 Methods and Steps

[0113] 12 Methods and Steps

[0114] 14 Methods and Steps

[0115] 16 Methods and Steps

[0116] 18 Methods and Steps

[0117] 20 Methods and Steps

[0118] 22 Methods and Steps

[0119] 24 Methods and Steps

[0120] 26 Methods and Steps

[0121] 400 devices

[0122] 40 users

[0123] 41 User Interface

[0124] 42 Measuring equipment

[0125] 43 Target coating

[0126] 44 Computer

[0127] 45 Network

[0128] 46 servers

[0129] 47 Database

[0130] 501 Directory

[0131] 502 Flash Image Segmentation

[0132] 503 Outlines overlaid on HDR image

[0133] 504 Creation of sub-images for each flash point

[0134] 504-1 Flashing Point Sub-Image

[0135] 504-2 Flashing Point Sub-Image

[0136] 504-3 Flashing Point Sub-Image

[0137] 505 Associating sub-images with pigments

[0138] 506 folders

[0139] 507 Directory

[0140] 601 Directory

[0141] 602 Steps for moving frames on a digital image

[0142] 603 Creation of Sub-Images for Each Flash Point

[0143] 603-1 Sub-image

[0144] 603-2 Sub-image

[0145] 603-3 Sub-image

[0146] 604 Associating sub-images with pigments

[0147] 605 folder

[0148] 606 Directory

[0149] 700 Neural Networks

[0150] 701 Input Layer

[0151] 702 fully connected layers

[0152] 703 Convolution + RELU layer

[0153] 704 Pooling Layer

[0154] 705 sub-image

[0155] 706 softmax layer

Claims

1. A computer-implemented method comprising at least the following steps: - obtaining color values, texture values ​​and digital images of the target coating using at least one measuring device, - based on the color value and / or the texture value obtained for the target coating, retrieving one or more preliminary matching formulas from a database comprising formulas for coating compositions and associated color values, associated texture values ​​and associated digital images, - performing image analysis on each of the obtained images of the target coating and the images associated with the one or more preliminary matching formulations to determine at least one sparkle point within the respective images using a computer processor operatively combined with at least one filtering technique, - creating a sub-image of each flash point from the corresponding acquired image and from said corresponding image associated with said one or more preliminary matching formulas, - providing the created sub-images to a convolutional neural network, said convolutional neural network being trained to associate the respective sub-images of the respective glitter spots with pigments and / or pigment classes and to identify said pigments and / or pigment classes based on said respective sub-images of said respective glitter spots, - determining and outputting statistical data of the identified pigments and / or pigment classes for the target coating and for each preliminary matching formula, respectively, wherein, in the case of n flash points in a captured digital image of the target coating or associated with a preliminary matching formula and retrieved from a database, n sub-images are created, wherein a number S1 of sub-images are associated with pigment 1, a number S2 of sub-images are associated with pigment 2, and so on, up to a number Sk of sub-images associated with pigment k, wherein k is greater than 2 and less than or equal to n, wherein both k and n are integers, wherein the numbers S1, S2, ..., Sk together with the respective pigments 1, 2, ..., k are used for determining, for the target coating or for the respective preliminary matching formula, statistical data on the respective fractions of different pigments 1, 2, ..., k within the target coating or within the respective preliminary matching formula, - using a computer processor to compare the statistical data determined for the target coating with the statistical data determined for the one or more preliminary matching formulations, and - determining at least one of the one or more preliminary matching formulas as the formula that best matches the target coating.

2. The method of claim 1, further comprising deriving a correlation of at least one pigment from each sub-image, wherein The correlation is indicative of a contribution of the at least one pigment to the distribution of sparkle points within the respective image from which the sub-image has been cut out.

3. The method according to claim 1 or 2, wherein: The image analysis for each image includes: creating an image mask that identifies the locations of flash point locations within the corresponding image based on color, texture and their gradients; detecting the outline of the mask image by identifying the boundaries of the connected pixels of each individual flash point location and overlaying the identified flash point outlines on the corresponding image; and creating a sub-image of each flash point identified in the mask by extracting the RGB pixel data of each flash point at the associated pixel location from the corresponding image and placing those extracted RGB pixel data in the center of a standard image frame, in which the RGB pixel data of the standard image frame was previously initialized to "0", i.e., black, to provide a defined background.

4. The method according to claim 1 or 2, wherein: The correlation of each sub-image of each measurement geometry with at least one pigment is derived with the aid of the convolutional neural network, which is configured to classify each sub-image of the corresponding flash point of each measurement geometry to a specific pigment and / or pigment class with a predetermined probability.

5. The method according to claim 4, wherein Each derived correlation for each measurement geometry at which the corresponding sub-image is acquired is used to adjust the contribution of the at least one pigment and / or pigment class when determining the optimally matching recipe, wherein determining the optimally matching recipe includes providing a list of pigments with corresponding quantities and / or concentrations.

6. The method according to claim 1 or 2, wherein: Each sub-image is created using an image area of ​​a maximum size based on the at least one flash point in the black background.

7. A device for identifying effect pigments in a target coating, comprising at least: a database comprising formulations of coating compositions and associated color values, associated texture values ​​and associated digital images, - at least one processor in communication with at least one measurement device, the database, at least one filtering technique, and the convolutional neural network, and programmed to perform at least the following steps: a. receiving color values, texture values ​​and digital images of the target coating from the measuring device, b. obtaining one or more preliminary matching formulas from the database based on the color value and / or the texture value obtained for the target coating, c. performing image analysis on each of the obtained images of the target coating and the images associated with the one or more preliminary matching formulas to determine at least one sparkle point within the corresponding image by using the filtering technique, d. creating a sub-image of each flash point from the received image and from the image associated with the one or more preliminary matching recipes, e. providing the created sub-images to the convolutional neural network, the convolutional neural network being trained to associate the corresponding sub-images of the corresponding glitter spots with pigments and / or pigment classes, and identifying the pigments and / or pigment classes based on the corresponding sub-images of the corresponding glitter spots, f. determining and outputting statistical data of the identified pigments and / or pigment classes for the target coating and for each preliminary matching formula, respectively, wherein, in the event that n sparkles are present in a digital image of the target coating captured or associated with a preliminary matching formula and retrieved from a database, n sub-images are created, wherein a number S1 of sub-images are associated with pigment 1, a number S2 of sub-images are associated with pigment 2, and so on, up to a number Sk of sub-images associated with pigment k, wherein k is greater than 2 and less than or equal to n, wherein both k and n are integers, wherein the numbers S1, S2, ..., Sk together with the corresponding pigments 1, 2, ..., k are used to determine, for the target coating or for the corresponding preliminary matching formula, statistical data regarding the respective fractions of different pigments 1, 2, ..., k within the target coating or within the corresponding preliminary matching formula, g. comparing the statistical data determined for the target coating with the statistical data determined for the one or more preliminary matching formulations, and h. determining at least one of the one or more preliminary matching formulas as a formula that best matches the target coating.

8. The device according to claim 7, further comprising the at least one measurement device, the filtering technique and / or the convolutional neural network.

9. The apparatus according to claim 7 or 8, wherein The processor is further configured to perform the step of deriving a correlation of at least one pigment from each sub-image, wherein the correlation indicates a contribution of the at least one pigment to the distribution of sparkle points within the corresponding image from which the sub-image has been cut out.

10. The apparatus according to claim 7 or 8, wherein The processor is further configured to derive a correlation of each sub-image of each measurement geometry with at least one pigment by means of the convolutional neural network, wherein the convolutional neural network is configured to classify each sub-image of the corresponding flash point of each measurement geometry to a specific pigment and / or pigment class with a predetermined probability.

11. The apparatus according to claim 10, wherein The processor is further configured to use each derived correlation for each measurement geometry at which the corresponding sub-image is acquired to adjust the contribution of the at least one pigment and / or pigment class when determining the best-matching recipe, wherein determining the best-matching recipe includes providing a list of pigments having corresponding quantities and / or concentrations. 12 . The apparatus according to claim 7 , further comprising an output unit configured to output the determined best-matching recipe.

13. The apparatus according to claim 7 or 8, configured to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable medium having a computer program, the computer program having program code configured to, when the computer program is loaded and executed by at least one processor communicatively coupled to at least one measurement device, a database, a filtering technique, and a convolutional neural network, perform at least the following steps: A. receiving color values, texture values ​​and digital images of the target coating from the measuring device, B. retrieving one or more preliminary matching formulas from the database comprising formulas for coating compositions and associated color values, associated texture values ​​and associated digital images based on the color values ​​and / or the texture values ​​obtained for the target coating, C. performing image analysis on each of the obtained images of the target coating and the images associated with the one or more preliminary matching formulas to determine at least one sparkle point within the corresponding image by using the filtering technique, D. creating a sub-image of each flash point from the received image and from images associated with the one or more preliminary matching recipes, E. providing the created sub-images to the convolutional neural network, the convolutional neural network being trained to associate the corresponding sub-images of the corresponding glitter spots with pigments and / or pigment classes, and to identify the pigments and / or pigment classes based on the corresponding sub-images of the corresponding glitter spots, F. Determining and outputting statistical data of the identified pigments and / or pigment classes for the target coating and for each preliminary matching formula, respectively, wherein: In the case where there are n sparkles in the captured digital image of the target coating or associated with the preliminary matching formula and retrieved from the database, n sub-images are created, wherein a number S1 of sub-images are associated with pigment 1, a number S2 of sub-images are associated with pigment 2, and so on, until a number Sk of sub-images are associated with pigment k, wherein k is greater than 2 and less than or equal to n, wherein both k and n are integers, wherein the numbers S1, S2, ..., Sk together with the corresponding pigments 1, 2, ..., k are used to determine, for the target coating or for the corresponding preliminary matching formula, statistical data on the respective scores of different pigments 1, 2, ..., k within the target coating or within the corresponding preliminary matching formula, G. comparing the statistical data determined for the target coating with the statistical data determined for the one or more preliminary matching formulations, and H. Determining at least one of the one or more preliminary matching formulas as the formula that best matches the target coating.

Images