Methods and apparatus for employing deep learning deployment and image similarity metrics

Abstract

The invention relates to a method and apparatus that can search for one or more optimal matching coating formulas in a database based on measured data of a target coating, i.e. search for one or more preliminary matching formulas within the database; and can refine the search using deep learning techniques using an image similarity measure between an image of the one or more optimal matching coating formulas on one side and an image of the target coating on the other side.

Description

Technical Field

[0001] The present invention relates to a computer-implemented method and apparatus for employing deep learning to deploy and use an image similarity metric between two HDR effect coated images. Background Technology

[0002] Today, in the color search and acquisition process, in addition to color information, additional features such as roughness, flash area, flash intensity, flash grade, and / or flash color variation / distribution are used as auxiliary conditions to find the optimal solution for a given target color / coating. These additional features are measures of different visual characteristics of the color texture appearance.

[0003] These additional features are typically derived from modern spectrometer instruments (such as Xrite). Or Byk Mac The raw image data of the target coating is captured. This raw image data is then processed by image processing algorithms. As the output of these algorithms, texture values ​​that should represent the optical properties of the target coating texture are obtained. These texture values ​​are classified according to known industry standards.

[0004] Due to the nature of complex coating mixtures, it can sometimes be difficult to formulate, identify, and / or search for acceptable matching formulations and / or pigment deposits. Ideally, a human could examine a complex coating mixture and determine the appropriate pigments within it. However, in practice, the pigments in a coating mixture may not be readily available in the set of tints used to create a matching coating in a paint system. Therefore, color-matching technicians must determine whether the paint system contains an appropriate offset, and if so, given that the offset is not exactly the same as the original pigment deposit, they must determine the additional modifications needed to accommodate the offset.

[0005] The reflectance (spectral data) and texture (image data) of the colored coating are measured from several geometries. Chromaticity data is derived from spectral data, and texture features are derived from image data. Known techniques using cameras and / or spectrometers, optionally combined with microscopic evaluation of the target coating, are generally not adequately defined as effective for resolving novel effects of pigment deposition or complex mixtures, and primarily focus on the individual evaluation of the target coating—that is, on a case-by-case analysis—a very time-consuming process because each new, unknown target coating needs to go through all analytical steps. Therefore, this time-consuming process may not satisfactorily solve application problems requiring efficient analysis of the target coating in conjunction with matching formulas.

[0006] Further strategies exist that use paint or virtual samples representing various textures and then compare them to unknown target coatings. However, this technique typically requires significant user intervention and is therefore subjective, which can lead to inconsistent results.

[0007] Color measurements of existing colors and their corresponding formulas are stored in a formula database. The color search / retrieval process is typically initiated by searching this database.

[0008] Then, formulas whose color measurements are close are selected as preliminary matching results / formulas and displayed to the user in a specific order, determined by a ranking function that uses chromaticity data and texture feature data as input. The purpose of this ranking function is to place the optimal formula for overall appearance at the top of the results list.

[0009] Individual spectral data provides acceptable matching quality from multiple perspectives, but often results in an unacceptable overall appearance due to texture and flash bias. As described by Byk-Gardner, existing techniques use spectrophotometers with black-and-white image capabilities to calculate texture values ​​Gdiff (diffuse graininess or roughness), Si (flash intensity), and Sa (flash area). This is a good first step for characterizing the texture of a target coating, but has limited ability to identify the overall appearance of the target coating. Recently, color images have become available on X-Rite spectrophotometers. The patent “PIGMENTIDENTIFICATION OF COMPLEX COATING MIXTURES WITH SPARKLE COLOR” (US 2017 / 0200288 A1) improves the technique by adding hue analysis to determine the flash color distribution. This advancement improves color acquisition performance, but utilizes only a limited set of features in the image, rather than all embedded image and effect features. Summary of the Invention

[0010] There is a need for a method and an apparatus that can measure an unknown target coating, search a database for one or more optimal matching coating formulas based on the measurement data of the target coating (i.e., search for one or more initial matching formulas within the database), and refine the search using deep learning techniques by deploying and using an image similarity metric between images of one or more optimal matching coating formulas on one side and images of the target coating on the other side. However, currently known systems can only use known (i.e., measurable) features of the target coating, such as spectral and textural features, but generally cannot help consider the hidden features required to match an unknown target coating.

[0011] The above-mentioned objective is achieved by the following method and apparatus. Further embodiments are given in the following description.

[0012] This disclosure proposes a computer-implemented method to improve the color matching acceptance rate of applied effect coatings when searching / retrieving a recipe database with spectral and HDR image sets.

[0013] This disclosure relates to a computer-implemented method comprising at least the following steps:

[0014] a) Using at least one measuring device, obtain at least one color value, digital image, and optional texture value for each of at least one measuring geometry, of the training target coating.

[0015] b) Provide a database comprising formulas for coating compositions and, for each of at least one measurement geometry, interrelated color values ​​(spectral data), interrelated digital images (e.g., HDR images), and optionally interrelated texture values.

[0016] c) For each training target coating, the processor retrieves a list of multiple preliminary matching formulas from the database based on the obtained color values ​​and optionally on the obtained texture values ​​for the corresponding training target coating.

[0017] d) Using visual inspection of digital images of at least one training target coating and digital images correlated with preliminary matching formulas and obtained from a database, divide the list of multiple preliminary matching formulas into two sublists. The first sublist includes visually good matching formulas from the multiple preliminary matching formulas, and the second sublist includes visually poor matching formulas from the multiple preliminary matching formulas.

[0018] e) For each training target coating, a processor creates multiple triples, each triple including a digital image of the corresponding training target coating for at least one measurement geometry, a digital image associated with visually good matching formulas from a database for that measurement geometry in at least one measurement geometry, and a digital image associated with visually poor matching formulas from a database for that measurement geometry in at least one measurement geometry.

[0019] f) The convolutional neural network is trained by feeding the created triples one after another as corresponding inputs and optimizing an n-dimensional cost function that defines the similarity distance to at least one training target coating as an image similarity metric, such that the cost function minimizes the formula for visually good matches and maximizes the formula for visually bad matches.

[0020] g) Make the trained neural network available in the processor for sorting digital images of the coating composition with respect to digital images of the target coating.

[0021] The terms “formula” and “recipe” are used as synonyms in this document. The term “operational combination” refers to the communication of corresponding components with each other in a manner in which the corresponding components can exchange data with each other. The terms “associated with” and “interrelated with” are used synonymously. Both terms indicate the wholeness of components that are associated / interrelated with each other.

[0022] Color values ​​(spectral data) are obtained by analyzing the spectral curves of at least one coating (e.g., at least one training target coating) using at least one measuring device. These spectral curves are measured with respect to different measurement geometries of the respective coating surface (e.g., the training target coating applied to the surface of a sample substrate). Typically, the spectral measurement geometry is defined by the illumination direction / angle and the observation direction / angle. A typical spectral measurement geometry is a fixed illumination angle of 45° and viewing angles of -15°, 15°, 25°, 45°, 75°, and 110° relative to the surface normal of the coating, each angle being measured relative to the specular reflection angle (i.e., the specular reflection direction), defined as the angle between the specular reflection direction and the coating surface normal, and the outgoing direction of the corresponding ray incident at the same angle to the coating surface normal.

[0023] Image and / or texture values ​​are obtained by capturing multiple digital images (particularly HDR color images) using an image capture device, each digital image being obtained with respect to the surface of a corresponding coating (e.g., a corresponding training target coating) using a different measurement geometry. A typical image-based texture measurement geometry is a fixed position of the image capture device (i.e., the camera) at 15° to the nominal (i.e., normal) of the training target coating surface. Irradiation angles are selected as r15as-15, r15as15, r15as-45, r15as45, r15as80, and from X-Rite. The defined semi-diffuse angle. "Semi-diffuse" here means "as diffuse as possible" with respect to the measuring device and its spatial dimensions. Regarding the geometric nomenclature, at least one measuring device (e.g., a camera) and the position of illumination are opposite. This means that the specular reflection angle is defined here by the fixed position of the camera. Specifically, this means, for example, that the name "r15 as-15" uses "r" to indicate "reverse," "15" to indicate the fixed position of the camera, i.e., at a 15° angle to the nominal (i.e., normal) surface of the training target coating, "as" to indicate "aspecular," and "-15" to indicate the angle of illumination measured relative to the specular reflection angle.

[0024] Texture values / parameters, particularly the flash rating SG, flash color variation CV, roughness C or grain size G, flash intensity Si, and flash area Sa of the target coating.

[0025] The at least one measuring device can be selected as a spectrophotometer, such as, for example, Xrite. Xrite Or Byk Mac This spectrophotometer can also be combined with other suitable equipment (such as microscopes) to obtain more image data, such as microscope images.

[0026] This database is a formulation database, which includes formulas for coating compositions and interrelated chromaticity data. For each formula, the interrelated chromaticity data includes spectral data, i.e., color values ​​of the sample coating (applied to the surface of the sample substrate) based on the corresponding formula, digital images, and optional texture values.

[0027] The proposed method provides a model with higher learning capabilities compared to models based on handcrafted features. This method identifies an effective image similarity metric that allows for the efficient finding of images similar to the target image during a per-example search process. The resulting image similarity metric should be correlated with human perception. The method is capable of learning fine-grained image similarity using a deep learning model.

[0028] A preliminary matching formula is selected from multiple formulas in the database based on a first matching metric. The first matching metric is defined / calculated by a color difference metric (e.g., CIE dE*) between the target coating (e.g., the training target coating) and the corresponding sample coating of all or at least some of the aforementioned spectral measurement geometries, and optionally supplemented by at least one texture difference metric, such as at least one of texture differences dSi, dSa, dG as defined by Byk-Gardner [“Beurteilung von Effektlackierungen,Den Gesamtfarbeindruckobjektiv messen”, Byk-Gardner GmbH]. The color difference metric and at least one texture difference metric can be added, optionally by weighted addition. The color difference metric can be a CIE with three color values. Description: Where L* represents the brightness from black (0) to white (100), a* represents the brightness from green (-) to red (+), and b* represents the brightness from blue (-) to yellow (+). Therefore, formulas whose measurements are close in terms of color and optional texture are selected as the initial matching formulas and displayed to the user in a list in a specific order, given by a sorting function that uses chromaticity data and optional texture feature data as input. The sorting function is chosen such that the optimal formula for spectral values ​​and optional texture values ​​is at the top of the list.

[0029] In addition, the database includes multiple digital images, particularly HDR color images, each obtained with respect to the surface of the corresponding coating with a different measurement geometry.

[0030] The phrase “communicate with” indicates that a communication connection exists between the relevant components.

[0031] For each training target coating, after obtaining a list of multiple preliminary matching formulas from the formula database based on the obtained color values ​​and optionally also based on the obtained texture values ​​for the corresponding training target coating, the list of multiple preliminary matching formulas for the corresponding training target coating is divided into two sublists using visual inspection of digital images of the corresponding training target coating and digital images associated with the preliminary matching formulas and obtained from the database. The first sublist includes visually good matching formulas of the multiple preliminary matching formulas (i.e., the associated digital images are visually close to the corresponding digital images of the training target coating), and the second sublist includes visually poor matching formulas of the multiple preliminary matching formulas (i.e., the associated digital images are visually not close to the corresponding digital images of the training target coating).

[0032] According to one aspect, 500 to 1000, specifically 1000, different training target coatings are selected, and for each training target coating, a number of N preliminary matching formulas are chosen, where N is an integer, specifically a natural number. This means using a set of approximately 500 to 1000 training target images to train a convolutional neural network (CNN). For each training target image, a list with multiple (N) preliminary matching formulas is obtained from a formula database. This list includes, for example, 20 preliminary matching formulas, i.e., N = 20. Typically, these 20 preliminary matching formulas are the 20 optimal matching formulas with acceptable matching and similar pigment deposition for the corresponding target images of the training target coatings. Multiple preliminary matching formulas can be selected such that for each preliminary matching formula, the color-based distance (such as dE) and optional texture-based distances (such as dS, dG, dSa, dSi) to the corresponding training target coating, or the sum of color- and texture-based distances (such as ∑(dE, dS, dG, dSa, dSi)), is less than a first threshold.

[0033] The list of target images for each training target coating is then subdivided into a first sublist and a second sublist. The first sublist includes visually good matching formulas from multiple preliminary matching formulas (i.e., the correlated digit images are visually close to the corresponding digit images of the training target coating), and the second sublist includes visually poor matching formulas from multiple preliminary matching formulas (i.e., the correlated digit images are visually less close to the corresponding digit images of the training target coating). Multiple triples are then created, each triple including the corresponding target image of the corresponding training target coating, the digit images from the first sublist, and the digit images from the second sublist. For each target image, the images from the first and second sublists can be randomly combined and fused with the corresponding target image to form a triple.

[0034] A model, namely a neural network, was developed to address both image similarity (complete image) and effect pigment similarity (image showing effect pigments against a black background). Effect pigments were then identified using image segmentation methods. Possible segmentation techniques included thresholding methods, edge-based methods, clustering methods, histogram-based methods, neural network-based methods, and hybrid methods.

[0035] Use the first sublist D of each training objective coating i+ Second sublist D i- Create multiple triples, where each triple includes a corresponding training target coating C for at least one measurement geometry. i digital image d i (where i is a natural integer, i>0), for at least one measurement geometry, the measurement geometry is obtained from the database along with the first sublist D. i+ Visually well-matched formulas for correlated digital images d i+ And, for at least one measurement geometry, the measurement geometry obtained from the database along with the second sublist D i- Visually poor matching formulas for correlated digital images d i- Image similarity relationships are characterized by the relative similarity ranking within the created triples; that is, image similarity relationships are labeled using the created triples. The created triples (d...) i ,d i+, d i- The data is fed into the convolutional neural network, which then creates triples (d...). i ,d i+, d i- ) as input. Triples (d i ,d i+, d i- ) characterizes the corresponding digital image d i ,di+ ,d i- The relative similarity ranking order. The goal is to learn an n-dimensional embedding function f that assigns smaller distances F (where n is a natural integer in the n-dimensional parameter / feature space) to more similar image pairs, i.e., for all available training target coatings C. i , F(f(d) i ),f(d i+ )) <F(f(d i ),f(d i- This means that (by considering all extractable, measurable, and hidden features of the corresponding image) an n-dimensional cost function F is to be optimized, defining the similarity distance with at least one training target coating, such that the cost function F minimizes the formula for the corresponding visually good match and maximizes the formula for the corresponding visually poor match. It should be understood that the n-dimensional embedding function f will embed each image d... i ,d i+ ,d i- Mapping to n-dimensional parameter / feature space R n The corresponding points in the image space are represented by feature maps, where each dimension represents a feature, and each feature may be weighted by corresponding factors, and F is the distance in this space. The smaller the distance F between two images, the more similar the two images are. The goal of the proposed method is to learn an embedding function f that assigns smaller distances F to more similar image pairs, i.e., considering as many features as possible—measurable and hidden features—that influence visual appearance in the embedding function f. A deep neural network f computes the distance between two images, d. i Embedding: f(d) i )∈R n , where n is the dimension of the feature embedding.

[0036] According to the present invention, a reverse image search algorithm with convolutional neural networks for deep learning is proposed to compare the similarity between two HDR effect coating images (i.e., two digital images that are respectively associated with the target coating and the sample coating).

[0037] For image classification, convolutional neural networks have the following advantages over previously disclosed parametric methods:

[0038] 1. Current industry standard color and effect parameters do not fully describe visual evaluation;

[0039] 2. Convolutional neural networks automatically and adaptively learn multi-level hidden image features, and are not limited to defined color and effect parameters;

[0040] 3. Convolutional neural networks can continue to learn from a large and ever-growing amount of available recipes and measurement data.

[0041] A convolutional neural network was trained to compute the embedded image and effect pigment features of the target coating and implemented in a search / retrieval algorithm based on a similarity metric.

[0042] The trained neural network f is eventually available in the processor to sort digital images of the coating composition (i.e., the sample coating) based on digital images of the target coating.

[0043] According to one embodiment of the proposed method, step e. includes an additional step ee., which ee. performs the following operations for at least one of the created triples: rotating the digital image of the corresponding training target coating by a given angle around the central image rotation axis, rotating the digital image associated with the corresponding visually good matching formula obtained from the database by a given angle around the central image rotation axis, and rotating the digital image associated with the corresponding visually bad matching formula obtained from the database by a given angle around the central image rotation axis, thereby obtaining at least one additional triple for the corresponding training target coating and based on at least one triple.

[0044] According to another embodiment, step ee is repeated several times by changing the given angle.

[0045] Therefore, the given angles can be selected from the following groups: 0°, 30°, 60°, 90°, 120°, 150°, 180°, 210°, 240°, 270°, and 300°.

[0046] According to another embodiment of the proposed method, step e. includes an additional step ee., which is performed for at least one of the created triples: dividing the digital image of the corresponding training target coating into multiple blocks, dividing the digital image obtained from the database and associated with the corresponding visually good matching formula into the same number of corresponding blocks, and dividing the digital image obtained from the database and associated with the corresponding visually poor matching formula into the same number of corresponding blocks, thereby obtaining additional triples corresponding to the number of blocks for the corresponding training target coating and based on at least one triple.

[0047] According to another embodiment of the proposed method, step f. includes setting an n-dimensional cost function F for training the convolutional neural network, such that it is applied to a vector f(b) and defined by n parameters a. i The n-dimensional embedding function f of the digital image b defined by the corresponding preliminary matching formula (where 0≤i≤n) is mapped to a scalar s(b) (i.e., distance values), and each component f of the vector f(b) is... i(That is, each dimension) includes the values ​​of the features of the corresponding preliminary matching formula, where each parameter a i Define the weights for the corresponding features.

[0048] According to another aspect, in step e, the digital image is preprocessed using an image segmentation method for identifying effect pigments, and the digital image is imaged using only a black background for the effect pigments.

[0049] The present invention also relates to a computer-implemented method comprising at least the following steps:

[0050] A. Using at least one measuring device, for at least one measuring geometry, obtain the color values, digital image, and optional texture values ​​of the target coating.

[0051] B. Provide a database comprising formulas for coating compositions and, for the at least one measurement geometry, interrelated color values ​​(e.g., spectral values), interrelated digital images (e.g., HDR images), and optionally interrelated texture values ​​(e.g., graininess, particle size, flash intensity, flash area, ...).

[0052] C. Provide the computer with a trained convolutional neural network.

[0053] D. For the target coating of at least one measurement geometry, based on the obtained color values ​​and optionally based on the obtained texture values, select one or more preliminary matching formulas from the database.

[0054] E. For each of the one or more preliminary matching formulas and for the at least one measurement geometry, retrieve digital images from the database that are associated with the corresponding preliminary matching formula.

[0055] F. For each of one or more preliminary matching formulas, provide the trained convolutional neural network with (input) a corresponding digital image associated with the corresponding preliminary matching formula, along with digital images obtained for the target coating and for the at least one measurement geometry.

[0056] G. For each of one or more preliminary matching formulas, a trained neural network is used to determine the similarity distance between the target coating and the corresponding preliminary matching formula. This trained neural network is trained to compute the similarity distance in an embedded feature layer between two digital images: the corresponding acquired digital image associated with the corresponding preliminary matching formula and the digital image acquired for the target coating.

[0057] H. Using an output device integrated with computer operation, output to the user one or more preliminary matching formulas to determine the similarity distance.

[0058] According to one aspect, the method further includes displaying the determined similarity distance of one or more preliminary matching formulas on a display interface of a display monitor connected in communication with a computer.

[0059] According to another embodiment of the proposed method, the determined similarity distances of one or more preliminary matching formulas are displayed in the form of a sorted list, wherein the lower the similarity distance, the better the corresponding preliminary matching formula matches the target coating. The preliminary matching formula with the minimum similarity distance is placed at the top of the sorted list.

[0060] This disclosure also relates to a computer-implemented method for ranking digital images of a sample coating and a target coating using a computer with a trained convolutional neural network, the trained convolutional neural network being trained to calculate similarity distances in embedded feature layers of the trained convolutional neural network between the two digital images.

[0061] Provided is a trained convolutional neural network, namely a trained convolutional neural network as described herein.

[0062] The present invention also relates to a device comprising at least:

[0063] - A database comprising formulas for coating compositions and interrelated color values, interrelated digital images, and optionally interrelated texture values ​​for at least one measurement geometry.

[0064] - At least one processor, which is communicatively connected to at least one measurement device, a database, and a convolutional neural network, and is programmed to perform at least the following steps:

[0065] 1. Receive color values ​​of the target coating, at least one digital image, and optional texture values ​​from at least one measuring device, and

[0066] 2. Performing steps c. to g. of the above-described method for training a convolutional neural network and / or steps D. to H. of the above-described method for using a trained neural network.

[0067] A convolutional neural network can be implemented on at least one processor, or it can be mounted on a separate computing device that is communicatively connected to at least one processor. The terms "computer," "processor," and "computing device" are used synonymously.

[0068] This disclosure further relates to a non-transitory computer-readable medium having a computer program having program code configured to perform at least the following steps when the computer program is loaded and executed by at least one processor, wherein the at least one processor is communicatively connected to at least one measuring device, the at least one measuring device being configured to provide color values, digital images, and optional texture values ​​of at least one target coating for each of at least one measuring geometry; communicatively connected to a database including formulas for coating compositions, and interrelated color values, interrelated digital images, and optionally interrelated texture values ​​for the at least one measuring geometry; and communicatively connected to a convolutional neural network.

[0069] 1. Receive from a measuring device color values, a digital image, and optional texture values ​​for at least one target coating for at least one measuring geometry, and

[0070] 2. Perform steps c. to g. of the above-described method for training a convolutional neural network and / or steps D. to H. of the above-described method for using a trained neural network.

[0071] The proposed method can be performed additionally, particularly after other pigment identification methods using flash color distribution and / or flash size distribution. This method is described, for example, in European applications US 2017 / 0200288A1 and 19154898.1, the contents of which are incorporated herein by reference in their entirety.

[0072] Finally, the optimal matching formula is identified (e.g., at the top of the sorted list) and forwarded to the mixing unit, which is configured to produce / mix the paint / coating composition based on the identified optimal matching formula. The mixing unit produces this paint / coating composition, which can then be used at the location of the target coating. The mixing unit can be a component of the proposed device.

[0073] The convolutional neural network that can be used with the proposed method is based on a learning process called backpropagation. The neurons in a convolutional neural network are arranged in layers. These layers include layers with input neurons (input layers), layers with output neurons (output layers), and one or more inner layers. Downstream of the neural network is a ranking layer that outputs a corresponding similarity distance (similarity difference), which determines / predicts the target coating against the corresponding sample coating. The convolutional neural network used is based on a learning process called backpropagation. Backpropagation should be understood here as a general term for a supervised learning process via error feedback. Several backpropagation algorithms exist: for example, Quickpropagation, Resilient Propagation (RPROP). This process uses a neural network consisting of at least three layers: a first layer with input neurons, an nth layer with output neurons, and (n-2) inner layers, where n is a natural number greater than 2. The terms "convolutional neural network" and "neural network" are used synonymously here.

[0074] During the training phase, the input neurons for the neural network, used as training data, are triplets of images of sample coatings and training target coatings. Each sample coating is based on a formula previously identified from the database as a preliminary matching formula for the training target coating. Each triplet contains the target image d of the training target coating. i A positive image d with a preliminary (visually) good matching formula i+ The negative image d of the initial (visual) poor matching formula i- Triplet (d) i ,d i+ ,d i- This represents the relative similarity relationship among the three images.

[0075] The inner layers of a convolutional neural network consist of all or a subset of convolutional layers, max-pooling layers, local normalization layers, and fully connected dense layers. A convolutional layer takes an image or feature map from another layer as input, convolves it with a set of p learnable kernels, and passes it through an activation function to generate p feature maps. A convolutional layer can be thought of as a set of local feature detectors. Max-pooling layers are responsible for reducing the dimensionality of the convolutional features. Local normalization layers normalize the feature maps around their local neighborhood, giving them unit norm and zero mean. This results in feature maps that are robust to differences in illumination and contrast. Stacked convolutional, max-pooling, and local normalization layers act as translation- and contrast-robust local feature detectors. Dense layers are the standard fully connected set of neurons in a neural network that map the feature maps from the convolutional and max-pooling layers to points in an n-dimensional feature space, obtained by an n-dimensional embedding function f defined by the neural network and applied to a sample layer of the digital image. The similarity distance value of the target coating (i.e. the value of the cost function F) is the distance between two points in the n-dimensional feature space, that is, the distance between the point assigned to the target coating and the point assigned to the corresponding sample coating.

[0076] The phrase "formula for coating composition and associated image" refers to the formula for coating composition and the captured image of the corresponding coating. The phrase "image associated with one or more preliminary matching formulas" refers to the captured image of the corresponding coating (applied to the substrate) for one or more preliminary matching formulas.

[0077] The proposed device may include an output unit configured to output the determined similarity distance.

[0078] The proposed device is specifically configured to perform the above-described method.

[0079] Typically, at least a database (also known as a recipe database) and at least one processor are networked to each other via corresponding communication connections. In cases where at least one measuring device and the convolutional neural network are independent components (i.e., not implemented on at least one processor), whether internal or external components of the device, the database and at least one processor are also networked to these components via corresponding communication connections, i.e., they communicate with each other. Each of the communication connections between the different components can be a direct or indirect connection, respectively. Each communication connection can be wired or wireless. Every suitable communication technology can be used. The recipe database and at least one processor can each include one or more communication interfaces for communicating with each other. This communication can be performed using wired data transmission protocols such as Fiber Distributed Data Interface (FDDI), Digital Subscriber Line (DSL), Ethernet, Asynchronous Transfer Mode (ATM), or any other wired transmission protocol. Alternatively, the communication can be wireless via a wireless communication network using any of a variety of protocols, such as General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), Code Division Multiple Access (CDMA), Long Term Evolution (LTE), Wireless Universal Serial Bus (USB), and / or any other wireless protocol. The corresponding communication can be a combination of wireless and wired communication.

[0080] The processor may include one or more input devices, such as touchscreens, audio inputs, mobile inputs, mice, keypad inputs, etc., or may communicate with them. Additionally, the processor may include one or more output devices, such as audio outputs, video outputs, screen / display outputs, etc., or may communicate with them.

[0081] Embodiments of the present invention can be used with or incorporated into a computer system, which may be a standalone unit or include one or more remote terminals or devices that communicate with a central computer located, for example, in the cloud, via a network (such as, for example, the Internet or an intranet). Therefore, the processors and related components described herein may be part of a local computer system, a remote computer, an online system, or a combination thereof. The recipe database and software described herein may be stored in the computer's internal memory or in a non-transitory computer-readable medium.

[0082] Within the scope of this disclosure, a database may be part of a data storage unit or may represent the data storage unit itself. The terms "database" and "data storage unit" are used synonymously.

[0083] The invention is further defined in the following examples. It should be understood that these examples are given by way of illustration only, indicating preferred embodiments of the invention. Through the above discussion and examples, those skilled in the art can determine the essential features of the invention, and various changes and modifications can be made to adapt it to various uses and conditions without departing from its spirit and scope. Attached Figure Description

[0084] Figure 1 A flowchart illustrating an embodiment of the proposed method for training neural networks is shown schematically.

[0085] Figure 2 schematically illustrates a flowchart of an embodiment of the proposed method for using a trained neural network trained according to an embodiment of the proposed training method.

[0086] Figure 3 Triples of images are shown for training a neural network according to an embodiment of the proposed training method.

[0087] Figure 4 An embodiment of the proposed device is illustrated schematically. Detailed Implementation

[0088] Figure 1 A flowchart schematically illustrates an embodiment of the proposed method for training a neural network 100.

[0089] Based on the color and / or texture values ​​obtained for the corresponding training target coating, for each training target coating C i (where 1≤i≤l) After obtaining a list of multiple preliminary matching formulas from the formula database, the digital image d of the corresponding training target coating is used. i The images are correlated with the preliminary matching formula and obtained from the database. Visual inspection (where 1≤k≤m, 1≤j≤p) divides the list of multiple preliminary matching formulas for the corresponding training target coating into two sublists D. i+ D i- The first sublist D i+ This includes visually good matching formulas for multiple preliminary matching formulas for the corresponding training target coatings, and a second sublist D. i- Visually poor matching formulas, including multiple preliminary matching formulas.

[0090] Based on one aspect, select 500 to 1000, especially 1000, different training target coatings C. i That is, 500≤i≤1000, and for each training target, coating C iWe select N initial matching formulas, where N is an integer and m + p = N. This means that the convolutional neural network 100 is trained on a set of approximately 500 to 1000 target images i. For each target image d i A list of multiple (N) preliminary matching formulas is obtained from a formula database. This list includes, for example, 20 preliminary matching formulas, i.e., N=20. Typically, these 20 preliminary matching formulas are 20 optimal matching formulas, which, considering color values ​​and optional texture values, have an acceptable match and similar pigment deposition for the corresponding training target coating. Multiple preliminary matching formulas can be selected such that, for each preliminary matching formula, the color-based distance (such as dE) and optional texture-based distances (such as dS, dG, dSa, dSi) to the corresponding training target coating, or the sum of color- and texture-based distances (such as ∑(dE, dS, dG, dSa, dSi)), is less than a first threshold.

[0091] Then the coating C used for the corresponding training target i and its corresponding target image d i The list is subdivided into the first sublist D i+ Second sublist D i- The first sublist D i+ Including m visually good matching formulas from multiple preliminary matching formulas, the digital image 110 is designated as And the second sublist D i- Including p visually poor matching formulas in multiple preliminary matching formulas, whose digital image 110 is specified as Then, in step 111, multiple triples 120 are created, each triple... (where 1≤k≤m and 1≤j≤p) includes the corresponding target image d. i First sublist D i+ Digital images Second sublist D i- Digital images For each target image d i The first sublist D can be... i+ Second sublist D i- Image Randomly combine and match with the corresponding target image d i Fusion to form triples Where 1 ≤ k ≤ m and 1 ≤ j ≤ p. Preferably, the same measurement geometry is used to capture the corresponding three digital images combined into triplets.

[0092] Models, i.e., neural networks, have been developed for both image similarity (for full images) and effect pigment similarity (for effect pigments with only a black background). Image segmentation methods are used to identify effect pigments. Possible segmentation techniques include thresholding methods, edge-based methods, clustering methods, histogram-based methods, neural network-based methods, hybrid methods, etc.

[0093] For each training target coating C i Using the first sublist D i+ And the second sublist D i- , multiple triples are created Where 1 < k ≤ m and 1 < j ≤ p and m, p > 1, as shown by arrow 111, where each triple Includes the corresponding training target coating C for one of the at least one measurement geometries i Of the digital image d i (where i is a natural integer, i > 0), the digital image obtained from the database for this measurement geometry in the at least one measurement geometry and correlated with the visually good matching formula of the first sublist D i+ And the digital image obtained from the database for this measurement geometry in the at least one measurement geometry and correlated with the visually poor matching formula of the second sublist D And the digital image obtained from the database for this measurement geometry in the at least one measurement geometry and correlated with the visually poor matching formula of the second sublist D i- Of the digital image The visual image similarity relationship is characterized by the relative similarity ranking in the created triples, i.e., the similarity relationship of the images is marked with the created triples. The created triples Are fed into the convolutional neural network 100, i.e., the convolutional neural network takes the created triples As input. Thus, the triples Are fed into the neural network 100 one by one, as shown by arrow 112. This means that the running parameters k, j increase sequentially, where k = 1,..., m and j = 1,..., p. The triples Characterize the relative similarity ranking order for the correspondingly included digital image d i , Of

[0094] The aim is to learn the n-dimensional embedding function f101, i.e., the neural network 100 that assigns a smaller distance defined by the similarity distance function F 106 (in the n-dimensional parameter / feature space R n In which n is a natural integer) to more similar image pairs, i.e., for all available training target coatings C i , The n-dimensional embedding function f101 is implemented by a set of network layers of neural network 100. The distance function F106 represents the cost function and is implemented by a ranking layer, which is the final output layer of neural network 100. The n-dimensional embedding function f101 first sets the corresponding triples... Each digital image d i , In n-dimensional space R n Assign exactly one point f(d) i ), This means that in n-dimensional space R n In the middle, d i Assigned to f(d) i 102, Assigned to 103, and Assigned to 104. Assign the corresponding training target coating C i digital image d i The point f(d) i )102 and the digital image assigned to the corresponding initially well-matched coating point The distance between 103 and / or assigned to the corresponding training target coating C i digital image d i The point f(d) i )102 and the digital image of the matching coating assigned to the corresponding initial defect point The distance between 104 These are the corresponding training target coatings C i A measure of similarity between the coating and the corresponding preliminary matching coating. This distance is defined by an n-dimensional cost function F = 10⁶.

[0095] This means optimizing the definition of the coating C at least one training target distance (by considering all extractable, measurable, and hidden features of the corresponding image). i The n-dimensional cost function F10 of the similarity distance is minimized by the cost function F10 for the corresponding visually good matching formula. And maximize the matching formula for the corresponding visual defects. As mentioned above, it should be understood that the n-dimensional embedding function f 101 will embed each image d i , Mapping to n-dimensional parameter space R n The corresponding point f(d) in i ), Each dimension represents a feature, and each feature may be weighted by corresponding factors:

[0096]

[0097]

[0098]

[0099] F106 represents the distance / distance metric in this space. The smaller the distance F106 between two images, the more similar the two images are. When training the neural network 100, the goal of the proposed method is to learn an embedding function f101 that assigns smaller distances F106 to more similar image pairs; that is, the embedding function f101 considers as many features as possible—measurable features and hidden features—that influence visual appearance. Layer f101 of the deep neural network 100 computes the distance between images d. i Embedding: f(d) i )∈R n , where n is the dimension of the feature embedding.

[0100] When training neural network 100, it is known that after passing through neural network 100, the first sublist D... i+ All digital images will be assigned to category 130 of visually good matching coatings from a plurality of initial matching coatings, and a second sublist D i- All digital images will be assigned to category 140 of visually poor matching coatings from a specified plurality of initial matching coatings. With this knowledge, and in particular by using the concept of backpropagation, the cost function F 106 can be minimized for visually good matching coatings and maximized for visually poor matching coatings simultaneously. Therefore, the embedding function f 101 and its defining parameters are determined.

[0101] When the embedding function f 101 is determined, it will be provided as a good matching formula for searching the available digital image p 211 of the target coating schematically shown in Figure 2, and used in the trained neural network of device 200.

[0102] When the digital image p211 (=target image) of the target coating is analyzed and the corresponding digital images q1,q2,…,q of the corresponding sample coatings for which the similarity to the target coating is to be analyzed are obtained… i When i≥1, the output 250 of the device 200 implementing the trained neural network 202 is the corresponding distance value F(f(p), f(q)). j )), where 1≤j≤i206 (see Figure 2a When analyzing the similarity of multiple digital images q1, q2, ..., q of multiple sample coatings corresponding to a target coating, iAs 210 is fed one after another (as shown at time scale 201) into neural network 202 for evaluation, neural network 202 applies the trained embedding function f(·) to each digital image and outputs the corresponding point / feature map f(p), f(q1), f(q2), ... f(q) for each digital image. i ), as indicated by arrow 203 (see Figure 2b The cost function f(·)204 is applied to the corresponding feature map pairs, each pair consisting of a feature map assigned to the target image p and a digital image q assigned to the sample coating. j The feature map. The corresponding distance values ​​F(f(p), f(q) j (where 1 ≤ j ≤ i 206) allows for sorting among sample coatings for which similarity analysis of the target coating is to be performed. This sorting order 250 can be presented in a table (e.g. Figure 2a and 2b (as shown in the image), output or display in the form of a list, bar chart, or similar format.

[0103] Figure 3 Schematic illustration of triples (d i ,d i+ ,d i- 300. Each triplet 300 contains the target image d. i 301. Positive image d i+ 302, and the negative image d i- 303. The triple 300 represents the relative similarity relationship between three images 301, 302, and 303. A model (i.e., a neural network) was developed for both image similarity (complete images) and effect pigment similarity (effect pigments with only a black background). Effect pigments are identified using image segmentation methods. Possible segmentation techniques include thresholding methods, edge-based methods, clustering methods, histogram-based methods, neural network-based methods, hybrid methods, etc.

[0104] Figure 4An embodiment of an apparatus 400 for performing the methods described herein is shown. A user 40 may utilize a user interface 41, such as a graphical user interface, to operate at least one measuring device 42 to measure the properties of a target coating 43, i.e., by capturing digital images of the target coating using a camera, each image obtained with a different measurement geometry (e.g., at different angles), and determining color values ​​and optional texture values ​​for the different spectral measurement geometries, for example, using a spectrophotometer. Data from at least one measuring device (e.g., camera 42) may be transmitted to a computer 44, such as a personal computer, mobile device, or any type of processor. The computer 44 may communicate with a server 46 via a network 45, i.e., establish a communication connection. The network 45 may be any type of network, such as the Internet, a local area network, an intranet, or a wireless network. The server 46 communicates with a database 47, which may store data and information used by the methods of embodiments of the invention for comparison purposes. In various embodiments, the database 47 may be used, for example, in a client-server environment or, for example, in a web-based environment, such as a cloud computing environment. Various steps of the methods of embodiments of the invention may be performed by the computer 44 and / or the server 46. In another aspect, the present invention can be implemented as a non-transitory computer-readable medium comprising software for causing a computer or computer system to perform the methods described herein. This software may include various modules for enabling a processor and user interface to perform the methods described herein.

[0105] Reference Symbol List

[0106] 100 Neural Networks

[0107] 101 Embedded Functions

[0108] 102 f(d i )

[0109] 103

[0110] 104

[0111] 106 Cost function F

[0112] 107

[0113] 108

[0114] 110 Images

[0115] 111 Arrow

[0116] 112 arrows

[0117] 120 Triples

[0118] 130 categories "Good"

[0119] 140 categories of "poor"

[0120] 200 Devices with implemented trained neural networks

[0121] 201 Time Scale

[0122] 202 Trained Neural Network

[0123] 203 arrows

[0124] 204 Cost function F(·)

[0125] Digital images of the coating on sample 210

[0126] 211 Digital image of the target coating

[0127] 212 is fed into the neural network.

[0128] The output of the 206 neural network, the distance values ​​F(f(p), f(q)). j ))

[0129] 250 Sorting order, neural network output

[0130] 301 d i

[0131] 302 d i+

[0132] 303 d i-

[0133] 400 equipment

[0134] 40 users

[0135] 41 User Interface

[0136] 42 Measuring equipment

[0137] 43 Target Coating

[0138] 44 Computers

[0139] 45 Network

[0140] 46 servers

[0141] 47 Databases

Claims

1. A computer-implemented method, the method comprising at least the following steps: a) Using at least one measuring device, obtain at least one color value, digital image, and optional texture value for each of at least one measuring geometry, targeting the coating. b) Provide a database comprising formulas for coating compositions, and for each of the at least one measurement geometry, including interrelated color values, interrelated digital images, and optionally interrelated texture values. c) For each training target coating, the processor retrieves a list of multiple preliminary matching formulas from the database based on the obtained color values ​​and optionally on the obtained texture values ​​for the corresponding training target coating. d) Using visual inspection of the digital images of the at least one training target coating and the digital images associated with the preliminary matching formulas and obtained from the database, divide the list of the plurality of preliminary matching formulas into two sublists, wherein, The first sublist includes visually good matching formulas from the plurality of preliminary matching formulas, and the second sublist includes visually poor matching formulas from the plurality of preliminary matching formulas. e) For each training target coating, the processor creates multiple triples, each triple including a digital image of the corresponding training target coating for one of the at least one measurement geometry, a digital image associated with visually good matching formulas from the database for the at least one measurement geometry and a digital image associated with visually poor matching formulas from the database for the at least one measurement geometry and a second sublist. f) The convolutional neural network is trained by feeding the created triples one after another as corresponding inputs and optimizing an n-dimensional cost function, wherein the cost function defines the similarity distance to the at least one training target coating, such that the cost function minimizes the corresponding visually good matching formula and maximizes the corresponding visually bad matching formula. g) Make the trained neural network available in the processor to sort digital images of the coating composition with respect to digital images of the target coating.

2. The method of claim 1, further comprising step e) an additional step ee), said additional step ee) performing for at least one of the created triples: rotating the digital image of the corresponding training target coating by a given angle around a central image rotation axis, rotating the digital image associated with the corresponding visually good matching formula obtained from the database by the given angle around the central image rotation axis, and rotating the digital image associated with the corresponding visually bad matching formula obtained from the database by the given angle around the central image rotation axis, thereby obtaining at least one additional triple for the corresponding training target coating and based on the at least one triple.

3. The method according to claim 2, comprising repeating step ee) several times by changing the given angle.

4. The method according to claim 2 or 3, wherein, The given angle is selected from the following group: 0°, 30°, 60°, 90°, 120°, 150°, 180°, 210°, 240°, 270°, 300°.

5. The method of claim 1, further comprising step e) an additional step ee), said additional step ee being performed for at least one of the created triples: dividing the digital image of the corresponding training target coating into a plurality of blocks, dividing the digital image associated with the corresponding visually good matching formula obtained from the database into the same number of corresponding blocks, and dividing the digital image associated with the corresponding visually bad matching formula obtained from the database into the same number of corresponding blocks, thereby obtaining, for the corresponding training target coating and based on the at least one triple, additional triples corresponding to the number of blocks.

6. The method according to claim 1, wherein, Step f) includes setting an n-dimensional cost function F for training the convolutional neural network, such that it can be represented by a corresponding vector and by n parameters. Defined n-dimensional embedding function Mapped to scalar ,vector Each component This includes the feature values ​​of the corresponding preliminary matching formula, where each parameter... Define the weights for the corresponding features.

7. The method of claim 1, wherein in step e), the digital image is preprocessed using an image segmentation method for identifying effect pigments, and the digital image of the effect pigments is formed using only a black background.

8. The method according to claim 1, wherein, Select 500 to 1000, especially 1000, different training target coatings, and for each training target coating, select a number of N initial matching formulas, where N is an integer.

9. The method according to claim 1, wherein, The plurality of preliminary matching formulas are selected such that, for each preliminary matching formula, the color-based distance, optionally texture-based distance, or color-and-texture-based distance to the corresponding training target coating is less than a first threshold.

10. A computer-implemented method, the method comprising at least the following steps: A) Obtain the color values, digital image, and optional texture values ​​of the target coating using at least one measuring device for at least one measuring geometry. B) Provide a database comprising formulas for coating compositions and interrelated color values, interrelated digital images, and optionally interrelated texture values ​​for the at least one measurement geometry. C) Provide the computer with a trained convolutional neural network. D) For the target coating of the at least one measurement geometry, based on the obtained color values ​​and optionally based on the obtained texture values, select one or more preliminary matching formulas from the database. E) For each of the one or more preliminary matching formulas and for the at least one measurement geometry, obtain digital images from the database that are associated with the corresponding preliminary matching formula. F) For each of the one or more preliminary matching formulas, provide the trained convolutional neural network with a corresponding acquired digital image associated with the corresponding preliminary matching formula, along with digital images acquired for the target coating and for the at least one measurement geometry. G) For each of the one or more preliminary matching formulas, the trained neural network is used to determine the similarity distance between the target coating and the corresponding preliminary matching formula. The trained neural network is trained to compute the similarity distance in an embedded feature layer between two digital images, i.e., the corresponding acquired digital image associated with the corresponding preliminary matching formula and the digital image acquired for the target coating. H) Using an output device integrated with the computer operation, output the determined similarity distance of the one or more preliminary matching formulas to the user. in, The trained convolutional neural network is provided by the method according to any one of claims 1 to 9.

11. The method of claim 10, further comprising displaying the determined similarity distance of the one or more preliminary matching formulas on a display interface of a display monitor communicatively connected to the computer.

12. The method according to claim 11, wherein, The similarity distances determined by the one or more preliminary matching formulas are displayed in the form of a sorted list, wherein the lower the similarity distance, the better the corresponding preliminary matching formula matches the target coating.

13. A computer-implemented method for ranking digital images of a sample coating and a target coating using a computer with a trained convolutional neural network, the trained convolutional neural network being trained to calculate similarity distances between two digital images in an embedded feature layer of the trained convolutional neural network. in, The trained convolutional neural network is provided by the method according to any one of claims 1 to 9.

14. An apparatus comprising at least: - A database comprising formulas for coating compositions and interrelated color values, interrelated digital images, and optionally interrelated texture values ​​for at least one measurement geometry. - At least one processor, communicatively connected to at least one measuring device, the database, and the convolutional neural network, and programmed to perform at least the following steps: 1) Receive color values, digital images, and optional texture values ​​of the target coating from the at least one measuring device, and 2) Perform steps c) to g) of claim 1 or steps D) to H) of claim 10.

15. A non-transitory computer-readable medium having a computer program having program code configured to perform at least the following steps when the computer program is loaded and executed by at least one processor, wherein the at least one processor is communicatively connected to at least one measuring device, the at least one measuring device being configured to provide color values, digital images, and optionally texture values ​​of at least one target coating for each of at least one measuring geometry; communicatively connected to a database including formulas for coating compositions and interrelated color values, interrelated digital images, and optionally interrelated texture values ​​for the at least one measuring geometry; and communicatively connected to a convolutional neural network: 1) Receive from the measuring device color values, digital images, and optional texture values ​​of at least one target coating for at least one measuring geometry, and 2) Perform steps c) to g) of claim 1 or steps D) to H) of claim 10.

Images