Visualizing the appearance of at least two materials

Abstract

In a computer-implemented method for visualizing the appearance of at least two materials, a first instance of an appearance model is obtained (56), the first instance being indicative of the appearance of a first material, and a second instance of the appearance model is obtained (66), the second instance (66) being indicative of the appearance of a second material. A virtual object (72) having a continuous three-dimensional surface geometry is visualized together using a display device (70). The virtual object has first and second portions (72a, 72b) that are visually separated by a virtual separation element (74) or a blending zone. The first portion of the virtual object is visualized using the first instance of the appearance model, and the second portion of the virtual object is visualized using the second instance of the appearance model.

Description

TECHNICAL FIELD

[0001] The present invention relates to a method of visualizing the appearance of at least two materials. The invention further relates to an apparatus for performing such a method and a corresponding computer program. BACKGROUND

[0002] Finding a color formulation that matches a given color of a physical object to be color matched (“target object”) can be a lengthy trial and error process. A set of colorants is selected, a candidate formulation using the colorants is retrieved or formulated, and a test object, such as a test panel, is prepared by applying a material prepared according to the candidate formulation to the test object. The test object is then visually compared to the target object. If the color of the test object does not match the color of the target object, the candidate formulation is modified as needed, often repeatedly, until the color of the test object matches the color of the target object to within a desired tolerance range.

[0003] In some applications, the target object can be a vehicle part coated with an existing coating, and it is desired to find a coating formulation whose color matches the appearance of the existing coating. Even if a paint code or vehicle identification number associated with the target object is known, merely retrieving the corresponding reference formulation can not yield an acceptable match. This is because, even if a paint code specification and coating formulation exist, the color and appearance of a given target object will vary slightly from batch to batch, formulator to formulator, or year to year. Modifications to the reference formulation need to be considered.

[0004] Traditionally, the process of finding a matching formulation has been a fully manual process. Whether formulating from scratch or modifying a reference formulation, success is largely dependent on the experience of the color professional selecting the colorants and defining the candidate formulation. Even experienced experts often need several iterations until a satisfactory match is obtained.

[0005] Over the past decades, increasingly sophisticated color formulation software has been introduced to help color professionals in formulating a color within a defined tolerance range with fewer iterations. Examples include the software “Color iMatch TM ” available from X-Rite, Grand Rapids, MI, USA, the software “PaintManager” available from PPG Industries, Strongsville, OH, USA, the software “Match Pigment” available from Datacolor, Lawrenceville, NJ, USA, or the software “Colibri TMColorMatch”. Color formulation software typically has three main components (see [Ber19] for example): The first is a database of the optical properties of colorants. The second is a set of algorithms that select colorants and predict candidate formulations. The third is a set of algorithms that correct the initial candidate formulation when the match is not within the tolerance range.

[0006] To populate the database with optical properties of colorants, the optical properties need to be determined by measurement. Depending on the type of application envisaged, this can be done in different ways. For example, for coatings as they are typically used in the automotive industry, each colorant can be prepared in a mix with a base formulation at different concentrations, and possibly with different amounts of added white and black pigments, and a reference object in the form of a so-called drawdown (for example, in the form of a black and white opaque card coated with the mix) can be created for each mix. Then, an instrument like a spectrophotometer can be used to determine the color properties of the drawdown in the form of spectral data or in a predefined color space (for example, in the form of tristimulus values or CIELAB values), and this can be fed to the database.

[0007] To perform color matching on a physical target object, the color properties of the target object are determined as well. The formulation software then predicts one or more candidate formulations that are expected to produce approximately the same color properties as the target object. To assess the quality of the match, a trial object (for example, a drawdown in the case of a coating) can be created for the selected candidate formulation. Then, the trial object can be visually compared to the target object. If the match is not visually satisfactory yet, the color properties of the trial object can be taken, and the color formulation software can correct the formulation based on a comparison of the calculated color properties of the candidate formulation with the measured color properties of the trial object. This process can be repeated as often as necessary until an acceptable match is obtained.

[0008] While an acceptable result can typically be obtained with fewer iterations in this way compared to with traditional manual methods, there is still room for improvement. In particular, it would be desirable to be able to judge the quality of the match between a candidate formulation and a target object with increased confidence already before the trial object has been actually produced. This is particularly desirable if the location where the trial object is produced is different from the location where the target object is located. For example, the target object can be a part of a damaged vehicle that needs to be repaired in a body shop. However, the body shop itself can not be equipped with color mixing facilities. Rather, the body shop can have to order ready-to-spray paint from a remote paint supplier. In such a case, it can be necessary that the match is within the tolerance range on the first shot.

[0009] Known methods of color matching can not produce satisfactory results, especially for paint coatings or other materials exhibiting gonioapparent properties. For example, some materials exhibit color flop. "Color flop" is a change in color value, hue, or chroma in the appearance of a material when the illumination and viewing directions are changed. Other examples of materials with gonioapparent properties are materials containing effect pigments, such as metallic flakes producing a sparkling effect or interference flakes producing a pearlescent effect, and materials with non-planar surface microstructures. In such cases, matching the color alone can not be sufficient. Rather, the entire appearance, including the angle-dependent color and texture, needs to be matched. Known techniques often do not provide satisfactory visualizations or measurements to accurately match the appearance of such materials.

[0010] US20150026298A1 discloses a method of using a mobile device to select the most likely variant of a matching paint candidate color standard for a vehicle repair. In this method, a user at a body shop enters information about the color of a vehicle into a mobile device and transmits the information to a remote central computer. The central computer selects a candidate color standard and transmits information about the candidate color standard to the mobile device. The mobile device displays information about the selected candidate color standard. The user visually compares a physical chip representing the selected candidate color standard to the color of the vehicle to be repaired. A disadvantage of this method is that a large number of physical chips of color standards are needed at the body shop. Alternatively, an image of the selected candidate color standard is displayed on a color display of the mobile device and the user visually compares the displayed image to the color of the vehicle to be repaired. However, this requires careful calibration of the display in order to enable a meaningful comparison.

[0011] US20050128484A1 discloses a method for determining a repair paint formula for color matching. Color characteristics of a target color to be matched are identified, entered and processed in such a way that the target color can be visually displayed. Alternative colors are selected from a database. The alternative colors can be displayed on a display as several virtual chips, each representing a different viewing angle, or as a curved panel. An image showing the appearance characteristics of the flake can be superimposed on the color. The virtual chips can be viewed in combination with the target color.

[0012] US20070097119A1 discloses a method for displaying a simulation of a paint coating on a display device. RGB color values are calculated over a range of aspecular angles. A statistical texture function of the paint coating is determined. The statistical texture function is applied to the RGB values and color pixels are displayed using these values. The statistical texture function does not depend on the illumination and viewing directions.

[0013] US20080291449A1 and US20080235224A1 both disclose methods for displaying images to select a matching formulation to match the appearance of an item, such as a target coating of a vehicle. In one embodiment, color data of the item is obtained using a colorimeter or spectrophotometer. Texture data of the item is obtained using an imaging device. A target image is created based on the color and texture data and displayed on a display device. A preliminary matching formulation is manually retrieved from a database. A matching image is generated for the preliminary matching formulation. The matching image is displayed side-by-side with the target image on the display device. The target image and the matching image can be displayed for multiple non-specular angles. The documents do not mention how to generate and display texture data for multiple non-specular angles.

[0014] US20200089991A1 discloses a system for displaying one or more images to select one or more matching formulations to match the appearance of a target coating of an item. A first database contains repair formulations and associated appearance characteristics. A second database contains identification information or three-dimensional geometry data of at least one item. A preliminary matching formulation is retrieved from the first database, the item or its three-dimensional geometry data is selected from the second database, and a marked portion of a surface of the item is received. An individual matching image is generated containing the marked portion and an unmarked portion adjacent to the marked portion, and the individual matching image is displayed on a display device. In the marked portion, the individual matching image is generated based on the appearance characteristics of the preliminary matching formulation. In the unmarked portion, the individual matching image is generated based on the appearance characteristics of the item. The appearance characteristics can include texture, metallic or pearlescent effects, gloss, sharpness of the image, flake appearance such as texture, sparkle, shine, and shimmer, and enhancement of perceived depth imparted by flakes. The document does not specify how to display the appearance of the item for different illumination and viewing directions.

[0015] US8,872,811B1 discloses a method of digitally generating data indicative of a synthetic appearance of a simulated material having physically plausible appearance attributes. The data set indicative of the synthetic appearance of the simulated material is determined based in part on data associated with a physically tangible source material and in part on data of measured attributes of a physically tangible reference material.

[0016] US2007291993A1 discloses an apparatus for measuring a spatially undersampled bidirectional reflectance distribution function (BRDF) of a surface.

[0017] US2018285481A1 discloses a system and process for manufacturing, visualizing and distributing stone slabs. A 3D scene can be generated which depicts a portion of a major surface of a stone slab at a target area of a slab installation environment. SUMMARY

[0018] It is an object of the present invention to provide a method of visualizing the appearance of at least two materials in such a way that a user is enabled to determine with increased confidence whether the appearance of the materials match without the need to produce a physical object.

[0019] The present disclosure provides a computer-implemented method for visualizing the appearance of at least two materials. The method comprises:

[0020] obtaining a first instance of an appearance model, the first instance being indicative of the appearance of a first material;

[0021] obtaining a second instance of the appearance model, the second instance being indicative of the appearance of a second material; and

[0022] obtaining a geometric model of a virtual object, the geometric model defining a three- dimensional macroscopic surface geometry of the virtual object; and

[0023] visualizing a scene comprising the virtual object using a display device, using the first and second instance of the appearance model and the geometric model together with a virtual separation element or a blending region,

[0024] wherein the virtual object has a first and a second portion,

[0025] wherein the first portion of the virtual object is visualized using the first instance of the appearance model,

[0026] wherein the second portion of the virtual object is visualized using the second instance of the appearance model, and

[0027] wherein the first and second portion are visually separated by the virtual separation element or by the blending region.

[0028] In the present invention a scene comprising an arbitrary three-dimensional virtual object is visualized, wherein two different portions of the virtual object are visualized side by side, which are separated by a virtual separation structure or a blending region. Two instances of the same appearance model are used to visualize the two portions. The first instance is for a first material and the second instance is for a second material. In this way a direct comparison of the appearance of the first and second material is facilitated. The object can have an arbitrary shape and an arbitrary orientation and can be visualized under arbitrary illumination and observation conditions.

[0029] The term "appearance" as used in this disclosure includes both color and texture, "texture" being broadly understood as describing the spatial variation of appearance across the surface of an object. An "appearance model" is a formal construct that describes appearance in mathematical terms using a number of material-related parameters called "appearance attributes". Appearance attributes can include color attributes and texture attributes. In a simple embodiment, an appearance model describes only the dependency of color on illumination and viewing conditions, without considering the spatial variation of appearance across the surface of an object. For example, in a simple embodiment, an appearance model can be a BRDF model that describes only the spectral reflectance as a function of illumination and viewing directions, without considering spatial variations. Thus, an appearance model can describe appearance only in terms of color attributes. In a more complex embodiment, an appearance model can be a model that includes "texture", i.e. in addition to the angular dependency of reflection and / or transmission, there is also a spatial variation of appearance across the surface of an object. In particular, an appearance model can be a spatially-varying bidirectional reflectance distribution function ("SVBRDF") model, a bidirectional texture function ("BTF") model, a bidirectional surface scattering distribution function ("BSSRDF model"), a specialized model for automotive paints, etc. Many such models are known in the art. Thus, an appearance model can describe appearance in terms of both color attributes and texture attributes.

[0030] Texture can depend to a large extent on the illumination and viewing directions. For example, in a metallic effect paint that typically contains high-reflective flakes, when the illumination and / or viewing directions are varied continuously, the surface locations where a strong reflection is observed can vary discontinuously, as the flakes across the surface will reflect under different combinations of illumination and viewing directions at different locations. Thus, the observed texture can be very different between illumination and viewing directions that only differ by a small angular value.

[0031] In the present invention, at least one virtual object is visualized. The geometry of the virtual object is described by a three-dimensional geometric model. The geometric model defines a three-dimensional macroscopic surface geometry of the virtual object. Preferably, the macroscopic surface geometry is curved along at least two mutually orthogonal directions. In some embodiments, a combination of small polygonal planar surfaces (e.g. a polygonal mesh) can be used in the geometric model to represent the curved surface of the virtual object. Preferably, the curved three-dimensional macroscopic surface geometry includes both convex and concave portions. Preferably, the macroscopic surface geometry of the virtual object has surface normals covering a large solid angle, i.e. having a large range of directions in three-dimensional space. Preferably, the solid angle covered by the directions of the surface normals of the virtual object is at least 50% of the solid angle of a hemisphere, i.e. it is preferably at least πsr. In this way, effects such as color shift, gloss, sparkle, texture, etc. between two materials can be compared simultaneously for a large number of illumination and viewing directions relative to the surface normals of the virtual object.

[0032] In some embodiments, the first and second materials can be substantially opaque materials, such as paint layers as they are used in the automotive industry for coating vehicle parts, or opaque plastics. In other embodiments, the first and second materials can be translucent materials, such as translucent plastics. The appearance model can be particularly adapted to a particular class of materials. For example, for vehicle paint layers, the appearance model can include a model of the effect of a transparent clear coat on top of an opaque or translucent paint layer and / or a model of the effect of reflective flakes in a paint layer. As another example, for translucent materials, the appearance model can use volume absorption and scattering coefficients and phase function parameters to model subsurface light transport of the material to solve the radiative transfer equation.

[0033] If the scene is visualized with a virtual dividing element, the first and second portions of the virtual object can be adjacent to each other and visually separated by the virtual dividing element, such that the first and second portions appear to meet at the location of the virtual dividing element.

[0034] If the scene is visualized with a blending zone, visualizing the blending zone can include interpolating between the appearance properties of the first and second instances of the appearance model, thereby creating a smooth transition between the appearances of the first and second portions of the virtual object.

[0035] The appearance model can require each instance to include at least one texture, in particular a discrete texture sheet including a plurality of textures, each texture being associated with a different combination of illumination and viewing directions. Interpolating between the appearance properties of the first and second instances of the appearance model can then include computing at least one interpolated texture in the blending zone by interpolating between at least one texture of the first instance of the appearance model and at least one texture of the second instance of the appearance model. Computing each interpolated texture can include:

[0036] (i) assigning interpolation weights to the at least one texture of the first instance of the appearance model and the at least one texture of the second instance of the appearance model; and

[0037] (ii) synthesizing the interpolated texture using the at least one texture of the first instance of the appearance model and the at least one texture of the second instance of the appearance model and the assigned interpolation weights,

[0038] wherein synthesizing the interpolated texture includes:

[0039] (a) randomly selecting one of the textures of the first and second instances of the appearance model with a probability proportional to the interpolation weight of said texture;

[0040] (b) randomly extracting a texture patch from the selected texture;

[0041] (c) modifying the extracted texture patches by modifying pixel values in the extracted texture patches to obtain modified texture patches, the modification of the pixel values being performed in such a way that at least one statistical property of the modified texture patches approximates a corresponding average statistical property, the average statistical property being determined by performing a weighted average over the textures of the first and second instances of the appearance model, the textures of the first and second instances of the appearance model being weighted by the interpolation weights;

[0042] (d) inserting the modified texture patches into the interpolated texture in such a way that the modified texture patches seamlessly fit into existing texture content in the interpolated texture; and

[0043] (e) repeating steps (a)-(d) until the interpolated texture is completely filled.

[0044] The display device can be a screen, e.g. an LCD screen, a projector or any other type of display device as they are well known in the art. If the display device comprises a screen, the screen can be touch sensitive as it is well known in the art. The display device can be capable of creating a 3D impression as it is well known in the art. For example, the display device can be a VR headset or a 3D display. In such a case, the virtual partition element can be a virtual plane. However, in a preferred embodiment, the display device displays the visualization of the virtual object as a two-dimensional projection in a two-dimensional viewing plane. In this case, the virtual partition element can be a simple line, in particular a straight line or a curve, in the viewing plane.

[0045] The method can comprise moving the virtual partition element or the fusion zone relative to the virtual object. This can be done interactively, i.e. in response to a user input, e.g. in response to a user moving a pointing device such as a mouse or trackpad, in response to a user hitting a key on a keyboard, or in response to a user swiping a finger or a pointing device such as a digital pen across a touch screen. When moving the virtual partition element or the fusion zone, the first and second portions are permanently redefined in such a way that the first and second portions are visually separated by the virtual partition element or the fusion zone even after the virtual partition element or the fusion zone has been moved. In particular, if there is a virtual partition element, the first and second portions can still appear to meet at the location where the virtual partition element is located when the virtual partition element is moved. In this way, the user can visually assess how the appearance of the virtual object changes in the selected region of the virtual object when the material of the virtual object changes.

[0046] In some embodiments, the scene is visualized under some defined illumination conditions, which can be described, e.g., by an environment map, as it is well known in the art. The method can optionally comprise changing the lighting conditions, in particular one or more of the lighting directions or the type of light source, i.e., the spectral properties of the light used for illumination, in response to a user input, and visualizing the virtual object under the changed illumination conditions, in order to facilitate comparing the appearance of the first and second materials under different illumination conditions. For example, in this way, metamerism can be detected.

[0047] In some embodiments, the virtual object is visualized in a defined pose (e.g., orientation, position, size, and distance from the observation point), and the method comprises changing the pose of the virtual object, i.e., rotating, moving, or rescaling the virtual object, in response to a user input, and visualizing the virtual object in the pose as it appears after the change in the pose. Thereby, differences in appearance that depend subtly on the direction of illumination and observation can be better discerned. The visualization can be performed in the form of a video, which visualizes the continuously rotated virtual object. The method can further comprise changing the shape of the virtual object.

[0048] An instance of the appearance model of the first and / or second material can be generated from a finite set of available appearance attributes associated with each material. The finite set of values can have been measured or obtained in some other way (e.g., by computation or database retrieval). The appearance model can typically require many more appearance attributes than are actually available in the finite set, and it can require different types of appearance attributes than those available. In particular, the set of appearance attributes required by the appearance model can have a greater cardinality (i.e., contain a greater number of appearance attributes) than the finite set of available appearance attributes. In other words, the set of available appearance attributes can be sparse compared to the more dense set of appearance attributes required by the appearance model. Mathematical operations that can involve transformations, fitting operations, and / or interpolation and extrapolation can be necessary to generate an instance of the appearance model from the finite set of available appearance attributes. For example, the finite set of available appearance attributes can contain color attributes (e.g., in the form of trichromatic or spectral data) for only a limited number of illumination and viewing direction pairs. The finite set of available appearance attributes can also include texture attributes (e.g., in the form of image data) for only one or a limited number of illumination and viewing direction pairs. In contrast, the appearance model can describe reflectance and / or transmittance as a function of angle and position in a different form. In particular, the appearance model can require much more spectral data and / or texture data for illumination and viewing direction pairs than those available, and the required illumination and viewing directions can be different than those available. Examples of how an instance of the appearance model with a more dense set of appearance attributes can be generated from a sparse set of available appearance attributes will be described in more detail below.

[0049] In some embodiments, the values of the finite set of available appearance attributes of the first material are determined by performing measurements of a target object including the first material using the (first) appearance capture device, and a first instance of the appearance model is generated from the finite set of appearance attributes. In this case, the first material can also be referred to as the “target material.”

[0050] In some embodiments, the second material is a candidate material intended to have a similar appearance to the first material. Thus, the second instance of the appearance model can be generated based on candidate appearance attributes associated with the candidate material. In some embodiments, the candidate material is a known reference material, and the candidate appearance attributes are appearance attributes associated with the known reference material. In other embodiments, the candidate material is composed of a plurality of component materials, and the appearance attributes can be available for a plurality of reference materials including the component materials of the candidate material. Thus, the candidate appearance attributes can be based on predetermined appearance attributes associated with the plurality of reference materials using a calculation. The formulation can be determined in such a way that the candidate material has an intended appearance attribute that at least approximately matches the appearance attributes of the target material. In particular, the presently proposed method can comprise the step of determining the formulation. This can involve minimizing a difference measure between the intended appearance attributes of the candidate material and the measured appearance attributes of the target material. Suitable difference measures and optimization algorithms are well known in the field of color formulation.

[0051] In particular, obtaining the second instance of the appearance model can comprise:

[0052] receiving appearance attributes of the first material;

[0053] determining a formulation of the second material based on the received appearance attributes of the first material and predetermined appearance attributes of the reference material; and

[0054] generating the second instance of the appearance model based on the formulation and measured appearance attributes of the reference material.

[0055] In some embodiments, the method further comprises obtaining a physical, tangible test object comprising the candidate material, for example by producing the test object. The method can comprise determining measured appearance attributes of the test object by performing measurements on the test object using the appearance capture device. The method can further comprise visualizing at least a portion of the at least one virtual object using the measured appearance attributes of the test object. In this way, the appearance of the test object can be compared to the appearance of the target object even though the target object does not exist at the same location as the test object. Additionally or alternatively, the method can comprise determining a modified formulation using the measured appearance attributes of the test object and the calculated appearance attributes of the candidate material. Suitable algorithms for determining a modified formulation once the measured appearance attributes of the test object are available are well known in the field of color formulation. Such algorithms are often referred to as “correction algorithms”. The method can then be repeated with the modified formulation.

[0056] The appearance capture device for determining the measured appearance properties of the test object can be the same device as the device for determining the measured appearance properties of the first material, or it can be a different device. The device is preferably configured to determine the same type of appearance properties under the same pair of illumination and viewing conditions as the appearance capture device for the target object. However, it is also conceivable that the appearance capture device for the test object is configured to determine a different type of appearance properties, a different number of appearance properties and / or a different pair of illumination and viewing directions than the appearance capture device for the target object.

[0057] In each case, the appearance capture device can be a multi-angle spectrophotometer configured to determine color properties for a plurality of combinations of illumination and viewing directions. The appearance capture device can also have imaging capabilities to determine texture properties in addition to color properties.

[0058] The set of available appearance properties of the first and / or second material can include appearance properties for a plurality of combinations of illumination and viewing directions, in particular color properties under two or more combinations of illumination and viewing directions and / or texture properties under two or more combinations of illumination and viewing directions.

[0059] In some embodiments, the available (e.g., measured) appearance properties of the first material (target material) include a plurality of measured image data sets, each associated with a different combination (e.g., pair) of illumination and viewing directions. To this end, the appearance capture device for determining the appearance properties of the first material can include one or more cameras configured to determine image data under a plurality of combinations (e.g., pairs) of illumination and viewing directions. In some embodiments, all combinations of illumination and viewing directions of the measured image data include the same viewing direction, but a plurality of different illumination directions (or, equivalently, the same illumination direction and a plurality of different viewing directions). This can be the case, for example, when using an appearance capture device with a single camera and a plurality of different light sources. In other embodiments, the combinations of illumination and viewing directions include two or more viewing directions.

[0060] In some embodiments, the available appearance properties of the first material can lack texture properties or can only contain a limited set of texture properties. This can occur, for example, if the appearance capturing device used to measure the target object lacks imaging capabilities or only has very limited imaging capabilities. In this case, the measured appearance properties of the first material can be supplemented by texture properties from a different source. In particular, the set of appearance properties associated with the first material can include texture properties, in particular in the form of image data, which have been determined based on texture properties associated with one or more reference materials and / or with the second material (candidate material). In particular, some or all of the texture properties of the first material can be determined based on computed texture properties (in particular in the form of image data) associated with the second material (candidate material) or based on measured texture properties (in particular in the form of image data) associated with a test object comprising the second material. Thus, the method can comprise determining texture properties (in particular in the form of image data) of the first material based on texture properties (in particular in the form of image data) associated with one or more reference materials and / or with the second material. This can involve modifying pixel values of the image data associated with the reference materials and / or with the second material. In this way, the first material can be visualized with the impression of realistic texture, even though the available texture information of the material itself would not be sufficient for this purpose.

[0061] In still other embodiments, the available appearance attributes of the candidate material or of the reference material can lack texture attributes, or can only contain a limited set of texture attributes. This can occur, for example, if the database containing the appearance attributes of the reference material was created using a simpler or older appearance capture device that lacks imaging capabilities or only has limited imaging capabilities. In this case, the appearance attributes associated with the second material can be supplemented by texture attributes, in particular in the form of image data, from a different source. In particular, the appearance attributes associated with the second material can include texture attributes, in particular in the form of image data, that have been determined based on the available texture attributes, in particular in the form of image data, associated with the first material and / or with the test object comprising the second material. In particular, some or all of the texture attributes associated with the second material can be determined based on the measured texture attributes, in particular in the form of image data, associated with the first material or based on the measured texture attributes, in particular in the form of image data, associated with the test object comprising the second material. Thus, the method can include determining texture attributes associated with the second material based on the measured texture attributes associated with the first material and / or based on the texture attributes associated with the test object comprising the second material. This can involve modifying pixel values of the image data associated with the first material and / or with the test object. In this way, the second material can be visualized with the impression of a realistic texture, even if the available texture information would not be sufficient for this purpose on its own.

[0062] As already outlined above, the set of available appearance attributes for each material can be sparse compared to the set of dense appearance attributes in the associated appearance model. The first and / or the second instance of generating an appearance model for a material can include at least one of the following operations:

[0063] interpolating between the available appearance attributes of the material under different combinations of illumination and viewing directions; and

[0064] extrapolating from the available appearance attributes of the material under the selected combination of illumination and viewing directions.

[0065] The available appearance attributes can include a plurality of source textures, each source texture being associated with a different set of source coordinates, each set of source coordinates being indicative of a combination of illumination and viewing directions for which the respective source texture is indicative of the spatial variation of the appearance. The appearance model can need to include a discrete texture table of a plurality of target textures, each target texture being associated with a different set of target coordinates, the set of target coordinates being indicative of a particular combination of illumination and viewing directions for which the target texture is indicative of the spatial variation of the appearance. The source textures can be associated with sets of source coordinates that are different from the target coordinates of the discrete texture table. The first and / or the second instance of generating an appearance model can then include determining at least one of the target textures by performing a statistical texture synthesis operation based on:

[0066] (i) a target set of coordinates associated with the target texture;

[0067] (ii) a source texture, and

[0068] (iii) a source set of coordinates.

[0069] Each source and target set of coordinates indicates a particular combination of illumination and viewing directions. Each source and target set of coordinates can in particular be a set of two or more angular coordinates. The source and target coordinates need not directly indicate the illumination and viewing directions in a reference frame determined by the surface of the material. Instead, they can be derived from these directions by an appropriate transformation. In particular, the source and / or target coordinates can be expressed in a Rusinkiewicz parameterization [Rus98]. In particular, in a Rusinkiewicz parameterization, each source or target set of coordinates can comprise or consist of a polar angle.

[0070] Each source and each target texture can be represented by two-dimensional image data, which will typically be in the form of an array of pixels, each pixel having a pixel value representing the reflection at the position of that pixel. The source and target textures need not be of the same size. Each of the source textures can represent the spatial variation of the reflective properties of the material for a particular combination of illumination and viewing directions. The target texture can thus represent the spatial variation of the reflective properties of the material for another combination of illumination and viewing directions. The spatial variation can represent, for example, the effect of an effect pigment in the material, which produces a sparkling effect.

[0071] By performing a statistical texture synthesis operation, a target texture is created that has similar statistical properties to the source textures, but on a pixel-by-pixel basis is dissimilar to any of the source textures and to the target texture at the different target coordinates. This mimics the properties of a metallic effect paint.

[0072] The statistical texture synthesis operation can comprise:

[0073] (i) assigning to each of the source textures an interpolation weight based on the target coordinates and the target coordinates; and

[0074] (ii) synthesizing the target texture using the source textures and the assigned interpolation weights.

[0075] In some embodiments, assigning to each of the source textures an interpolation weight comprises the following procedure:

[0076] creating a Delaunay triangulation of the source set of coordinates;

[0077] finding a simplex in the Delaunay triangulation comprising the target coordinates, the found simplex having a plurality of corners; and

[0078] using the barycentric coordinates of the target coordinates with respect to the found simplex as interpolation weights for the source textures at the corners of the found simplex.

[0079] The interpolation weights for source textures outside the found simplex are preferably set to zero, i.e. source textures outside the found simplex are not used for synthesizing the target texture.

[0080] Synthesizing the target texture can comprise the following steps:

[0081] (a) randomly selecting one of the source textures with a probability proportional to the interpolation weight of the source texture;

[0082] (b) randomly extracting a texture patch from the selected source texture, the texture patch being a portion of the source texture;

[0083] (c) modifying the extracted texture patch by modifying pixel values in the extracted texture patch to obtain a modified texture patch, the modification of pixel values being performed in such a way that at least one statistical property of the modified texture patch approximates a corresponding average statistical property, the average statistical property being determined by performing a weighted average over the source textures, said source textures being weighted by the interpolation weights;

[0084] (d) inserting the modified texture patch into the target texture such that the modified texture patch seamlessly fits into the texture content already present in the target texture; and

[0085] (e) repeating steps (a) - (d) until the target texture is completely filled.

[0086] Each texture patch is to be understood as a contiguous portion of the source texture from which it has been extracted, i.e. a contiguous image patch that has been cut out of the source texture. Each texture patch can have an arbitrary shape. A rectangular shape, in particular a square shape, is preferred. Each texture patch preferably has an edge length that is at least twice the size of the largest effect pigment (e.g. largest reflective flake) in the paint that is modeled. On the other hand, each texture patch preferably has an area that is not larger than 10% of the area of the respective source texture.

[0087] The texture patches are randomly extracted, i.e. the position of each extracted texture patch in the selected source texture is determined randomly, preferably with a constant weight over the area of the source texture. By randomly extracting the texture patches, it is ensured that the resulting target texture at different target coordinates is not similar.

[0088] In a preferred embodiment, the statistical property for which an approximate match is desired is the histogram of pixel values. The histogram of pixel values is a dataset comprising, for each of a range of possible intensity values in the image, an indicator of the relative frequency of that range in the image. Histograms of pixel values are widely used in digital image processing for visualizing properties such as contrast and brightness of digital images. Their generation from image data and their interpretation are well known in the art.

[0089] The modification of the pixel values preferably comprises applying a pointwise transformation to the pixel values, i.e. a transformation that is applied individually to each pixel value, the pointwise transformation being monotonically non-decreasing. In this way, it is ensured that the same pixel values before the modification are still the same after the modification, and the relative brightness order of the pixels does not change, i.e. a pixel value that was brighter than another pixel value before the modification of the texture patch will not be darker than the other pixel value after the modification. Suitable transformation functions are well known in the field of digital image processing. In particular, histogram matching algorithms for determining a transformation function that will cause the histogram of an image to match the histogram of another image are well known in the art.

[0090] Inserting the modified texture patch into the target texture such that it fits seamlessly into the texture content already present in the target texture can be achieved by using techniques such as Graphcut / MinCut [Kw03] that are well known in the art. In these techniques, a seam is computed that strengthens the visual smoothness between the existing texture content and the newly placed patch. The texture patch and the already present texture content are stitched together along this seam.

[0091] In addition to a discrete texture table, the appearance model can comprise a monochromatic brightness BRDF model for describing the angle dependence of the overall reflective properties averaged over surface location and wavelength. An example of generating an appearance model of a material can then comprise determining the parameters of the brightness BRDF model.

[0092] The appearance model can further comprise a discrete color table having a plurality of entries in the form of color values, each entry being associated with a specific set of coordinates, each set of coordinates indicating the illumination and viewing directions at which the material is observed. The color values can be expressed in an arbitrary color space, such as a trichromatic color space such as RGB or CIEXYZ, or any other color space such as CIELAB (L*a*b*), or in the form of spectral data representing the spectral response of the material to incident light in an arbitrary format. An example of generating an appearance model can then comprise at least one of the following operations:

[0093] (i) determining the entries of the discrete color table by interpolating between available color attributes at different combinations of illumination and viewing directions; and / or

[0094] (ii) determining entries of a discrete color table by extrapolation from available color attributes under different combinations of illumination and viewing directions.

[0095] Instead of a monochromatic BRDF model in combination with a color table, in alternative embodiments the appearance model can comprise any other kind of model for describing the angular dependence of the luminance and color. For example, the appearance model can comprise a trichromatic BRDF model that models the angular dependence of the reflectance properties for each of three different color channels in a trichromatic color space, such as CIEXYZ or RGB, or that models the angular dependence of the L*a*b* values in the CIELAB color space.

[0096] The appearance model can further comprise a model of the effect of a transparent coating on top of an opaque or semi-transparent paint layer.

[0097] The present disclosure also provides a system for visualizing the appearance of at least two materials. The system comprises a display device, at least one processor and at least one memory comprising program instructions configured to cause the processor to perform the above-described method using the display device. The system can further comprise the above-described first and / or second appearance capture device, and can be correspondingly configured for receiving the appearance attributes from these appearance capture devices.

[0098] The present disclosure also provides a computer program product comprising program instructions that, when executed by a processor, cause the processor to perform the above-described method. The computer program product can comprise a non-volatile computer readable medium having the program instructions stored thereon. The non-volatile medium can comprise a hard disk, a solid state drive, a memory card, or any other type of computer readable medium as it is well known in the art. BRIEF DESCRIPTION OF DRAWINGS

[0099] In the following preferred embodiments of the present application are described with reference to the accompanying drawings, which are for explanatory purposes only and are not intended to limit the present application to the presently preferred embodiments. In the drawings,

[0100] Figure 1 a schematic illustration of a method of visualizing the appearance of two materials is shown;

[0101] Figures 2-4 a schematic illustration of a display device showing a virtual object in an alternative way according to an embodiment of the present application is shown;

[0102] Figure 5 a flowchart illustrating Figure 1 the method;

[0103] Figure 6A schematic diagram showing a hardware-oriented view of an example color formulation system is shown.

[0104] FIG. 7 shows a perspective view of an example appearance capture device according to the prior art;

[0105] FIG. 8 shows a perspective view of a measurement array of the appearance capture device in FIG. 7;

[0106] Figure 9 A graph illustrating an example discrete color table is shown.

[0107] Figure 10 An example texture of a discrete texture table is shown.

[0108] Figure 11 A graph illustrating an example discrete texture table is shown.

[0109] Figure 12 A schematic illustration of a method for generating a target texture based on a plurality of source textures is shown.

[0110] Figure 13 A schematic pixel value histogram is shown.

[0111] Figure 14 A schematic illustration of inserting a texture patch into a target texture is shown.

[0112] Figure 15 A flowchart of a method for generating an instance of an appearance model is shown.

[0113] Figure 16 A flowchart of a method for generating a target texture based on a plurality of source textures is shown; and

[0114] Figure 17 A schematic diagram illustrating the variation of information content along three dimensions is shown. DETAILED DESCRIPTION

[0115] Definitions

[0116] In this disclosure, reference to a singular can include the plural unless the context otherwise indicates. Specifically, the word “a” or “an” can mean one, or one or more.

[0117] The term “colorant” is to be understood as a constituent of a material that provides the appearance of color when light is reflected from or transmitted through the material. Colorants include pigments and dyes. A “pigment” is a colorant that is generally insoluble in the base constituent material of the material. Pigments can be from natural or synthetic sources. Pigments can include both organic and inorganic constituents. The term “pigment” also includes so-called “effect pigments” that produce special effects in the material. Examples include interference pigments and reflective particles or flakes. A “dye” is a colorant that is generally soluble in the base constituent material of the material.

[0118] The term "recipe" is to be understood as relating to a collection of information that determines how a material is to be prepared. The material can comprise a coating material, such as an automotive paint, a solid material, such as a plastic material, a semi-solid material, such as a gel, and combinations thereof. The recipe particularly comprises the concentrations of the ingredients from which the material is composed, such as a base and a colorant. A material that has been prepared according to a recipe can also be referred to as a "formulation".

[0119] The term "visual appearance" or simply "appearance" is to be understood broadly as the way in which an object reflects and transmits light, including but not limited to how an individual observing the object under various observation conditions perceives the color and surface texture of the object. The appearance also includes instrumental measurements of how the object reflects and transmits light.

[0120] One aspect of the visual appearance is color. The "color" of an object is determined by the portion of the spectrum of incident white light that is reflected or transmitted without being absorbed. The color of an object can be described by a "color attribute". In general, a color attribute indicates the spectral response of an object when the object is illuminated by incident light. In the context of the present disclosure, the term "color attribute" is to be understood broadly as encompassing any form of data that indicates the spectral response of an object when the object is illuminated by incident light. The color attribute can take the form of a color value in an arbitrary color space, for example in a trichromatic color space such as RGB or CIEXYZ, or in any other color space such as CIELAB (L*a*b*), or in the form of spectral data representing the spectral response of a material to incident light in an arbitrary format. In the context of the present disclosure, a color attribute can particularly comprise the reflectance values and / or absorption and scattering coefficients of a material at a plurality of wavelengths. A "discrete color table" is to be understood as relating to a collection of a plurality of sets of color attributes, each set of color attributes being associated with a different combination of illumination and observation directions.

[0121] Another aspect of visual appearance is texture. The term "texture" is to be understood broadly as referring to spatial variations in appearance across the surface of a material, both to variations on a micro- or meso-scale (i.e. variations on a scale where individual structural elements are generally not discernible to the naked eye), and also to variations on a macro-scale (i.e. variations on a scale where individual structural elements are discernible to the naked eye). Texture as understood in the present disclosure includes phenomena such as roughness, sparkle, and surface topography. Texture can be described by "texture attributes". In the context of the present disclosure, the term "texture attribute" is to be understood broadly as encompassing any form of data that is capable of quantifying at least one aspect of texture. Examples of texture attributes include global texture attributes such as a global roughness parameter or a global sparkle parameter. In some embodiments, texture attributes can include normal maps or height maps. In some embodiments, texture attributes can include image data. In some embodiments, image data can be associated with a particular combination of illumination and viewing directions. In such embodiments, texture attributes can include multiple sets of image data, each set of image data being associated with a different combination of illumination and viewing directions. A "discrete texture atlas" is to be understood as referring to a collection of multiple sets of texture attributes, preferably in the form of image data, each set of texture attributes being associated with a different combination of illumination and viewing directions. In some embodiments of the present disclosure, the images in a discrete texture atlas are generated from a set of source textures, and these images are therefore referred to as "target textures".

[0122] The term "appearance model" is to be understood as referring to a formal construction that describes appearance in mathematical terms using a plurality of material-related parameters referred to as "appearance attributes". Appearance attributes can include color attributes and texture attributes. An appearance model is preferably device- and platform-independent, i.e. it is independent of a particular measurement device that can have been used to determine the appearance attributes, and it is independent of a particular rendering platform for visualization. An appearance model provides a mathematical description of appearance in such a form and at such a level of completeness that it is possible to generate a visualization (i.e. a rendering and display) of a virtual object under arbitrary illumination and viewing conditions using the appearance model in combination with a geometric model of the virtual object. For any portion of a surface of a virtual object whose geometry is defined by a geometric model, and for any given illumination and viewing angle, the appearance model provides the necessary information to compute the appropriate appearance attributes.

[0123] A "virtual object" is an object that exists only virtually in a computer. A virtual object can or can not correspond to a real, tangible object. A virtual object can be mathematically defined by a geometric model and by associated appearance information.

[0124] A "geometric model" of a real or virtual object is to be understood as at least an approximate representation of the geometric structure of any surface of the object in three dimensions. For example, in some embodiments, the geometric model defines a curve along at least two mutually orthogonal directions. In other embodiments, the geometric model defines a plurality of polygons or faces. The geometric model can for example be represented by a CAD file.

[0125] The expression "instance of an appearance model" is to be understood as referring to a set of values of all appearance attributes of a particular appearance model, complemented with information enabling to identify the underlying appearance model. In other words, while an appearance model itself is a formal construction defining how to describe an appearance in terms of a set of appearance attributes, an instance of an appearance model comprises actual values of these appearance attributes specific to a given material, for example as determined by measurements and / or as derived from appearance attributes of individual components of a material. In particular, an instance of an appearance model can be provided in the form of a data file. Preferably, the data file is in a format independent of devices and platforms, such as the AxF TM format proposed by X-Rite. Thus, an AxF file can be considered as a representation of an "instance" of an appearance model. However, the present disclosure is not limited to a particular format of an instance of an appearance model, and an instance of an appearance model can be provided in another file format (e.g. MDL format), or it can even be provided in a form different from a file, for example as a data stream.

[0126] The luminance attribute can be represented in an appearance model having a bidirectional reflectance distribution function. A "bidirectional reflectance distribution function" (BRDF) is to be understood in the usual sense as a function defining how light is reflected at an opaque surface depending on the illumination and observation directions, thereby providing a ratio of reflected radiance exiting along the observation direction to the irradiance incident on the surface from the illumination direction. If the surface exhibits a spatial variation of this ratio, the BRDF is understood to provide an average of this ratio over the surface area. A "monochromatic luminance BRDF" is a BRDF providing a (possibly weighted) average of the ratio over all visible wavelengths of light, thereby modeling the overall luminance variation of the surface.

[0127] “Image data” is data representing an image. An image includes an image of an actual target surface or object as well as a synthetic image derived from one or more geometric models in combination with one or more sets of appearance properties. An image can take the form of a two-dimensional array of image elements (“pixels”), each pixel having a certain pixel value. The pixel value can represent the reflectance at the location of the pixel at a particular wavelength, the average reflectance over a particular wavelength range, or the average reflectance over all visible wavelengths. Thus, in some embodiments, the image data can be provided in the form of an array of pixel values. In other embodiments, the image data can be provided in compressed form or in transformed form. An “image data set” is a data set comprising or consisting of image data of at least one image, i.e. a data set representing one or more images.

[0128] Two images X, Y of the same size are considered “dissimilar” on a pixel-by-pixel basis if their pixel values are not correlated. Dissimilarity can be quantified by a suitably defined correlation measure of the pixel values. For the purposes of the present disclosure, two images are considered dissimilar if the absolute value of the Pearson product-moment correlation coefficient r XY of the pixel values in the images is not larger than 0.1, preferably not larger than 0.02. The Pearson product-moment correlation coefficient r XY is defined as the quotient of the covariance of the pixel values divided by the product of the standard deviations (i.e. square roots of the variances) of the pixel values of the two images:

[0129]

[0130] Here, x i denotes the pixel value in image X, denotes the arithmetic mean of the pixel values in image X, y i denotes the pixel value in image Y, denotes the arithmetic mean of the pixel values in image Y, N indicates the number of pixels in each image (which is the same for both images since the images have the same size), and the summation is over all pixels. The arithmetic mean of the pixel values in image X is defined in the usual way as

[0131]

[0132] The "Rusinkiewicz parameterization" is a description of the illumination and viewing directions in terms of a "halfway vector" and a "difference vector", the "halfway vector" being defined as the vector halfway between the incident and reflected light rays, the "difference vector" being the illumination direction in a reference frame in which the halfway vector lies at the north pole. The halfway vector in spherical coordinates in a sample-fixed reference frame in which the surface normal lies at the north pole can be denoted as (θ h , φ h ), where θ h is called the polar "halfway angle". The difference vector in spherical coordinates in a reference frame in which the halfway vector lies at the north pole can be denoted as (θ I , φ I ), where θ I is called the polar "difference angle". The remaining angles that define the illumination and viewing directions are the azimuth angles φ h and φ I . For details, see [Rus98], the content of which is incorporated herein in its entirety by reference.

[0133] In some embodiments of the present disclosure, barycentric coordinates are used to determine texture attributes associated with an appearance model. A "barycentric coordinate" is a coordinate in a coordinate system in which the position of a point of a simplex (i.e. a triangle in the case of two dimensions) is specified by the weights assigned to its vertices. In two dimensions, the simplex is a triangle. The barycentric coordinates of a point r in two dimensions with respect to vertices r1, r2, and r3 are given by three numbers λ1, λ2, λ3, such that

[0134] (λ1+λ2+λ3)r = λ1r1+λ2r2+λ3r3,

[0135] where at least one of the numbers λ1, λ2, λ3 is non-zero. It can be required that the barycentric coordinates are non-negative (i.e. it can be required that the point lies within the convex hull of the vertices). It can be required that the barycentric coordinates satisfy the condition λ1+λ2+λ3 = 1. They are then called "absolute barycentric coordinates".

[0136] An "appearance capture device" is a device capable of determining one or more appearance attributes of an object. Depending on the appearance attribute to be determined, the appearance capture device can take the form of, for example, a camera, a colorimeter, a spectrophotometer, or an imaging spectrophotometer.

[0137] A “spectrophotometer” is a device for determining the reflection and / or transmission properties of a surface or material under illumination with visible light as a function of wavelength, i.e. the spectral response of the object. Different types of spectrophotometers are known, which have different geometries and are optimized for different purposes. One important type is the “integrating sphere spectrophotometer”. An integrating sphere spectrophotometer comprises an “integrating sphere”, i.e. a hollow spherical cavity bounded by a diffusely reflecting white inner surface, which has at least one inlet port for illumination and at least one outlet port for observation. The integrating sphere produces a uniform scattering or diffusion effect. By multiple scattering reflections, light rays that hit any point on the inner surface are uniformly distributed to all other points. The influence of the original direction of the light is minimized. Examples of integrating sphere spectrophotometers are the models Ci7860 and Ci7500 by X-Rite. Other types of spectrophotometers only determine spectral information for a single narrow range of illumination directions (e.g. 45° to the surface normal) and a single narrow range of observation directions (e.g. 0° to the surface normal). Examples include the models 962 and 964 available from X-Rite. Yet other spectrophotometers, called “gonio-spectrophotometers” or “multi-angle spectrophotometers”, are able to determine spectral information for multiple combinations of different illumination and observation directions. An “imaging spectrophotometer” additionally has imaging capabilities, i.e. it can comprise one or more cameras to take one or more digital images of the object. An example of a multi-angle spectrophotometer with imaging capabilities includes the benchtop model TAC7 or the handheld models MA-T6 or MA-T12 available from X-Rite.

[0138] A material can be transparent, translucent or opaque. A material is “transparent” if it allows light to pass through the material without significant absorption and scattering of light. A material is “translucent” if it allows light to pass through, but light can be scattered inside or at any of the two interfaces. A material is “opaque” if it does not transmit light. A material can be opaque only in some spectral regions, while it is translucent or transparent in other spectral regions, and vice versa. For example, a material can strongly absorb red light, i.e. it is essentially opaque to red light, while it only weakly absorbs blue light, i.e. it is transparent to blue light. Some more complex materials, especially angle-appearance changing materials, can comprise a combination of transparent, translucent and opaque materials. For example, a paint coating can comprise an opaque base layer and a transparent clear coat. Opaque (reflective or interference flakes) or translucent pigments can be contained in the opaque, transparent or translucent layers of the paint coating.

[0139] For the purposes of the present disclosure, a material will be broadly considered to be "translucent" if a reasonably thin slice of the material transmits a appreciable portion of the incident radiant flux in at least a portion of the visible spectrum, e.g., if a slice having a thickness of 0.1 mm transmits at least 1% of the incident radiant flux in at least a portion of the visible spectrum. In this sense, the term "translucent" encompasses the term "transparent", i.e., for the purposes of the present disclosure, a transparent material is also considered to be translucent. In this sense, examples of translucent materials include many common plastic materials based on polymers, including but not limited to organic polymers (such as PET, PP, PE, PMMA, PS, PC, PVC, PTFE, Nylon), organic copolymers (such as styrene-butadiene copolymers), inorganic polymers (such as polysiloxanes), and many natural polymers. The translucent plastic materials can contain pigments and additives. However, other classes of materials can also be translucent in the sense of the present disclosure, including, e.g., silicate glasses or paper.

[0140] For the purposes of the present disclosure, a material is to be understood to be "homogeneous" if its subsurface light transport properties are invariant on a macroscopic or mesoscopic scale, e.g., on a scale larger than 1 pm. In particular, a homogeneous material does not include mesoscopic or macroscopic objects of varying appearance, such as flakes.

[0141] The term "macroscopic surface geometry" is to be understood as referring to the overall geometry of the product, excluding microscopic or mesoscopic surface structure, i.e., excluding variations in surface geometry on a microscopic or mesoscopic scale below, e.g., 1 mm. For example, a local variation in surface height from the local average of less than, e.g., 1 mm, can be considered a microscopic or mesoscopic surface structure, and thus the macroscopic surface geometry can be equivalent to the surface geometry after averaging over a length scale of at least 1 mm. If in mathematical terms, the surface geometry corresponds at least locally to a two-dimensional differentiable manifold in three-dimensional Euclidean space, then the surface geometry is "continuously curved".

[0142] The term "rendering" refers to the automatic process of generating a photo-realistic image of a scene by a computer program. In the present disclosure, the scene comprises at least one virtual object. The input information for the rendering operation comprises a 3D geometric model of the at least one virtual object, at least one set of appearance attributes associated with the virtual object, information on the position and orientation of the at least one virtual object in the scene, the lighting conditions, which can take the form of an environment map, and parameters characterizing the observer, such as the viewpoint, the focal distance, the field of view, the depth of field, the aspect ratio and / or the spectral sensitivity. The output of the rendering operation is an image of the scene, which comprises an image of at least part of the virtual object. Many different rendering algorithms are known, at different levels of complexity, and the software for rendering can employ a variety of different techniques to obtain the final image. It is generally impractical to keep track of every grain of light in the scene, as it would require too much computation time. Therefore, simplified techniques for modeling the transport of light are generally used, such as ray tracing and path tracing...

[0143] The term "visualization" encompasses rendering a scene comprising a virtual object and displaying the rendered scene. The term "display device" or simply "display" is to be understood as referring to an output device of a computer for presenting information in visual form. The display device can take the form of a computer monitor, a TV screen, a projector, a VR headset, a screen of a handheld device such as a smartphone or a tablet computer. The display device can be a touch screen. In some embodiments, the display device can be a "virtual light booth" as disclosed in EP 3 163 358 Al, in order to provide a particularly realistic impression of the rendered scene.

[0144] The term "database" refers to a collection of organized data that can be electronically accessed by a computer system. In simple embodiments, the database can be a searchable electronic file in any format. Examples include Microsoft Excel TM spreadsheet or a searchable PDF document. In more complex embodiments, the database can be a relational database maintained by a relational database management system using a language such as SQL.

[0145] The term “computer” or “computing device” refers to any device that can be instructed via a program to automatically perform a sequence of arithmetic or logical operations. Without limitation, a computer can take the form of a desktop computer, a notebook computer, a tablet computer, a smart phone, a programmable digital signal processor, etc. A computer typically includes at least one processor and at least one memory device. A computer can be a subunit of another device, such as an appearance capture device. A computer can be configured to establish a wired or wireless connection to another computer, including a computer comprising a database for querying. A computer can be configured to be coupled to a data input device, such as a keyboard or computer mouse, or to a data output device, such as a display or a printer, via a wired or wireless connection.

[0146] A “computer system” is to be understood broadly as encompassing one or more computers. If the computer system comprises more than one computer, these computers need not necessarily be at the same location. The computers within a computer system can communicate with each other via wired or wireless connections.

[0147] A “processor” is an electronic circuit that performs operations on external data sources, in particular memory devices.

[0148] A “memory device” or simply “memory” is a device for storing information for use by a processor. A memory device can include volatile memory, such as random access memory (RAM), and non-volatile memory, such as read-only memory (ROM). In some embodiments, a memory device can include a non-volatile semiconductor memory device, such as an (E)EPROM or a flash memory device, which can take the form of a memory card or a solid-state disk, for example. In some embodiments, a memory device can include a mass storage device with mechanical components, such as a hard disk. A memory device can store programs for execution by a processor. A non-volatile memory device can also be referred to as a non-transitory computer readable medium.

[0149] A “program” is a collection of instructions that can be executed by a processor to perform a particular task.

[0150] A “wired connection” is a connection via an electrical conductor. A wired connection can include one or more cables. A “wireless connection” is a connection that includes electromagnetic transfer of information between two or more points that are not connected by an electrical conductor. A wireless connection includes a connection via WiFi TM , Bluetooth TM , 3G / 4G / 5G mobile networks, optical communication, infrared, etc.

[0151] Exemplary embodiments of the method using color formulation software

[0152] Figure 1An exemplary embodiment of a method of visualizing the appearance of two or more materials in the context of vehicle repair is illustrated.

[0153] Suppose that a damaged vehicle needs to be repaired in a body shop. A technician at the body shop uses a handheld appearance capture device 52 to determine a set of appearance attributes 54 of a paint coating on an intact vehicle part. The paint coating is an example of a target material, and the intact vehicle part is an example of a target object comprising the target material. The appearance attributes 54 of the paint coating comprise on the one hand color attributes in the form of spectral data for a plurality of pairs of illumination and viewing directions, and on the other hand texture attributes in the form of image data for a plurality of pairs (possibly different pairs) of illumination and viewing directions.

[0154] The appearance capture device 52 transmits the measured appearance attributes 54 to a computer system. The computer system can comprise a local client computer at the premises of the body shop. The computer system can further comprise one or more remote computers at one or more locations different from the body shop. The local client computer can be for example a mobile electronic device, such as a notebook computer or a tablet computer. The remote computers can act as servers for the local client computer at the body shop.

[0155] The computer system executes several elements of software. In a first aspect, the computer system executes model generation software 102. The model generation software 102 generates a first instance 56 of a selected form appearance model based on the measured appearance attributes 54, the first instance 56 representing the appearance of the target material. The first instance of the appearance model is stored in a first AxF file in a device- and platform-independent form.

[0156] In a second aspect, the computer system executes color formulation software 104. The color formulation software 104 determines one or more candidate formulations 60 from a database of reference formulations. Each candidate formulation defines a candidate material (in the present example, a candidate paint coating) whose appearance attributes can match the measured appearance attributes 54 of the target material, the candidate material comprising one or more colorants in a base formulation. To determine the candidate formulations, the formulation software 104 retrieves predetermined appearance attributes associated with different reference materials from a database 106. In particular, the predetermined appearance attributes can be associated with individual colorants and / or with reference formulations comprising one or more colorants dispersed in a base formulation. In particular, the database 106 can comprise two sub-databases: a colorant database storing appearance attributes associated with individual colorants and base materials, and a formulation database storing appearance attributes associated with reference formulations. The appearance attributes in the database 106 can have been determined beforehand by performing measurements on reference objects comprising reference materials made according to the reference formulations or comprising ingredients of the reference formulations.

[0157] For example, to determine appearance attributes associated with individual colorants in the colorant database, reductions coated with formulations containing individual colorants at different concentrations can be prepared and measured; to determine appearance attributes in the formulation database, reductions coated with formulations that have been prepared according to reference formulations can be measured. The measurements can be performed using an appearance capture device, which can be the same type as the appearance capture device 52, or it can be a different type, as will be discussed in more detail below.

[0158] The formulation software uses the predetermined appearance attributes retrieved from the database 106 to compute candidate formulations whose associated appearance attributes are expected to "approximate" the measured appearance attributes of the target material. The appearance attributes of a candidate formulation "approximate" the measured appearance attributes of the target material if the appearance difference between the appearance attributes of the candidate formulation and the measured appearance attributes is small according to some predefined difference norm. In other words, the formulation software performs a minimization algorithm to determine candidate formulations that approximate a minimum of the difference norm. The formulation software 104 outputs one or more of the candidate formulations 60 thus determined. A technician selects and optionally manually modifies one of the candidate formulations. The formulation software 104 provides a set of candidate appearance attributes 64 for the selected (and optionally modified) candidate formulation 60. In some use cases, the candidate formulation corresponds to a single reference material, i.e., the candidate appearance attributes include the appearance attributes associated with the single reference material. In other use cases, the candidate appearance attributes are computed from measured appearance attributes associated with multiple reference materials. The reference materials can include individual ingredients of the candidate formulation in a manner that enables determination of appearance attributes associated with these individual ingredients. This is particularly useful when the candidate formulation is modified by changing the proportions of the individual ingredients of the formulation. The model generation software 102 receives the candidate appearance attributes 64 and generates a second instance 66 of the appearance model representing the expected appearance of a paint coating that has been prepared according to the candidate formulation 60. The second instance 66 of the appearance model is stored in a second AxF file.

[0159] In a third aspect, the computer system executes the rendering software 108. The rendering software renders the virtual object 72, i.e. it creates a photo-realistic digital image of the virtual object 72 based on a geometric model of the surface geometry of the virtual object 72, at least one instance of the appearance model, and the illumination and viewing conditions. The rendered virtual object 72 is displayed in the scene on the display device 70. The geometric model defines a continuous three-dimensional macroscopic surface geometry with surface normals distributed over a relatively large solid angle, i.e. it comprises curved or straight surface portions with a direction normal to the surface portions pointing in many different directions. In 3D computer graphics it is well known to use polygonal modeling as a method for modeling objects by representing or approximating their surfaces using polygonal meshes. These polygonal meshes are also considered to be continuous curved three-dimensional macroscopic surfaces if the polygonal meshes essentially behave as continuously curved surfaces when rendered. The rendering software generates a two-dimensional image of the virtual object 72 in a particular direction and under particular illumination conditions (assuming particular viewing conditions). The surface geometry, the direction and the illumination conditions are chosen in such a way that the rendered image gives a good impression of the appearance of the virtual object for a large range of angles between the surface normal, the illumination direction and the viewing direction, respectively, in order to allow the observer to assess the appearance of the rendered virtual object 72 for a large range of these directions at the same time.

[0160] The virtual object 72 has first and second portions 72a, 72b adjacent to each other. The first portion 72a is rendered using a first instance 56 of the appearance model, while the second portion 72b is rendered using a second instance 66 of the appearance model. A virtual dividing line 74 is visualized between the first and second portions 72a, 72b. The first and second portions 72a, 72b appear to meet at the virtual dividing line 74. The virtual dividing line can be visible or invisible. The rendering software can be configured to switch between a first mode in which the virtual dividing line is visible and a second mode in which the virtual dividing line is invisible based on a user input.

[0161] The display device 70 can be located at a body repair shop. For example, the display device 70 can be a touch screen display of a local client computer at the body repair shop. A technician at the body repair shop can use his finger or use a pointing device such as a digital pen, trackpad or mouse to move the virtual line 74 across the display device 70 and observe how the appearance matches or differs between the first and second instance of the appearance model as rendered on the virtual object 72, i.e. between the appearance of the actual paint finish of the car as measured and the expected appearance of the paint finish that has been prepared according to the candidate recipe. Optionally, the technician can change the shape of the virtual object, rotate the virtual object in space while changing the illumination and viewing angles. Optionally, the technician can change the illumination conditions, including the illumination direction and the selection of light sources.

[0162] In some embodiments, the virtual object 72 can have the shape of an actual part of the damaged car, e.g. the shape of the target object 50. To this end, the rendering software 108 can retrieve from a suitable memory, e.g. from a database 110 storing geometric data of a plurality of car parts, a three-dimensional geometric model corresponding to the geometry of the car part. To this end, the technician can provide, for example, the manufacturer and model of the damaged car or another type of vehicle information, such as a unique vehicle identification number, and one or more components to be rendered as the virtual object 72.

[0163] In other embodiments, the rendered virtual object 72 is a three-dimensional object which is different from an actual car part of the car to be repaired, but has a three-dimensional shape which is useful to check the appearance properties of various materials. Also in such embodiments, the rendering software 108 can retrieve the three-dimensional geometric model of the virtual object from a memory, e.g. from the database 110. Irrespective of whether the virtual object 72 represents an actual car part, the virtual object 72 preferably allows to check color and texture differences for a large number of angles between the surface normal, the illumination direction and the viewing direction at the same time.

[0164] The appearance of the first part 72a of the rendered virtual object does not need to perfectly match the appearance of the actual target object 50 when observed on the display device 70. In particular, the display device 70 does not need to be calibrated. What is important is that the appearance of the first and second part 72a, 72b is directly comparable to each other in both color and texture. This is possible even if the colors on the display are not true colors. By using the same appearance model for both parts, the direct comparability of the appearance of the two parts 72a, 72b is ensured.

[0165] It should also be noted that neither the appearance model itself nor the rendering software need to be perfect in order to ensure direct comparability. It is sufficient that the appearance model and the rendering software enable a reasonably realistic judgment of the difference between the two instances representing the actual coating material on the target object 50 and the coating material according to the candidate recipe 60.

[0166] If the visual comparison of the first and second portions 72a, 72b shows that the match is still not satisfactory, the technician can modify the candidate recipe 60 in the formulation software 104 either by selecting a different candidate recipe or by modifying the previously selected candidate recipe. The formulation software 104 provides the appearance properties of the modified candidate recipe to the model generation software 102, which creates a modified second instance of the appearance model based on these properties. The rendering software 108 can be instructed to render the second portion of the virtual object using the modified second instance of the appearance model, thereby replacing the previous candidate recipe in the visualization by the modified recipe, or the rendering software can be instructed to split the virtual object into three portions 72a, 72b, 72c in order to visualize the target material together with two candidate recipes, thereby adding another movable dividing line 76 to the visualization ("recursive splitter control").

[0167] When a satisfactory match has been obtained, an actual trial object 80, e.g. a mockup, can be produced with the selected candidate recipe 60. The appearance properties of the trial object 80 can be determined using the appearance capture device 82. The appearance properties of the trial object 80 can be compared to those of the target object 50 in order to determine whether the match is objectively within the tolerance range. This can be done by evaluating an appropriate difference norm between the measured appearance properties of the trial object 80 and those of the target object 50. Additionally or alternatively, a third instance of the appearance model can be generated using the appearance properties of the trial object 80, and the target material and the trial material can be visualized side by side on the display device 70 using the associated instances of the appearance model. Of course, it is also possible to directly compare the two objects if both the physical trial object 80 and the physical target object 50 are located at the same location. If the match is still not within the tolerance range, the formulation software 104 can further modify the candidate recipe 60 by taking into account the difference between the predicted appearance properties of the candidate recipe 60 and those that have actually been determined for the trial object 80.

[0168] The trial object 80 can be produced by the technician in the body shop. However, most often, the trial object 80 will be produced by a paint supplier at a location remote from the body shop. Therefore, the appearance capture device 82 used to determine the appearance properties of the trial object 80 can be different from the appearance capture device 52 used to determine the appearance properties of the target object 50.

[0169] In the above embodiment, it is assumed that a technician at the auto repair shop not only operates the appearance capture device 52 to determine a set of appearance attributes 54 of the target object 50, but also operates the formulation software 104 to define one or more candidate formulations and uses the display device 70 to compare the first and second parts 72a, 72b of the virtual object 72. However, these tasks can also be distributed among different people working at different locations. For example, the technician at the auto repair shop can transmit the appearance attributes 54 to a remote paint supplier, and a paint expert at the paint supplier can use the formulation software 104 to define candidate formulations and compare different parts of the virtual object to determine whether the expected match is satisfactory. Therefore, the computer system can include at least one computer under the control of the paint supplier, such as a computer executing the formulation software 104, model generation software 102, and rendering software 108. Many other possibilities exist for the division of tasks among the different entities involved in the process. Therefore, it will be apparent to the technician that the computer system can include several computers that may be located in different locations and may be under the control of different entities, communicating via wireless or wired data communication connections.

[0170] Alternative embodiments of the segmenter control

[0171] Figures 2-4 An alternative embodiment of how virtual objects can be displayed on a display device 70 is illustrated.

[0172] exist Figure 2 In this embodiment, the virtual object 72 is again divided into three parts 72a, 72b, and 72c to visualize the target material and the two candidate formulations. However, compared with... Figure 1 Compared to the previous embodiment, the second movable separator 76 is oriented perpendicular to the first movable separator 74 and extends horizontally between the vertical movable separator 74 and the vertical boundary of the display scene. In this way, the target material can be visualized side by side with two different candidate formulations on the virtual object 72. By moving the first and second separators, the user can not only easily assess the similarity between the target material and the candidate formulations, but also compare the candidate formulations with each other.

[0173] exist Figure 3In this embodiment, a blending zone 78 is defined around a virtual dividing line 74 that separates portion 72a from portions 72b and 72c. Within the blending zone 78, the virtual object 72 is visualized as having appearance properties that smoothly vary between the appearance properties of portion 72a and the appearance properties of the corresponding adjacent portions 72b or 72c. The appearance properties in the blending zone 78 can be calculated by interpolating between the appearance properties of the target material and the appearance properties of the corresponding candidate formulation. The blending zone 78 is associated with the virtual dividing line 78, and when the user moves the virtual dividing line 78, the blending zone 78 moves with the virtual dividing line 78, thereby always ensuring a smooth blend around the virtual dividing line 74. In this way, a more realistic impression of how the virtual object 72 would look is obtained if a portion of the target coating is over-sprayed with a coating according to the corresponding candidate formulation.

[0174] Interpolation between the appearance properties of the target coating and candidate formulations can be performed in a manner very similar to the interpolation process described below in the context of generating an appearance model from a finite set of available appearance properties. In particular, interpolation between the color properties of the target coating and candidate formulations can be performed directly in any linear color space. Interpolation between the texture properties of the target coating and candidate formulations can be performed using a modified statistical texture synthesis process, which will be described in more detail below.

[0175] like Figure 4 As illustrated, the arrangement of one or more dividing lines and / or one or more merging zones between two or more parts of the virtual object 72 is conceivable. The dividing lines do not have to be straight lines. They can also be curved. For example, in Figure 4 The diagram shows a ring-shaped blending region 78 that completely surrounds the first portion 72a of the virtual object. Users can move the ring-shaped blending region across the virtual object to evaluate the difference between a target coating and one or more candidate formulations. The target coating is visualized within the circular area defined by the blending region 78, while the one or more candidate formulations are visualized outside the blending region. Again, by associating different appearance attributes with different portions of the virtual object 72, more than one candidate formulation can be visualized. Figure 4 In the example, in addition to the first part 72a, two additional parts 72b and 72c of the virtual object 72 are also illustrated. These additional parts 72b and 72c are separated by a virtual separator line 76. The virtual separator line 76 can also be moved by the user.

[0176] A much larger arrangement of virtual partition elements and / or merging zones is conceivable.

[0177] Other possible ways of visualizing objects using instances of appearance models

[0178] In alternative embodiments, instances of the appearance model can be used to visualize scenes other than scenes that include only a single virtual object having two or more portions that are visualized using different instances of the appearance model. For example, a scene that includes two or more identical or different virtual objects, each of which is visualized using a different instance of the appearance model, can be visualized. In general, a scene includes one or more virtual objects, and different portions of the scene are visualized using different instances of the appearance model. For example, two identical shaped objects (e.g., two identical shaped vehicle parts, such as rear view mirrors) can be visualized side-by-side using first and second instances of the appearance model in combination with a common geometry model.

[0179] Flowchart

[0180] Figure 5An exemplary flowchart illustrating a method of visualizing the appearance of two materials is shown. In step 501, a user (e.g., a technician at a body shop) determines the appearance attributes 54 of a target object 50 by measuring the light reflection and / or transmission properties and optionally the texture properties of the target object with a first appearance capture device 52. In step 502, the appearance capture device 52 transmits these measured appearance attributes to a component of a computer system, e.g., to a handheld computer in the body shop. In step 503, the computer system generates a first instance 56 of an appearance model from the measured appearance attributes using model generation software 102. In step 504, the computer system reads the appearance attributes of one or more reference materials from a database 106. In step 505, the computer system determines one or more candidate formulations 60 using formulation software 104. The formulation software 104 can perform this step 505 by simply selecting one or more reference materials having appearance attributes similar to the measured appearance attributes 54 of the target object 50, by modifying the formulations of the retrieved reference materials, or by generating new candidate formulations 60 if no close matches are found in the database 106. In step 506, the computer system determines the appearance attributes 64 associated with the candidate formulations 60 using formulation software 104. In step 507, the computer system generates a second instance 66 of the appearance model from the appearance attributes 64 associated with the candidate formulations 60 using model generation software 102. In step 508, the computer system visualizes a virtual object 72 based on both instances 56, 66 of the appearance model, a geometric model of the virtual object 72, and illumination and viewing conditions using rendering software 108. In step 509, the user compares the rendered virtual object portions 72a, 72b to obtain an acceptable appearance match. In step 510, if the visualized virtual object portions 72a, 72b do not provide an acceptable visual match, the user can modify the formulation or select a different candidate formulation 60. Once an acceptable rendered match is obtained, in step 511, the user prepares a test object 80 using the candidate formulation. In step 512, the same or a different user determines the appearance attributes of the test object 80 using a second appearance capture device 82. In step 513, the appearance attributes of the test object 80 are transmitted to the computer system, and the computer system determines a modified formulation, if needed, to refine the match using formulation software 104. Steps 506 through 513 can then be repeated for the modified formulation.

[0181] Computer system: exemplary hardware

[0182] Figure 6 FIGS. 1-3 illustrate a method of visualizing the appearance of two materials that can be implemented in a system such as the system 10 of FIG. 1. Figure 1 and 2An exemplary hardware-oriented block diagram of a computer system used in the methods illustrated in the middle. In this example, the computer system comprises two main components: a local client computer (e.g. a laptop or tablet computer) 300, which can be located at a car body repair shop, and a remote server computer 360, which can be located at the premises of a paint supplier.

[0183] As it is well-known in the art, the various components of the local client computer 300 communicate with each other via one or more buses 301. The local client computer 300 comprises one or more processors 310. As it is well-known in the art, the processor(s) 310 can comprise e.g. a single- or multi-core CPU and a GPU. The local client computer 300 further comprises one or more non-volatile memory devices 320, such as a flash memory device and / or a hard disk drive. The non-volatile memory 320 stores, among other things, an operating system 321 of the local client computer 300 and several applications, including the model generation software 102 and the rendering software 108. The non-volatile memory 320 also stores user data. The local client computer 300 further comprises a random access memory (RAM) 330, and an input / output (I / O) interface 340 and a communication interface 350. Attached to the communication interface are a display device 70 and a pointing device 90. The communication interface 350 can comprise one or more of e.g. an Ethernet interface, a WiFi interface, a Bluetooth interface, etc. The communication interface can be used for communication with the remote server 360. TM

[0184] The remote server computer 360 can be set up similarly to the client computer 300. It stores the formulation software 104 for execution. The client computer 360 can comprise or be connected to the database 106. Communication between the server computer 360 and the client computer 300 can take place via a wired or wireless network, e.g. via a LAN or WAN, in particular via the Internet.

[0185] The client computer also communicates with the first and / or second appearance capture device 52, 82 via the communication interface 350.

[0186] In other embodiments, certain functions of the client computer 300 are instead transferred to the server computer 360. In particular, the server computer (rather than the client computer) can execute the model generation software 102 and / or the rendering software 108. This can be useful if the client computer is a "thin" client with limited computing power.

[0187] In yet other embodiments, the computer system consists of only a single computer executing all of the above-mentioned software components.

[0188] ​Exemplary appearance capture device

[0189] In the above examples, the appearance capturing devices 52 and 82 are preferably multi-angle spectrophotometers with imaging capabilities. Such devices are known per se. For example, the MA-TX series available from X-Rite can be used.

[0190] Figure 4 and Figure 5 An exemplary handheld appearance capturing device that can be used in connection with the present application is illustrated in Fig. 1 of US20140152990A1. The appearance capturing device of US20140152990A1 is described in more detail in US20140152990A1, the content of which is incorporated herein by reference in its entirety for the teaching of a handheld appearance capturing device. Figure 4 and Figure 5 The appearance capturing device of US20140152990A1 is described in more detail in US20140152990A1, the content of which is incorporated herein by reference in its entirety for the teaching of a handheld appearance capturing device.

[0191] The appearance capturing device is configured for capturing a visual impression of a measurement object. In the following, the appearance capturing device is also referred to as "measurement device" or simply as "device". In the following, the term "measurement array" is understood to mean the sum of components of the handheld measurement device for illuminating a measurement point on the surface of a measurement object and capturing the light reflected by the measurement point and converting it into a corresponding electrical signal. The term "device normal" is understood to mean an imaginary straight line which is fixed relative to the device and which extends substantially through the center point of the measurement opening of the measurement device and which is perpendicular to the surface of the measurement object when the measurement device is positioned on a planar surface of the measurement object. The plane of the measurement opening is usually parallel to the surface of the measurement object so that the device normal is also perpendicular to the measurement opening. The term "vertical" is understood to mean the direction of the device normal. A vertical cross-section is thus understood to mean a planar cross-section in a plane which contains the device normal or which is parallel to the device normal. In the following description of the measurement device, directions and / or angles are relative to the device normal which is spatially fixed relative to the measurement device.

[0192] The handheld measurement device shown in Fig. 7 is indicated as a whole by the reference HMD. It comprises a housing H which accommodates the measurement array, as shown in Fig. 7, and an electronic control array (not shown) which controls the measurement array. Two grip parts 1 and 2 are realized laterally on the housing H. A wrist strap 3 is arranged on the upper side of the housing H. A display array 4 is provided on the front side of the housing H. Operating members (not shown) are arranged on the upper side of the housing H.

[0193] The underside of the housing H comprises a housing base 5 which is reinforced by a base plate 7, which is equipped with a measurement opening 6. The housing base 5 comprises a hole (not indicated with a reference) in the region of the measurement opening 6, so that light can exit the interior of the housing through the hole and the measurement opening 6 and, conversely, light from the outside can enter the interior of the housing through the measurement opening 6 and the hole. Three support members 7a, 7b and 7c are arranged around the measurement opening 6 on the base plate 7 and contribute to enabling the measurement device to be positioned correctly even on a curved measurement surface, so that the device normal coincides completely or at least largely with the normal of the measurement surface in the center point of the measurement point.

[0194] The device normal is indicated in Figure 7 by the reference DN. It is perpendicular to the base plate 7 and extends through the center point of the measurement opening 6.

[0195] The arrangement of the measurement array is illustrated in Figure 8. It comprises an arcuate body 10 which is fixedly held in the housing H and in which the optical and / or optoelectronic components of the measurement array are arranged. In the exemplary embodiment shown, these components comprise seven illumination devices 21, 22, 23, 24, 25, 26 and 27 and three pick-up devices 31, 32 and 33. In addition, an illumination device 28 for diffuse illumination is provided immediately adjacent to the measurement opening 6.

[0196] The seven illumination devices 21 to 27 illuminate measurement points on the surface of the measurement object in different illumination directions relative to the device normal DN. For example, the optical axes of the illumination devices 21 to 27 can be oriented at angles of -60°, -45°, -30°, -20°, 0°, +30° and +65° relative to the device normal. All seven illumination devices 21 to 27 are arranged so that their optical axes lie in a common plane which contains the device normal DN, which plane is referred to hereinafter as the system plane SP.

[0197] Two of the three pick-up devices 31 to 33 are implemented as spectral measurement channels; the third pick-up device is implemented as a spatially resolved color measurement channel. The pick-up devices receive the measurement light reflected in the area of the illuminated measurement point of the measurement object in the system plane SP at observation angles of +15° and +45°. The two pick-up devices 31, 32 forming the spectral measurement channels comprise two spectrometers 31a and 32a to which the measurement light is fed via lenses and optical fibers 31c and 32c. The pick-up device 33 forming the spatially resolved measurement channel comprises a color-enabled (RGB) camera 33a to which the measurement light is applied via a beam splitter and a lens (not shown). The beam splitter is located in the pick-up light path of pick-up device 32 and directs a portion of the measurement light from the arc-shaped body 10 laterally onto the camera 33a at an observation angle of +15°. Thus, the pick-up devices 32 and 33 share the measurement light and receive it at exactly the same observation angle.

[0198] The measurement geometry is contrary to the ASTM standards E2194 and E2539, where two specular illuminations at 15° and 45° and six specular spectral channels at 0°, 30°, 65°, -20°, -30° and -60° are defined for measurements on metallic and pearlescent pigments. An additional illumination device 22 at an angle of -45° is provided for measuring gloss in combination with pick-up device 31.

[0199] The two spectrometers 31a and 32a spectrally resolve the measurement light fed to them at observation angles of 45° and 15°, respectively, and each time of measurement generate a set of spectral measurement values, each corresponding to an intensity in a different wavelength range. The spectrometers 31a, 32a do not spatially resolve the measurement light, i.e. they spectrally resolve the entire measurement light they receive.

[0200] The RGB camera 33a resolves the measurement light fed to it at an observation angle of 15° both spatially and spectrally. The spectral resolution is limited to three channels according to the three colors RGB. The RGB camera accordingly generates a raw data set of 3*n measurement values per time of measurement, where n is the number of resolved pixels.

[0201] An illumination device 28 providing diffuse illumination is provided so that the measurement device also supports a measurement mode with diffuse illumination conditions. The illumination device 28 is configured as a LED background illumination which directly illuminates the measurement object from a large spatial angle. It comprises two rows of white light emitting diodes arranged on both sides of the measurement opening 6 and two inclined diffusion films each assigned to one row for homogenizing the illumination. The two rows of LEDs can be controlled separately by a control array.

[0202] Of course, other types of illumination devices can also be employed depending on the envisaged application.

[0203] The aforementioned handheld measuring device is equipped with seven illumination devices and three pickup devices for measurement purposes. Other combinations of illumination devices and pickup devices are possible. The illumination devices do not necessarily need to be arranged in a plane.

[0204] The output of the measuring device includes a set of appearance attributes. Some of these appearance attributes are color attributes, such as RGB, CIELAB, or CIEXYZ values, either in the form of spectral measurements determined by a spectrometer or color values ​​derived from it. Other appearance attributes are texture attributes, such as texture data, either in the form of raw image data obtained from an RGB camera or in the form of image data derived from it, such as monochrome grayscale image data. This set of appearance attributes obtained from the measuring device is typically sparse compared to the appearance attributes required to generate an instance of the appearance model, as will be discussed in more detail in the next section.

[0205] Examples of appearance models

[0206] As outlined above, many different appearance models have been proposed in the art. An example, particularly useful for describing the appearance of coatings on modern vehicles, will be described below. This exemplary appearance model is an improvement upon the model proposed in [RMS+08]. The contents of [RMS+08] are incorporated herein by reference in their entirety for the purpose of teaching the exemplary appearance model. The appearance model currently discussed is also referred to as the "CPA2 model".

[0207] The appearance model assumes the coating consists of two outermost layers: an opaque or translucent color layer, which is covered by a smooth, transparent coating. The color layer may contain highly reflective particles (so-called "flakes"). Incident light with direction i travels along its "refracted" incident direction. The light is refracted into the transparent coating before reaching the color layer. The reflected light travels along the outgoing direction before being refracted into the surrounding air in one direction (o). Propagation occurs through the transparent coating. Refraction is modeled using standard geometric optics, assuming the typical refractive index of the transparent coating.

[0208] The appearance model of the color layer is divided into three parts:

[0209] (a) Monochromatic luminance BRFD component, which models the spatial and wavelength-averaged overall reflection behavior (luminance variation) of the material. The luminance BRFD component can represent high frequencies in the angular domain.

[0210] (b) The spectral portion in the form of a discrete color table, which describes the low-frequency angular variations (color shifts) of the color.

[0211] (c) a BTF part in the form of a texture map that captures the effect of the reflective flakes in the color layer. The BTF part is represented by a plurality of fixed-size images, each image representing the texture at a specific combination of illumination and viewing angles.

[0212] The three parts are combined into a spatially varying BRDF function that represents the appearance of the paint surface as follows:

[0213]

[0214] Here,

[0215] i denotes the direction of the incoming light in air,

[0216] o the direction of the reflected light in air,

[0217] x a position on the surface of the object,

[0218] n the surface normal at position x,

[0219] is the (refracted) direction of the incoming light in the transparent coating,

[0220] is the (refracted) direction of the reflected light in the transparent coating,

[0221] F(n, d, η) is a function that approximates the Fresnel reflection at the medium boundary for relative refractive index η (or the Fresnel formula itself),

[0222] is a monochromatic BRDF model, see part (a) below,

[0223] is an interpolated color map, see part (b) below,

[0224] is an interpolated texture map, see part (c) below,

[0225] δ(d1, d2) is the Dirac delta function on O(3), which is non-zero only when d1 = d2, and

[0226] r(n, d) is the direction of the reflection at the surface for normal n.

[0227] (a) Example of a luminance BRDF part

[0228] The monochromatic luminance BRDF part is modeled using a multi-lobe microfacet based BRDF model:

[0229]

[0230] Here,

[0231] denotes the halfway direction (also called "halfway vector") between and ,

[0232] a denotes the diffuse albedo,

[0233] K denotes the number of BRDF lobes,

[0234] s k denotes the multiplier of the k-th lobe,

[0235] D denotes the microfacet normal distribution function according to the chosen model,

[0236] a k denotes the roughness parameter of the k-th lobe,

[0237] F 0,k denotes the Fresnel function parameter of the k-th lobe, which can be the index of refraction or the exponent of the reflection at normal incidence, and

[0238] G denotes the so-called geometric structure term according to the chosen model.

[0239] In some embodiments, the Cook Torrence BRDF model with Schlick-Phong approximation is used. Here, D, F and G are defined as follows:

[0240]

[0241] F(n, d, F 0,k ) = F 0，k + (1 - F 0,k )(1 - 〈n, d>) 5

[0242]

[0243] Here,

[0244] 0 h denotes the polar angle of the halfway vector in the fixed object reference frame, where the surface normal n defines the z-axis ("north pole"),

[0245] denote the z-components of the vectors and in said reference frame, respectively.

[0246] The free parameters of this model are a, s k , a k and F 0，k (for k = [1..K]). Typically, K is chosen to be between 1 and 3.

[0247] (b) Example of a spectral part

[0248] The spectral portion is modeled using a discrete color table, which comprises values ​​of one or more color attributes (e.g., hue and saturation) for combinations of multiple directions. The values ​​of these color attributes will be simply referred to as "color values" below. Advantageously, the Rusinkiewicz parameterization [Rus98] is used to indicate the directions. In a simplified embodiment, the color table is bivariate, meaning the values ​​of the color attributes depend only on the polar angles in the Rusinkiewicz parameterization, as will be explained below. These polar angles will also be referred to below as the "coordinates" of the color table.

[0249] Figure 9 An exemplary representation of a bivariate discrete color table is shown. On the horizontal axis, the mid-range vector... polar angle θ h It is used as the first coordinate. This polar angle is also called the "flake angle" because it defines the angle between the surface normal n of the paint layer and the normal of the flakes in the paint layer, which will cause incident light to change from the incident direction. Reflected in the direction of emission On the vertical axis, the mid-way vector and incident direction The range angle θ between I Used as the second coordinate. Range angle θ I Incident direction in the transformed reference frame The polar angle, in the transformed reference frame, the mid-way vector Define the z-axis. It can be interpreted as indicating the direction of incidence relative to the slice normal. The color table contains each pair of coordinates (i.e., the polar angle θ). h and θ I Each combination of coordinates represents one or more color values. The color table is defined for each pair of coordinates containing one or more color values. Figure 9 The circle in the diagram represents such a pair of coordinates (polar angle θ). h and θ I The combination of these (positions) will be referred to as “positions” in the color table below. Positions in the color table are spaced regularly. Therefore, the color table is represented by a regular rectangular grid of positions and their associated color values. The positions of two exemplary entries in the color table are labeled “a” and “b”.

[0250] The appearance model assumes that the paint material has isotropic reflection. For isotropic materials, color does not depend on the mid-range vector in a fixed object reference frame. azimuth φ h Therefore, the color table does not consider the azimuth angle φ. h .

[0251] Furthermore, the model assumes that the effect particle ("flake") itself is also isotropically reflecting, and that the color shift is dominated by the specular reflection component of the reflected light. Thus, the color only weakly depends on the incident direction in the transformed reference frame of the azimuth angle φ I . Thus, the color table does not consider the azimuth angle φ I either.

[0252] In Figure 9 the color table at larger values of θ h and θ I , the blank areas indicate combinations of angles that are "forbidden" due to refraction and total reflection at the interface of the transparent coating and the surrounding air. The color table does not contain any color values for these combinations of angles.

[0253] (c) Example of a BTF part

[0254] The appearance model comprises a bidirectional texture function (BTF) model. Generally, a BTF model is a multidimensional function that depends on the planar texture coordinates (x, y) as well as the observation and illumination spherical angles. The BTF part is modeled by a discrete texture table. Each entry in the texture table is a texture slice, i.e. an image that represents the effect of a flake for a particular combination of illumination and observation directions. In a simple embodiment, the Rusinkiewicz parameterization is also used for the texture table, and the texture table is again bivariate, which is again parameterized by the angles θ h and θ I .

[0255] Figure 10 A simplified example of a texture slice is shown in Fig. 2. The bright spots in this figure indicate the reflections of the flake whose flake normal is oriented at the associated flake angle φ h and which is available at the associated difference angle φ I .

[0256] Figure 11 The discrete nature of the bivariate texture table is illustrated in Fig. 3. In the appearance model, a texture slice is provided for the discrete locations (defined by their coordinates θ h , θ I ) marked by the circles. Due to the higher storage space requirement for each texture slice as compared to the color values in the color table, the spacing between the discrete values of θ h and θ I in the texture table can be larger than in the color table, respectively. As in the color table, there are no entries for "forbidden" angles for the texture table. The locations of three exemplary texture slices are marked as a', b' and c'.

[0257] Generating instances of appearance models

[0258] In the presently proposed method, only a limited set of appearance attributes for a limited number of combinations of directions is available from the measurements with the handheld appearance capture device or from the formulation software These appearance attributes are typically not available for the same combinations of directions of incident and outgoing light as those required by the discrete color and texture patches in the appearance model. Furthermore, the available appearance attributes are not necessarily of the same type as those required by the appearance model. For example, the available appearance attributes typically do not include the parameters of the single-color luminance BRDF model as described above.

[0259] Therefore, in order to generate an instance of the appearance model, it is necessary to determine the appearance attributes of the appearance model from the limited set of available appearance attributes. It should be noted that the set of appearance attributes of the appearance model can have a much larger cardinality than the limited set of available appearance attributes. Therefore, generating an instance of the appearance model can involve interpolation and extrapolation.

[0260] As already discussed above, the free parameters of the luminance BRDF part are a, s k , a k and F 0,k (for k = [1..K]). These parameters can be determined using non-linear optimization as known in the art [RMS+08].

[0261] Next, the bi-variate color table needs to be populated. The color attributes can only be available for a few pairs of angles θ h and θ I , which are marked with crosses in the diagram of Figure 9 Each pair of angles (θ h , θ I ) for which a color attribute is available will be referred to in the following as a "sample point". The sample points define a convex hull, which is shown as the dashed area in Figure 9 Each position (θ h , θ I ) for which a color value is needed in the bi-variate color table will be referred to as a "target position" with "target coordinates". In the present example, the target positions are distributed on a regular grid, and therefore these positions can also be referred to as "grid points". The color values at the target positions will be referred to as "target color values". They can be of the same format as the available color attributes, or they can be of a different format. For example, the available color attributes can include spectral data, while the color table can include reduced values (e.g. trichromatic values such as RGB or CIEXYZ, or values in another color space such as CIELAB). For the following discussion, it is assumed that a transformation between the format of the available color attributes at the sample points and the format of the color values in the color table is known.

[0262] By standard interpolation and extrapolation procedures, color values at target locations can easily be interpolated and extrapolated from the available color attributes at the sample points. If the format of the color values in the color table is different from the format of the available color attributes at the sample points, the necessary transformation can be applied before or after the interpolation / extrapolation, as long as both the available color attributes at the sample points and the color values in the color table are expressed in a linear color space. If the color values in the color table are not expressed in a linear color space (e.g., as CIELAB values), the interpolation / extrapolation is first performed in a linear color space, and the transformation is applied only then.

[0263] For example, for interpolation within the convex hull, nearest-neighbor interpolation, inverse distance weighting [Sh68] or Gaussian process regression [W98] between the color attributes at the available sample points or between the color values obtained from these color attributes by said transformation can be used, as it is well known in the art.

[0264] In a preferred embodiment, for extrapolation at target locations with a theta h lower than the minimum theta h of the convex hull or a theta h higher than the maximum theta h of the convex hull, the color attribute (or the derived color value) at the nearest neighbor sample point can be used. Nearest neighbor sample point is to be understood as the sample point with the smallest Euclidean distance to the target location. In Figure 9 , this is illustrated for a target location "a". The color value at the target location "a" is derived from the available color attribute at sample point A, which is the nearest neighbor sample point to the target location "a".

[0265] For extrapolation at all other target locations outside the convex hull, in particular those locations with a low or high theta I , the preferred embodiment will not only use the nearest neighbor sample point, but will interpolate between the color attributes (or the derived color values) of the two sample points closest to the target location. In Figure 9 , this is illustrated by the entry at target location "b". The two closest sample points are points B and C. Thus, the color value at the target location "b" will be determined by interpolation between the color attributes (or the derived color values) at sample points B and C.

[0266] Of course, other "well-behaved" interpolation and extrapolation methods can be used as well, as they are known in the art.

[0267] Finally, the bivariate texture table needs to be populated. The texture samples in the form of images can only be available for a few pairs of angles theta h and theta I(Hereinafter referred to again as “sample points”) are available. For example, as described above, the appearance capture device 52 may have one or more cameras. The available texture samples may be image data for multiple illumination angles and a single viewing angle (e.g., in the case of a single RGB camera) or image datasets for multiple illumination angles and multiple viewing angles (e.g., in the case of multiple RGB cameras).

[0268] exist Figure 11 In the example, only six texture samples are available. The corresponding sample points are in Figure 11 The points are marked with a cross and labeled A', B', C', D', E', and F'. The convex hulls of these sample points are again shown as dashed lines. The texture samples at the sample points are called the "source textures." Texture slices need to be determined to fill the bivariate texture table with respect to the angle θ. h and θ I This is again referred to as the "target location" with "target coordinates." Similar to the case of the color table, in this example, these points are arranged on a regular grid and therefore can also be called "grid points." Therefore, the texture slice at the target location is called the "target texture." They can be determined from the available source textures as follows:

[0269] -For each target location (θ) in the texture table h ,θ I This determines whether the target location is inside the convex hull.

[0270] - If the target location is inside the convex hull, then "Statistical Texture Composition" is used to interpolate the target texture at that location from the surrounding source textures, as explained in more detail below. For example, for Figure 11 Position a' in the text will be synthesized using statistical texture.

[0271] - If the target location is not within the convex hull, the target texture is defined using the source texture of the nearest neighbor sample point (“First Strategy”), or a constant texture without any spatial variation is used, i.e., an “empty” texture that does not contain any reflections from the slab (“Second Strategy”). In one possible implementation, for a maximum θ smaller than the available sample points... h The value θ h The first strategy can be adopted, and for the maximum θ greater than the available sample points... h θ h Adopt the second strategy. For example, for Figure 11 For position b', the first strategy would be appropriate, while for position c', the second strategy could be used. It is also possible to use statistical texture synthesis for target positions outside the convex hull, provided that at least two sample points are sufficiently close to the target position. For example, using the source textures at sample points B' and D', statistical texture synthesis for target position b' might be appropriate.

[0272] Statistical texture synthesis

[0273] An exemplary embodiment of a statistical texture synthesis in Figures 12 to 16 will now be explained with reference to Figure 11 a target position a'.

[0274] Initially, three source textures at different sample points are selected for the target position a'. To this end, a Delaunay triangulation is created for all sample points A' to F'. In two dimensions, as in the present example, the result of a Delaunay triangulation is a set of triangles. In Figure 11 , the edges of the resulting triangles are illustrated by straight lines between the respective sample points A' to F'. Now, the triangle containing the target position a' is selected, and the sample points at the corners of this triangle are determined. In the present example, these are the sample points B', D' and E'. For the subsequent process, the source textures at these three sample points are used.

[0275] Next, interpolation weights for the three source textures are determined. To this end, absolute barycentric coordinates of the target position a' with respect to the selected triangle are determined, i.e. non-negative barycentric coordinates λ1, λ2, λ3 that satisfy the condition λ1+λ2+λ3=1. These barycentric coordinates are then used as interpolation weights for the source textures at the corners of the selected triangle. In Figure 11 the example, the absolute barycentric coordinates with respect to sample point B' are rather large, while the absolute barycentric coordinates with respect to sample points D' and E' are much smaller. Therefore, the source texture at sample point B' will receive a relatively large interpolation weight, while the interpolation weights of sample points D' and E' will be much smaller. For all other source textures, the interpolation weight is set to zero.

[0276] Now, a target texture is synthesized for the target position a' from the three source textures at sample points B', D' and E'. This will be explained with respect to Figure 9 .

[0277] To this end, one of the three source textures is randomly selected, with probabilities proportional to the interpolation weights of the source textures. In Figure 12 the example, the source texture 211 at sample point B' has been randomly selected.

[0278] From this source texture 211, now a texture patch, i.e. a small portion of the source texture, is extracted at a random position within the source texture, which is indicated by a small rectangle in the source texture. In Figure 12 the example, a texture patch 212 is extracted. Now, the extracted texture patch 212 is modified such that one of its statistical properties at least approximately matches the corresponding average statistical property. In the present example, the extracted texture patch 212 is modified such that its pixel value histogram approximately matches the average pixel value histogram 202. This will now be explained in more detail.

[0279] Figure 13 An exemplary pixel value histogram is illustrated in FIG. 3. A pixel value histogram contains a number of relative frequency values, one such value corresponding to each discrete pixel value in the image. For example, if the pixel values range from 0 to 255, then for each pixel value between 0 and 255, the pixel value histogram will include the relative frequency of that pixel value in the image. In some embodiments, the pixel values can be binned, each bin corresponding to a range of pixel values, and the pixel value histogram can accordingly include a reduced number of relative frequency values, each value representing the relative frequency of pixel values in one of the bins. The pixel value histogram can be used to assess the distribution of brightness in the image, including overall brightness and contrast. For example, Figure 13 A pixel value histogram of the image in FIG. 1 would indicate that the image contains some bright spots or regions before the relatively dark background.

[0280] In Figure 12 In FIG. 3, all pixel value histograms are shown in a very simplified manner with only five bins for each pixel value histogram. The pixel value histogram of the source texture 211 from which the texture patch 212 is extracted is shown as histogram 213. The average pixel value histogram is shown as histogram 202. This average histogram has been obtained as a weighted average of the pixel histograms 213, 222 and 232 of the three source textures 211, 221 and 231 at sample points B', D' and E', respectively, the weights being the interpolation weights as determined above. The pixel values of the texture patch 212 are now modified in order to obtain a pixel value histogram that more closely matches the average pixel value histogram 202. The modification is performed by applying a monotonic, non-decreasing point-wise transformation to each pixel value in the texture patch 212. Histogram matching algorithms for finding a suitable transformation are well known in the field of digital image processing associated with brightness and contrast modification. The modified texture patch is illustrated as texture patch 214, and the resulting pixel value histogram is illustrated as histogram 215. The resulting pixel value histogram 215 now closely matches the average pixel value histogram 202.

[0281] The modified texture patch 214 is now inserted into the target texture 201 so that it seamlessly fits the texture content already in the target texture 201. To this end, the first modified texture patch 214 can simply be placed in a corner of the target texture 201. Each subsequent texture patch is inserted onto the target texture by a technique known as "MinCut" or "Graphcut". Reference [Kw03]. Very simply, a seam is computed that maximizes the visual smoothness between the existing pixels and the newly placed patch. The texture patch and the existing content are stitched together along this seam.

[0282] In Figure 14Fig. 6 illustrates one such insertion step. In this example, a modified texture patch 214 is to be inserted into a target texture 201 which already includes a texture patch 203. The position 204 of the newly inserted patch 214 is chosen so that the newly inserted patch 214 overlaps the existing patch 203. In the overlapping region, a seam 205 is calculated so that the existing pixels and the newly inserted pixels along the seam appear visually smooth, i.e. no visible boundary is created along the seam, as explained in detail in [Kw03]. The existing patch 203 and the newly inserted patch 214 are stitched together along this seam, and the remaining parts of each patch are discarded. This results in a larger patch 206.

[0283] The process is now repeated as often as needed to completely fill the target texture 201.

[0284] Flowchart for filling a discrete texture atlas

[0285] Figure 15 An exemplary process illustrating an example of generating an appearance model is shown in the flowchart of Fig. 6. In step 601, the model generation software receives a set of available appearance attributes, which can have been determined by a capturing device that measures the appearance of a target object 50 or by formulation software that determines the ingredients of a candidate formulation 60, including the proportional amounts of the ingredients in the candidate formulation 60. In step 602, the software performs a fit of a luminance BRDF model to the available appearance attributes. In step 603, the software populates a discrete color table based on the available appearance attributes. In step 604, the software populates a discrete texture table based on the available appearance attributes. To do this, it determines the target textures that form the entries of the discrete texture table.

[0286] Flowchart for statistical texture synthesis

[0287] Figure 16A flowchart illustrating an exemplary process for determining a target texture associated with a target coordinate set is shown. In step 701, the model generation software receives a plurality of source textures and their associated coordinates. In step 702, the software creates a Delaunay triangulation. In step 703, the software identifies the simplex of the Delaunay triangulation that contains the target coordinate. In step 704, the software determines the barycentric coordinates of the target coordinate with respect to the identified simplex and stores these as interpolation weights. In step 705, the software randomly selects one of the source textures at a corner of the selected simplex with a probability according to the interpolation weights of the source textures. In step 706, the software randomly extracts a texture patch from the selected source texture. In step 707, the software modifies the patch to match its pixel value histogram to the average pixel value histogram. In step 708, the software inserts the modified patch into the target texture so that it seamlessly fits into the existing texture content in the target texture. Steps 705 through 708 are repeated until the target texture is completely filled. The completed target texture can then be used as an entry in the discrete texture table.

[0288] Computing the texture atlas of a mixture

[0289] When computing an entry of a discrete texture table for a material that is a mixture of several components (e.g., several components of a candidate material that is a mixture of components according to a recipe determined by the formulation software), statistical texture synthesis can likewise be employed. In this case, source patches can be randomly sampled from source textures associated with different components with a probability according to their concentration in the mixture.

[0290] Generating texture attributes of a fusion zone

[0291] If a blend zone is displayed on a display device, the texture properties to be used in the blend zone can likewise be computed using a statistical texture synthesis process. The statistical texture synthesis process employed can interpolate between a texture in the bivariate texture table of a first instance of the appearance model (hereinafter referred to as "first texture") and a corresponding texture in the bivariate texture table of a second instance of the appearance model (hereinafter referred to as "second texture"). The first texture is associated with a target material to be displayed on one side of the blend zone and the second texture is associated with a candidate recipe to be displayed on the other side of the blend zone. The resulting synthesized texture can be used to generate a bivariate texture table for the blend appearance model associated with a particular location within the blend zone. Each synthesized texture is synthesized from the first and second textures in the same spirit as explained above for the synthesized target texture, with the only difference being that interpolation is performed between the first and second textures rather than among three source textures. The interpolation weights between the first and second textures can be chosen between 0 and 1 depending on the location within the blend zone.

[0292] Combination of different appearance capture devices

[0293] In the workflow discussed above, appearance properties of three types of objects are determined by measurements: the target object, the trial object, and reference objects that have been produced for various reference materials. These measurements will typically be performed at different locations. For example, in the case of vehicle repair, measurements on the target object are performed in a body shop, measurements on the trial object will typically be performed by a paint supplier, and measurements on the abridgement of the reference materials will typically be performed in a paint development laboratory.

[0294] Therefore, these measurements will typically not be performed by the same appearance capture device. If the same type of appearance capture device is used for measurements of all three types of objects, only inter-instrument calibration is needed. However, typically different types of appearance capture devices, possibly with different degrees of complexity, are used at different locations. For example, a body shop can use only a relatively simple hand-held multi-angle spectrophotometer (such as the MA-5 instrument by X-Rite) to measure the target object, a paint supplier can use a more complex hand-held multi-angle spectrophotometer with imaging capabilities (such as the MA-T6 or MA-T12 instruments by X-Rite) to measure the trial object, and a paint development laboratory can use a highly complex fixed appearance capture device (such as the TAC7 instrument by X-Rite) to determine the properties of the reference materials, the device having a large number of illumination and viewing directions and highly complex imaging capabilities.

[0295] Therefore, the available information content associated with the three types of objects can be different. This is illustrated in Figure 17 Fig. 3, Figure 17 schematically illustrates the variation of information content along three dimensions: the first dimension is the available information content for the target object, the second dimension is the available information content for the trial object, and the third dimension is the available information content for the reference materials. The available information content can vary along each of these dimensions between “L” (low, color properties are available only for a small number of combinations of illumination and viewing directions, no texture properties are available) and “H” (high, both color properties and texture properties are available, each for a large number of combinations of illumination and viewing directions).

[0296] Possible strategies for handling different levels of information content will now be discussed.

[0297] (a) the information content is high for the target object, the test object and the reference material Figure 17 block "H / H / H" in the table

[0298] Ideally, for each of the three dimensions, the information content is high, i.e. for the target material, for the test material and for each reference material, a large number of combinations of color attributes of illumination and observation directions and a considerable number of combinations of texture attributes in image form of illumination and observation directions are available. In this case, sufficient information is directly available to visualize the appearance of the target material based on the measured appearance attributes of the target object and to generate a first instance of the appearance model for this purpose. There is also sufficient information to determine suitable candidate formulations for which a good match to the appearance of the target material is expected, to visualize the appearance of the candidate formulations based on the available appearance attributes of the reference materials and to generate a second instance of the appearance model for this purpose. Finally, there is also sufficient information to visualize the test material and to generate another instance of the appearance model for this purpose and to assess whether the appearance of the test material actually matches the appearance of the target material by determining a suitable difference norm and / or by visually comparing the visualizations of the target material and the test material.

[0299] (b) For the target object, the information content is low, but for the test object and the reference material, the information content is high ( Figure 17 (in the block "L / H / H")

[0300] In some embodiments, the measurements of the test object and the reference objects are performed using appearance capturing devices having both spectrophotometric and imaging capabilities, i.e. the associated information content is high, while the measurements of the target object are performed using simpler devices having only spectrophotometric capabilities without imaging capabilities, i.e. the associated information content is low. The question then arises how to visualize the appearance of the material of the target object in a realistic way, although there are no measured texture attributes of this material.

[0301] In this case, at least two conceivable strategies are available. In a simple strategy, the entire virtual object is simply visualized without texture. For this purpose, both the first and the second instance of the appearance model can be generated with an "empty" texture table, i.e. with a texture table whose entries are all identical and do not exhibit any spatial variations. Equivalently, an appearance model without texture can be used to visualize the virtual object.

[0302] However, this procedure can not result in a rather realistic impression of the appearances of the materials of the target object and the candidate formulations and thus it can be difficult to judge whether these appearances actually match.

[0303] Thus, in a preferred strategy, the texture of the target object is predicted based on known texture attributes associated with the reference material. Once the test object has been produced, the texture prediction for the test object can additionally be based on texture attributes of the test object. In other words, the texture information associated with the reference material and / or the test object is used to visualize the appearance of the target material. To this end, the texture information belonging to another instance of the appearance model can be used to "edit" the first instance of the appearance model, i.e. the texture information associated with the target material is replaced or modified by the texture information associated with a different material. A method for "editing" appearance attributes in this way is described in US20150032430A1, the content of which is incorporated herein by reference in its entirety.

[0304] If a limited number of texture attributes are available for the target material, however, these texture attributes are insufficient for generating the texture part of the first instance of the appearance model in a meaningful way, or contain less texture information than the texture information available for the reference material and / or the test object, a similar strategy can also be used. For example, the first appearance capture device can be a multi-angle spectrophotometer configured to determine color attributes and only one or a few global texture attributes, e.g. a global roughness parameter or a global sparkle parameter. In contrast, the available texture attributes for the reference material and / or for the test object can include image data for several combinations of illumination and viewing directions. In this case, the texture attributes associated with the target object can be modified to include image data derived from the available image data of the reference material and / or the test object. The image data associated with the target object should of course be consistent with the available measured texture attributes of the target object, e.g. with the measured global parameters such as roughness or sparkle parameters. This can be ensured by appropriately modifying the pixel values in the available image data of the reference and / or test material, such that the global parameters computed from the modified image data correspond to the measured global parameters of the target object, respectively.

[0305] The method for modifying the pixel values can be very similar to the above described method in the context of statistical texture synthesis, where a texture patch is modified such that its histogram of pixel values approximately matches the average histogram of pixel values. Note that both the histogram of pixel values and the global roughness or sparkle parameter are examples of statistical properties associated with the image data. Similar to the above example of statistical texture synthesis, in the present example, the image is also modified such that one of its statistical properties (here: the global parameter computed from the image data) approximately matches a measured statistical property of the target object (here: the measured global parameter of the target object).

[0306] The limited texture properties measured for the target object can also differ from the global parameters. For example, the measured texture properties can comprise a single image for one single combination of illumination and viewing direction. Irrespective of the exact nature of the measured texture properties, additional images of the target material can be generated by modifying the pixel values of the available images associated with the target material, the reference material and / or the test object in such a way that at least one statistical property of the modified image data approximates the respective statistical property associated with the measured texture properties of the target object.

[0307] The process of the proposed texture prediction is based on the assumption that the texture of a target material will be fairly similar to the texture of a candidate formulation as long as the ingredients of the candidate formulation belong to the same class of materials as the ingredients of the target material. For example, if it is known that the target material comprises a certain type of flakes, and if the candidate formulation comprises flakes of this type, it is reasonable to assume that the texture of the target material will be at least close to the texture of the candidate formulation.

[0308] A similar strategy can also be employed if the available color properties of the target material are insufficient for reliably fitting the color properties required by the appearance model used. This can be the case, for example, if the multi-angle spectrophotometer used does not have a sufficient number of illumination and viewing direction pairs to determine the color properties of the target material. In this case, the color properties of the appearance model can also be determined by “editing” using a process as disclosed in US20150032430A1 in a similar way as the editing process of the texture properties described above.

[0309] (c) For the target object and the reference material, the information content is high, but for the test object, the information content is low of( Figure 17 (in the block "H / L / H")

[0310] In some embodiments, image-based texture information is available for the target material and the reference material, but no or only limited texture information (e.g. one or more global texture parameters or image information for only one pair of illumination and viewing direction) is available for the test object. In this case, a similar strategy as in case (b) can be employed. In particular, the texture of the test material can be predicted based on the known texture properties associated with the target material and / or with the reference material.

[0311] (d) For the target object, the information content is low, and for the test object, the information content is low, but for the Reference material, information content is high Figure 17 Block "L / L / H" in

[0312] In some embodiments, image-based texture information is available for the reference material only, while no or only limited texture information is available for the target object and the test object. In this case, again a similar strategy as in case (b) can be employed. In particular, the texture of the target material and the test material can be predicted based on the known texture properties associated with the reference material.

[0313] (e) For the target object and the test object, the information content is high, but for the reference material, the information content is low of "H / H / L" (not visible in Figure 17 the middle)

[0314] In some embodiments, image-based texture information is available for both the target object and the trial object, but no or only limited texture information is available for the reference material. This can be the case, for example, if a relatively old colorant database is used, which was populated at a time when only less sophisticated instrumentation was available, while more modern appearance capturing devices are available for measuring the target material and the trial material.

[0315] In such a case, the texture information of the target object and / or the trial object (once the trial object has been produced) can be used to visualize the appearance of the candidate formulation. To this end, the texture information pertaining to the target material and / or the trial material can be used to "edit" a second instance of the associated appearance model. This can be done in very much the same way as the first instance of the appearance model was edited in case (b) described above.

[0316] Once these texture properties are available, it is advantageous to use the texture properties of the trial object to visualize the candidate formulation, since it can be safely assumed that relatively small differences between the composition of the further candidate formulation and the trial material will only negligibly affect the texture.

[0317] (f) For the target object and the reference material, the information content is low, but for the test object, the information content is high of "L / H / L" (not visible in Figure 17 the middle)

[0318] If image-based texture information is available for the trial material, while no or only limited texture information is available for the target material and the reference material, the virtual object can be visualized without texture or using some generic texture, as long as the trial object is not yet available. Once the trial object is available, the texture properties of the trial object can then be used to visualize the target material and the candidate formulation. To this end, the texture properties in the associated instances of the appearance model can be edited as described above for case (b).

[0319] (g) For the target object, the information content is high, but for the test object and the reference material, the information content is low of( Figure 17 (in the block "H / L / L")

[0320] If image-based texture information is available for the target material, while no or only limited texture information is available for the trial material and the reference material, a similar strategy as for case (e) described above can be employed. The texture information of the target object can be used to visualize the appearance of the candidate formulation and the appearance of the trial object. To this end, the texture information pertaining to the target material can be used to "edit" the associated instances of the appearance model.

[0321] (h) the information content is low for the target object, the test object and the reference material Figure 17 block "L / L / L" in Fig. 1

[0322] If no image-based texture information is available for any of the objects, the virtual object can simply be visualized without a texture or using a generic texture of the class of materials the target object belongs to.

[0323] References

[0324] [Ber19] Roy S. Berns, “Billmeyer and Saltzman’s Principles of Color Technology”, 4th edition, Wiley, 2019, p. 184.

[0325] [Kw03] Kwatra, V., Essa, I., Turk, G., Bobick, A., “Graphcut textures: image and video synthesis using graph cuts”, ACM Transactions on Graphics (ToG), 2003, 22(3), pp. 277-286.

[0326] [MTG+19] Gero Muller, Jochen Tautges, Alexander Gress, Martin Rump, Max Hermann, Francis Lamy, “AxF - Appearance exchange Format”, version 1.7, April 11, 2019, which is available on request from X-Rite GmbH.

[0327] [RMS+08] Martin Rump, Gero Muller, Ralf Sarlette, Dirk Koch, Richard Klein, “Photo-realistic Rendering of Metallic Car Paint from Image-Based Measurements”, in: R. Scopigno, E. Gobbetti, K. Yagi, G. Wimmer (Eds.), Computer Graphics Forum, Eurographics, April 2008, 27:2 (527-536).

[0328] ​​[RSK09] Martin Rump, Ralf Sarlette, and Richard Klein, "Efficient Resampling, Compression and Rendering of Metallic and Pearlescent Paint", in: M. Magnus, B. Rosenhahn, H. Theisel (Eds.), Proceedings of Vision, Modeling, and Visualization, November 2009, pp. 11-18.

[0329] [Rus98] Szymon M. Rusinkiewicz, "A New Change of Variables for Efficient BRDF Representation", in: G. Drettakis G., N. Max (Eds.), Rendering Techniques '98, Eurographics, Springer, Vienna, 1998, https: / / doi.org / 10.1007 / 978-3-7091-6453-2_ 2 .

[0330] [Sh68] Shepard D., "A two-dimensional interpolation function for irregularly- spaced data", in: Proceedings of the 1968 23rd ACM national conference, 1968, pp. 517-524.

[0331] [W98] Williams, C. K., "Prediction with Gaussian processes: From linear regression to linear prediction and beyond", in: Learning in graphical model, Springer, Dordrecht, 1998, pp. 599-621.

[0332] List of references signs

[0333] 1, 2 holding member

[0334] 3 wristband

[0335] 4 display array

[0336] 5 housing base

[0337] 6 measurement opening

[0338] 7 substrate

[0339] 7a, 7b, 7c support member

[0340] 10 arcuate body

[0341] 21, 22, 23, 24, 25, 26, 27, 28 illumination device

[0342] 31, 32, 33 pick-up device

[0343] 31a, 32a spectrometer

[0344] 33a RGB camera

[0345] 31c, 32c optical fiber

[0346] 50 target object

[0347] 52 appearance capture device

[0348] 54, 64 appearance attribute

[0349] 56, 66 instance of appearance model

[0350] 60 candidate formulation

[0351] 70 display device

[0352] 72 virtual object

[0353] 72a, 72b, 72c portion

[0354] 74, 76 virtual dividing line

[0355] 78 fusion zone

[0356] 80 test object

[0357] 82 appearance capture device

[0358] 90 pointing device

[0359] 102 model generation software

[0360] 104 formulation software

[0361] 106 database (colorants and formulations)

[0362] 108 rendering software

[0363] 110 database (geometric data)

[0364] 201 target texture

[0365] 211, 221, 231 source texture

[0366] 203, 206, 212, 214 texture patch

[0367] 202, 213, 215, 222, 232 pixel value histogram

[0368] 204 position

[0369] 205 seam

[0370] 300 client computer

[0371] 310 processor

[0372] 320 non-volatile memory

[0373] 321 operating system software

[0374] 330 RAM

[0375] 340 I / O interface

[0376] 350 communication interface

[0377] 360 server computer

[0378] 501-511 steps

[0379] HMD handheld measuring device

[0380] H housing

[0381] DN device normal

[0382] SP system plane

[0383] A, B, C sample points (color)

[0384] a, b target positions (color table)

[0385] A', B', C', D', E', F' sample points (texture)

[0386] a', b', c' target positions (texture table)

Claims

1. A computer-implemented method for visualizing the appearance of at least two materials, the method comprising: obtaining a first instance of an appearance model (56), the first instance (56) being indicative of the appearance of a first material, wherein obtaining the first instance of the appearance model (56) comprises generating the first instance (56) using a set of appearance attributes (54) of the first material; obtaining a second instance of the appearance model (66), the second instance (66) being indicative of the appearance of a second material, wherein obtaining the second instance of the appearance model (66) comprises generating the second instance (66) using a set of appearance attributes (64) of the second material or using predetermined appearance attributes of one or more reference materials, the predetermined appearance attributes being stored in a database (106); and obtaining a geometric model of a virtual object (72), the geometric model defining a three-dimensional macroscopic surface geometry of the virtual object (72); and visualizing a scene comprising the virtual object (72) using a display device (70), using the first and second instances of the appearance model (56, 66) and the geometric model and a virtual separation element (74) and / or a blending zone (78), wherein the virtual object (72) has first and second portions (72a, 72b), wherein the first portion (72a) of the virtual object (72) is visualized using the first instance of the appearance model (56), and wherein the second portion (72b) of the virtual object (72) is visualized using the second instance of the appearance model (66), and wherein the first and second portions (72a, 72b) are visually separated by the virtual separation element (74) and / or by the blending zone (78), wherein the appearance model comprises a discrete texture table, the discrete texture table comprising a plurality of target textures (201), each target texture (201) being associated with a different combination of illumination and viewing directions.

2. The computer-implemented method according to claim 1, wherein, the scene is visualized with the virtual separation element (74), and wherein the first and second portions (72a, 72b) are adjacent to each other and visually separated by the virtual separation element (74) such that the first and second portions (72a, 72b) appear to meet at the location where the virtual separation element (74) is located.

3. The computer-implemented method of claim 2, wherein, The display device (70) defines a two-dimensional viewing plane, and wherein the virtual separation element (74) is a line in the viewing plane.

4. The computer-implemented method according to any one of the preceding claims, wherein the scene is visualized with the blending zone (78), and wherein visualizing the blending zone (78) comprises interpolating between the appearance attributes of the first and second instances of the appearance model (56, 66).

5. The computer-implemented method according to claim 4, wherein interpolating between the appearance attributes of the first and second instances of the appearance model (56, 66) comprises computing at least one interpolated texture in the blending zone (78) by interpolating between at least one texture of the first instance of the appearance model (56) and at least one texture of the second instance of the appearance model (66), wherein computing each interpolated texture comprises: (i) assigning interpolation weights to at least one texture of the first instance (56) of the appearance model and to at least one texture of the second instance (66) of the appearance model; and (ii) synthesizing the interpolated texture using the at least one texture of the first instance (56) of the appearance model and the at least one texture of the second instance (66) of the appearance model and the assigned interpolation weights, synthesizing the interpolated texture comprising: (a) randomly selecting one of the textures of the first and second instances (56, 66) of the appearance model with a probability proportional to the interpolation weight of said texture; (b) randomly extracting a texture patch from the selected texture; (c) modifying the extracted texture patch by modifying pixel values in the extracted texture patch to obtain a modified texture patch, the modification of pixel values being performed in such a way that at least one statistical property of the modified texture patch approximates a corresponding average statistical property, the average statistical property being determined by performing a weighted average over the textures of the first and second instances (56, 66) of the appearance model, the textures of the first and second instances (56, 66) of the appearance model being weighted by the interpolation weights; (d) inserting the modified texture patch into the interpolated texture in such a way that the modified texture patch seamlessly fits into existing texture content in the interpolated texture; and (e) repeating steps (a) - (d) until the interpolated texture is completely filled.

6. The computer-implemented method according to any one of claims 1 to 3, comprising moving the virtual dividing element (74) or the fusion zone (78) relative to the virtual object (72) in response to a user input, and redefining the first and second portions (72a, 72b) in such a way that the first and second portions (72a, 72b) remain visually separated by the virtual dividing element (74) or the fusion zone (78) even after the virtual dividing element (74) or the fusion zone (78) has been moved.

7. The computer-implemented method according to any one of claims 1 to 3, wherein the scene is visualized with defined illumination conditions, and wherein the method comprises changing the illumination conditions in response to a user input, and visualizing the scene with the changed illumination conditions.

8. The computer-implemented method according to any one of claims 1 to 3, wherein, the scene is visualized with a virtual object (72) in a defined pose, and wherein the method comprises changing the pose of the virtual object (72) in response to a user input, and visualizing the scene with the virtual object (72) in the changed orientation.

9. The computer-implemented method according to any one of claims 1 to 3, comprising: determining the set of appearance properties (54) of the first material by performing measurements on a target object (50) comprising the first material using a first appearance capturing device (52).

10. A computer-implemented method for visualizing the appearance of at least two materials, the method comprising: obtaining a first instance of an appearance model (56), the first instance (56) being indicative of an appearance of a first material, wherein obtaining the first instance of the appearance model (56) comprises generating the first instance (56) using a set of appearance properties (54) of the first material; obtaining a second instance of the appearance model (66), the second instance (66) being indicative of an appearance of a second material, wherein obtaining the second instance of the appearance model (66) comprises generating the second instance (66) using a set of appearance properties (64) of the second material or using predetermined appearance properties of one or more reference materials, the predetermined appearance properties being stored in a database (106); and obtaining a geometry model of the virtual object (72), the geometry model defining a three-dimensional macroscopic surface geometry of the virtual object (72); and visualizing a scene comprising the virtual object (72) using a display device (70), using the first and second instances of the appearance model (56, 66) and the geometry model and a virtual dividing element (74) and / or a blending zone (78), wherein the virtual object (72) has first and second portions (72a, 72b), wherein the first portion (72a) of the virtual object (72) is visualized using the first instance of the appearance model (56), and wherein the second portion (72b) of the virtual object (72) is visualized using the second instance of the appearance model (66), and wherein the first and second portions (72a, 72b) are visually separated by the virtual dividing element (74) and / or by the blending zone (78), wherein obtaining the second instance of the appearance model (66) comprises: receiving appearance properties (54) of the first material; determining a recipe (60) for the second material based on the received appearance properties (54) of the first material and the predetermined appearance properties of the reference material; and generating the second instance of the appearance model (66) based on the recipe (60) and the predetermined appearance properties of the reference material.

11. The computer-implemented method according to claim 10, further comprising: preparing a physical test object (80) using a material prepared according to the recipe (60); determining a set of measured appearance properties of the test object (80) by performing measurements on the test object (80); calculating appearance properties of the second material based on the recipe (60) and the predetermined appearance properties of the reference material; and using the measured appearance properties of the test object (80) and the calculated appearance properties of the second material to determine a modified recipe.

12. A computer-implemented method for visualizing appearances of at least two materials, the method comprising: obtaining a first instance of an appearance model (56), the first instance (56) being indicative of an appearance of a first material, wherein obtaining the first instance of the appearance model (56) comprises generating the first instance (56) using a set of appearance properties (54) of the first material; obtaining a second instance of the appearance model (66), the second instance (66) being indicative of an appearance of a second material, wherein obtaining the second instance of the appearance model (66) comprises generating the second instance (66) using a set of appearance properties (64) of the second material or using predetermined appearance properties of one or more reference materials, the predetermined appearance properties being stored in a database (106); and obtaining a geometry model of the virtual object (72), the geometry model defining a three-dimensional macroscopic surface geometry of the virtual object (72); and visualizing a scene comprising the virtual object (72) using a display device (70), using the first and second instances of the appearance model (56, 66) and the geometry model and a virtual dividing element (74) and / or a blending zone (78), wherein the virtual object (72) has first and second portions (72a, 72b), wherein the first portion (72a) of the virtual object (72) is visualized using the first instance of the appearance model (56), and wherein the second portion (72b) of the virtual object (72) is visualized using the second instance of the appearance model (66), and wherein the first and second portions (72a, 72b) are visually separated by the virtual dividing element (74) and / or by the blending zone (78), wherein obtaining the second instance of the appearance model (66) comprises: receiving appearance properties (54) of the first material; determining a recipe (60) for the second material based on the received appearance properties (54) of the first material and the predetermined appearance properties of the reference material; and generating the second instance of the appearance model (66) based on the recipe (60) and the predetermined appearance properties of the reference material.

11. The computer-implemented method according to claim 10, further comprising: preparing a physical test object (80) using a material prepared according to the recipe (60); determining a set of measured appearance properties of the test object (80) by performing measurements on the test object (80); calculating appearance properties of the second material based on the recipe (60) and the predetermined appearance properties of the reference material; and using the measured appearance properties of the test object (80) and the calculated appearance properties of the second material to determine a modified recipe.

12. A computer-implemented method for visualizing appearances of at least two materials, the method comprising: obtaining a first instance of an appearance model (56), the first instance (56) being indicative of an appearance of a first material, wherein obtaining the first instance of the appearance model (56) comprises generating the first instance (56) using a set of appearance properties (54) of the first material; obtaining a second instance of the appearance model (66), the second instance (66) being indicative of the appearance of the second material, wherein obtaining the second instance of the appearance model (66) comprises generating the second instance (66) using the set of appearance properties (64) of the second material or using predetermined appearance properties of one or more reference materials, the predetermined appearance properties being stored in a database (106); and obtaining a geometry model of the virtual object (72), the geometry model defining a three-dimensional macroscopic surface geometry of the virtual object (72); and visualizing a scene comprising the virtual object (72) using the display device (70), using the first and second instances (56, 66) of the appearance model and the geometry model and the virtual separation element (74) and / or the blending zone (78), wherein the virtual object (72) has a first and a second portion (72a, 72b), wherein the first portion (72a) of the virtual object (72) is visualized using the first instance (56) of the appearance model, and wherein the second portion (72b) of the virtual object (72) is visualized using the second instance (66) of the appearance model, and wherein the first and second portions (72a, 72b) are visually separated by the virtual separation element (74) and / or by the blending zone (78), wherein the appearance properties (54) of the first material, the appearance properties (64) of the second material and / or the predetermined appearance properties of the reference materials are associated with a plurality of combinations of illumination and viewing directions, and wherein generating the first and / or second instance (56, 66) of the appearance model comprises at least one of the following operations: interpolating between available appearance properties under different combinations of illumination and viewing directions; and extrapolating from available appearance properties under a selected combination of illumination and viewing directions.

13. The computer-implemented method of claim 12, wherein the available appearance properties comprise a plurality of source textures (211, 221, 231), each source texture being associated with a different set of source coordinates A' to F', each set of source coordinates A' to F' being indicative of a combination of illumination and viewing directions for which the respective source texture (211, 221, 231) is indicative of the spatial variation of the appearance, wherein the appearance model comprises a discrete texture table, the discrete texture table comprising a plurality of target textures (201), each target texture (201) being associated with a different set of target coordinates a', the set of target coordinates a' being indicative of a particular combination of illumination and viewing directions for which the target texture (201) is indicative of the spatial variation of the appearance, and wherein generating the first and / or second instance (56, 66) of the appearance model comprises determining at least one of the target textures (201) by performing a statistical texture synthesis operation based on: (i) the set of target coordinates associated with the target texture (201); (ii) the sets of source coordinates A' to F'; and (iii) the source textures (211, 221, 231).

14. The computer-implemented method of claim 13, wherein, the statistical texture synthesis operation comprises: (i) assigning an interpolation weight to each of the source textures (211, 221, 231) based on the target coordinate set a' and the source coordinate sets A' - F'; and (ii) synthesizing the target texture (201) using the source textures (211, 221, 231) and the assigned interpolation weights.

15. The computer-implemented method of claim 14, wherein, Synthesizing the target texture (201) comprises: (a) randomly selecting one of the source textures (211, 221, 231) with a probability proportional to the interpolation weight of the source texture (211, 221, 231); (b) randomly extracting a texture patch (212) from the selected source texture (211); (c) modifying the extracted texture patch (212) by modifying pixel values in the extracted texture patch to obtain a modified texture patch (214), the modification of the pixel values being performed in such a way that at least one statistical property (215) of the modified texture patch (214) approximates a corresponding average statistical property (202), the average statistical property (202) being determined by performing a weighted average over the source textures (211, 221, 231), the source textures (211, 221, 231) being weighted by the interpolation weights; (d) inserting the modified texture patch (214) into the target texture in such a way that the modified texture patch (214) seamlessly fits into existing texture content (203) in the target texture; and (e) repeating steps (a) - (d) until the target texture is completely filled.

16. The computer-implemented method of any one of claims 1 to 3, wherein, The appearance model comprises a model for describing the angular dependence of the brightness and the color.

17. The computer-implemented method of any one of claims 1 to 13, wherein, The appearance model comprises a monochromatic brightness BRDF model for describing the angular dependence of the overall reflectance properties averaged over surface locations and wavelengths.

18. The computer-implemented method of claim 17, wherein, The appearance model comprises a discrete color table having a plurality of entries in the form of color values, each entry being associated with a specific coordinate set, each coordinate set indicating an illumination and a viewing direction in which the material is observed.

19. A computer-implemented method for visualizing the appearance of at least two materials, the method comprising: obtaining a first instance (56) of an appearance model, the first instance (56) being indicative of the appearance of a first material, wherein obtaining the first instance (56) of the appearance model comprises generating the first instance (56) using a set of appearance properties (54) of the first material; obtaining a second instance (66) of the appearance model, the second instance (66) being indicative of the appearance of a second material, wherein obtaining the second instance (66) of the appearance model comprises generating the second instance (66) using a set of appearance properties (64) of the second material or using predetermined appearance properties of one or more reference materials, the predetermined appearance properties being stored in a database (106); obtaining a geometric model of a virtual object (72), the geometric model defining a three-dimensional macroscopic surface geometry of the virtual object (72); and visualizing a scene comprising the virtual object (72) using a display device (70), using the first and second instances (56, 66) of the appearance model and the geometric model and a blending region (78), wherein the virtual object (72) has first and second portions (72a, 72b), wherein a first instance (56) of the appearance model is used to visualize a first portion (72a) of the virtual object (72), and wherein a second instance (66) of the appearance model is used to visualize a second portion (72b) of the virtual object (72), and wherein the first and second portions (72a, 72b) are visually separated by a blending zone (78), wherein visualizing the blending zone (78) comprises interpolating between appearance properties of the first and second instances (56, 66) of the appearance model, wherein interpolating between appearance properties of the first and second instances (56, 66) of the appearance model comprises computing at least one interpolated texture in the blending zone (78) by interpolating between at least one texture of the first instance (56) of the appearance model and at least one texture of the second instance (66) of the appearance model, wherein computing each interpolated texture comprises: (i) assigning an interpolation weight to at least one texture of the first instance (56) of the appearance model and to at least one texture of the second instance (66) of the appearance model; and (ii) synthesizing the interpolated texture using the at least one texture of the first instance (56) of the appearance model and the at least one texture of the second instance (66) of the appearance model and the assigned interpolation weight, synthesizing the interpolated texture comprising: (a) randomly selecting one of the textures of the first and second instances (56, 66) of the appearance model with a probability proportional to the interpolation weight of said texture; (b) randomly extracting a texture patch from the selected texture; (c) modifying the extracted texture patch by modifying pixel values in the extracted texture patch to obtain a modified texture patch, the modification of pixel values being performed in such a way that at least one statistical property of the modified texture patch approximates a corresponding average statistical property, the average statistical property being determined by performing a weighted average over the textures of the first and second instances (56, 66) of the appearance model, said textures of the first and second instances (56, 66) of the appearance model being weighted by the interpolation weight; (d) inserting the modified texture patch into the interpolated texture in such a way that the modified texture patch seamlessly fits into the existing texture content in the interpolated texture; and (e) repeating steps (a)-(d) until the interpolated texture is completely filled.

20. The computer-implemented method of claim 19, comprising moving the blending zone (78) relative to the virtual object (72) in response to a user input, and redefining the first and second portions (72a, 72b) in such a way that the first and second portions (72a, 72b) remain visually separated by the blending zone (78) even after the blending zone (78) has been moved.

21. An apparatus for visualizing the appearance of at least two materials, comprising a display apparatus, a processor and a memory, the memory comprising program instructions configured to cause the processor to perform the method of any of the preceding claims.

22. A computer program product comprising a computer readable medium having stored thereon program instructions that, when executed by a processor, cause the processor to carry out the method according to any one of claims 1 to 20.

Images