Displaying virtual objects in real scenes

Abstract

In a method for displaying a virtual object (140), a scene (100) is imaged. The scene includes a target device (120). The target device includes a set of color target elements (121) having known reflectance properties and at least one illumination target element (122) having a plurality of known surface normals. At least one image of the scene is processed to determine measured colors of the color target elements and illumination and viewing conditions of the scene. Color transform parameters are computed by comparing the measured colors of the color target elements to the known reflectance properties. The virtual object is rendered for the determined illumination and viewing conditions. The rendered virtual object is aligned with the scene. A color transform is applied to the imaged scene and / or to the virtual object, and a superposition of the rendered virtual object and the imaged scene is displayed in real time on a display device (220) as illumination and / or viewing conditions change.

Description

Technical Field

[0001] The present invention relates to a method for displaying virtual objects in an imaged real-life scene, a system configured to perform the method, and a corresponding computer program. Background Technology

[0002] The common expectation is to find a material whose visual appearance precisely matches that of an existing sample object. For example, when repairing a vehicle in a body shop, one might expect to find a paint coating for replacing the body part whose visual appearance precisely matches the original paint coating on the rest of the vehicle. Even if the paint code or vehicle identification number of the original paint coating is known, the corresponding paint may not be available. Sometimes, a reference formula for the original paint is available. However, simply mixing paint according to the reference formula may not produce an acceptable match. This is because the appearance of the original paint coating will vary slightly depending on the batch, formulator, or year. The challenge then becomes defining a modified paint formula that ensures the appearance of the paint coating on the replacement body part perfectly matches the appearance of the original paint coating on the rest of the vehicle.

[0003] A fan deck is known to be used to find materials that match the appearance of a sample object. A fan deck is a collection of sample sheets with known colors. The sample sheets in the fan deck are compared with the sample object, and suitable candidate materials are selected based on this comparison. However, fan decks are not always available in the field. Even if fan decks are available, it is possible that none of the colors in the fan deck will achieve an acceptable match.

[0004] Because sample objects may possess gonioapparent properties—meaning their appearance can vary depending on the direction of illumination and observation—defining the composition of the test object's material makes the task of precisely matching its appearance to that of the physically tangible sample object's material more complex. For example, some modern car paints exhibit color flops when the direction of illumination or observation changes. A color flop is a change in the color value, hue, or chromaticity of a material's appearance when the direction of illumination and observation changes. Other examples of materials with gonioapparent properties include materials containing effect pigments, such as metallic flakes that produce a shimmering effect or interference flakes that produce a pearlescent effect, as well as materials with non-planar surface microstructures. In such cases, simply matching the color may not be sufficient. Instead, it is necessary to match the entire appearance, including angle-dependent color and texture.

[0005] It is known to describe the appearance of an object based on one or more of several appearance attributes, such as color, gloss, gloss level, roughness, etc. It is also known to use dedicated appearance capture devices to measure the appearance attributes of sample objects. Different types of appearance capture devices are known in the art. The type of device best suited for this task depends largely on the environment, and particularly on the type of sample object. For example, different types of capture devices may be best suited for determining the appearance attributes of a vehicle body section rather than a piece of fabric or a translucent sample object. Known appearance capture devices include highly complex fixed multi-angle appearance capture devices (such as the TAC7 device available from X-Rite, Grand Rapids, Michigan, USA), integrating sphere spectrophotometers (such as the fixed Ci7800 device or the handheld Ci64 device available from X-Rite), and handheld multi-angle spectrophotometers with or without imaging capabilities (such as the imaging multi-angle spectrophotometer MA-T12 available from X-Rite). Once the appearance attributes of the sample object have been measured, paint formulation software can be used to determine the formulation of candidate materials whose appearance attributes are expected to best match the measured appearance attributes of the sample object. For example, in the case of replacing a body part of a vehicle, a formulation of a candidate paint can be determined that is expected to closely match the measured appearance properties of the existing paint coating on the vehicle body, and the replacement body part can be coated with such paint.

[0006] However, dedicated appearance capture devices are not always available in the field. Furthermore, even if the appearance attributes of a sample object have been measured by a dedicated appearance capture device, the number of measured appearance attributes is typically limited, and this limited number of appearance attributes may not always fully define the actual appearance of the sample object. Therefore, it is not always possible to reliably predict how well the actual appearance of the test object will match the appearance of the sample object. Consequently, there is a risk of producing test objects whose appearance does not match the appearance of the sample object well enough, and subsequently having to discard these test objects, resulting in undesirable waste and associated costs.

[0007] In other applications, it may not be necessary to precisely match the appearance of the test subject to that of an existing object; rather, it may be sufficient to simply explore how the test subject would look in a real-world environment. For example, the test subject could be a new product that has not yet been manufactured, such as a toaster, and it might be desirable to explore how the product would look in a specific environment (e.g., on a specific table) before actual production.

[0008] US20200089991A1 discloses a system for displaying one or more images to select one or more matching formulas to match the appearance of a target coating on an article. A first database contains repair formulas and associated appearance characteristics. A second database contains identification information or three-dimensional geometric data of at least one article. A preliminary matching formula is retrieved from the first database, an article or its three-dimensional geometric data is selected from the second database, and marked portions of the article's surface are received. An individual matching image is generated, including the marked portions and unmarked portions adjacent to the marked portions, and displayed on a display device. In the marked portions, the individual matching image is generated based on the appearance characteristics of the preliminary matching formula. In the unmarked portions, the individual matching image is generated based on the appearance characteristics of the article. Appearance characteristics can be calculated from images acquired from the article using an imaging device. However, suitable imaging devices are not always available.

[0009] US10,375,148B2 discloses a method for selecting the most likely variant of a matching candidate paint color standard for vehicle repair using a mobile device with a color display and an input unit. In the first step, an identification standard associated with the vehicle's color is input into the mobile device. This may involve acquiring a digital image of the vehicle to extract information such as the car brand or color group. The identification standard is sent to a central computer that includes a color database. Software on the central computer selects one or more candidate color standards that match the input identification standard and sends information about these candidate color standards to the mobile device. The information is displayed on the mobile device. One or more physical samples painted with the candidate color standards are visually compared to the color of the vehicle to be repaired. A matching color standard is selected. This selection is input into the mobile device and transmitted to the central computer. The software identifies variants of the selected candidate color standard and transmits a representation of the variant to the mobile device and displays it. The best-matching variant is selected and transmitted to the central computer. The software returns the corresponding paint formulation. Displaying information about the candidate color standards may involve displaying images of the candidate color standards on the color display of the mobile device. The user can then visually compare the images with the color of the vehicle to be repaired. For such comparisons to be meaningful, the display needs to have a large color gamut, a wide dynamic range, and very high color fidelity. Such displays are expensive and are often unavailable in the field. Furthermore, lighting and viewing conditions need to be known with high precision, as the perceived color of the sample object will depend on both.

[0010] WO2010 / 125023A1 discloses a method for displaying an image of an effect coating having texture and color properties. This method can be used to select candidate coatings with properties that match those of a target coating. To this end, the document suggests displaying an image of the target coating and images of at least one candidate coating. However, this method is affected by unavoidable color distortion introduced by the camera used to capture the image of the target coating.

[0011] US10,049,294B2 (a member of the same family as EP3051793A1) discloses a method for determining color transformation parameters. A target device is placed in the scene to be imaged. The target device includes, on one hand, a set of color target elements with known colors, and on the other hand, at least one illumination target element with multiple different known surface normals. An image of the scene is captured. A local environment map of the global illumination of the scene is constructed using the imaged illumination target elements, and the colors of the color target elements are captured. The expected colors of the color target elements in the image are determined based on the known colors of the color target elements and based on the local environment map of global illumination. Color transformation parameters are calculated based on the difference between the expected colors and the captured colors. The color transformation can then be applied to the imaged sample surface. This process can be repeated across multiple different illumination or observation conditions to estimate the reflectance model of the sample surface. The document does not mention the visual evaluation of the difference between the two objects.

[0012] HPALensch et al.'s "Image-Based Reconstruction of Spatial Appearance and Geometric Detail" (ACM Transactions on Graphics, Vol. 22, No. 2, April 2003, pp. 234-237) discloses an image-based method for reconstructing the appearance of sample objects. Different materials of the object are detected, and the average bidirectional reflectance distribution function (BRDF) is fitted to each material. Multiple reflective steel spheres are placed in the scene to determine the position of the light source. This paper only concerns itself with reconstructing the appearance of sample objects. It does not mention the prediction or evaluation of visual differences between two objects.

[0013] US6,768,814B1 discloses a method for determining a color scheme to match a selected color. An electronic imaging device is calibrated by measuring color signals of at least two calibration colors having known chromaticity data. The selected color is measured using the imaging device. A mathematical model is used to calculate parameters for converting the measured color signals of the calibration colors into their known chromaticity data. The measured color signal of the selected color is converted into chromaticity data using the mathematical model and the calculated parameters. A database is used to determine a color scheme whose chromaticity data most closely matches the calculated chromaticity data of the measured selected color. This document does not mention predicting or evaluating visual differences between two objects.

[0014] US2005 / 0128484A1 discloses a method for determining a repair paint formulation for color matching. The color characteristics of the target color to be matched are identified, input, and processed in such a way that the target color can be visually displayed. Alternative colors are selected from a database. The alternative colors can be displayed on a monitor as several virtual patches, each representing a different viewing angle, or displayed as a curved panel. An image showing the appearance characteristics of the patches can be superimposed on the color. Interpolation can be performed on colors and images obtained for multiple non-mirror angles to show changes in the appearance of the patches as the non-mirror angle changes. These patches can be observed in conjunction with the target color. The target color can be projected onto the entire image of the vehicle, and an available paint formulation can be superimposed on it. A more realistic impression of the target and alternative colors is desired compared to the impression obtained by uniformly projecting the target color onto the entire image of the vehicle and superimposing the alternative colors.

[0015] G. Klein et al.'s paper, "Simulating Low-Cost Cameras for Augmented Reality Compositing," IEEE Transactions on Visualization and Computer Graphics, Vol. 16, No. 3, pp. 369-380 (2010), discloses an augmented reality method in which graphics are overlaid onto a live video feed. A compositing method is described that models artifacts produced by small, low-cost cameras. Summary of the Invention

[0016] In one aspect, the present invention provides a method that enables a user to better judge the appearance of a test subject in a real-world setting before the test subject is actually produced.

[0017] The proposed method is a way to display virtual objects in an imaged real-world scene. This method includes:

[0018] a) Imaging a scene using an imaging device, the scene including a target device, the target device including a set of color target elements having known reflective properties and a set of one or more illumination target elements having multiple different known surface normals relative to the color target elements;

[0019] b) Process at least one image of the scene using an imaging target device to determine the illumination and viewing conditions of the scene;

[0020] c) Process at least one image of the scene to determine the measured color of a color target element;

[0021] d) Taking into account the determined illumination and observation conditions, calculate the color transformation parameters based on a comparison between the measured color of the color target element and its known reflectance properties;

[0022] e) Render the virtual object according to the determined illumination and observation conditions;

[0023] f) Align the rendered virtual object with the imaged scene;

[0024] g) Applying a color transformation to the imaged scene using calculated color transformation parameters or applying an inverse color transformation to the virtual object; and

[0025] h) After applying color transformation or inverse color transformation, display on a display device at least a portion of the rendered virtual object superimposed on at least a portion of the imaged scene.

[0026] In the proposed method, an imaging device such as a digital camera is used to image a real scene. Virtual objects are rendered using a computer algorithm, assuming illumination conditions identical or similar to those present in the scene, and observation conditions identical or similar to those of the imaging device. The rendered virtual objects are displayed on a display device as an overlay of the imaged scene. To ensure that the appearance of the rendered virtual objects in the imaged scene can be judged with high confidence, a color transformation is applied. The color transformation takes into account the effects of limitations of the imaging device in capturing the colors of objects. Specifically, the color transformation corrects for unknown spectral sensitivity of the imaging device and unknown nonlinearities introduced by image processing within the imaging device. The color transformation can transform color values ​​already determined by the imaging device (e.g., RGB values ​​per pixel) into device-independent corrected color values, and thus can be directly compared with the device-independent color attributes of the virtual objects. The color transformation can be applied to all or part of the imaged scene. Alternatively, an inverse color transformation can be applied to all or part of the virtual objects before, during, or after the rendering operation. The color transformation ensures that the display colors of the imaged scene and the display colors of the rendered virtual objects become directly comparable on the display device. Therefore, even if the displayed absolute color deviates from the actual color due to defects in the display device, it is possible to reliably determine the appearance of virtual objects in the imaged scene. Thus, due to color transformation, neither the imaging device nor the display device requires pre-calibration.

[0027] According to this disclosure, parameters of color transformation and illumination and observation conditions are determined by means of a target device. Suitable exemplary embodiments of a target device and a method for determining illumination and observation conditions and color transformation parameters using such a target device are disclosed in patent document US10,049,294B2, the contents of which are incorporated herein by reference in their entirety. The target device comprises, on the one hand, a set of color target elements having known reflective properties (e.g., known parameters of a model of the BRDF for each color target element). Therefore, the expected color of each color target element can be calculated for any illumination and observation conditions. The color target elements are preferably matte, i.e., they primarily exhibit diffuse reflection. On the other hand, the target device includes a set of one or more illumination target elements having multiple known surface normals relative to the color target elements. Each illumination target element is preferably glossy or semi-glossy, i.e., it exhibits considerable specular reflection. The illumination target elements preferably have a neutral color; in particular, it can be black.

[0028] This group of one or more target elements facilitates the determination of illumination conditions. For example, illumination conditions can be determined using an imaged group of one or more target elements in the form of at least a partial environmental map of global illumination, as disclosed in more detail in US10,049,294B2. As detailed in that document, the group of one or more target elements may include at least one target element having a non-planar surface, which may be characterized by a known surface normal map including multiple different surface normals. In some embodiments, the at least one target element having a non-planar surface may be a sphere or a portion of a sphere, particularly a hemisphere. In some embodiments, the group of one or more target elements may include multiple target elements, for example, each target element being characterized by a different surface normal or a different surface normal map, wherein each of the multiple target elements includes a flat surface characterized by a different surface normal. In some embodiments, the group of target elements may include two or more target elements with different gloss levels.

[0029] By comparing the geometry of an image of the target device with the known geometry of the target device, observation conditions (in particular, the location of the imaging device and the nature of the scene-to-image mapping, including scale and any distortion) can be determined. To facilitate the determination of observation conditions, the target device may include one or more alignment marks (as are well known in the art) and / or distortion target elements that facilitate the determination of distortion based on the image (e.g., distortion target elements with a checkerboard pattern).

[0030] Color transformation parameters can be determined using the measured color of the color target element. To this end, based on the known reflectivity of the color target element, and considering the significant correlation between reflectivity and illumination and viewing angles, the expected color of the color target element in the image can first be calculated, taking into account the determined illumination conditions (e.g., a local environment map of global illumination) and viewing conditions. Next, the color transformation parameters can be determined based on the difference between the calculated expected color and the measured color of the color target element.

[0031] It should be noted that this disclosure is not limited to embodiments using the color target element disclosed in US10,049,294B2. For example, the target device may additionally include a filter as disclosed in US9,823,131B2 and / or a reflection reference map as disclosed in US2013 / 0093883A1.

[0032] Based on the location of the target device in the scene image, it is easy to align the rendered virtual object with the imaged scene, so that the virtual object appears to maintain its position within the scene even when the viewing orientation changes, as is well known in the art. This task is particularly simple if the rendered virtual object is displayed in the imaged scene at the location of the target device.

[0033] In an advantageous embodiment, the scene includes at least one sample object, for example, for the purpose of directly comparing the visual appearance of a rendered virtual object and the visual appearance of an imaged sample object. The sample object may have non-planar surface portions. In such embodiments, it is advantageous if the non-planar surface portion of the virtual object is displayed adjacent to the non-planar surface portion of the sample object. By simultaneously displaying the non-planar surface portions of both the sample object and the virtual object, a more realistic impression of the appearance of both the sample object and the rendered virtual object is obtained. This is especially true if one or both of the surfaces exhibit effects dependent on the direction of illumination and / or the viewing direction, such as color variations, pearlescent sheen, glitter, etc. To facilitate direct comparison of the appearance of the non-planar surface portions, the displayed non-planar surface portion of the virtual object may have a continuous shape corresponding to the displayed non-planar surface portion of the sample object. In other words, the sample object and the virtual object are advantageously displayed in such a way that they appear to merge seamlessly, without any edges or kinks at the line where the objects meet on the display. This generates a highly realistic impression of the appearance of both objects, and differences between their appearances can be easily detected.

[0034] In some embodiments, the non-planar surface portion of the displayed virtual object may have a shape consistent with the shape of the corresponding surface portion of the sample object, which obviously replaces the surface portion of the sample object, thereby producing an impression of an even more realistic appearance.

[0035] The non-planar surface portion of the virtual object preferably has a three-dimensional macroscopic surface geometry that curves along at least two mutually orthogonal directions. Preferably, the macroscopic surface geometry of the virtual object includes both protrusions and depressions. Preferably, the macroscopic surface geometry of the virtual object has surface normals covering a large solid angle, i.e., having a wide range of directions in three-dimensional space. Preferably, the solid angle covered by the direction of the surface normal of the virtual object is at least 20% or even at least 50% of the solid angle of a hemisphere, i.e., it is preferably at least 0.4πsr or even at least πsr. In this way, effects such as color shift, gloss, pearlescent, and glitter can be simultaneously compared between the sample object and the virtual object for a large number of illumination and observation directions relative to the surface normal of the virtual object.

[0036] The impression becomes more realistic if the non-planar surface portion of the displayed virtual object has a closed perimeter, and is displayed in such a way that it is surrounded by the displayed non-planar surface portion of the sample object along its entire closed perimeter. In this case, it is preferable that the display surface portion of the virtual object has a shape corresponding to the continuous continuation of the display surface portion of the sample object along its entire perimeter.

[0037] This is advantageous if the target device is located on the surface of the sample object. In this way, the illumination and observation conditions determined using the imaging target device are likely to match the illumination and observation conditions of adjacent surface portions of the sample object, and the color transformation parameters are likely to adequately describe not only the applicable color transformation at the location of the target device, but also the applicable color transformation at the locations of adjacent surface portions of the sample object.

[0038] Advantageously, the rendered virtual object is displayed on the display device at the location where the target device is positioned within the imaged scene. Specifically, the display portion of the rendered virtual object can be displayed on the display device, rather than on the imaged target device. In this way, the display portion of the rendered virtual object is precisely displayed at a location for which illumination conditions and color transformation parameters have already been determined. This further contributes to obtaining a realistic impression of the difference between the appearance of the rendered virtual object and the sample object.

[0039] The target device may be flexible along at least one bending direction to conform to the shape of the non-planar surface of the sample object. To ensure that irradiation and observation conditions can still be determined when the target device is bent, it may be advantageous to provide at least one rigid portion on the target device or to ensure that the target device can be bent only along a single bending direction. The target device may have the shape of a flat card on which color target elements and irradiation target elements are provided. The card may be made of any material.

[0040] In a preferred embodiment, the scene is imaged for multiple different illumination and / or observation conditions. Then, for the multiple different illumination and / or observation conditions, virtual objects are advantageously rendered, and the rendered virtual objects are displayed on a display device overlaid with the imaged scene for the different illumination and / or observation conditions. In other words, in some embodiments, the illumination and / or observation conditions are changed, and steps a), b), and e) through h) of the method are repeated at least multiple times. Specifically, the imaging device can be moved relative to the scene to image the scene from multiple different viewing directions. By displaying virtual objects in the imaged scene for multiple different illumination and / or observation conditions, a more reliable impression of the visual appearance of the virtual objects within the context of the imaged scene is obtained. This is particularly useful if the virtual objects exhibit effects such as color changes when illumination or observation conditions change.

[0041] In a particularly preferred embodiment, at least steps a), b), and e) through g) are repeated sequentially, such that the rendered virtual object is displayed on the display device as a continuous video stream overlaid with the imaged scene. This is advantageously performed in real time (i.e., as illumination and / or viewing conditions change).

[0042] This method is advantageously implemented using a computer system that cooperates with an imaging device and a display device. The method is advantageously executed by at least one processor of the computer system, which receives program instructions from a corresponding computer program. In some embodiments, a mobile electronic device is employed for both imaging and display, the mobile electronic device comprising the imaging device and the display device. In particular, the mobile electronic device may be a mobile phone, tablet computer, laptop computer, or virtual reality headset. In other embodiments, the imaging device and the display device may be separate units. The at least one processor performing the method of the invention may include at least one processor of the mobile electronic device (e.g., CPU and / or GPU), and / or it may include at least one processor of a separate computing device.

[0043] The display device can be a screen, such as an LCD screen, a projector, or any other type of display device as is well known in the art. If the display device includes a screen, the screen can be touch-sensitive, as is well known in the art. The display device can be capable of creating 3D impressions, as is well known in the art. For example, the display device can be a VR headset or a 3D display.

[0044] If the scene being imaged includes a sample object, the method may include retrieving appearance attributes and / or geometric parameters that indicate the geometry of the sample object from one or more databases, and using the retrieved appearance attributes and / or 3D geometric parameters to render a virtual object.

[0045] Specifically, the method may include:

[0046] Transmit metadata describing the sample object to the server; and

[0047] In response to the transmitted metadata, return the appearance attributes of the virtual object from the server, and optionally return the geometric parameters.

[0048] The returned appearance attributes can then be used, and if applicable, the returned geometric parameters, to render the virtual object. In other embodiments, the virtual object is rendered based on appearance attributes and / or geometric parameters that have already been provided in another manner (e.g., through analysis of images captured by an image capture device).

[0049] For example, if the imaged scene includes a sample object that is part of a vehicle, the metadata may include a vehicle identification number and / or paint identifier that identifies the paint on the sample object. In other embodiments, the metadata may include appearance attributes of the sample object obtained by an appearance capture device (e.g., a multi-angle spectrophotometer). In yet another embodiment, the metadata includes one or more images of the sample object, or is derived from one or more images of the sample object. The server processes the metadata and returns appearance attributes of the virtual object based on the metadata. For example, in the case of a vehicle component, the server may return appearance attributes of candidate paints expected to match the appearance of the paint on the sample object. For this purpose, the server may execute or access formulation software. Furthermore, the server may return geometric parameters of the virtual object based on the metadata.

[0050] In another aspect, the present invention provides a system for displaying virtual objects in an imaged real scene, the system being configured to perform the method of the present invention.

[0051] Specifically, the system may include:

[0052] The target device includes a set of color target elements having known reflective properties and at least one irradiation target element having a plurality of known surface normals;

[0053] Imaging devices;

[0054] Display device; and

[0055] A computer system configured to use an imaging device and a display device to perform the method of the present invention.

[0056] Specifically, the computer system can be configured to perform the following steps:

[0057] a) To image a scene using an imaging device, the scene including the target device;

[0058] b) Process at least one image of the scene to determine the lighting and viewing conditions of the scene;

[0059] c) Process at least one image of the scene to determine the measured color of the color target element;

[0060] d) Taking into account the determined irradiation conditions, calculate the color transformation parameters based on the comparison between the measured color and the known color of the color target element;

[0061] e) Render virtual objects based on the determined illumination conditions;

[0062] f) Align the rendered virtual object with the imaged scene;

[0063] g) Applying the calculated color transformation parameters to the imaged scene or applying the inverse color transformation to a virtual object; and

[0064] h) After applying a color transformation or inverse color transformation, the display device displays at least a portion of the rendered virtual object superimposed on at least a portion of the imaged scene.

[0065] As discussed in more detail above, the computer system can be configured to display, adjacent to, the non-planar surface portion of a rendered virtual object on a display device, such that the non-planar surface portion of the rendered virtual object has a shape corresponding to a continuous continuation of the non-planar surface portion of the sample object. As discussed in more detail above, the computer system can be configured to display a portion of the rendered virtual object on a target device at a location within the imaged scene. As discussed in more detail above, the computer system can be configured to at least repeat steps a), b), and e) through h) of the method, preferably continuously, such that the superposition of at least a portion of the rendered virtual object with at least a portion of the imaged scene is displayed as a continuous video stream, preferably in real-time. As discussed in more detail above, the system can include a mobile electronic device comprising an imaging device, a display device, and a processor.

[0066] In another aspect, the present invention provides a computer program product comprising program instructions that, when executed by at least one processor of a computer system, cause the at least one processor to perform the method of the present invention. Specifically, the instructions can cause the at least one processor to perform the following actions:

[0067] a) To image a scene including a target device, the target device comprising a set of color target elements having known reflective properties and at least one illumination target element having a plurality of known surface normals;

[0068] b) Process at least one image of the scene to determine the lighting and viewing conditions of the scene;

[0069] c) Process at least one image of the scene to determine the measured color of the color target element;

[0070] d) Calculate the color transformation parameters based on a comparison between the measured color of the target color element and its known reflectance properties;

[0071] e) Render virtual objects based on the determined illumination and observation conditions;

[0072] f) Align the rendered virtual object with the imaged scene;

[0073] g) Applying the calculated color transformation parameters to the imaged scene or applying the inverse color transformation to a virtual object; and

[0074] h) causes the display device to overlay at least a portion of the rendered virtual object with at least a portion of the imaged scene.

[0075] Computer program products may include non-volatile computer-readable media on which program instructions are stored. Non-volatile media may include hard disks, solid-state drives, memory cards, or any other type of physically tangible computer-readable media as is well known in the art. Attached Figure Description

[0076] Preferred embodiments of the invention are described below with reference to the accompanying drawings, which are for the purpose of illustrating the presently preferred embodiments of the invention and not for the purpose of limiting the presently preferred embodiments of the invention. In the drawings,

[0077] Figure 1 A schematic diagram of a scene is shown, which includes sample objects and target devices, as well as an illuminant for illuminating the scene and a mobile electronic device for imaging the scene.

[0078] Figure 2 A schematic block diagram of a mobile electronic device is shown, focusing on its hardware.

[0079] Figure 3 A flowchart illustrating a method for comparing the visual appearance of a real-world sample object with the expected appearance of a virtual object according to a first embodiment is shown; and

[0080] Figure 4 A flowchart illustrating a method for comparing the visual appearance of a real-world sample object with the expected appearance of a virtual object according to a second embodiment is shown. Detailed Implementation

[0081] definition

[0082] In this disclosure, references in the singular form may also include the plural. Specifically, unless the context otherwise indicates, the words “a” or “one” may refer to one or more.

[0083] The term "visual appearance," or simply "appearance," should be broadly understood as the way an object reflects and transmits light, including but not limited to how individuals observing the object perceive its color and surface texture under various viewing conditions. Appearance also includes instrumental measurements of how an object reflects and transmits light.

[0084] One aspect of visual appearance is color. The “color” of an object is determined by the portion of the spectrum of incident white light that is reflected or transmitted but not absorbed. The color of an object can be described by “color attributes.” Generally, color attributes indicate the spectral response of an object when it is illuminated by incident light. In the context of this disclosure, the term “color attribute” is to be understood broadly as including any form of data indicating the spectral response of an object when it is illuminated by incident light. Color attributes can take the form of color values ​​in any color space, such as in a three-color space like RGB or CIEXYZ, or in any other color space like CIELAB (L*a*b*), or in the form of spectral data representing the spectral response of a material to incident light in any format. In the context of this disclosure, color attributes may in particular include the reflectance values ​​and / or absorption and scattering coefficients of a material at multiple wavelengths.

[0085] Another aspect of visual appearance is texture. The term "texture" is broadly understood to refer to spatial variations in the appearance of a material's surface, both at microscopic or mesoscopic scales (i.e., scales at which individual structural elements are typically not perceptible to the naked eye) and at macroscopic scales (i.e., scales at which individual structural elements are perceptible to the naked eye). Texture, as understood in this disclosure, includes phenomena such as roughness, luster, and surface morphology. Texture can be described by "texture properties." In the context of this disclosure, the term "texture property" is broadly understood to include any form of data capable of quantifying at least one aspect of texture. Examples of texture properties include global texture properties, such as global roughness parameters or global luster parameters. In some embodiments, texture properties may include normal maps or height maps. In some embodiments, texture properties may include image data. In some embodiments, image data may be associated with a specific combination of illumination and viewing directions. In such embodiments, texture properties may include multiple image datasets, each associated with a different combination of illumination and viewing directions.

[0086] "Image data" is data representing an image. An image includes images of an actual target surface or object, as well as composite images of virtual objects derived from one or more geometric models combined with one or more sets of appearance attributes. An image may take the form of a two-dimensional array of image elements ("pixels"), each pixel having a specific pixel value. Pixel values ​​may represent the reflection at the pixel's location at a specific wavelength, the average reflection within a specific wavelength range, or the average reflection across all visible wavelengths. Therefore, in some embodiments, image data may be provided as an array of pixel values. In other embodiments, image data may be provided in a compressed or transformed form. An "image dataset" is image data comprising at least one image or a dataset consisting of image data of at least one image, i.e., a dataset representing one or more images.

[0087] The "bidirectional reflectance distribution function" (BRDF) is generally understood as a function that defines how light is reflected at an opaque surface depending on the direction of illumination and observation, thus providing the ratio of reflected radiation emitted along the observation direction to the irradiance incident on the surface from the direction of illumination. If the surface exhibits spatial variation in this ratio, the BRDF is understood to provide the average value of this ratio over the surface area.

[0088] An "appearance capturing device" is a device capable of determining one or more appearance attributes of an object. Depending on the appearance attribute to be determined (e.g., color attribute, texture attribute, or other attribute), the appearance capturing device may take the form of, for example, a camera, a colorimeter, a spectrophotometer, or an imaging spectrophotometer.

[0089] A spectrophotometer is a device used to determine the reflectance and / or transmittance properties (as a function of wavelength, i.e., the spectral response of the object) of a material or surface when illuminated with visible light. Different types of spectrophotometers are known, each with different geometries and optimized for different purposes. One important type is the integrating sphere spectrophotometer. An integrating sphere spectrophotometer comprises an integrating sphere, a hollow spherical cavity defined by a diffuse white inner surface, having at least one inlet port for illumination and at least one outlet port for observation. The integrating sphere produces a uniform scattering or diffusion effect. Through multiple scattering and reflection, light incident on any point on the inner surface is uniformly distributed to all other points. The influence of the original direction of light is minimized. Examples of integrating sphere spectrophotometers are X-Rite models Ci7860 and Ci7500. Other types of spectrophotometers determine only spectral information for a single narrow range of illumination direction (e.g., 45° to the surface normal) and a single narrow range of observation direction (e.g., 0° to the surface normal). Examples include models 962 and 964, available from X-Rite. Several other spectrophotometers, referred to as "gonometric spectrophotometers" or "multi-angle spectrophotometers," are capable of determining spectral information for multiple combinations of different illumination and observation directions. Spectrophotometers may additionally have imaging capabilities, meaning they can include one or more cameras to capture one or more digital images of an object. Examples of multi-angle spectrophotometers with imaging capabilities include the benchtop model TAC7 or the handheld models MA-T6 or MA-T12, available from X-Rite.

[0090] "Imaging apparatus" is broadly understood as a device configured to capture still or moving images, such as a digital camera with or without video capture capabilities.

[0091] In the context of this disclosure, the term "target device" refers to an apparatus comprising a set of color target elements having known reflective properties and a set of one or more irradiating target elements having a plurality of known surface normals relative to the color target elements, as described in more detail elsewhere in this disclosure.

[0092] In this disclosure, virtual objects are rendered and displayed. The geometry of the virtual object can be described by a three-dimensional geometric model. The three-dimensional geometric model defines the three-dimensional macroscopic surface geometry of the virtual object. The geometric model can be represented by a CAD file. Preferably, the macroscopic surface geometry has at least one portion that bends (preferably continuously bends) along at least two mutually orthogonal directions. In mathematical terms, the surface geometry is “continuously curved” if it at least approximately and at least locally corresponds to a two-dimensional differentiable manifold in three-dimensional Euclidean space. In some embodiments, the curved portions of the virtual object can be represented in the geometric model using a combination of small polygonal planar surfaces (e.g., polygonal meshes). In 3D computer graphics, polygonal modeling is well known as a method for modeling objects by representing or approximating their surfaces using polygonal meshes. If the polygonal mesh behaves as a continuous surface during rendering, the geometry defined by the polygonal mesh is considered continuously curved. Preferably, the curved three-dimensional macroscopic surface geometry includes both convex and concave portions. Preferably, the virtual object has surface normals covering a large solid angle, i.e., having a wide range of directions in three-dimensional space. Preferably, the solid angle covered by the direction of the surface normal of the virtual object is at least 50% of the solid angle of the hemisphere, that is, it is preferably at least 1πsr. In this way, effects such as color shift, gloss, glitter, and texture can be compared between two materials simultaneously in a large number of illumination and observation directions relative to the surface normal of the virtual object.

[0093] The term "macroscopic surface geometry" should be understood to refer to the overall geometry of a product, excluding micro or mesoscopic surface structures, i.e., variations in surface geometry at micro or mesoscopic scales less than, for example, 1 mm. For example, local variations in surface height and local average values ​​less than, for example, 1 mm can be considered micro or mesoscopic surface structures, and therefore macroscopic surface geometry can be equivalent to the surface geometry averaged over a length scale of at least 1 mm.

[0094] "Illumination conditions" may include the characteristics of the luminescent body and the direction of illumination or the distribution of such directions. "Observation conditions" may include the observation direction and observation distance.

[0095] "Color transformation" converts a first set of color attributes into a second set of color attributes. The color attributes can be color values ​​or spectral values ​​in any color space. This transformation can involve changing from one color space to a different color space (e.g., from measured spectral values ​​or RGB to a device-independent color space such as CIEXYZ or CIELAB).

[0096] The term "rendering" refers to the automated process of generating photorealistic images of objects or scenes using computer programs. Input information for rendering operations includes a 3D model of the object, a set of appearance attributes associated with the object, information about the object's position and orientation, lighting conditions (which may take the form of an environment map), and the viewpoint. Many different rendering algorithms are known at varying levels of complexity, and software used for rendering can employ a variety of techniques to obtain the final image. Tracking every single light particle in a scene is generally impractical because it requires excessive computational time. Therefore, simplified techniques for modeling light transport are typically used. One of these techniques is ray tracing.

[0097] The phrase "aligning a virtual object with the imaged scene" refers to the process of determining where to place the rendered object relative to other objects in the scene. In the context of this disclosure, the virtual object is preferably aligned with at least one reference object in the scene in such a way that even if the imaging device moves relative to the imaged scene, the virtual object will not appear to have moved relative to the reference object.

[0098] The term "visualization" encompasses both rendering and display. For display, a display device is used. The term "display device," or simply "monitor," is to be understood as referring to a computer output device used to present information in a visual form. A display device can take the form of a computer monitor, TV screen, projector, VR headset, or the screen of a handheld device such as a smartphone or tablet. A display device can be a touchscreen. In some embodiments, the display device can be a "virtual light booth," as disclosed in EP 3163 358 A1, to provide a particularly realistic impression of the rendered scene.

[0099] The term "database" refers to a collection of organized data that can be accessed electronically by a computer system. In a simple embodiment, a database can be a searchable electronic file in any format. Examples include Microsoft Excel. TM A spreadsheet or a searchable PDF document. In more complex embodiments, the database can be a relational database maintained by a relational database management system using a language such as SQL.

[0100] The term "computer" or "computing device" refers to any device that can be instructed by a program to automatically perform a sequence of arithmetic or logical operations. Without limitation, a computer can take the form of a desktop computer, laptop computer, tablet computer, smartphone, programmable digital signal processor, etc. A computer typically includes at least one processor and at least one memory device. A computer can be a subunit of another device (such as an appearance capture device). A computer can be configured to establish a wired or wireless connection to another computer, including a computer used for querying databases. A computer can be configured to be coupled to a data input device (such as a keyboard or computer mouse) or a data output device (such as a monitor or printer) via a wired or wireless connection.

[0101] The term "computer system" should be broadly understood to include one or more computers. If a computer system includes more than one computer, these computers do not necessarily need to be located in the same place. Computers within a computer system can communicate with each other via wired or wireless connections.

[0102] A processor is an electronic circuit that performs operations on external data sources, particularly memory devices.

[0103] A "memory device," or simply "memory," is a means for storing information for use by a processor. Memory devices can include volatile memory, such as random access memory (RAM), and non-volatile memory, such as read-only memory (ROM). In some embodiments, a memory device can include a non-volatile semiconductor memory device, such as (E)EPROM or flash memory, which may take the form of, for example, a memory card or a solid-state drive. In some embodiments, a memory device can include a high-capacity storage device with mechanical components, such as a hard disk. A memory device can store programs for execution by a processor. Non-volatile memory devices can also be referred to as non-volatile computer-readable media.

[0104] A "program" is a collection of instructions that can be executed by a processor to perform a specific task.

[0105] A "wired connection" is a connection via an electrical conductor. A wired connection may include one or more cables. A "wireless connection" is a connection involving the electromagnetic transmission of information between two or more points that are not connected by an electrical conductor. Wireless connections include those via WiFi. TM ,Bluetooth TM Connections to 3G / 4G / 5G mobile networks, optical communication, infrared, etc.

[0106] Scene imaging

[0107] Figure 1A schematic diagram of a scene 100 including a sample object 110 and a target device 120 is shown. The scene is illuminated by a natural or artificial light source (light emitter) 300. It is imaged using a mobile electronic device 200.

[0108] Sample object 110 defines a non-planar sample surface. For simplicity, Figure 1 Only the surface is shown. The sample surface can have any shape. In this example, the non-planar sample surface is illustrated as curved in only one direction. However, the sample surface can be curved in more than one direction. In some embodiments, the sample surface may be formed by paint or coating on the sample object.

[0109] The target device 120 includes multiple color target elements 121, illumination target elements 122, and multiple alignment marks 123.

[0110] Each color target element 121 has predetermined, known reflectance properties. In particular, the correlation between the spectral reflectance of each color target element 121 and the direction of illumination and observation is known. For example, the parameters of the BRDF model for each color target element 121 are known.

[0111] The irradiated target element 122 has multiple surface normals that vary along at least two non-parallel directions (i.e., along the inclination and azimuth in polar coordinates). Its surface is smooth or at least semi-smooth, meaning its BRDF has a significant specular component. In this example, the irradiated target element 121 has a hemispherical shape.

[0112] For possible embodiments of the target device 120, particularly possible embodiments of the color target element 121 and the illumination target element 122, refer to US 10,049,294 B2. More complex color target elements may be employed, such as color target elements including texture, glitter, etc.

[0113] Alignment marks 123 can be simple marks such as crosses, or they can be configured as reference marks such as those disclosed, for example, in “Generation of fiducial marker dictionaries using mixed integer linear programming”, Pattern Recognition 51 (2016) 481-491, by S. Garrido-Jurado et al.

[0114] In this example, the target device 120 takes the form of a flat card. It can be flexible, allowing it to bend in one direction. It is positioned directly on the surface of the sample object 110 and can be held on the sample surface by adhesive or by magnetic force.

[0115] The light source 300 can be any type of light source that provides a sufficiently broad spectrum to distinguish different colors. For example, the light source 300 may include sunlight, one or more incandescent lamps, one or more discharge lamps, one or more LEDs, etc. The illumination conditions of each object in the scene are the result of the interaction between the light source and the object's environment. These may differ across scenes.

[0116] The mobile electronic device 200 includes an imaging device in the form of a digital camera 210 and a display 220. Figure 1 In one example, the mobile electronic device 200 is a tablet computer or smartphone, and the display is a multi-touch display that serves as both an input and output device. In other embodiments, the mobile electronic device 200 may be a laptop computer or a VR headset with an integrated display and camera. In yet another embodiment, the camera 210 and the display 220 may be included in different devices. For example, the display may be a standard LCD display for a desktop computer or a display for a laptop computer, and a separate digital camera may be connected to the desktop or laptop computer. However, it is preferred that the camera and display be included in a single device, as this makes more realistic evaluations possible.

[0117] Hardware of mobile electronic devices

[0118] Figure 2 A hardware-oriented schematic block diagram of a mobile electronic device 200 is shown. The mobile electronic device includes a camera 210 and a display 220. Furthermore, the mobile electronic device includes electronic components forming a computer system 201. As is well known in the art, various components of the computer system 201 communicate via one or more buses 202. The computer system 201 includes one or more processors 230. As is well known in the art, processors 230 may include, for example, a single-core or multi-core CPU and a GPU. The computer system 201 also includes one or more non-volatile memory devices 240, such as flash memory devices and / or hard disk drives. The non-volatile memory 240 stores, in particular, the operating system 241 of the computer system 201 and one or more application programs 242. The non-volatile memory 240 also stores program data and user data. The data may, in particular, include calibration data 243, as will be explained in more detail below. The computer system 201 also includes random access memory (RAM) 250 and a communication interface 260. The communication interface 260 may include, for example, an Ethernet interface, a WiFi interface, or Bluetooth. TMOne or more of the following interfaces. The communication interface can be used to communicate with a remote server 400, which in turn can access the database 410. Communication can be conducted via wired or wireless networks, such as via a LAN or WAN, and particularly via the Internet. The computer system 201 also includes an input / output interface 270 connected to the camera 210 and the display 220.

[0119] Display virtual objects and compare their visual appearance with that of imaged sample objects.

[0120] Figure 3 An embodiment of a computer-implemented method for displaying a virtual object 140 in an imaged scene 100 using a mobile electronic device 200 so as to be able to compare the expected visual appearance of the virtual object 140 with the visual appearance of a sample object 110 is illustrated.

[0121] In step 501, the user inputs metadata into the mobile electronic device 200, which identifies and / or describes the sample object 110. For example, if the sample object is a car part, the user can input the vehicle identification number (VIN) of the car and an identifier for the relevant car part (e.g., an identifier specifying that the part is the right rear door). In other embodiments, the user uses the camera 210 of the mobile electronic device 200 to acquire at least one image of the sample object 110, and the mobile electronic device 200 derives metadata from the image of the sample object 110. In yet another embodiment, the user uses an appearance capture device, such as a multi-angle spectrophotometer, to determine the appearance attributes of the sample object 110 and inputs these appearance attributes as metadata into the mobile electronic device 200. These and other possibilities can also be combined to generate more detailed metadata.

[0122] In step 502, the mobile electronic device 200 transmits metadata to the server 400 using the communication interface 260. The server 400 receives the metadata and uses it to define suitable virtual objects and uses the database 410 to determine associated appearance attributes and geometric parameters. For example, if the sample object is a car part painted with existing paint, the database may store known appearance attributes of selected paints recently provided by one or more car manufacturers, and the metadata may identify the existing paint. The server may then return the appearance attributes of the existing paint. In an alternative embodiment, the server may determine a formulation of a new candidate paint that is expected to match the existing paint. Color configuration algorithms for finding candidate formulations are generally known in the art. Particularly advantageous methods for finding candidate formulations are disclosed in European patent applications filed on 7 July 2020, EP20184560, EP20184561 and EP20184562, the contents of which are incorporated herein by reference in their entirety. The server may return the appearance attributes of the candidate paint. The same or different databases may also store, for example, the geometric parameters of all relevant automotive parts of one or more car manufacturers in the form of CAD data, and the server may be configured to return the geometric data of the automotive parts identified and / or described by metadata from the database.

[0123] In step 503, the mobile electronic device 200 receives the appearance attributes and geometric data of the virtual object from the server 400.

[0124] In step 504, the user registers the target device 120 with the mobile electronic device 200. For example, the user can enter a unique identifier for the target device 120 into the mobile electronic device 200, or scan a barcode or QR code on the back of the target device 120, or preferably, identify it using a reference mark on the front of the target device. The mobile electronic device 200 can then retrieve calibration data for the thus identified target device 120 from a local or remote database and store it as calibration data 243 in non-volatile memory 240. The calibration data 243 may specifically include known color properties of the color target element 121 (e.g., in the form of parameters of the BRDF model of these color target elements). The calibration data may also include geometric data describing the surface normal map of the irradiated target element 122 and the reflection data of the element, which may again be in the form of parameters of the BRDF model of the surface of the irradiated target element 122. The user then places the target device 120 on the sample object 110, thereby creating scene 100.

[0125] In step 511, the user uses the camera 210 of the mobile electronic device 200 to image the scene 100.

[0126] In step 512, the mobile electronic device 200 processes the portion of the imaged scene containing the target device 120, for example, using alignment marks 123 and / or distorted target elements, to determine viewing conditions. The mobile electronic device 200 also uses the irradiated target element 122 (e.g., in the form of a local environment map of global illumination) to determine illumination conditions. For details, refer to US10,049,294B2.

[0127] In step 513, the mobile electronic device 200 processes a portion of the scene containing the image of the target device 120 to determine the measured color of the color target element 121.

[0128] In step 514, the mobile electronic device 200 calculates color transformation parameters based on a comparison of the measured color of the color target element 121 and its known reflectivity, taking into account illumination and observation conditions. For details, see US10,049,294B2.

[0129] In step 515, the mobile electronic device 200 renders a virtual object using the geometric parameters and appearance attributes received in step 503, based on the illumination and observation conditions determined in step 512. It also aligns the rendered virtual object with the imaged sample object in such a way that, when displayed on the display 220, the surface of the rendered virtual object 140 matches the imaged surface of the sample object 110.

[0130] In step 516, the mobile electronic device 200 applies a color transformation to the image of scene 100 using the color transformation parameters determined in step 514. For details, refer again to US10,049,294B2. As a result of the color transformation, the image is now represented by device-independent color values ​​that have been corrected for, for example, the nonlinear radiometric response and unknown spectral sensitivity of camera 210.

[0131] In step 517, portions of the imaged sample object 110 and the rendered virtual object 140 are displayed together on the display 220 as an overlay (see [link]). Figure 1 ).like Figure 1 As illustrated, the display portion of the rendered virtual object 140 can be displayed in such a way that it replaces the imaging target device 120 on the display 220 and is completely surrounded by the display portion of the imaging sample object 110 along its entire perimeter. Because the surface geometry of the virtual object 140 is consistent with the surface geometry of the sample object 110, the display portion of the rendered virtual object 140 is seamlessly integrated into the display portion of the imaging sample object 110.

[0132] Augmented Reality Environment

[0133] In some embodiments, the scene is imaged for multiple different illumination and / or observation conditions. In particular, the mobile electronic device 200 moves relative to the scene to image the sample object 110 from multiple different viewing directions. By acquiring image data from the sample object 110 and rendering the virtual object 140 from different viewing angles, the scene will provide a more reliable impression of the visual appearance of the virtual object in the context of the imaged scene when the virtual object is rendered.

[0134] Figure 4 The figure illustrates a preferred embodiment. Although not shown in Figure 4 As shown in the diagram, but initially it can be combined as shown in the diagram. Figure 3 Execute as described Figure 3 Steps 501 and 504. Then, steps 511 through 517 are performed at least once. Subsequently, steps 511, 512, 515, 516, and 517 are repeated consecutively to create a continuous video stream on display 220, while the user moves the mobile electronic device 200 around the sample object 110 in step 518 to continuously change the viewing conditions. Figure 1 (See arrow M in the image). Furthermore, the illumination conditions can be changed, for example, by activating and / or moving one or more light sources and / or by rotating the sample object 110.

[0135] When viewing and / or illumination conditions change, the user observes display 220 and compares the visual appearance of the displayed portions of the imaged sample object 110 and the rendered virtual object 140. Therefore, the user compares the visual appearance of these objects, which are seamlessly displayed together on display 220 in a highly realistic augmented reality environment, enabling highly reliable prediction of the appearance of the virtual object 140 from different viewing angles and illumination conditions, and how well the appearances of the sample object 110 and the virtual object 140 match.

[0136] Steps 513 and 514, namely determining the measured color of the color target element 121 and calculating the color transformation parameters, do not need to be repeated in each iteration. In principle, performing these steps only once is sufficient. However, it may be advantageous to repeat these steps at least periodically or even in each iteration. For example, the determined color transformation parameters may change slightly when illumination conditions and / or observation conditions change. In some embodiments, the color transformation parameters determined for different iterations can be averaged, and the averaged color transformation parameters can be used to apply the color transformation.

[0137] Optional production of test subjects

[0138] If the match is deemed good enough, the appearance attributes of the virtual object can optionally be used to produce an actual test object. Therefore, the actual appearance of the test object and the sample object can be compared in reality. It should be noted that the geometric parameters of the test object do not necessarily need to correspond to the geometric parameters of the virtual object. For example, if the test object is a replacement part of a damaged car, then the sample object 110 would be the complete part of the car, and the virtual object would be a model of that complete part. On the other hand, the replacement part will be used to replace the damaged part of the car; that is, the geometry of the replacement part (test object) will generally differ from the geometry of the sample object. Nevertheless, the method disclosed herein facilitates a reliable prediction of whether the appearance of the replacement part will blend well into the appearance of the remaining complete parts of the car.

[0139] Revise

[0140] Although the invention has been described with reference to preferred embodiments, various modifications are possible without departing from the scope of this disclosure.

[0141] For example, color transformation does not necessarily need to be applied to the imaged scene (or a portion thereof). In an alternative embodiment, instead, an inverse color transformation can be applied to the virtual object (or a portion thereof). This can be done before, during, or after the rendering process. Applying an inverse color transformation to the virtual object may be particularly advantageous if the camera 210 and display 220 are calibrated in such a way that the color impression on display 220 is fairly realistic for the image captured by camera 210, whereas if device-independent color values ​​are provided, there may be considerable color deviation on the display. In yet another embodiment, both the imaged scene (or a portion thereof) and the virtual object undergo individual color transformations such that the color values ​​of the imaged scene 100 and the rendered virtual object 140 are actually correlated with each other by the color transformation defined by the color transformation parameters determined in step 507.

[0142] When comparing the appearance of a virtual object with that of a sample object, the geometry of the virtual object does not need to be exactly the same as that of the sample object. It may be sufficient if the display surface of the virtual object is a continuous continuation of the display surface of the sample object. While it is advantageous for the display surface of the virtual object to be completely surrounded by the display surface of the sample object, this is not strictly necessary. It may be sufficient if the display portions of the virtual object and the sample object meet along (straight or curved) boundary lines.

[0143] While applications in vehicle repair scenarios have been referenced, many other applications of the proposed methods, systems, and computer programs are possible. For example, the proposed methods can also be applied to visualize new products in a realistic environment prior to actual production.

[0144] Virtual objects may include one or more materials having a robust texture, such as fabric. If such an object is to be compared to a sample object, the target device may include textured target elements with known properties, in addition to solid-color target elements, to better characterize the texture of the sample object. The target device may also include filters as disclosed in US9,823,131.

[0145] Further modifications are possible without departing from the scope of this disclosure.

[0146] List of reference markers

[0147] 100 scenes

[0148] 110 sample objects

[0149] 120 target equipment

[0150] 121 color target element

[0151] 122 Irradiation Target

[0152] 123 Alignment Marks

[0153] 200 mobile electronic devices

[0154] 201 bus system

[0155] 210 camera

[0156] 220 monitor

[0157] 230 processor

[0158] 240 non-volatile memory

[0159] 241 Operating System Data

[0160] 242 program data

[0161] 243 calibration data

[0162] 250 RAM

[0163] 260 communication interface

[0164] 300 light-emitting bodies

[0165] 400 server

[0166] 410 Database

[0167] 501-518 Process Steps

Claims

1. A method for displaying a virtual object (140) in an imaged scene (100), the method comprising: a) Image a scene (100) using an imaging device (210), the scene including a target device (120), the target device (120) including a set of color target elements (121) having known reflective properties and a set of one or more illumination target elements (122) having a plurality of known surface normals relative to the color target elements (121). b) Using an imaging target device (120), process at least one image of the scene (100) to determine the illumination and observation conditions of the scene (100); c) Process at least one image of the scene (100) to determine the measured color of the color target element (121); d) Taking into account the determined illumination and observation conditions, calculate the color transformation parameters based on the comparison between the measured color of the color target element (121) and the known reflectance properties; e) Render the virtual object for the determined illumination and observation conditions (140); f) Align the rendered virtual object (140) with the imaged scene (100); g) Applying a color transformation to at least one image of the scene without applying the color transformation to a virtual object using calculated color transformation parameters, or applying an inverse color transformation to a virtual object without applying the inverse color transformation to at least one image of the scene, wherein the color transformation transforms a first set of color attributes into a second set of color attributes, such that the colors of the imaged scene and the colors of the rendered virtual object become directly comparable when displayed; and h) Display at least a portion of the rendered virtual object (140) superimposed on at least a portion of the imaged scene (100) on the display device (220).

2. The method according to claim 1, wherein, The scene (100) includes a sample object (110), wherein a non-planar surface portion of a rendered virtual object (140) is displayed adjacent to a non-planar surface portion of the sample object (110), the non-planar surface portion of the rendered virtual object (140) having a shape that is a continuous continuation corresponding to the non-planar surface portion of the sample object (110).

3. The method according to claim 2, wherein, The target device (120) is located on the surface of the sample object (110).

4. The method according to any one of the preceding claims, wherein, The display portion of the rendered virtual object (140) is displayed on the display device (220) at the location where the target device (120) is located in the imaged scene (100).

5. The method according to any one of claims 1 to 3, comprising: j) Change the lighting and / or observation conditions of the scene; as well as k) For altered irradiation and / or observation conditions, repeat steps a), b) and e) through g at least.

6. The method according to claim 5, wherein, At least steps a), b), and e) through g) are repeated continuously, such that at least a portion of the rendered virtual object (140) is superimposed on at least a portion of the imaged scene (100) and displayed on the display device (220) as a continuous video stream, particularly in real time when illumination conditions and / or observation conditions change.

7. The method according to any one of claims 1 to 3, wherein, A mobile electronic device (200) is used for both imaging and display, the mobile electronic device (200) including an imaging device (210) and a display device (220).

8. The method according to any one of claims 1 to 3, comprising: The metadata describing the sample object (110) is transmitted to the server; as well as In response to the transmitted metadata, the appearance properties of the virtual object (140) are returned from the server.

9. A system for displaying a virtual object (140) in an imaged scene (100), the system comprising: The target device (120) includes a set of color target elements (121) having known reflective properties and at least one illumination target element (122) having a plurality of known surface normals. Imaging device (210); Display device (220); and A computer system configured to perform the following steps: a) The imaging device (210) images a scene (100), the scene (100) including the target device (120). b) Process at least one image of the scene (100) to determine the illumination and viewing conditions of the scene (100); c) Process at least one image of the scene (100) to determine the measured color of the color target element (121); d) Taking into account the determined illumination and observation conditions, calculate the color transformation parameters based on the comparison between the measured color of the color target element (121) and the known reflectance properties; e) Render virtual objects for the determined illumination and observation conditions (140); f) Align the rendered virtual object (140) with the imaged scene (100); g) Applying a color transformation to at least one image of the scene without applying the color transformation to a virtual object using calculated color transformation parameters, or applying an inverse color transformation to a virtual object without applying the inverse color transformation to at least one image of the scene, wherein the color transformation transforms a first set of color attributes into a second set of color attributes, such that the colors of the imaged scene and the colors of the rendered virtual object become directly comparable when displayed; and h) The display device (220) displays at least a portion of the rendered virtual object (140) superimposed on at least a portion of the imaged scene (100).

10. The system according to claim 9, wherein, The computer system is configured to display the non-planar surface portion of a rendered virtual object (140) adjacent to the non-planar surface portion of a sample object (110) in the scene (100), the non-planar surface portion of the rendered virtual object (140) having a shape corresponding to the continuous continuation of the non-planar surface portion of the sample object (110).

11. The system according to claim 9 or 10, wherein, The computer system is configured to cause the display device (220) to display a portion of the rendered virtual object (140) at the location of the target device (120) in the imaged scene (100).

12. The system according to claim 9 or 10, wherein, The computer system is configured to repeat steps a), b), and e) through g) at least multiple times.

13. The system according to claim 12, wherein, The computer system is configured to repeat steps a), b) and e) to g) at least continuously, such that at least a portion of the rendered virtual object (140) is superimposed on at least a portion of the imaged scene (100) as a continuous video stream, particularly in real time.

14. The system according to claim 9 or 10, comprising a mobile electronic device (200), the mobile electronic device (200) including an imaging device (210) and a display device (220).

15. A computer program product comprising program instructions, wherein the program instructions, when executed by at least one processor (230), cause the at least one processor (230) to perform the following actions: a) The imaging device (210) images a scene (100) including a target device (120), the target device (120) including a set of color target elements (121) having known reflective properties and at least one illumination target element (122) having a plurality of known surface normals. b) Process at least one image of the scene (100) to determine the illumination and viewing conditions of the scene (100); c) Process at least one image of the scene (100) to determine the measured color of the color target element (121); d) Taking into account the determined illumination and observation conditions, calculate the color transformation parameters based on the comparison between the measured color of the color target element (121) and the known reflectance properties; e) Render virtual objects for the determined illumination and observation conditions (140); f) Align the rendered virtual object (140) with the imaged scene (100); g) Applying a color transformation to at least one image of the scene using calculated color transformation parameters without applying the color transformation to the virtual object, or applying an inverse color transformation to the virtual object without applying the inverse color transformation to at least one image of the scene, wherein, The color transformation converts the first set of color attributes into a second set of color attributes, making the colors of the imaged scene and the rendered virtual objects directly comparable during display; and h) The display device (220) simultaneously displays at least a portion of the imaged sample object (110) and at least a portion of the rendered virtual object (140).

Images