Color rendering

Abstract

To provide a method, system and software for improved color rendering of an item or scene, such as an item for sale through websites or printed brochures, e.g., color catalogs.SOLUTION: A method for improving color rendering includes: improving accuracy in rendering an online item, such as on a user's laptop, smartphone, a tablet computing device, an electronic-book reader or desktop computer showing a webpage; customizing rendering of an item to fit accurate rendering of the item or a buyer's intended use for the item; and improving color renderings for a print item through use of a user's device to capture an image of the printed item using a device's camera, and then display the item more accurately or congruently than it was originally displayed in print.SELECTED DRAWING: Figure 4

Description

Background Art

[0001] When viewing an item online or in the pages of a catalog sent as a postal item, the true appearance of the item may not be accurately presented. For example, consider an off-white chair sold in a catalog on paper. Suppose a buyer wants to purchase the chair in an off-white color but is unsure whether the displayed color matches their current off-white interior decoration and ultimately decides not to purchase it. This not only results in the seller missing out on a revenue opportunity but also the buyer missing out on the benefits of owning the chair. Moreover, there are cases where the buyer purchases the chair believing it will match their interior decoration but discovers after delivery that it does not. In such cases, the buyer or the seller has to pay for the return shipping, and both have to spend time and effort on the return process. The above-mentioned problems caused by inaccurate color rendering of items on catalogs and similar papers lead to high costs.

[0002] Regarding the accuracy of rendering (e.g., the depiction of an item's image), generally, the rendering in online sales is of lower quality than that in catalogs. In many cases, color rendering depends on an image capture device and an image rendering device, and both devices can be accompanied by color errors, resulting in inaccurate color rendering. Due to such inaccurate rendering, items are often displayed in colors that are slightly or significantly different from their actual colors. Return problems frequently occur only based on such defects, causing huge losses due to the shipping cost of the product to the buyer and the return shipping cost from the buyer, dissatisfaction among buyers, and further losses of time and money for both the buyer and the seller.

Summary of the Invention

[0003] The present disclosure relates to methods, systems, and / or software configured to improve the color rendering of items and scenes such as products sold through websites, printed brochures (such as color catalogs), etc. This improvement in color rendering can be achieved by improving the accuracy when displaying web pages on a user's laptop, smartphone, or desktop computer to showcase products online. Further, the present disclosure relates to a method for solving many problems in the field of online sales where accurate representation of products and customized representations according to the intended use of the product by the buyer are desired. Additionally, the present disclosure also describes a method for improving the color rendering of a printed product through the use of a user's device, such as capturing an image of the printed product with the device's camera and then displaying it more accurately or realistically than its original printed display state.

Brief Description of the Drawings

[0004] This application includes at least one color drawing. Copies of the color drawing(s) are available from the United States Patent and Trademark Office upon request and payment of the required fee. The following figures will be referred to for an explanation of the technology and device aspects enabling machine learning-based filtering searches. The same reference numerals are used for similar features and components throughout each figure.

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Mode for Carrying Out the Invention

[0005] Overview of the Invention The present disclosure relates to a method, system, and / or software configured to improve color rendering. For example, color rendering in photographs of families taken with a camera, products and presentations provided through a website, or products and presentations published in printed brochures such as colored catalogs. In this technique, the effects of various means such as a user's laptop, smartphone, or desktop computer that displays a web page are improved, thereby making it possible to improve the accuracy when presenting products online. Further, the present disclosure relates to a method for solving many problems in the field of online presentation where an accurate description of a product or a scene is desired. Furthermore, the technique of the present disclosure also enables customization of effects according to the intended use of the product by the buyer. The present disclosure also describes a method for improving the description of a product included in a printed image through the use of a user's device, such as capturing an image of a printed product with the camera of the user's device and displaying it more accurately on the display unit of the device than in the original printed display state. When the original image is not captured correctly

[0006] There may be cases where the original image of an item or scene is not accurately captured. Therefore, the image of a product may already be inaccurate before it is displayed on a website for selling the product or provided for printing. The techniques described in this disclosure utilize physical color standards to improve accuracy. These physical color standards provide a means to verify the color of the captured image. To utilize such physical color standards, an image of an item is captured along with its physical color standard. This can be done using the physical color standard within the image of the item, or within an image captured simultaneously, or within an image captured under the same or similar ambient conditions. Image providers such as sellers can, by adopting physical color standards, modify the captured image while comparing the displayed physical color standard with the physical color standard that can be perceived with their own eyes. Although this is done by humans and there may be a possibility of adding new errors, it is also possible to correct the errors and differences between the reflected light actually emitted from an object (the measurable actual physical characteristics of the reflected light) and that based on human perception.

[0007] Physical color standards can be replicated or standardized and can actually be the same as the actual physical color standard imaged together with the item. Alternatively, physical color standards can also be captured using a computer program that stores effects or data related to the physical color standards. In any case, or in either situation, adopting physical color standards can improve the accuracy of images of items or scenes. As a result, entities (real persons or things) such as those taking the images, like sellers, can provide the modified and improved images to a web page or a printer. With the techniques described in this disclosure, the accuracy of the depiction of items or scenes is improved, so that subsequently, when finally displayed to other entities (for example, viewers of the scene or buyers of the item, etc.), the presentation of the item can also be naturally improved. Furthermore, in the adoption of physical color standards by these techniques, the image of an item or scene can be automatically corrected using the color information recorded for the physical color standards. These techniques will be further described in more detail below.

[0008] Moreover, in the adoption of physical color standards by this technique, the image of an item or scene can be automatically corrected using the color information recorded for the physical color standards. These techniques will be further described in more detail below. Incorrect or unconvincing rendering

[0009] When rendering a color image on paper or a display, there are numerous reasons why it may not be accurately rendered. As previously mentioned, the original cause may be that the image of the item or scene was not accurately captured, but this is not always the only problem.

[0010] Suppose the seller takes the original image of the product with a camera that cannot accurately capture colors or with ambient light that causes the product to be depicted in a color different from normal ambient light (e.g., when taking an image in a studio illuminated by fluorescent lights or when taking an image of a product intended for outdoor use indoors, with different home lighting or sunlight than normal).

[0011] In such cases, the product shown in the original image does not resemble its appearance when viewed under different ambient conditions. For example, it is different from the appearance when viewed in the location where the buyer intends to use the product (e.g., outdoors for outdoor furniture, indoors for outdoor / indoor clothing, paintings, cushions, and carpets). As previously mentioned, the original image may further lose accuracy on the surface even without errors due to lighting differences.

[0012] The techniques described in this disclosure provide a physical color standard as a means to correct this defect. This physical color standard provides a means to collate the color of the rendered image through calibration performed simultaneously with or prior to the rendering of the image (which will be described in detail later). To utilize this physical color standard, an entity on the shooting side such as the seller captures the image of the product together with the physical color standard. As previously mentioned, this can be done using the physical color standard included in the image of the product taken. (However, until there is a rendering requirement accompanied by a physical color standard so that the description space of the product does not become complex, the physical color standard can also be extracted from the image and provided in response to a request from the web page without being displayed.)

[0013] To explain in more detail, assume the case of a bright teal shirt. The color of the shirt is an element that is extremely important to the buyer. Therefore, rather than speculating on whether the seller was able to take an accurate image of the shirt or an image that resembles it, or speculating on whether the buyer's own display unit can accurately display the color, instead, the seller uses a physical color standard to capture an image of the shirt, and then, the buyer is given an opportunity to attempt an accurate presentation. Note that the terms "accurate" and "accuracy" used in this disclosure refer to the fact that when a product is presented as an image, the color of the product is accurately represented as compared to the actual appearance of the product at the time of shooting. On the other hand, "congruent" is a term regarding how the product looks at the same or a different location from where the original image was taken. Thus, users may sometimes seek accuracy, congruence, or a combination of both. When purchasing a product, the buyer hopes that an accurate image of the product is presented in a catalog or on a website. Also, it is desirable that the image of the product is displayed in the same way as the assumed posture when the user is looking at the catalog or the display unit. This As will be detailed in this disclosure, the user may wish to modify and display an image of the product or scene so that the user's surrounding environment, lighting, and further, the lighting angle, etc. are reflected in the product. Also, although congruence is not very important, it is also common to desire accurate color rendering.

[0014] As described above, (since the physical color standard is reproducible or standardizable and can be used for calibrating an image captured with the help of a computer program or a person or both,), the seller can modify the captured image based on the physical color standard, and thus, supply the modified and improved image to a web page or a catalog printer. This is a partial solution in the process of finally improving the image presented to the buyer. As described above, this applies to cases such as more accurately displaying the color of an image (here, a teal shirt) in presentations on a website, a colored catalog, or the like.

[0015] Second, regardless of whether the seller modifies the captured image based on the physical color standard, the seller can capture and provide the physical color standard together with images of the shirt and other items. Therefore, in the image finally presented to the buyer, even if it is the display on the buyer's side (for example, the display of the item in the website displayed on the buyer's smartphone or the display of the item in the catalog captured by the buyer's camera), the physical color standard can be displayed or accessed. If the image for the physical color standard is captured at a time different from when the image was taken, potential errors may occur. Therefore, it is better to capture the physical color standard simultaneously within the image or under the same or similar conditions as when the image was captured, which enhances the function of correcting the captured image. (However, even in the former case, if it is done under the same or similar ambient light and using the same or almost the same image capture device, the color rendering can be improved according to the physical color standard.)

[0016] In this example, in this technique, using the image of the item and the physical color standard (such as the physical color standard within the same captured image), the image can be corrected to accurately show the true color of the item. Here, the true color includes at least the correct hue (such as red, green, blue, etc.). In this technique, through correction, accurate lightness, chroma (chrominance, vividness, saturation), and luminance can also be included in the true color. To do this, it is also possible to compare the physical color standard within the image with another physical color standard (such as the color standard accessible from the rendering device or a copy that accurately reproduces the physical color standard on a physical medium) by an application on the buyer's computer, etc. In this technique, for the color rendering of the color of the item (such as hue, chroma <vividness / saturation>, lightness, and / or luminance, etc.) and other attributes, changes are also made based on the correction for rendering to match or almost match the physical color standard of the buyer (such as the physical color standard assuming the buyer's own situation) with the physical color standard captured together with the item.

[0017] For example, when the physical color standard of an image is not correctly (accurately or similarly) depicted, the color of the image can be changed by an application on the device of the person on the production side (such as the buyer's smartphone) to match the physical color standard of the image. This is effective when correcting the color of an item by changing the entire image of the item and the color standard until the physical color standard presented on the buyer's smartphone matches the physical color standard in the image assuming the item is viewed at the location of the production side. If the captured physical color standard can be appropriately depicted according to the actual physical color standard, the depiction of the item will also be appropriate. This is because color changes can be made to both the item and the captured physical color standard at once (not essential but easier than performing them individually). Thus, assuming the red / green balance of the captured item and the physical color standard is poor or there is too little blue, the application can be used to restore the balance to match the physical color standard of a specific location until the captured physical color standard is appropriately depicted (e.g., until it looks the same to the viewer). When the captured physical color standard is appropriately depicted, the color of the item will also be (even more) appropriately depicted.

[0018] Furthermore, in this technique, based on information such as the type and age of the display unit and each setting, it is possible to check whether the correct color is displayed on the user's display unit. In this technique, it is also possible to then make changes to the displayed image based on the correct physical color standard and the color matching state on the buyer's device. By calibrating the display unit to match the standard after producing an image of the physical color standard (e.g., obtained using the camera of the device itself or received by receiving a provided one), such incorrect calibration of the buyer's device can also be corrected. By doing so, in this technique, when appropriate depiction of a scene or item is desired, it is possible to record how the settings of the display unit should be changed. Note that it should be noted that many devices are calibrated to intentionally interfere with the accurate or appropriate depiction of a scene, such as blue light reduction, dimming for night display, and changes to the display unit for power consumption savings.

[0019] More specifically, in this technique, an image of a certain color (here, some or all of the colors of the physical color standard) is taken, displayed to the user of the device, and a calibration sequence such as having the user calibrate to match the displayed color is also executed. To achieve a more accurate rendering of the image, this process can be performed several times with different colors and lighting (it is not necessary to perform a comparison of the physical color standard for each rendering of an item or scene). Adjustment according to the color of ambient light

[0020] The technique of the present disclosure also solves the problems of different ambient light (such as color and angle). For example, even if the entity on the shooting side accurately captures the image and the image on the display of another person is accurately displayed, this means that when the item is imaged by the entity on the shooting side (such as the seller's own eyes), the item appears the same as it does to a normal human eye. However, the rendering of the same item by the entity that performs the rendering later (such as the rendering on the buyer's display by the buyer) may not resemble the appearance of the item under the buyer's current ambient conditions.

[0021] Suppose a buyer is considering a certain product on a smartphone. In this example, assume that such a product is accurately rendered on the buyer's display. However, even if it is accurately rendered, it may be different from the situation where the buyer is considering the placement of the product. Returning to the example of the off-white chair below for explanation. The light when taking an image of the item was bright blue-white light (that is, light that is not natural light), or an image of the item was captured with the wavelength light of natural light, but assume that the buyer's house is illuminated by incandescent bulbs (pale yellow), fluorescent lights, or light-emitting diodes (LEDs). In either case, in the rendering of the buyer's smartphone, it is not possible to appropriately depict the item in the color assuming the actual ambient conditions of the location where the buyer is currently located or the location where the buyer wants to place the item (in this example, the off-white chair). In such a case, to appropriately depict the item, it is not the accuracy of the color of the item or scene at the time of shooting, but rather to depict based on the surrounding conditions.

[0022] In this technique, the image presented on the buyer's display unit can be appropriately depicted to show an appearance assuming the situation as seen from the buyer's location. As one of the methods for this, there is a method of detecting the surrounding conditions by a camera (such as a smartphone) (performing spectral analysis of the ambient light for color components such as the wavelength and intensity of light). This can be done by using the camera of a smartphone or other sensors to obtain the wavelength characteristics (wavelength, energy, etc.) and color characteristics of the current ambient light. In this technique, using such information, changes are made to the presentation of the product on the smartphone considering the difference between the current ambient light and the ambient light when the image was taken (if known), or at least the difference from the ambient light that is estimated or predicted to be the case when the image of the product was taken.

[0023] For example, if an image of a product is taken in an environment with strong blue light and thus the image is particularly saturated with blue, the technique can present the image of the product with the blue intensity weakened. Similarly, when the ambient light is from a fluorescent lamp (here, it is assumed that the color rendering index <CRT:Color Rendering Index> is relatively low), the technique can correct the spectral difference (such as a state where the red of an object in the image is too pale due to the light from some fluorescent lamps). Also, in the case of incandescent lighting, it is adjusted to reduce yellow from the image after presentation (assuming a correlated color temperature CCT of 2700K). It is possible to use the sensors of a smartphone or other devices, and it is also possible to use information regarding the image of the product (the known or assumable surrounding environment of the product in the original image taken and the image after subsequent changes), but the other methods described in this disclosure are also useful when making changes to the presentation according to the surrounding conditions.

[0024] However, when an image of an item is captured together with a physical color standard, if it is assumed that the entity on the production side (such as the buyer) has (or had) the physical color standard, the physical color standard (the one captured together with the original image of the item and the one the buyer has) can be adjusted to match the current production of the image of the item. By doing so, the item can be presented to the buyer in a state similar to the buyer's conditions (however, strictly speaking, such similarity may be regarded as having low accuracy in other production situations). For example, even if the item captured can be accurately represented by the production on the buyer's smartphone, it may not be what the buyer wants. The buyer may wish to confirm an image that assumes the case of seeing the product on-site, such as when actually seeing it or at least an image produced according to the surrounding conditions around oneself.

[0025] According to this technique, (assuming that the buyer's camera and / or display unit are correctly calibrated as described in the present disclosure,) such adjustment can be automatically performed based on the actual production of the physical color standard by the buyer's device. Alternatively, through the interface provided to the user by this technique, the user can view the item and the physical color standard captured together with the item along with other physical color standards for visual comparison with the captured physical color standard, and based on that (or automatically as described in the present disclosure), make changes to the image of the item being produced. Thus, the user interface can present the hue of red, yellow, and blue, as well as the chroma (saturation) and luminance (and other attributes of the color), so that the user can change each of them and adjust until the two types of standards match based on the perception of the buyer's eyes. (However, such attributes are only a small part of the several color elements that can be selected and changed by this technique.) Also, since the perception by the human eye may not always match the technical measured values regarding light in a completely linear manner, this technique also corrects for differences in the way each individual perceives color. Adjustment according to the surrounding luminance and / or light intensity

[0026] In this technique, in addition to correcting the hue of the ambient light due to the difference between the hue of the light when an image is captured or modified and the hue of the ambient light, the luminance of the periphery can also be corrected. Similarly, as described above, in this technique, an image of an item can be rendered to match the luminance of the periphery. The photography of many products on the market is carried out under extremely bright lighting conditions compared to the conditions under which the buyer intends to use the item. Therefore, the image, regardless of whether it has been accurately captured or rendered, is presented in a state that does not resemble the current luminance of the buyer. However, as described above, this technique also corrects the difference in luminance. Here, an example of a decorative item such as a cushion used on the buyer's existing sofa will be described. Such cushions are usually imaged using a white background in a high-luminance environment. Therefore, in this technique, the luminance (and other light indicators) of the rendered cushion image can be reduced to match the luminance of the ambient light. By doing so, the entity on the rendering side (here, the buyer) can appropriately depict the rendering of the item or scene. Augmented Reality

[0027] Continuing with the example of the cushion described above, assume that in a high-luminance environment where the hue of the ambient light has a slightly excessive blue tint, the cushion is placed on a sofa that does not match the buyer's brown sofa, and the original image of the cushion is photographed. According to conventional practice, if the buyer purchases the cushion Before making a decision, usually, it is required to: 1) trust that the image is accurately taken; 2) trust that the image is accurately rendered by the buyer's device; 3) "infer how it generally looks" in the actual situation while correcting the differences in ambient lighting (such as red, green, blue, etc.) in mind; and / or 4) "infer how it generally looks" in the actual situation while correcting the brightness in mind. Therefore, in addition to trusting that the image displayed in front of them is accurate, buyers must also have an idea of how the cushion looks when the blue color or brightness is reduced. Moreover, many buyers also want to see how it looks when installed in their own interior decoration (here, a brown sofa). Even if only one of the above difficulties occurs, it may pose a problem for the buyer. However, as seen in many cases, when the number of difficulties reaches two, three, four, or five, it will hinder a comfortable purchasing experience.

[0028] With this technique, one or all of these five problems can be solved, enabling a more accurate and / or more lifelike rendering, and by more appropriately depicting the item or scene, the experience of users who use catalogs, books, and websites can be improved. For example, it can enhance the buyer's experience and the confidence when making a decision on whether to purchase.

[0029] This can be achieved by using augmented reality with this technique. In this technique, in addition to, or instead of, one or more of the solution means of the present disclosure, an image of an item can be presented superimposed on the buyer's own situation. Many smartphones can render in real time or almost in real time (e.g., in real time on the display of the smartphone, etc.) a current image of the user or the user's location taken with the camera mounted thereon. An image of an item that has been previously subject to correction can be displayed superimposed on the rendered real space. The image related to this item can be corrected in advance as described above, or can also be corrected within the augmented reality interface. As one method for doing this, a method of using a physical color standard having some three-dimensionality can be mentioned (this method is useful when customizing the rendering of an item according to the lighting angle or selecting an image at an optimal angle from a plurality of images of the item taken at various lighting angles). Examples of physical color standards having three-dimensionality include a cube, a tetrahedron, a sphere, a hemisphere, etc.

[0030] Continuing the explanation with reference to the above example. Suppose a buyer is interested in salmon-colored cushions for autumn interior decoration. A buyer who has a brown sofa wants to see what it actually looks like when the cushion is placed on their brown sofa. By using augmented reality, the buyer can move or overlay the item of interest (in this example, the salmon-colored cushion when placed on the buyer's sofa) while viewing their surrounding environment and interior decoration displayed on a mobile device such as a tablet or smartphone. For this purpose, if the item has already been corrected and the augmented reality space of the state of a specific location is accurately produced, the buyer can generally grasp how the item will look. Subsequently, the buyer can correct the image presented on the display unit while comparing with their own eyes the sofa imaged on the display unit and its physical color standard, and the actual sofa seen with the naked eye from the buyer's location. However, many inaccurate or unconvincing points in the image of the item can be corrected using augmented reality technology. Suppose the buyer has a physical color standard and places it on their sofa. The buyer can adjust the color (hue, brightness, chroma <vividness / saturation>, etc.) of the displayed sofa so that the standard seen with the naked eye matches the standard displayed on the buyer's display unit by voice instructions, touch interface, or other means. Thus, the buyer's location can be accurately reproduced on the buyer's device. Furthermore, such a standard does not need to be an international standard or a consistent standard in order to perform the reproduction while viewing the same item. Thus, with this technique, the user can adjust the color and brightness through the user interface so that the image of the sofa to be reproduced matches the sofa seen by the buyer with the naked eye, and there may even be cases where it is not necessary to use a physical color standard.

[0031] For example, as described above, images can be overlaid to represent the current situation so that the item is appropriately depicted, or the image of the item can be modified to match it using physical color standards in the augmented reality space. Thus, the buyer can adjust the color, such as the hue, brightness, illumination angle, and chroma (saturation) of red, green, and blue, while viewing the image of the item and the physical color standard imaged together with the image, and can produce an image of the item in a state that more closely resembles the appearance of the item in the actual room, taking into account the hue, brightness, and angle of the actual illumination.

[0032] In a display unit incorporating an augmented reality space, a realistic salmon-colored cushion can be displayed together with the state of a specific location (such as a sofa or lighting). Therefore, the buyer can place the image of the cushion on top of the image of the sofa, or simply arrange or fix it in the center. Also, by moving the device and / or the camera, it is possible to display the state where the cushion is placed on the sofa in a desired orientation. With this augmented reality, in addition to the various advantages already described, the buyer can utilize the "walk-around" function, enabling a better grasp of the feel and suitability of the item. In this technique, the user can change the illumination angle of the item while "walking around" (assuming that multiple images of the item at different illumination angles are provided). Also, in this technique, the image of the item can be edited by adding shadows or highlights to the item to approximate the appearance at different illumination angles (when the buyer walks around in various directions such as left, right, and behind). As part of realizing this, in this technique, the difference in illumination angle when the user moves is detected, for example, with respect to the 3D physical color standard at the user's position.

[0033] Although an example of furniture is described here, it is also possible to appropriately depict clothing and other items. According to this technique, even without an objective color standard, a buyer can, in an augmented reality space or using a snapshot, for example, image the color of a person's arm and then match the produced arm image with the arm seen with the buyer's naked eye. By doing so, it is possible to match the production of the buyer's current situation (such as the arm and light). In this technique, moreover, the image of the item is made to look even more lifelike by any of the methods described in the present disclosure, and the said item is presented in accordance with the state of a specific location. As an example, there is a case of checking whether a scarf sold on the Internet matches well with a certain individual's favorite jacket, hair color, or skin tone. Another example is a case of checking whether the color of a shirt matches well with, interferes with, or enhances the buyer's skin tone, hair, or the color of cosmetics. Furthermore, the color of cosmetics can also be the item to be imaged. By producing the color of cosmetics in an even more lifelike state, it becomes possible to promote the purchase decision-making.

[0034] With this technique, the captured scenes and objects can be depicted more appropriately, thus improving the experience of users who view websites, catalogs, social media, etc. When the user is a buyer, it can lead to the promotion of the purchasing decision-making process. For example, consider using a small physical color standard with a certain degree of three-dimensionality. After imaging cosmetics such as lipsticks, foundations, or blushes, (regardless of the presence or absence of still images or augmented reality effects), adopt this technique to more accurately or realistically depict the color of the item and how it would look when applied to a particular individual's skin. For example, cosmetics retailers may provide a foldable small physical color standard for each purchase, or simply offer it for free in online stores or physical stores. And when a buyer wants to confirm how an item displayed in a catalog or online store would look on their own skin, they can fold the physical color standard into some three-dimensional shape and then use this technique to correct the color, brightness, and even lighting angle of the cosmetics, that is, make it even more realistic. The buyer can make changes while comparing the image of the physical color standard displayed with the cosmetics to the physical color standard in their hand, and thereby change the image to make it resemble the situation where the physical color standard in their hand exists. This enables the purchase process, which usually requires the person to visit a store directly, to be carried out remotely. This is particularly useful for both buyers and sellers as there are many buyers with restrictions on movement due to health concerns (such as the spread of COVID-19) or economic or ecological considerations (such as not wanting to drive to the store to protect the environment or save the buyer's expenses). Size adjustment

[0035] In addition to, or instead of, the above techniques, the size of an image of an item can be correctly set. This can be done according to some of the procedures described above, but can also be done by the following procedure. In this technique, first, an image (including the angle at which the image is presented) is corrected using information about the item such as height, width, and depth, and its dimensions are associated with the image. The sizes of the location and other conditions (such as the buyer's body and interior decoration) can be confirmed by directly inputting the dimensions or by using the function of a mobile device that can measure objects in the image. Examples of such measurement functions include the Apple (registered trademark) Measure app that can measure the dimensions of an item produced using augmented reality or a snapshot of a specific location's condition or object. In this technique, instead of, or in addition to, using the above procedure, the dimensions of the physical color standard can be utilized. Assuming that it is known whether the dimensions of the physical color standard in the image of the captured item are the same as or different from those of the physical color standard existing at the buyer's location, in this technique, the image of the item sold on the web page (or catalog by paper) can be enlarged or reduced, so that it can be displayed on the buyer's mobile device in the correct scale. (For example, a salmon-colored cushion can be displayed in an appropriate size according to the size of the sofa, thereby further promoting the buyer's decision-making.)

[0036] This can be particularly useful in the case of furniture, interior decoration, jewelry, ornaments, and clothing. (However, regarding clothing, it refers to the case where it is imaged while worn on a model or mannequin. This is because it doesn't make much sense to change the scale and display when the clothing item is placed flat or folded.) Suppose the buyer wants to check if a particular bracelet looks good when worn on the arm. In this technique, by improving the presentation of the product, the buyer's purchase decision can be promoted. The bracelet can be overlaid on a snapshot of the buyer's wrist or real-time augmented reality and displayed in a similar color, similar brightness, similar lighting angle, and accurate scale. This brings a great improvement for both the buyer and the seller in sales other than on websites such as catalogs. Environmental example

[0037] FIG. 1 illustrates an example of a system 100 that can implement techniques for performing more accurate or more lifelike rendering of an imaged item. The system 100 includes a computer device 102, which is illustrated by way of example as four types of mobile devices: a laptop computer 102-1, a tablet computer device 102-2, a smartphone 102-3, and an e-book reader 102-4. However, other computer devices and systems, such as a desktop computer or a netbook, may also be used.

[0038] The computer device 102 includes a computer processor 104, a computer-readable storage medium 106 (hereinafter, "medium 106"), a display unit 108 (s), and an input mechanism 110. The medium 106 includes computer-executable instructions that, when executed by the computer processor 104, perform operations such as an operating system 112 and an image module 114.

[0039] The image module 114 has functions that enable or assist the techniques described in this disclosure, such as improving the accuracy or lifelikeness of an image at, for example, the location where the image is taken (such as the seller described above in this disclosure) or the location where the image is ultimately rendered (such as the buyer). and so on.

[0040] In addition, the image module 114 may include, or have access to, a history 116, a user interface 118, and a three-dimensional module 120 (hereinafter, "3D module 120"). The user interface 118 enables the image module 114 to present on the user interface 118 on the display unit 108 a rendered image (e.g., the current location of the user rendered as an augmented reality including an item). Further, the user interface 118 enables a user of the computer device 102 to make changes to the rendered image via the input mechanism 110. The 3D module 120 enables the image module 114 to modify the image so as to show an item at different angles, different lighting angles, and / or different scales according to the situation represented in the augmented reality. The 3D module 120 can determine and change the scale of the item using the physical color standard in the image in which the item is displayed together with the physical color standard existing at the location of the device. Alternatively, the scale of the item can be appropriately adjusted by measuring the dimensions of the location by the 3D module 120 using a measurement sensor.

[0041] For example, as described below, the image module 114 can provide a user interface that receives a user selection for modifying an image that captures a physical color standard. The image module 114 receives the user's selection and modifies the image that captures the physical color standard and the image that captures the item or scene. In addition to, or instead of, this, when a location on the physical color standard of a color within the physical color standard that matches a part of an item or scene in the image is known (or can be identified), the image module 114 can also associate the part of the item or scene in the image with the matching color within the physical color standard. Also, by doing so, based on the color information recorded for a known location, the actual color of the part of the item can be recorded. This enables more accurate or more lifelike rendering. Further, changes for matching the color of the part rather to the recorded color information can be applied to all of the item, scene, or image, thereby improving the accuracy or lifelikeness of the entire item or scene.

[0042] The history 116 may include various data described in the present disclosure. For example, information regarding ambient light at the user's location, selections previously made by the user (such as a buyer or a seller), information regarding the current display unit (such as calibration data generated in accordance with the present disclosure), and further data from other information sources such as selections of other users and display unit data may also be included.

[0043] The computer device 102 includes, or has access to, one or more display units 108 and an input mechanism 110. FIG. 1 shows examples of four types of display units, all of which are configured integrally with respective devices (however, they do not necessarily have to be configured integrally). Examples of the input mechanism 110 include, but are not limited to, a touch sensor, a motion tracking sensor (such as one utilizing a camera), a mouse (either standalone or configured integrally with a keyboard), a track pad, a touch pad, a capacitance sensor (such as one installed on the surface of the computer device 102), and a microphone with voice recognition software, and may include gesture sensing sensors or devices. The input mechanism 110 may be configured separately from or integrally with the display unit 108. An example of an integral configuration is a gesture sensing display unit having an integral touch sensing or motion sensing function.

[0044] The imaging unit 122 may include a visible light sensor or an invisible light sensor such as a camera or an IR camera (and a projector). Also, the imaging unit 122 may be able to sense ambient conditions even if it cannot necessarily capture an image. Furthermore, the imaging unit 122 may function together with other members to provide the above-described augmented reality and other effects. Example of manual image modification

[0045] FIG. 2 is a diagram showing an example of a technique for a user to modify an image of an item. By this modification operation, the user's device can perform operations such as modifying the rendered image of the item according to the ambient conditions at a specific location, improving the resemblance, or simply correcting inaccurate parts or the rendering of the original image.

[0046] As shown in the figure, the image module 114 presents an image 202 of the item 204 to the user interface 208 of the user device 210 together with the captured physical color reference 206. The user interface 208 is configured to receive manual changes to the image through various control elements such as voice control, gesture control, and other hue ring controls or slider controls 212 (such as red, green, and blue hues, brightness, etc.). The manual changes in the present disclosure are made based on a visual comparison between the captured physical color reference 206 and the physical color reference 214 existing at a specific location.

[0047] As an example, refer to FIG. 3 showing the initial state of a set of slider control elements 212 and two consecutive user selections (see 212-1 and 212-2) below it. The two user selections are shown as a first selection 302 and a second selection 304. In the first user selection, the gray line 306 (initial state of the slider control element) on the blue slider bar 308 is lowered to reduce the blue hue. The gray line 310 shows the gray line in the state where the blue hue has been reduced. In the second user selection, the blue hue is further reduced, and that state is shown by the gray line 312 located below the gray line 310. In the initial rendering of the image 202, the blue-tinged salmon color with a pink tint fades to a salmon color with an orange tint as the blue decreases. Incidentally, the physical color reference 206 is changed simultaneously with the item (however, this may be done before changing the color of the item). In the original rendering of the physical color reference, the blue is further reduced from the blue reduction state shown by the line 314 to the blue reduction state shown by the line 316. Here, the blue reduction state reference 316 will coincide with the existing physical color reference 214. Therefore, as can be seen from the rendering 318 based on the improvement by the first selection and the rendering 320 based on the further improvement by the second selection, the closer the rendered reference is to the actual reference, the closer and / or more accurate the item 204 becomes to the state at the user's location.

[0048] Thus, the image module 114 receives manual changes via the user interface 208 and makes changes to the image 202 based on the received manual changes. After the change, the image can be rendered or saved for later use. Such changes can be executed in real time, with a delay, or simply provided to another interface (such as the augmented reality interface 216 in FIG. 10). Note that this user interface 208 may be configured as augmented reality (by presenting a color image indicating the user's location together with the imaged object and the physical color standard), or may be an interface that does not display the user's location. Example of color rendering method

[0049] Hereinafter, a method for improving the accuracy and resemblance of the current or future rendering of an object or a scene (for example, anything that can be imaged) by this technique will be described using the method 400 shown in FIG. 4 as an example. For each method described in this disclosure, the steps can be rearranged (for example, executed in an order different from the illustrated order), repeatedly executed, iterated, or combined, as long as there are no problems in the context. The illustrated order is not essential unless otherwise specified in this disclosure.

[0050] In step 402, a user interface is provided that receives a user selection for making changes to the image capturing the physical color standard. Examples of the user interface will be described with reference to FIGS. 2, 3 (partial), 5, 6, 9, and the description of the above technique. For example, in FIGS. 2, 3, and 9, the user interface 208 includes a slider-type control element 212 for changing the color of the image.

[0051] Also, in this method, as described above, a user interface can be provided that includes a plurality of selectable control elements presented in different expressions based on physical color criteria. This makes it easier for the user to select quickly or repeatedly. For example, consider the example of FIG. 5 showing five types of brightness that can be selected via the user interface 502 (an example of the user interface 118) of the smartphone 102-3.

[0052] In step 404, a user selection for changing the image that captures the physical color criteria is received via the user interface. Examples of selections via the user interface are as described above. For example, when the user manually makes a selection regarding the color change of the image that captures the physical color criteria through the operation of a control element, the user interface receives that user selection. As described above, the color change may refer to a change in the hue, chroma (vividness or saturation), lightness, or brightness of the image that captures the physical color criteria. Regarding the selection of manual color change, as described above with reference to FIG. 3, an example was used where the user makes a selection to lower the blue hue of the captured image.

[0053] Continuing to refer to the example of FIG. 5, in the user interface 502, for example, by simply tapping and selecting any one of the images of five physical color criteria with different brightness levels on the display unit, a significant (in the visible range) selection from brightness 504-1 to brightness 504-5 can be made. As described above, FIG. 5 also shows the physical color criteria 506 at the user's location. These operations can be repeated, re-executed, or executed with changes. In this example, as also shown in FIG. 5, with this technique, finer gradation options (for example, a fine brightness range 510 with five-step fine gradations of selectable brightnesses 510-1, 510-2, 510-3, 510-4, 510-5) and, for example, after the selection of brightness, different color change classifications can present options for different hues and different chromas (saturations), etc., at the selected brightness or a brightness close to it.

[0054] In operation 406, the modified image capturing the physical color reference is presented via a user interface or another user interface and based on the received user selection. Although many of these examples are described in this disclosure, for example, as shown in FIG. 3, further modification is made to the modified image 314 capturing the physical color reference to obtain an image 316 with less blue than image 314 or image 206. Continuing with the example of FIG. 5, selection of one of the luminance ranges (here 504-3) will result in the presentation of physical color references with slightly different gradations (indicated by the reference numerals 510-1, 510-2, 510-4, and 510-5) (along with a copy of the selected luminance 504-3). Although it is not necessary to adopt this method or use the above specific elements, there are also entities such as remote entities or the image module 114 that modify the image prior to presentation in operation 406.

[0055] In operation 408, the image of the item or scene captured at the same location as the location where the image of the physical color reference was captured is modified. Such modification can be made based on one or multiple selections received regarding the modification of the image of the physical color reference captured at the same location as the item or scene (as previously described with reference to FIGS. 3 and 5). Such modification of the image of the item or scene and the resulting more accurate or more lifelike rendering of the item or scene can be performed individually, simultaneously, or contemporaneously. In the example shown in FIG. 3, when the image module 114 incorporates the item or scene into the same image as the physical color reference, the image capturing the physical color reference and the image of the item or scene are modified in a single operation contemporaneously. Thus, as shown in FIG. 3, the physical color reference is modified and the item is also modified, and both are displayed on the user interface 208 (not shown in FIG. 3 for the sake of simplicity of the visual depiction).

[0056] In Project 410, the changed images of objects or scenes are recorded or displayed. In FIG. 3, the changed images of objects or scenes are indicated with the reference numeral 320. In addition to, or instead of, this, the details of the change can be recorded, thereby recording the color change due to the change in the physical color standard. The recording of such a change is useful when making future changes to future images to present the future images more accurately or more realistically. Thus, if the images of objects or scenes are transmitted to another viewer together with the recording, a more accurate rendering of the image or scene can be achieved based on the recording. Thereby, if the display unit finally used for rendering the objects or scenes has a similar setting or display method, it is easier to further improve the accuracy or realism of the rendering. As described in the present disclosure, further changes can be made.

[0057] If such a record or a combination of such records is made to indicate the changes to the images selected by the user for the display unit associated with the user, the record can be used later to automatically correct the images to be received or captured in the future. This is a form of automatically making changes based on the correction of the user's display unit, and also based on the calibration of the user's image 122 and the display unit 108 when the user is the image taker. This can be stored in the history 116 of FIG. 1. By doing so, in Method 400, changes can be automatically made to other images of other objects or scenes captured at the same location. This automatic change is made based on the difference between the image captured with the physical color standard and the changed image captured with the user selection and the physical color standard described in the present disclosure.

[0058] The physical color reference is shown as a 3D box with a large hue range, but other physical color references can also be used. For example, when the item or scene is cosmetics, the physical color reference can be a high-resolution range of human skin colors or a reduced physical dimension of such a reference. As another example, there can be a use of a physical color reference having a higher resolution than the illustrated physical color reference for human hair, skin color tones, brightness, and further chroma (saturation). Still further, when the item is a decorative item, another reference according to the environmental conditions of the home can be used. Also, when the scene is a photo of a person, including ranges of various colors of hair, skin color tones, and further clothing in the physical color reference enables a more accurate rendering. When the scene is a landscape image, the range of colors existing in an outdoor location may be represented by the physical color reference.

[0059] As described to some extent above with reference to FIG. 5, in this method, a plurality of selectable control elements consisting of physical color references presented in different representations can be provided. Generally, in step 412, different representations of the physical color reference are determined. This can be determined in various ways including the following steps. In step 414, a portion of the image capturing the physical color reference is associated with a matching color in a previously recorded image of the physical color reference or a copy of the physical color reference (provided that the position of the matching color within the previously recorded image of the physical color reference or the copy of the physical color reference is known). The matching color can be any of a matching hue, a matching chroma (vividness or saturation), a matching lightness, or a matching brightness. In step 416, based on the color information recorded for the known position, the actual color of the portion of the image capturing the physical color reference is determined. Further, in step 418, based on the actual color of the portion, different representations of the physical color reference (for example, a plurality of expressions providing color options in a finer range) are determined, enabling faster and more precise selection. Therefore, the image module 114 can present various options that are more likely to match and are more comprehensive.

[0060] For example, referring to FIG. 6, nine ranges of hue and luminance levels presented via the user interface 602 of the smartphone 102-3 (which is merely an example of the user interface 118) are shown. Here, since images of nine physical color standards each having a different color and luminance are presented on the user interface 602, any one of the nine color / luminance ranges 604 can be selected simply by performing a tap operation or the like on the display unit. As shown in the figure, red is stronger in the left column, blue is stronger in the right column, and the luminance increases toward the upper row. In these examples, different selectable expressions are determined by the alternative process 412, and as a result, nine different expressions are created based on the actual color determination, presenting options of colors close to or around the actual color using similar hue balances, luminance, and chroma (saturation; not shown).

[0061] As described above, the method described in the present disclosure can be implemented in a repetitive or iterative format. For example, by a user selecting different characteristics, or multiple types of characteristics (e.g., the luminance and hue grid shown in FIG. 6) on one interface, the image is changed until it matches the physical color standard (e.g., 506). As described above, the user can also, in addition to or instead of this, manually change the color via the interface (e.g., by instructing "increase the luminance and reduce the red" by voice input or using a slider-type control element 212 or the like).

[0062] Referring back to the operation of step 412, instead of or in addition to performing steps 414, 416, and 418, in step 412, different representations are determined by the previously performed calibration. This calibration can be based on a comparison that is clearly selectable by a human to calibrate a display unit that displays a user interface, or one or more changes previously made to the color of an image of a physical color standard can be made to the same or a similar display unit based on the record of the production. As an example, there is an example where the image module 114 captures an image of a physical color standard existing at the user's location by the imaging unit 122. Next, the user manually changes the production of the image to match it while looking at the actual physical color standard according to the steps of method 400. Therefore, such a change can be said to be a kind of display unit calibration. However, in such a change, since the inaccuracy of the imaging unit 122 may be taken into account, the calibration may not always be perfect. Example of a color rendering method by a color matching method

[0063] Hereinafter, a method for improving the accuracy and resemblance of the current or future production of an item or a scene by using a color matching method according to this technique will be described by taking method 700 shown in FIG. 7 as an example. For each method described in this disclosure, within a range where no problem occurs in the context, each operation can be rearranged (for example, executed in an order different from the illustrated order), repeatedly executed, iterated, or combined. The illustrated order is not essential unless otherwise specified in this disclosure. Note that since all or part of method 700 can be executed without the operation of the user, it can also be automatically executed by the image module 114 shown in FIG. 1 or the like. (However, this does not include the case where the operation of the user is explicitly described.)

[0064] In operation 702, an image of a physical color standard is received. The physical color standard includes known colors at known positions within a particular physical color standard. This physical color standard can be calibrated, known to be accurate such as the exact standard 800 of FIG. 8 described below, or data representing the standard in a form usable by a machine. FIG. 8 shows known colors 802 and positions 804 on the exact standard 800. It is not necessary to display this to the user; rather, it can be made non - visible and / or executed only by a computer process (such as an image module 114 that provides instructions to cause the processor 104 to execute the operations of method 700) that uses a data file or image including the known positions and colors. The position can be in Cartesian coordinates or the like.

[0065] The known color 802 has at least one known hue and may also have other known color attributes. The position 804 is also known and is associated with the known color 802. Although not essential, the known color 802 existing at the known position 804 within the standard 800 can be determined by calibration based on a human - selectable comparison for the purpose of calibrating a display presenting a user interface. In operation 704, an image of an item or a scene is received. The image of the item or scene and the image of the physical color standard are the same image, separate images captured contemporaneously, or separate images captured in the same location context such as the same location or the same ambient conditions.

[0066]

[0067] ​In step 706, a portion of an item or scene displayed in an image of the item or scene is associated with a matching color within the physical color standard (provided that the position of the matching color on the physical color standard is known). The physical color standard is a physical color standard for an item or scene captured at the same or a similar location, such as within the same image as the item or scene. The association of a portion of the item may include sampling a plurality of portions of the image of the item or scene together with known colors of the physical color standard, or vice versa, and when a match is found based on the sampling, the corresponding portion becomes the subject of the association. Further, this association can be performed so as to match the hue, chroma (vividness / saturation), lightness, or luminance of the portion.

[0068] Hereinafter, as an example, an explanation will be given based on FIG. 8 showing an image 202 of an item 204 displayed together with a physical color standard 206 in the same image 202 in FIG. 2. In step 706, the image module 114 associates a portion 806 of the item 204 (i.e., a portion having a color that matches a color within a portion 808 of the image of the physical color standard 206). Note that the color rendered in the image of the physical color standard 206 may or may not be correctly rendered, but if neither the item 204 nor the image of the physical color standard 206 is correctly rendered in the same way, it is also possible to correct the color of the item.

[0069] In step 708, based on the color information recorded for a known position, the actual color of the portion of the item or scene is determined. Here, the position of the portion 808 is the position 810 within the image of the physical color standard 206, and this position 810 is mapped to the position 804 of an accurate reference 800 that is associated with the known color 802. The known color 802 is the color that is actually desired to be used for rendering the portion 806 of the item 204.

[0070] In step 710, the recorded color information and the position of the corresponding part of the item or scene are recorded. By recording in this way, it is effective in correctly representing accurate or similar colors in the future rendering of the image of the item or scene. In the example of the figure, as such color information, a known color 802 and a part 806 of the item 204 are recorded.

[0071] In step 712, according to this technique, an image of an item or scene having accurate or similar colors can be rendered based on the recorded color information and the position of the corresponding part of the item or scene. Although not essential, when an item or scene having such accurate or similar colors is rendered, the user can check its accuracy and resemblance and make changes according to the various methods described above (for example, step 404 of method 400). Example of a color rendering method using the recorded color information

[0072] Hereinafter, a method for improving the accuracy and resemblance of rendering of an item or scene by this technique will be described by taking method 900 shown in FIG. 9 as an example. For each method described in the present disclosure, within the range where no problem occurs in the context, each operation can be rearranged (for example, executed in an order different from the illustrated order), executed repeatedly, iterated, or combined. The illustrated order is not essential unless otherwise specified in the present disclosure. Note that since method 900 can be executed without any operation by the user for all or part of it, it can also be automatically executed by the image module 114 shown in FIG. 1, etc. (However, this does not include the case where the user's operation is explicitly described.)

[0073] In operation 902, an image of an item or scene is received. Although not required, it is also possible to configure the recorded color information and the position of a portion of the item or scene to be generated as a result of the operations of method 700. In such a case, it is also possible to receive the image from a remote device that has executed method 700, or (for example, when method 700 is executed on the same user device such as the same computer device 102) to receive the image from the same device that is executing method 900.

[0074] In operation 904, the recorded color information and the position of a portion of the item or scene are received. The recorded color information and the position of the portion of the item or scene indicate the exact color or a color similar thereto of the portion located at the position of the item or scene. The recorded color information and position indicate the color of the portion of the item or scene with the exact color at that position rather than a color similar to the user's ambient conditions, and further involves receiving, via a user interface, a selection made by the user to further modify the rendering of the modified image of the item or scene. In such a case, the further modification may be based on the physical color criteria at a specific location of the point visible to the user and is effective to improve the resemblance of the modified image of the item or scene to the ambient lighting conditions at that point.

[0075] In operation 906, a difference between the color of a portion of the item or scene in the image and the recorded color information is determined.

[0076] In operation 908, changes are made to an image of an item or a scene based on a recorded color or a color difference. Note that the changes made to the color of this part so as to match the recorded color can be made only to the relevant part of the item or the scene, or to the entire item or scene, or to any intermediate part thereof. Thus, changes such as reducing the red hue of a part of an item and increasing the chroma (saturation) so as to match the recorded color can be applied to parts other than the relevant part. Thereby, changes can be made without performing a detailed analysis of each part of the item or the scene. In the present disclosure, the term “alter” means various changes made to an image, for example, creation of a new image, changes to an image file, and further use of the altered image. However, the language “alter” should not be construed as necessarily meaning that the same image or data file must be used. Rather, a new image or data file can be created, and in that case, the altered image becomes a new image.

[0077] In operation 910, the altered image of the item or the scene is recorded or presented. As described above, the altered image can be presented on the display unit 108 using the user interface 118 of the computer device 102. However, the user interface can be an augmented reality interface that can be generated by the 3D module 120.

[0078] As somewhat described above, in this technique, the display unit can be calibrated to more accurately depict an image. In such a case, further changes can be automatically made to the image based on the calibration. Alternatively, the user can manually make a selection regarding the calibration of the display unit that presents the altered image so as to receive the changes. This further change improves the resemblance or accuracy when presenting the altered image of the item or the scene on the display unit. Examples of Augmented Reality

[0079] FIG. 10 is a diagram showing an example of an augmented reality interface. Here, via the augmented reality interface 216, an item 204, an imaged physical color standard 206 (optionally, which may have been previously corrected as shown in FIG. 2 for example), and a physically existing location 220 can present an image 218 of the location where the location has been imaged. The image 202 of the item 204 and the physical color standard 206 are displayed superimposed on the image 218 of the location. This example also includes a slider-type control element 212, and the user can operate this slider-type control element 212 to change the brightness and color of the image 218 of the location (and, separately from that, the image 204 of the item that may have been previously corrected by this technique). As described above, the physical color standard 222 of a specific location can be imaged as an augmented reality image 224 in the augmented reality interface 216. While the user perceives the physical color standard 222 of the specific location with the naked eye, changes can be made to the augmented reality image 224 until it matches that state. By doing so, the imaged item 204 and the imaged location 220 (i.e., the image 218 of the location) can be made to resemble the user's location 220. Thus, the user can visually confirm whether a salmon-colored cushion product matches their interior decoration (here, the sofa and table provided at the physically existing location 220). Therefore, by this technique, it becomes possible to correct inaccurate points and unappealing points by changing the image of the product and the image of the location where the product is intended to be used. Alternatively, instead, the user can make changes to the image 202 and the item 204, and the user's display unit may also be placed on the sofa (not shown). However, in this case, it becomes easier to perform similar coloring, but (since most cushions are larger than the display units of smartphones and tablets), it may become difficult to adjust the item 204 to the correct size. Another example of the seller and the buyer

[0080] Hereinafter, taking the method 1100 shown in FIG. 11 as an example, a method for enabling a seller to more accurately produce an image of an item such as an item for sale by this technique will be described. In each method described in the present disclosure, each procedure can be combined, rearranged (for example, executed in an order different from the illustrated order), repeatedly executed, iterated, or combined. The illustrated order is not essential unless otherwise specified in the present disclosure.

[0081] In step 1102, an image of the item is captured together with the physical color standard. This can be done by any of the above methods, such as the user capturing it using the imaging unit 122.

[0082] In step 1104, an application (for example, the image module 114, etc.) receives the captured image of the item and the physical color standard.

[0083] In step 1106, the application compares the physical color standard captured in the captured image with an image regarding the recorded physical color standard or data that can be used for accuracy determination of the captured physical color standard.

[0084] In step 1108, the application makes changes to the image of the item and the physical color standard based on the comparison. Such changes to the image are effective in improving the accuracy of the image of the item. Note that such improvement in accuracy may involve correcting errors by the camera during production by improving the accuracy according to the color of the ambient light when the image was captured, and / or correcting the image according to the color of another ambient light, such as an image of the recorded physical color standard captured under favorable ambient illumination (for example, white light based on sunlight).

[0085] Alternatively, or in addition, in step 1110, the application may present an item to the user interface, thereby enabling the seller to adjust the image to match what is visible to the naked eye while visually inspecting it. (For example, it becomes possible to adjust the captured image to match the physically existing item, or to adjust the captured physical color standard to match the physically existing color standard.) In step 1112, the application receives manual changes to the image (e.g., based on the user's visual comparison of the physical color standard within the image with the physical color standard actually present at a specific location). By doing so, the application (such as Image Module 114) modifies the image of the item in response to the user's input and records the modified image in step 1114. This can be performed after, before, or in conjunction with other steps in the method.

[0086] In this technique, alternatively, or in addition, in step 1116, assistance can also be provided when the seller selects a plurality of modified physical color standards for the purpose of correcting the image of the item. As will be described later with respect to the examples in FIGS. 5, 6, and 12, the seller can more easily select an image of a physical color standard that is more suitable by this technique, and thus a more accurate image of the related item. Each physical color standard to be modified is determined in step 1116, presented in step 1118, and the selection is received in step 1120. The methods for determining these images, selecting them, and the iteration / arrangement methods of each step of the method will be described in the description regarding FIG. 12 with reference to FIGS. 5 and 6 as well. For example, if the past change history of the system settings or personal settings of the seller's own device or camera is known, by this technique, it becomes easier to modify and present the image so that the physical color standard appears accurate and close to the naked-eye vision, or to present a more significant standard and repeatedly make selections to enable an accurate determination of the modification.

[0087] Returning to step 1114 for explanation, the changed image or the input made to change the image is provided or recorded in the display unit as described above with respect to the user interface 208 shown in FIG. 2 (and FIG. 3), the augmented reality interface shown in FIG. 10, and the user interface 502 shown in FIG. 5. Method Example (Buyer Side)

[0088] Hereinafter, a method by which a buyer can more accurately or more realistically produce an image in an electronic form (such as an email, a text message, a social medium, a web page, etc.) or a printed state of an item or a scene such as an item for sale will be described using the method 1200 shown in FIG. 12 as an example. In each method described in the present disclosure, each procedure can be combined, rearranged (for example, executed in an order different from the illustrated order), repeatedly executed, iterated, or combined. The illustrated order is not essential unless otherwise specified in the present disclosure.

[0089] FIG. 12 shows a method example 1200 that simplifies a technique for making an image in an electronic form (such as an email, a text message, a social medium, a web page, etc.) or a printed state of an item to be sold more accurate or more realistic, and the following will be described with reference thereto.

[0090] In step 1202, the captured image is received by an application. This captured image may be improved or unimproved as described for the seller's method. In this method (i.e., other than some of the other methods described in the present disclosure), an image of an item is captured together with a physical color standard. In the case of an image included in a printed material such as a catalog or an advertisement, the buyer may use his or her device to capture an image of the item in the printed state and the physical color standard (for example, take a photo of it) and receive the image.

[0091] Also, in step 1204, it is optionally possible to have the application compare the physical color standard imaged within the captured image with a physical color standard of the same type as the physical color standard imaged together with the item (for example, its copy) imaged at a specific location. By comparing the captured image with an image regarding the physical color standard imaged at a specific location (for example, by the image module 114), the difference between the two can be determined according to the ambient light when the image captured at the specific location was taken, and then the image of the item and the physical color standard can be automatically adjusted. As a result, the accuracy and / or resemblance of colors such as hue, luminance, lightness, and chroma (saturation) regarding the item are improved. Examples include lowering the luminance to match the luminance of the slightly darker surroundings, or changing the hue of red, green, or blue to make it closer to or further from the current state. In this technique, it is also possible to use historical data in this step to improve the image of the item described in the present disclosure.

[0092] In step 1206, the application makes changes to the image of the item based on the comparison. In step 1208, the image of the item is provided to the buyer's device or the like. If there are defects such as the inability to perform accurate rendering on the display unit of the device, it is also possible to correct such defects, or the user can compare the modified image rendered on the user interface provided by the user with the physical color standard with the naked eye and manually make further changes to the image of the item in steps 1212 and 1214 (or as part of steps 1206 and 1208).

[0093] Incidentally, this method can also be executed iteratively or repeatedly in real time when changes occur in the ambient conditions or when the buyer's perspective on the physical color standard changes.

[0094] Also, in some cases, it can be configured to perform spectral analysis of light by the seller's application when the image is captured. Further, such a configuration may be applied to the buyer's application (which may be the same or different from the seller's application) to be useful when correcting the image of the item.

[0095] Also, in this technique, in some cases, for example, when making changes to an image before the buyer arbitrarily makes some adjustments to the image related to the item, information regarding the type, age, and defects of the buyer's display unit can be utilized. Further, when using the image module 114, before the implementation of the method, etc., after a viewer (such as a buyer) calibrates the display unit based on the image of the physical color standard (the image acquired or taken by the buyer) by this technique, the user interface can be used to change the calibration of the display unit while visually observing the physical color standard at a specific location. As a result, the display can be generally improved or calibration data can be provided, so that when the buyer receives an image (such as an image of the product for sale) imported for color selection, etc., in this technique, by using such calibration (for example, automatically in steps 1216 and 1218, or by other methods described in this disclosure), the presentation of the product can be improved. This procedure can also be executed together with or separately from other procedures. As an example, a method of first executing this procedure and then asking the buyer to make further improvements or changes to the image via the user interface can be mentioned.

[0096] As an alternative or in addition thereto, in step 1216, a plurality of modified physical color criteria are determined. These plurality of physical color criteria can be determined in various ways as described above. For example, device settings (e.g., in some display units, as a known feature of the device, settings for "nighttime" color rendering of yellow by reducing blue for the purpose of preventing eye fatigue, or user personal settings regarding the hue of the device, etc.), user history (e.g., historical information such as the display unit of the device being set to produce colors with unbalanced chrominance, hue, luminance, etc. according to the user's prior selection), history related to the seller (e.g., the seller is known for imaging the item under specific ambient light), and further, the way of using the current ambient conditions detected by sensors (e.g., dark room, fluorescent lamp, LED lighting, etc.). Based on these, to some extent, a hue range (or other characteristics) is determined in step 1216. These can be the final or initial steps when the buyer selects, among the physical color criteria imaged by the device, those that are closest to the buyer's naked-eye vision of the physical color criteria at that location. However, in some cases, it may be a wide range with little or no analysis, but rather provide a wide range of options that can be selected by the buyer, and (when the buyer wants to further improve the granularity, accuracy, and / or resemblance) narrow down based on previous selections. It is also possible to do so.

[0097] In this technique, in step 1218, some of the renderings of the determined modified physical color criteria are presented as options. For example, consider the example of FIG. 5 showing five levels of luminance that can be selected via the user interface 502 of the smartphone 102-3 (an example of the user interface 118). Here, using the user interface 502, for example, by simply tapping and selecting on the display unit any one of the images of five physical color criteria with different luminances respectively, a wide selection (in the naked-eye region) from luminance 504-1 to luminance 504-5 can be made.

[0098] In this technique, in step 1220, one rendering selected from among the renderings of the determined modified physical color standard is received. In this example, via the user interface 502, selections of any of the images with various brightnesses 504 are continuously received. In FIG. 5, it is assumed that the buyer selects brightness 504-3 as the one closest to it based on the naked-eye observation of the physical color standard 506 at the buyer's location 508. At this point, with a more refined gradation selection (for example, a fine brightness range 510 where brightness can be selected from a more refined five-step gradation) or different selections, steps 1216, 1218, 1220, and 1208 can be repeatedly executed one or more times. Therefore, for example, in this technique, after the brightness is selected, options with different colorings such as different hue balances (or other characteristics) can be presented at the selected brightness or a brightness close to it.

[0099] For example, looking at FIG. 6 again, nine different hue / brightness ranges presented via the user interface 602 of the smartphone 102-3 (which is just an example of the user interface 118) are shown. Here, images of nine physical color standards with different hues and brightnesses are presented on the user interface 602, so that any of the nine hue / brightness ranges 604 can be selected simply by performing a tap operation or the like on the display unit. As shown in the figure, red is stronger in the left column, blue is stronger in the right column, and the brightness increases as going up the upper row.

[0100] The buyer can continue this operation until satisfied with the matching state with the physical color standard (for example, 214, 222, 506) by selecting different characteristics or multiple types of characteristics (for example, the brightness and hue grid shown in FIG. 6) on one interface. Instead, the buyer can also manually change the characteristics through the interface (for example, by instructing "increase the brightness and reduce the red" by voice input or using a slider-type control element 212, etc.).

[0101] Referring back to the method of FIG. 12, in this technique, optionally in step 1210, the buyer may record any input and / or changes to the hue and brightness made by the buyer on the buyer's device. This helps the application to further improve the performance it will play for future items. Thus, when the buyer makes changes to the image, the application can learn what to change in the future. For example, if the buyer makes a choice to increase the red in the image and correlates this increase in red with the data regarding the image of the item and the data regarding the surrounding conditions, such a change can be made to be learned by the application. By continuously recording the changes made to the image by the buyer in this way, the application can be made to learn a more appropriate way to change when rendering further images on the buyer's display. This same data can be provided to other instances of the application or received from other instances and used to improve the automatic correction by the application. For example, assume that another buyer changes the blue hue of an image of an item (the same or a different item) by a specific amount under the condition that the ambient light of the other buyer's device has specific characteristics. At this time, if the ambient light of the other buyer's device is similar (or the contrast is similar to the ambient light when the original image was taken), such a change in hue is recorded and it is possible to automatically change the image by delivering it to the application of the buyer and / or using it by the application. However, this data can also be optionally shared according to the buyer's choice. Disclosure of additional information

[0102] ​Due to the functions provided by this technique, it becomes possible to capture the image of a scene together with an established physical color standard (since the human eye can distinguish approximately ten million different colors, for example, a color wheel, color bar, or 3D physical color standard indicating dozens to millions of colors). Therefore, an image can be generated in colors that truly represent the real object. For example, in the case of an image provider such as a seller who sells products online or through a catalog, by using the image module 114, it is possible to adjust the image of the product to match the true or similar color tone, brightness, and other factors of the product.

[0103] For this purpose, the image module 114 calculates the difference between the value of the physical color standard of the captured image and the value of the correct physical color standard stored in or accessible by the computer device 102. For example, assume that a scene including a desired product and a physical color standard is captured. In that case, the physical color standard can be identified by analyzing the image according to this technique. The color of the physical color standard appearing in the image may vary depending on the surrounding factors at the time of image capture. For example, the lighting of the scene may be dim, or the auto white balance (AWB) function of the computer device may be distorted. The color of the physical color standard appearing in the image can be compared with the color of the established physical color standard stored in or accessible by an application (such as the image module 114) configured to execute this technique. Through this comparison, the difference in hue values is revealed. As an alternative or in addition, such a difference is recognized or determined by an application for rendering that includes information about the color of each pixel rendered in the image, or even by rendering hardware such as an image processing device (GPU). By comparing the information about the color of the pixels in the corrected image of the physical color standard with the pixels that render the image of the physical color standard captured, the difference in color characteristics at the rendering level is determined. These pixels can be mutually associated with the respective parts of these standards, for example, by mapping in Cartesian coordinates.

[0104] For example, the physical color standard in an image may include a red hue value of R231 at a specific location (or pixel), while in the physical color standard established (or corrected) in the application, the corresponding red hue at that location may be specified as R220. Thus, in the application, a difference of -11 is produced in the true-color red hue in the image (and thus in the item or scene). Also, in the application, until a representative image is generated, a difference in hue is produced for all physical color standard values. Also, the true-color hue bar and luminosity can be displayed in real time, and the user can adjust the hue using these bars. Further, spectral analysis of light can be performed (using a light sensor, etc.) to confirm that the hue, brightness, etc. are correct or similar.

[0105] Due to the functions provided by this technique to the prospective customer side, the user can visualize a more accurate representation or a representation similar to the color of the item. In this technique, after reading out the indicators of the display unit of the device (such as the type of screen, age, settings, etc.), the physical color standard provided by the seller can be analyzed. By comparing the indicators of the display unit of the device with the physical color standard provided by the seller, the color of the image can be accurately represented. For example, in the image module 114, it can be determined whether the blue hue of the device is displayed with a lower brightness than the red and green hues. Thus, in an application configured to execute this technique, the calculation and display of the image can be performed taking into account the settings of the display unit of the device and the physical color standard provided by the seller. Alternatively, by the application, for example, after taking and displaying a color photo, the user can be made to adjust and match the displayed color, etc. The color of the screen can also be determined by a calibration sequence. The calibration sequence can be repeated until a representative color is depicted. This calibration can be used to modify the image by the method described in the present disclosure. Further, when the application is used on a wireless communication device (such as a smartphone, tablet, etc.), the user can thereby capture an image of the screen on which an item (such as a product for sale) is displayed. Also, by using the application on a wireless communication device (such as a smartphone, tablet, etc.), based on the physical color standard provided with the image and the physical color standard established in the application (or based on the physical color standard by the buyer's visual observation instead of or after the initial correction by the application), the captured image can be modified.

[0106] Color has many characteristics. To name just a few, there are attributes such as hue, chroma (vividness and saturation), lightness, and luminance. Hue is also said to be "a parameter of color appearance", and can be defined based on the degree of color stimulus, or a single numerical value related to a chromaticity diagram (chromaticity coordinates) or a color wheel (see Figure 2). Alternatively, hue can also be defined based on the dominant wavelength or the wavelength of its complementary color. Chroma (vividness and saturation) is an attribute related to the intensity of the perceived color. These are formally defined by the International Commission on Illumination, but such a formal definition is not essential. More specifically, the vividness of a color can depend on both spectral reflectance and illumination intensity. Saturation indicates the vividness of the color in a given area judged in proportion to luminance. Luminance is also an attribute of color and is related to the visual perception of emitted or reflected light. Thus, an object with low luminance is not very bright. Lightness, sometimes called "color value" or "color tone", refers to the manifestation of luminance. There are various lightness models in the art, such as the Munsell, HSL, or HSV color models. As is clear to those skilled in the field of color science, the characteristics of color can sometimes overlap and are often based on the user's perception, so differences can occur among users. However, this difference can be addressed by the techniques described in this disclosure, and images of similar objects or scenes can be created for a specific user (even if they do not seem similar to others).

[0107] Also, as described above and in more detail in this section, in this technique, the characteristics of color can be used singly or in combination to change and improve the rendering of color images.

Examples

[0108] Various examples will be described below. Example 1: Providing a user interface that receives a user selection for changing an image that captures the physical color standard, Receiving, via the user interface, a user selection for changing an image that captures the physical color standard, Presenting a modified image that captures the physical color standard based on the received user selection via the user interface or another user interface; Modifying an image of an object or scene captured in the same context as the image capturing location of the image that captures the physical color standard based on the received selection or the modification to the image that captures the physical color standard; Presenting or recording the modified image of the object or scene; A method comprising the steps of:

[0109] Example 2: Presenting an image of the modified object together with the physical color standard by a user interface configured to receive a manual modification to the image made based on a visual comparison of the physical color standard existing at a specific location and the captured physical color standard; Receiving a manual modification via the user interface; Further comprising making a further modification to the image of the modified object based on the received manual modification, The method of Example 1, wherein the recording of the modified image is to record the modified image together with the manual modification.

[0110] Example 3: One or more computer-readable media that, when executed by one or more processors, include instructions for performing the steps of the method of Example 1 or Example 2 accordingly.

[0111] Example 4: A display unit, One or more computer processors, One or more computer-readable media that, when executed by the one or more processors, include instructions for performing the steps of the method of Example 1 or Example 2 accordingly, and further rendering the modified image on the display unit. A mobile computer device comprising:

[0112] Example 5: A mobile computer device comprising means for performing the steps of Example 1, Example 2, or Example 3.

[0113] Example 6: A method for improving the accuracy or resemblance of an image showing an item, receiving an image of the item together with a physical color standard, comparing the image of the physical color standard with a copy of the physical color standard imaged at a specific location, modifying the image of the item based on a comparison between the image of the physical color standard and the copy of the physical color standard imaged at the specific location, and providing the modified image for rendering. A method including these steps.

[0114] Example 7: A method for improving the accuracy or resemblance of an image showing an item, receiving an image of the item together with a physical color standard, presenting the image of the item together with the physical color standard by a user interface configured to receive manual modifications to the image made based on a visual comparison between the physical color standard existing at a specific location and the photographed physical color standard, receiving manual modifications via the user interface, and providing the modified image for storage or rendering. A method including these steps.

[0115] Example 8: One or more computer-readable media including instructions that, when executed by one or more processors, perform the steps of the method of Example 6 or Example 7 accordingly.

[0116] Example 9: a display unit, one or more computer processors, one or more computer-readable media including instructions that, when executed by the one or more processors, perform the steps of the method of Example 6 or Example 7 accordingly and further include rendering the modified image on the display unit. A mobile computer device including these components.

[0117] Example 10: A mobile computer device comprising means for performing the steps of Example 6, Example 7 or Example 8.

[0118] Example 11: Providing a user interface that receives a user selection for changing an image captured with a physical color reference, Receiving, via the user interface, a user selection for changing an image captured with a physical color reference, Presenting, via the user interface or another user interface, a modified image captured with the physical color reference based on the received user selection, Changing an image of an item or scene captured in the same context as the image captured with the physical color reference based on the received selection or the change to the image captured with the physical color reference, Presenting or recording the modified image of the item or scene, comprising a method.

[0119] Example 12: The method of Example 11 or any other preceding example, characterized in that providing the user interface provides a plurality of selectable control elements, each of the plurality of selectable control elements being presented in a different representation of the physical color reference, and receiving the user selection includes selecting any of the control elements.

[0120] Example 13: The method of Example 11 or any other preceding example, further comprising determining various representations of the physical color reference, the determination including: Associating a portion of an image captured with the physical color reference with a matching color in a previously recorded image of the physical color reference or a copy of the physical color reference (a color whose position in the previously recorded image of the physical color reference or the copy of the physical color reference is known), Determining the actual color of the portion of the image captured with the physical color reference based on the color information recorded for the known position, Determining different representations of the physical color reference based on the actual color of the portion, characterized in that the method includes this.

[0121] Example 14: The method of Example 11 or other examples preceding it, characterized in that the matching color is a matching hue, chroma (vividness / saturation), lightness or luminance.

[0122] Example 15: The method of Example 14, characterized in that an image of a previously recorded physical color standard or a copy of the physical color standard is determined by calibration based on a human-selectable comparison for the purpose of calibrating a display unit presenting the user interface.

[0123] Example 16: The method of Example 11 or other examples preceding it, further comprising modifying the image capturing the physical color standard based on the received selection before presenting the modified image capturing the physical color standard.

[0124] Example 17: The method of Example 16, characterized in that the image capturing the physical color standard and the image of an item or scene are present within the same captured image, and the modification to the image capturing the physical color standard and the modification to the image of the item or scene are performed in the same process at the same time.

[0125] Example 18: When the user manually makes a selection regarding the color change of the image capturing the physical color standard through the operation of a control element, receiving the user selection is made possible by the user interface, and receiving the user selection for modifying the image capturing the physical color standard includes changing the hue, chroma (vividness / saturation), lightness or luminance of the image capturing the physical color standard. The method of Example 11 or other examples preceding it.

[0126] Example 19: Further comprising recording the color change due to the change of the physical color standard, and the recording of the change has an effect on future modifications to future images for presenting the future images more accurately or more realistically. The method of Example 11 or other examples preceding it.

[0127] Example 20: Further including automatically making changes to another image of another item or scene, or another image of another item or scene captured in the same location context, and the automatic change being based on the difference between an image capturing the physical color standard or the user's selection and the state of the image of the item or scene after the change that captures the physical color standard, which is a method of Example 11 or other previous examples characterized thereby.

[0128] Example 21: The item or scene is cosmetics, and its physical color standard includes the color tone range of human skin, or the item or scene is a clothing product, and its physical color standard includes the color tone range of human hair and skin, or the item or scene is a landscape, and its physical color standard includes the range of colors seen in an outdoor location, which is a method of Example 11 or other previous examples characterized thereby.

[0129] Example 22: Receiving an image of the physical color standard including a known color at a known position within the physical color standard, Receiving an image of an item or scene, characterized in that the image of the item or scene and the image of the physical color standard are the same image, or separate images captured simultaneously, or separate images captured in the same location context, Associating a part of the item or scene shown in the image of the item or scene with a matching color within the physical color standard (a color whose position on the physical color standard is known) mutually, Determining the actual color of the part of the item or scene based on the color information recorded for the known position, A method including recording the color information and the position of the part of the item or scene, and using such recording to be effective in accurately or realistically representing the correct color in future presentations of the image of the item or scene.

[0130] Example 23: The mutual association of the items includes sampling a plurality of parts of the image of the item or scene together with the known colors of the physical color standard, and when a matching one is found based on the sampling, the corresponding part becomes the object of mutual association, which is a method of Example 22 characterized thereby.

[0131] Example 24: The method of Example 22 or other preceding examples, characterized in that it is automatically executed without interactive processing with the user.

[0132] Example 25: The method of Example 22 or other preceding examples, further comprising producing an image of an item or scene having an accurate or similar color based on the recorded color information and the position of the part of the item or scene.

[0133] Example 26: The method of Example 22 or other preceding examples, characterized in that the correlation between the part of the item or scene shown in the image of the item or scene and the matching color within the physical color standard is to match the hue, chroma (vividness / saturation), lightness or luminance.

[0134] Example 27: The method of Example 22 or other preceding examples, characterized in that the known color existing at a known position within the physical color standard is determined by calibration based on a human-selectable comparison for the purpose of calibrating a display unit presenting a user interface.

[0135] Example 28: The method of Example 22 or other preceding examples, characterized in that the item or scene is a cosmetic, and its physical color standard includes the color range of human skin, or the item or scene is a clothing product, and its physical color standard includes the color range of human hair and skin, or the item or scene is a landscape, and its physical color standard includes the range of colors seen in outdoor locations.

[0136] Example 29: Receiving an image of an item or scene, Receiving the recorded color information and the position of a part of the item or scene, wherein the recorded color information and the position of a part on the item or scene indicate the accurate color or a similar color of the part at the position of the item or scene, Determining the color difference between the color of the part of the item or scene in the image of the item or scene and the color of the recorded color information, Modifying the image of the item or scene based on the determined color difference A method including recording or rendering a modified image of the item or scene.

[0137] Example 30: The method of Example 29, wherein the recording or rendering of the modified image of the item or scene is a rendering in an augmented reality interface of the modified image.

[0138] Example 31: Characterized in that the exact color of the part of the item or scene at the position is shown by the recorded color information at the position, and further including receiving, via a user interface, a user's selection to make a further change effective in improving the resemblance of the modified image of the item or scene to the ambient lighting conditions at the location based on the physical color criteria at a specific location of the point visible to the user himself for the rendering of the modified image of the item or scene.

[0139] Example 32: The method of Example 29 or other preceding examples, further including receiving, via a user interface, a user's selection to make a further change effective in improving the accuracy or resemblance of the rendering when the modified image of the item or scene is rendered on a display unit based on the calibration of the display unit on which the modified image is rendered.

[0140] Example 33: The method of Example 29 or other preceding examples, further including automatically adding a change effective in improving the accuracy or resemblance of the rendering when the modified image of the item or scene is rendered on a display unit based on the calibration of the display unit on which the modified image is rendered. Conclusion

[0141] Although the various aspects of color rendering have been described in terms of specific functional characteristics and / or methods, the subject matter of each of the appended claims is not necessarily limited to the specific functional characteristics and methods described above. Rather, the specific functional characteristics and methods are merely disclosed as examples of color rendering. Therefore, other equivalent functional characteristics and methods are also intended to be included within the scope of the appended claims. Furthermore, although various different aspects have been described, it goes without saying that each of the above aspects can be implemented individually or in association with one or more of the other aspects described above.

Claims

**Claim 1** Providing a user interface that receives a user selection to modify an image captured with a physical color reference, where the image captured with the physical color reference is captured under a first ambient light, and the user interface is configured to receive a modification to the image captured with the physical color reference based on a visual comparison between the image captured with the physical color reference and a physical object at a specific location based on the physical color reference; Receiving, via the user interface, a user selection to modify an image captured with a physical color reference, where the user selection is received based on a visual comparison between the image captured with the physical color reference and a physical object at a specific location, the physical object at the specific location being under a second ambient light that is different from the first ambient light; Presenting, via the user interface or another user interface, a modified image captured with the physical color reference based on the received user selection, where the modified image captured with the physical color reference can be made more accurate than the image captured with the physical color reference via the user selection or resemble the second ambient light; Modifying an image of an item or scene captured at the same location as the location where the image captured with the physical color reference was captured, based on the received selection or the modification to the image captured with the physical color reference; Presenting or recording a modified image of the item or scene, where the modified image of the item or scene is more accurate than the image of the item or scene or resembles the second ambient light; comprising A method of presenting or recording a modified image of an item or scene by presenting the modified image of the item or scene within an augmented reality interface. **Claim 2** Providing a plurality of selectable control elements by providing the user interface, each of the plurality of selectable control elements being presented with a different representation of the physical color reference, where receiving the user selection includes selecting any of the plurality of selectable control elements, and further comprising determining various representations of the physical color reference, the determination including Associating a portion of an image capturing the physical color reference with a color that is the same color in a previously recorded image of the physical color reference or a copy of the physical color reference, and the position in the previously recorded image of the physical color reference or the copy of the physical color reference is known; Determining the actual color of the portion of the image capturing the physical color reference based on the color information recorded for the known position; Determining different representations of the physical color reference based on the actual color of the portion; The method according to claim 1, characterized in that it includes the above.

3. The method according to claim 2, characterized in that the previously recorded image of the physical color reference or the copy of the physical color reference is determined by calibration, and the calibration is based on a user-selectable comparison for the purpose of calibrating a display unit presenting the user interface.

4. The method according to any one of claims 1 to 3, characterized in that an image capturing the physical color reference and an image of an item or a scene are present in the same captured image, and the change to the image capturing the physical color reference and the change to the image of the item or the scene are in the same process performed simultaneously.

5. The method according to any one of claims 1 to 4, characterized in that it is possible through a user interface to receive a user selection when the user manually makes a selection regarding a change in the color of an image capturing the physical color reference through an operation of a control element, and receiving a user selection for changing the image capturing the physical color reference includes changing the hue, chroma (vividness / saturation), lightness, or luminance of the image capturing the physical color reference.

6. The method according to any one of claims 1 to 5, further including recording a color change due to the change in the physical color reference, and the recording of the change being effective in future changes to a future image for presenting the future image more accurately or more realistically.

7. The method according to any one of claims 1 to 6, further including automatically making a change to another image of another item or scene, or another image of another item or scene captured in the same location context, and the automatic change being based on the difference between the image capturing the physical color reference or the user's selection and the state of the image capturing the physical color reference after the change.

8. The method according to any one of claims 1 to 7, wherein the article or scene is a cosmetic, and the physical color standard thereof includes the color tone range of human skin, or the article or scene is a clothing product, and the physical color standard thereof includes the color tone range of human hair and skin, or the article or scene is a landscape, and the physical color standard thereof includes the range of colors seen in outdoor locations.

9. The method according to any one of claims 1 to 7, wherein the article or scene is a cosmetic, and presenting or recording the changed image of the article or scene means presenting the changed image of the article or scene within an interface having an image of the user's hair, skin or face, the cosmetic, a part of the cosmetic, or the color of the cosmetic.

10. The method according to any one of claims 1 to 9, wherein the physical object at a specific location is a physical color standard.

11. The method according to any one of claims 1 to 9, wherein the physical object at a specific location is part of the user's anatomical tissue, and the part of the user's anatomical tissue is associated with a specific position of the physical color standard.

12. The method according to claim 11, wherein the part of the user's anatomical tissue is the user's skin.

13. The method according to any one of claims 1 to 12, wherein presenting or recording the changed image of the article or scene means presenting or recording the changed image of the article or scene in real time.

14. A system comprising means for performing the method according to any one of claims 1 to 13.

15. A computer device, a display unit, a processor, and a computer-readable storage medium including instructions which, when executed by the processor, cause the computer device to perform the method according to any one of claims 1 to 13. A computer device comprising the above.

Images