Color rendering

Abstract

To provide improvement of color rendering by improving accuracy in rendering an item online while displaying a webpage on a notebook computer, smartphone or desktop computer of a user.SOLUTION: An image of a physical color standard is captured in a first ambient light. A user interface is configured to: receive a change to the captured image of the physical color standard based on a naked-eye comparison of the captured image of the physical color standard with a specific local physical object based on the physical color standard; receive the user selection to alter the captured image of the physical color standard; and alter the image of an item or scene captured in the same locale as that in which the image of the physical color standard was captured, based on the alteration to the image of the color standard, and present the image.SELECTED DRAWING: Figure 4

Description

[Background technology]

[0001] When an item is displayed online or in a printed catalog, its true appearance may not be accurately portrayed. For example, consider an off-white chair offered for sale in a printed catalog. A buyer may want to purchase the chair in an off-white color but ultimately forgo the purchase because they are unsure whether the displayed color matches their current off-white interior decor. This not only results in missed revenue opportunities for the seller, but also misses out on the benefits of owning the chair. Even worse, a buyer may purchase a chair they believe will match their interior decor only to discover after delivery that it does not. In such cases, the buyer or seller must pay return shipping costs, and both parties must expend time and effort processing the return. These inaccuracies, caused by inaccurate color rendering of items in catalogs and similar print catalogs, can be costly.

[0002] The accuracy of representations (e.g., image depictions of items) is typically lower in online sales than in catalog representations. Color representation often relies on image capture and image representation devices, all of which may have color errors, resulting in inaccurate color rendering. This inaccuracy in representation often results in items being displayed in a slightly or significantly different tone from their actual color. Return issues based solely on these defects are frequent, resulting in significant losses in shipping costs to and from buyers, buyer dissatisfaction, and lost time and money for both buyers and sellers. Summary of the Invention

[0003] The present disclosure relates to methods, systems, and / or software configured to enhance the color rendition of items or scenes, such as merchandise sold through websites or printed publications (e.g., color catalogs). This enhancement can be achieved by improving the accuracy of online representations of items, such as by displaying web pages on a user's laptop, smartphone, or desktop computer. The present disclosure also addresses many problems in the field of online sales, where accurate representations of items and customized representations tailored to a buyer's intended use of the item are desirable. Furthermore, the present disclosure describes methods for enhancing the color rendition of printed items through the use of a user's device, such as capturing an image of the printed item with a camera on the user's device and then displaying the image more accurately or similarly to the original printed representation. [Brief explanation of the drawings]

[0004] This application contains at least one color drawing. Copies of the color drawing(s) are available from the U.S. Patent and Trademark Office upon request and payment of the appropriate fee. Aspects of techniques and devices for enabling machine learning search refinement are described with reference to the following figures, in which like reference numerals refer to like features and components throughout the drawings. [Figure 1] FIG. 1 illustrates an example of an environment in which color rendering techniques can be implemented. [Figure 2] FIG. 1 illustrates an example of a user interface with an image of an item and object color criteria. [Figure 3] 1 shows the initial state of a set of slider-type control elements, followed by two successive user selections and the resulting changes to the item and object color criteria. [Figure 4] 10A-10C illustrate examples of methods for improving accuracy or likeness of items and scenes. [Figure 5] FIG. 10 illustrates various brightness options that can be selected using a user interface. [Figure 6]This figure shows the nine ranges of hue and brightness levels displayed on the user interface. [Figure 7] 1A-1C illustrate examples of how color matching techniques can be used to improve the accuracy and likeness of current or future renditions of items or scenes in accordance with the present techniques. [Figure 8] 1A and 1B show an item and a physical color standard captured on or near the item, as well as an accurate or calibrated color standard showing known colors and their locations. [Figure 9] 1A and 1B show examples of how the technique can be used to more accurately or realistically represent items or scenes. [Figure 10] FIG. 1 illustrates an augmented reality interface displaying an image of an item and object color criteria, and a user's location. [Figure 11] 1 illustrates an example of how the technique allows a seller to more accurately stage images of items or scenes. [Figure 12] FIG. 10 illustrates an example of how the technique allows buyers and other viewers to more accurately or realistically render images containing items or scenes in electronic formats (email, text message, social media, web pages, etc.) or when printed. DETAILED DESCRIPTION OF THE INVENTION

[0005] Overview of the Invention The present disclosure relates to methods, systems, and / or software configured to improve color rendition, such as in camera-captured family photos, products and presentations offered through websites, or products and presentations in printed publications such as color catalogs. The present techniques improve rendition across various avenues, such as a user's laptop, smartphone, or desktop computer, where the webpage is viewed, thereby enabling more accurate online product renditions. The present disclosure also addresses many problems in the field of online renditions, where accurate depictions of items and scenes are desirable. Furthermore, the disclosed techniques enable customized renditions tailored to a buyer's intended use of the item. The present disclosure also describes methods for improving the rendition of an item in a printed image through the use of a user's device, such as capturing an image of the item in its printed form with a camera on the user's device and then displaying it on the device's display more accurately than the original printed representation. If the original image is not imported correctly

[0006] Original images of items or scenes may not be accurately captured. Therefore, product images may already be inaccurate before they are displayed on a website for sale or before images of the items are provided for printing. The techniques described herein improve accuracy by utilizing an object color standard. This object color standard provides a means for matching the colors of captured images. To utilize an object color standard in this manner, an image of the item is captured along with the object color standard. This can be done using the object color standard in an image of the item, or in an image taken around the same time, or in an image taken under the same or similar ambient conditions. By employing the object color standard, an image provider, such as a seller, can modify the captured image by comparing the displayed object color standard with the object color standard perceived by their own eyes. Because this is performed by a human, additional errors may be introduced, but it can also correct for errors and discrepancies between the actual reflected light from an object (a measurable, actual physical property of reflected light) and human perception.

[0007] The object color standard may be replicated or standardized and may in fact be the same as the actual object color standard photographed with the item, or may be a representation or data representation of the object color standard. Object color standards can also be incorporated using computer programs that store object color data. In either or both cases, employing object color standards can improve the accuracy of images of items or scenes. This allows an entity (real person or thing) capturing an image, such as a seller, to provide a modified or improved image to a web page or printer. The techniques described in this disclosure improve the accuracy of the depiction of an item or scene, which may then naturally improve the presentation of the item when ultimately displayed to other entities (e.g., viewers of the scene or buyers of the item).

[0008] Furthermore, the employment of object color standards in accordance with the present techniques allows for the color information recorded for the object color standards to be used to automatically correct images of items or scenes. These techniques are described in more detail below. Inaccurate or inappropriate presentation

[0009] There are many reasons why a color image may not be accurately rendered on paper or display. As mentioned above, the original cause may be that the image of the item or scene was not accurately captured, but this is not necessarily the only problem.

[0010] Suppose a seller takes original images of a product with a camera that does not capture color accurately, or in ambient lighting that causes the product to appear in colors different from normal ambient light (e.g., unusual home lighting or sunlight, such as when an image is taken in a fluorescent-lit studio, or when a product is photographed indoors for outdoor use).

[0011] In such cases, the image of the product in the original image may not represent the product's intended appearance under different ambient conditions, such as the location where the buyer intends to use the product (e.g., outdoors for outdoor furniture, indoors for outdoor / indoor clothing, paintings, cushions, or carpets, etc.). As noted above, the original image may suffer from additional apparent inaccuracies even without the inaccuracies of lighting differences.

[0012] The techniques described in this disclosure provide a physical color standard as a means to remedy this deficiency. This physical color standard provides a means to match the color of a rendered image, either contemporaneously with the rendering of the image or through prior calibration (described in more detail below). To utilize this physical color standard, a capturing entity, such as a seller, captures an image of the item along with the physical color standard. As previously mentioned, this may be done using the physical color standard contained within the captured image of the item. (However, to avoid cluttering the depiction space of the item, the physical color standard may not be displayed until a rendering request with the physical color standard is made, and may instead be extracted from the image and provided upon request from a web page.)

[0013] To further illustrate, consider the case of a bright teal shirt. The color of the shirt is almost always an important factor to the buyer. Thus, rather than guessing whether the seller captured an accurate or comparable image of the shirt, or whether the buyer's own display can accurately display the color, the seller captures an image of the shirt using a physical color standard, and then the buyer is given the opportunity to further attempt an accurate rendering. As used in this disclosure, the terms "accurate" and "precise" refer to the accurate representation of the color of the item when rendered as an image, relative to the actual appearance of the item at the time of the image capture. Congruent, on the other hand, refers to how the item appears at a different location, the same or different from the location where the original image was captured. Thus, users may desire accuracy, preconceived imagery, or sometimes a combination of both. When purchasing a product, buyers expect accurate product images to be presented in catalogs and on websites. It is also desirable for the product images to appear as they would appear in the location where the user is viewing the catalog or display. As detailed in the disclosure, users may wish to display images of items or scenes that are altered to reflect their surroundings, lighting, and even lighting angles, etc. Also, users often desire accurate color rendition, although similarity is less important.

[0014] As noted above, a seller can modify a captured image based on a physical color standard (because the physical color standard can be copied or standardized and thus used to calibrate captured images with the aid of a computer program or a human, or both), and then provide the modified / improved image to a printer for a web page or catalog. This is a partial solution in improving the image that is ultimately presented to the buyer. As noted above, this is the case, for example, for more accurately displaying the color of an image (here, a teal shirt) on a website, in a color catalog, or other presentation.

[0015] Second, a seller can capture and provide an object color standard along with an image of a shirt or other item, regardless of whether the seller modifies the captured image based on the object color standard. Thus, the object color standard can be displayed or accessed in the image ultimately presented to the buyer, even on the buyer's display (e.g., a website display of the item on the buyer's smartphone or a catalog display captured by the buyer's camera). Because potential errors can occur when capturing an image for the object color standard at a time other than when the image was captured, capturing the object color standard simultaneously with the image and / or under the same or similar conditions as when the image was captured provides a greater ability to correct the captured image (although even in the former case, the object color standard can improve color rendition if captured under the same or similar ambient lighting and with the same or nearly the same image capture device).

[0016] In this example, the technique uses an image of an item and a physical color standard (e.g., a physical color standard in the same captured image) to correct the image to accurately represent the item's true color. Here, true color includes at least the correct hue (e.g., red, green, blue), but the technique can also correct the correct lightness, chrominance (saturation), and brightness through correction. To do so, the technique can compare the physical color standard in the image to another physical color standard (e.g., a color standard accessible from a rendering device or an exact copy of the physical color standard on a physical medium), such as through an application on the buyer's computer. The technique also alters the rendition of the item's color (e.g., hue, chroma, lightness, and / or brightness) and other attributes based on the correction to render the physical color standard captured with the item to match or nearly match the buyer's physical color standard (e.g., a physical color standard intended for the buyer's own location).

[0017] For example, if the object color standard in the image is not properly (accurately or appropriately) depicted, an application on the device of the image staging party (e.g., a buyer's smartphone) may alter the color of the image to match the object color standard in the image. This is effective for correcting the color of the item by modifying the image and color standard of the entire item until the object color standard depicted on the buyer's smartphone matches the object color standard in the image as it would appear in the staging party's location. If the captured object color standard can be properly depicted according to the actual object color standard, the item will also be properly depicted. This is because color changes can be made to both the item and the captured object color standard at once (which is not required, but is easier than performing them separately). Thus, if the captured item and object color standard have poor red / green balance or too little blue, the application can be used to rebalance the captured object color standard to match the object color standard in the particular location until it is properly depicted (e.g., until they appear the same to the viewer). If the imaged physical color standard is properly depicted, the color of the item will also be properly depicted (even more so than before).

[0018] Furthermore, in this technique, the user's preferences are determined based on information such as the type of display, age, and settings. The display can be checked to see if it displays the correct colors. The technique can then modify the displayed image based on the correct physical color standard and the color match on the buyer's device. Miscalibration of the buyer's device can also be corrected by staging an image of the physical color standard (e.g., obtained using the device's own camera or by receiving one provided) and then calibrating the display to match the standard through device settings or visual comparison with the naked eye as described above. In this way, the technique can record how to modify the display settings if a proper depiction of a scene or item is desired. Note that many devices are intentionally calibrated to prevent an accurate or proper depiction of a scene, such as by reducing blue light, dimming for nighttime viewing, or modifying the display to conserve power.

[0019] More specifically, the technique also performs a calibration sequence, such as taking an image of a color (here, the color of some or all of the physical color standards), displaying it to the user of the device, and allowing the user to calibrate the displayed colors to match. To achieve a more accurate rendition of the image, this process can be performed several times with different colors and lighting (rather than having to compare the physical color standards for each rendition of an item or scene). Adjusting for ambient light color

[0020] The techniques of this disclosure also address the issue of different ambient lighting (color, angle, etc.). For example, even if the taking entity captures the image accurately and the image on the other person's display appears accurately, this only means that the item will match what the normal human eye sees when imaged by the taking entity (e.g., the seller's own eyes), but it does not necessarily mean that a subsequent rendition of the same item by a rendition entity (e.g., by a buyer on their display) will be appropriate for how the item appears in the buyer's current ambient conditions.

[0021] Suppose a buyer is reviewing an item on their smartphone. In this example, we assume that the item is accurately rendered on the buyer's display. However, even accurate rendering may differ from the context in which the buyer is considering placing the item. Returning to the example of the off-white chair, we assume that the light used when the item was captured was bright blue-white (i.e., not natural light), or that the item was captured using natural wavelengths of light, but the buyer's home is lit with incandescent (light yellow) bulbs, fluorescent lights, or light-emitting diode (LED) light. In either case, the rendering on the buyer's smartphone does not adequately portray the item in the colors expected when viewed under the actual ambient conditions where the buyer is currently located or where the buyer intends to place the item (in this example, an off-white chair). In such cases, the appropriate rendering of the item relies on ambient conditions rather than the color accuracy of the item or scene at the time of capture.

[0022] The technique can appropriately render the image presented to the buyer's display to reflect how the item will appear in the buyer's location. One method for doing this is to use a camera (e.g., a smartphone) to sense the ambient conditions (i.e., perform a spectral analysis of the ambient light for color components such as wavelength and intensity of light). This can be done by using the smartphone's camera or other sensors to determine the wavelength characteristics (e.g., wavelength, energy) and color characteristics of the current ambient light. The technique uses this information to modify the presentation of the item on the smartphone to account for differences between the current ambient light and the ambient light under which the image was captured (if known), or at least the estimated or predicted ambient light under which the image of the item was captured.

[0023] For example, if an image of an item is taken in an environment with strong blue light, resulting in an image that is particularly saturated with blue, this technique can be used to produce an image of the item with a reduced intensity of blue. Similarly, if the ambient light is from a fluorescent lamp (assuming the CRT has a relatively low color rendering index), this technique can be used to produce an image of the item with a reduced intensity of blue. Techniques can correct for spectral differences (such as the way some fluorescent lighting makes objects in an image appear too red), or adjust for incandescent lighting to reduce yellow in the re-staging (assuming a correlated color temperature CCT of 2700K). Sensors on smartphones or other devices can be used, as can information about the image of the item (such as the known or likely surroundings of the item at the time of original capture and in any subsequently altered images), but other methods described in this disclosure can also aid in altering the re-staging to suit ambient conditions.

[0024] However, if an image of the item is captured along with a physical color standard, and assuming the entity performing the rendition (e.g., a buyer) has (or had) a physical color standard, the current rendition of the item's image can be modified to match the physical color standard (as captured with the original image of the item and as possessed by the buyer). This allows the item to be presented to the buyer in a manner that is appropriate to the buyer's requirements (although such appropriateness may, strictly speaking, be considered less accurate in other rendition situations). For example, while a rendition on a buyer's smartphone may accurately represent the photographed item, it may not be what the buyer desires. A buyer may wish to see the item as it would appear in person, for example, or at least as rendition of the item in accordance with the buyer's surroundings.

[0025] The technique can make such adjustments automatically based on the buyer's device's actual rendition of the object color standard (assuming the buyer's camera and / or display are properly calibrated as described herein). Alternatively, the technique provides a user with an interface that allows the user to view the item and its captured object color standard, along with other object color standards for visual comparison with the captured object color standard, and to modify the rendition of the item image accordingly (or automatically, as described herein). Thus, the user interface can present red-yellow-blue hues, as well as saturation and brightness (and other color attributes), which the user can modify until the two standards match based on the buyer's visual perception (though these are only a few of the many color elements that can be selected and modified using the technique). The technique also compensates for differences in individual color perception, since the human eye's perception may not match technically measured values ​​of light in a linear fashion. Adjustment to ambient brightness and / or luminosity

[0026] In addition to correcting for the hue of the ambient light due to differences between the hue of the ambient light and the light when an image is captured or altered for rendering, the technique can also correct for ambient luminance. Similarly, as discussed above, the technique can render an image of an item to match the ambient luminance. Many items for sale are photographed under lighting that is significantly brighter than the conditions under which the buyer intends to use the item. Thus, regardless of whether the image was accurately captured or rendered, the image will be presented in a manner that is inappropriate for the buyer's current luminance. Similarly, the technique also corrects for luminance differences. Consider the example of a decorative item, such as a cushion for a buyer's existing sofa. Such cushions are typically photographed in a high-luminance environment, often against a white background. Thus, the technique can reduce the luminance (and other light indicators) of a rendered image of the cushion to match the luminance of the ambient light. This allows the rendering entity (here, the buyer) to properly portray the rendered item or scene. Augmented reality

[0027] Continuing with the cushion example above, suppose an original image of the cushion was taken in a high-brightness environment where the ambient light hue was slightly too blue, and the cushion was placed on a different sofa that did not match the buyer's brown sofa. Conventional practice would require the buyer to purchase the cushion. Before making a purchase decision, buyers typically must 1) trust that the image was accurately captured, 2) trust that the image was accurately rendered by the buyer's device, 3) mentally "guess what it will look like" in real life, correcting for differences in ambient lighting (e.g., red, green, blue), and / or 4) mentally "guess what it will look like" in real life, correcting for brightness. So, in addition to trusting the accuracy of the image displayed in front of them, buyers must also know what the cushions will look like with reduced blue and brightness. Furthermore, many buyers want to see how they will look in their own home décor (in this case, a brown sofa). While even one of these challenges can be problematic for a buyer, adding two, three, four, or even five of these challenges, as is often the case, can detract from a pleasant buying experience.

[0028] The present technique may address one or all of these five problems, thereby enabling more accurate and / or realistic renderings and better depictions of items and scenes, improving the user experience of catalogs, books, and websites, for example, improving the buyer's experience and confidence in making a purchase decision.

[0029] This can be achieved through the use of augmented reality using the present technique. In addition to or instead of one or more of the solutions of the present disclosure, the present technique can present an image of an item overlaid on the buyer's own situation. Many smartphones can render a current image of the user or the user's location taken with their onboard camera in real time or near real time (e.g., real time on the smartphone display). Images of previously calibrated items can be overlaid on the rendered real space. These item images can be pre-calibrated as described above or calibrated within the augmented reality interface. One way to do this is to use a somewhat three-dimensional object color fiducial (which can be useful for customizing the lighting angle and the rendition of the item to that lighting angle, or selecting the best angle from multiple images of the item taken at various lighting angles). Examples of three-dimensional object color fiducials include cubes, tetrahedrons, spheres, and hemispheres.

[0030] Continuing with the example above, consider a buyer interested in salmon-colored cushions for fall home décor. The buyer owns a brown sofa and wants to see how the cushions would actually look on their brown sofa. Using augmented reality, the buyer can move and overlay the item of interest (in this case, the salmon-colored cushions on the buyer's sofa) while viewing their surroundings and home décor on a mobile device, such as a tablet or smartphone. The item is already calibrated, and the augmented reality space accurately depicts the specific location, giving the buyer a general idea of ​​how the item will look. The buyer can then calibrate the image displayed on their display by comparing the image of the sofa and its physical color standard with the actual sofa as seen with the naked eye from their location. However, many inaccuracies or inconsistencies in the image of the item can be corrected using augmented reality technology. Suppose a buyer has a physical color standard and wants to place it on their sofa. The buyer can adjust the color (hue, brightness, saturation, etc.) of the displayed sofa by voice command, touch interface, or other means to match the criteria seen by the naked eye with the criteria displayed on their display. In this way, the buyer's location can be accurately represented on the buyer's device. Furthermore, because the representation is done while viewing the same item, such criteria do not need to be international or consistent standards. Thus, this technique may even obviate the need for physical color criteria, since the user can adjust the color and brightness via the user interface so that the rendered sofa image matches the sofa seen by the buyer's naked eye.

[0031] For example, as described above, an image can be overlaid to represent the current situation so that the item is depicted appropriately, or an object color standard can be used in the augmented reality space to change the image of the item to match it. Thus, by viewing the image of the item and the object color standard captured with it and adjusting the color such as red, green, and blue hue, brightness, lighting angle, and saturation, the buyer can create an image of the item that more closely matches the appearance of the item in the actual room, taking into account the hue, brightness, and angle of the actual lighting.

[0032] A display incorporating an augmented reality space can display a matching salmon-colored cushion alongside a specific location (e.g., sofa, lighting), allowing the buyer to place the image of the cushion on top of the sofa image, simply center or anchor it, or move the device and / or camera to display the cushion on the sofa in a desired orientation. In addition to the benefits already discussed, this augmented reality allows the buyer to "walk around" the item, providing a better understanding of the item's feel and fit. The technique allows the user to change the lighting angle of the item as they "walk around" (assuming multiple images of the item are available with different lighting angles). The technique also allows the user to edit the image of the item by shading or highlighting it to approximate how it would look under different lighting angles (e.g., as the buyer walks around in various directions, such as left, right, or back). This is accomplished by detecting the different lighting angles as the user moves, e.g., relative to a three-dimensional object color reference at the user's location.

[0033] While furniture is used as an example, clothing and other items can also be appropriately depicted. According to this technique, even without a physical color reference, a buyer can capture the color of a person's arm, for example, in an augmented reality space or using a snapshot, and then match the rendered image of the arm to the buyer's physical view of the arm. This allows the rendering to match the buyer's current situation (e.g., arm, lighting, etc.). The technique then uses one of the methods described herein to make the image of the item more suitable and present the item in a specific location. An example would be checking whether a scarf sold online matches a person's favorite jacket, hair color, or skin tone. Another example would be checking whether a shirt color matches, or interferes with, or complements, the buyer's skin tone, hair color, or makeup color. Furthermore, the color of cosmetics can also be an item to be captured. By rendering the color of cosmetics in a more suitable state, purchasing decisions can be facilitated.

[0034] This technique can improve a user's experience viewing websites, catalogs, social media, and so on by providing a more accurate depiction of the captured scene or item. For a buyer, this technique can facilitate a purchasing decision. For example, consider the use of a small, physical color standard with some degree of three-dimensionality. After capturing an image of a cosmetic product, such as lipstick, foundation, or blush, the technique can be employed (whether as a still image or with or without augmented reality) to more accurately or realistically depict the color of the product and how that color will appear on a specific individual's skin. For example, a cosmetics retailer might provide a small, foldable physical color standard with each purchase, or simply provide it free of charge in their online or physical store. Then, when a buyer wants to see how a product displayed in a catalog or online store will look on their own skin, the physical color standard can be folded into some kind of three-dimensional shape, and the technique can be used to correct for the color, brightness, and even lighting angle of the cosmetic product, thus providing a more accurate or realistic appearance. The buyer can modify the image of the cosmetic product and the physical color standard displayed therewith while comparing it with the physical color standard in the buyer's possession, thereby modifying the image so that it is appropriate for the situation in which the physical color standard in the buyer's possession exists. This allows purchases that would normally require an in-person visit to a store to be completed remotely, which is useful for both buyers and sellers, especially as many buyers have limited mobility due to health concerns (such as the spread of COVID-19) or economic or ecological considerations (such as not wanting to drive to a store to protect the environment or save the buyer money). Adjusting the size

[0035] In addition to or instead of the above techniques, item images can be properly sized. This can be done using several steps described above, but can also be done by the following procedure. This technique first calibrates the image (including the angle at which the image is presented) using information about the item, such as height, width, and depth, and associates those dimensions with the image. The size of the location and other conditions (such as the buyer's body or interior décor) can be determined by manually entering dimensions or by using mobile device features that allow measuring objects in the image. Examples of such measurement features include the Apple® Measure app, which allows users to measure the dimensions of items created using augmented reality or snapshots of specific location conditions or objects. Instead of or in addition to using the above steps, this technique can also use the dimensions of object color standards. Assuming the dimensions of the object color standards in the captured image of the item are the same or different from those present at the buyer's location, this technique allows users to scale the image of the item for sale on a webpage (or paper catalog) so that it appears at the correct scale on the buyer's mobile device. (For example, salmon-colored cushions can be displayed in the appropriate size depending on the size of the sofa, thereby further facilitating the buyer's decision-making.)

[0036] This can be particularly useful for furniture and interior décor, jewelry, accessories, and clothing (although in the case of clothing, this refers to images taken on models or mannequins, since scaling does not make much sense when clothing is laid flat or folded). Suppose a buyer wants to see how a particular bracelet looks on their wrist. This technique can improve the product presentation and help the buyer make a purchase decision. The bracelet can be overlaid on a snapshot of the buyer's wrist or in real-time augmented reality, displayed in the right color, at the right brightness, at the right lighting angle, and at the correct scale. This is a significant improvement for both buyers and sellers, even in non-website sales, such as catalogs. Example environment

[0037] 1 illustrates an example system 100 that can implement techniques for providing a more accurate and / or more realistic representation of a photographed item. System 100 includes computing devices 102, which are shown as four exemplary mobile devices: a laptop computer 102-1, a tablet computing device 102-2, a smartphone 102-3, and an e-reader 102-4, although other computing devices or systems, such as desktop computers or netbooks, may also be used.

[0038] Computing device 102 includes a computer processor 104, a computer-readable storage medium 106 (hereinafter "medium 106"), a display(s) 108, and an input mechanism 110. Medium 106 includes computer-executable instructions that, when executed by computer processor 104, perform operations such as an operating system 112 and an imaging module 114.

[0039] The image module 114 may, for example, improve the accuracy and likeness of the image at the location where the image is taken (e.g., the seller as described above in this disclosure) or at the location where the image is ultimately staged (e.g., the buyer). and the like, have functions that enable or assist the techniques described in this disclosure.

[0040] The image module 114 may also include or have access to a history 116, a user interface 118, and a three-dimensional module 120 (hereinafter, "3D module 120"). The user interface 118 allows the image module 114 to present a rendered image (e.g., a representation of the user's current location in augmented reality, including an item) on the display 108. The user interface 118 also allows the user of the computing device 102 to modify the rendered image via the input mechanism 110. The 3D module 120 allows the image module 114 to modify the image to show the item from different angles, lighting angles, and / or scales depending on the context depicted in the augmented reality. The 3D module 120 can determine and modify the scale of the item using object color references in the image in which the item is displayed along with object color references present at the device's location. Alternatively, the 3D module 120 can use a measurement sensor to measure the dimensions of the location and appropriately adjust the scale of the item.

[0041] For example, the image module 114 can provide a user interface for receiving user selections to apply changes to the image capturing the object color standard, as described below. The image module 114 receives the user selections and applies changes to the image capturing the object color standard and the image capturing the item or scene. Additionally or alternatively, if the location on the object color standard of a color in the object color standard that matches a portion of an item or scene in an image is known (or can be identified), the image module 114 can correlate that portion of the item or scene in the image with the matching color in the object color standard. This can also record the actual color of that portion of the item based on the color information recorded for the known location, allowing for a more accurate or plausible rendering. Furthermore, changes to match the color of that portion to the recorded color information can be applied to the entire item, scene, or image, thereby improving the accuracy or plausibility of the entire item or scene.

[0042] History 116 may include various data as described in this disclosure, such as information about ambient light at the user's location, previous selections made by users (e.g., buyers or sellers), information about the current display (e.g., calibration data generated in accordance with this disclosure), and even data from other sources, including other users' selections and display data.

[0043] Computing device 102 may include or have access to one or more display units 108 and input mechanisms 110. FIG. 1 shows four exemplary display units, all of which may be integrated with their respective devices (although not necessarily). Input mechanisms 110 may include touch sensors and motion tracking sensors (e.g., camera-based), as well as gesture-sensitive sensors and devices such as a mouse (standalone or integrated with a keyboard), a trackpad or touchpad, capacitive sensors (e.g., mounted on the surface of computing device 102), and a microphone with voice recognition software, to name just a few. Input mechanisms 110 may be separate from or integrated with display unit 108. An example of an integrated configuration is a gesture-sensitive display with integrated touch or motion sensors.

[0044] The image capture unit 122 may include a visible or invisible light sensor, such as a camera or an IR camera (and / or a projector). The image capture unit 122 may also sense ambient conditions, even if it cannot necessarily capture an image. Furthermore, the image capture unit 122 may function in conjunction with other components to provide the augmented reality and other effects described above. Example of manually changing an image

[0045] Figure 2 illustrates an example technique by which a user can modify an image of an item on their device to alter or enhance the rendering of the item to suit the ambient conditions of a particular location, or simply correct inaccuracies in the original image or rendering of the image.

[0046] As shown, the image module 114 presents an image 202 of an item 204 along with a captured physical color standard 206 on a user interface 208 of a user device 210. The user interface 208 is configured to accept manual changes to the image through various controls, such as voice control, gesture control, or other color wheel or slider controls 212 (e.g., red, green, blue hue, brightness, etc.). In the present disclosure, manual changes are made based on a visual comparison of the captured physical color standard 206 with a physical color standard 214 present at a particular location.

[0047] For example, refer to FIG. 3, which illustrates the initial state of a set of slider-type controls 212 and two successive user selections (see 212-1 and 212-2) below the slider-type controls. The two user selections are shown as first selection 302 and second selection 304. The first user selection reduces the blue hue by lowering gray line 306 on blue slider bar 308 (the initial state of the slider-type controls). Gray line 310 represents the gray line with the blue hue reduced. The second user selection further reduces the blue hue, as represented by gray line 312 below gray line 310. Note that the initial rendition of image 202 changes from a pinkish-bluish salmon color to an orange-salmon color, reducing the blue. Note that the object color standard 206 is changed simultaneously with the item (although this can be done before changing the color of the item). In the original rendition of the physical color standard, the blue is further reduced from the blue-reduced state indicated by line 314 to a blue-reduced state indicated by line 316. Note that the blue-reduced standard 316 now matches the actual physical color standard 214. Thus, as can be seen from rendition 318 based on the first selection refinement and rendition 320 based on the second selection further refinement, the closer the rendition to the actual standard, the closer and / or more accurate the item 204 will be to the condition at the user's location.

[0048] Thus, the image module 114 receives manual modifications via the user interface 208 and modifies the image 202 based on the received manual modifications. After modifications, the image can be staged or saved for later use, and such modifications may be performed in real time, delayed, or simply provided in another interface (such as the augmented reality interface 216 in FIG. 10 ). Note that this user interface 208 may be configured as an augmented reality (by presenting a color image showing the user's location along with the imaged item and object color reference) or may be an interface that does not display the user's location. Examples of color rendering methods

[0049] The following describes a method for improving the accuracy and likeness of current or future renditions of items or scenes (e.g., anything that can be photographed) using the present techniques, using method 400 shown in Figure 4 as an example. For each method described in this disclosure, steps may be rearranged (e.g., performed in a different order than shown), performed iteratively, repeated, or combined within the context. The order shown is not required unless otherwise specified in this disclosure.

[0050] In step 402, a user interface is provided that receives user selections for modifying the image capturing the physical color reference. Examples of the user interface are described with reference to FIGS. 2, 3 (partial), 5, 6, and 9, as well as the description of the techniques above. For example, in Figures 2, 3 and 9, the user interface 208 includes a slider-type control element 212 for changing the color of the image.

[0051] This method also allows for providing a user interface with multiple selectable control elements, each presented with a different representation of a physical color reference, as described above, facilitating rapid or repetitive user selection. For example, consider the example of FIG. 5, which illustrates five intensities selectable via user interface 502 (an example of user interface 118) of smartphone 102-3.

[0052] In step 404, a user selection for modifying the image capturing the physical color standard is received via a user interface. Examples of selections via a user interface are described above, for example, when a user manually selects a color modification of the image capturing the physical color standard through manipulation of a control element. As previously described, a color modification may refer to a change in the hue, saturation (vividness or saturation), lightness, or brightness of the image capturing the physical color standard. A manual color modification selection is described above with reference to FIG. 3 using the example of a user selecting to reduce the blue hue of a captured image.

[0053] Continuing with the example of FIG. 5 , user interface 502 allows for large-scale (naked-eye) selection of intensities 504-1 through 504-5, for example, by simply tapping on the display to select one of five physical color standard images, each with a different luminance. As noted above, FIG. 5 also shows physical color standard 506 at the user's location. Each of these actions can be repeated, replayed, modified, and so on. In this example, as also shown in FIG. 5 , the technique may present finer gradation options (e.g., a fine luminance range 510 with five fine gradations of selectable intensities 510-1, 510-2, 510-3, 510-4, 510-5) and, after a luminance selection, different hue options or different saturations, for example, with different color change classifications, at or near the selected luminance.

[0054] In step 406, the modified image capturing the physical color reference is presented via the user interface or another user interface and based on the received user selection. While many examples of this are described throughout this disclosure, for example, as shown in FIG. 3, modified image 314 capturing the physical color reference is further modified to produce image 316, which is less blue than images 314 and 206. Continuing with the example of FIG. 5, selection of one of the luminance ranges (here, 504-3) results in the presentation of subtle variations in the physical color reference (labeled 510-1, 510-2, 510-4, and 510-5) (along with copies of the selected luminance 504-3). While not required to employ this method or use the specific elements described above, some entities, such as remote entities or image module 114, may modify the image prior to presentation in step 406.

[0055] In step 408, an image of an item or scene captured at the same location as the image of the object color standard is altered. Such alterations may be based on one or more selections received (as described above with reference to FIGS. 3 and 5) regarding alterations to the image of the object color standard captured at the same location as the item or scene. Such alterations to the image of the item or scene, and thereby to provide a more accurate or plausible representation of the item or scene, may be made individually, simultaneously, or contemporaneously. In the example shown in FIG. 3, the image module 114 alters the image of the object color standard and the image of the item or scene contemporaneously in a single action, such as when the item or scene is captured in the same image as the object color standard. Thus, as shown in FIG. 3, the object color standard is altered and the item is also altered, both of which are displayed in the user interface 208 (not shown in FIG. 3 for clarity of visual representation).

[0056] At step 410, the altered image of the item or scene is recorded or displayed. In FIG. 3, the altered image of the item or scene is shown at 320. Additionally or alternatively, the alterations can be recorded, thereby recording the color changes due to the altered object color reference. Such a record of the alterations can be useful in making future alterations to future images to present those future images more accurately or similarly. Thus, if the image of the item or scene is transmitted to another viewer along with the record, a more accurate rendition of the image or scene can be based on the record. This can facilitate further improvement in the accuracy or similarity of the rendition if the display ultimately used to rendition the item or scene has a similar setup or display format. Further alterations can be made as described herein.

[0057] Such a record, or a combination of such records, can be used later to automatically correct future received or captured images, such that the record indicates the modifications made to the user's selected image for the display associated with the user. This is one form of automatic modification based on the user's display calibration and, if the user is the image photographer, the calibration of the user's image 122 and display 108. This can be saved in history 116 of FIG. 1. In doing so, method 400 can automatically modify other images of different items or scenes captured at the same location. The automatic modification is based on the image capturing the object color standard or the difference between the user selection and the modified image capturing the object color standard as described herein.

[0058] While the object color reference is shown as a 3D box with a large range of hues, other object color references can be used. For example, if the item or scene is cosmetics, the object color reference could be a high-resolution range of human skin tones or a scaled-down version of that reference. Another example would be to use an object color reference with higher resolution human hair and skin tone, brightness, and even saturation than the object color reference shown. Furthermore, if the item is a decorative item, a different reference could be used that reflects the environmental conditions of a home. Furthermore, if the scene is a photograph of a person, the object color reference could include a range of hair and skin tones and even clothing colors to achieve a more accurate representation. If the scene is a landscape image, the object color reference could represent the range of colors present in an outdoor location.

[0059] As described in part above with reference to FIG. 5, the present technique can provide a plurality of selectable control elements consisting of different representations of the physical color standard. Generally, in step 412, the different representations of the physical color standard are determined. This can be determined in a variety of ways, including the following steps: In step 414, a portion of the image capturing the physical color standard is correlated with a matching color in a previously recorded image or copy of the physical color standard (provided the location of the matching color in the previously recorded image or copy of the physical color standard is known). The matching color can be a matching hue, a matching chroma (vividness or saturation), a matching lightness, or a matching luminance. In step 416, the actual color of the portion of the image capturing the physical color standard is determined based on the color information recorded for the known location. Additionally, in step 418, different representations of the object color criteria (e.g., multiple representations offering a finer range of color choices) are determined based on the actual color of the part, allowing for faster and more precise selection, thereby allowing image module 114 to present a wider variety of options with higher likelihood of matching.

[0060] For example, referring to Figure 6, a range of nine hues and brightness levels is shown presented via a user interface 602 of smartphone 102-3 (which is merely one example of user interface 118). Here, nine physical color reference images, each with a different color and brightness, are presented on user interface 602, allowing users to simply tap on the display. By simply manipulating the screen, one of nine color / brightness ranges 604 can be selected. As shown, the left column has more red, the right column has more blue, and the upper columns have increasing brightness. In these examples, substitution process 412 determines different selectable representations, resulting in nine different representations created based on the actual color determination, offering color choices close to or surrounding the actual color with similar hue balance, brightness, saturation (not shown), etc.

[0061] As noted above, the methods described herein can be performed in an iterative or iterative manner, such as by allowing a user to select different characteristics or characteristics in an interface (e.g., the brightness and hue grid shown in FIG. 6 ), which alters the image until it matches a physical color standard (e.g., 506). As noted above, the user can also or alternatively alter the color manually through the interface (e.g., by voice input such as "increase brightness and decrease red," or by using slider control element 212, etc.).

[0062] Referring back to the operation of step 412, instead of or in addition to performing steps 414, 416, and 418, step 412 determines a different representation based on a previous calibration. This calibration can be based on a comparison that is clearly human-selectable to calibrate the display displaying the user interface, or based on a recording of one or more modifications previously made to the colors of an image of an object color standard on the same or a similar display. For example, the image module 114 captures an image of an object color standard present at the user's location using the image capture unit 122. The user then manually modifies the image rendition to match the actual object color standard, following the steps of method 400. This modification can thus be considered a type of display calibration. However, such modifications may take into account imprecision in the image capture unit 122, and therefore the calibration may not always be perfect. Example of color rendering method using color matching method

[0063] The following describes a method for utilizing color matching techniques to improve the accuracy and likeness of current or future renditions of items or scenes, using method 700 shown in FIG. 7 as an example. For each method described in this disclosure, the operations may be rearranged (e.g., performed in a different order than shown), performed iteratively, repeated, or combined without causing problems in context. The order shown is not required unless otherwise specified in this disclosure. Note that method 700 can be performed in whole or in part without user interaction, and may therefore be performed automatically, such as by image module 114 shown in FIG. 1 (unless user interaction is explicitly stated).

[0064] In step 702, an image of a physical color standard is received. The physical color standard includes known colors at known locations within the specific physical color standard. This physical color standard may be calibrated or known to be accurate, such as precision standard 800 in FIG. 8, described below, or data representing the standard in a machine-usable format. FIG. 8 shows known colors 802 and locations 804 on precision standard 800. This need not be displayed to the user, but rather could be hidden and / or performed solely by a computer process (e.g., image module 114, which includes instructions that cause processor 104 to perform the operations of method 700) using a data file or image containing the known locations and colors. The locations may be Cartesian coordinates, etc.

[0065] A known color 802 has at least one known hue and may also have other known color attributes. A location 804 is also known and correlated to the known color 802. Although not required, a known color 802 that resides at a known location 804 within the reference 800 may be used. The color 802 can be determined by calibration based on a human selectable comparison for purposes of calibrating a display that presents a user interface.

[0066] In step 704, an image of an item or scene is received. The image of the item or scene and the image of the object color reference may be the same image, different images captured at the same time, or different images captured at the same location or with the same ambient conditions, etc.

[0067] In step 706, portions of an item or scene displayed in the image of the item or scene are correlated with matching colors in the object color standard, provided that the location of the matching color on the object color standard is known. The object color standard is an object color standard captured at the same or similar point, such as in the same image as the item or scene. Correlating portions of an item may involve sampling portions of the image of the item or scene with known colors in the object color standard, or vice versa, and, upon finding a match based on the sampling, the corresponding portion is correlated. Furthermore, the correlation may be performed to match the hue, chroma (saturation), lightness, or brightness of the portion.

[0068] 8, which shows an image 202 of an item 204 displayed in the same image 202 as an object color reference 206 in FIG. 2. In step 706, the image module 114 correlates a portion 806 of the item 204 (i.e., a portion having a color that matches a color in a portion 808 of the image of the object color reference 206). Note that the color rendered in the image of the object color reference 206 may or may not be correctly rendered, and if the image of the item 204 and the image of the object color reference 206 are not rendered correctly in the same way, the color of the item can be corrected.

[0069] In step 708, the actual color of the portion of the item or scene is determined based on the color information recorded for the known location, where the location of the portion 808 is a location 810 in the image of the object color reference 206, which is mapped to a location 804 of the precise reference 800 that correlates to a known color 802. The known color 802 is the color that is actually desired to be used to render the portion 806 of the item 204.

[0070] In step 710, the recorded color information and the location of that portion of the item or scene are recorded. This recording helps ensure that the correct or similar color is correctly represented in future renditions of the item or scene. In the illustrated example, such color information includes known color 802 and portion 806 of item 204.

[0071] In step 712, the technique may render an image of the item or scene with the correct or likely color based on the recorded color information and the location of the portion of the item or scene. Although not required, once this correct or likely color image of the item or scene has been rendered, a user may verify its accuracy or suitability and make changes according to the various techniques described above (e.g., step 404 of method 400). Examples of color rendering methods using recorded color information

[0072] The following describes a method for improving the accuracy and likeness of the presentation of items and scenes using this technique, taking the method 900 shown in FIG. 9 as an example. For each method described in this disclosure, the operations can be rearranged (e.g., performed in a different order than shown), performed iteratively, repeated, or combined, as long as the context remains consistent. The order shown is not required unless otherwise specified in this disclosure. Note that method 900 can be performed in whole or in part without user interaction, and can therefore also be performed automatically by the image module 114 shown in FIG. 1, for example. (However, it is possible to perform the method 900 automatically by using the image module 114 shown in FIG. 1 without explicit user interaction.) (Except as stated in

[0073] In step 902, an image of an item or scene is received. Although not required, the recorded color information and location of the portion of the item or scene may be generated as a result of the operations of method 700. In such cases, the image may be received from a remote device that performed method 700, or may be received from the same device that performs method 900 (e.g., if method 700 is performed on the same user device, such as the same computing device 102).

[0074] Step 904 involves receiving recorded color information and a location of a portion of an item or scene, the recorded color information and the location of the portion of the item or scene indicating the exact color or a suitable color for the portion of the item or scene at that location, the recorded color information and location indicating the color of the portion of the item or scene in the exact color at that location, rather than the suitable color for the user's ambient conditions, and receiving, via a user interface, a selection by the user to make further modifications to the rendering of the modified image of the item or scene. In such cases, the further modifications may be based on object color criteria at a particular location of the point of view from the user, effective to improve the suitability of the modified image of the item or scene for the ambient lighting conditions at the location.

[0075] In step 906, the difference between the color of the item or portion of the scene in the image and the recorded color information is determined.

[0076] In step 908, an image of the item or scene is altered based on the recorded color or color difference. Note that the change made to the color of this portion to match the recorded color can be made to only that portion of the item or scene, or to the entire item or scene, or any portion in between. Thus, an alteration, such as reducing the red hue and increasing the saturation of a portion of an item to match the recorded color, can be applied to more than that portion. This allows the alteration to be made without detailed analysis of each portion of the item or scene. In this disclosure, the term "alter" refers to various changes made to an image, such as creating a new image, making changes to the image's file, or even using an altered image. However, the term "alter" should not be interpreted as necessarily requiring the use of the same image or data file; rather, a different image or data file can be created, in which case the altered image becomes a new image.

[0077] In step 910, the modified image of the item or scene is recorded or rendered. As previously described, the modified image can be rendered on the display 108 using the user interface 118 of the computing device 102. However, the user interface can be an augmented reality interface, which can be generated by the 3D module 120.

[0078] As described somewhat above, the present technique allows a display to be calibrated to more accurately depict an image. In such a case, further modifications to the image can be made automatically based on the calibration. Alternatively, the modifications may be received by a user manually selecting the calibration of the display to render the modified image. The further modifications improve the likeness or accuracy with which the modified image of the item or scene is rendered on the display. Augmented Reality Examples

[0079] 10 is a diagram illustrating an example of an augmented reality interface, in which an item 204, an imaged object color reference 206 (which may optionally have been previously calibrated, e.g., as shown in FIG. 2), and an object location 220 are displayed via an augmented reality interface 216. The example may present an image 218 of the location where the image 218 was captured. The image 202 of the item 204 and the object color reference 206 are overlaid on the image 218 of the location. This example also includes a slider control 212 that the user can manipulate to change the brightness and color of the image 218 of the location (and, separately, the image 204 of the item, which may have previously been corrected using the present techniques). As previously described, the object color reference 222 of the particular location may be captured as an augmented reality image 224 in the augmented reality interface 216. The user may then modify the augmented reality image 224 until it matches the object color reference 222 of the particular location as perceived by the user's naked eye. This allows the captured item 204 and the captured location 220 (i.e., the image 218 of the location) to appear more representative of the user's location 220. Thus, the user can visually verify whether the salmon-colored cushion product matches their interior décor (here, the sofa and table at physical location 220). The technique thus allows the user to modify the image of the product and the image of the location where the product will be used to correct any inaccuracies or incompatibility. Alternatively, the user can modify image 202 and item 204 and place their display on the sofa (not shown). While this facilitates color matching, it can be difficult to scale item 204 to the correct size (since most cushions are larger than the display of a smartphone or tablet). Another example of a buyer and seller

[0080] The following describes how the present techniques can enable sellers to more accurately stage images of items, such as items for sale, using the example method 1100 shown in Figure 11. In each of the methods described in this disclosure, the steps can be combined, rearranged (e.g., performed in a different order than shown), performed iteratively, repeated, or combined. The order shown is not required unless otherwise specified in this disclosure.

[0081] In step 1102, an image of the item along with the physical color reference is captured, which can be done by a user using image capture unit 122 or by any of the methods described above.

[0082] In step 1104, an application (eg, image module 114) receives a captured image of an item and an object color reference.

[0083] In step 1106, the application compares the object color standard captured in the captured image to recorded images of the object color standard or data that can be used to determine the accuracy of the captured object color standard.

[0084] In step 1108, the application modifies the image of the item and the object color standard based on the comparison. Such modifications to the image are effective to improve the quality of the image of the item. Note that such improvement may involve correcting for camera-induced errors in staging by improving the quality to the color of the ambient light under which the image was captured, and / or correcting the image to the color of a different ambient light under which the image was captured, such as a recorded image of the object color standard taken under preferred ambient lighting (e.g., sunlight-based white light).

[0085] Alternatively, or in addition, the application may present the item in a user interface in step 1110, allowing the seller to adjust the image to match what they see while viewing with their own eyes (e.g., by adjusting a captured image to a physically present item or by adjusting a captured object color standard to a physically present color standard). In step 1112, the application receives manual changes to the image (e.g., based on a user's visual comparison of the object color standard in the image with the object color standard actually present at a particular location). In doing so, the application (e.g., image module 114) modifies the image of the item in response to the user's input and records the modified image in step 1114. This can be performed after, before, or in conjunction with other steps in the method.

[0086] Alternatively, or in addition, the technique can assist the seller in selecting multiple modified object color standards in step 1116 to enhance the image of the item. As described below with respect to the examples of Figures 5, 6, and 12, the technique similarly facilitates the seller's selection of more suitable object color standard images, and thus more accurate images of the associated item. Each object color standard to be modified is determined in step 1116, presented in step 1118, and a selection is received in step 1120. The method for determining, selecting, and repeating and sequencing these steps is described in the description of Figure 12, also with reference to Figures 5 and 6. For example, if the seller's device or camera system settings or personalization history is known, the technique can facilitate the presentation of images modified to approximate the accurate, human-eye appearance of the object color standard, or it can present more drastic standards and allow for iterative selection to determine the precise modifications.

[0087] Returning to step 1114, the modified image and the inputs made to modify the image are provided on or recorded in a display as described above with respect to user interface 208 shown in FIG. 2 (and FIG. 3), the augmented reality interface shown in FIG. 10, and user interface 502 shown in FIG. 5. Example method (buyer side)

[0088] The following describes a method 1200 shown in FIG. 12 that allows a buyer to more accurately or realistically represent an item or scene, such as an item for sale, in an electronic format (e.g., email, text message, social media, web page, etc.) or in print. In each method described herein, steps may be combined, rearranged (e.g., performed in a different order than shown), performed iteratively, repeated, or combined. The order shown is not required unless otherwise specified in this disclosure.

[0089] A simplified example method 1200 illustrating a technique for producing more accurate or more representative images of items for sale in electronic (email, text message, social media, web pages, etc.) or printed formats is shown and described below with reference to FIG. 12.

[0090] In step 1202, a captured image is received by an application. This captured image may be enhanced as described for the seller's method, or may be unenhanced. In this method (i.e., other than some other method described in this disclosure), an image of the item is captured along with the physical color standard. In the case of an image contained in a printed publication such as a catalog or advertisement, the buyer may use their device to capture (e.g., take a photo of) the printed image of the item and the physical color standard and receive the image.

[0091] Optionally, in step 1204, the application may compare the object color reference captured in the captured image with a copy of an object color reference of the same type (e.g., a copy thereof) captured at a particular location. By comparing the captured image with an image of the object color reference captured at a particular location (e.g., by image module 114), differences between the two may be determined based on the ambient light at the time the image captured at the particular location was captured, and then the image of the item and the object color reference may be automatically adjusted. This may improve color accuracy and / or similarity, such as hue, brightness, lightness, and saturation, associated with the item. For example, brightness may be reduced to match slightly darker ambient lighting, or the current This may involve making changes to the red, green, or blue hues to make them closer to or further away from the original image. The technique may also use historical data in this process to improve the image of the item described in this disclosure.

[0092] In step 1206, the application modifies the image of the item based on the comparison. In step 1208, the image of the item is provided to a buyer's device or other device, where imperfections such as an inaccurate rendering of the display of the device can be corrected, or the user can manually make further modifications to the image of the item in steps 1212 and 1214 (or as part of steps 1206 and 1208) by visually comparing the modified image rendered in the provided user interface to a physical color standard.

[0093] Incidentally, this method can be performed iteratively or repeatedly in real time as ambient conditions change or as buyers' perspectives on physical color criteria change.

[0094] Additionally, in some cases, the seller's application may be configured to perform a spectral analysis of the light at the time the image is captured, and such configuration may be applied to the buyer's application (which may be the same or different from the seller's application) to assist in correcting the image of the item.

[0095] Additionally, the technique may optionally utilize information regarding the buyer's display type, age, and imperfections when making changes to an image, such as before the buyer optionally makes any adjustments to the image of the item. Furthermore, the image module 114 may allow a viewer (e.g., a buyer) to calibrate the display based on an image of a physical color standard (such as an image acquired or photographed by the buyer) using the technique, and then use a user interface to make changes to the display calibration while viewing the physical color standard with the naked eye at a specific location, such as before performing the method. This may improve the display generally or provide calibration data so that when a buyer receives an image (e.g., an image of an item for sale) such as by importing it for color selection, the technique may use such calibration to improve the presentation of the item (e.g., automatically in steps 1216 and 1218, or by other methods described herein). This step may also be performed in conjunction with or separately from other steps, such as by first performing this step and then prompting the buyer to make further improvements or changes to the image via a user interface.

[0096] Alternatively or additionally, in step 1216, multiple modified object color standards are determined. These multiple object color standards can be determined in a variety of ways, including the aforementioned, including device settings (e.g., some displays have a known device feature of a yellow "nighttime" color rendering, such as reducing blue to prevent eye strain, or the user's personal device hue settings), user history (e.g., a user's previous selection of a device display to render colors with an unbalanced saturation (chrominance), hue, and luminance), seller-related history (e.g., the seller is known to capture items in a particular ambient light), and even current ambient conditions detected by a sensor (e.g., a dark room, fluorescent lighting, LED lighting, etc.). Based on these, a certain hue range (or other characteristic) is determined in step 1216. These may be the final or first step in the buyer's selection of object color standards captured by the device that best approximate the buyer's visual perception of the object color standard at the location. However, in some cases, the scope may be broad with little or no analysis, but rather provide a broad range of options for the buyer to choose from, and then refine based on previous selections (if the buyer desires further granularity, precision and / or similarity). It is also possible to incorporate it.

[0097] In step 1218, the technique presents several renditions of the determined modified physical color reference as options. For example, consider the example of FIG. 5, which shows five intensities selectable via user interface 502 (an example of user interface 118) of smartphone 102-3. Here, using user interface 502, a large range of intensities (in the human eye range) can be selected, for example, from intensities 504-1 to 504-5, simply by tapping on the display to select one of five physical color reference images, each with a different intensity.

[0098] The technique receives a selected rendition of the determined modified physical color standard in step 1220. In this example, the technique continues to receive a selection of images at various intensities 504 via user interface 502. In FIG. 5, assume that the buyer selects intensities 504-3 as the closest match to physical color standard 506 based on a visual observation of the physical color standard at buyer location 508. At this point, steps 1216, 1218, 1220, and 1208 can be performed one or more times with finer gradational selections (e.g., finer luminance range 510, which allows for a finer five-level luminance selection) or different selections. For example, after a luminance is selected, the technique may present differently colored options, such as with different hue balances (or other characteristics), at or near the selected luminance.

[0099] For example, referring again to Figure 6, nine different hue / luminance ranges are shown presented via user interface 602 (which is merely one example of user interface 118) of smartphone 102-3. Here, nine physical color reference images, each with a different hue and luminance, are presented on user interface 602, allowing a user to select one of the nine hue / luminance ranges 604 simply by tapping on the display. As shown, the left column has stronger reds, the right column has stronger blues, and the luminance increases toward the top.

[0100] The buyer can continue to do this by selecting different properties or properties (e.g., the brightness and hue grid shown in FIG. 6) in a single interface until they are satisfied with the match to the physical color criteria (e.g., 214, 222, 506). Alternatively, the buyer can manually change the properties through the interface (e.g., by voice commanding "increase brightness and decrease red," or by using slider control element 212, etc.).

[0101] Referring back to the method of FIG. 12 , the technique may optionally record any hue and brightness inputs and / or changes made by the buyer on the buyer's device at step 1210. This helps the application further improve performance for future items. Thus, as the buyer makes changes to the image, the application can learn what to change in the future. For example, if the buyer chooses to increase the red in the image and correlate this increase with data about the item image and data about ambient conditions, the application can learn from that change. This ongoing recording of the buyer's changes to the image can teach the application how to more appropriately modify future images when rendering them on the buyer's display. This same data can be provided to and received from other instances of the application to improve automatic corrections by the application. For example, another buyer may change the blue hue of an item image (of the same or a different item) by a specific amount under certain ambient lighting conditions on the buyer's device. In this case, if the ambient light of the buyer's device is similar (or if the original image is taken If the image is captured in a different color (e.g., if the contrast is similar between the image and the ambient light at the time of capture), such changes in hue are recorded and can be passed to and / or used by the buyer's application to automatically modify the image, although this data can also be shared at the buyer's option. Disclosure of additional information

[0102] The functionality provided by the present technique allows an image of a scene to be captured along with an established physical color standard (e.g., a color wheel, color bar, or 3D physical color standard showing tens to millions of colors, since the human eye can distinguish approximately 10 million different colors), allowing the image to be generated with colors that are truly representative of the real thing. For example, for image providers such as sellers selling products online or via catalogs, image module 114 can be used to adjust images of items to match the true or likely color tone, luminosity, and other factors of the item.

[0103] To do this, the image module 114 calculates the difference between the object color standard values ​​in the captured image and the correct object color standard values ​​stored in or accessible by the computing device 102. For example, suppose a scene containing a desired product and the object color standard is captured. The image can then be analyzed using the present technique to identify the object color standard. The color of the object color standard as it appears in the image may vary depending on ambient factors at the time of image capture. For example, the scene may be poorly lit, or the computing device's auto white balance (AWB) function may be distorted. The color appearing in the image of the object color standard can be compared to the color of an established object color standard stored in or accessible by an application (e.g., the image module 114) configured to execute the present technique. The comparison reveals a difference in hue values. Alternatively, or in addition, such differences can be recognized or determined by a rendering application or even rendering hardware, such as a graphics processing unit (GPU), that contains information about the color of each pixel rendered in the image. Differences in color characteristics at the rendition level are determined by comparing color information for pixels in the corrected image of the physical color standard with pixels renditions of the image capturing the physical color standard, where these pixels can be correlated to portions of each of these standards, such as by mapping in Cartesian coordinates.

[0104] For example, an object color standard in an image may contain a red hue value of R231 at a particular location (or pixel), while an object color standard established (or calibrated) in an application may specify the corresponding red hue at that location as R220. Thus, the application generates a -11 difference in the true color red hue of the image (and thus the item or scene). The application also generates hue differences for all object color standard values ​​until a representative image is generated. It also displays true color hue bars and luminance in real time, allowing the user to adjust the hue using these bars. Furthermore, spectral analysis of the light (e.g., with a light sensor) can be performed to ensure that the hue, luminance, etc. are correct or appropriate.

[0105] The technique provides a potential buyer with functionality that allows the user to visualize a more accurate and / or similar representation of the color of an item. The technique can read device display indicators (e.g., screen type, age, settings) and then analyze a seller-provided physical color standard. Comparing the device display indicators to the seller-provided physical color standard allows for an accurate representation of the color of an image. For example, the image module 114 can determine whether blue hues on the device are displayed with less intensity than red and green hues. Thus, an application configured to implement the technique can calculate and display an image that takes into account the device display settings and the seller-provided physical color standard. Alternatively, the application can, for example, take and display a photograph of the color and then allow the user to adjust or match the displayed color. The calibration sequence can also determine the color of the screen. The calibration sequence can be repeated until a representative color is depicted. This calibration can be used to modify the image in the manner described herein. Furthermore, when used on a wireless communication device (e.g., a smartphone, a tablet, etc.), the application allows a user to capture an image of a screen displaying an item (e.g., a product for sale). The application can also be used on the wireless communication device (e.g., a smartphone, a tablet, etc.) to modify the captured image based on physical color standards provided with the image and physical color standards established by the application (or based on physical color standards obtained by a buyer's visual observation instead of or after initial correction by the application).

[0106] Color has many characteristics, including hue, chroma (vividness or saturation), lightness, and brightness, to name just a few. Hue, also known as a "color appearance parameter," can be defined based on the intensity of a color stimulus or a single numerical value related to a chromaticity diagram (chromaticity coordinates) or color wheel (see Figure 2). Alternatively, hue can be defined based on the dominant wavelength or the wavelength of its complementary color. Chroma (vividness or saturation) is an attribute related to the perceived chromatic intensity of a color. These have been formally defined by the International Commission on Illumination, but such formal definitions are not required. More specifically, color vividness can depend on both spectral reflectance and illuminant intensity. Saturation refers to the vividness of a color in a given area, judged relative to its brightness. Brightness is also a color attribute related to the visual perception of emitted or reflected light. Thus, objects in low light are less bright. Brightness, sometimes called "color value" or "hue," refers to the manifestation of brightness. Various lightness models exist in the art, including the Munsell, HSL, or HSV color models. Those familiar with the field of color science will recognize that color characteristics sometimes overlap and are often based on user perception, resulting in user-specific variations. However, these variations can be addressed by the techniques described in this disclosure to create images of items or scenes that look good to a particular user (even if they do not look good to others).

[0107] Additionally, as previously mentioned and further described herein, the present techniques can utilize one or more characteristics of color to alter and improve the rendition of a color image. [Example]

[0108] Various embodiments will now be described. Example 1: providing a user interface for receiving user selections for modifying an image capturing a material color reference; receiving, via the user interface, a user selection for modifying an image capturing a material color reference; presenting a modified image capturing the material color criteria based on the user selection received via the user interface or another user interface; modifying an image of an item or scene captured at the same location as the image capturing the object color standard based on the received selection or modification to the image capturing the object color standard; and presenting or recording an altered image of the item or scene; A method comprising:

[0109] Example 2: presenting the modified image of the item along with the physical color standard through a user interface configured to receive manual modifications to the image based on a visual comparison of the photographed physical color standard with a physical color standard present at a particular location; receiving manual changes via the user interface; and applying further modifications to the modified item image based on the received manual modifications; 2. The method of example 1, wherein recording the modified image comprises recording the modified image along with the manual modifications.

[0110] Example 3: One or more computer-readable media comprising instructions that, when executed by one or more processors, accordingly perform the steps of the method of Example 1 or Example 2.

[0111] Example 4: A display unit; one or more computer processors; and one or more computer-readable media containing instructions that, when executed by the one or more processors, perform the steps of the method of Example 1 or Example 2 accordingly, and further present the modified image on the display.

[0112] Example 5: A mobile computing device comprising means for performing the steps of example 1, example 2, or example 3.

[0113] Example 6: A method for improving the accuracy or likeness of an image showing an item, receiving an image of an item along with an object color standard; comparing the image of the physical color standard with a copy of the physical color standard taken at a particular location; modifying the image of the item based on a comparison of the image of the physical color standard with a copy of the physical color standard captured at the particular location; and providing the modified image for presentation.

[0114] Example 7: A method for improving the accuracy or similarity of an image showing an item, receiving an image of an item along with an object color standard; presenting an image of the item along with the physical color standard through a user interface configured to receive manual modifications to the image based on a visual comparison of the photographed physical color standard with a physical color standard present at a particular location; receiving manual changes via the user interface; and providing the modified image for storage or presentation.

[0115] Example 8: One or more computer-readable media comprising instructions that, when executed by one or more processors, perform the steps of the method of Example 6 or Example 7 accordingly.

[0116] Example 9: A display unit; one or more computer processors; and one or more computer-readable media comprising instructions that, when executed by the one or more processors, perform the steps of the method of Example 6 or Example 7 accordingly, and further comprising: presenting the modified image on the display.

[0117] Example 10: A mobile computing device comprising means for performing the steps of example 6, example 7, or example 8.

[0118] Example 11: providing a user interface for receiving user selections for modifying an image capturing a material color reference; receiving, via the user interface, a user selection for modifying an image capturing a material color reference; presenting a modified image capturing the object color criteria based on the user selection received via the user interface or another user interface; modifying an image of an item or scene captured at the same location as the image capturing the object color standard based on the received selection or modification to the image capturing the object color standard; and and presenting or recording an altered image of the item or scene.

[0119] Example 12: The method of Example 11 or any other preceding example, wherein the user interface provides a plurality of selectable control elements, each of the plurality of selectable control elements being presented with a different representation of the physical color criterion, and wherein receiving the user selection includes selecting one of the control elements.

[0120] Example 13: The method of example 11 or any other preceding example, further comprising determining various representations of the physical color standard, the determining including: Correlating a portion of the image capturing the physical color standard with a matching color in a previously recorded image or copy of the physical color standard (the color having a known location in the previously recorded image or copy of the physical color standard); determining the actual color of the portion of the image capturing the physical color standard based on the color information recorded for the known location; and and determining different representations of the physical color standard based on the actual color of the part.

[0121] Example 14: The method of example 11 or any other example preceding it, wherein the matching colors have matching hue, chroma (vividness, saturation), lightness, or brightness.

[0122] Example 15: The method of Example 14, wherein a previously recorded image of a physical color standard or a copy of the physical color standard is determined by human-selectable comparison-based calibration for the purpose of calibrating a display unit presenting the user interface.

[0123] Example 16: The method of Example 11 or any other preceding example, further comprising modifying the image capturing the physical color standard based on the received selection before presenting the modified image capturing the physical color standard.

[0124] Example 17: The method of Example 16, wherein the image capturing the physical color reference and the image of the item or scene are present in the same captured image, and wherein the changes to the image capturing the physical color reference and the changes to the image of the item or scene are made simultaneously in the same process.

[0125] Example 18: A method of Example 11 or any other preceding example, characterized in that the user interface allows for receiving a user selection when the user manually makes a selection regarding a change to the color of an image capturing a physical color standard through the operation of a control element, and receiving the user selection for making a change to the image capturing the physical color standard includes changing the hue, saturation (vividness / saturation), lightness or brightness of the image capturing the physical color standard.

[0126] Example 19: The method of Example 11 or any other preceding example, further comprising recording the color change due to the change in the physical color standard, wherein recording the change effects future changes to the future image to present the image more accurately or in a more suitable manner.

[0127] Example 20: The method of Example 11 or any other preceding example, further comprising automatically applying changes to other images of other items or scenes, or other images of other items or scenes captured in the same location, wherein the automatic changes are based on differences between the image capturing the object color reference or a user selection and the modified state of the image capturing the object color reference.

[0128] Example 21: The method of Example 11 or any other preceding example, wherein the item or scene is a cosmetic product and the object color standard includes a range of human skin tones, or the item or scene is a clothing product and the object color standard includes a range of human hair and skin tones, or the item or scene is a landscape and the object color standard includes a range of colors found in an outdoor location.

[0129] Example 22: receiving an image of a physical color standard that includes known colors at known locations within the physical color standard; receiving an image of the item or scene, wherein the image of the item or scene and the image of the object color reference are the same image, or different images captured at the same time or in the same location; Correlating portions of items or scenes shown in the image of the items or scenes with matching colors in an object color standard (colors whose locations on the object color standard are known); determining the actual color of the portion of the item or scene based on the recorded color information for the known location; A method including recording the recorded color information and the location of that portion of the item or scene, and using such recording to effect accurate or similar color representation in future renditions of images of that item or scene.

[0130] Example 23: The method of Example 22, wherein the correlation of the item includes sampling multiple portions of the image of the item or scene with known colors of the object color standard, and when a match is found based on the sampling, the corresponding portion is the subject of correlation.

[0131] Example 24: The method of example 22 or any other example preceding it, wherein the method is performed automatically without user interaction.

[0132] Example 25: The method of Example 22 or any other preceding example, further comprising: rendering an image of the item or scene having the correct or suitable color based on the recorded color information and the location of the portion of the item or scene.

[0133] Example 26: The method of Example 22 or any other preceding example, wherein the correlation between the portion of the item or scene shown in the image of the item or scene and a matching color in the object color standard is by matching hue, chroma (vividness, saturation), lightness or brightness.

[0134] Example 27: The method of Example 22 or any other preceding example, wherein the known colors present at known locations within the physical color standard are determined by calibration based on human-selectable comparisons for the purpose of calibrating a display that presents a user interface.

[0135] Example 28: The method of Example 22 or any other preceding example, wherein the item or scene is a cosmetic product and the object color standard includes a range of human skin tones, or the item or scene is a clothing product and the object color standard includes a range of human hair and skin tones, or the item or scene is a landscape and the object color standard includes a range of colors found in an outdoor location.

[0136] Example 29: receiving images of items and scenes; receiving recorded color information and a location of a portion of an item or scene, wherein the recorded color information and the location of the portion on the item or scene indicate the exact color or a likely color of the portion at the location of the item or scene; determining a color difference between the color of the portion of the item or scene in the image of the item or scene and the recorded color information; modifying the image of the item or scene based on the determined color difference; and recording or staging an altered image of said item or scene.

[0137] Example 30: The method of example 29, wherein the recording or rendition of the modified image of the item or scene is a rendition of the modified image in an augmented reality interface.

[0138] Example 31: A method of Example 29 or any other preceding example, characterized in that the recorded color information and at the location indicate the exact color of the portion of the item or scene at that location, and further including receiving via a user interface a user selection to make further modifications to the rendering of the modified image of the item or scene that are effective to improve the suitability of the modified image of the item or scene to the ambient lighting conditions at the location based on object color criteria at the particular location from the user's perspective.

[0139] Example 32: The method of Example 29 or any other preceding example, further comprising receiving, via a user interface, a user selection to make further modifications to the rendering of the modified image of the item or scene that are effective to improve the accuracy or likeness of the rendering when the modified image of the item or scene is rendered on a display, based on a calibration of the display on which the modified image is rendered.

[0140] Example 33: The method of Example 29 or any other preceding example, further comprising automatically making modifications to a rendition of the modified image of the item or scene that are effective to improve the accuracy or likeness of the rendition when the modified image of the item or scene is renditioned on a display based on a calibration of the display on which the modified image is rendition. Conclusion

[0141] Although color rendering aspects have been described in terms of specific functional characteristics and / or methods, the subject matter of the appended claims is not necessarily limited to the specific functional characteristics and / or methods described above. Rather, the specific functional characteristics and / or methods are disclosed merely as examples of color rendering. Accordingly, other equivalent functional characteristics and / or methods are intended to be encompassed within the scope of the appended claims. Furthermore, while various different aspects have been described, it should be understood that each of the above aspects can be practiced individually or in conjunction with one or more of the other aspects described above.

Claims

1. providing a user interface for receiving user selections for making modifications to an image captured of a physical color standard, the image captured of the physical color standard being captured in a first ambient light, the user interface being configured to receive modifications to the image captured of the physical color standard based on a visual comparison of the image captured of the physical color standard to physical objects in a particular location based on the physical color standard; receiving, via the user interface, a user selection for modifying the image capturing the physical color standard, the user selection being based on a visual comparison of the image capturing the physical color standard with a physical object at a particular location, the physical object at the particular location being in a second ambient light, the second ambient light being different from the first ambient light; presenting, via the user interface or another user interface, a modified image capturing the physical color reference based on the received user selection, wherein the modified image capturing the physical color reference is enabled via the user selection to be more accurate or more representative of a second ambient light than the image capturing the physical color reference; modifying an image of an item or scene captured at the same location as the image capturing the object color standard based on the received selection or modification to the image capturing the object color standard; and presenting or recording an altered image of the item or scene, the altered image of the item or scene being more accurate or more representative of the second ambient lighting than the image of the item or scene; A method comprising:

2. providing the user interface to provide a plurality of selectable control elements, each of the plurality of selectable control elements being presented with a different representation of the physical color standard; receiving the user selection includes selecting one of the plurality of selectable control elements; and further comprising determining different representations of the physical color standard, the determining including: Correlating a portion of the image capturing the physical color standard with a matching color in a previously recorded image or copy of the physical color standard, the location of which in the previously recorded image or copy of the physical color standard is known; determining the actual color of the portion of the image capturing the physical color standard based on the color information recorded for the known location; and determining different representations of the physical color standard based on the actual color of the part; 2. The method of claim 1 .

3. 3. The method of claim 2, wherein a previously recorded image of a physical color standard or a copy of the physical color standard is determined by calibration, the calibration being based on a human-selectable comparison for the purpose of calibrating a display presenting the user interface.

4. A method according to any one of claims 1 to 3, characterized in that the image capturing the physical color reference and the image of the item or scene are present in the same captured image, and the changes to the image capturing the physical color reference and the changes to the image of the item or scene are made simultaneously in the same process.

5. 5. The method according to claim 1, wherein the user interface is capable of receiving a user selection when the user manually selects a color change for the image capturing the physical color standard through the operation of a control element, and the receiving of the user selection for the change for the image capturing the physical color standard includes changing the hue, chroma (vividness / saturation), lightness or brightness of the image capturing the physical color standard. method.

6. 6. The method of claim 1, further comprising recording color changes due to changes in the physical color standard, wherein recording the changes effects future modifications to the future image to present the image more accurately or in a more suitable manner.

7. 7. The method of claim 1, further comprising automatically applying modifications to other images of other items or scenes, or other images of other items or scenes captured at the same location, the automatic modifications being based on differences between the image capturing the object color reference or a user selection and the modified state of the image capturing the object color reference.

8. 8. The method of any one of claims 1 to 7, wherein the item or scene is a cosmetic product and the physical color standard comprises a range of human skin tones, or the item or scene is a clothing product and the physical color standard comprises a range of human hair and skin tones, or the item or scene is a landscape and the physical color standard comprises a range of colors found in outdoor locations.

9. A method according to any one of claims 1 to 7, characterized in that the item or scene is a cosmetic product, and presenting or recording the modified image of the item or scene comprises presenting the modified image of the item or scene within an interface having an image of the user's hair, skin or face, the cosmetic product, a portion of the cosmetic product, or the color of the cosmetic product.

10. The method of claim 1 , wherein the physical object at the particular location is a physical color standard.

11. 10. The method of claim 1, wherein the physical object at the particular location is a part of the user's anatomy, and the part of the user's anatomy is associated with a particular position of the physical color reference.

12. The method of claim 11 , wherein the portion of the user's anatomy is the user's skin.

13. A method according to any one of claims 1 to 12, wherein presenting or recording the modified image of the item or scene comprises presenting the modified image of the item or scene within an augmented reality interface.

14. A method according to any one of claims 1 to 13, wherein presenting or recording a modified image of an item or scene comprises presenting or recording a modified image of an item or scene in real time.

15. A system comprising means for carrying out the method according to any one of claims 1 to 14.

16. a computing device, A display unit; a processor; a computer-readable storage medium containing instructions that, when executed by a processor, accordingly cause the computing device to perform the method of any one of claims 1 to 14; A computing device comprising:

Images