Method and system for maintaining color calibration using common objects

Abstract

Systems and methods for maintaining color calibration using common objects are described herein. In an exemplary embodiment, an AR system includes a forward camera, an AR display, a processor, and a user interface. The processor is configured to receive image data from the forward camera and identify any known object depicted in the image data. The processor then determines at least one piece of test rendered RGB information of the identified known object, and displays the RGB information through the AR display. Input from the user interface is received by the processor and used to update an AR display color calibration model, wherein the input indicates which one of the at least one test renders is a closest match to the real-world object and indicates a level of satisfaction with the match. More test renders may be provided iteratively to improve the accuracy of the calibration.

Description

This application is a divisional application of Chinese Patent Application No. 201880084415.8, titled "Method and System for Maintaining Color Calibration Using Common Objects", with a filing date of December 21, 2018. The content of the parent application is incorporated herein by reference in its entirety. Cross - Reference to Related Applications

[0001] This application is a non - provisional application and claims the benefit of priority under 35 U.S.C.§119(e) to: U.S. Provisional Patent Application Serial No. 62 / 612,140, titled "Method and System for Maintaining Color Calibration Using Common Objects", filed on December 29, 2017, which is incorporated herein by reference in its entirety. Background of the Invention

[0002] The purpose of color calibration is to measure and / or adjust the color response of a device (input or output) to a known state. In International Color Consortium (ICC) terminology, this is the basis for the additional color characterization and subsequent profiling of the device. In non - ICC workflows, calibration sometimes refers to establishing a known relationship with a standard color space on a one - time basis. Color calibration is a requirement for all devices that are part of an effective color - management workflow.

[0003] Input data can come from device sources such as digital cameras, image scanners, or any other measurement device. These inputs can be monochromatic (in which case only the calibration response curve is required, although in some selected cases, the color or spectral power distribution corresponding to that single channel must also be specified) or specified in multi - dimensional color (most commonly in the three - channel RGB model). In most cases, the input data is calibrated against the Profile Connection Space (PCS).

[0004] Color calibration is used in many industries such as television production, gaming, photography, engineering, chemistry, medicine, etc.

[0005] Traditional computer monitors require separate characterization when used in applications that require the most precise color reproduction. Augmented reality systems would similarly benefit from separate calibration to ensure an accurate and believable presentation of the mixed content (real - world and synthetic imagery). Augmented reality systems may require even better accuracy because the real - world and synthetic content are, by definition, in the same scene and are typically adjacent in the field of view. This adjacency presents the worst - case scenario for color matching, so the strictest color reproduction is advantageous.

[0006] Frequent calibration of an augmented reality display system is beneficial for maintaining the highest level of color reproduction accuracy. If the user needs to view color charts or other traditional characterization targets to maintain this high precision, the user experience will be diminished. One-time factory calibration is not sufficient because it cannot account for changes in the display over time. Summary of the Invention

[0007] A calibrated forward camera or spectrometer continuously captures image data of the real-world scene of an augmented reality (AR) system. In some embodiments, the camera system (or alternatively, a second camera system) is inside the AR head-mounted device (e.g., AR goggles), and can also detect inserted synthetic images. A calibration process (such as eye tracking) can be used to align the camera image with the image seen by the observer. A processor controlling the AR system communicates with a color database of known objects, such as products, logos, or even natural artifacts, such as grass and the sky. When an object from the database is recognized in the real-world field of view, the processor uses at least one of the following two methods to recalibrate the display: 1) With user intervention: Present the user with two versions of the known object (e.g., display one version on either side of the real-world object). The user selects the closest visual match. This process can be iterated. 2) Without user intervention: The processor displays the best current estimate of the matching color overlaid on the real-world object. The inner camera captures the scene, and the processor compares the color of the real-world object with the (synthetic) estimate. The measured color and the desired color are compared, and if their values are within a threshold color difference, the calibration is complete. This process is fully automated. The color difference threshold can be a built-in default value or set / controlled by the user (e.g., depending on their specific application).

[0008] The recalibration of the AR display is operable to improve the color accuracy of the synthetic image. The fully automated embodiments disclosed herein are not mandatory for the user. Embodiments with user intervention provide color calibration results tuned to the user's specific color vision. Brief Description of the Drawings

[0009] Figure 1 Depicts an overview of a processing flow for identifying common objects and calculating their colors according to at least one embodiment.

[0010] Figure 2 Depicts a chart relating various usage scenarios and their conditions to applicable processes according to at least one embodiment.

[0011] Figure 3Depicts an overview of a color calibration process for user engagement in a traditional AR display system according to at least one embodiment.

[0012] Figure 4 Depicts an overview of a color calibration process for user engagement in an AR display system including an internal measurement imaging device according to at least one embodiment.

[0013] Figure 5 Depicts a sequence diagram of a process for user engagement to maintain color calibration using a common object in the field of view according to at least one embodiment.

[0014] Figure 6 Depicts an overview of a fully automatic color calibration process for an AR display system including an internal measurement imaging device according to at least one embodiment.

[0015] Figure 7 Depicts a sequence diagram of a fully automatic process for using a common object in the field of view to maintain color calibration according to at least one embodiment.

[0016] Figure 8A Depicts a visual overview of a first example scenario involving a traditional HMD according to at least one embodiment, and Figure 8B Depicts the corresponding AR color calibration interface view presented to the user.

[0017] Figure 9A Depicts a visual overview of a first example scenario involving an HMD with an internal measurement camera according to at least one embodiment, and Figure 9B Depicts the corresponding AR color calibration interface view presented to the user.

[0018] Figure 10A Depicts a second example scenario involving a traditional AR display according to at least one embodiment, and Figure 10B Depicts the corresponding AR color calibration interface view presented to the user.

[0019] Figure 11A Depicts a second example scenario involving an AR display with an internal measurement camera according to at least one embodiment, and Figure 11B Depicts the corresponding AR color calibration interface view presented to the user.

[0020] Figure 12A And 12B Is a schematic diagram of an AR display according to an exemplary embodiment.

[0021] Figure 13 Illustrates an exemplary wireless transmit - receive unit that can be used to implement an AR display in some embodiments. Detailed Description Abbreviation AR Augmented Reality 3D Three-Dimensional HMD Head-Mounted Display FOV Field of View RGB Red-Green-Blue (color pixel) Introduction

[0022] This disclosure presents methods and systems for maintaining color calibration using common objects. Such methods and systems can be implemented as processes occurring in an AR system, such as an AR HMD or an AR server, and as the AR system itself. Various embodiments take the form of programmatic methods. In the embodiments described herein, a calibrated forward camera or spectrometer continuously captures image data of the real-world scene of the AR system. In some embodiments, the camera system (or alternatively, a second camera system) is inside the AR glasses and can detect the real-world scene as well as the inserted synthetic image. The processor in charge of control of the AR system communicates with a color database of known objects, such as products, logos, or even natural artifacts such as grass and sky. When an object from the database is recognized in the real-world field of view, the processor recalibrates the display. In Figure 12A and 12B exemplary configurations of AR systems in some exemplary embodiments are shown. In these diagrams, the camera is mounted behind the display surface of the glasses but outside the observer's field of view (e.g., above, below, or to the side of the observer's field of view). In some embodiments, the AR system is provided with eye tracking to align the camera image with the user's field of view. Advantages

[0023] The recalibration of the AR display helps improve the color accuracy of the synthetic image. The fully automated embodiments disclosed herein are not mandatory for the user, while embodiments including user feedback provide a means for tuning the display for the user's specific color vision.

[0024] Before proceeding with the detailed description, note that the entities, connections, arrangements, etc., depicted in the various figures and described in connection with the various figures are presented as examples and not as limitations. Accordingly, any and all statements or other indications regarding what is "depicted" in a particular figure, what a particular element or entity "is" or "has" in a particular figure, and any and all similar statements - which may be isolated and construed out of context as absolute and thus limiting - can only be properly construed with a clause such as "in at least one embodiment" constructively added in front of them.

[0025] In addition, any variations and permutations described in subsequent paragraphs and anywhere else in this disclosure can be implemented with respect to any embodiment, including with respect to any method embodiment and with respect to any system embodiment. Exemplary color database

[0026] The exemplary methods described herein utilize a database that can identify objects and their color or spectral properties. Prior to performing the disclosed methods, the database is created or identified and its contents are made available to the devices or processes embodying the teachings herein. The database can be constructed to include data used by one or more known object recognition techniques. The data can be collected in a manner that takes into account the fact that objects can be imaged from unknown viewpoints and under unknown and / or complex illumination both spectrally and spatially. For the sake of balance in this disclosure, exemplary methods are described for cases involving diffuse illumination and directional detection. However, such conditions are not meant to be limiting in any way, as the database can be readily extended to include more complex illumination conditions. The omission of references to more complex scenarios is for the sake of brevity and clarity.

[0027] The data in the database of identifiable objects and their color or spectral properties can be obtained using several means, including: actual measurements of specific materials; estimates from product trade literature (e.g., Pantone colors); other databases (for traditional materials such as grass, sky, brick, skin, etc.); and so on. In at least one embodiment, the data set includes color coordinates (CIELAB or others). In some embodiments, spectral reflectance data can be captured. The data for certain materials or products can be measured by the entity planning to implement the process. In such cases, the entity planning to implement the process may or may not make the data publicly available. The attributes associated with each object can include one or more of the following: · Spectral reflectance coefficient. In some embodiments, the spectral reflectance coefficient is data measured under known standard reference conditions, such as measurements at known illumination and detection angles, such as bidirectional reflectance (e.g., illumination at 45° and measurement at 0°) or hemispherical reflectance measurements. In some embodiments, the bidirectional reflectance distribution function (BRDF) or more generally the bidirectional scattering distribution function (BSDF) is used to characterize · Fluorescent behavior. In some embodiments, fluorescence can be characterized by the Donaldson matrix as described in R. Donaldson, Spectrophotometry of fluorescent pigments, Br. J. Appl. Phys. 5 (1954) 210-214. The matrix can be determined using appropriate measurement equipment and fully characterizes the spectral reflectance as a function of the wavelength of the incident light. In some embodiments, fluorescent information can be determined based on the material properties of the identified object. For example, fluorescent paper typically exhibits fluorescent behavior similar to that of ordinary "daylight" fluorescent security objects. · Glossiness. Glossiness data can be data collected by a gloss meter under a set of standard reference conditions. Depending on the object, the specification can be 80° glossiness (for diffuse materials); 60° glossiness (semi-glossy materials) or 20° glossiness (glossy materials). Other angles are possible, but these angles are the most common. · Logo font.

[0028] In some embodiments, the database can be extended by storing the measured properties of new objects encountered by the user. In this case, a validation mechanism can be applied because there is no ground truth color on which any calibration is necessarily based. In this case, it is appropriate to ask the user to ensure that the calibrated color match is sufficient. Then, the color can be estimated by the front camera and the calibration model. For better ground truth color data, an integrating sphere spectroradiometer can be used to measure the data and / or an integrated spectroradiometer can be included in the AR HMD device. Detailed processes and structures

[0029] A set of processes and devices corresponding to various usage scenarios are disclosed herein. Figure 1 Relates to processes and depicts an exemplary processing flow 100. Figure 1 Depicts a processing flow start 102, which includes using a front camera to identify a common object and calculate its actual color within the current AR scene.

[0030] In process 100, a calibrated forward camera images a current field of view to generate color image data and detect the wearer's field of view 104. In some embodiments, the forward camera is mounted to the AR HMD. In some embodiments, the forward camera is embedded within the AR HMD. The color image data is received 106 at a processor (either in the HMD or an AR server), and at step 108, the processor uses the data to identify objects in the scene that match objects in a known object database. Process 100 also includes determining 116 the current scene illumination incident on the identified objects at least in part by using the image data received at the processor. The determined scene illumination, along with color or spectral properties obtained from the known object database, is used by the processor to calculate the actual color of the identified objects under the lighting conditions in the current scene. Step 110 includes retrieving the properties of any known objects. It should be noted that the actual color here is only a function of the known object properties obtained from the database and the real-world illumination estimated using the image data. In at least one embodiment, the real-world illumination is estimated at least in part by comparing the image data from the forward camera with the properties obtained from the known object database. Step 112 provides the processor to calculate the actual color of any known objects in the scene. At this point, the "actual color" only takes into account the real-world illumination and the object properties. Thus, this is the color of the light incident on the glasses (HMD) after reflection from the object. The properties of the glasses have not been considered. Therefore, the display / rendering properties of the AR device worn by the user are not involved in determining the actual color of the object.

[0031] In Figure 2 The usage scenarios 1 - 3 are outlined in the diagram depicted Figure 2 200 depicts a diagram relating various usage scenarios and their conditions to applicable processes according to at least one embodiment. The figure shows which usage scenarios are applicable based on whether there is user intervention and whether there is an image sensor (inner measurement device) within the field of view of the HMD display surface.

[0032] Usage scenario 1 corresponds to an AR HMD device that does not have a camera sensor within the field of view of the AR display surface. The user interacts with the AR HMD to request and control color calibration. The process 300 applicable to usage scenario 1 in diagram 20 will be discussed in the following Figure 3 description.

[0033] Both usage scenario 2 and usage scenario 3 correspond to AR HMD devices that have a camera sensor within the field of view of the AR display surface. If Figure 1 the forward camera has a field of view (FOV) that includes the AR display surface, then the camera sensor can be with respect toFigure 1 The forward camera under discussion. Alternatively, for example when Figure 1 the forward camera does not include the position / orientation of the AR display surface in its FOV, in addition to Figure 1 the forward camera, the camera sensor can be included in the HMD. In Use Case 2, the user interacts with the AR HMD to request and / or control color calibration, and moreover, the camera sensor located within the field of view of the AR display surface sends image data to the processor, which correlates this data with the user input data for tuning the automatic color calibration model. Figure 3 Process C applicable to Use Case 2 is outlined in. In Use Case 3, there is no user interaction or intervention. The color calibration process is fully automated by using the automatic color calibration model.

[0034] Regarding Figure 6 the process for Use Case 3 is described.

[0035] Note that nothing precludes a particular AR system from using multiple of the above methods, and in fact, a large number of possible embodiments not listed for the sake of brevity may include various combinations of certain elements from these processes. For example, the fully automated method associated with Use Case 3 can run continuously in the background, and if the user senses that the color reproduction is not optimal via the user interface of the AR device, the user can trigger manual calibration. Then, knowing that they do not want visual disruptions in the FOV during an upcoming moment, the user can re - engage in the fully automated process. The processes outlined in Table 200 can be used sequentially to first tune and then run the fully automated AR color calibration process.

[0036] Figure 3 A summary of the color calibration process for user engagement in a conventional AR display system according to at least one embodiment is shown. Figure 3 The AR display system used in is consistent with Use Case 1; it does not include an internal measurement imaging device for viewing the content presented on the AR display surface (i.e., it is a conventional AR display system and thus does not include an inner camera). Figure 3 The process of is shown as Flowchart 300.

[0037] In process 300, step 302 shows the actual color of a known object within a scene. At step 304, the actual color of the known object in the AR scene is input into an inverse display model to convert the color data into RGB data. Thereby, at step 306, the RGB data of the known object is estimated. Next, the process includes selecting alternative (nearby) RGB coordinates as test colors 308. This selection can be performed by a processor using a test color generation algorithm. The number of selected alternative coordinates may be unrestricted, except to maintain a reasonable interface for the user at step 310. A larger number of selected alternative coordinates provides faster convergence towards a preferred display calibration. Next, the process includes rendering the known object using each selected test color at step 312 and visually displaying each rendering near the known object. In some embodiments, the entire known object may be rendered. In some embodiments, only a representative portion of the known object is rendered. In various embodiments, the amount of the known object that is rendered is based on the size of the known object. At step 314, the process then prompts the user to select which of the displayed renderings is the closest visual match to the known object. In at least one embodiment, step 314 includes a further prompt requesting the user to select the degree of closeness (i.e., acceptability) of the match. If, at decision 316, the user is not satisfied with the degree of closeness of the match, process 300 includes updating the estimate 318 of the RGB data of the known object based on the RGB data of the preferred rendering, and process 300 repeats starting from the corresponding step 306. If the user is satisfied with the degree of closeness of the match, then process 300 includes: at step 320, updating the display model using the RGB data of the selected rendering and the actual color of the known object. Then, process 300 ends, and Figure 1 process 100 in

[0038] Figure 4 shows an overview of a user - involved color calibration process for an AR display system including an internal measurement imaging device according to at least one embodiment. Figure 4 The AR display system used in Figure 4 is consistent with usage scenario 2; it does include an internal measurement imaging device (e.g., an inward camera) for viewing content rendered on an AR display surface.

[0039] In process 400, at step 402, the actual color of a known object in the AR scene (as determined by process 100) is input into the inverse display model to convert the color data to RGB data. In this way, the RGB data of the known object is estimated through the inverse display model (color to RGB) 404 and the RGB 406 of the estimated known object. Next, process 400 includes selecting alternative (nearby) RGB coordinates as test colors 408. The number of selected alternative coordinates is not limited, except to maintain a reasonable interface for the user. A larger number of selected alternative coordinates provides faster convergence towards a preferred display calibration. Next, process 400 includes rendering the known object 410 using each selected test color and visually displaying each rendering near the known object at step 414. In some embodiments, the entire known object may be rendered. In some embodiments, only a representative portion of the known object may be rendered. In various embodiments, the amount of the known object rendered is based on the size of the known object. At step 414, the user selects which of the displayed renderings is the closest visual match (i.e., is preferred) to the known object. In at least one embodiment, the prompt further requests the user to select the degree of closeness (i.e., acceptability) of the match at decision 416. If the user is not satisfied with the degree of closeness of the match, at step 418, process 400 includes updating the estimate of the RGB data of the known object based on the RGB data of the preferred rendering, and process 400 repeats starting from the corresponding step. If the user is satisfied with the degree of closeness of the match, then at step 420, process 400 includes updating the display model using the RGB data of the preferred rendering and the actual color of the known object. Process 400 differs from process 100 in that: process 400 further includes updating the inner camera model at step 422 using the actual color of the known object and the RGB data of the preferred rendering. Then, process 400 ends, and process 100 resumes at step 104, as indicated by the identifier "A".

[0040] Figure 5 Sequence diagram 500 depicts a sequence of processes involved by a user for maintaining color calibration using a common object in the field of view, according to at least one embodiment. Figure 5Shows operations performed by an image processing component 502, an AR display component 504, and a user component 506. The image processing component estimates the current lighting conditions 508. Real-world objects are visible to the user through the AR display 510. The image processing component captures an image of the real-world objects 512. Then, the image processing component identifies the real-world objects and their colors. The image processing component calculates a candidate color rendering 516 and sends the rendering to the AR display component 518. The AR display component displays the synthetic content (i.e., the rendering) 520. The synthetic content is visible to the user via the AR display 522. Then, the user compares the candidate rendering with the real-world object in step 524 and selects a preferred rendering 526. The user selection is sent to the image processing component, where, at 528, the image processing component uses the feedback to update the display color model.

[0041] Figure 6 Shows an overview of a fully automated color calibration process 600 for an AR display system including an internal measurement imaging device according to at least one embodiment. Figure 6 The AR display system used therein is consistent with usage scenario 3 shown in Table 200; it may include an internal measurement imaging device (e.g., an inner camera) for viewing content rendered on the AR display surface.

[0042] In process 600, the actual color 602 of a known object in the AR scene (as determined in process 100) is input into the inverse display model to convert this color data into RGB data 604. Thereby, in step 606, the RGB data of the known object is estimated. Next, the process includes, in step 608, selecting alternative (nearby) RGB coordinates as test colors. This selection can be performed by the processor using a simple test color generation algorithm. The number of selected alternative coordinates is not limited, except to maintain a reasonable interface for the user. A larger number of selected alternative coordinates provides faster convergence towards the preferred display calibration. Next, the process includes, in step 610, using each selected test color to render a portion of the known object, and in step 612 visually displaying each rendering near the known object. In embodiments where multiple alternative colors are displayed, the processor selects different regions of the known object to be rendered. Process 600 then includes: in step 614, the inward camera detects the colors of the known object and all rendered segments. In step 616, the processor selects the alternative RGB data that most closely matches the color of the known object. In decision 618, it is determined whether the color match is less than a target maximum color difference. If the match is not close enough, in step 620, process 600 includes updating the estimate of the RGB data of the known object based on the RGB data of the preferred rendering, and process 600 repeats starting from the corresponding step 606. If the match is close enough, then process 600 includes updating the display model using the actual color of the known object and the RGB data of the selected rendering. Then process 600 ends, and process 100 resumes at step 104.

[0043] The steps listed below are a supplementary description of process 600.

[0044] Step 1: Identify candidate objects within the field of view:

[0045] The processor determines whether an object in the current field of view of the forward camera is in the known object database by using an image processing algorithm. This algorithm can consider any of the previously listed attributes. In some embodiments, several of these attributes are compared. For example, after detecting a bright red object, the number of potential objects in the database can be greatly reduced, and then a second attribute can be compared, and so on. The literature on object detection and recognition is very extensive, and any number of publicly available methods can be applied and potentially combined to achieve the necessary performance level for a given AR application.

[0046] Step 2: Estimate the spectral power distribution SPD of the illumination:

[0047] The estimation of the SPD can be performed by many methods established in the literature. The forward camera or other components can be used for accurate estimation of the current illumination. In some embodiments, the techniques described in Cheng, Price, Cohen, and Brown's "Effective Learning-Based Illuminant Estimation Using Simple Features" (IEEE CVPR2015) can be used to perform the illumination estimation.

[0048] Step 3: Estimate the displayed RGB coordinates of the known object:

[0049] The estimation of the effective reflectance of the object can be done by using the forward camera and considering the illumination estimated in Step 2 and the known reflectance of the object from Step 1. An exemplary estimate (radiance) of the light reaching the observer is the product of the illumination and the reflectance. If more complex geometric properties are available (e.g., BRDF), they can be applied here to improve the estimate of what the observer sees. The radiance calculated in this way does not yet take into account the glasses properties. After applying the transmittance of the glasses, the resulting radiance is a useful estimate of the light incident on the observer's eyes and the inner camera. Using the estimated light source and the known CIE transformation, this spectral radiance can be converted to a color. This color can be processed through the inverse display model to estimate the RGB required to match the color.

[0050] Step 4: Process the calibration and update the display color reproduction model:

[0051] Before performing a single recalibration, several objects can be identified and their properties estimated. For each object, the color coordinates are mapped to the estimated RGB coordinates of the AR display. The estimated RGB coordinates are displayed by the AR glasses inside or near the object, and the forward camera detects the colors of both. The difference between the target color and the actual color is determined for several objects, and then the display model is updated. This process can be repeated as needed until the finally estimated color and the measured color are below the color difference threshold. The color distance threshold can be a predefined value or a user-adjustable value. The website of the International Color Consortium (ICC), www.color.org / displaycalibration.xalter, is a useful reference for camera calibration techniques. The ICC has established various methods for calibrating displays and has outlined the processes and communications for operating such calibrated displays.

[0052] The rendering of accurate colors is limited and complicated by the transparent nature of the AR glasses. Techniques that account for these complications are described below.

[0053] Step 5: Update the object color and potentially iterate:

[0054] First, the system updates the RGB coordinates of the object, redisplay the rendering, and re-images the rendering and the real-world object with the camera. Then, the system checks the color difference between the rendering and the target object color. If the color difference is below the threshold, the process is complete. If the color difference is above the threshold, the process repeats steps 4 and 5.

[0055] Figure 7 Depicts a sequence diagram 700 of a fully automated process for maintaining color calibration using a common object in the field of view according to at least one embodiment. Figure 7 Includes an image processing component 702, an AR display component 704, and an internal camera component 706. The image processing component 702 estimates the current lighting conditions 708. The real-world object is visible to the internal camera component through the AR display 710. The image processing component captures an image 712 of the real-world object. Then, the image processing component identifies the real-world object and its color 714. The image processing component calculates a candidate color rendering and sends the rendering to the AR display component 716. The AR display component displays the synthetic content (i.e., the rendering) 720. This synthetic content is visible to the internal camera component 722. An image of the displayed object is provided to the internal camera 722. Then, the internal camera component captures / records images 724 of various candidate renderings and the real-world object, both of which are visible at the AR display. The image of the AR display is sent to the image processing component 726, which uses this feedback to update the display color model 728.

[0056] Figure 8A Depicts a visual overview 800 of a first example scenario involving a traditional HMD according to at least one embodiment, and Figure 8B Depicts a corresponding AR color calibration interface view 850 presented to the user. Figure 8A The overview contains a viewer 806, an AR visor 804, an overlay of synthetic content visible to the viewer and displayed by the AR visor 804, and the identified real-world object 802 that the viewer can also see through the AR visor 804. The AR visor 804 is part of an HMD that performs the color calibration process taught herein. The AR system presents to the viewer Figure 8BThe AR color calibration interface 850. At this time, the composite content overlay includes the AR color calibration interface. In some embodiments, the interface includes a prompt to select the closest color or a preferred option and at least one candidate RGB color option in the form of a rendering of the identified real-world object. In other embodiments, the interface includes a set of candidate RGB color options as a rendering of the identified real-world object. Figure 8B An interface is shown that allows a user to select the color closest to a "real" object 854 by selecting option A 852 or option B 856. The AR system receives viewer input identifying the selected option via the user interface. The AR system can iterate the calibration process by presenting new options to the viewer and receiving new viewer input. In each iteration, the display calibration model is updated using the color parameters selected by the viewer compared to the known parameters of the identified real-world object. In one embodiment, the display color calibration model can be adjusted such that when the AR system renders a composite content overlay including the known colors of the identified real-world object, the RGB values of the selected option are used to render those colors. Figure 8A - 8B Embodiments of the systems and methods of the present disclosure involving a conventional HMD are described. The conventional HMD does not include a camera that can view the composite content overlay displayed by the AR mask. Such a system can perform the above processes 100 and 300.

[0057] Figure 9A Depicts a visual overview of a first example scenario involving an HMD with an internal measurement camera according to at least one embodiment, and Figure 9B Depicts a corresponding view of the AR color calibration interface presented to the user. Figure 9A The overview shows the identified real-world object 902, the AR mask 904 of the HMD, the composite content overlay visible to the viewer and displayed by the AR mask 904, and the identified real-world object 902 that the viewer can also see through the AR mask 904. The HMD includes an inner camera 906 that can view a combined image of the composite content overlay at the AR mask 904 and the identified real-world object. Such an AR system presents to the viewer Figure 9B The AR color calibration interface. At this time, the composite content overlay includes the AR color calibration interface. In some embodiments, the interface includes a prompt for the user to select the closest color or a preferred option and at least one candidate RGB color option in the form of a rendering of the identified real-world object. In other embodiments, the interface includes only a set of candidate RGB color options as a rendering of the identified real-world object. In such as Figure 9BIn the fully automated embodiment depicted, the interface 950 includes a single color option rendering, and the inner camera images the real-world object and the color option rendering. This image data is sent to the processor of the HMD to be used as input for updating the AR display color calibration model. The AR system can iterate the calibration process by presenting new options to the inner camera and receiving new image data as input. In each iteration, if the color match between the rendered option 954 and the real-world object 952 determined by an image analysis algorithm using the inner camera image data is better, the color parameters of the rendered option are used to update the display calibration model. In one embodiment, the display color calibration model can be adjusted such that when the AR system renders a composite content overlay including the known colors of the identified real-world objects, the RGB values of the selected option are used to render those colors. Figure 9A - 9B Embodiments of the systems and methods of the present disclosure include an HMD having an inner camera capable of viewing an AR mask. Such systems can perform the processes 100, 300, 400, and 600 described above.

[0058] Figure 10A Depicts a second example scenario 1000 involving a conventional AR display according to at least one embodiment, and Figure 10B Depicts a corresponding AR color calibration interface view 1050 presented to the user. Figure 10A The example scenario described includes an observer 1002 viewing an object 1012 on a table through an AR display (e.g., a glass see-through AR mask) 1004. A light source 1006 provides incident light that is reflected by the object 1008 and propagates towards the observer 1010. The reflected light passes through the AR display 1004.

[0059] In Figure 10A - 10B the object is a Rubik's cube. Figure 10B Describes the view 1050 of the observer through the AR mask. The observer's view includes the real-world Rubik's cube 1052, two composite renderings 1054, 1056 of the Rubik's cube each having a slightly altered color, and a composite rendering of a question mark prompting the observer to select which composite Rubik's cube more closely matches the real-world Rubik's cube. The question mark is inserted into the image to indicate to the observer that a choice needs to be made. The Rubik's cubes 1054 and 1056 represent composite images with slightly altered colors.

[0060] Figure 11A Depicts a second example scenario 1100 involving an AR display 1104 having an internal measurement camera 1106 according to at least one embodiment, and Figure 11B Depicts a corresponding AR color calibration interface view presented to the user. Figure 11AThe example scenario 1100 described in Figure 11A - 11B includes an observer viewing an object 1114 on a table through an AR display (e.g., a see-through glass AR headset 1104). A light source 1108 provides incident light that reflects from the object 1110 and propagates towards the observer 1112. The reflected light passes through the AR display 1104 and is viewed by the observer 1102 and the inward-facing camera 1106. In Figure 11B the object is a Rubik's Cube.

[0061] Now referring to Figure 12A and 12B exemplary configurations of AR systems 1200 and 1250 are shown. In the schematic diagrams, cameras 1202 and 1252 are respectively mounted behind the display surfaces of the glasses but outside the observer's field of view (e.g., above, below, or to the side of the observer's field of view). In some embodiments, the AR system is provided with eye tracking to align the camera images with the user's field of view. In Figure 12A it is shown that the camera is coupled to a control module 1204, which is coupled to a waveguide 1206 and an LCD 1208. Similarly, in Figure 12B the camera 1252 is coupled to a control module 1254 as is the LCD 1256. Figure 12B A partially reflective surface 1258 is also shown in Rendering display colors in the presence of an environmental background

[0062] Traditional color display models relate device RGB coordinates to the output radiance (or color) of a display. More advanced models also take into account environmental room conditions (flare). AR glasses present additional complications caused by spatially varying ambient light from the scene that passes through the glasses and can be observed by or overlap with the AR display imagery by the user. Exemplary embodiments can address this problem as follows.

[0063] Consider an embodiment where there is an outer front camera that detects the ambient light seen by an observer, and a processor is operable to determine the spatial relationship between the ambient light and an internal AR display. As a result, the processor can access the aligned radiance or color of the light seen by the observer at each pixel location in the AR display. Since the AR display is aligned with the real world, the following spatial coordinates x, y are used for the two systems.

[0064] According to a traditional display model, the spectral radiance L can be calculated such that the AR display faces the observer from a given pixel x, y and input R, G, B color coordinates:

[0065] Note that the subscript λ indicates that the parameter is spectrally quantified. It should also be noted that the display model f1 is independent of the position on the display. The contribution from the ambient light is based on the camera model f2:

[0066] Again, note that the camera model f2 is independent of the position of the pixel. represents the spectral transmittance of the display at a given pixel. The total radiance seen by the observer is the sum of these two parts:

[0067] Thus, the input color to the final display model is operable to account for environmental effects. This will place some limitations on the available display colors. Even when the AR display is completely off, the ambient light passing through the glasses will impose a lower limit on the radiance that can be presented to the observer.

[0068] In some embodiments, for color-critical applications, the observer can be instructed to keep the viewing point free of bright real-world regions. In fact, the benefit of guiding the observer to a dim area depends on the type and quality of the light blocking available in a particular pair of glasses. In some embodiments, the AR system includes a technique that allows light to pass completely where there is no AR image and blocks light completely anywhere there is an AR image. In such embodiments, for the regions where the light is completely blocked, the value of the ambient radiance as described above can be zero. Some variations of the solution

[0069] Figure 2 Table / Chart 200 in provides three exemplary scenarios in which the teachings disclosed herein can be applied. Other combinations regarding user input and imaging capabilities can also be considered.

[0070] In more complex embodiments, the database of object attributes includes bidirectional reflectance distribution functions for some or all of the objects. Utilizing this data improves the accuracy in estimating the effective reflectance of a given object. However, this is a computationally much more expensive application because it takes into account the various directional aspects of the real-world lighting of the object. Other discussions

[0071] Figure 13 is a system diagram showing an exemplary wireless transmit receive unit (WTRU) 1302 that can be used as a head-mounted AR display in an exemplary embodiment. As Figure 13 shown, the WTRU 1302 can include a processor 1318, a transceiver 1320, a transmit / receive component 1322, a speaker / microphone 1324, a numeric keypad 1326, a display / touchpad 1328, a non-removable memory 1330, a removable memory 1332, a power supply 1334, a global positioning system (GPS) chipset 1336, and / or peripheral devices 1338. It should be understood that the WTRU 1302 can also include any sub-combination of the foregoing components while remaining compliant with the embodiment.

[0072] The processor 1318 can be a general-purpose processor, a dedicated processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), and a state machine, etc. The processor 1318 can perform signal encoding, data processing, power control, input / output processing, and / or any other function that enables the WTRU 1302 to operate in a wireless environment. The processor 1318 can be coupled to the transceiver 1320, and the transceiver 1320 can be coupled to the transmit / receive component 1322. Although Figure 13 the processor 1318 and the transceiver 1320 are described as separate components, it should be understood that the processor 1318 and the transceiver 1320 can also be integrated together in an electronic component or chip.

[0073] The transmit / receive component 1322 may be configured to transmit or receive signals to or from a base station (e.g., base station 1314a) via the air interface 1316. For example, in one embodiment, the transmit / receive component 1322 may be an antenna configured to transmit and / or receive RF signals. As an example, in another embodiment, the transmit / receive component 1322 may be a radiator / detector configured to transmit and / or receive IR, UV, or visible light signals. In yet another embodiment, the transmit / receive component 1322 may be configured to transmit and / or receive RF and optical signals. It should be understood that the transmit / receive component 1322 may be configured to transmit and / or receive any combination of wireless signals.

[0074] Although the transmit / receive component 1322 is described as a single component in Figure 13 the WTRU 1302 may include any number of transmit / receive components 1322. More specifically, the WTRU 1302 may utilize MIMO technology. Thus, in one embodiment, the WTRU 1302 may include two or more transmit / receive components 1322 (e.g., multiple antennas) that transmit and receive wireless signals via the air interface 1316.

[0075] The transceiver 1320 may be configured to modulate the signals to be transmitted by the transmit / receive component 1322 and to demodulate the signals received by the transmit / receive component 1322. As described above, the WTRU 1302 may have multi-mode capabilities. Therefore, the transceiver 1320 may include multiple transceivers that allow the WTRU 1302 to communicate over multiple RATs (e.g., NR and IEEE 802.11).

[0076] The processor 1318 of the WTRU 1302 may be coupled to a speaker / microphone 1324, a numeric keypad 1326, and / or a display / touchpad 1328 (such as a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit), and may receive user input data from these components. The processor 1318 may also output user data to the speaker / microphone 1324, the keypad 1326, and / or the display / touchpad 1328. In addition, the processor 1318 may access information from any suitable memory such as a non-removable memory 1330 and / or a removable memory 1332, and store information in these memories. The non-removable memory 1330 may include random access memory (RAM), read only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 1332 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 1318 may access information from memories that are not actually located within the WTRU 1302, and store data in these memories. By way of example, such memories may be located on a server or a home computer (not shown).

[0077] The processor 1318 may receive power from a power supply 1334, and may be configured to distribute and / or control power for other components in the WTRU 1302. The power supply 1334 may be any suitable device for powering the WTRU 1302. For example, the power supply 1334 may include one or more dry cell battery packs (such as nickel cadmium (Ni-Cd), nickel zinc (Ni-Zn), nickel metal hydride (NiMH), lithium ion (Li-ion), etc.), solar cells, and fuel cells, among others.

[0078] The processor 1318 may also be coupled to a GPS chipset 1336, which may be configured to provide location information (such as longitude and latitude) related to the current location of the WTRU 1302. As a supplement or replacement to the information from the GPS chipset 1336, the WTRU 1302 may receive location information from a base station via an air interface 1316, and / or determine its location based on the signal timing received from two or more nearby base stations. It should be understood that the WTRU 1302 may obtain location information by means of any suitable positioning method while remaining compliant with the embodiments.

[0079] The processor 1318 may also be coupled to other peripheral devices 1338, where the peripheral devices may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connections. For example, the peripheral devices 1338 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and / or videos), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, modules, a frequency modulation (FM) radio unit, a digital music player, a media player, a video game console module, an Internet browser, a virtual reality and / or augmented reality (VR / AR) device, and an activity tracker, among others. The peripheral devices 1338 may include one or more sensors, and the sensors may be one or more of the following: a gyroscope, an accelerometer, a Hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor, a geographical location sensor, an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and / or a humidity sensor, etc.

[0080] The WTRU 1302 may include a full-duplex radio device, where for this radio device, the reception or transmission of some or all signals (e.g., associated with a specific subframe for UL (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent and / or simultaneous. The full-duplex radio device may include an interference management unit that reduces and / or substantially eliminates self-interference either by means of hardware (e.g., a choke coil) or by signal processing by a processor (e.g., a separate processor (not shown) or by the processor 1318). In an embodiment, the WTRU 1302 may include a half-duplex radio device that transmits and receives some or all signals (e.g., associated with a specific subframe for UL (e.g., for transmission) or downlink (e.g., for reception)).

[0081] Note that the various hardware elements of one or more of the described embodiments are referred to as "modules" that implement (i.e., perform, run, etc.) the various functions described herein in connection with the corresponding modules. As used herein, a module includes hardware that those skilled in the relevant art deem suitable for a given implementation (e.g., one or more processors, one or more optical processors, one or more SLMs, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices). Each of the described modules may also include instructions executable to perform one or more of the functions described as being performed by the corresponding module, and note that these instructions may take the form of, or include, hardware (i.e., hardwired) instructions, firmware instructions, and / or software instructions, etc., and may be stored in any suitable non-transitory computer-readable medium or media, such as those commonly referred to as RAM, ROM, etc.

[0082] Although the features and elements have been described above in specific combinations, those of ordinary skill in the art will understand that each feature or element may be used separately or in any combination with other features and elements. Additionally, the methods described herein may be implemented in part by using a computer program, software, or firmware embodied in a computer-readable medium to be executed by a computer or processor. Examples of computer-readable media include, but are not limited to, read only memory (ROM), random access memory (RAM), registers, buffer memories, semiconductor storage devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM discs and digital versatile discs (DVDs). Processors associated with software may be used to implement an image analysis engine, an image rendering engine, a controller, a timing module, an operating system, etc. for an AR display system.

Claims

1. A method for color calibration of an augmented reality display device, the method comprising: Estimate the real-world illuminance; Capture an image of a real-world object; Using object recognition based on the image of the real-world object, identify the real-world object among objects in a database of recognizable objects, and access the known color characteristics of the real-world object from the database; Use the real-world illuminance and the color characteristics of the real-world object to calculate the color rendering of a synthetic version of the real-world object; Display the color rendering on an augmented reality display, and record an image of the display containing the color rendering; And Based on the image of the displayed color rendering, adjust the display color model for rendering synthetic content.

2. The method according to claim 1, wherein, Estimating the real-world illuminance includes: comparing the captured image of the real-world object with the known color characteristics of the real-world object.

3. The method according to claim 1, wherein the augmented reality display device is a head-mounted display including an internal camera, and wherein the image of the real-world object and the image of the display are obtained using the internal camera.

4. The method according to claim 3, wherein the image of the real-world object and the image of the display are obtained simultaneously.

5. The method according to claim 1, wherein the known color characteristics of the real-world object are accessed from a database.

6. The method according to claim 1, wherein adjusting the display color model is based on a comparison between the image of the real-world object and the image rendered with the displayed color.

7. A method for color calibration of an augmented reality display device, the method comprising: Using a camera installed inside an augmented reality head-mounted device, capture an image of a real-world object through at least partially transparent display of the augmented reality head-mounted device, the display being locatable above the user's eyes, and when the display is located above the user's eyes, the camera is installed at a position outside the field of view of the user's display and facing the display; Display the color rendering of the real-world object on the display; Using the camera, capture an image of the display containing the color rendering; And Based on the comparison between the image of the real-world object and the image of the displayed color rendering, adjust the display color model for rendering synthetic content.

8. The method according to claim 7, wherein the capturing of the image of the real-world object and the capturing of the image of the display are performed when the display is positioned above the user's eyes.

9. An augmented reality device, comprising: A head-mounted device, including a display that is at least partially transparent and capable of being positioned above the user's eyes; A first camera installed inside the head-mounted device, the first camera being installed at a position such that when the display is positioned above the user's eyes, the first camera faces outward through the display, outside the field of view of the user's display, and towards the display; And At least one processor configured to calibrate the display based on an image of the display captured by the first camera.

10. The device according to claim 9, wherein, The processor is further configured to perform: Capture an image of a real-world object; Display the color rendering of the real-world object on the display; and Capture an image of the display containing the color rendering; Wherein the calibration is based on a comparison between the image of the real-world object and the image of the displayed color rendering.

11. The apparatus according to claim 10, wherein the image of the real-world object is captured by the first camera.

12. The apparatus according to claim 10, further comprising a second camera mounted externally to the head-mounted device on the display, wherein the image of the real-world object is captured by the second camera.

13. The apparatus according to claim 9, wherein the calibration of the display is a color calibration.

Images