Method and system for maintaining color calibration using common objects

Abstract

Systems and methods for maintaining color calibration using common objects are described herein. In an exemplary embodiment, an AR system includes a forward-facing camera, an AR display, a processor, and a user interface. The processor is configured to receive image data from the forward-facing camera and identify any known objects depicted in the image data. The processor then determines RGB information of at least one test rendering of the identified known object and displays the RGB information through the AR display. Input from the user interface is received by the processor and used to update the AR display color calibration model, wherein the input indicates which of the at least one test rendering is the closest match to the real-world object and indicates a level of satisfaction with the match. More test renderings can be iteratively provided to improve the accuracy of the calibration.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a non-provisional application of and claims the benefit of 35 U.S.C. §119(e) of U.S. Provisional Patent Application Serial No. 62 / 612,140, ​​filed on December 29, 2017, entitled “Method and System for Maintaining Color Calibration Using Common Objects,” which is incorporated herein by reference in its entirety. Background Art

[0003] The purpose of color calibration is to measure and / or adjust the color response of a device (input or output) to a known state. In International Color Consortium (ICC) terminology, this is the basis for additional color characterization and subsequent profiling of the device. In non-ICC workflows, calibration sometimes refers to a one-time establishment of a known relationship to a standard color space. Color calibration is a requirement for all devices that are an active part of a color management workflow.

[0004] The input data can come from a device source such as a digital camera, image scanner, or any other measurement device. These inputs can be monochromatic (in which case only the calibration response curve is required, although in some selected cases the color or spectral power distribution corresponding to this single channel must also be specified) or specified in multidimensional color (most commonly in a three-channel RGB model). In most cases, the input data is calibrated against a Profile Connection Space (PCS).

[0005] Color calibration is used by many industries such as television production, gaming, photography, engineering, chemistry, medicine, etc.

[0006] Conventional computer displays require separate characterization when used in applications that require the most accurate color reproduction. Augmented reality systems will similarly benefit from separate calibration to ensure accurate and believable presentation of mixed content (real world and synthetic imagery). Augmented reality systems may require even better accuracy because the real world and synthetic content are by definition in the same scene and are typically in close proximity in the field of view. This proximity presents a worst-case scenario for color matching, so the strictest color reproduction is advantageous.

[0007] Frequent calibration of augmented reality display systems is beneficial to maintain the highest level of color reproduction accuracy. If the user needs to view a color chart or other traditional characterization target in order to maintain this high accuracy, the user experience will be diminished. A one-time factory calibration is not sufficient because it does not account for changes in the display over time. Summary of the invention

[0008] A calibrated forward-facing camera or spectrometer continuously captures image data of a real-world scene for an augmented reality (AR) system. In some embodiments, the camera system (or alternatively, a second camera system) is inside an AR head-mounted device (e.g., AR goggles) and can also detect inserted synthetic images. A calibration process (such as eye tracking) can be used to align the camera image with the image seen by the observer. A processor controlling the AR system communicates with a color database of known objects, such as products, signs, and even natural artifacts such as grass and sky. When an object from the database is identified in the real-world field of view, the processor recalibrates the display using at least one of the following two methods:

[0009] 1) With user intervention: Two versions of the known object are presented to the user (eg, one version is displayed on either side of the real-world object). The user selects the closest visual match. This process can be iterated.

[0010] 2) Without user intervention: The processor displays the best current estimate for matching color overlaid on a real-world object. The inside camera captures the scene, and the processor compares the color of the real-world object to the (synthesized) estimate. The measured color and the expected color are compared, and if their values ​​are within a threshold color difference, calibration is complete. This process is fully automated. The color difference threshold can be a built-in default or set / controlled by the user (e.g., depending on their specific application).

[0011] Recalibration of the AR display is operable to improve the color accuracy of the synthetic image. The fully automatic embodiments disclosed herein are not mandatory for the user. Embodiments with user intervention provide color calibration results tuned for the user's specific color vision. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 An overview of a process flow for identifying common objects and calculating their colors in accordance with at least one embodiment is depicted.

[0013] Figure 2 Depicted is a diagram relating various use cases and their conditions to applicable processes in accordance with at least one embodiment.

[0014] Figure 3 An overview of a user-involved color calibration process for a conventional AR display system is depicted in accordance with at least one implementation.

[0015] Figure 4 An overview of a user-involved color calibration process for an AR display system including an internal measurement imaging device is depicted in accordance with at least one embodiment.

[0016] Figure 5 Depicted is a sequence diagram of a process for maintaining user involvement in color calibration using common objects in a field of view in accordance with at least one embodiment.

[0017] Figure 6 An overview of a fully automated color calibration process for an AR display system including an internal measurement imaging device in accordance with at least one embodiment is depicted.

[0018] Figure 7 Depicted is a sequence diagram of a fully automated process for maintaining color calibration using common objects in a field of view in accordance with at least one embodiment.

[0019] Fig. 8A depicts a visual overview of a first example scenario involving a conventional HMD in accordance with at least one embodiment, and Figure 8B Depicted is the corresponding AR color calibration interface view presented to the user.

[0020] Fig. 9A depicts a visual overview of a first example scenario involving an HMD with an internal measurement camera in accordance with at least one embodiment, and Fig. 9B Depicted is the corresponding AR color calibration interface view presented to the user.

[0021] Fig. 10A depicts a second example scenario involving a conventional AR display according to at least one embodiment, and Fig. 10B Depicted is the corresponding AR color calibration interface view presented to the user.

[0022] Fig.11A depicts a second example scenario involving an AR display with an internal measurement camera in accordance with at least one embodiment, and Fig. 11B Depicted is the corresponding AR color calibration interface view presented to the user.

[0023] Fig. 12A and 12B is a schematic diagram of an AR display according to an exemplary embodiment.

[0024] Fig.13 An exemplary wireless transmit-receive unit is shown that may be used to implement an AR display in some embodiments. DETAILED DESCRIPTION

[0025] abbreviation

[0026] AR Augmented Reality

[0027] 3D

[0028] HMD Head-mounted Display

[0029] FOV Field of View

[0030] RGB Red-Green-Blue (color pixel)

[0031] introduction

[0032] Disclosed herein are methods and systems for maintaining color calibration using common objects. Such methods and systems can be implemented as processes occurring in an AR system, such as an AR HMD or an AR server, and as the AR system itself. Various embodiments take the form of procedural methods. In the embodiments described herein, a calibrated forward-facing camera or spectrometer continuously captures image data of a real-world scene of an AR system. In some embodiments, the camera system (or alternatively, a second camera system) is inside the AR glasses and can detect the real-world scene as well as the inserted synthetic image. A processor responsible for control of the AR system communicates with a color database of known objects, such as products, signs, and even natural artifacts such as grass and sky. When an object from the database is identified in the real-world field of view, the processor recalibrates the display. Fig. 12A and 12B , . In some exemplary embodiments, exemplary configurations of AR systems are shown in . In these schematic diagrams, the camera is mounted behind the display surface of the glasses, but outside the viewer's field of view (e.g., above, below, or to the side of the viewer's field of view). In some embodiments, the AR system is provided with eye tracking to align the camera image with the user's field of view.

[0033] advantage

[0034] The recalibration of the AR display helps improve the color accuracy of the synthetic image.The fully automatic embodiments disclosed herein are not mandatory for the user, and embodiments that include user feedback provide a means for tuning the display to the user's specific color vision.

[0035] Before continuing with this detailed description, it is noted that the entities, connections, arrangements, etc. depicted in and described in connection with the various figures are presented as examples and not as limitations. Therefore, any and all statements or other indications as to what a particular figure "depicts," what a particular element or entity in a particular figure "is" or "has," and any and all similar statements—which might be read in isolation and out of context as absolute and therefore limiting—may only be properly read if they are constructively preceded by a clause such as "in at least one embodiment."

[0036] Furthermore, any variations and permutations described in the following paragraphs and anywhere else in this disclosure may be implemented with respect to any embodiment, including with respect to any method embodiment and with respect to any system embodiment.

[0037] Exemplary Color Database

[0038] The exemplary method described herein utilizes a database of identifiable objects and their color or spectral attributes. Before performing the disclosed method, a database is created or identified, and its contents are made available for use in a device or process embodying the teachings of this article. The database can be constructed to include data used by one or more known object recognition technologies. Collecting this data can be performed in a manner that takes into account the fact that objects can be imaged from unknown viewpoints and in both spectrum and space under unknown and / or complex illumination. For the balance of the present disclosure, exemplary methods are described for situations involving diffuse illumination and directional detection. However, this condition is not meant to be limiting in any way, because the database can be easily expanded to include more complex lighting conditions. Omitting references to more complex situations is only for brevity and clarity.

[0039] The data in the database of identifiable objects and their color or spectral properties can be obtained using several means, including: actual measurements of specific materials; estimates from product trade literature (e.g., Pantone colors); other databases (for traditional materials such as grass, sky, brick, skin, etc.), etc. In at least one embodiment, the data set includes color coordinates (CIELAB or other). In some embodiments, spectral reflectance data can be captured. The data for certain materials or products may be measured by the entity that plans to implement the process. In this case, the entity that plans to implement the process may or may not make the data publicly available. The properties associated with each object may include one or more of the following:

[0040] Spectral reflectance. In some embodiments, the spectral reflectance is data measured under known standard reference conditions, for example, measurements under known illumination and detection angles, such as bidirectional reflectance (e.g., illumination at 45° and measurement at 0°) or hemispherical reflectance. In some embodiments, the spectral reflectance is characterized using a bidirectional reflectance distribution (BRDF) or more generally a bidirectional scattering distribution function (BSDF).

[0041] · Fluorescence behavior. In some embodiments, fluorescence can be characterized by a Donaldson matrix as described in R. Donaldson, Spectrophotometry of fluorescent pigments, Br. J. Appl. Phys. 5 (1954) 210-214. This matrix can be determined using appropriate measurement equipment and fully characterizes the spectral reflectance as a function of the wavelength of the incident light. In some embodiments, fluorescence information can be determined based on the material properties of the object being identified. For example, fluorescent paper typically exhibits fluorescent behavior similar to that of ordinary "daylight" fluorescent security objects.

[0042] Gloss. Gloss data may be data collected by a gloss meter under a set of standard reference conditions. Depending on the object, the specification may be 80° gloss (for diffuse materials); 60° gloss (semi-gloss materials) or 20° gloss (glossy materials). Other angles are possible, but these are the most common.

[0043] Logo font.

[0044] In some embodiments, the database may be expanded by storing measured properties of new objects encountered by the user. In this case, a verification mechanism may be applied since there is no necessarily existing base real color on which any calibration is based. In this case, it is appropriate to ask the user to ensure that the color match after calibration is adequate. The color may then be estimated by the forward-facing camera and the calibration model. For better base real color data, an integrating sphere spectroradiometer may be used to measure the data and / or an integrated spectroradiometer may be included in the AR HMD device.

[0045] Detailed process and structure

[0046] A set of processes and apparatus corresponding to various use cases are disclosed herein. Figure 1 An exemplary process flow 100 is described and described. Figure 1 A process flow is depicted starting 102 , which includes using a forward-facing camera to identify common objects and calculate their actual colors within the current AR scene.

[0047] In process 100, a calibrated forward camera images the current field of view to generate color image data and detect the wearer's field of view 104. In some embodiments, the forward camera is mounted to an AR HMD. In some embodiments, the forward camera is embedded in the AR HMD. The color image data is received 106 at a processor (of the HMD or AR server), and at step 108, the processor uses the data to identify objects in the scene that match objects in a known object database. Process 100 also includes determining the current scene illumination incident on the identified object 116 at least in part by using the image data received at the processor. The determined scene illumination is used by the processor together with the color or spectral properties obtained from the known object database to calculate the actual color of the identified object under the lighting conditions in the current scene. Step 110 includes retrieving the properties of any known object. It should be noted that the actual color here is only a function of the known object properties obtained by the database and the real-world illumination estimated using the image data. In at least one embodiment, the real-world illumination is estimated at least in part by comparing the image data from the forward camera with the properties obtained from the known object database. Step 112 provides for the processor to calculate the actual color of any known objects in the scene. At this point, the "actual color" only takes into account the real world lighting and the object properties. Therefore, this is the color of the light incident on the glasses (HMD) after reflection from the object. The properties of the glasses are not considered yet. Therefore, the display / rendering properties of the AR device worn by the user are not involved in determining the actual color of the object.

[0048] exist Figure 2 The use cases 1-3 are outlined in the diagram depicted in FIG. Figure 2 200 depicts a graph relating various use cases and their conditions to applicable processes in accordance with at least one embodiment. The graph shows which use cases are applicable based on whether there is user intervention and whether there is an image sensor (inside measurement device) located within the field of view of the HMD display surface.

[0049] Use case 1 corresponds to an AR HMD device that does not have a camera sensor within the field of view of the AR display surface. The user interacts with the AR HMD to request and control color calibration. The process 300 applicable to use case 1 in Figure 20 will be as follows Figure 3 is discussed in the description.

[0050] Use Case 2 and Use Case 3 both correspond to an AR HMD device with a camera sensor located within the field of view of the AR display surface. Figure 1The forward-facing camera has a field of view (FOV) that includes the AR display surface, then the camera sensor may be about Figure 1 The forward-facing camera discussed. Alternatively, for example, when Figure 1 The forward-facing camera does not include the position / orientation of the AR display surface in its FOV, except Figure 1 In addition to the forward-facing camera of the AR HMD, the camera sensor may be included in the HMD. In use case 2, the user interacts with the AR HMD to request and / or control color calibration, and the camera sensor located within the field of view of the AR display surface sends image data to the processor, which correlates this data with user input data for tuning the automatic color calibration model. Figure 3 The process applicable to use case 2 is outlined in C. In use case 3, no user interaction or intervention is involved. By using the automatic color calibration model, the color calibration process is fully automated.

[0051] about Figure 6 The flow for use case 3 is described.

[0052] Note that nothing precludes a particular AR system from using multiple of the above methods, and indeed, a large number of possible embodiments, not listed for the sake of brevity, may include various combinations of certain elements from these processes. For example, the fully automatic method associated with use case 3 may run continuously in the background, and if the user senses via the user interface of the AR device that the color reproduction is not optimal, the user may trigger a manual calibration. The user may then re-engage the fully automatic process, knowing that they do not want visual distractions in the FOV during the upcoming moments. The process outlined in Table 200 may be used sequentially to first tune and then run the fully automatic AR color calibration process.

[0053] Figure 3 An overview of a user-involved color calibration process for a conventional AR display system is shown in accordance with at least one embodiment. Figure 3 The AR display system used in is consistent with Use Case 1; it does not include an internal measurement imaging device for viewing content presented on the AR display surface (i.e., it is a traditional AR display system and therefore does not include an internal camera). Figure 3 The process is shown as flow chart 300 .

[0054] In process 300, step 302 shows the actual color of the known object in the scene. In step 304, the actual color of the known object in the AR scene is input to the inverse display model to convert the color data into RGB data. Thus, in step 306, the RGB data of the known object is estimated. Next, the process includes selecting alternative (nearby) RGB coordinates as test colors 308. This selection can be performed by a processor using a test color generation algorithm. The number of selected alternative coordinates may be unlimited, unless it is to maintain a reasonable interface for the user in step 310. A larger number of selected alternative coordinates provides a faster convergence towards the preferred display calibration. Next, the process includes rendering the known object using each selected test color in step 312, and visually displaying each rendering near the known object. In some embodiments, the entire known object can be rendered. In some embodiments, only a representative portion of the known object is rendered. In various embodiments, the amount of the rendered known object is based on the size of the known object. In step 314, the process then prompts the user to select which displayed rendering is the closest visual match to the known object. In at least one embodiment, step 314 includes a prompt further requesting the user to select a closeness (i.e., acceptability) of the match. If, at decision 316, the user is not satisfied with the closeness of the match, process 300 includes updating the estimate 318 of the RGB data of the known object based on the RGB data of the preferred rendering, and process 300 repeats from corresponding step 306. If the user is satisfied with the closeness of the match, process 300 includes, at step 320, updating the display model using the RGB data of the selected rendering and the actual color of the known object. Process 300 then ends, and Figure 1 The process 100 in is restarted. The display model parameters can be re-optimized to take into account the new information (the newly determined RGB to color matching parameters).

[0055] Figure 4 An overview of a user-involved color calibration process for an AR display system including an internal measurement imaging device is shown in accordance with at least one embodiment. Figure 4 The AR display system used in is consistent with Use Case 2; it does include an internal measurement imaging device (e.g., an inside camera) for viewing content rendered on the AR display surface. Figure 4 The process of FIG. 4 shows a flow 400 .

[0056] In process 400, at step 402, the actual color of the known object in the AR scene (as determined by process 100) is input to the inverse display model to convert the color data into RGB data. In this way, the RGB data of the known object is estimated by the inverse display model (color to RGB) 404 and the RGB 406 of the known object. Next, process 400 includes selecting alternative (nearby) RGB coordinates as test colors 408. In addition to maintaining a reasonable interface for the user, the number of selected alternative coordinates is not limited. A larger number of selected alternative coordinates provides a faster convergence towards the preferred display calibration. Next, process 400 includes rendering the known object 410 using each selected test color, and visually displaying each rendering near the known object in step 414. In some embodiments, the entire known object can be rendered. In some embodiments, only a representative portion of the known object can be rendered. In various embodiments, the amount of the rendered known object is based on the size of the known object. In step 414, the user selects which displayed rendering is the closest visual match to the known object (i.e., preferred). In at least one embodiment, the prompt further requests the user to select the closeness (i.e., acceptability) of the match at decision 416. If the user is not satisfied with the closeness of the match, at step 418, process 400 includes updating the estimate of the RGB data of the known object based on the RGB data of the preferred rendering, and process 400 repeats from the corresponding step. If the user is satisfied with the closeness of the match, at step 420, process 400 includes updating the display model using the RGB data of the preferred rendering and the actual color of the known object. Process 400 differs from process 100 in that process 400 also includes updating the inner camera model using the actual color of the known object and the preferred rendering RGB data at step 422. Then, process 400 ends and process 100 restarts at step 104, as indicated by identifier "A".

[0057] Figure 5 Depicted is a sequence diagram 500 of a user involved process for maintaining color calibration using common objects in a field of view in accordance with at least one embodiment. Figure 5Operations performed by an image processing component 502, an AR display component 504, and a user component 506 are shown. The image processing component estimates current lighting conditions 508. Real-world objects are visible to the user through the AR display 510. The image processing component captures an image of the real-world object 512. The image processing component then identifies the real-world object and its color. The image processing component calculates candidate color renderings 516 and sends the renderings to the AR display component 518. The AR display component displays the synthetic content (i.e., the rendering) 520. The synthetic content is visible to the user via the AR display 522. The user then compares the candidate renderings with the real-world objects at step 524 and selects a preferred rendering 526. The user selection is sent to the image processing component, which uses the feedback to update the display color model at 528.

[0058] Figure 6 An overview of a fully automated color calibration process 600 for an AR display system including an internal measurement imaging device is shown in accordance with at least one embodiment. Figure 6 The AR display system used in is consistent with use case 3 shown in Table 200; it may include an internal measurement imaging device (e.g., an inside camera) for viewing content rendered on the AR display surface.

[0059] In process 600, the actual color 602 of a known object in an AR scene (as determined by process 100) is input to an inverse display model to convert the color data into RGB data 604. Thus, in step 606, the RGB data of the known object is estimated. Next, the process includes selecting alternative (nearby) RGB coordinates as test colors in step 608. This selection can be performed by a processor using a simple test color generation algorithm. In addition to maintaining a reasonable interface for the user, the number of selected alternative coordinates is not limited. A larger number of selected alternative coordinates provides a faster convergence towards a preferred display calibration. Next, the process includes rendering a portion of a known object using each selected test color in step 610, and visually displaying each rendering near the known object in step 612. In an embodiment where multiple alternative colors are displayed, the processor selects different areas of the known object to be rendered. Process 600 then includes: in step 614, the inner camera detects the color of the known object and all rendered fragments. In step 616, the processor selects the alternative RGB data that matches the color of the known object the closest. At decision 618, it is determined whether the color match is less than a target maximum color difference. If the match is not close enough, at step 620, process 600 includes updating the estimate of the RGB data of the known object based on the RGB data of the preferred rendering, and process 600 repeats from corresponding step 606. If the match is close enough, process 600 includes updating the display model using the actual color of the known object and the RGB data of the selected rendering at step 622. Process 600 ends, and process 100 restarts at step 104.

[0060] The steps listed below are a supplemental description of process 600 .

[0061] Step 1: Identify candidate objects within the field of view:

[0062] The processor determines whether an object in the current field of view of the forward-facing camera is in a database of known objects by using an image processing algorithm. The algorithm may take into account any of the attributes listed previously. In some embodiments, several of these attributes are compared. For example, after a bright red object is detected, the number of potential objects in the database may be greatly reduced, and then a second attribute may be compared, and so on. The literature on object detection and recognition is very in-depth, and any number of published methods may be applied and potentially combined to achieve the necessary performance level for a given AR application.

[0063] Step 2: Estimate the spectral power distribution SPD of the lighting:

[0064] The estimation of the SPD can be performed by many methods established in the literature. The forward-facing camera or other components can be used to accurately estimate the current lighting. In some embodiments, the lighting estimation can be performed using the techniques described in Cheng, Price, Cohen, and Brown, "Effective Learning-Based Illuminant Estimation Using Simple Features," (IEEE CVPR 2015).

[0065] Step 3: Estimate the display RGB coordinates of the known object:

[0066] Estimating the effective reflectivity of the object can be done by using the forward-facing camera and taking into account the illumination estimated in step 2 and the known reflectivity of the object from step 1. An exemplary estimate of the light reaching the observer (radiance) is the product of the illumination and the reflectivity. If more complex geometric properties are available (such as BRDF), they can be applied here to improve the estimate of what the observer sees. The radiance so calculated does not yet take into account the properties of the glasses. After applying the transmittance of the glasses, the resulting radiance is a useful estimate of the light incident on the observer's eyes and the inside camera. Using the estimated illuminant and the known CIE transform, the spectral radiance can be converted to color. The color can be processed by an inverse display model to estimate the RGB required to match the color.

[0067] Step 4: Process the calibration and update the display color reproduction model:

[0068] Before performing a single recalibration, several objects can be identified and their properties estimated. For each object, the color coordinates are mapped to estimated RGB coordinates for the AR display. The estimated RGB coordinates are displayed by the AR glasses in or near the object, and the forward-facing camera detects the colors of both. The difference between the target color and the actual color is determined for several objects, and then the display model is updated. This process can be repeated as needed until the final estimated color and the measured color are below a color difference threshold. The color distance threshold can be a predefined value or a user-adjustable value. The International Color Consortium (ICC) website www.color.org / displaycalibration.xalter is an effective reference for camera calibration technology. The ICC has established a variety of methods by which displays can be calibrated and outlines the processing and communications used to operate such calibrated displays.

[0069] The rendering of accurate colors is limited by and complicated by the transparent nature of AR glasses. The following description takes these complications into account.

[0070] Step 5: Update object color, and potentially iterate:

[0071] First, the system updates the RGB coordinates of the object, redisplays the rendering, and re-images the rendering and the real-world object with the camera. Then, the system checks the color difference between the rendering and the target object color. If the color difference is below the threshold, the process is complete. If the color difference is above the threshold, the process repeats steps 4 and 5.

[0072] Figure 7 Depicted is a sequence diagram 700 of a fully automated process for maintaining color calibration using common objects in a field of view, in accordance with at least one embodiment. Figure 7 Includes an image processing component 702, an AR display component 704, and an internal camera component 706. The image processing component 702 estimates the current lighting conditions 708. Real-world objects are visible to the internal camera component through the AR display 710. The image processing component captures an image 712 of a real-world object. Then, the image processing component identifies the real-world object and its color 714. The image processing component calculates a candidate color rendering and sends the rendering to the AR display component 716. The AR display component displays synthetic content (i.e., the rendering) 720. The synthetic content is visible to the internal camera component 722. The image of the displayed object is provided to the internal camera 722. Then, the internal camera component captures / records images 724 of various candidate renderings and real-world objects, both of which are visible at the AR display. The image of the AR display is sent to the image processing component 726, which uses the feedback to update the display color model 728.

[0073] Fig. 8A depicts a visual overview 800 of a first example scenario involving a conventional HMD in accordance with at least one embodiment, and Figure 8B A corresponding AR color calibration interface view 850 presented to a user is depicted. Fig. 8A The overview of FIG. 8 includes a viewer 806, an AR visor 804, a synthetic content overlay visible to the viewer and displayed by the AR visor 804, and recognized real-world objects 802 that the viewer can also see through the AR visor 804. The AR visor 804 is part of the HMD that performs the color calibration process taught herein. The AR system presents the viewer with Figure 8BThe AR color calibration interface 850 of the synthetic content overlay includes an AR color calibration interface. In some embodiments, the interface includes a prompt to select the closest color or preferred option and at least one candidate RGB color option in the form of a rendering of the identified real-world object. In other embodiments, the interface includes a set of candidate RGB color options as a rendering of the identified real-world object. Figure 8B An interface is shown that allows a user to select the closest color compared to a "real" object 854 by selecting option A 852 or option B 856. The AR system receives viewer input identifying the selected option via the user interface. The AR system can iterate the calibration process by presenting new options to the viewer and receiving new viewer input. In each iteration, the display calibration model is updated using the viewer-selected color parameters compared to the known parameters of the identified real-world object. In one embodiment, the display color calibration model can be adjusted so that when the AR system renders a synthetic content overlay that includes the known colors of the identified real-world object, those colors are rendered using the RGB values ​​of the selected option. Figures 8A-8B Embodiments of the systems and methods of the present disclosure are directed to a conventional HMD. The conventional HMD does not include a camera that can view the synthetic content overlay displayed by the AR mask. Such a system can perform the above-described processes 100 and 300.

[0074] Fig. 9A depicts a visual overview of a first example scenario involving an HMD with an internal measurement camera in accordance with at least one embodiment, and Fig. 9B Depicted is the corresponding AR color calibration interface view presented to the user. Fig. 9A The overview of FIG. 1 shows an identified real-world object 902, an AR mask 904 of the HMD, a synthetic content overlay visible to the viewer and displayed by the AR mask 904, and the identified real-world object 902 also visible to the viewer through the AR mask 904. The HMD includes an inside camera 906 that can view a combined image of the synthetic content overlay at the AR mask 904 and the identified real-world object. This AR system presents the viewer with Fig. 9B In this case, the synthetic content overlay includes the AR color calibration interface. In some embodiments, the interface includes a prompt for the user to select the closest color or preferred option and at least one candidate RGB color option in the form of a rendering of the identified real-world object. In other embodiments, the interface only includes a set of candidate RGB color options as a rendering of the identified real-world object. In embodiments such as Fig. 9BIn the fully automatic embodiment depicted in , the interface 950 includes a single color option rendering, and the inner camera images the real-world object and the color option rendering. This image data is sent to the processor of the HMD to be used as input to update the AR display color calibration model. The AR system can iterate the calibration process by presenting new options to the inner camera and receiving new image data as input. In each iteration, if the color match between the rendered option 954 and the real-world object 952 is better, as determined by the image analysis algorithm using the inner camera image data, the display calibration model is updated using the color parameters of the rendered option. In one embodiment, the display color calibration model can be adjusted so that when the AR system renders a synthetic content overlay of known colors including the identified real-world objects, those colors are rendered using the RGB values ​​of the selected option. Figures 9A-9B Embodiments of the systems and methods of the present disclosure include an HMD having an inner camera capable of viewing an AR mask. Such a system may perform the above-described processes 100, 300, 400, and 600.

[0075] Fig. 10A depicts a second example scenario 1000 involving a conventional AR display according to at least one embodiment, and Fig. 10B A corresponding AR color calibration interface view 1050 presented to a user is depicted. Fig. 10A The example scene depicted in FIG. 1 includes an observer 1002 viewing an object 1012 on a table through an AR display (e.g., a glass see-through AR mask) 1004. A light source 1006 provides incident light, which is reflected by an object 1008 and propagates toward an observer 1010. The reflected light passes through the AR display 1004.

[0076] exist Figures 10A-10B In the example, the object is a Rubik's cube. Fig. 10B A view of an observer through an AR mask is depicted 1050. The observer's view includes a real-world Rubik's Cube 1052, two synthetic renderings 1054, 1056 of the Rubik's Cube, each with a slightly altered color, and a synthetic rendering of a question mark prompting the observer to select which synthetic Rubik's Cube more closely matches the real-world Rubik's Cube. The question mark is inserted into the image to indicate to the observer that a selection needs to be made. Rubik's Cubes 1054 and 1056 represent synthetic images with slightly altered colors.

[0077] Fig.11A depicts a second example scenario 1100 involving an AR display 1104 with an internal measurement camera 1106 in accordance with at least one embodiment, and Fig. 11B Depicted is the corresponding AR color calibration interface view presented to the user. Fig.11AThe example scene 1100 depicted in FIG. 1 includes an observer viewing an object 1114 on a table through an AR display (e.g., a glass see-through AR mask 1104). A light source 1108 provides incident light, which reflects from the object 1110 and travels toward the observer 1112. The reflected light passes through the AR display 1104 and is viewed by the observer 1102 and the inside camera 1106. Figures 11A-11B , the object is a Rubik's Cube. Fig. 11B Depicted are the observer's views through the AR mask before and after a synthetic test rendering of the Rubik's Cube is inserted into the observer's FOV via the AR display (i.e., View (A) and View (B)). First, the observer's view (A) 1152 contains the real-world Rubik's Cube on the table, and this view is also visible to the inside camera. The inside camera or another camera sends image data to the processor, which then identifies the Rubik's Cube as an object in a database of known objects. Next, the synthetic test image of the Rubik's Cube is overlaid using the AR mask, and this new view (B) 1154 is visible to both the observer and the inside camera. The inside camera sends image data to the processor, which then adjusts the color rendering scheme of the AR mask based on the difference between View (A) and View (B).

[0078] Reference now Fig. 12A and 12B , shows an exemplary configuration of AR systems 1200 and 1250. In the schematic diagram, cameras 1202 and 1252 are respectively mounted behind the display surface of the glasses, but outside the viewer's field of view (e.g., above, below, or to the side of the viewer's field of view). In some embodiments, the AR system is provided with eye tracking to align the camera image with the user's field of view. Fig. 12A , the camera is shown coupled to a control module 1204, which is coupled to a waveguide 1206 and an LCD 1208. Similarly, in Fig. 12B , the camera 1252 is coupled to the control module 1254 as is the LCD 1256. Fig. 12B Also shown is a partially reflective surface 1258.

[0079] Rendering display colors in the presence of an ambient background

[0080] Traditional color display models relate device RGB coordinates to the output radiance (or color) of the display. More advanced models also take into account ambient room conditions (flare). AR glasses have the additional confusion of spatially varying ambient light from the scene and passing through the glasses, which can be observed by users adjacent to or overlapping the AR display imagery. Exemplary embodiments can address this issue as follows.

[0081] Consider an embodiment where there is an outside forward-facing camera that detects ambient light seen by an observer, and a processor is operable to determine the spatial relationship between that ambient light and an internal AR display. The result is that the processor has access to the aligned radiant brightness or color of the light seen by the observer at each pixel location in the AR display. Since the AR display and the real world are aligned, the following spatial coordinates x, y are used for both systems.

[0082] According to the traditional display model, the spectral radiance L can be calculated for an AR display towards the viewer from a given pixel x,y and input R,G,B color coordinates:

[0083]

[0084] Note that the subscript λ indicates that the parameter is spectrally quantized. Also note that the display model f1 is independent of the position on the display. The contribution from ambient light is based on the camera model f2:

[0085]

[0086] Note again that the camera model f2 is independent of the location of the pixels. represents the spectral transmittance of the display at a given pixel. The total radiant brightness seen by the observer is the sum of these two components:

[0087]

[0088] Therefore, the input colors to the final display model can be manipulated to take into account environmental effects. This will place some limits on the available display colors. Even when the AR display is completely turned off, the ambient light passing through the glasses will impose a lower limit on the radiance that can be presented to the observer.

[0089] In some embodiments, for color-critical applications, the observer may be instructed to keep the viewpoint clear of bright real-world areas. In practice, the benefit of directing the observer to dim areas depends on the type and quality of light blocking available in the particular glasses. In some embodiments, the AR system includes technology that allows light to pass completely where there is no AR image, and completely blocks light anywhere there is an AR image. In such an embodiment, the value of ambient radiance as described above may be zero for areas where light is completely blocked.

[0090] Some variations of the solution

[0091] Figure 2 The table / chart 200 in provides three exemplary scenarios in which the teachings disclosed herein may be applied.Other combinations of user input and imaging capabilities are also contemplated.

[0092] In a more complex embodiment, the database of object properties includes a bidirectional reflectance distribution function for some or all objects. Utilizing this data improves the accuracy in estimating the effective reflectivity of a given object. However, this is a much more computationally expensive application because various directional aspects of the real-world lighting of the object are taken into account.

[0093] Other Discussions

[0094] Fig.13 1 is a system diagram illustrating an exemplary wireless transmit receive unit (WTRU) 1302 that may be used as a head mounted AR display in an exemplary embodiment. Fig.13 As shown, the WTRU 1302 may include a processor 1318, a transceiver 1320, a transmit / receive element 1322, a speaker / microphone 1324, a numeric keypad 1326, a display / touchpad 1328, non-removable memory 1330, removable memory 1332, a power source 1334, a global positioning system (GPS) chipset 1336, and / or peripherals 1338. It should be appreciated that the WTRU 1302 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0095] The processor 1318 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, and the like. The processor 1318 may perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 1302 to operate in a wireless environment. The processor 1318 may be coupled to the transceiver 1320, which may be coupled to the transmit / receive element 1322. Although Fig.13 The processor 1318 and the transceiver 1320 are depicted as separate components, however it should be appreciated that the processor 1318 and the transceiver 1320 may also be integrated together in an electronic component or chip.

[0096] The transmit / receive component 1322 may be configured to transmit or receive signals to or from a base station (e.g., base station 1314a) via air interface 1316. For example, in one embodiment, the transmit / receive component 1322 may be an antenna configured to transmit and / or receive RF signals. As an example, in another embodiment, the transmit / receive component 1322 may be an emitter / detector configured to transmit and / or receive IR, UV, or visible light signals. In yet another embodiment, the transmit / receive component 1322 may be configured to transmit and / or receive RF and light signals. It should be appreciated that the transmit / receive component 1322 may be configured to transmit and / or receive any combination of wireless signals.

[0097] Although in Fig.13 1302 as a single component, but the WTRU 1302 may include any number of transmit / receive components 1322. More specifically, the WTRU 1302 may employ MIMO technology. Thus, in one embodiment, the WTRU 1302 may include two or more transmit / receive components 1322 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 1316.

[0098] The transceiver 1320 may be configured to modulate signals to be transmitted by the transmit / receive component 1322 and to demodulate signals received by the transmit / receive component 1322. As described above, the WTRU 1302 may have multi-mode capabilities. Thus, the transceiver 1320 may include multiple transceivers that allow the WTRU 1302 to communicate via multiple RATs (e.g., NR and IEEE 802.11).

[0099] The processor 1318 of the WTRU 1302 may be coupled to a speaker / microphone 1324, a numeric keypad 1326, and / or a display / touchpad 1328 (e.g., a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit), and may receive user input data from these components. The processor 1318 may also output user data to the speaker / microphone 1324, the keypad 1326, and / or the display / touchpad 1328. In addition, the processor 1318 may access information from and store information in any suitable memory such as a non-removable memory 1330 and / or a removable memory 1332. The non-removable memory 1330 may include a random access memory (RAM), a read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 1332 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 1318 may access information from, and store data in, memory that is not physically located on the WTRU 1302, such as on a server or a home computer (not shown).

[0100] The processor 1318 may receive power from the power source 1334 and may be configured to distribute and / or control power for other components in the WTRU 1302. The power source 1334 may be any suitable device for powering the WTRU 1302. For example, the power source 1334 may include one or more dry cell batteries (e.g., nickel-cadmium (Ni-Cd), nickel-zinc (Ni-Zn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, and the like.

[0101] The processor 1318 may also be coupled to the GPS chipset 1336, which may be configured to provide location information (e.g., longitude and latitude) related to the current location of the WTRU 1302. In addition to or in lieu of the information from the GPS chipset 1336, the WTRU 1302 may receive location information from a base station via the air interface 1316 and / or determine its location based on the timing of signals received from two or more nearby base stations. It will be appreciated that the WTRU 1302 may acquire location information via any suitable positioning method while remaining consistent with an embodiment.

[0102] The processor 1318 may also be coupled to other peripherals 1338, wherein the peripherals may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 1338 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and / or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, Modules, FM radio units, digital music players, media players, video game console modules, Internet browsers, virtual reality and / or augmented reality (VR / AR) devices, and activity trackers, etc. The peripherals 1338 may include one or more sensors, which may be one or more of the following: gyroscopes, accelerometers, Hall effect sensors, magnetometers, orientation sensors, proximity sensors, temperature sensors, time sensors, geographic location sensors, altimeters, light sensors, touch sensors, magnetometers, barometers, gesture sensors, biometric sensors, and / or humidity sensors, etc.

[0103] The WTRU 1302 may include a full-duplex radio in which the reception or transmission of some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) and downlink (e.g., for reception)) may be concurrent and / or simultaneous for the radio. A full-duplex radio may include an interference management unit that reduces and / or substantially eliminates self-interference by means of hardware (e.g., chokes) or by signal processing by a processor (e.g., a separate processor (not shown) or by means of the processor 1318). In an embodiment, the WTRU 1302 may include a half-duplex radio that transmits and receives some or all signals (e.g., associated with specific subframes for UL (e.g., for transmission) or downlink (e.g., for reception)).

[0104] Note that the various hardware elements of one or more described embodiments are referred to as "modules," which implement (i.e., execute, run, etc.) the various functions described herein in conjunction with the corresponding modules. As used herein, a module includes hardware that a person skilled in the relevant art deems suitable for a given implementation (e.g., one or more processors, one or more optical processors, one or more SLMs, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application specific integrated circuits (ASICs), one or more field programmable gate arrays (FPGAs), one or more memory devices). Each described module may also include executable instructions for performing one or more functions described as being performed by the corresponding module, and note that these instructions may take the form of or include hardware (i.e., hardwired) instructions, firmware instructions, and / or software instructions, etc., and may be stored in any suitable non-transitory computer-readable medium or media, such as those commonly referred to as RAM, ROM, etc.

[0105] Although the features and elements are described above in specific combinations, it will be understood by those of ordinary skill in the art that each feature or element may be used alone or in any combination with other features and elements. In addition, the methods described herein may be implemented in part by using a computer program, software, or firmware incorporated in a computer-readable medium so as to be executed by a computer or processor. Examples of computer-readable media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, buffer memory, semiconductor storage devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM discs and digital versatile discs (DVDs). A processor associated with the software can be used to implement an image analysis engine, an image rendering engine, a controller, a timing module, an operating system, etc. for use in an AR display system.

Claims

1. A method for color calibration of an augmented reality display device, the method comprising: Estimate real-world illumination; capturing an image of a real-world object, identifying the object and accessing known color characteristics of the real-world object; computing a first candidate color rendering of a synthesized version of the real-world object using the real-world illuminance and the color characteristic of the real-world object; computing a second candidate color rendering of a synthesized version of the real-world object using the real-world illuminance and the color characteristic of the real-world object; displaying the two candidate color renderings using the transparent AR display while the real-world object is visible through the transparent AR display, and prompting a user to indicate a preferred color rendering; as well as In response to the user indication, a display color model used to generate future candidate renderings and synthesized AR content is updated based on the indicated characteristics of the rendering.

2. The method of claim 1 , wherein estimating the real-world illumination comprises: The captured image of the real-world object is compared to the known color characteristics of the real-world object.

3. The method of claim 1 , wherein capturing the image of the real-world object comprises: The image is captured with the HMD's forward-facing camera. 4 . The method of claim 1 , wherein the first candidate color rendering and the second candidate color rendering are based on expected temporal changes in the display rendering characteristics. The method of claim 1 , wherein the candidate color rendering comprises a primary color patch of the real-world object. The method of claim 1 , wherein the candidate color rendering comprises a synthesized version of an entire real-world object.

7. The method according to claim 1, wherein prompting the user comprises: The user is prompted via at least one of a synthetic overlay displayed by the AR device, an audible request generated by the AR device, and a tactile notification emitted by the AR device. The method of claim 1 , wherein the display color model is used to adjust a display color calibration of the AR system.

9. The method of claim 1, wherein the steps of calculating the first candidate color rendering, calculating the second candidate color rendering, displaying the two candidate color renderings, and updating the display color model are performed iteratively.

10. The method of claim 9, wherein the iterative execution is terminated after a predetermined number of iterations.

11. The method of claim 9, wherein the iterative execution is terminated in response to a user input indicating a calibration termination request.

12. A method for color calibration of an augmented reality display device, the method comprising: Estimate real-world illumination; capturing an image of a real-world object, identifying the real-world object and accessing known color characteristics of the real-world object; computing a candidate color rendering of a synthesized version of the real-world object using the real-world illuminance and the color characteristic of the real-world object; displaying the candidate color rendering using the transparent AR display while the real-world object is visible through the transparent AR display, and recording an image including both the real-world object and the candidate color rendering; as well as A display color model used to render composite content is adjusted based on a calculated difference between the image of the real-world object and the image rendered in the displayed candidate color.

13. The method of claim 12, wherein estimating the real-world illumination comprises: The captured image of the real-world object is compared to the known color characteristics of the real-world object.

14. The method of claim 12, wherein capturing the image of the real-world object comprises: The image is captured with a forward-facing camera of an HMD, and recording the image of the display including both the real-world object and the candidate color rendering includes recording the image with an inside camera of the HMD.

15. The method of claim 12, wherein capturing the image of the real-world object comprises: The image is captured with a forward-facing camera of the HMD, and recording the image of the display including both the real-world object and the candidate color rendering includes recording the image with the forward-facing camera.

16. An apparatus for color calibrating an augmented reality display, the apparatus comprising a processor configured to at least perform: Estimate real-world illumination; capturing an image of a real-world object, identifying the object and accessing known color characteristics of the real-world object; computing a first candidate color rendering of a synthesized version of the real-world object using the real-world illuminance and the color characteristic of the real-world object; computing a second candidate color rendering of a synthesized version of the real-world object using the real-world illuminance and the color characteristic of the real-world object; displaying the two candidate color renderings using the transparent AR display while the real-world object is visible through the transparent AR display, and prompting a user to indicate a preferred color rendering; as well as In response to the user indication, a display color model used to generate future candidate renderings and synthesized AR content is updated based on the indicated characteristics of the rendering.

17. The apparatus of claim 16, wherein estimating the real-world illumination comprises comparing the captured image of the real-world object to the known color characteristics of the real-world object.

18. The apparatus of claim 16, wherein capturing the image of the real-world object comprises capturing the image with a forward-facing camera of an HMD. The apparatus of claim 16 , wherein the display color model is used to adjust display color calibration of the AR system.

20. The apparatus of claim 16, wherein the steps of calculating the first candidate color rendering, calculating the second candidate color rendering, displaying the two candidate color renderings, and updating the display color model are performed iteratively.

Images