Method for rendering augmented reality content in conjunction with an external display

Abstract

Systems and methods for improving color reproduction are described. In some embodiments, a video of a real-world scene is captured, and a screen content region is detected in the captured video. A processor selectively applies screen content color conversion on the screen content region to generate a processed video, and displays the processed video in substantially real-time, e.g., on a video see-through head-mounted display. The screen content color processing can be different for different types of external displays. The screen content color processing can also be determined based at least in part on lighting conditions. In some embodiments, the color of a displayed virtual object is adjusted based on a visual parameter of the screen content.

Description

[0001] Cross-reference to related applications

[0002] This application is a non-provisional filing of U.S. Provisional Patent Application Serial No. 62 / 851476 entitled “Method for Improving Rendering of a Remote Display for Viewing on a Local Display”, filed May 22, 2019, and claims the benefit of the patent application under 35 U.S.SC §119(e), the entire contents of which are incorporated herein by reference. Background Technology

[0003] Augmented reality (AR) displays and mixed reality (MR) displays, such as head-mounted displays (HMDs), provide real-time visual interaction with the real world. Figure One It has the ability to display virtual content. In the case of optical see-through displays, users view the real world through the optical components of the display.

[0004] Video perspective augmented reality displays and mixed reality displays capture video of real-world environments and display that video to the user in real time. These displays are typically designed to provide a realistic reproduction of real-world environments, which themselves may include other display devices such as television or computer screens. Accurate color reproduction contributes to the desired level of realism. These displays also allow the display of virtual objects, making them or other virtual content appear to the user as part of the real world. Accurate color reproduction of virtual objects contributes to a satisfactory integration of those objects with the real world. Summary of the Invention

[0005] In some implementations, a method includes: capturing at least one image of a real-world scene; identifying a captured screen content region in the at least one image; determining at least one visual parameter of the screen content from the captured screen content region; determining a transformation of virtual content based on the at least one visual parameter; applying the transformation to the virtual content; and causing the transformed virtual content to be displayed on an augmented reality display.

[0006] In some implementations, at least one visual parameter includes the brightness of the screen content, and the transformation includes a transformation of the brightness of the virtual content. The determined brightness of the screen content may be the maximum brightness in the captured screen content area. The transformation may include a gain factor applied to the color code value of the virtual content, wherein the gain factor increases as the determined brightness of the captured screen content area increases.

[0007] In some implementations, at least one visual parameter includes the white point of the screen content, and the transformation includes a transformation of the white point of the virtual content. Determining the white point of the screen content may include applying a gray-world algorithm to the captured area of ​​screen content.

[0008] In some embodiments, an image of a real-world scene is captured using a forward-facing camera of an augmented reality display. The augmented reality display can be an optical see-through head-mounted display.

[0009] In some embodiments, the method further includes identifying content displayed in the captured screen content area using the captured screen content area; and selecting virtual content based on the identified content.

[0010] In some embodiments, the method further includes identifying a type of display used to display the screen content; determining at least one additional visual parameter based on the type of display, and further transforming the virtual content based on the additional visual parameter.

[0011] Some embodiments include an apparatus comprising a processor configured to perform the methods described herein. The apparatus can include a non-transitory computer- readable medium storing instructions to perform the functions described herein. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 is a system diagram illustrating an example wireless transmit / receive unit (WTRU) that can be used as a client device in some embodiments.

[0013] Figure 2 is a schematic diagram of a scene including both real-world elements and virtual elements displayed on an augmented reality display.

[0014] Figure 3 is a schematic diagram of a real-world scene including screen content displayed on an external video display.

[0015] Figure 4 is a flowchart of a method performed by an external video display.

[0016] Figures 5A-5C is a flowchart of a method performed by a client device (e.g., an AR display) in some embodiments.

[0017] Figure 6 is a message flow diagram illustrating steps performed in some embodiments.

[0018] Figure 7 is a flowchart illustrating a method performed in some embodiments.

[0019] Figure 8 is a schematic diagram of an optical see-through head-mounted display that can be used in some embodiments.

[0020] Figure 9is a schematic diagram of a video see-through head mounted display that can be used in some embodiments.

[0021] Figure 10 determination of an angular size of an object displayed on a client device is schematically illustrated.

[0022] Figure 11 is a flowchart of a method of performing size-based color adjustment in accordance with some embodiments.

[0023] Figure 12 is a schematic diagram of a face swap application in which size-based color adjustment can be used.

[0024] Figure 13 is a flowchart of a method of adjusting the appearance of virtual content in accordance with some embodiments.

[0025] Exemplary devices for implementing embodiments

[0026] Figure 1 is a system diagram illustrating an example wireless transmit / receive unit (WTRU) 102. As shown, the WTRU 102 can include a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138, among others. It will be appreciated that the WTRU 102 can include any sub-combination of the foregoing elements while remaining consistent with embodiments. Figure 1

[0027] The processor 118 can be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Array (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor 118 can perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 can be coupled to the transceiver 120, which can be coupled to the transmit / receive element 122. While Figure 1 The processor 118 and the transceiver 120 are depicted as separate components, but it will be understood that the processor 118 and the transceiver 120 can be integrated together in an electronic package or chip.

[0028] ​The transmit / receive element 122 can be configured to transmit signals to, or receive signals from, a base station 1 14 over the air interface 1 16. For example, in one embodiment, the transmit / receive element 122 can be an antenna configured to transmit and / or receive RF signals. In an embodiment, the transmit / receive element 122 can be an emitter / detector configured to transmit and / or receive IR, UV, or visible light signals, for example. In yet another embodiment, the transmit / receive element 122 can be configured to transmit and / or receive both RF and light signals. It will be appreciated that the transmit / receive element 122 can be configured to transmit and / or receive any combination of wireless signals.

[0029] Although the transmit / receive element 122 is depicted in the embodiments herein as a single element, the WTRU 102 can include any number of transmit / receive elements 122. More specifically, the WTRU 102 can employ MIMO technology. Thus, in one embodiment, the WTRU 102 can include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116. Figure 1

[0030] The transceiver 120 can be configured to modulate the signals that are to be transmitted by the transmit / receive element 122 and to demodulate the signals that are received by the transmit / receive element 122. As noted above, the WTRU 102 can have multi-mode capabilities. Thus, the transceiver 120 can include multiple transceivers for enabling the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11, for example.

[0031] The processor 118 of the WTRU 102 can be coupled to, and can receive user input data from, the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or organic light-emitting diode (OLED) display unit). The processor 118 can also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. In addition, the processor 118 can access information from, and store data in, any type of suitable memory, such as the non-removable memory 130 and / or the removable memory 132. The non-removable memory 130 can include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 can include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 can access information from, or store data in, memory that is not physically located on the WTRU 102, such as on a server or a home computer (not shown). ​

[0032] The processor 118 may receive power from the power supply 134 and may be configured to distribute and / or control power to other components in the WTRU 102. The power supply 134 may be any suitable device for powering the WTRU 102. For example, the power supply 134 may include one or more dry cell battery packs (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0033] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) about the current location of the WTRU 102. In addition to or instead of the information from the GPS chipset 136, the WTRU 102 may receive location information from base stations (e.g., base stations 114a, 114b) via air interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It should be understood that, while remaining consistent with the implementation, the WTRU 102 may acquire location information using any suitable location determination method.

[0034] The processor 118 may also be coupled to other peripheral devices 138, which may include one or more software and / or hardware modules providing additional features, functions, and / or wired or wireless connectivity. For example, peripheral device 138 may include an accelerometer, electronic compass, satellite transceiver, digital camera (for photos and / or video), Universal Serial Bus (USB) port, vibration device, television transceiver, hands-free headset, etc. Modules, FM radio units, digital music players, media players, video game player modules, internet browsers, virtual reality and / or augmented reality (VR / AR) devices, activity trackers, etc. Peripheral devices 138 may include one or more sensors, which may be one or more of the following: gyroscopes, accelerometers, Hall effect sensors, magnetometers, orientation sensors, proximity sensors, temperature sensors, time sensors; geolocation sensors; altimeters, light sensors, touch sensors, magnetometers, barometers, gesture sensors, biometric sensors, and / or humidity sensors.

[0035] The WTRU 102 can include a full duplex radio for which transmission and reception of some or all signals (e.g., associated with particular subframes for both the UL (e.g., for transmission) and the downlink (e.g., for reception) can be concurrent and / or simultaneous. The full duplex radio can include an interference management unit to reduce and / or substantially eliminate self-interference and / or cross- interference that can occur during concurrent transmission and reception. In one embodiment, the WTRU 102 can include a half duplex radio for which transmission and reception of some or all signals (e.g., associated with particular subframes for either the UL (e.g., for transmission) or the downlink (e.g., for reception)) are time divided. DETAILED DESCRIPTION

[0036] Color reproduction for real-world displays .

[0037] It is desirable for a color reproduction system to have the ability to capture and display all inputs relevant to the application. In the case of a video pass-through head mounted display (HMD) or phone / tablet, the viewer’s field of view can include a second digital display (referred to herein as an “external display”), such as a television, computer monitor, or movie screen.

[0038] A computer-controlled digital display exhibits different properties than most real-world objects and light sources. Thus, from a color management perspective, the external display can benefit from a different treatment than the balance of the real-world scene in the viewer’s field of view. This is especially true if the external display contains content intended to have consistent coloration, such as a logo. Inconsistent display of such content can be easily noticed by the user and can detract from the viewing experience.

[0039] The relevant challenges are illustrated using a specific example. Consider a viewer wearing an HMD who is watching a video presentation on a regular television screen. An application on the HMD is providing virtual content (e.g., an animated character) associated with the video. The virtual character is being augmented into the HMD wearer’s field of view. Thus, the HMD wearer sees the primary video and the virtual character. Problems can arise due to the different color capabilities and settings that a television can have and the different color reproduction capabilities of the HMD. Without coordination, the video content and the virtual content can not share similar colors. The color differences make the AR content not fully integrated with the primary video.

[0040] Figure 2An example of augmented reality presentation in an environment with an external display is shown. The external display 202 is a real-world object. In the case of a video perspective augmented reality display, the video of the external display 202 and the video of the screen content displayed thereon (presented as 203) are captured by the augmented reality client system and displayed to the user in real time. Additionally, the augmented reality client system renders and displays virtual objects 204 and 206 to the user. Properties of the external display 202, the client device's camera, and the client device itself can all affect the colors of the external display 202, as seen on the client device's display. This can lead to inconsistencies between the colors displayed on the external display 202 and the colors of the virtual objects 204 and 206. Even in the absence of any virtual objects, the color reproduction of the external display 202 as seen through the client device may be unsatisfactory in the absence of color reproduction techniques as described herein.

[0041] In the case of an optical see-through augmented reality display, the user can directly view an external display 202 through the optics of the augmented reality display. It is desirable that the displayed virtual objects 204 and 206 have an appearance consistent with the content on the external display 202. However, the external display 202 and the user's augmented reality display may have different display attributes, such as different primary colors and different brightness levels. Furthermore, the content displayed on the augmented reality display may have been generated using a different white point or other attributes than the content displayed on the external display. Such differences can lead to an inconsistency in appearance between the content on the external display and the virtual content, which can diminish the user's enjoyment of the content.

[0042] In some implementations, the color reproduction method performed by the augmented reality client device operates as follows. An exemplary process involves detecting the presence of a digital display (“external display”) in the field of view of a video perspective augmented reality display.

[0043] The position of the external display is established and tracked within the observer's field of view. Based on this tracking process, dynamic (changing) areas on the client device that overlap (blur) with the external display are identified.

[0044] The client device receives out-of-band information about its attributes from an external display, such as the spectral power distribution of its RGB primary colors; ICC or other profiles; and possibly more. In some implementations, this information can be transmitted using the techniques described in the following documents: T. Li; C. An; X. Xiao, A.T. Campbell; and X. Zhou, “Real-Time Screen-Camera Communication Behind Any Scene,” MobiSys'15, May 18-22, 2015, Florence, Italy.

[0045] The client device applies out-of-band information to improve the appearance of the external display and seamlessly displays the improved rendering of the external display to the observer.

[0046] The exemplary methods described herein provide improved color reproduction for external displays, such as those rendered within a local display. In some implementations, color reproduction performed by the client device takes into account environmental interference between the observer and the external display (e.g., fog, smoke) and other processing that takes into account the larger color gamut on the HMD display (if appropriate) or can improve appearance by rendering the colors of the external display closer to the expected colors. This is especially true for color information intended to have a consistent appearance, such as security signals or brand / logo colors.

[0047] In embodiments where virtual objects are displayed near an image on an external display, the methods described herein can provide a more consistent appearance of the external display relative to the virtual objects.

[0048] Some implementation schemes can be achieved using the following steps.

[0049] Part 1 : Establish and track the position of the external display in the field of view of the observer .

[0050] An exemplary method is used to identify an external display. For the purposes of this specific embodiment, it can be assumed that the display is rectangular but can be viewed at an angle; therefore, the processor will identify and track a projected rectangle whose shape is limited to a constrained quadrilateral. Some embodiments use the technology described in US20160014350A1 to identify an external display.

[0051] Once the external display has been identified, the processor is used to accurately track its dynamic position. Some implementations use the technique described in the following literature to perform markerless tracking: G. Klein, “Visual Tracking for Augmented Reality,” PhD dissertation, University of Cambridge, January 2006. Once the initial position is known, the processor continues to track the position of the external display and determines the position of the corresponding pixels on the local display. Therefore, the tracking subsystem is used to provide the display system with information about where to insert the rendered external display so that it appears correct from the observer's perspective.

[0052] Part 2: Determine the properties of the external display .

[0053] The client device uses camera data and / or other methods to estimate the properties of the external display. In some implementations, this includes the chromaticity or spectral power distribution (SPD) and / or peak brightness of the RGB (or other) primary colors. Some implementations use this to determine as many of these properties as possible, and to represent the actual properties. Various techniques can be used to determine the properties of the external display.

[0054] In some cases, the brand and model of the display can be directly read via the HMD camera. The database is then accessed and camera attributes extracted. Similarly, the camera can be used to scan the shape, logo, or other distinguishing features of the display case, and the database can be searched based on these parameters.

[0055] In some implementations, point spectroradiometers or imaging spectroradiometers are used to directly measure the SPD of the display. In some cases, the display may contain a certain amount of pure red, green, and blue regions, which simplifies processing. In other cases, a few seconds of spectral measurements are collected and used to derive the SPD of each primary color. For example, principal component analysis can be used to extract fundamental (basic) spectral data from a series of measurements.

[0056] In some implementations, a colorimeter is used instead of a spectroradiometer as described above. A series of measurements can be collected. Once the measurements cover a sufficient percentage of the display's color gamut, the chromaticity diagram can be used to determine an estimate of the display's primary colors.

[0057] Once the SPD or color is measured (e.g., using a spectroradiometer or colorimeter as described above), the database can be searched to find the display's brand and model. Additional display attributes can be retrieved from the database.

[0058] In some implementations, the client device (e.g., HMD) determines the properties of the external display by communicating directly with it. In some implementations, the method described in T. Li et al. cited above is used to transmit information within the display scene in a manner that does not attract the user's attention or otherwise distract them. The information that may be transmitted in some implementations includes properties of the external display, such as: the spectral power distribution of its RGB primary colors; ICC or other profiles; and possibly more.

[0059] Part 3: Apply the determined properties to improve the appearance of the external display .

[0060] Figure 3 A real-world scene including an external display 302 is shown, which in this example is a large outdoor advertising video display.

[0061] Figure 4 This is a flowchart representing the color processing path of external display 302. Any specified pixel in the external display has a specified target color. This example references CIELAB coordinates, but other coordinate systems can be used. The external display processor converts the CIELAB coordinates to device RGB coordinates via its display profile. Once a given pixel is activated by the display system, the RGB coordinates result in predictable spectral radiation emitted by the external display at that pixel location.

[0062] Figure 5A This is a flowchart illustrating the steps performed at a client device (e.g., an HMD). The spectral radiation of the external display is detected by the client device's camera and converted into device (camera) RGB values. Using a camera profile, the client device's processor converts the RGB to CIELAB or other device-independent color coordinates. Note that a hypothetical scene white point is built into this CIELAB calculation. The white point can be determined by the client device from the scene content.

[0063] Figure 5B This is a flowchart illustrating additional steps performed at the client device. Based on the client device's camera data, the processor estimates the maximum brightness from the display. This is the CIELAB (and effective spectral radiance) obtained from the region with the maximum digital count (e.g., R=G=B=255). Based on the CIELAB pixel data, the maximum brightness, and the camera profile (inferred from the external display), the digital count of the pixels on the external display is estimated.

[0064] Figure 5CThis is a flowchart illustrating the additional steps performed at the client device. The expected spectral radiance is estimated using the RGB and spectral primary colors transmitted from the external display. The actual radiance can be reduced due to degradation of the external display or more likely interfering environmental factors (e.g., smoke, fog) or exposure settings on the HMD camera that respond to more than just the external display. Based on the spectral radiance, the expected CIELAB value is determined and used as the target display color for the local display. The local display profile converts this target CIELAB to device RGB and displays it on the local display. Thus, the viewer experiences the expected color.

[0065] Part 4: Seamlessly display the improved rendering of the external display to the observer on the local display .

[0066] Using the tracking described in Part 1, and by continuously updating the HMD display as described in Part 3, the appropriate colors of an external display are shown to the HMD user, which is appropriately positioned within its field of view to blend seamlessly into other real-world content.

[0067] Overview of external content adjustment .

[0068] Examples of methods implemented in some implementation schemes are in Figure 6 As shown in the diagram. A content source (e.g., a streaming video source) provides content to an external display, such as a television screen. The external display shows the content. A client AR display device near the external display captures video of a real-world scene, which includes the screen content being displayed on the external display. The client device determines the area of ​​the video that includes the external display device. The client AR display also determines the properties of the external display. In some embodiments, the properties of the external display may be transmitted from the external display itself. Based on the properties of the external display, the client device determines the color processing to be performed on the portion of the video corresponding to the external display. The client device selectively processes the colors in the portion of the video corresponding to the external display and displays the processed video to the user substantially in real time. In some embodiments, the client device also displays virtual content, which in some cases may be retrieved from the content source.

[0069] As described above, in some implementations, color processing is performed as follows. RGB values ​​are captured by a camera for an area corresponding to screen content on an external display. In a first transformation, these RGB values ​​are converted to device-independent color coordinates (e.g., XYZ color coordinates or CIELAB*a*b* coordinates). For simplicity, an example of CIELAB coordinates will be used here. This first transformation may be performed based on the camera's profile. In a second transformation, these CIELAB coordinates are converted to remote display RGB values, which represent estimates of the RGB values ​​that produce the colors indicated by the CIELAB coordinates displayed on the external display. In a third transformation, the remote display RGB values ​​are converted to the expected spectral radiance. This third transformation may be based on the remote display RGB values ​​and the radiance of the spectral primary colors. In a fourth transformation, the expected spectral radiance is converted to the expected CIELAB coordinates. In a fifth transformation, the expected CIELAB coordinates are converted to client display RGB values ​​using the local display profile for display on the client device.

[0070] In some implementations, two or more transformation steps can be combined instead of performing separate transformation steps. For example, consider an implementation where the RGB values ​​of the client camera are represented as R... c G c B c Furthermore, the RGB values ​​of the client monitor are represented as R. d G d B d In some implementations, the transformation steps can be combined into processing steps, such as the following linear transformation, where A is a 3×3 matrix.

[0071]

[0072] In some implementations, such linear transformations can replicate the results of performing individual transformation steps. In other implementations, linear transformations can be used as first-order approximations of the results of performing individual transformation steps. In some implementations, transformation steps can be combined into a single nonlinear transformation step.

[0073] In some implementations, the transformation matrix A (or other color processing parameters) can be calculated in response to the detection of an external display device and the determination of the device's attributes. In other implementations, one or more transformation matrices are pre-stored. For implementations that pre-store more than one transformation matrix, different matrices can be stored for different types of external displays. For example, different transformation matrices can be stored for use with an LCD screen, but not with an OLED screen, and so on. In some implementations, the transformation matrix can be adjusted or selected based at least in part on lighting conditions, such as the white point of the scene.

[0074] In some implementations, screen content color transformation is chosen (whether implemented as a single transformation or a series of transformations) such that a user viewing screen content through a video see-through head-mounted display will perceive on the screen the colors that are substantially the same as those he or she would perceive if he or she were watching the same video directly without a head-mounted display.

[0075] Figure 7 A flowchart of an exemplary method is provided. In this example, a client device, such as a video perspective head-mounted display, captures video of a real-world scene and identifies screen content regions within the captured video. The client device determines color processing parameters, such as elements of a transformation matrix A or other parameters described herein for color coordinate transformation. The client device selectively processes colors (e.g., RGB values) within the screen content regions. In some implementations, screen content color processing is not performed outside the screen content regions (although other color processing may be performed outside the screen content regions). The processed video is then displayed to the user substantially in real time (although some processing latency is unavoidable).

[0076] Adjusting the content of the external display to match the virtual content of the local display .

[0077] In some implementations, a client device associated with an augmented reality or mixed reality (AR / MR) display is used to modify, enhance, or re-render content seen on an external display to be compatible with the virtual content to be displayed in the AR / MR display. Compatibility may include adjusting the brightness and / or color of the content on the external display to match or otherwise align with the brightness or color attributes of the displayed virtual content.

[0078] In the first example, the user may have a head-mounted display equipped with a forward-facing camera and a "see-through" AR / MR display. The AR / MR display allows the user to see their surroundings, including any nearby external display devices. In some implementations, the following steps can be used to determine the properties of the content displayed on the external display and to adjust the appearance of the content on the external display to be compatible with other content to be displayed on the AR / MR display (e.g., virtual content, including augmented or reality-rendered objects).

[0079] Use a forward-facing camera to capture images of the surrounding environment.

[0080] The captured image is analyzed to detect nearby external displays. This can be accomplished using various computer vision techniques for object recognition, such as those described in the following literature: Nacenta, Miguel, “Computer Vision Approaches to Solve the Screen Pose Acquisition Problem for Perspective Cursor,” Technical Report HCI-TR-06-01, Department of Computer Science, Saskatchewan University, December 5, 2005. Such techniques can be used to identify the presence of external display devices and provide the location and extent, or “occupancy area,” of the external display device within the captured image.

[0081] The range of content displayed on an external display device can be identified. For example, this can be done by starting with the area occupied by the external display device and removing its frame or border from consideration, so that only the active screen area of ​​the external display is considered in subsequent steps. Alternatively, captured images can be compared over time to determine which rectangular area within the external display device's footprint changes over time, and this area can be identified as the active screen area to be considered in subsequent steps.

[0082] Samples captured within a range of content displayed on an external display device (e.g., within an "active screen area") can be analyzed to determine the visual properties of the externally displayed content. For example, a luminance histogram of these samples can be created to determine luminance properties such as minimum, maximum, and average brightness. Similarly, the color properties of these samples can be analyzed to determine the color properties of the content displayed by the external display device. Such analysis can be performed on a single captured frame or computed over a time interval, wherein multiple (or many) captured frames are used as input within that time interval.

[0083] The properties of the externally displayed content can be used to determine modifications to that content, improving compatibility with other content to be displayed in the AR / MR display (e.g., virtual content, including augmented or reality-rendered objects). For example, the brightness or color properties of the externally displayed content can be adjusted so that the user sees the same or similar brightness and color properties between the user's view of the externally displayed content and other content displayed in the AR / MR display. This modification can take the form of adjusting the brightness or color of captured samples within the range of content displayed on the external display device. The captured samples can be adjusted, and the resulting adjusted samples can be rendered as an "overlay" in the AR / MR display, allowing the user to see the adjusted sample of the externally displayed content replacing the real-world (or "perspective") view of that content.

[0084] In some implementations, an AR / MR HMD user may encounter an external display already showing content from an external display. Detection of the external display can trigger additional steps to determine and adjust the brightness and / or color attributes of the content on the external display, as seen by the user through the AR / MR HMD. Other content (e.g., virtual content, augmented and / or reality-rendered objects) can be “accompanying content” generated for use with the content on the external display. For example, if the external display is a football game, other content to be displayed on the AR / MR display could be scores, statistics, and commentary about the game. Similarly, if the external display is a movie, other content could be a virtual rendering of one of the characters appearing alongside the screen in the movie, along with commentary on the film. Regarding accompanying content, additional steps may exist to identify the content currently playing on the external display, such that corresponding other content (e.g., virtual content, augmented and / or reality-rendered objects) is generated, retrieved from the network, or otherwise obtained and prepared for rendering in the AR / MR display device along with the content on the external display.

[0085] In another variation of the above implementation, an AR / MR HMD user may encounter an external display, but the content displayed on the external display may not yet be playing on it. Instead, the detection of the external display device (visually as described above, or via network connectivity and device discovery processes, such as via UPnP) allows the client device (e.g., the AR / MR display or device or its associated controller device) to instruct the external display device to begin playing content selected by the client device. For example, the client device may have access to virtual content to be displayed as an accompaniment to a given movie, and the client device may (after detecting the presence of an external display device in the environment) request the external display device to begin playing the movie, allowing the client device to then enhance the movie-watching experience by displaying the accompanying content on the AR / MR display. In this case, the client device does not need to identify what content is playing on the external device (because it already knows what is being played). The client device can be used to identify the location and extent (e.g., “occupying area”) of the external display device in the user's view, and in the case of multiple external display devices in the environment, it can visually verify that the expected content is playing on the detected external display device. Client devices can be used to adjust the brightness and / or color attributes of externally displayed content to be compatible with other content to be displayed in the AR / MR HMD (e.g., virtual content, annotations, and / or accompanying content of realistically rendered objects).

[0086] In some implementations, other content to be displayed in the AR / MR HMD may not be supplementary content and may not have been specifically prepared for use with externally displayed content. For example, the brightness and / or color attributes of the externally displayed content may be adjusted to be compatible with user interface (UI) elements and annotations typically displayed in the AR / MR HMD. This can be done as a style point or to ensure that UI elements and annotations displayed in the AR / MR HMD remain visible in the presence of potentially bright externally displayed images.

[0087] In the second example, the user may have a head-mounted display equipped with a forward-facing camera and a "see-through" AR / MR display. As previously mentioned, the AR / MR display allows the user to see their surroundings, including nearby external display devices. The following steps can be used to determine the properties of the external display and use such properties to adjust the appearance of the content on the external display to be compatible with other content to be displayed in the AR / MR display (e.g., virtual content, including augmented or reality-rendered objects).

[0088] Use a forward-facing camera to capture images of the surrounding environment.

[0089] The captured image is analyzed to detect nearby external displays. This can be accomplished using various computer vision techniques for object recognition, such as those described in Nacenta cited above. Such techniques can be used to identify the presence of external display devices and provide the location and extent (“occupancy”) of the external display device within the captured image.

[0090] Identify the extent of content displayed on the external display device. This can be done, for example, by starting with the area occupied by the external display device and removing its frame or border from consideration, so that only the active screen area of ​​the external display is considered in subsequent steps. In some implementations, captured images can be compared over time to determine which rectangular area within the area occupied by the external display device changes over time, and this area can be identified as the active screen area to be considered in subsequent steps.

[0091] Determine the attributes of the external display that will affect the displayed content. The attributes of the external display can be determined by identifying the brand and model of the external display and retrieving attributes corresponding to that brand and model from a database. The database may be local to the client device (e.g., an AR / MR HMD device or its associated controller). Alternatively, the database may be accessible by the client device via a network. The attributes of the external display may also be determined by communicating with the external display (e.g., using point-to-point wireless communication, or communicating with the display device via a network). The determined attributes of the external display may include fixed attributes of the external display, such as maximum achievable brightness, primary colors, and color space. The determined attributes of the external display may include variable attributes that are not inherent to the brand and model of the display, such as current brightness or color settings, the current "preset" mode of the display in use, or an indication of the lifespan of the display device.

[0092] The identified properties of the external display can be used to determine modifications to the content displayed on the external display, which will improve compatibility with other content to be displayed in the AR / MR display (e.g., virtual content, including augmented or reality-rendered objects). For example, the brightness or color properties of the content on the external display can be adjusted so that the user sees the same or similar brightness and color properties between the user's view of the content on the external display and other content displayed on the AR / MR display. This modification may include brightness or color adjustments to the captured samples within the scope of the content displayed on the external display device. The captured samples can be adjusted, and the resulting adjusted samples can be rendered as an "overlay" in the AR / MR display, so that the user sees the adjusted sample of the content on the external display replacing the real-world (or "perspective") view of that content.

[0093] Additional features of some embodiments .

[0094] In some implementations, the client device is used to simultaneously track and process multiple external displays.

[0095] In some implementations, the colors used to render the virtual objects are selected to match the colors on an external display that is reproduced through a video-perspective AR display.

[0096] In some implementations, to enhance the user experience, additional content determined to be relevant and related to the content being displayed on the external display is inserted, in addition to simply and correctly rendering the external display. For example, if a movie scene is being displayed, the client device can be used to find relevant content (e.g., the location of nearby stores selling related goods) and display the content adjacent to the external display in the user's field of view. To align the color experience with the content on the display, the client may follow the color processing path described in Section 3 above ("Applying inferred attributes to improve the appearance of the external display").

[0097] In some implementations, the methods described herein are used to compensate for changes in the color of an external display caused by variations in local environmental conditions and to correct the color for the user.

[0098] In very bright or dark conditions, a local camera can be configured to appropriately capture the colors of an external display (e.g., by reducing or increasing the exposure time). Using the methods described herein, an external display can be appropriately rendered if it might otherwise be washed out or too dark to be viewed by the user (or viewer).

[0099] Color reproduction based on the angular size of objects .

[0100] When content is inserted into an augmented reality system or a video pass-through system (e.g., a phone or tablet), the apparent size of the object can affect the color perceived by the observer. This can be particularly noticeable when the inserted object is a face. For example, face-swapping applications or other face-modification technologies can alter the size of the input face before final display. Given the reasonable range of object size variations and the available range of screen sizes, noticeable color shifts can become apparent. Therefore, the angular size of an object can affect its color. That is, the perceived color of an artifact is a function of the size of the artifact in the observer's field of vision. If the angular size of an object changes (e.g., if the augmented reality content is modified to appear closer to or further away from the observer), the perceived color can change even when displaying the same RGB values.

[0101] In some implementations, the processor detects the angular size (relative to the observer) of the content of interest. In general, this is augmented reality content, but for video passthrough (phones or tablets), it can be anything within the field of view. An example of video passthrough could be detecting a logo or other color-critical item and taking its visual size into account when presented to the display. For either application, the application model compensates for the observer's visual attributes with a size based on the field of view, thereby adjusting the color of the content of interest to reflect its apparent size.

[0102] Once the angular size of the content is determined, this parameter, along with other parameters (such as environmental parameters like white point), is used to generate a rendering model. The inserted content is then rendered (or re-rendered) to set the color based on that apparent size. If the inserted object remains in the field of view but has changed size relative to the observer, a new rendering model is determined, and the object is re-rendered using forward and backward color modeling techniques. In some implementations, the rendering model is pre-calculated, at least in part, using a range of expected object sizes.

[0103] In some implementations, display systems such as AR glasses or phones / tablets are used to indicate the apparent visual size of any rendered content. Such methods can be used to employ a color model that considers several factors, including the observer's age and the visual size of the content. A model that can be used in some implementations is the CIE 2006 model. Existing techniques modify the geometry of objects but do not consider color changes.

[0104] The methods described herein can create more realistic and less conspicuous renderings of objects that change size within the field of view or are rendered to a size different from what the observer initially sees (e.g., in a face-swapping application). Such methods can be used not only with inserted content but also to detect known objects within the field of view and correct their colors after taking their visual size into account.

[0105] Such methods may be particularly suitable when presenting color matching to an observer, such as when the rendered objects are intended to match real-world objects. The matching can be decomposed when one or both of these objects change their apparent size relative to the observer, in the absence of a color compensation method as described in this paper.

[0106] Figure 10 This is a schematic perspective view illustrating how the angular size of the rendered content is determined in some implementations. The angle θ will be used as one of the parameters in the CIE 2006 model or other models of the observer's spectral sensitivity. The angle θ can be calculated as...

[0107] θ=2tan -1 (s / 2d)

[0108] In some implementations, a spectral sensitivity model is used to adjust the rendering pipeline.

[0109] When inserting an object, or if the object's size increases or decreases (e.g., if the observer is to experience the inserted content moving closer or further away), a process can be applied that specifies the visual angle size of the rendered content. During initial rendering, or during re-rendering when the object moves, a step can be performed to determine the distance from the display to the observer, which allows the angle size to be calculated, and rendering can continue from there.

[0110] Figure 11 This is a flowchart illustrating a rendering method performed in some implementations. The display device identifies the content to be rendered. This content can be a newly inserted object or an object whose size or position has changed. The processor, for example, determines the angular size of the object based on its dimensions and the distance from the object to the viewer's eye, such as... Figure 10 As shown. For example, when the display device is an HMD, other techniques can be implemented to determine the angular size of the object. Some implementations take into account the observer's age, which can affect the user's color perception. Other implementations may use average age or other representative age, without specifying the current user's actual age. An appropriate color matching function is determined at least in part based on the object's angular size and / or the observer's age. The object can then be rendered (or the rendered color can be updated) based on the determined color matching function.

[0111] In some implementations, the use of observer-dependent color matching functions can be achieved using techniques described in the following literature: Patent application No. 16 / 280866, filed February 20, 2019, entitled “Method and Apparatus for Increasing Color Accuracy of Display by Compensating for Observer’s Color Vision Properties,” the entire contents of which are incorporated herein by reference.

[0112] When colors are matching between an RGB display and the real world, an implementation using angular size-based color reproduction can be employed. This type of matching might be conditional isochromatic, where the physical properties (spectral properties of the materials) don't match, but the colors do. This is a characteristic of visual systems that allows the observer to perceive a perfect color match between a rendered object and a real-world object.

[0113] Conditional isochromatic coloring is used when an observer is searching for a product or object of a specific color and comparing the rendered color to real-world colors. In this case, changing the apparent size can affect color matching. A small render of an object that is far away (and therefore visually smaller) may be perceived as a perfect match, but this match can be broken down as the object gets closer and fills more of the field of view. If the rendered RGB colors are not adjusted, they will look identical and may not match larger real-world objects.

[0114] Another application of conditional color matching is color coordination (commonly known as "matching," as described in "Does this shirt match this tie?"). This type of matching is done for many color-critical purposes, such as fashion (clothing), furnishings, pigments, etc. For these applications, hue is often the most critical dimension being evaluated. In the same way that precise matching can be decomposed, the hue of a real-world object can vary with the size of the field of view. In this case, the match chosen by the observer will no longer appear pleasing, or will better appear as different color groups.

[0115] Changes in the color matching function due to changes in object size are analogous to changes in observer size and should be expected to have the same effect on conditional isochromaticity as other changes. For example, changing the illuminator may alter the conditional isochromaticity. In some embodiments described herein, the color of the displayed object can be adjusted to maintain a match. The color matching function described in CIE 2006 can be used to adjust the color.

[0116] like Figure 10 As shown, to accurately determine the angular size of the inserted content, some implementations use the distance to the observer. This can be accomplished using methods for detecting object distances using a monocular camera, as described in, for example, the following literature: Peyman Alizadeh, Object Distance Measurement Using a Single Camera for Robotic Applications, MSc Dissertation, Laurentian University, Sudbury, Ontario, Canada (2015). Methods such as using a single forward-facing camera or using ranging techniques (e.g., time-of-flight sensors) can also be employed. Figure 10 A client device with a display 1002 viewed by an observer 1006 is shown. In this example, the client device has a forward-facing camera 1004. The client device is used to determine the viewing distance d from the observer based on, for example, the size of an image of the observer captured by the camera 1004 (e.g., based on the assumption that the user's pupillary distance is approximately 60 mm, or based on other relatively consistent facial feature structures) or based on other distance measurement techniques (such as those described by Alizadeh). Some implementations may use a default distance from the observer based on the intended use of the client device. The observer observes an object with an angular size θ, where θ can be determined trigonometrically based on the viewing distance d and the physical size s of the object as rendered on the screen.

[0117] Some implementations perform size-based color reproduction of real-world objects in a scene. Logos and other color-sensitive items can be identified in the scene (e.g., by comparison with a database) and then color-corrected based on size. This utilizes information about the actual color of the logo or item, which may be known (e.g., available in a database). Because all content is rendered to a display, such implementations can be performed in video-pass-through systems (e.g., phones / tablets).

[0118] Popular phone applications include several that perform "face swapping" or other facial adjustment methods. It has been established that people are very sensitive to the accuracy of facial and skin tone colors. Therefore, these applications particularly benefit from more accurate rendering that takes into account the size of the inserted content. Figure 12 As shown, performing a "face swap" on the original image (left) to generate a swapped image (right) can involve changing the face size by two times or more. Implementation schemes that adjust colors based on object size can be implemented to provide more accurate color matching. Regarding more traditional parameters for accurate rendering, adjustments may have shifted some skin tones to areas of the viewfinder with different illuminators (white points). In this case, the rendering engine can be used to produce more accurate colors by taking into account this difference in addition to the size effect.

[0119] Client devices used in some embodiments .

[0120] The implementation scheme described in this article can be achieved using various types of augmented reality display devices. Figure 8 and Figure 9 An example of a head-mounted display that may be employed in an exemplary implementation is illustrated.

[0121] Figure 8 An optically transparent head-mounted display is shown. Figure 8 In this implementation, a liquid crystal display (LCD) 802 is used to render virtual content. A partially transparent reflector 804 (e.g., a semi-silvered reflector or beam splitter) allows the user to see both an image of the real world and an image of the content displayed on the LCD. Viewing optics 806 are provided to allow the user to focus on the virtual content. The LCD, viewing optics, and partially transparent reflector are components of the main display. A protective cover 808 is mounted externally to the main display.

[0122] Figure 8The head-mounted display includes a forward-facing camera 810, which helps track the current orientation of the user's head. In some embodiments, the forward-facing camera is also used to measure the optical properties of the current real-world scene (e.g., illuminance). In other embodiments, one or more separate illuminance sensors are used to measure the scene's optical properties. A control module 812 controls the display 802. The control module 812 can use information about the scene's illuminance (and, in some embodiments, white point) to control the brightness of the virtual content displayed on the LCD. It should be noted that while an LCD display is used as an example herein, other types of displays may be used alternatively. Figure 8 The optical perspective display provides the user with a direct view of the real world (through a partial reflector 804). The user can also view virtual objects displayed on the LCD 802.

[0123] Figure 9 The head-mounted display is a video-perspective head-mounted display device. The display includes a forward-facing camera 910 that captures real-time images of the real-world environment. The real-time video of the real-world environment (possibly after processing) is displayed on an LCD display 902. Virtual content may also be displayed on the LCD display 902. Viewing optics 906 are provided to allow the user to focus on the LCD display. A control module 912 controls the LCD display 902. The control module 912 can use information about scene illumination (and, in some embodiments, white point) to control the brightness of the virtual content displayed on the LCD. It should be noted that while an LCD display is used as an example herein, other types of displays may be used alternatively. Figure 9 The video perspective display provides users with a real-time video view of the real world along with any virtual objects displayed on the LCD 902.

[0124] Adjustment of virtual content .

[0125] Some implementations adjust the colors of virtual content to improve compatibility between the virtual content's colors and those displayed on a physical monitor. Such implementations can be used with both optical see-through and video see-through head-mounted displays.

[0126] Consider a video being displayed on a regular television screen while a viewer wearing an HMD is watching. An application on the HMD is providing virtual content associated with the video. Ideally, the colors of the virtual content should be a reliable match to the colors displayed on the television. Color discrepancies can cause the virtual content to not fully integrate with the main video.

[0127] Some exemplary implementations operate as follows: The presence of a digital display (e.g., an external display) in the field of view is detected. The position of the external display in the observer's field of view is determined. Thus, the dynamic (changing) area in the local display that overlaps with (blurs) the external display is known (or can be determined).

[0128] Client devices associated with an HMD or other local display can receive out-of-band information describing the properties of the external display, such as the spectral power distribution of its RGB primary colors; International Color Consortium (ICC) or other profiles; and possibly more. This information can be transmitted, for example, using the techniques described in Real-Time Screen-Camera Communication Behind Any Scene. T Li; C An; X. Xiao, ATCampbell; and X. Zhou, MobiSys '15, May 18-22, 2015, Florence, Italy.

[0129] Out-of-band information is used to generate virtual content that is visually compatible with what is seen on an external display. The virtual content is then displayed on the local display using colors that have been generated or adjusted to match the content displayed on the external display.

[0130] Some implementations are used to identify external displays. In some cases, the display may be rectangular but can be tilted for viewing, and the client device can therefore be used to identify and track the projected rectangle, the shape of which may approximate a constrained quadrilateral.

[0131] The client device can also be used to track the dynamic position of an external display, for example, using markerless tracking. Once the initial position is known, the client device can continue tracking the position of the external display and determine the position of the corresponding pixel on the local display.

[0132] Using camera data and other methods, the client device can estimate the properties of the external display. Examples of display properties may include the chromaticity or spectral power distribution (SPD) and / or peak brightness of the RGB (or other) primary colors. This can be summarized in an ICC color profile corresponding to the operation of the external display.

[0133] In some implementations, the brand and model of the display can be read directly via an HMD camera. The database is then accessed and the external display attributes are extracted. Similarly, a camera can be used to scan the shape, logo, or other distinguishing features of the display case, and the database can be searched based on these parameters.

[0134] In some implementations, point spectroradiometers or imaging spectroradiometers can be used to directly measure the SPD of the display. In some cases, the display may contain a certain amount of pure red, green, and blue regions. In other cases, a few seconds of spectral measurement can be used to infer the SPD of individual primary colors. For example, principal component analysis can be used to extract fundamental (basic) spectral data from a series of measurements.

[0135] In some implementations, a colorimeter is used instead of a spectroradiometer. A series of measurements can still be advantageous. A chromaticity diagram can allow for a good estimate of the display's primary colors, provided the measurements cover a sufficient percentage of the display's color gamut.

[0136] Once the SPD or other color attributes have been measured, the measured SPD or color information can be used as query input to search the database. Therefore, the brand and model of the monitor can be identified, and / or additional attributes of the monitor can be retrieved.

[0137] In some implementations, the color attributes of the external display are retrieved via direct communication between the external display and the HMD or other client devices. Such communication may be conducted via wireless radio communication between the HMD and the external display (e.g., Bluetooth or NFC), or via wireless networks (e.g., one or more of cellular and WiFi networks). In some implementations, the communication may be embedded in visible light from the display. For example, the technology described in Real-Time Screen-Camera Communication Behind Any Scene can be used to transmit this information. (TLi; C An; X. Xiao, ATCampbell; and X. Zhou, MobiSys '15, May 18-22, 2015, Florence, Italy). Relevant attributes of the external display may include: the spectral power distribution of its RGB primary colors; the peak brightness of the external display; or ICC or other profiles. In some implementations, in addition to the fixed attributes of the external display device, the current device settings of the external display (e.g., current color settings, current brightness settings, the current display "preset" mode in use, etc.) may also be transmitted.

[0138] The determined display properties can be used to adjust the appearance of virtual content, which can then be displayed to enhance the content on the external display.

[0139] The HMD can generate or adjust virtual content for display within the HMD to match determined attributes of the external display (e.g., color and brightness attributes).

[0140] Adjusting virtual content based on a captured view of the external display content .

[0141] In some implementations, a client device associated with an augmented reality or mixed reality (AR / MR) display is used to modify, adjust, or enhance virtual content to be displayed on the AR / MR display to ensure compatibility with content displayed on an external display device. The external display device may be close to the AR / MR display device and visible to the user of the AR / MR display device. Compatibility may include adjusting the brightness and / or color of the virtual content to match or otherwise align with the brightness or color attributes of the externally displayed content.

[0142] In one example, a user may have a head-mounted display equipped with a forward-facing camera and a "see-through" AR / MR display. The AR / MR display allows the user to see their surroundings, including any nearby external display devices. The following steps can be used to determine the properties of the content displayed on the external display and adjust the appearance of other content to be displayed on the AR / MR display (e.g., virtual content, including augmented or reality-rendered objects) to be compatible with the user's view of the externally displayed content.

[0143] Use a forward-facing camera to capture images of the surrounding environment.

[0144] The captured image is analyzed to detect nearby external displays. This can be accomplished using various computer vision techniques for object recognition, such as those described in Nacenta cited above. Such techniques can be used to identify the presence of external display devices and provide the location and extent (or “occupancy”) of the external display devices within the captured image.

[0145] Identify the extent of the content displayed on the external display device. This can be done, for example, by starting with the area occupied by the external display device (obtained from previous steps) and removing the frame or border of the external display device from consideration, so that only the active screen area of ​​the external display is considered in subsequent steps. In some implementations, captured images can be compared over time to determine which rectangular area within the area occupied by the external display device changes over time, and this area can be identified as the active screen area to be considered in subsequent steps.

[0146] Samples captured within a range of content displayed on an external display device (e.g., within an active screen area) can be analyzed to determine the visual properties of the externally displayed content. For example, a luminance histogram of these samples can be created to determine luminance properties such as minimum, maximum, and average brightness. Similarly, the color properties of these samples can be analyzed to determine the color properties of the content displayed by the external display device. Such analysis can be performed on a single captured frame or computed over a time interval, wherein multiple (or many) captured frames are used as input within that time interval.

[0147] The properties of the externally displayed content can be used to determine modifications to other content to be displayed in the AR / MR display (e.g., virtual content, including augmented or reality-rendered objects). Modifications can be determined to make the other content compatible with the user view of the externally displayed content. For example, brightness or color properties of the other content can be adjusted so that the user sees the same or similar brightness and color properties between the user view of the externally displayed content and the other content displayed in the AR / MR display. Modifications can take the form of brightness or color adjustments to the other content to be displayed in the AR / MR display.

[0148] In some implementations, AR / MR HMD users may encounter an external display already showing content from an external display. Detection of the external display can trigger additional steps to determine and adjust the brightness and / or color attributes of other content to be displayed in the AR / MR HMD (e.g., virtual content, augmented and / or reality-rendered objects). This other content can be supplementary content generated to be used with the content on the external display. For example, if the external display is a rugby game, the other content to be displayed on the AR / MR display could be scores, statistics, and commentary about the game. Similarly, if the external display is a movie, the other content could be a virtual rendering of one of the characters appearing alongside the screen in the movie, along with commentary on the film. Regarding supplementary content, some implementations can be used to identify content currently playing on the external display such that corresponding supplementary content (e.g., virtual content, augmented and / or reality-rendered objects) can be generated, retrieved from the network, or otherwise obtained and then modified or adjusted according to the steps described above before being displayed on the AR / MR display device.

[0149] In some implementations, AR / MR HMD users may encounter external displays, but the content displayed on these external displays may not yet be playing on them. Instead, the detection of external display devices (visually or via network connectivity and device discovery processes, such as via UPnP) allows the AR / MR device or its associated controller device to instruct the external display device to begin playing content selected by the AR / MR device. For example, the AR / MR device may access virtual content to be displayed as an accompaniment to a given movie, and the AR / MR device may (after detecting the presence of an external display device in the environment) request the external display device to begin playing the movie, allowing the AR / MR device to then enhance the movie-watching experience by displaying the accompanying content. In this case, the AR / MR device or its associated controller device does not need to identify what content is playing on the external device. The AR / MR device or its associated controller device can be used to identify the location and extent (e.g., “occupying area”) of the external display device in the user's view, and, in the case of multiple external display devices in the environment, can visually verify that the expected content is playing on the detected external display device. AR / MR HMDs or associated controller devices can be used to adjust the brightness and / or color attributes of accompanying content (which may be virtual content, annotations, and / or realistically rendered objects) to be compatible with the content displayed externally as seen by the AR / MR HMD user.

[0150] In some implementations, other content to be displayed in the AR / MR HMD may not be supplementary content and may not have been specifically prepared for use with externally displayed content. For example, the brightness and / or color attributes of UI elements and annotations typically displayed in the AR / MR HMD may be adjusted to be compatible with the determined brightness and / or color attributes of the externally displayed content. In some implementations, this is done to ensure that UI elements and annotations displayed in the AR / MR HMD remain visible in the presence of potentially bright externally displayed images.

[0151] Exemplary implementations in Figure 13As shown in the diagram. A client device (e.g., an HMD and / or a controller associated with the HMD) captures at least one image of a real-world scene (1302). The client device identifies (1304) a captured screen content area in one or more images that will be displayed on an external display (such as a computer monitor, television, or movie screen). The client device determines (1306) at least one visual parameter of the screen content from the captured screen content area. For example, the client device may determine a brightness (1308), which may be a maximum brightness. The client device may determine a white point of the screen content (1310), which may be performed using a gray-world algorithm. In some embodiments, the client device may determine more than one visual parameter. The client device determines a transformation (1312) based on the determined one or more visual parameters. In some embodiments where the visual parameter includes the brightness of the screen content, the transformation includes a transformation of the brightness of the virtual content. The brightness transformation may be a gain factor applied to the color code value of the virtual content. The gain factor may increase as the determined brightness of the captured screen content area increases. In some implementations where visual parameters include the white point of screen content, the transformation may include a transformation of the white point of virtual content. In some implementations, an offset is applied in addition to the gain factor.

[0152] The client device applies one or more determined transformations to virtual content (1314), and the client device causes the transformed virtual content to be displayed (1316) on an augmented reality display. To display the transformed virtual content, if the client device itself is equipped with an augmented reality display device such as an HMD, the client device may display the transformed virtual content. If the client device is a controller for an associated display device, the client device may cause the transformed virtual content to be displayed by transmitting a signal representing the transformed virtual content to the associated augmented reality display device.

[0153] In some implementations, virtual content is selected based on content displayed on an external display. In one example of such an implementation, the client device identifies (1318) the content displayed in the captured screen content area and selects (1320) the virtual content to be displayed based on the identified content.

[0154] Adjusting virtual content based on the identified properties of the external display device .

[0155] In some implementations, augmented reality or mixed reality (AR / MR) displays are used to modify, adjust, or enhance virtual content to be displayed on the AR / MR display to be compatible with content displayed on an external display device. The external display device may be close to the AR / MR display device and visible to the user of the AR / MR display device. Compatibility may include adjusting the brightness and / or color of the virtual content to match or otherwise align with the brightness or color attributes of the externally displayed content.

[0156] In one example, a user might have a head-mounted display equipped with a forward-facing camera and a "see-through" AR / MR display. The AR / MR display allows the user to see their surroundings, including any nearby external display devices. The following steps can be used to determine the properties of the external display and use these properties to adjust the appearance of other content to be displayed in the AR / MR display (e.g., virtual content, including augmented or reality-rendered objects). Other content can be visually adjusted to be compatible with the user's view of the externally displayed content.

[0157] A forward-facing camera is used to capture images of the surrounding environment. The captured images are then analyzed to detect nearby external displays. This can be accomplished using various computer vision techniques for object recognition, such as those described in Nacenta cited above. Such techniques can be used to identify the presence of external display devices and to provide the location and extent (or area occupied) of the external display devices within the captured images.

[0158] The attributes of the external display that will affect the externally displayed content are determined. The attributes of the external display can be determined by identifying the brand and model of the external display and retrieving attributes corresponding to that brand and model from a database. The database may be local to the AR / MR HMD device or its associated controller. Alternatively, the AR / MR HMD device or its associated controller may access the database via a network. The attributes of the external display may also be determined by communicating with the external display (e.g., using point-to-point wireless communication, or communicating with the display device via a network). The determined attributes of the external display may include fixed attributes of the external display, such as maximum achievable brightness, primary colors, and color space. The determined attributes of the external display may include variable attributes that are not inherent to the brand and model of the display, such as current brightness or color settings, the current "preset" mode of the display in use, or an indication of the lifespan of the display device.

[0159] The identified display properties can be used to determine modifications to other content to be displayed in the AR / MR display (e.g., virtual content, including augmented or reality-rendered objects). Modifications can be determined to make the other content compatible with the user view of the externally displayed content. For example, brightness or color attributes of the other content can be adjusted so that the user sees the same or similar brightness and color attributes between the user view of the externally displayed content and the other content displayed in the AR / MR display. Modifications can take the form of brightness or color adjustments to the other content to be displayed in the AR / MR display.

[0160] In some implementations, users of client devices, including AR / MR HMDs, may encounter an external display that is already showing content from an external display. Detection of the external display can trigger additional steps to determine and adjust the brightness and / or color attributes of other content to be displayed in the AR / MR HMD (e.g., virtual content, augmented and / or reality-rendered objects). This other content can be supplementary content generated to be used with the content on the external display. For example, if the external display is a rugby game, the other content to be displayed on the AR / MR display could be scores, statistics, and commentary about the game. Similarly, if the external display is a movie, the other content could be a virtual rendering of one of the characters appearing alongside the screen in the movie, along with commentary on the film. Regarding supplementary content, additional steps may exist to identify the content currently playing on the external display, such that corresponding other content (e.g., virtual content, augmented and / or reality-rendered objects) can be generated, retrieved from the network, or otherwise obtained and then modified or adjusted according to the steps described above before being displayed on the AR / MR display device.

[0161] In another implementation, the AR / MR HMD user may encounter the external display before the content being played on it is. In this case, the detection of the external display device (optically as described above, or via a network connection and device discovery process, such as via UPnP) allows the client device to instruct the external display device to begin playing content selected by the client device. For example, the client device may access virtual content to be displayed as an accompaniment to a given movie, and the client device may (after detecting the presence of an external display device in the environment) request the external display device to begin playing the movie, allowing the client device to then enhance the movie-watching experience by displaying the accompanying content. In this case, the client device does not need to identify what content is being played on the external device. The client device can be used to identify the location and extent (e.g., “occupying area”) of the external display device in the user’s view, and in the case of multiple external display devices in the environment, it can visually verify that the expected content is being played on the detected external display device. The client device can then use, as described above, to adjust the brightness and / or color attributes of the accompanying content (which may be virtual content, annotations, and / or realistically rendered objects) to be compatible with the content seen by the AR / MR HMD user on the external display.

[0162] In another alternative implementation, the content to be displayed in the AR / MR HMD may not be incidental content and may not yet be specifically prepared for use with externally displayed content. However, in some such implementations, the brightness and / or color attributes of the UI elements and annotations displayed in the AR / MR HMD can be adjusted to be compatible with the determined brightness and / or color attributes of the externally displayed content. This can be done as a style point, or to ensure that the UI elements and annotations displayed in the AR / MR HMD remain visible in the presence of potentially bright externally displayed images.

[0163] Adjusting virtual content using a hybrid approach .

[0164] In some implementations, the client device associated with the AR / MR HMD can be used to adjust the virtual content to match the externally displayed content based on both analysis of captured views of the externally displayed content and the attributes of the identified external display device. This can be accomplished by determining how to adjust the virtual content based on the captured camera views analyzing the externally displayed content, and by determining how to adjust the virtual content based on the attributes of the external display device obtained (e.g., via database lookup or via communication with the external display device).

[0165] In some implementations, when the HMD camera has a good view of the content displayed on an external display device, the client device can adjust the virtual content based on analysis of the captured view of the content on the external display device. When the HMD camera does not have a good view of the content displayed on the external display device, the client device can adjust the virtual content based on the identified attributes of the external display device. The latter situation may occur when the external display is not yet displaying content (e.g., before a movie begins), or when the external display is displaying a portion of the content that is not suitable for visual analysis (e.g., it is displaying an opening title card with only monochrome words on a black background, making useful color analysis impossible). The latter situation may also occur when the user turns their head away from the external display, which may be temporary.

[0166] Other techniques described herein can be used in combination. For example, the model of the external display can take some fixed properties of the identified external display device as input, but it can also take properties determined by analyzing captured HMD forward camera images of the content of the external display to improve the results.

[0167] Determination of brightness adjustment .

[0168] The method described below can be used to determine the brightness adjustment of virtual content for display in an AR / MR HMD.

[0169] In some implementations, brightness adjustment is determined by determining the gain applied to the color code values ​​of the content to be displayed on the HMD. Given the display brightness L of the external display... Real and the display brightness L of augmented reality displays AR The color code value X displayed on an external monitor AR Generate the following output brightness Y real :

[0170] Y real =L real ·(X AR ) γ

[0171] And the color code value X displayed on the HMD AR Generate the following output brightness Y AR :

[0172]

[0173] For color code values ​​X that achieve the same output brightness AR and X AR Set Y real Equal to Y AR This gives

[0174]

[0175] It can be represented as

[0176]

[0177] in

[0178]

[0179] Therefore, in some implementations, in order to adjust the brightness of the content displayed on the HMD to match the content displayed on an external display, the color code value X of the content to be displayed on the HMD (e.g., the pixels of the content) is... AR Multiply by Gain CodeValue In some implementations, gain can be limited. CodeValue The value should be chosen to avoid exceeding the display's parameters. For example, in an implementation with a maximum displayable color code value of 255, the gain can be selected as follows:

[0180]

[0181] CodeValuePeak is the X to be displayed in the virtual content. AR The peak value.

[0182] Alternative implementations may use other techniques to determine the gain value, or other techniques to adjust the brightness of the content displayed on the HMD. For example, the formula given above assumes the same value for γ, but if the value of γ is different, alternative techniques may be applied.

[0183] Exemplary brightening algorithms that can be used in some implementations are described in Xu, Xinyu, and Louis Kerofsky. “Improving content visibility for high-ambient-illumination viewable display and energy-saving display.” Journal of the Society for Information Display 19.9 (2011): 645-654.

[0184] In some implementations, the brightness L of the external display real This is determined by a method that includes capturing an image including a display, determining a region within the image corresponding to content displayed on the display, and determining a maximum brightness value from the determined region. The determined maximum brightness value can be used as a lower limit for the maximum brightness of a real-world display.

[0185] Determination of color adjustment .

[0186] In some of the implementations described herein, color adjustments are applied to virtual content for display in an AR / MR HMD.

[0187] In some implementations, to perform color adjustment, a gray world algorithm is applied to an image captured by an AR / MR HMD on an external display to determine an estimate of the white point of the content displayed on the external display. Some implementations use the techniques described in the following literature: Ebner, Marc, “The Gray World Assumption,” ColorConstancy, Chichester, West Sussex: John Wiley & Sons, 2007. The virtual content can be modified to be displayed using the estimated white point of the content displayed on the external display. In some implementations, the white point can be determined based on a temporal average over multiple frames. In some implementations, the white point of the content displayed on the external display can be determined based on measured colors of known objects detected in the content (such as the color of branded items like soda cans). In some implementations, the external display can transmit white point information to a client device. In some implementations, the client device can send a command to the external display to display specific content (such as a test pattern), which the client device can then use to measure the display's color attributes. In some implementations, the client device can query the external display to retrieve information about user-adjustable settings of the display, such as brightness, contrast, etc.

[0188] White point modification of virtual content can be performed using one or more techniques known in the art. For example, the virtual content can be converted to a linear light domain, and color adaptation can be used to modify the linear pixel values. Some implementations employ techniques described in one or more of the following publications: Ebner, Marc, “Color Constancy,” Chichester, West Sussex, John Willie & Son Publishing, 2007; Johannes von Kries, *Beitrag zurphysiologie dergesichtsempfindung*. *Arch. Anat. Physiol*, 2: 505-524, 1878; King Manlam, *Metamerism and Color Constancy*, Ph.D. Thesis, University of Bradford, 1985; H. Helson, “Object-color changes from daylight to incandescent filament illumination,” *Illum. Engng.*, 47: 35-42, 1957; Graham D. Finlayson and Sabine Süsstrunk, “Performance of a chromatic adaptation transform based on spectralsharpening,” *In Color and Imaging*. Conference, Vol. 2000, pp. 49-55, 2000; Changjun Li, M. Ronnier Luo, Bryan Rigg, and Robert WG. Hunt, “CMC 2000 chromatic adaptation transform: Cmccat2000,” Color Research & Application, 27(1): 49-58, 2002. The virtual content can then be converted from the linear RGB domain back to the non-linear RGB domain for display.

[0189] Overview of virtual content adjustment .

[0190] Various methods can be used to adjust the virtual content to match the content displayed on the external monitor.

[0191] Some implementations may employ brightness matching. Such implementations can be used to estimate the peak brightness of a real-world display, for example, using a histogram of one or more images from a real-world display. The brightness (or dimming) factor of the virtual display can be determined based on the determined peak brightness of the external display. Alternatively, brightness (or dimming) can be applied to the external display to match the displayed virtual content.

[0192] Some implementations may employ white point matching. Such implementations can be used to estimate the white point of content displayed on an external monitor, for example, using a gray-world assumption. White point adjustments can be performed to modify the virtual content so that the content's white point has been modified. Alternatively, white point adjustments can be applied to the external monitor to match the displayed virtual content.

[0193] Some implementation schemes use the aforementioned technology to match both the brightness and white point of virtual content with the enhanced content.

[0194] Additional embodiments .

[0195] In some implementations, a method for visually adjusting virtual content includes: detecting an external display device near an AR device; capturing an image of the external display device using the AR device's camera; determining the extent of a display area associated with the external display device; obtaining a set of sample values ​​from the image that fall within the range of the display area; analyzing the sample values ​​to determine attributes of the externally displayed content (e.g., white point or brightness); and adjusting the virtual content based on the determined attributes for display within the AR device.

[0196] In some implementations, a method for visually adjusting virtual content includes: capturing an image near an AR display device; detecting an external display device near the AR device; determining a set of pixels in the image corresponding to a display area associated with the external display device; obtaining a set of sample values ​​from the image within a range of the display area falling within the image; analyzing the sample values ​​to determine attributes of the external display (e.g., white point or brightness); and adjusting the virtual content based on the determined attributes for display within the AR device.

[0197] In some implementations, a method includes: capturing video of a real-world scene; detecting areas of screen content in the captured video; selectively applying screen-content color conversion to those areas to generate a processed video; and displaying the processed video substantially in real time. Some such implementations are performed on video-perspective augmented reality displays (such as video-perspective head-mounted displays). In some implementations, screen-content color conversion is not performed on areas of the video outside the detected screen content areas.

[0198] In some embodiments, determining screen-content color transitions may include determining attributes of the display on which the screen content is displayed. In some embodiments, determining the attributes of the display includes receiving at least one communication from the display delivery attributes of the display. In some embodiments, determining the attributes of the display includes analyzing video in the detected screen content area.

[0199] In some implementations, determining the screen-content color transition includes determining the lighting conditions of the real-world scene, such as determining the white point.

[0200] In some implementations, a method for improving the rendering of an image of a real-world display viewed on a second display includes: capturing an image of a real-world scene; estimating the lighting of the real-world environment; detecting the presence and location of the real-world display in the real-world image; estimating the capability of the real-world display; determining enhancement attributes corresponding to the real-world display based on the estimated capability and real-world lighting; calculating an enhanced image by enhancing the real-world scene in the area containing the location of the real-world display based on the estimated capability and estimated lighting; and rendering the enhanced real-world image to the second display.

[0201] In some such implementations, the capabilities of the real-world display include information about the display's brightness or color gamut.

[0202] In some implementations, the capability of the real-world display is received from the real-world display via a communication channel. In some implementations, the capability of the real-world display is estimated by identifying the model of the real-world display and querying the Internet (e.g., a database accessible via the Internet).

[0203] In some implementations, estimating the lighting of the real-world environment includes measuring the ambient light level. This can be done using an image of the real-world environment.

[0204] In some implementations, the content displayed on the real-world display is coordinated with the content displayed on the second display.

[0205] In some implementations, a method includes: determining the angular size of an object to be displayed to a user; determining a color adjustment for the object based on the determined angular size; applying the color adjustment to the object; and rendering the color-adjusted object. The color-adjusted object may be rendered on, for example, a head-mounted display, a telephone, or a tablet computer.

[0206] In some such implementations, determining the angular size of the object includes determining the on-screen size of the object and determining the distance of the user from the screen.

[0207] In some embodiments, a method is provided for improving the rendering of an image of a real-world display viewed on a second display. An exemplary method includes: capturing an image of a real-world scene; estimating the lighting of the real-world environment; detecting the presence and location of a real-world display in the real-world image; estimating the capabilities of the real-world display; determining an adjustment attribute corresponding to the real-world display based on the estimated capabilities and real-world lighting; calculating an adjusted image by adjusting the real-world scene in the area containing the location of the real-world display based on the estimated capabilities and estimated lighting; and rendering the adjusted real-world image to the second display. In some such embodiments, the capabilities of the real-world display are estimated by identifying the model of the real-world display and querying a database of the capabilities of the real-world display based on the identified model.

[0208] In some implementations, a system includes a processor and a nontransitory computer-readable medium storing instructions for performing any of the functions described herein.

[0209] It should be noted that the various hardware elements of one or more embodiments described in the implementation schemes are referred to as “modules” that perform (i.e., execute, implement, etc.) the various functions described herein in conjunction with the respective modules. As used herein, a module includes hardware (e.g., one or more processors, one or more microprocessors, one or more microcontrollers, one or more microchips, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more memory devices) that a person skilled in the art would consider suitable for a given specific implementation. Each of the said modules may also include executable instructions for performing one or more functions described as being performed by the respective module, and it should be noted that these instructions may take the form of or include the following instructions: hardware (i.e., hardwired) instructions, firmware instructions, software instructions, etc., and may be stored in any suitable one or more non-transitory computer-readable media (such as commonly referred to as RAM, ROM, etc.).

[0210] Although features and elements have been described above in specific combinations, those skilled in the art will understand that each feature or element may be used alone or in any combination with other features and elements. Furthermore, the methods described herein may be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media (such as internal hard disks and removable disks), magneto-optical media, and optical media (such as CD-ROM disks and digital versatile optical discs (DVDs)). A processor associated with the software may be used to implement a radio frequency transceiver for a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. An augmented reality display method, the augmented reality display method comprising: Capture at least one image of a real-world scene; Identify the captured screen content region in the at least one image; Measure at least one visual parameter of the screen content from the captured screen content area in the image; Based on the at least one visual parameter, determine the transformation of the virtual content; Apply the transformation to the virtual content; as well as This enables the transformed virtual content to be displayed on an augmented reality display.

2. The method of claim 1, wherein the at least one visual parameter includes the brightness of the screen content, and wherein the conversion includes a conversion of the brightness of the virtual content.

3. The method of claim 2, wherein the measured brightness of the screen content is the maximum brightness in the captured screen content area.

4. The method of claim 2 or 3, wherein the conversion includes a gain factor applied to the color code value of the virtual content, and wherein the gain factor increases as the measured brightness of the captured screen content area increases.

5. The method according to any one of claims 1 to 3, wherein the at least one visual parameter includes the white point of the screen content, and wherein the transformation includes the transformation of the white point of the virtual content.

6. The method of claim 5, wherein measuring the white point of the screen content comprises applying a gray world algorithm to the captured screen content area.

7. The method according to any one of claims 1 to 3, wherein the image of the real-world scene is captured using the forward-facing camera of the augmented reality display.

8. The method according to any one of claims 1 to 3, wherein the augmented reality display is an optical see-through head-mounted display.

9. The method according to any one of claims 1 to 3, further comprising: Using the captured screen content area, identify the content displayed in the captured screen content area; as well as The virtual content is selected based on the identified content.

10. The method according to any one of claims 1 to 3, further comprising: Identify the type of display used to show the screen content; The virtual content is further transformed based on the type of the display and at least one additional visual parameter.

11. An augmented reality device, the augmented reality device comprising a processor, the processor being configured to perform at least: Capture at least one image of a real-world scene; Identify the captured screen content region in the at least one image; Measure at least one visual parameter of the screen content from the captured screen content area in the image; Based on the at least one visual parameter, determine the transformation of the virtual content; Apply the transformation to the virtual content; as well as This enables the transformed virtual content to be displayed on an augmented reality display.

12. The apparatus of claim 11, wherein the at least one visual parameter includes the brightness of the screen content, and wherein the conversion includes a conversion of the brightness of the virtual content.

13. The apparatus of claim 12, wherein the measured brightness of the screen content is the maximum brightness in the captured screen content area.

14. The apparatus of claim 12 or 13, wherein the conversion includes a gain factor applied to the color code value of the virtual content, and wherein the gain factor increases as the measured brightness of the captured screen content area increases.

15. The apparatus according to any one of claims 11 to 13, wherein the at least one visual parameter includes the white point of the screen content, and wherein the transition includes the transition of the white point of the virtual content.

Images