A method and an apparatus

Abstract

Systems and methods are described for capturing images of portions of first and second display devices within an environment using a forward-facing camera associated with a head-mounted augmented reality (AR) head-mounted display (HMD), the first and second display devices displaying first and second portions of content related to an AR presentation; and displaying a third portion of content related to the AR presentation on the AR HMD, the third portion being determined based on the images of portions of the first and second display devices captured using the forward-facing camera. Furthermore, the first and second display devices may be active stereoscopic displays, and the AR HMD may simultaneously function as shutter glasses.

Description

[0001] This application is a divisional application of the Chinese invention patent application with application number 201680058637.3, application date September 30, 2016, and invention name “Method and system for automatic calibration of dynamic display configuration”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to U.S. Provisional Application No. 62 / 239,143, filed on October 8, 2015, entitled “METHODS AND SYSTEMS OF AUTOMATIC CALIBRATION FOR DYNAMIC DISPLAY CONFIGURATIONS,” U.S. Provisional Application No. 62 / 260,069, filed on November 25, 2015, entitled “METHODS AND SYSTEMS FOR OPTIMIZING DISPLAY DEVICE OUTPUT USING A SECONDARY DISPLAY DEVICE,” and U.S. Provisional Application No. 62 / 261,029, filed on November 30, 2015, entitled “METHODS AND SYSTEMS FOR OPTIMIZING DISPLAY DEVICE OUTPUT USING A SECONDARY DISPLAY DEVICE.” Technical Field

[0004] The present application is generally related to immersive 3D content using a combination of light-transmissive augmented reality (AR) head-mounted displays (HMDs) and traditional displays. Background Art

[0005] There is a growing interest and activity in the field of Virtual Reality (VR) and AR. Major industry players are working on consumer-facing AR / VR HMD devices, while various stakeholders in the Hollywood entertainment industry appear to be actively developing activities around AR and VR content. If consumer AR / VR takes off, there will be a huge demand for solutions that can consume content on a larger number of devices beyond just the AR or VR HMDs that are currently driving development.

[0006] One of the illusions that VR aims to create is that the user can feel that they exist in a synthetic virtual world rather than in physical reality. This illusion can be achieved by replacing the user's visual and auditory perception of the real world with synthetic images and sounds generated by computer simulation. In order to achieve a convincing illusion, the images and sounds generated by the computer simulation should have a consistent spatial relationship with the user and respond to visual changes caused by the user's actions, so that the user can explore the virtual world as if they were exploring the real physical world. Similar to VR, AR aims to generate illusions, creating an alternative version of the physical reality around the user, where virtual elements can be attached to the physical environment.

[0007] As AR and VR HMDs prepare for immersive experiences in the living room, it can be observed that the size of consumer displays is growing. Users want larger displays in their living rooms. However, despite the increasing size, current flat or slightly curved displays do not fully support immersive experiences, such as those created for AR / VR HMDs. However, as display technology evolves, in the near future, users may have even more display areas in their living rooms, displays may become non-planar and occupy several walls, and / or displays may be deformable, so that the shape and configuration of the display can be changed to match the content being consumed and the usage context.

[0008] AR and VR HMDs are also not suitable for displaying fully immersive content on them due to the semi-transparent nature of the light-transmissive display and the limited field of view. The transparency of the display causes the virtual content to appear as a ghost image in the user's visual world, and the limited field of view further disrupts the illusion when the field of view switches sharply towards the edge of vision.

[0009] Currently, several manufacturers (including major industry giants such as Microsoft, Sony, Epson, etc.) are promoting light-transmissive AR HMDs to the consumer market. These first-generation devices aimed at the consumer market have some drawbacks, especially in terms of limited field of view, brightness, and transparency of the pictures they produce. However, it is realistic to foresee that the next generation of devices will remedy at least some of these drawbacks and provide high-resolution AR visualization with good image quality in a comfortable form factor.

[0010] Although future improvements in AR HMDs are a high probability event, it will be very difficult to reproduce the resolution and color / brightness of traditional displays using light-transmissive HMDs. In addition, current technology for AR HMDs only provides a fixed human eye accommodation distance, and by doing so, another drawback to be considered occurs.

[0011] AR HMDs also have some features that allow them to surpass traditional displays in some ways. AR HMDs can enable users to move freely within an environment without compromising overall image quality. Further, the latest smart glasses HMDs can track user movement and create estimates about the geometry of the environment, which in turn can enable environmental augmentation and rich interaction with the environment. In addition, with the HMD display close to the user's eyes, the user can have a very clear line of sight to the display.

[0012] In the area of ​​shape-changing displays, the most relevant examples are mass-produced display devices that can be adjusted between curved and flat shapes. In addition, there are many speculations about spliced ​​display technology. In addition to displays that include multiple flat displays or slightly curved displays, display manufacturers have long been working on truly flexible displays, and mass production solutions are becoming a reality. In the academic research community, there are already some examples of relevant prototype installations. For example, the following documents describe robotic displays and small spliced ​​displays in an exemplary manner: A Shape-Shifting Wall Display that Supports Individual and Group Activities (TAKASHIMA, Kazuki, et al. A Shape-Shifting Wall Display that Supports Individual and Group Activities. 2015) and a Tilt Display: Designing Display Surfaces with Multi-axis Tilt and Actuation (ALEXANDER, Jason; LUCERO, Andrés; SUBRAMANIAN, Sriram. Tilt displays: designing display surfaces with multi-axis tilting and actuation (Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services. ACM, 2012. p. 161-170)). Some related devices and methods can be found in the following patent documents: U.S. Patent No. 8988343; U.S. Patent No. 8711091; and U.S. Patent Application Publication No. US20100321275A1. However, there appear to be few examples of deformable or tiled display systems that would allow for dynamic adjustments during an experience. Summary of the invention

[0013] The methods and systems herein describe a process that receives sensor-based data about display information and user information (e.g., user position, user eye position) as input, and modifies a graphics output stream generated by a currently active application based on the received input, thereby adjusting the graphics output to match the current display configuration and usage context. An overall view of the components of the process is shown in FIG. Figure 2 shown.

[0014] The described embodiments may automatically adjust an application's rendering output to match a display configuration that may be dynamically changed, such as the shape of a display, the orientation of a display, the relative position of two or more displays, etc., which may change during application runtime execution.

[0015] Some embodiments are based on analyzing sensor data from sensors observing the user and display configuration. The rendering changes needed to match the display configuration can be injected into the intercepted graphics API calls made by the application. The goal is to achieve highly immersive rendering of unmodified applications, while taking into account the viewing content and context to improve the results.

[0016] In some embodiments, a display setup is equipped with a sensor (one or more) configured to detect a current display layout configuration. In addition to the display configuration sensor, a sensor that enables user detection in front of the display can also be used to observe the area around the display configuration. An embodiment describes the following process, in which graphics API calls generated by an unmodified application can be intercepted, a view transformation is calculated based on display configuration information and head tracking, and then the view transformation is injected into the graphics API call sent to the display via a display driver. This method can achieve immersive rendering of unmodified applications and dynamic changes in display configuration (even during application runtime). In the method described below, changes in the display configuration can be monitored at all times and output parameters can be adjusted to account for changes in the configuration detected during application runtime without user involvement.

[0017] As a result of the process, the user will be provided with an immersive rendering of the currently active application adapted to the current display configuration. For example, a user playing a 3D first-person shooter game on a setup with multiple displays can adjust the arrangement of the screens around them during game play. As the orientation of the displays changes, the user will experience the correct immersive rendering of the game play, just as if they were viewing the game world through the combination of multiple displays. Similarly, for a flexible display, a user can change the shape of the display from a flat shape on a wall to a more immersive shape during application runtime simply by moving the flexible display to a corner, so that it can extend from one wall to another while gradually bending 90 degrees between walls.

[0018] In methods according to some examples, the quality and immersive experience achieved is better than what can be achieved with a separate light-transmissive AR HMD or traditional display. In some embodiments, an auxiliary display device may be used to fill in gaps that exist when rendering on a primary device or a set of primary devices. For example, where multiple large displays are available in the space, an AR HMD may be used to bridge any optical gaps between these displays. In other cases where the AR HMD is the primary display device for content, a traditional display may be used to fill in areas that fall outside the limited field of view of the AR HMD. For example, the primary device(s) may be one or more devices that are determined or selected to be most suitable for displaying the content or a portion of the content.

[0019] Some embodiments described herein may utilize a method based on a combined approach that can selectively divide content onto different displays while also enabling immersive rendering based on head tracking. In addition, the method can monitor the output all the time and correct for unwanted occlusions and shadows caused by physical objects between the display and the viewer. Where there are multiple display devices available, one or more of them can be used in combination to fill in the gaps that exist when rendering on an AR HMD device. For example, where there are multiple large displays available in the space, an AR HMD can be used to bridge the optical gaps between these display devices. In other cases where the AR HMD is the primary display device for the content, a conventional display can fill in areas that fall outside the limited field of view of the AR HMD. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] A more detailed understanding may be obtained through the following description given in an exemplary manner in conjunction with the accompanying drawings.

[0021] Figure 1A An exemplary communication system is depicted within which one or more disclosed embodiments may be implemented.

[0022] Figure 1B It shows the Figure 1A An exemplary client device of a communication system.

[0023] Figure 2 An overall view of components that may be used within a system is depicted in accordance with at least one embodiment.

[0024] Figure 3 A flow chart of a process according to at least one embodiment is depicted.

[0025] Figure 4 A flow chart of a process according to at least one embodiment is depicted.

[0026] Figure 5A and 5B An exemplary configuration of two flat-panel displays according to at least one embodiment is depicted.

[0027] Fig. 6A , 6B 6C illustrate various calculations for projection on two flat panel displays according to at least one embodiment.

[0028] Fig. 7A and 7B An exemplary configuration of a flexible display according to at least one embodiment is illustrated.

[0029] Figure 8 An exemplary calculation of flexible display projection according to at least one embodiment is depicted.

[0030] Fig. 9 A user is depicted in an immersive environment including two external display devices in accordance with some embodiments.

[0031] Fig.10 Depicted is a shading correction example according to some embodiments.

[0032] Fig.11 An example of occlusion correction is depicted in accordance with some embodiments.

[0033] Fig.12 Depicted is an example of assigning virtual elements according to eye accommodation distance in accordance with some embodiments.

[0034] Fig.13 Depicted is an example of a user within an immersive environment in accordance with some embodiments.

[0035] Fig.14 Depicted is an example of detecting artifacts in accordance with some embodiments.

[0036] Fig.15 Depicted is an example of inpainting a detected artifact in accordance with some embodiments.

[0037] Fig.16 Depicted is an example of an AR HMD rendering content outside of a display in accordance with some embodiments.

[0038] Fig.17 Depicted is an example of rendering a virtual element selected to be displayed on an AR HMD in accordance with some embodiments.

[0039] Fig.18 An embodiment utilizing an AR HMD as shutter glasses is depicted in accordance with some embodiments.

[0040] Fig.19 is a flow diagram of a process according to some embodiments. DETAILED DESCRIPTION

[0041] A detailed description of illustrative embodiments will now be provided with reference to the various drawings. Although the description provides detailed examples of possible implementations, it should be noted that the details provided are for exemplary purposes only and are not intended to limit the scope of the present application. Systems and methods related to immersive augmented reality may be used with reference to Figure 1A and 1B The wired and wireless communication systems. First, the wired and wireless systems will be described.

[0042] Figure 1A is a diagram of an exemplary communication system 100 in which one or more disclosed embodiments may be implemented. The communication system 100 may be a multiple access system that provides content such as voice, data, video, messaging, broadcast, etc. to multiple wireless users. The communication system 100 allows multiple wired and wireless users to access such content by sharing system resources including wired and wireless bandwidth. As an example, the communication system 100 may use one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single carrier FDMA (SC-FDMA), etc. The communication system 100 may also use one or more wired communication standards (e.g., Ethernet, DSL, radio frequency (RF) over coaxial cable, fiber optic technology, etc.).

[0043] like Figure 1AAs shown, the communication system 100 may include client devices 102a, 102b, 102c, 102d and / or 102e, a personal area network (PAN) 106, and communication links 114 / 115 / 116 / 117 / 118, but it should be understood that the disclosed embodiments contemplate any number of client devices, base stations, networks and / or network components. Each of the client devices 102a, 102b, 102c, 102d, 102e can be any type of device configured to operate and / or communicate in a wired or wireless environment. For example, client device 102a is depicted as a tablet / touch screen smartphone, client device 102b is depicted as a speaker, client device 102c is depicted as a light emitting device, client device 102d is depicted as a television, and client device 102e is depicted as an HMD (in some embodiments, AR or VR).

[0044] Some or all of the client devices 102a, 102b, 102c, 102d, 102e within the communication system 100 include multi-mode capabilities, i.e., the client devices 102a, 102b, 102c, 102d and 102e include multiple transceivers for communicating with different wired or wireless networks over different communication links.

[0045] In some embodiments, client devices 102a, 102b, 102c, 102d, and 102e may communicate with each other using a local communication protocol via PAN 106. For example, the client devices may communicate using Bluetooth, Wi-Fi, wireless LAN (WLAN), or other forms of wireless local communication protocols.

[0046] Figure 1B It shows the Figure 1A An exemplary client device within a communication system of . Specifically, Figure 1B FIG. 1 is a system diagram of an exemplary client device 102. Figure 1B As shown, client device 102 may include a processor 118, a transceiver 120, a transmit / receive component 122, a speaker / microphone 124, a numeric keypad 126, a display / touchpad 128, a non-removable memory 130, a removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and other peripherals 138. It should be appreciated that client device 102 may represent any of client devices 102a, 102b, 102c, 102d, and 102e, and may include any sub-combination of the above components while remaining consistent with an embodiment.

[0047] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, and the like. The processor 118 may perform signal encoding, data processing, power control, input / output processing, and / or any other functionality that enables the client device 102 to operate in a wired or wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. Although Figure 1B The processor 118 and the transceiver 120 are depicted as separate components, but it should be appreciated that the processor 118 and the transceiver 120 may be integrated into one electronic component or chip.

[0048] The transmit / receive component 122 may be configured to transmit or receive signals to or from the PAN 106 via the communication links 114 / 115 / 116 / 117 / 118. For example, in one embodiment, the transmit / receive component 122 may be an antenna configured to transmit and / or receive RF signals. In another embodiment, as an example, the transmit / receive component 122 may be an emitter / detector configured to transmit and / or receive IR, UV or visible light signals. In yet another embodiment, the transmit / receive component 122 may be configured to transmit and receive RF and light signals. In yet another embodiment, the transmit / receive component may be a wired communication port, such as an Ethernet port. It should be appreciated that the transmit / receive component 122 may be configured to transmit and / or receive any combination of wired or wireless signals.

[0049] In addition, although Figure 1B 102 as a single component, the client device 102 may include any number of transmit / receive components 122. More specifically, the client device 102 may use MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive components 122 (e.g., multiple antennas) for transmitting and receiving wireless signals via the communication links 114 / 115 / 116 / 117 / 118.

[0050] The transceiver 120 may be configured to modulate signals to be transmitted by the transmit / receive component 122 and to demodulate signals received by the transmit / receive component 122. As described above, the client device 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers that allow the client device 102 to communicate via multiple RATs such as UTRA and IEEE 802.11.

[0051] The processor 118 of the client device 102 can be coupled to a speaker / microphone 124, a numeric keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit), and can receive user input data from these components. The processor 118 can also output user data to the speaker / microphone 124, the numeric keypad 126, and / or the display / touchpad 128. In addition, the processor 118 can access information from any suitable memory (e.g., a non-removable memory 130 and / or a removable memory 132), and store information in these memories. The non-removable memory 130 can include a random access memory (RAM), a read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 can include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, etc. In other embodiments, the processor 118 can access information from and store data in memories that are not actually located in the client device 102, where, for example, the memory can be located on a server or a home computer (not shown).

[0052] The processor 118 may receive power from the power source 134, and may be configured to distribute and / or control power for other components in the client device 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cell battery packs (e.g., nickel-cadmium (Ni-Cd), nickel-zinc (Ni-Zn), nickel-metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, a wall outlet, etc.

[0053] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) related to the current location of the client device 102. In addition to or in lieu of the information from the GPS chipset 136, the WTRU 102 may determine its location based on location information received via the communication links 114 / 115 / 116 / 117 / 118 and / or based on the timing of signals received from two or more nearby base stations. It should be appreciated that the client device 102 may acquire location information using any suitable positioning method while remaining consistent with an embodiment. According to one embodiment, the client device 102 may not include a GPS chipset and may not require location information.

[0054] The processor 118 may also be coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functionality, and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos or videos), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, modules, FM radio units, digital music players, media players, video game console modules, Internet browsers, and more.

[0055] The displays that people currently have in their homes, despite their increasing size, still cannot provide a truly immersive experience. Currently, the most serious limitations of VR HMDs are limited resolution and issues related to head tracking and associated motion sickness. However, these issues will improve with the next generation of VR HMDs. Another issue with VR HMDs is the complete visual isolation they impose on the user. This isolation severely limits the experience both spatially and socially.

[0056] VR setups consisting of several display walls, called VR-Caves, have been built and used in many research institutions. This cave-like installation enables natural exploration of VR environments, but at the expense of high space and device requirements and costs. However, these cave systems allow intuitive exploration of virtual content, strong immersion, and even limited support for multiple concurrent users.

[0057] It would be very beneficial if the user could have the ability to change shape and behavior depending on content and context to support an immersive virtual experience in a 3D mode where the display would surround the user similar to a VR cave, whereas for traditional content (e.g., movies), the display could provide a maximum size flat or slightly curved screen.

[0058] To improve the usability of the deformable display, the system that generates the output to the display can automatically detect changes in the display configuration and adjust the output characteristics accordingly. Current graphics output configurations supported by operating systems generally support static display configurations, where the configuration process can be performed as a separate step. The display surface can be considered as just a 2D plane, and its position and orientation are manually defined by the user.

[0059] Such dynamic display behavior requires changes in how displays and outputs are managed at the operating system and software application layers.The present disclosure provides a scheme for managing dynamically changing display configurations in real time during application execution.

[0060] Automatic calibration for dynamic display configuration

[0061] The embodiments described herein can enable a display configuration including multiple display blocks to fully function, or the configuration and shape of the display can be changed automatically or manually during operation. This enables more diverse content display and can provide an improved user experience in terms of content viewing and usage context.

[0062] Figure 2 An overall view of components for a system is depicted, the system being configured to receive sensor-based data regarding display configuration and user position as input, and to modify a graphics output stream generated by a currently active application based on the received input, thereby adjusting the graphics output to match the current display configuration and usage context. Figure 2 As shown, Figure 2 Depicted are a user tracking sensor 202 , displays 204 a and 204 b , a display configuration sensor 206 , a graphics processing unit (GPU) 208 , a graphics driver 210 , a dynamic display configuration manager module 212 , and an application module 214 .

[0063] Displays 204a and 204b may be television screens, computer monitors, smartphone / tablet screens, LCD displays, LED displays, or any other common display modules known to those skilled in the art. In some embodiments, at least one display may be a flat panel. In some embodiments, at least one display may be a curved display. In some embodiments, at least one display may be a flexible display.

[0064] The user sensor tracking sensor 202 may be a camera, a camera with a depth sensor (RGB-D), or any type of sensor that can observe the area around the display and provide data that can be analyzed to detect and track users and analyze the approximate position of the users' eyes. When multiple users are detected viewing the display, the user detection and tracking module may be instructed to use some heuristics to determine the primary user, or find the best average eye position (which produces an acceptable viewing position for all users), rather than using some static predefined default eye positions for the user eye positions.

[0065] Display configuration sensors can be simple sensors embedded in the display structure that can measure the angle of connection between display components, or they can be optical, acoustic, or magnetic sensors that can detect the relative orientation and position of the display by observing the setup from outside the mechanical structure.

[0066] In some embodiments, the process of managing dynamic display configuration can be implemented as a software module. The software module can be installed as a driver as part of the operating system, or it can be integrated as an additional feature of the graphics driver, or it can be integrated with the actual application that generates the graphical output. Figure 2 In the overall view of the related components of FIG. 2 , the software module dynamic display configuration manager 212 that implements the processes described below is shown as a separate driver module within the operating system layer.

[0067] The data from these sensors is fed into the dynamic display configuration manager module 212 which may implement a process having the following steps:

[0068] 1. Receive sensor data from a sensor that monitors the display configuration.

[0069] 2. Identify the current display configuration.

[0070] 3. Receive sensor data about users in the area.

[0071] 4. Detect the number of users and the eye position of the main user;

[0072] 5. Using the current display configuration, solve the 3D projection and transformation that will be used to correctly render content to the user.

[0073] 6. Inject projections and transformations into the captured graphics call stream generated by the currently running application.

[0074] A more detailed explanation of the process will be given below.

[0075] In some embodiments, the process is tasked with taking data from sensors 202 and 206 that observe the user and display configuration, respectively, and modifying the rendering application 214 to match the display configuration (and, if necessary, the user's eye position). Figure 3 The process is schematically shown. Based on the sensor data, the display configuration manager 212 can identify the shape and layout of the current display configuration, such as display shape, position and orientation. The user sensor 202 can provide the display configuration manager with information about the number of users viewing the display and the position of the users' eyes.

[0076] Figure 4An embodiment of the steps of a continuous process performed by the dynamic display configuration manager 212 is shown in detail. The first step of the process is to capture and analyze sensor data. The process can track information about the shape, position and orientation of the display, as well as the user viewing the display and the position of the user's eyes. In the case where the relative position between the various display components is known, the use of angle detection sensors (e.g., electronic potentiometers) is sufficient. In the case where the various display screens can be moved with more degrees of freedom, other tracking methods can be used, such as sound or magnetic tracking.

[0077] When a camera or depth sensor (RGB-D) is used to observe the display configuration, computer vision-based methods can be used to detect changes in the shape, position, or orientation of each individual display component. Since the visual data sent to the display is known, any image detection and tracking method (for example, the image detection and tracking method used in the markerless tracking scheme given for AR) can be used to detect the position and orientation of the display from the RGB data alone. It can also be detected by using, for example, SALZMANN, Mathieu; HARTLEY, Richard; FUA, Pascal. Convex optimization for deformable surface 3-d tracking (in Computer Vision, 2007. ICCV2007. IEEE 11th International Conference on. IEEE, 2007., pp. 1-8 (Computer Vision, 2007. ICCV2007. IEEE 11th International Conference on. IEEE, 2007. p. 1-8)) (hereinafter referred to as "Salzmann et al."). The proposed convex optimization method or similar methods detect the shape change of the display from the RGB data. Where RGB-D data is present, a depth component included with that data may be used in conjunction with the RGB data to further refine the shape, position, and orientation detection.

[0078] In addition to receiving sensor data about the display configuration and user position, the process also intercepts all graphics API calls generated by the currently active application. Graphics API call interception can be accomplished by routing the graphics call processing through the module, for example by replacing the default graphics driver shared object with the module and plugging the output of the module into the input of the default graphics driver shared object. Graphics API call interception can operate according to principles similar to those used by, for example, NVidia 3D Vision and NVidia 3DTV Play modules. These NVidia modules can capture all graphics API calls generated by unmodified applications, cache the graphics API calls, and inject modified projection and model view matrices into the calls for stereo rendering. Stereo rendering involves the scene being rendered twice, once for each eye, with slightly different offset viewpoints that match the user's approximate interpupillary distance.

[0079] In addition to being able to intercept and modify graphics API calls made by applications, the dynamic display configuration manager 212 can also inspect graphics API calls to identify the type of content being displayed. The content type can be used to decide how the rendering should be modified to best suit the display configuration and usage context. For example, 3D content can be converted to an immersive rendering that takes into account the position of the user's glasses, while for video content, it may be sufficient to simply crop correctly for each display and convert the cropped area of ​​the video to produce a seamless combined display area.

[0080] The viewing parameter calculator can calculate the correct viewport setting, projection, and model view transformation used to render the 3D content to each display for each different eye position. An example of calculating the projection and model view transformation is given below in the simple case of two non-planar displays and a single flexible display. The same method can be extended to calculate parameters for any number of displays with arbitrary orientations and shapes.

[0081] As a final step in the process, the view renderer may render the entire content separately for each display, and in the case of stereo rendering, may render twice for each individual display. While rendering, the view renderer may send view-dependent viewport settings and graphics API calls, and the correct projection and model view are injected into the graphics driver. The graphics driver 210 may then forward these commands to the graphics processing unit (GPU) 208, which may output graphics to displays 204a and 204b. In the case of displays with non-planar surfaces, the rendering step may require rendering the scene twice, the first time the entire scene may be rendered to a renderer buffer using a standard perspective projection covering the entire area covered by the display. The rendered image data in the renderer buffer may then be warped to correct for display geometry distortion, and the warped image may be sent to the display.

[0082] Two flat-screen monitors

[0083] In one embodiment, a user may view a display configuration consisting of two displays. The angle between the two displays may be dynamically adjusted, and a sensor may be added to the connection between the displays, which may send angle readings to the dynamic display configuration manager 212. Figure 5A The starting situation is shown where a user is viewing 2D graphics from a display configured in planar mode.

[0084] Fig. 6A The projections calculated by the dynamic display configuration manager for two display monoscopic rendering when the screens are arranged in parallel are shown. A user may launch an application that generates a 3D rendering of a virtual world. The dynamic display configuration manager identifies changes in the 3D rendering mode and switches to an immersive output mode by inspecting the graphics API calls made by the application.

[0085] In immersive rendering mode, the dynamic display configuration manager can calculate projections for both eyes of the user, such as Figure 6B The eye position can be estimated based on the user sensor data, and the required viewpoint transformation and projection can be injected into the captured graphics API calls generated by the application.

[0086] Figure 5B Shown is a user viewing a display configuration after changing the angle between displays. This new configuration resembles a VR cave display, where the user is surrounded by display walls. In this case, the viewpoint transformation used to correctly render the two display views is different. Figure 6C The projection used for a non-planar configuration is shown.

[0087] Flexible Displays

[0088] In some embodiments, the output may be calibrated for a configuration consisting of a display having a non-planar display surface, or a display surface that can change shape (e.g., from a planar surface to a curved or non-uniformly shaped display surface, which may be achieved with a flexible display). In an exemplary embodiment, a user is viewing a flexible display that is first placed on a wall as a planar surface, such as Fig. 7A In this case, the projection used for 3D rendering is the same as before. Fig. 6A Rendering in this case requires no additional steps and can be performed by calculating projection and transformation matrix modifications based on the display orientation and position and the user's head or eye position received from head tracking.

[0089] While the application is running, the user can change the shape of the display by bending the display to a certain angle. The shape of the display after deformation caused by the user action is as follows: Figure 7B As shown. It may not be possible to perform correct immersive rendering on a deformed display shape caused by display curvature simply by modifying the model view and projection matrices. Instead, the rendering requires the use of additional geometric corrections to compensate for the image distortion caused by the non-planar display plane.

[0090] Geometric correction can be achieved by first detecting the new shape of the display. In cases where an RGB-D sensor can be used for configuration detection, optical matching can be used as an aid to detect the display shape through depth reconstruction, or in cases where only RGB data is available, the display shape can be detected using the techniques described herein (e.g., the techniques given by Salzmann et al.). Optical detection of the display configuration is only an available option. In some embodiments, the display configuration sensor can be a simple sensor embedded in the display structure, which can measure the connection angle between the display components, or it can be an optical, acoustic, magnetic, etc. sensor that can detect the relative shape, orientation and position of the display by observing the configuration from outside the mechanical structure.

[0091] When the deformed display shape, position and orientation are determined, the rendered area may be calculated to cover the entire area of ​​the display. Figure 8It shows how the projection covering the entire display area looks like from top to bottom after visualization. For non-planar display surfaces, immersive 3D rendering can be performed in two steps. In the first rendering path, the scene can be rendered normally using the calculated projection. The first rendering path is aimed at rendering the image into the rendering buffer. In the second rendering path, the deformation geometry can be used to define the distortion of pixels from the image plane coordinates generated by the first rendering path to the deformed display screen coordinates. This step can be thought of as texturing the deformed geometry by projecting the rendering buffer pixels onto the deformed geometry from the user's viewpoint, and then stretching the textured deformed geometry back to a flat image, which is then sent to the display. The result of displaying the distorted image on the display is that the user can see the correct non-distorted view of the virtual environment from his viewpoint.

[0092] Additional Embodiments

[0093] Several possible variations of the above embodiment are possible for slightly different use cases, which will be briefly described below.

[0094] The first possible variation is a setup where no user tracking is included. This somewhat more limited approach can provide correct content rendering for a single predefined viewpoint for dynamic display configurations. This approach may be sufficient in cases where the application handles the head tracking required to produce a truly immersive rendering, or where truly immersive rendering is not required (such as where the display is only used for video content, such as outputting scalable 360-degree video to a display configuration with a display covering a large field of view but which does not support changing viewpoints based on head tracking).

[0095] In addition to simply adjusting the rendering to suit the display configuration, another embodiment may integrate features to actively change the display configuration. In this case, the exemplary process includes an additional module that can identify the primary display configuration based on the content type, number of users, and user positions, and then command the display to change configuration to the detected ideal position. In some embodiments, this utilizes a motorized display device to be used and an interface for the process that can communicate with the display setup, which can be used to command the display configuration to change shape as desired.

[0096] In addition to embodiments where the display configuration only changes shape, the processes described herein can also be used for situations where displays dynamically appear and disappear within an environment. For example, when using dynamic output mapping as described in International Patent Application PCT / US16 / 47370, filed on August 17, 2016, entitled "Systems and Methods for Enhancing Augmented Reality Experience with Dynamic Output Mapping" (hereinafter referred to as [Harviainen]), displays can be found from the environment during runtime and then provided to the user for output. In this case, the display configuration sensor can be configured to cope with displays that become available / unavailable during runtime, and can detect their shape, position, and orientation. In some embodiments, this can be handled by display visual tracking as described in [Harviainen]. If the position of the display can be detected for each rendering step, some further embodiments can be extended to displays with dynamic positions, such as head-mounted AR displays. This can use, for example, inertial measurement unit (IMU) or vision-based tracking, which can detect where the user is looking, e.g., where the AR screen is located, and this tracking information can be combined with the process so that the correct part of the content can be displayed on the ARHMD in combination with other displays.

[0097] As described above, module 212 can be implemented as a separate module at the operating system level or integrated as a feature on the graphics driver. In both embodiments, the module can be run as a component that can change the output generated by the application without the application knowing the process, or can provide the information required by the application by itself to generate the correct rendering. The process has been described above in a mode where the application 214 is unaware of dynamic display rendering. However, in some embodiments, module 212 may collaborate with the application so that the application 214 can know the dynamic display configuration manager and can control the rendering. When immersive rendering is required, the application 214 can request the current display configuration and associated rendering settings, and then control the rendering by selecting how to apply the information, and directly flow the graphics API to the default graphics driver. In this way, the application 214 can have more control over the rendering, so that the best rendering method of the UI element can be selected, for example, depending on which type of display configuration is currently being used.

[0098] Output immersive content on a combination of displays

[0099] By using an AR HMD in combination with conventional display technologies (e.g., display screens and projectors), new ways of outputting immersive real-time 3D graphics are possible. When using the newly invented method, some embodiments may integrate any of the following features added to general 3D graphics output. In some embodiments, head tracking may make it possible to render immersively on a conventional display using motion parallax. In some embodiments, stereoscopic viewing may be integrated. When coupled to an AR HMD that blocks viewing, a display with a sufficiently high frame rate may use an active stereo mode, which may be used as active shutter glasses in addition to being used as a display device. In some embodiments, a conventional display may be extended by rendering an area outside a conventional display onto an AR HMD display. In some embodiments, a shadow cast by a viewer or by an object (e.g., when a front projection display is used), and corresponding content (e.g., content blocked or occluded by the shadow) may be displayed on the display of the AR HMD. In some embodiments, object occlusion caused by an object between the viewer and the display may be detected, and corresponding content (e.g., content blocked or distorted by an intermediate object) may be displayed on the display of the AR HMD. In some embodiments, by compartmentalizing content (i.e., virtual elements may be displayed on a display that is closer to the correct eye adaptation distance), the AR HMD may provide a better match to natural eye adaptation.

[0100] Merging the benefits of AR HMDs with those of traditional displays will not only improve the viewing experience, but will also enable entirely new ways of consuming immersive content using traditional displays.

[0101] According to at least one embodiment, a method can combine multiple methods to maximize the immersion that can be created by combining a light-permeable AR HMD with a traditional display device. In some embodiments, the method can perform a process including the following steps:

[0102] As a pre-processing step, the method may perform the following steps:

[0103] 1. Use streaming protocols to connect to external display devices available in the environment.

[0104] 2. Create a rendering buffer for each display and choose whether to use stereo mode and which stereo mode to use. When using stereo rendering, create two rendering buffers for each display (one for each eye). When the displays in the environment support active stereo mode, the AR HMD can be used to block viewing (for example, as shutter glasses), with one buffer for each eye.

[0105] The rendering / blocking of the view seen by the eye on the AR HMD is synchronized with the environment display.

[0106] For each rendering step, the method may perform the following steps:

[0107] 1. Divide the content into sections that are displayed on the AR HMD or on a connected external display.

[0108] 2. Track the user's head position and calculate additional model views, projections, and viewport transformations used to render the virtual environment from the user's viewpoint onto an external display device within the environment.

[0109] 3. Render images to be displayed on each external display within the environment and stream the images to the display. Mask image areas where overlap occurs between the external display and the AR HMD. The masking can minimize color spillover from the external display to the AR HMD display.

[0110] 4. Use the AR HMD camera to detect the image displayed on the external display to detect artifacts caused by object occlusion, and render the part lost due to the occlusion or the part with artifacts on the AR HMD.

[0111] 5. Render the area outside the external display surface area onto the AR HMD.

[0112] 6. Render the elements selected to be displayed onto the AR HMD.

[0113] 7. In case of active stereo mode, repeat the process by using the conversion of the other eye and switch the eye that is being blocked using the AR HMD.

[0114] In some embodiments, the AR HMD may be connected to an output display device within the environment. The AR HMD device may be connected to an output display device present in the environment using a variety of available technologies. In some embodiments, the AR HMD may automatically poll the output display devices within the environment and connect to any output display device detected. In some embodiments, a list of available output display devices may be displayed to the user, and the user may select one or more output display devices for connection. The actual connection between the output display device and the AR HMD may be formed using any connection protocol, such as Bluetooth, Apple's Airplay, Miracast, Chromecast, or any other known method that can wirelessly display content on an external display device.

[0115] In some embodiments, the AR HMD may use sensors to build a virtual model of the environment and output devices. The AR HMD may poll available output devices (e.g., using wireless networking technology such as WiFi or Bluetooth) and may connect to any output devices detected. A test signal generator within the AR HMD may build test signals and content signals and send them to the connected output devices. The output of the connected output devices may be observed using a camera embedded in the AR HMD, and for each connected output device, the determination module may be configured to discard any output device that does not produce the desired results.

[0116] In some embodiments, in addition to discovering and connecting to available displays in the environment, a "virtual environment model" can also be constructed. A "virtual environment model" is a model that describes the structure of the environment, maps the location of output devices in the environment, and outputs device characteristics. The virtual environment model can be used in the runtime step of the process of tracking the user's movement in the environment, and can also track the position of the external display relative to the user (for example, in some embodiments, relative to the user's eyes). In some embodiments, the virtual environment model can be updated in real time to reflect the user's movement, and the virtual environment model can also be used to calculate the view transformation to apply to the original 3D content. In some embodiments, calculating the virtual environment model includes using one or more RGB-D sensors, cameras, depth sensors, or any combination thereof to determine the relative position of the object and the display in the user's environment. In some embodiments, the arrangement of the sensor / camera on the AR HMD relative to the user's eyes is known (for example, known during some calibration processes). In some embodiments, the object and display distance can be determined by comparing the images captured by the camera on the AR HMD. In some embodiments, the orientation of the display can be determined by analyzing the known image of the external display detected in the environment.

[0117] In some embodiments, the next step in the process is to create a rendering buffer for each connected external display device. In embodiments utilizing stereoscopic rendering, the process may generate two rendering buffers for each connected display, one for each eye. If supported by the AR HMD and the display, active stereoscopic imaging (e.g., active filtering) may be used to generate a stereoscopic image on the connected display. In some embodiments, the external display may support automatic stereoscopic imaging, for example, it may be equipped with a crystalline lens or similar beam splitter technology that can automatically divide the image for the user's eyes. For active stereo, the AR HMD may be configured to synchronize sequential display switching between views of different eyes, blocking the viewing of each eye at a very high rate. In embodiments utilizing active stereoscopic viewing, the AR HMD may block the viewing of an eye (where the content is not currently being rendered), for example, the process may switch between rendering content for the first eye when blocking the second eye and blocking the first eye when rendering content for the second eye.

[0118] For each rendering buffer, the 3D content can be divided between the connected external display device and the AR HMD. In order to divide the 3D content to the external display device and render the content correctly, the processing module can intercept the 3D graphics API call originally created by another application. Intercepting the graphics API call can allow the process to analyze and modify the 3D content without making any modifications to the application that originally generated the 3D content.

[0119] In some embodiments, the virtual environment can respond naturally to user movement to create the illusion that the user is in the virtual environment. User head tracking can enhance the sense of immersion, so that virtual elements in the virtual world can respond to the movement of the user's head in a natural way, for example, allowing the user to examine the elements by moving around the object, just as the user is looking at the real object in the physical environment. Motion parallax is a phenomenon that can activate this sense of immersion, which means that when the user moves his head, the object moves in a very natural way. In some embodiments, the process can perform tracking based on RGB-D data captured by a camera or a sensor embedded in the AR HMD (e.g., a gyro sensor, an infrared (IR) sensor, or various other types of sensors and / or combinations thereof). Based on the sensor data detected by head tracking, the process can calculate how the model view matrix, projection matrix, and / or viewport of the original application should be changed to correctly render the virtual environment on the external display when viewed from the user's new viewpoint. In some embodiments, the view transformation can be calculated based on sensor data representing the shape, position, and orientation of the display. In order to calculate the view transformation, the process can receive the spatial relationship between the user's eye position and the display in the environment from the above-mentioned virtual environment model. OpenGL may provide various functions that may be used to calculate view transformation matrices (such as the model view matrix, the projection matrix, and the viewport settings described above). In some embodiments, some of these functions may accept processed sensor data and may update existing transformation matrices, settings, etc. In some embodiments, an initialization process may determine a first transformation matrix by comparing captured images using a camera. In further embodiments, the first transformation matrix may be continuously updated based on user / head tracking sensor data to provide natural motion parallax from the user's viewpoint.

[0120] Once the 3D content is captured, the process can divide the content by selecting a portion of the content to be displayed on an external display device and a portion of the content to be displayed on the AR HMD. The purpose of this step is to divide the content into: (i) elements to be displayed on an available external display within the environment; and (ii) elements to be displayed on the AR HMD, so that the natural eye adaptation distance of the AR HMD display can match the distance of the user to the external display device, while minimizing the error between the virtual element distance and the natural eye focal length required to view the external display device. For each selected element, this can be done by first determining the virtual distance of the virtual 3D element and comparing this distance to (i) the eye adaptation distance of each available external display device and (ii) the natural eye usage distance of the AR HMD, and selecting the display (e.g., external display or AR HMD) for which the eye adaptation distance of the element best matches the distance of the given virtual element. Fig.12 An example of selectively rendering virtual content on an AR HMD and an external display device to provide optimal natural eye adaptation for viewing the virtual content is depicted. As shown, a virtual element with a better virtual distance match to the display location can be rendered on the external display device. On the other hand, a virtual element close to the AR HMD eye adaptation distance can be rendered using the AR HMD. Various techniques can be used to determine which display is the best match for which virtual distance. For example, the best match can be determined by using a ratio, where the best match is a match where the ratio between the display distance and the virtual distance is close to 1. It can also be based on, for example, the display distance d d The virtual distance d v The matching function m(d d ,d v ) to determine the best match, wherein the display is selected so as to maximize the value of the matching function m(). The matching function m() may be empirically determined based on user feedback regarding comfort and / or the real level of virtual objects at different virtual distances on displays at different physical distances.

[0121] In some embodiments, an exemplary method of dividing data between an external display and an AR HMD may include: for each virtual object in a plurality of virtual objects, determining a virtual distance at which the object will appear from the user's perspective. The object may be rendered on the AR HMD at a virtual distance within an acceptable range of the user's AR HMD. If the virtual distance at which the object will appear is greater than a maximum acceptable distance that can be rendered on the user's AR HMD, the object may be rendered on the external display, wherein the actual distance of the external display from the user is greater than the maximum acceptable distance.

[0122] Fig. 9A user is depicted using an AR HMD in a room with two external display devices. As shown, the content has been divided between display 1 and display 2, and the AR HMD renders part of the content between the displays. In some embodiments, the process may provide a first portion of a video presentation on a first external display, a second portion of the video presentation on a second external display, and a third portion of the video presentation on the AR HMD, wherein the third portion is displayed in the space between the first external display and the second external display. In some embodiments, a virtual environment model may be used to determine what content is displayed on the external display device and what content is displayed on the AR HMD. For example, if the system knows the physical position, orientation, and shape of the external display, as well as the aspect ratio and the desired size of the video presentation to be displayed, the process may calculate the amount (and what portion) of the video presentation that needs to be displayed on the first and second external displays, and then render the remaining portion of the video presentation on the ARHMD. In some embodiments, the camera and / or sensor on the AR HMD may analyze the portions displayed on the first and second displays, and use this information to determine the gap between the displays that needs to be filled with the third portion.

[0123] The next step in the process is to provide the 3D content for each external display to a dedicated rendering cache. The dedicated rendering cache can render the 3D content by applying model view transformations, projections, and viewport settings to the 3D graphics API calls originally generated by the default graphics application. Applying the transformations can change the viewpoint and projection of the original 3D content to compensate for the viewpoint as if the user was looking toward the external display.

[0124] Once the rendering buffer has rendered the selected display element, the overlapping area between the external display device and the AR HMD can be masked to avoid unnecessary color bleeding from the external display to the picture displayed on the AR HMD. In some embodiments, masking can be performed by projecting the elements rendered on the AR HMD to the external display image plane and then replacing the area of ​​the image plane, for example, by rendering the area of ​​the buffer with a masking color (such as black). In some embodiments, a virtual environment model can be used to determine potential areas that can be masked by virtual objects displayed on the AR HMD. Fig.14 An example of masking a portion of a first external display is depicted, and Fig.17An object rendered on the AR HMD is depicted, the object being displayed on the masked area. When an area of ​​the rendering buffer that overlaps with an element to be rendered on the AR HMD has been masked, the image buffer may be streamed to the external display device for output.

[0125] After the rendering buffer has been streamed to the external display, the AR HMD can check the output of each external display by analyzing the image captured using the camera sensor on the AR HMD. By analyzing the camera view of the external display, unwanted anomalies (e.g., artifacts) can be detected. In some embodiments, the captured camera view of the external display can be compared with a version of the original data (e.g., which can be converted). In some embodiments, if part of the content is missing from the captured camera view, an artifact can be detected, which can indicate: (i) the presence of an occluding object; and (ii) the location of the occluded area. An example of the occluded area may be a shadow and occluded area on the display behind the object due to the object being between the viewer and the display or between the projector and the projection area. In some embodiments, the artifact / occluded portion of the image can be rendered on the AR HMD to repair the erroneous area in the output image. Fig.10 An example of a user wearing an AR HMD in a room with a projector is depicted. As shown in the figure, since the user stands between the screen and the projector, a shadow is cast on the screen. In some embodiments, the AR HMD may be configured to detect the area on the screen covered by the user's shadow by using a front-facing camera, and render part of the video content on the AR HMD on the screen area covered by the user's shadow. Fig.11 An example of occlusion is depicted, where an object (depicted as a chair) may block a portion of a user's view of a display. In some embodiments, the AR HMD may be configured to detect such occlusion and may render the occluded portion on the AR HMD.

[0126] In some embodiments, the system may be aware of the image buffer data sent to each external display used, and may detect artifacts within the expected output of the display by comparing the actual display output with the original image buffer sent to the external display device. In some embodiments, this may be achieved by capturing images from a camera sensor embedded with the AR HMD, applying a view transformation to the original image buffer data based on user tracking data, and then comparing the output of the display device within the image captured by the AR HMD camera with the transformed image buffer data. If there are significant differences between areas of the captured image and those areas of the transformed image buffer data, the area may be considered to be an artifact within the output of the external display device (e.g., a shadow or occlusion caused by an object in the environment). In some embodiments, the area affected by the artifact may be corrected by isolating the artifact area within the transformed output image buffer and displaying the transformed output image buffer using the AR HMD.

[0127] Since shadows and occlusion artifacts need to be detected from the display output before they are corrected, users may periodically see delay issues. When an artifact is first detected from the output of the display, there may be an initial delay from detection to repairing it. In some embodiments, the delay depends on the performance of the following aspects: AR HMD camera capture, artifact detection, conversion and output of the corrected image buffer data rendered on the AR HMD. The delay between artifact detection and correction can be minimized by efficient computing performance, the use of high frame rate cameras, and memory bandwidth optimization. Further, since real-time rendering on various display devices (including AR HMD) is expected to be higher than 20Hz, artifacts caused by object occlusion and shadows in the display output can be expected to have relatively low spatial differences between consecutive output frames. Therefore, once a ghost area is detected, it can be assumed that the ghost area will remain the same as the previous frame when rendering the next frame, thereby improving ghost correction, so that the expected abnormal area can be automatically rendered on the AR HMD. Changes in the ghost area can be detected similarly to the first detection of the ghost area, but the updated ghost area can be used for subsequent frames. By this method, the artifacts and abnormal areas visible to the user can be minimized. In some embodiments, more sophisticated methods can be used to estimate how the artifact area will change from frame to frame. This can be achieved by, for example, using a method similar to Kalman filtering to model and estimate the motion of the abnormal area.

[0128] In some embodiments, a method may include the following steps: receiving information about a plurality of objects to be rendered into an augmented reality experience, wherein the objects are to be rendered so as to appear at locations at different distances from a user's augmented environment location; determining a virtual distance range of the objects from the user's augmented environment location to be rendered using a head mounted display (HMD) worn by the user; and rendering the plurality of objects to be rendered into the augmented reality experience within the distance range of the objects from the user's augmented environment location to be rendered using the HMD; rendering at least some of the plurality of objects at a distance exceeding a maximum distance of the distance range of the objects from the user's augmented environment location to be rendered using the HMD by using a display at an actual distance from the user's real world location (the actual distance exceeding a maximum distance of the distance range of the objects from the user's augmented environment location to be rendered using the HMD); and rendering portions of some of the plurality of objects that are occluded, poorly rendered, or intermixed on the user's HMD to improve the quality and integrity of the overall experience.

[0129] In some embodiments, when the AR HMD product features an RGB-D sensor, depth data from the RGB-D sensor can be used to predict occlusion caused by objects within the environment. In some embodiments, object geometry can be modeled and object positions estimated (e.g., a virtual environment model), so that occlusion caused by objects can be estimated without first observing the display output.

[0130] In some embodiments, virtual elements within areas not covered by an external display may be rendered on the AR HMD. In this embodiment, the AR HMD may be used to extend the entire display area. In some embodiments, content may be rendered on the AR HMD for areas that appear within the field of view of the user wearing the ARHMD but do not have available external display surface area. While the image quality may not match that of the external display, this approach may help improve how the user perceives the virtual world and may serve as a bridge between different external displays, creating a better sense of immersion together. In some embodiments, as Fig. 9 As shown, the AR HMD may bridge the spatial gap between two or more external displays, or it may simply increase the effective display area of ​​one or more displays. This embodiment may use a virtual environment model and image analysis techniques to determine what content can be displayed on the AR HMD and where to display the content on the AR HMD. In some embodiments, knowing the user's physical distance from the gap between the displays and what objects will appear and how far away they are virtually, the process can generate stereoscopic output by using eye parallax, resulting in an accurate display of 3D content.

[0131] In some embodiments, while using stereo output, the rendering process described in the above paragraph may be performed for the first eye, and then repeated for the second eye. In some embodiments, the AR HMD may also block the eye for which the rendering process is not currently being performed.

[0132] In some embodiments, an AR HMD that can block the view (e.g., by using a liquid crystal display layer or similar device that can be configured to switch display pixels between an opaque state and a transparent state and can switch at a sufficiently high frame rate) can be used as active shutter glasses in addition to being an AR display device. In addition, the AR HMD can be configured to synchronize the blocking of the view (e.g., by adjusting the shutter frame rate to synchronize with the display) so that the 3D stereo output of the active stereo display can be properly supported. In some embodiments, the active stereo display can provide a timing signal for the shutter glasses via a wired signal or by an infrared or wireless radio frequency (e.g., Bluetooth, DLP link) transmitter. In order to enable correct synchronization, the AR HMD can detect the timing signal provided by the display, or detect the eye switching frequency by other means.

[0133] In some embodiments, the AR HMD may use an IR camera to detect an infrared (IR) sync signal. Many AR HMDs may have an IR camera for 3D tracking and may be used to detect a sync signal. For example, an RGBD sensor provided on an AR HMD may have an RGB camera that is sensitive to infrared light or a separate camera that operates in the infrared range.

[0134] In some embodiments, the product feature of the AR HMD may be Bluetooth connectivity. Therefore, hardware that can be configured to detect synchronization signals within the Bluetooth frequency range (usually around 2.4 GHz) is usually embedded in the AR HMD.

[0135] In addition to detecting the synchronization signal, an AR HMD with a camera running at a sufficiently high frame rate can detect the switching frequency of the active stereo display by analyzing video frames captured from the AR HMD camera looking at the active stereo display. This embodiment can enable the AR HMD to synchronize shutter operation with any active stereo display by using any synchronization signal transmission technology.

[0136] Detecting the eye view switching frequency by analyzing video frames captured from an AR HMD camera looking towards an active stereoscopic display can be broken down into multiple steps for synchronizing the AR HMD shutter function.

[0137] In some embodiments, information indicating whether a display is outputting in active stereo mode may not be provided to the system. The system may detect whether a display is outputting in active stereo mode. If a display with active stereo output is detected, the next step may be to detect the frequency and timing used by the display to switch between left / right eye views.

[0138] In some embodiments, the system may be aware that media is being displayed and may be able to determine the eye order of active stereo output (i.e., which image is for the left eye and which image is for the right eye) by comparing the image output by the external display device with the media content. However, in some embodiments, the system may not always have access to the media content. In this embodiment, the system may perform additional contextual analysis to determine the correct eye order. In some embodiments, the system may compare occlusions that occur before display elements. In some embodiments, the system may be operable to identify known objects (e.g., faces, bodies, etc.) within the image output by the external display. If a known object is detected, knowledge of the geometry of this object may help the system determine which image order may produce more correct depth cues. In some embodiments, a less heuristic method using a neural network trained with a large number of samples to detect the correct left / right eye order may be used to estimate the correct eye order.

[0139] In some embodiments, to solve the above listed sub-problems, it is necessary to store and analyze the captured display images in a time sequence of viewing timestamps. An exemplary process includes: (i) detecting whether the display is outputting in active stereo mode; and (ii) setting synchronization if active stereo output is detected.

[0140] By using an AR HMD as shutter glasses as well as an active stereo display, several benefits can be provided. One benefit is that it can simplify the use of various types of content. In some embodiments, the AR HMD can display AR content within the environment and switch to use as active shutter glasses when needed. This eliminates the need to use different glasses or switch glasses depending on which type of content is being consumed.

[0141] The AR HMD can detect and track the active stereoscopic display surface area by analyzing images captured from sensors embedded in the AR HMD. This allows the AR HMD itself to block the portion of the user's view that overlaps with the active stereoscopic display image when the active stereoscopic display is playing stereoscopic content. This can reduce view flickering because only part of the image is flickered at a high frame rate. This can also reduce the loss of light received by the user's eyes because, unlike current active shutter glasses (which effectively block the entire viewing area of ​​one eye at a time), only the portion of the view that overlaps with the display is turned off. In some embodiments, the AR HMD can provide shutter glasses functionality using a liquid crystal (LC) layer (which can be used to block light from reaching the user's eyes). In some embodiments, the LCD can be a thin layer or film arranged on the AR HMD display, which can be controlled by the system to selectively block the area overlapping the display based on the display position and orientation information obtained using the above method.

[0142] In some embodiments, using the AR HMD as shutter glasses may enable viewing of multiple active stereoscopic displays simultaneously. Different active stereoscopic displays may also potentially even operate at different frequencies. Because the ARHMD can detect and track several display regions, the shutter effect can be limited to only the regions where the various active stereoscopic displays overlap, and the shutter regions can operate at their own individual frequencies, as long as the frame rate update of the AR HMD display is synchronized with the combined frequency of all displays. In addition, the AR HMD may detect multiple synchronization signals in multiple different transmission techniques, thereby being able to synchronize with multiple active stereoscopic displays simultaneously using different synchronization signal transmission techniques.

[0143] In some embodiments, the AR HAM can be used as an active shutter, and the RGB display capabilities of the AR HMD can enhance the main image displayed on the active stereo display. In some embodiments, this can improve the luminosity of the active stereo display by combining the RGB rendering on the AR HMD with the image rendered on the active stereo display. In some embodiments, the AR HMD can selectively block areas of the active stereo display so that areas of the image that are intended to be displayed as black can be displayed darker. At this time, this embodiment can extend the natural dynamic range of the active stereo display and increase the maximum luminosity beyond the maximum luminosity that the active stereo display alone can produce.

[0144] In some embodiments, crosstalk between an active stereo display and an AR HMD can be reduced relative to crosstalk between an active stereo display and conventional active shutter glasses. Due to slower pixel off times, some active stereo displays may experience crosstalk when switching eye views while the ghost image displayed to the other eye remains on the display. In some embodiments, this can be reduced by detecting potential crosstalk areas using the AR HMD and compensating for potential crosstalk errors by displaying an RGB image on the AR HMD, thereby minimizing the crosstalk (e.g., smoothing high contrast in potential crosstalk areas).

[0145] Some embodiments may minimize flicker by rendering an RGB image for the eye that is currently blocked from viewing the image on the active stereo display. The AR HMD may be configured to render the content displayed on the active stereo display using the AR HMD RGB display at a lower resolution or with a poorer color range for the currently blocked eye, thereby reducing the flicker experience presented by the active shutter glasses and reducing the overall brightness drop of the view.

[0146] In the main embodiment, the process can be performed using a light-transmissive AR HMD that can constantly check the environment using an embedded camera or RGB-D sensor. However, in an alternative embodiment, the calculations, communication between components, content management, and rendering can be arranged in an alternate manner. In this alternative embodiment, the above process may require some modifications and can be performed on various other types of devices instead of the AR HMD described in the above embodiment.

[0147] In some embodiments, occlusion caused by objects can be detected by using a virtual environment model, rather than by analyzing images output from the display to detect objects that block the view of the external display from the user's viewpoint. In this embodiment, the virtual environment model may have been created within the initialization method and may have a sufficiently high resolution while also covering the entire operating area. In this alternative method, areas that are occluded by objects in front of the external display may be automatically selected to be rendered on the AR HMD rather than on the external display. In this embodiment, at the first run-time processing step where content is divided to be displayed on various displays, areas of the external display that are blocked by other objects may be detected and selected to be rendered on the AR HMD.

[0148] In some embodiments, instead of using AR HMD sensors for head tracking and display artifact detection, an alternative embodiment may receive sensor data from sensors embedded in the environment. In this embodiment, a mobile application performing the process may request data from an external sensor, perform user head tracking based on the external sensor data, and determine head tracking based on the known external sensor position. In some embodiments, the external sensor may be an RGB-D sensor that observes the user. In some embodiments, the external sensor may be an optical sensor, a magnetic sensor, a depth sensor, or a sound tracking sensor, etc.

[0149] In some embodiments, shadows and occlusions caused by objects and / or viewers may also be detected by analyzing external sensor data. In this embodiment, problem areas (e.g., occluded areas) may already be considered during the content segmentation step of the process, thereby avoiding situations where erroneous images need to be initially displayed to be detected. In some embodiments, an RGB-D sensor observing a user may detect shadow areas based on an approximate estimate of the user geometry, rather than detecting shadow areas based on the output image.

[0150] In some embodiments, an external server may be configured to create an immersive experience and perform the above process. In some embodiments, the server may receive sensor data from the AR HMD and perform the entire rendering process based on the received sensor data. In some embodiments, when streaming sensor data from the AR HMD to the external server performing the process, the network interface may introduce a certain level of latency. In embodiments that experience sufficiently high network latency, some features described herein may not be implemented (e.g., image error detection and correction based on camera data analysis during runtime may not be implemented).

[0151] In an alternative embodiment, in addition to performing the rendering process for the combined display described above, the application that generates the display content can also be executed on a dedicated server. In this embodiment, the server can be connected to all displays in the environment. Based on the knowledge of the location of these displays relative to the user's location, the server can stream content to each display (including the AR HMD device worn by the user).

[0152] Fig.13 A plan view of a user viewing virtual 3D content in a room with a large forward-projecting projector and a regular monitor is shown. Fig.13 It includes users in the environment and various virtual 3D elements, projectors, projection screens, physical objects, and external display devices. As shown in the figure, Fig.13The following occlusions are included: the viewer blocking the projector and the physical object blocking the display screen. In some embodiments, after the user launches an application that generates 3D content (e.g., a virtual world viewer), the module implementing the above rendering process can poll the available output devices in the environment (e.g., external display devices and / or projectors) and connect to any available output display device. In some embodiments, the rendering process can create the required rendering buffer for each external display.

[0153] Once the external output devices and their relative positions relative to the user's eye position are known, the mobile application can divide the intercepted graphics API calls into 3D content to be sent to different display devices. In some embodiments, the 3D content can be divided based on matching eye adaptation distances, which in some embodiments results in individual virtual elements closest to the user being selected for display as a whole on the AR HMD, while other elements at relatively large virtual distances can be displayed primarily on the available external display. In some embodiments, for each rendering step, the external display position relative to the user's eye position can be updated based on 3D tracking of the user's movement in space (e.g., user head tracking) (the tracking is based on sensor data received from sensors embedded with the AR HMD).

[0154] In some embodiments, elements selected to be rendered on the external display can be rendered by injecting the desired model view, projection, and / or viewport settings into the intercepted graphics API call. Once the view has been rendered to a render buffer, overlapping areas of objects to be rendered on the AR HMD can be masked from the render buffer, and the render buffer can be streamed to the external display. Fig.14 The output on the external display after masking the selected area to be rendered on the AR HMD is shown. As shown, due to the different 3D elements to be displayed on the AR HMD, part of the 3D elements to be displayed on the projection screen can be masked.

[0155] In some embodiments, after the display image is streamed to and displayed on the external display, the mobile application can analyze the camera sensor data captured by the camera embedded in the AR HMD for artifacts within the output image. In some embodiments, shadows cast by the viewer on the projected image and occlusions caused by a chair in front of the display screen can be detected. Fig.15 As shown, for the erroneous image area, the correct part of the view can be rendered and displayed on the AR HMD.

[0156] In some embodiments, the AR HMD may render a 3D content area outside the area covered by the external display device. Fig.16 An exemplary embodiment is shown. Fig.16 As shown, 3D content rendered on an AR HMD can bridge the area between the two displays.

[0157] In some embodiments, Fig.17 As shown, 3D content elements selected to be displayed on the AR HMD may be rendered and displayed on the AR HMD.

[0158] Shutter Glasses Changes

[0159] Using an HMD to present a full VR / VR experience may produce eye strain, headaches, nausea, and fatigue, due in part to convergence-accommodation conflict. Further, using stereoscopic projection will limit opportunities for personalization for the user and privacy. In some embodiments, the above process can produce a combined AR HMD / stereoscopic projection, where the AR HMD can also be used as active shutter glasses for the projection. Fig.18 An exemplary embodiment is shown. As shown in the figure, Fig.18 The invention comprises a user wearing an AR HMD and two external display devices, each external display having a corresponding shutter timing synchronization signal transmitter. In some embodiments, the display can activate a 3D effect while using an active shutter method.

[0160] In some embodiments, the method may include the following steps: capturing an image of a portion of a first projection screen using a forward-facing camera associated with an AR HMD; responsively determining that the captured image represents a first portion of content associated with an AR display; and configuring the AR HMD to generate an image at a frequency associated with the first portion of content while the AR HMD also displays a second portion of content associated with the AR display. In some embodiments, the first portion of content associated with the AR display may be displayed as alternating frames of a stereoscopic projection. In some embodiments, the generated image may be used as an alternating left and right shutter. In some embodiments, a left-eye image of the second portion of content associated with the AR display is displayed in a portion of the AR HMD associated with the left-eye display, while an image for generating a shutter effect for the right eye is displayed in a portion of the AR HMD associated with the right-eye display, and vice versa.

[0161] In some embodiments, synchronization can be determined by analyzing a camera signal that captures the projection screen image. In some embodiments, the method further comprises detecting the frequency of the left / right display. In some embodiments, the detection is performed by image analysis. In some embodiments, the detection is performed by occlusion analysis.

[0162] In some embodiments, a method for distributing content display among display devices within an AR system includes the following steps: capturing images of portions of first and second display devices within an environment using a forward-facing camera associated with a head-mounted AR display, the first and second display devices displaying first and second portions of content associated with an AR presentation; and displaying a third portion of content related to the AR presentation on the head-mounted AR display, the third portion being determined based on the images of portions of the first and second display devices captured using the forward-facing camera of the head-mounted AR display, wherein the determination of the third portion of content related to the AR presentation includes determining a portion of content related to the AR presentation associated with a portion of the field of view not occupied by the first or second display device.

[0163] In some embodiments, a method for improving an immersive multi-device virtual / augmented experience includes the following steps: identifying available display devices with content streaming interfaces within an experience environment; allocating a rendering buffer for each available display; splitting the content into portions displayed on an AR HMD or on a connected external display; tracking the user's head position and calculating additional view transformations (e.g., model views, projections, and viewports) used to render the virtual environment viewed from the user's viewpoint to an external display within the environment; rendering images to be displayed on each external display within the environment and streaming the images to the displays; masking areas of the image to be displayed on the external display that will overlap with elements to be rendered on the AR HMD to minimize color bleeding from the external display to the AR HMD display.

[0164] In some embodiments, the method further includes checking the image displayed on the external display with the AR HMD camera to detect artifacts caused by object occlusion. In some embodiments, the method further includes rendering the missing portion or artifact of the image on the AR HMD. In some embodiments, the method further includes rendering areas outside the surface area of ​​the external display on the AR HMD. In some embodiments, the method further includes rendering elements selected to be displayed on the AR HMD. In some embodiments, the AR HMD may utilize an active stereo mode, and the process may be repeated using a switch of one eye while blocking the second eye. In some embodiments, the method further includes switching the eye that is being blocked by the AR HMD.

[0165] In some embodiments, the method further comprises creating two rendering buffers for each display, one for each eye, or if not in stereo mode, creating a single rendering buffer for each display.

[0166] Fig.19A flow chart of a process according to some embodiments is depicted. As shown in the figure, Fig.19 The method includes an image capture process, which includes the following steps: capturing frames from an AR HMD; detecting a display image from the captured frames; isolating and compensating for perspective transformation of the image on the HMD based on corner points of the display area detected from the captured images; and adding the undistorted image visible on the display to a display image queue to be analyzed. Once the number of images in the display image queue exceeds a preset requirement, the image queue analysis process is started. Fig.19 As shown, the image queue analysis process includes the following steps: calculating the difference between consecutive images in the queue and detecting the frequency based on the difference between different frames (active stereo output should appear as a clear high-frequency band). In some embodiments, Fourier transform can be used to detect the frequency distribution. The next step is to pair the images as a stereo pair based on the detected high-frequency band. The pixel or feature conversion between the images forming the stereo pair can be compared and the first and second elements between the stereo pairs can be compared. When the first or second elements between different pairs are compared, the offset is very small, and the offset between the images of a single stereo pair should be very similar to the situation between pairs. In a preferred embodiment, the offset between the images within the stereo pair will be very small. If this is met, it can be determined that the output of the display is in an active stereo mode at a known frequency. The next step can then be to detect which image in the stereo pair image is for which eye. In some embodiments, this can be done by comparing the picture with the original media or using object recognition (occlusion, etc.). Finally, synchronization can be set for active shutter glasses.

[0167] In an embodiment, there is a method comprising: detecting a configuration of a switchable display; detecting a position of a user; and based on the position and the configuration, switching graphics output to the switchable display.

[0168] In one embodiment, there is a system comprising: a first sensor operable to detect a configuration of a convertible display; a second sensor operable to detect a position of a user; and a display configuration manager operable to convert graphics output to the convertible display based on the position and the configuration.

[0169] In an embodiment, there is a method comprising: receiving sensor data from a plurality of sensors; detecting a display configuration of at least one display; based on the display configuration, generating a projective transformation; generating an output video stream that applies the projective transformation to an unmodified video stream; and sending the output video stream to the at least one display. In some embodiments, the method may further include wherein the plurality of sensors include a user tracking sensor. In some embodiments, the method may further include wherein the user tracking sensor is an RGB-D sensor. In some embodiments, the method may further include wherein the user tracking sensor is a camera. In some embodiments, the method may further include using the user tracking sensor to determine a primary user from within a group of users. In some embodiments, the method may further include determining a user eye position using the user tracking sensor. In some embodiments, the method may further include wherein the plurality of sensors include a display configuration sensor. In some embodiments, the method may further include wherein the display configuration sensor is an angle detection sensor. In some embodiments, the method may further include wherein the display configuration sensor is an electronic potentiometer. In some embodiments, the method may further include wherein detecting the display configuration includes detecting an angle between two or more displays. In some embodiments, the method may further include wherein detecting the display configuration includes detecting a shape between at least one display. In some embodiments, the method may further include wherein detecting the display configuration includes detecting a relative position of two or more displays. In some embodiments, the method may further include wherein the at least one display includes a flat display. In some embodiments, the method may further include wherein the at least one display includes a curved display. In some embodiments, the method may further include wherein the at least one display includes a flexible display. In some embodiments, the method may further include wherein generating the projective transformation further includes determining an area rendered via the output video stream. In some embodiments, the method may further include wherein the display configuration is determined for a predetermined user position. In some embodiments, the method may further include wherein the display configuration is continuously updated. In some embodiments, the method may further include transmitting a configuration control to at least one display. In some embodiments, the method may further include wherein the configuration control adjusts the display configuration. In some embodiments, the method may further include wherein the configuration control is input via a touch screen. In some embodiments, the method may further include wherein the configuration control is input via a keyboard. In some embodiments, the method may further include wherein the configuration control is input via a voice command. In some embodiments, the method may further include wherein the configuration control is input via a controller.In some embodiments, the method may further include wherein the configuration control is input via a joystick. In some embodiments, the method may further include wherein the display dynamically appears. In some embodiments, the method may further include wherein the display dynamically disappears. In some embodiments, the method may further include wherein the output video stream is based on stereo rendering. In some embodiments, the method may further include detecting user head tracking. In some embodiments, the method may further include wherein detecting the display configuration includes detecting the orientation of the at least one display. In some embodiments, the method may further include wherein detecting the display configuration includes detecting the number of displays within a user's viewing range. In some embodiments, the method may further include wherein detecting the display configuration includes detecting the position of the at least one display. In some embodiments, the method may further include wherein detecting the display configuration includes detecting the shape of the at least one display.

[0170] In one embodiment, there is a device comprising: a user sensor configured to detect user information; a display configuration sensor configured to detect a display configuration of at least one display; a dynamic display configuration manager configured to receive the user information, the display configuration, and a graphics API call, the dynamic display configuration manager configured to calculate a projection transformation and responsively render a corresponding video output for each of the at least one display by applying the projection transformation to each unmodified video stream; and a graphics driver configured to output the corresponding rendered video output. The device may further include wherein the user sensor is configured to detect a user eye position. The device may further include wherein the user sensor is configured to determine a primary user. The device may further include wherein the user sensor is configured to provide head tracking information. The device may further include wherein the user sensor includes an RGB-D sensor. The device may further include wherein the user sensor includes an inertial measurement unit (IMU). The device may further include wherein the user sensor includes a camera. The device may further include wherein the display configuration sensor is configured to detect the number of available displays. The device may further include wherein the display configuration sensor is configured to detect the position of the at least one display. The device may further include wherein the display configuration sensor is configured to detect the relative position of two or more displays. The device may further include wherein the display configuration sensor is configured to detect the orientation of the at least one display. The device may further include wherein the display configuration sensor is an angle detector configured to detect the angle between two displays. The device may further include wherein the display configuration sensor is configured to determine the shape of the at least one display. The device may further include wherein the display configuration sensor is a depth sensor. The device may further include wherein the display configuration sensor is a sound sensor. The device may further include wherein the display configuration sensor is an optical sensor. The device may further include wherein the display configuration sensor is a magnetic sensor. The device may further include wherein the at least one display comprises a planar display. The device may further include wherein the at least one display comprises a non-planar display. The device may further include wherein the dynamic display configuration manager is configured to: read back a first rendered image based on a normal projection, responsively warp the first rendered image to correct for display geometry deformation, and send the warped image to the at least one display. The device may further include wherein the at least one display comprises a flexible display. The device may further include wherein the at least one display comprises an organic light emitting diode (OLED) display. The apparatus may further include wherein the at least one display comprises a liquid crystal display (LCD).The apparatus may further include wherein the display configuration sensor is configured to detect at least one display that appears dynamically. The apparatus may further include wherein the display configuration sensor is configured to detect at least one display that disappears dynamically. The apparatus may further include wherein the dynamic display configuration manager is configured to receive user configuration controls. The apparatus may further include wherein the display configuration is adjusted according to the user configuration controls. The apparatus may further include wherein the user configuration controls include display orientation change requests. The apparatus may further include wherein the user configuration controls include display on / off requests. The apparatus may further include wherein the user configuration controls include display warping requests.

[0171] In some embodiments, there is a method comprising: receiving a captured frame from a forward-facing camera embedded on an AR HMD; identifying a distorted display image within the captured frame; forming an undistorted display image using the display image; and comparing the undistorted display image to an original display image and determining whether an artifact exists. In some embodiments, the method may further include wherein the artifact comprises a shadow occlusion. In some embodiments, the method may further include wherein the artifact comprises an object occlusion. In some embodiments, the method may further include wherein identifying the distorted display image comprises identifying a corner of the distorted image. In some embodiments, the method may further include wherein forming the undistorted image comprises: identifying perspective information based on the identified corner; and applying a first transformation to the distorted display image based on the perspective information. In some embodiments, the method may further include wherein the first transformation is based on a model view transformation / matrix. In some embodiments, the method may further include wherein the first transformation is a viewport transformation. In some embodiments, the method may further include: determining missing content obscured by the artifact; applying a second transformation to the missing content; and displaying the transformed missing content on a display on the AR HMD. In some embodiments, the method may further include converting wherein the second conversion is an inverse conversion of the first conversion.

[0172] In an embodiment, there is a method comprising: detecting an artifact within a virtual presentation, the artifact being at a first location; correcting the artifact within the virtual presentation for a first time appearing at the first location; and correcting the first artifact for a second time appearing at a second location, the second location being determined based at least in part on the first location. In some embodiments, the method may further comprise estimating the second location based on the first location. In some embodiments, the method may further comprise wherein estimating the second location is further based on a third location, the third location appearing at a third time prior to the first time. In some embodiments, the method may further comprise wherein estimating the second location is based at least in part on Kalman filtering.

[0173] In an embodiment, there is a method comprising: receiving a captured frame from a forward-facing camera embedded on an AR HMD; identifying a distorted display image within the captured frame; forming a distorted original image based on a transformation of the original image; and comparing the distorted display image with the distorted original image and identifying one or more artifacts. In some embodiments, the method may further include that the transformation is based on perspective information. In some embodiments, the method may further include that the perspective information is obtained by identifying a corner of the distorted display image. In some embodiments, the method may further include that the artifact is an occluded object. In some embodiments, the method may further include that the artifact is a shadow area. In some embodiments, the method may further include that the artifact is an overlap between a virtual object displayed on the AR HMD and the distorted display image. In some embodiments, the method may further include masking an area corresponding to the overlap on an external display that is currently displaying the distorted display image.

[0174] In one embodiment, there is a device comprising: a forward-facing camera embedded on an AR HMD configured to capture a frame; an image processor configured to receive the captured frame, identify a distorted display image within the captured frame, and use the display image to form an undistorted display image; and a comparison module configured to compare the undistorted display image with the original image and determine whether an artifact exists. In some embodiments, the device may further include wherein the artifact includes a shadow occlusion. In some embodiments, the device may further include wherein the artifact includes an object occlusion. In some embodiments, the device may further include wherein the image processor is configured to identify a corner of the distorted display image. In some embodiments, the device may further include wherein the processor is configured to form the undistorted image by: identifying perspective information based on the identified corner, and applying a first transformation to the distorted image based on the perspective information. In some embodiments, the device may further include wherein the transformation is based on a model view transformation / matrix. In some embodiments, the device may further include wherein the transformation is a viewport transformation. In some embodiments, the device may further include the image processor being configured to: determine lost content obscured by the artifact, apply a second transformation to the lost content; and display the transformed lost content on a display on the AR HMD. In some embodiments, the device may further include an inverse transformation in which the second transformation is an inverse transformation of the first transformation.

[0175] In an embodiment, there is a device comprising: an image detection module configured to detect an artifact within a virtual presentation, the artifact being at a first location; an image correction module configured to correct the artifact within the virtual presentation that appears at the first location at a first time, and to correct the first artifact that appears at a second location at a second time, the second location being based at least in part on the first location. In some embodiments, the device may further include wherein the image correction module is configured to estimate the second location based on the first location. In some embodiments, the device may further include wherein estimating the second location is further based on a third location that appears at a third time prior to the first time. In some embodiments, the device may further include wherein estimating the second location is based at least in part on Kalman filtering.

[0176] In one embodiment, there is a device comprising: a forward-facing camera embedded on an AR HMD configured to capture a frame; an image processor configured to receive the captured frame, identify a distorted display image within the captured frame, and form a distorted original image based on the original image; and a comparison module configured to compare the distorted display image with the distorted original image and determine whether there is an artifact. In some embodiments, the device may further include wherein the image processor is configured to generate the distorted original image by applying a transformation to the original image. In some embodiments, the device may further include wherein the transformation is based on perspective information. In some embodiments, the device may further include wherein the image processor obtains perspective information by identifying a corner of the distorted display image. In some embodiments, the device may further include wherein the artifact is an occluded object. In some embodiments, the device may further include wherein the artifact is a shadow area. In some embodiments, the device may further include wherein the artifact is an overlap between a virtual object displayed on the AR HMD and the distorted display image. In some embodiments, the device may further include a masking module configured to mask an area of ​​the external display corresponding to the overlapping portion between the virtual objects.

[0177] In an embodiment, there is a method comprising: providing a video presentation on an external display; determining a portion of the video presentation that is obscured from a user; and displaying the obscured portion of the video presentation on a head mounted display of the user. In some embodiments, the method may further include wherein the obscured portion of the video presentation displayed on the head mounted display is aligned with the video presentation on the external display from the perspective of the user. In some embodiments, the method may further include wherein the external display is a television screen. In some embodiments, the method may further include producing the video presentation by modifying an original video presentation according to a view transform. In some embodiments, the method may further include wherein the view transform is a model view transform. In some embodiments, the method may further include wherein the view transform is based on a viewport setting. In some embodiments, the method may further include wherein the view transform is based on user head tracking. In some embodiments, the method may further include wherein the view transform is based on a virtual environment model. In some embodiments, the method may further include wherein the determination of the obscured portion is at least partially accomplished by using a forward-facing camera on a head mounted display. In some embodiments, the method may further include: generating a virtual environment model of the user's environment; and determining the obscured portion based on the environment model. In some embodiments, the method may further include wherein the virtual environment model is generated at least in part by using a sensor on the head mounted display. In some embodiments, the method may further include wherein the sensor comprises a device selected from the group consisting of an optical sensor, a magnetic sensor, a depth sensor, and a sound tracking sensor.

[0178] In an embodiment, there is a method comprising: projecting a video presentation onto a surface; determining that a portion of the projected video presentation is shadowed; and displaying the shadowed portion of the video presentation on a head mounted display of a user. In some embodiments, the method may further comprise wherein the shadowed portion of the video presentation displayed on the head mounted display is aligned with the projected video presentation from the perspective of the user. In some embodiments, the method may further comprise turning off an area of ​​the projector corresponding to the video presentation occupied by the shadowed portion of the projected video presentation.

[0179] In an embodiment, there is a method comprising: providing a first portion of a video presentation on a first external display and providing a second portion of the video presentation on a second external display; and displaying a third portion of the video presentation on a head mounted display, wherein the third portion is displayed in a space between the first and second external displays. In some embodiments, the method may further include wherein the third portion of the video presentation is aligned with the first and second portions from the perspective of a user of the head mounted display. In some embodiments, the method may further include wherein the first and second external displays are active stereo displays. In some embodiments, the method may further include wherein the first and second external displays have corresponding first and second display frequencies, respectively. In some embodiments, the method may further include wherein the first and second display frequencies are different. In some embodiments, the method may further include detecting the first and second display frequencies using a camera on the head mounted display. In some embodiments, the method may further include independently blocking the first and second external displays for a first eye using the head mounted display. In some embodiments, the method may further include alternatingly blocking the first and second external displays for the first eye and the second eye at the first and second detected display frequencies, respectively. In some embodiments, the method may further include receiving the first and second display frequencies from first and second shutter timing synchronization signal transmitters, respectively.

[0180] In an embodiment, there is a method comprising: providing a first portion of a video presentation on a first external display and providing a second portion of the video presentation on a second external display; and displaying a third portion of the video presentation on a head mounted display, wherein the third portion at least partially overlaps an area on the first or second external display. In some embodiments, the method may further include masking the overlapping area. In some embodiments, the method may further include flipping pixels within the overlapping area. In some embodiments, the method may further include wherein masking includes displaying a masking color in the overlapping area. In some embodiments, the method may further include wherein the masking color is black.

[0181] In one embodiment, there is a method comprising: capturing images of portions of first and second display devices within an environment using a forward-facing camera associated with a head-mounted augmented reality (AR) head-mounted display (HMD), the first and second display devices displaying first and second portions of content related to an AR presentation; and displaying a third portion of content related to the AR presentation on the AR HMD, the third portion being determined based on the images of portions of the first and second display devices captured using the forward-facing camera. In some embodiments, the method may further include determining the third portion of content related to the AR presentation including determining a portion of content related to the AR presentation associated with a portion of a field of view not occupied by the first or second display device.

[0182] In an embodiment, there is a method comprising: connecting to an external display; capturing a set of 3D content; splitting the set of 3D content into a first virtual element and a second virtual element; displaying the first virtual element on the external display; and displaying the second virtual element on a local display. In some embodiments, the method may further include wherein the local display is a light-transmissive augmented display (AR) head-mounted display (HMD). In some embodiments, the method may further include wherein the external display is a television. In some embodiments, the method may further include wherein the external display is a projection screen. In some embodiments, the method may further include detecting an occlusion area on the external display. In some embodiments, the method may further include wherein the occlusion area is a shadow cast on the projection screen. In some embodiments, the method may further include wherein the occlusion area is an area on the television screen blocked by an object. In some embodiments, the method may further include rendering content blocked by the occlusion area on the local display. In some embodiments, the method may further include applying a view transformation to at least one of the first and second virtual elements. In some embodiments, the method may further include wherein the view transformation is a model view transformation. In some embodiments, the method may further include wherein the view transformation is based on a viewport setting. In some embodiments, the method may further include wherein the view transition is determined based on user head tracking. In some embodiments, the method may further include assigning the first and second virtual elements to the external display and the local display, respectively, based on natural eye adaptation. In some embodiments, the method may further include generating a virtual environment model, the model including a virtual layout of the external display relative to the user. In some embodiments, the method may further include wherein the virtual environment model includes a relative external display distance. In some embodiments, the method may further include wherein the virtual environment model includes position information of one or more objects relative to the external display and the user. In some embodiments, the method may further include wherein occlusion is determined based on the virtual environment model. In some embodiments, the method may further include wherein the virtual environment model is calculated based on sensor data from a depth sensor. In some embodiments, the method may further include detecting an overlap area between the local display and the external display. In some embodiments, the method may further include wherein displaying the content of the external display within the overlap area on the local display. In some embodiments, the method may further include rendering the overlap area of ​​the external display with a masking color. In some embodiments, the method may further include detecting image artifacts on the external and / or local display.In some embodiments, the method may further include wherein the image artifact is an occlusion. In some embodiments, the method may further include wherein the image artifact is an overlap between the local and external displays. In some embodiments, the method may further include wherein the local display may extend the display area of ​​the external display. In some embodiments, the method may further include displaying a third virtual element on a second external display, the third virtual element being segmented from the 3D content. In some embodiments, the method may further include wherein the local display bridges a gap between the external display and the second external display. In some embodiments, the method may further include wherein the external display and the second external display are active stereoscopic displays, each having a respective display frequency. In some embodiments, the method may further include detecting the respective display frequencies using a camera attached to the local display. In some embodiments, the method may further include wherein the respective display frequencies are different. In some embodiments, the method may further include wherein the local display is used as shutter glasses, and wherein the local display blocks a first eye for the external display at a first display frequency of the respective display frequencies, and wherein the local display blocks the first eye for the second external display at a second display frequency of the respective display frequencies. In some embodiments, the method may further include wherein the external display is an active stereoscopic display. In some embodiments, the method may further include wherein the local display blocks a first eye of a user and displays the first and second virtual elements for a second eye of the user. In some embodiments, the method may further include wherein blocking the first eye includes blocking a portion of the local display occupied by the external display when viewed from the first eye. In some embodiments, the method may further include displaying an RGB image on the local display for the first eye.

[0183] In one embodiment, there is a device comprising: a transmission unit configured to stream a video presentation to an external display; a camera mounted on a head mounted display configured to determine a portion of the video presentation that is obscured from a user; and a transparent screen on the head mounted display configured to display the obscured portion of the video presentation. In some embodiments, the device may further include wherein the obscured portion of the video presentation displayed on the head mounted display is aligned with the video presentation on the external display from the perspective of the user. In some embodiments, the device may further include wherein the external display is a television screen. In some embodiments, the device may further include a conversion module configured to generate the video presentation by modifying the original video presentation according to a view transformation. In some embodiments, the device may further include wherein the view transformation is a model view transformation. In some embodiments, the device may further include wherein the view transformation is based on a viewport setting. In some embodiments, the device may further include wherein the view transformation is based on user head tracking. In some embodiments, the device may further include wherein the view transformation is based on a virtual environment model. In some embodiments, the device may further include a processor configured to: generate a virtual environment model of the user environment; and determine the obscured portion based on the virtual environment model. In some embodiments, the device may further include wherein the processor generates the virtual environment model at least in part in response to receiving sensor data from a sensor on the head mounted display. In some embodiments, the device may further include a device wherein the sensor includes a device selected from the group consisting of: an optical sensor, a magnetic sensor, a depth sensor, and a sound tracking sensor.

[0184] In one embodiment, there is a system comprising: a projector configured to project a video presentation onto a surface; a sensor configured to determine that a portion of the projected video presentation is shadowed; and a head mounted display configured to display the shadowed portion of the video presentation. In some embodiments, the system may further include wherein the shadowed portion of the video presentation displayed on the head mounted display is aligned with the projected video presentation from the perspective of the user. In some embodiments, the system may further include wherein the projector is configured to close an area of ​​the video presentation corresponding to that occupied by the shadowed portion of the projected video presentation. In some embodiments, the system may further include wherein the projector is mounted to the head mounted display. In some embodiments, the system may further include wherein the projector is externally located relative to the head mounted display.

[0185] In one embodiment, there is a device comprising: a stream module configured to display a first portion of a video presentation on a first external display and a second portion of the video presentation on a second external display; and a head mounted display configured to display a third portion of the video presentation, wherein the third portion is displayed in a space between the first and second external displays. In some embodiments, the device may further include wherein the third portion of the video presentation is aligned with the first and second portions from the perspective of a user of the head mounted display. In some embodiments, the device may further include wherein the first and second external displays are active stereoscopic displays. In some embodiments, the device may further include wherein the first and second external displays have corresponding first and second display frequencies, respectively. In some embodiments, the device may further include wherein the head mounted display is a translucent display and is configured to independently block areas of the translucent display corresponding to the first and second external displays for a first eye at the first and second display frequencies, respectively. In some embodiments, the device may further include a first and a second shutter timing synchronization signal transmitter configured to receive the first and the second display frequencies, respectively, for the first eye and the second eye, alternately blocking the areas of the semi-transparent display corresponding to the first and the second external displays at the first and the second detected display frequencies, respectively. In some embodiments, the device may further include a first and a second shutter timing synchronization signal transmitter configured to receive the first and the second display frequencies, respectively.

[0186] In one embodiment, there is a device comprising: a stream module configured to display a first portion of a video presentation on a first external display and a second portion of the video presentation on a second external display; and a head mounted display configured to display a third portion of the video presentation, wherein the third portion overlaps at least a portion of an area on the first or second external display. In some embodiments, the device may further include wherein the stream module is configured to mask the overlapping area. In some embodiments, the device may further include wherein masking includes flipping pixels within the overlapping area. In some embodiments, the device may further include wherein masking includes displaying a masking color within the overlapping area. In some embodiments, the device may further include wherein the masking color is black.

[0187] In one embodiment, there is a device comprising: a forward-facing camera configured to capture images of portions of first and second display devices within an environment, the first and second display devices displaying first and second portions of content associated with an augmented display (AR) presentation; and a head-mounted display configured to display a third portion of content associated with the AR presentation, the third portion being determined based on the images of portions of the first and second display devices captured using the forward-facing camera. In some embodiments, the device further comprises wherein determining the third portion of content associated with the AR presentation comprises determining a portion of content associated with an AR presentation associated with a portion of a field of view not occupied by the first or second display device.

[0188] In one embodiment, there is a device comprising: a network module configured to connect to an external display; a processor running a mobile application, the mobile application executing instructions, the instructions comprising intercepting a 3D content set and splitting the 3D content set into a first virtual element and a second virtual element; a streaming module configured to display the first virtual element on the external display; and a head-mounted display configured to display the second virtual element. In some embodiments, the device may further include wherein the head-mounted display is a light-permeable augmented display (AR) head-mounted display (HMD). In some embodiments, the device may further include wherein the external display is a television. In some embodiments, the device may further include wherein the external display is a projection screen. In some embodiments, the device may further include a camera configured to detect an occlusion area on the external display. In some embodiments, the device may further include wherein the occlusion area is a shadow projected on a projection screen. In some embodiments, the device may further include wherein the occlusion area is an area on a television screen blocked by an object. In some embodiments, the device may further include rendering content blocked by the occlusion area onto the head-mounted display. In some embodiments, the device may further include wherein the mobile application is further configured to apply a view conversion to at least one of the first and second virtual elements. In some embodiments, the device may further include wherein the view conversion is a model view conversion. In some embodiments, the device may further include wherein the view conversion is based on a viewport setting. In some embodiments, the device may further include wherein the view conversion is determined based on user head tracking. In some embodiments, the device may further include wherein the mobile application is further configured to assign the first and second virtual elements to the external display and the local display, respectively, based on natural eyes. In some embodiments, the device may further include wherein the processor is configured to generate a virtual environment model, the model including a virtual layout of the external display relative to the user. In some embodiments, the device may further include wherein the virtual environment model includes a relative external display distance. In some embodiments, the device may further include wherein the virtual environment model includes position information of one or more objects relative to the external display and the user. In some embodiments, the device may further include a sensor configured to collect sensor data for determining the virtual environment model. In some embodiments, the device may further include wherein the sensor is a device selected from a set consisting of: an optical sensor, a magnetic sensor, a depth sensor, and a sound tracking sensor.In some embodiments, the device may further include a camera configured to detect an overlapping area between the local display and the external display. In some embodiments, the device may further include wherein the head-mounted display is configured to display the content of the external display within the overlapping area. In some embodiments, the device may further include wherein the mobile application renders the overlapping area of ​​the external display using a masking color. In some embodiments, the device may further include a camera configured to detect image artifacts on the external display and / or the head-mounted display. In some embodiments, the device may further include wherein the image artifact is an occlusion. In some embodiments, the device may further include wherein the image artifact is an overlapping portion between the local and external displays. In some embodiments, the device may further include wherein the local display can extend the display area of ​​the external display. In some embodiments, the device may further include wherein the stream module is configured to display a third virtual element on a second external display, the third virtual element being segmented from the 3D content. In some embodiments, the device may further include wherein the head-mounted display is configured to display the second portion between the external display and the second external display. In some embodiments, the device may further include wherein the external display and the second external display are active stereoscopic displays, each having a corresponding display frequency. In some embodiments, the device may further include a camera configured to detect the corresponding display frequencies using a camera attached to the local display. In some embodiments, the device may further include wherein the corresponding display frequencies are different. In some embodiments, the device may further include wherein the head mounted display is configured to block the first eye for the external display at a first frequency in the corresponding display frequencies, and wherein the head mounted display is configured to block the first eye for the second external display at a second frequency in the corresponding display frequencies. In some embodiments, the device may further include wherein the external display is an active stereo display. In some embodiments, the device may further include wherein the head mounted display is configured to block the first eye of the user and display the first and second virtual elements for the second eye of the user. In some embodiments, the device may further include wherein blocking the first eye includes blocking a portion of the head mounted display occupied by the display when viewed from the first eye by the external. In some embodiments, the device may further include instructions for displaying an RGB image for the first eye on the local display.

Claims

1. An automatic calibration method, comprising: detecting a first portion of content displayed on an external display as alternating frames of a stereoscopic projection, the first portion of the content relating to an augmented reality (AR) presentation; In response to detecting the first portion of the content being displayed on the external display as alternating frames of the stereoscopic projection, switching an AR head mounted display (HMD) to an active shutter mode; as well as In the active shutter mode: A shutter effect is generated in synchronization with the alternating frames of the stereoscopic projection while presenting a second portion of the content related to the AR presentation.

2. The method according to claim 1, wherein: The shutter effects include: Alternating between: blocking the left eye's view while content for the right eye is being rendered; and blocking the right eye's view while content for the left eye is being rendered.

3. The method according to claim 1, wherein: The shutter effects include: obscuring a first portion of the user's view corresponding to the view of the external display; and Light is passed through a second portion of the view of the user, the second portion corresponding to a view without the view of the external display.

4. The method according to claim 3, further comprising: In the active shutter mode, Selectively blocking only a first portion of a user's view, wherein the first portion of the user's view corresponds to and further overlaps with a view of the external display based on display position and orientation information about the external display.

5. The method according to claim 2, further comprising: determining a shutter frequency that alternates in the active shutter mode; as well as The alternation between blocking the view for the left eye and blocking the view for the right eye is arranged to occur at the shutter frequency.

6. The method of claim 5, wherein determining the shutter frequency comprises: An image captured by a front-facing camera of the AR head mounted display is analyzed.

7. The method of claim 6, wherein determining the shutter frequency further comprises: One or more of the images are transformed to compensate for a viewing angle between the AR head mounted display and an active stereoscopic display surface.

8. The method of claim 5, wherein determining the shutter frequency comprises: A synchronization signal associated with the external display is detected.

9. The method of claim 1 , wherein detecting a first portion of the content displayed on the external display comprises: An image captured by a front-facing camera of the AR head mounted display is analyzed.

10. The method of claim 1, wherein detecting a first portion of the content displayed on the external display comprises: A synchronization signal associated with an active stereoscopic display surface is detected.

11. The method according to claim 1, wherein: Presenting the second portion of the content related to the AR presentation includes: displaying a left-eye image of the second portion of the content related to the AR presentation in a portion of the AR head-mounted display associated with a left-eye display, while displaying a shutter effect for the right eye in a portion of the AR head-mounted display associated with a right-eye display, and displaying a right-eye image of the second portion of the content related to the AR presentation in a portion of the AR head-mounted display associated with the right-eye display, while displaying a shutter effect for the left eye in a portion of the AR head-mounted display associated with the left-eye display, Wherein, the display of the AR presentation is synchronized with the shutter effect.

12. An automatic calibration device, comprising: An augmented reality (AR) head mounted display configured to switch to an active shutter mode; Front camera; processor; as well as A non-transitory computer-readable medium storing instructions that, when executed by the processor, are operated to cause the device to: detecting a first portion of content displayed on an external display as alternating frames of a stereoscopic projection, the first portion of the content relating to an augmented reality (AR) presentation; In response to detecting the first portion of the content being displayed on the external display as alternating frames of the stereoscopic projection, switching an AR head mounted display (HMD) to an active shutter mode; as well as In the active shutter mode: A shutter effect is generated in synchronization with the alternating frames of the stereoscopic projection while presenting a second portion of the content related to the AR presentation.

13. The apparatus of claim 12, further comprising a spatially controllable shutter layer configured to block the first portion and further configured to allow light to pass through the second portion.

14. The apparatus according to claim 12, wherein: The instructions, when executed by the processor, are operable to cause the device to present the second portion of the content related to the AR presentation, including: displaying a left-eye image of the second portion of the content related to the AR presentation in a portion of the AR head-mounted display associated with a left-eye display, while displaying a shutter effect for the right eye in a portion of the AR head-mounted display associated with a right-eye display, and displaying a right-eye image of the second portion of the content related to the AR presentation in a portion of the AR head-mounted display associated with the right-eye display, while displaying a shutter effect for the left eye in a portion of the AR head-mounted display associated with the left-eye display, Wherein, the display of the AR presentation is synchronized with the shutter effect.

15. The device of claim 13, wherein the spatially controllable shutter layer comprises a liquid crystal display layer.

16. An automatic calibration device comprising: a head mounted display configured to display a first virtual element, the head mounted display having a portion associated with a left eye display and having a portion associated with a right eye display; a front-facing camera associated with the head-mounted display, the front-facing camera configured to capture an image from a portion of an external display; processor; as well as A non-transitory computer-readable medium storing instructions which, when executed by the processor, are operated to cause the device to: responsively determining that the captured images include a second virtual element displayed on the external display as an alternating image of the stereoscopic image sequence, the second virtual element being related to the AR presentation; detecting a frequency of the captured images by analyzing sequential images of the captured images; as well as The head-mounted display is turned off in synchronization with the image of the stereoscopic image sequence at the detected frequency, while also presenting a left-eye image of the first virtual element related to the AR presentation in the portion of the head-mounted display associated with the left-eye display, and presenting a right-eye image of the first virtual element related to the AR presentation in the portion of the head-mounted display associated with the right-eye display.

17. The device of claim 16, wherein the instructions, when executed by the processor, are further operable to further cause the device to connect to the external display.

18. The apparatus of claim 16, further comprising: a network module configured to connect to the external display; as well as A streaming module is configured to display the second virtual element on the external display.

19. The apparatus according to claim 16, wherein: The instructions, when executed by the processor, are further operated to further cause the apparatus to: intercepting a set of three-dimensional (3D) content; and The group of 3D contents is divided into the first virtual element and the second virtual element.

20. The apparatus of claim 19, wherein: The instructions, when executed by the processor, are further operated to further cause the apparatus to: further dividing the set of 3D contents into third virtual elements; and The third virtual element is displayed on a second external display.

21. The apparatus of claim 20, wherein the second external display is an active stereoscopic display having an associated display frequency.

22. The apparatus according to claim 16, wherein the head mounted display is configured to operate as shutter glasses, wherein the head mounted display is configured to occlude the first eye to the external display at a first display frequency, and in, The head mounted display is configured to occlude the first eye to a second external display at a second display frequency.

23. The apparatus of claim 16, wherein when executed by the processor, the instructions are further operated to further cause the apparatus to: The external display is caused to display the second virtual element on the external display.

Images