Correlation effect augmented reality system and method

Abstract

The augmented reality system includes a first scene and a second scene. The augmented reality system further includes a partially reflective surface positioned relative to the first scene, the second scene, and an audience member, the partially reflective surface facilitating viewing of one of the first scene and the second scene through the partially reflective surface and facilitating reflecting the other of the first scene and the second scene toward the audience member as an augmented reality image. A sensor of the augmented reality system is adapted to detect and generate data indicative of characteristics of the first scene, and a correlation effect system is operable to receive the data and adjust the second scene based thereon.

Description

[Technical field]

[0001] The present invention relates to a correlation effect augmented reality system and method. [Background technology]

[0002] Amusement park attractions, such as ride systems, can provide entertainment to guests in a number of ways, including displaying augmented reality (AR) images for guests to view. For example, an amusement park ride can include a display located adjacent to the guest and operate alone or in conjunction with other features to output AR images for guests to view. This can be done to create special effects, such as the illusion of a translucent ghost present in a stage scene.

[0003] One such method of providing AR images is traditionally called the Pepper's Ghost Illusion. This technology for providing AR images is believed to have been developed in the 19th century. The Pepper's Ghost Illusion uses the reflective properties of a translucent or transparent material (e.g., glass, plastic, or polyester foil) to virtually project an image onto a scene for an audience to view. For example, an angled glass plate can be placed in front of a stage, and imagery can be projected onto the plate from outside the audience's line of sight, which then partially reflects the imagery back towards the audience. Thus, the audience sees a scene presented behind the plate and in the audience's line of sight, and simultaneously perceives the reflected imagery. Because lighting allows light behind the plate to be seen through the reflected imagery, this can make the reflected imagery seem like a ghostly apparition. However, lighting techniques can make the reflected imagery appear more three-dimensional by reducing light competition with the background. This type of AR has been available for many years and is currently used in many amusement park attractions. However, it is recognized that audiences are becoming more sophisticated and are able to see through the true nature of this illusion, and therefore it is now recognized that there is a need for improved techniques to make the illusion more realistic and immersive.

[0004] This section is intended to introduce the reader to various aspects that may be related to the various aspects of the present disclosure that are described below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present disclosure. As such, it should be understood that this description is to be read in this light, and not as admissions of prior art. Summary of the Invention [Means for solving the problem]

[0005] Certain embodiments consistent with the disclosed subject matter are summarized below. These embodiments are not intended to limit the scope of the disclosure, but rather these embodiments are intended only to provide a brief summary of certain disclosed embodiments. Indeed, the disclosure may include a variety of forms that may be similar to or different from the embodiments set forth below.

[0006] One embodiment includes an augmented reality system comprising a first scene and a second scene. A partially reflective surface is positioned relative to the first scene, the second scene, and a spectator to facilitate viewing one of the first scene or the second scene through the partially reflective surface and to facilitate reflecting the other of the first scene or the second scene as an augmented reality imagery toward the spectator. A sensor is configured to detect a characteristic of the first scene and generate data indicative of the characteristic. A correlation effect system is configured to receive the data and adjust the second scene based on the data.

[0007] One embodiment includes an augmented reality system comprising a background scene including background scene illumination and an augmented reality scene including augmented reality scene illumination. A partially reflective surface is positioned relative to the background scene, the augmented reality scene, and a spectator to facilitate viewing the background scene through the partially reflective surface and to facilitate reflecting the virtual reality scene toward the spectator as an augmented reality imagery. A sensor is configured to detect lighting characteristics of the background scene and generate data indicative of the lighting characteristics. A correlation effects system is configured to receive the data and adjust the augmented reality scene illumination based on the data.

[0008] One embodiment includes an augmented reality system comprising a background scene including a three-dimensional stage area and background scene lighting configured to adjustably illuminate the three-dimensional stage area. The augmented reality system also includes an augmented reality scene including augmented reality scene lighting configured to provide augmented reality imagery. A partially reflective surface is positioned relative to the background scene, the augmented reality scene, and an audience member to facilitate viewing the background scene through the partially reflective surface and to facilitate reflecting the augmented reality imagery toward the audience member. A sensor is configured to detect a characteristic of one of the background scene or the augmented reality scene and generate data indicative of the characteristic. A correlation effect system is configured to receive the data and adjust an aspect of the other of the background scene or the augmented reality scene based on the data.

[0009] These and other features, aspects, and advantages of the present disclosure will become better understood when the following detailed description is read in conjunction with the accompanying drawings, in which like characters represent like parts throughout the drawings. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram of an augmented reality (AR) system positioned for viewing by an audience, according to one aspect of the present disclosure. [Diagram 2] FIG. 1 is a schematic perspective view of an embodiment of an AR system in which a projection source operates to adjust the lighting of a stage relative to a virtual light-emitting object and to adjust the positioning of the presentation of the virtual light-emitting object relative to a physical object, according to an embodiment of the present disclosure. [Diagram 3] FIG. 1 is a schematic perspective view of an embodiment of an AR system in which a lighting system is disposed proximate a boundary of a partially reflective surface and operable to adjust the lighting of the stage relative to the virtual light-emitting object, in accordance with an embodiment of the present disclosure; [Figure 4] FIG. 1 is a side schematic diagram of an AR system in which an image source operates to display or mimic a reflection of a virtual object on an item in a background scene, according to one embodiment of the disclosure. [Diagram 5]FIG. 1 is a side schematic diagram of an AR system in which stage lighting operates to mimic the reflection of a virtual image onto an object in a background scene, according to an embodiment of the present disclosure. [Figure 6] FIG. 1 is a side schematic diagram of an AR system in which a lighting array operates to mimic the reflection of a virtual image onto an object in a background scene, according to an embodiment of the present disclosure. [Figure 7] FIG. 1 is a schematic perspective view of an AR system in which lighting of multiple three-dimensional scenes involved in an augmented reality display is monitored by a lighting detector to facilitate adjusting lighting of one or more of the multiple scenes, according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] One or more specific embodiments are described below. In order to provide a concise description of these embodiments, not all features of an actual embodiment are described herein. As with any industrial design or engineering project, it will be appreciated that the development of any such actual embodiment will require numerous implementation-specific decisions to be made in order to achieve the specific goals of the developers, including compliance with system-related and business-related constraints that may vary between embodiments. It will be further appreciated that such development efforts may be complex and time-consuming, but will nevertheless be a routine undertaking of design, fabrication, and manufacture for those of ordinary skill in the art having the benefit of this disclosure.

[0012] When describing elements of various embodiments of the disclosure, the articles "a," "an," and "the" are intended to mean that there are one or more of the element. The terms "comprising," "including," and "having" are intended to be inclusive and mean that there may be additional elements other than the listed elements. In addition, it should be understood that references to "one embodiment" or "one embodiment" of the present disclosure are not intended to be interpreted as excluding the existence of additional embodiments that incorporate the referenced features.

[0013] According to this embodiment, the augmented reality (AR) system may include an AR imagery source (e.g., a projector or an illuminated scene) that operates to project AR imagery onto a partially reflective surface (e.g., glass, plastic, or polyester foil) that is disposed between an audience (e.g., a viewer) and a background scene, such as a stage, an electronic display, a screen, or a set. The AR imagery source and background may include a combination of the referenced features. The AR imagery may include any of a variety of images that are projected onto a screen that reflects the AR imagery toward the audience to make it appear to exist against the background scene. In some embodiments, the AR imagery may include characters (e.g., people), ghosts, objects (e.g., desks), text, virtual light-emitting objects (e.g., flames), and the like. In one embodiment, the partially reflective surface may include a semi-transparent mirror that partially reflects the AR imagery from the source toward the audience. In other words, the partially reflective surface reflects the AR scene toward the audience as an AR imagery. A spectator may view the AR imagery reflected from the semi-transparent mirror as being superimposed on a background scene, which may include physical props, electronic displays (e.g., a projector screen or liquid crystal display), or both. In this manner, the AR imagery may appear to the spectator as if it were located in close proximity to features of the background scene. To enhance this effect, the AR imagery may include three-dimensional (3D) imagery, which may be described as a two-dimensional imagery that appears three-dimensional when viewed through appropriate lenses (e.g., polarized or tinted lenses of 3D glasses).

[0014] A semi-transparent mirror, representative of various other partially reflective surfaces (e.g., glass, mesh), can be positioned at an angle to the audience and the AR imagery source to produce what has been described as the Pepper's Ghost effect. The Pepper's Ghost effect involves reflecting the AR imagery such that the viewer can simultaneously view the AR imagery of the partially reflective surface in combination with features located on the other side of the partially reflective surface. Specifically, the Pepper's Ghost effect allows the AR imagery to be seen to overlap an object or image located behind the partially reflective surface. For example, a human figure presented as an AR imagery and reflected by the partially reflective surface may appear to be sitting in an actual chair located on a stage behind the partially reflective surface, relative to the audience's perspective.

[0015] The scene behind the partially reflective surface that is seen directly by the audience may be referred to as the background scene. The scene that provides the AR imagery seen by the audience after reflection from the partially reflective surface may be referred to as the AR scene. The background scene and the AR scene may include physical components (e.g., stage props, actors, structures) and / or electronic displays (e.g., projectors, liquid crystal displays, lighting panels). For example, the background screen may include a screen (e.g., a movie screen) onto which an image is projected, while the AR scene may include illuminated props and actors arranged so that light reflects off the partially reflective surface. In this example, the AR scene may be located in a space below the stage that cannot be seen directly by the audience, but is arranged to direct light towards the partially reflective surface to reflect towards the audience.

[0016] In another example, a combination of opposite or different features for each of the background scene and the AR scene can be used. Since the imagery provided by the background scene and the AR scene are combined to provide the Pepper's Ghost effect, the relative lighting of each scene will affect the nature of the effect on how this is seen by the audience. For example, if the background scene is much brighter than the AR scene, the audience will barely be able to perceive the AR imagery reflected back at them. However, if the background scene is much darker than the AR scene, the AR imagery will dominate and the audience may not see the background scene. Furthermore, since the AR imagery is presented to the audience as seen by reflection, the features of the displayed AR imagery (e.g., a character moving around) will not illuminate the background scene in a way that the audience would expect the actual feature to do. For example, the AR imagery of a particular feature (e.g., a person, a lamp, or a car) will not adequately reflect that feature in an actual mirror in the background scene. Similarly, AR imagery that may typically resemble luminous features (eg, a lamp, flashlight, flame, or other luminous object) will not contribute light and corresponding shadows to the background scene.

[0017] The present embodiment includes a monitoring system that tracks lighting in one or both of the background scene and the AR scene to coordinate lighting effects to achieve a desired overlapping perception of the combined imagery from the background scene and the AR scene. Additionally, the present embodiment includes lighting features that mimic the expected lighting effects of the features provided by the AR imagery so that the background scene is affected in an expected manner by the audience, which enhances the immersiveness of the display. The present embodiment also includes a tracking system that operates to track features in the background scene to provide the desired AR imagery correlated thereto. For example, a physical lantern or the like can be tracked to move around in the background scene and provide positioning information for the lantern. This positioning information can be used to correlate to the position of the lantern so that the AR imagery is provided at a location on the partially reflective surface and the combination of the background scene and the AR imagery makes the lantern appear to the audience as if it is lit and glowing. Specifically, a light-emitting object such as a flame can be projected onto a partially reflective surface or screen in a location that will cause the audience to see the light-emitting object when tracking the physical location of the lantern. Additionally, the lighting of the background scene can be adjusted to cast shadows that appear to result from the false light of, for example, a lantern.

[0018] 1 is a schematic diagram of an AR system 10 according to one embodiment of the present disclosure. The AR system 10 includes a background scene 12, an AR scene 14, and a partially reflective surface 16 (which may also be referred to as a partially reflective layer or screen 16) disposed therebetween. An audience member 18 is positioned such that an AR imagery 20, including light reflected from the AR scene 14, is reflected from the partially reflective surface 16 toward the audience member 18 in a manner that makes the AR imagery 20 appear to be disposed in the background scene 12, as indicated by a dashed virtual image 21. Furthermore, the background scene 12 is positioned such that the audience member 18 can view it directly through the partially reflective surface 16, which is also partially transparent. The background scene 12 and the AR scene 14 may be defined by any combination of physical props, real actors, electronically generated imagery, and the like. For example, the background scene 12 may include a stage with physical features (e.g., chairs, tables, and live actors), while the AR scene 14 may include a projector that directs the AR imagery 20 onto the partially reflective surface 16. In other embodiments, a different combination of physical features and electronically generated imagery may be used, including both the background scene 12 and each of the AR scenes 14. The partially reflective surface 16 may include special foil, glass, plastic, a partial mirror, or the like, operable to allow the audience 18 to both see through it and observe the imagery reflected therefrom, in appropriate lighting conditions. The AR system 10 also includes a correlation effect system 22, which may be a control device designed to work with various sensors 24, 26 and lighting systems 28, 30 to provide the desired effects according to one embodiment. Furthermore, in one embodiment according to the present disclosure, the correlation effect system 22 may actually include the sensors 24, 26 and lighting systems 28, 30. In some embodiments, only one sensor 24 , 26 may be used to monitor one or both of the background scene 12 and the AR scene 14 .

[0019] Specifically, AR system 10 includes a correlation effects system 22 having features that can correlate aspects of AR scene 14 with aspects of background scene 12 in a manner that enhances the immersion of audience member 18 in the AR illusion being provided by AR system 10. AR system 10 can also include features such as actuators 19 that facilitate manipulation (e.g., repositioning) of partially reflective surface 16 to achieve particular correlation results.

[0020] The AR system 10 and / or the correlated effect system 22 may include one or more controllers 32, processors 34, and / or memories 36 for performing various functions (e.g., directing the operation of other system functions). The one or more memories 36 may include random access memory (RAM), read only memory (ROM), rewriteable non-volatile memory such as flash memory, hard drives, optical disks, and / or other types of memory. The one or more processors 34 of the AR system 10 may include one or more general purpose microprocessors, one or more application specific integrated circuits (ASICs), and / or one or more field programmable gate arrays (FPGAs). The one or more controllers 36 may include a programmable logic controller (PLC) or other computer-based controller. These features (e.g., the controller 36, the processor 34, and the memory 32) may be components of the correlated effect system 22 or may be separate features. Additionally, as can be appreciated, these features may operate using stored instructions (eg, code) that, when executed, cause an action (eg, dimming a light or operating an actuator).

[0021] In one embodiment, the AR system 10 can operate to correlate lighting characteristics between the AR scene 14 and the background scene 12 to improve coordination between the AR image 20 and directly viewable aspects of the background scene 12. For example, it may be desirable to adjust the AR image 20 to have similar coloring as the background scene 12. This can be accomplished by observing lighting characteristics in either the AR scene 14 or the background scene 12 via the sensors 24, 26 and making corresponding lighting changes in the other of the AR scene 14 or the background scene 12 via the lighting systems 28, 30. One or both of the sensors 24, 26 can be included and / or used by a particular system. Examples of lighting characteristics that can be detected and adjusted include illuminance, position / orientation, brightness, color, temperature, contrast, and quality. Characteristics of a particular scene can be altered based on different characteristics of the other scene. For example, the correlation effect system 22 can operate to adjust the brightness of the background scene 12 based on the contrast of the AR scene 14. In this embodiment, lighting can be detected and adjusted in either (or both) a physical scene (e.g., a theater stage) or a virtual scene (e.g., a scene provided by an electronic display such as a liquid crystal display or projector). For example, in an embodiment in which one or both of the background scene 12 and the AR scene 14 include an electronic display, the sensors 24, 26 can detect the display settings (e.g., color settings) and the lighting systems 28, 30 can control the display settings based on the output from the sensors 24, 26. In addition to changing the lighting, in this embodiment, the physical positioning of the partially reflective surface 16 can be adjusted. For example, the angle of the partially reflective surface 16 relative to the background scene 12, the AR scene 14, and the audience 18 can be adjusted with the actuators 19 based on instructions from the correlation effects system 22 to achieve a desired viewing result.

[0022] 2 is a schematic perspective view of an embodiment of an AR system 10 in which a projection source 60 operates to adjust the illumination of a stage 62 relative to a virtual light-emitting object 64 and to adjust the positioning of the virtual light-emitting object 64 relative to a physical object 66, according to an embodiment of the present disclosure. In the illustrated embodiment of FIG. 2, the background scene 12 includes a stage 62 and a physical object 66 that is a moving background object. The physical object 66 can be any number of movable objects. For example, the moving object can be a robot figure, a torch, a sword, an actor, or any other trackable object. However, in the illustrated embodiment, the physical object is a prop lantern that does not actually emit light by itself.

[0023] A tracking system 70 is included to facilitate tracking of physical movements within the background scene 12. For example, the exemplary tracking system 70 includes sensors (e.g., cameras) 72, a processor 74, and a memory 76 that cooperate to spatially track physical objects 66 and / or actors 80. By tracking these physical aspects (e.g., physical objects and actors 80) within the background scene 12, appropriate lighting adjustments can be made to the lighting system 60 to make the augmented reality presentation of the AR system 10 more immersive. For example, the AR imagery 20 can be presented by a projector 82 of the lighting system 60 on the partially reflective surface 16 such that the positioning of the AR imagery 20 correlates to the positioning of the physical object 66 relative to the line of sight 84 of the audience 18. The exemplary projector 82 is representative of a wide range of image generating devices that can be utilized to achieve a desired effect. For example, the projector 82 may represent a flat screen television that generates an image of a flame, which is then reflected by the partially reflective surface 16 to create the illusion that the flame is actually located on or within the physical object 66. In other embodiments, the projector 82 may include a screen that receives an image by front or rear projection. As with a flat screen television, the image presented on the screen may be reflected by the partially reflective surface 16 to create the desired illusion. The projector 82 may represent any device that would operate to provide the desired image. Additionally, in some embodiments, the projector 82 may be replaced by a physical feature (e.g., a lit candle) that can be moved around the space by an actuator to provide a reflected light that correlates to the desired positioning of the illusion image relative to the stage 62.

[0024] In view of the above, the combined action of the tracking system 77 and the lighting system 60 allows the AR system 10 to make the physical object 66 (a non-functional lantern prop in the illustrated embodiment) appear to be ignited and emitting a flame since the AR imagery 20 represents a flame and is appropriately positioned. Additionally, the stage lights 86 of the lighting system 60 can be operated to cast a shadow 88 that is correlated to the positioning of the AR imagery 20. In the illustrated embodiment, for example, the shadow 88 can be cast by operating one or more specific lamps 90 to make it appear as if the AR imagery 20 is causing the shadow 88. A correlation effect system 22 (e.g., an automated controller) can be used for this purpose by receiving input from the tracking system 70 and providing an output to the lighting system 60. The correlation effect system 22 can also use known positioning information (e.g., a predefined path of the physical object 66 and / or the AR imagery 20) to control the lighting system 60 to provide correlation effects and immersion. For example, a lantern prop can be moved across the stage 62 using automated control, and the traversal pattern can be coordinated with the projector 82 to provide a desired effect. Such an embodiment would improve efficiency and eliminate processing time and / or costs associated with tracking physical features (e.g., physical objects 66).

[0025] The physical objects 66 and / or actors 80 may be tracked by the tracking system 70 due to one or more characteristics (e.g., shape, color, facial features, or RFID) that enable detection or recognition by the tracking system 70. For example, in one embodiment, the tracking system 70 is programmed to identify a particular shape of the physical object 66 and track the movement of the physical object 66 such that position data (e.g., current location) can be readily identified essentially in real time. To accomplish this, the memory 76 and processor 74 may execute programming (e.g., object recognition or facial recognition programs) to interpret data and / or extrapolate from data obtained from the sensor 72. The position data obtained by the tracking system may be used to identify an appropriate location for use in controlling the lighting system 60.

[0026] FIG. 3 is a schematic perspective view of an embodiment of the AR system 10, in which the stage lights 86 of the lighting system 60 are positioned proximate the boundary 102 of the partially reflective surface 16 and are operable to adjust the illumination of the stage 62 of the background scene 12 with respect to the AR imagery 20, which in the illustrated embodiment is presented as a virtual light-emitting object, specifically a virtual fire. The audience 18 can perceive the virtual fire as being present in the center of the stage 62 and in front of the physical object 66, as generally indicated by the dashed virtual image 21. The embodiment illustrated in FIG. 3 is arranged to operate in a similar manner as described above with respect to the embodiment of the AR system 10 described with respect to FIG. 2. That is, the AR imagery 20, which is a virtual light-emitting object, is projected by the projector 82 onto the partially reflective surface 16 so as to appear to be located within the three-dimensional space of the background scene 12 and to interact with the background object 66 to generate a shadow 88. However, in this embodiment, the stage lights 86 are mounted on or at the proximate edge of the partially reflective surface 16. This positioning allows various lamps 90 (e.g., LED lights) of the stage lighting 86 to be activated based on the positioning of the AR imagery 20 to create a realistic effect on the background scene 12, which in the illustrated embodiment is a shadow 88 that appears to be caused by a virtual flame represented as the virtual image 21.

[0027] As can be appreciated, if the AR imagery 20 represents a flame moving around the partially reflective surface 16 to mimic a flickering flame moving around a three-dimensional scene, the stage lights 86 can move around to cast different shadows based on the mimicked positional relationship and can flicker to provide proper shadow formation. In some embodiments, the lamps 90 themselves do not move, but various lamps partially or completely positioned around the partially reflective surface 16 can be actuated to achieve an effect similar to the embodiment of FIG. 3. Using such techniques and systems, even observers of the audience 18 who have seen multiple Pepper's Ghost illusions will be more immersed in the effect. It should be noted that the positioning of the various lamps 90 can also be coordinated in combination with actuation of the various lamps 90 at various positions. For example, if the various lamps 90 are coupled to the boundary 102, the lamps 90 can be moved by moving the entire partially reflective surface 16 using the actuators 27 to achieve the desired viewing result. In other embodiments, separate actuators can be used for each lamp 90.

[0028] FIG. 4 is a schematic side view of the AR system 10, in which the projection source 112 (e.g., a movie projector, a television screen, a projection screen) operates to display or mimic a reflected light by displaying an image 116 of a virtual object 118 presented by the AR imagery 20 on a surface 124 in the background scene 12, which in the illustrated embodiment includes a faux mirror disposed on the stage 62. In other embodiments, the surface 124 may represent other items, such as shiny furniture, glass windows, dinnerware sets, metal panels, and the like. Specifically, in the illustrated embodiment of FIG. 4, the background scene 12 of the AR system 10 includes a faux mirror as the surface 124, which is actually a projection surface for presenting the image 116 to the audience. The virtual object 118 would not actually be reflected by a real mirror (or other shiny surface) in the background scene 12, since it would only appear to be disposed as a virtual image 21. Therefore, the present embodiment can mimic real-world reflections, thereby enhancing the audience's sense of immersion.

[0029] In reality, the fake mirror is a prop that mimics the reflective properties of a mirror by displaying an image received from a projection source 112, which in the illustrated embodiment is placed in front of a surface 124. In other embodiments, rear projection can be utilized. In still other embodiments, the projection source 112 and the surface 124 can be combined features such as a flat screen television. To make the background scene 12 appear realistic and enhance the audience's 18 immersion in the perceived subject, the correlation effects system 22 coordinates the motion of the projection source 112 and the projector 82. For example, the movement of the virtual object 118 presented by the projector 82 is coordinated with the image 116 presented by the projection source to give the illusion that the surface 124 is reflecting the virtual object 118. In some embodiments, this can include modifying data from the virtual object 118 to present the image 116. For example, depending on the surface being mimicked (e.g., a slightly glossy paint on a piece of furniture), the image 116 may need to appear blurry. It should be noted that projection source 112 and projector 82 may represent any number of image presentation characteristics. For example, any of these characteristics may represent a display screen, an illuminated stage, a conventional projector, etc.

[0030] FIG. 5 is a side schematic view of an AR system 10 according to an embodiment of the present disclosure, in which stage lighting 86 operates to mimic the reflection of a virtual image 21 on items in a background scene 12. In an illustrated embodiment, a physical reflective object 152 (e.g., a shiny ball, an hourglass, a shiny painting, a shiny piece of furniture) is placed in the background scene 12. In the AR scene 14, a physical lighted object 154 (e.g., a projector, a television screen, a candle, an object under light) is placed such that an imagery of the physical lighted object 154 reflects off the partially reflective surface 16 toward the audience 18 in such a way as to create the illusion that the physical lighted object 154 is actually present in the background scene 12 and placed as the virtual image 21. A sensor 24 (e.g., a camera) operates to detect lighting conditions that would be present on the surface of the physical lighted object 152 if the physical lighted object 154 were actually placed in the background scene 12 as the virtual image 21. In one embodiment, this can be based on the sensor 24 viewing a physical lighted object 154 from a position that would correspond to a physical reflective object 152 if the virtual image 21 were actually present in the background scene 12. For example, a shiny rubber ball is placed at a 45 degree angle relative to where the virtual image 21 would appear to be in the background scene 12, so the sensor can be placed at a 45 degree angle relative to the physical lighted object 154 in the AR scene. Furthermore, such information can be calculated using modeling techniques, and an approximation may be sufficient to provide the desired illusion. Such techniques can be applied when the virtual image 21 is based on an image displayed by a screen (e.g., a television screen). Thus, the data obtained from the sensor 24 can then be used to generate light from the stage lights 86 that will be projected or reflected in the background scene 12 in an appropriate manner. For example, in the illustrated embodiment of FIG. 5, a center set of the stage lights 86 has been selected to be active based on the data to create the desired reflective effect.The resulting lighting effect will match both the position and quality of light that would strike the physical reflective object 152 if it were positioned as perceived in relation to the virtual image 21, so that the physical reflective object 152 responds to the projected light in an accurate and reliable manner.

[0031] FIG. 6 is a side schematic view of an AR system 10 according to an embodiment of the present disclosure, in which an illumination array 162 (e.g., a planar array of collimated light, a light field display, a panel of lasers) operates to mimic the reflection of a virtual image 21 onto an object in a background scene 12. The AR system 10 of FIG. 6 is configured in a manner similar to the embodiment shown in FIG. 5. However, the stage lighting 86 in the example embodiment of FIG. 6 includes an illumination array 162. The illumination array 162 can include any of a variety of focused illumination systems that can direct points of light to specific locations without the associated light cone expanding substantially, causing blurring, etc. Using data obtained from the sensor 24, in a manner similar to that described with respect to FIG. 5, the illumination array 162 can direct light in a desired pattern onto the back surface of the partially reflective surface 16 such that the light is appropriately reflected onto the background scene 12. For example, in embodiments in which lighting array 162 includes a light field display, certain light rays may be directed to converge at the perceived location of virtual image 21 (based on data from sensor 24 indicating the perceived location) and then diverge therefrom in a manner similar to that which would occur if virtual image 21 actually existed. This may facilitate a highly accurate illusion of a specular reflection of virtual image 21 on physical reflective object 152.

[0032] 7 is a schematic perspective view of an AR system 10 according to an embodiment of the present disclosure, in which the illumination of a plurality of three-dimensional scenes associated with an augmented reality display is monitored by an illumination detector to facilitate adjusting the illumination of one or more of the plurality of scenes. In an exemplary embodiment, both the background scene 12 and the AR scene 14 are three-dimensional stage areas. The background scene 12 is illuminated by background scene illumination 202, which is monitored by a background scene sensor 204 (e.g., a camera). The AR scene is illuminated by AR scene illumination 212, which is monitored by an AR scene sensor 214. The props 218 of the AR scene 14 reflect light from the AR scene illumination 212 toward the partially reflective surface 16, where the light is directed toward the audience 18. The combined illumination of the background scene 12 and the AR scene 14 can be controlled by a correlation effects system 22, which in an exemplary embodiment is in wireless communication with the illumination system 60 and associated sensors 204, 214.

[0033] In one embodiment, the correlation effect system 22 can control the lighting system 60 so that the AR imagery 20 appears translucent to the audience 18, thereby providing a ghostly effect to the AR imagery 20. Such control of the correlation effect system 22 can be based on sensor data from the background scene sensor 204 and / or the AR scene sensor 214. Also, lighting adjustments of the background scene lighting 202 and the AR scene lighting 212 can be made by the correlation effect system 22 based on sensor data from the background scene sensor 204 and / or the AR scene sensor 214, and other adjustments can be made to the viewing experience. For example, the AR imagery 20 can be made to appear more stereoscopic, or various adjustments can be made to better correlate the background scene 12 and the AR scene 14 (e.g., colorization, contrast adjustments). It should be noted that while FIG. 7 illustrates a particular embodiment in which the background scene 12 and the AR scene 14 are provided by a three-dimensional stage setting, multiple different features can be combined to form the background scene 12 and the AR scene 14. For example, the projector 82 may be part of the lighting system 60 and may operate to facilitate the provision of additional AR imagery 20. As another example, the display 222 (e.g., an LCD screen or projection screen) may cooperate with the physical objects 66 of the stage 62 to facilitate the provision of the background scene 12. Each of these features (e.g., the display 222, the projector 82, the lighting 202, 212) may be coordinated by the interrelated effects system 22 to provide an immersive presentation of augmented reality to the audience by coordinating the effects to provide the image results expected by the present embodiment.

[0034] Various aspects of the present disclosure are illustrated by Figures 1-5 and their corresponding descriptions. For example, Figure 4 is an example of an AR system 10 providing a false reflection of an AR imagery 20 in a background scene 12, while Figure 5 provides an example of a particular arrangement of a background scene 12 and a three-dimensional stage area as an AR scene 12 and the scenes used to provide coordinated lighting in an AR presentation. These are particular aspects of the embodiments provided to convey the broader features covered by the present disclosure, and can be combined in various ways to achieve different combined results. For example, false reflections as described with respect to Figure 5 can be provided in conjunction with various embodiments of the present disclosure, for example, in conjunction with the lighting arrangement of Figure 3. Indeed, the present disclosure covers all combinations of the disclosed features of the AR system 10.

[0035] While only certain features of the disclosed embodiments have been illustrated and described herein, many modifications and changes will occur to those skilled in the art, and it is, therefore, to be understood that the appended claims are intended to cover all such modifications and changes which fall within the true spirit of the present disclosure.

[0036] The technology presented and claimed herein refers to and applies to tangible objects and specific examples of a practical nature that positively improve the art and are therefore not abstract, intangible, or purely theoretical. Moreover, if any claim appended at the end of this specification contains one or more elements designated as "means for [performing] ... [function]" or "step for [performing] ... [function]," such elements are to be construed pursuant to 35 U.S.C. 112(f). On the other hand, for any claim containing an element designated in any other manner, such elements are not to be construed pursuant to 35 U.S.C. 112(f). [Explanation of symbols]

[0037] 10. AR System 12 Background Scenes 14 AR Scenes 16 Partially reflective surface 18. Spectators 19 Actuators 20 AR Imagery 21 Virtual Images 22 Correlation Effect System 24 Sensors 26 Sensors 28 Lighting System 30 Lighting System

Claims

1. 1. An augmented reality system, comprising: The first scene, The second scene, a partially reflective surface positioned relative to the first scene, the second scene, and an audience member, the partially reflective surface facilitating viewing of the first scene through the partially reflective surface and facilitating reflecting the second scene as an augmented reality imagery towards the audience member; a sensor configured to detect at least one characteristic of the first scene, including a positioning of a physical object in the first scene, and generate data indicative of the at least one characteristic; a correlation effects system including one or more processors and a memory accessible to said one or more processors; the memory storing instructions that, when executed by the one or more processors, The augmented reality system receives the data and causes the one or more processors to adjust the second scene based on the data by commanding one or more actuators to reposition the partially reflective surface so that the augmented reality imagery is positioned relative to the positioning of the physical object.

2. 2. The augmented reality system of claim 1, wherein the first scene comprises a first illumination and the second scene comprises a second illumination, the at least one characteristic comprises an illumination characteristic, and the correlated effects system is configured to adjust the second illumination based on the illumination characteristic of the first scene.

3. The augmented reality system of claim 2 , wherein the first scene is a background scene and the first illumination is background scene illumination, and the second scene is an augmented reality scene and the second illumination is augmented reality illumination.

4. The augmented reality system of claim 2 , wherein the first scene is an augmented reality scene and the first illumination is augmented reality illumination, the second scene is a background scene and the second illumination is background scene illumination.

5. 2. The augmented reality system of claim 1, wherein the second scene comprises an image generator, and the correlation effects system is configured to adjust color, contrast, or a combination thereof, of an image provided by the image generator based on the data, the data being indicative of lighting of the first scene.

6. The augmented reality system of claim 1 , comprising an image generator configured to mimic a reflection of the augmented reality imagery in the first scene based on the data.

7. The augmented reality system of claim 1 , wherein the first scene includes a three-dimensional stage apparatus and the sensor comprises a camera configured to detect lighting characteristics of the three-dimensional stage apparatus.

8. The augmented reality system of claim 1 , wherein the first scene comprises an electronic display.

9. The augmented reality system of claim 1 , wherein the sensor is one of a plurality of sensors configured to monitor one or both of the first scene and the second scene for motion and lighting characteristics.

10. 2. The augmented reality system of claim 1, wherein the correlation effects system is configured to adjust lighting of the first scene to cast shadows in the first scene based on a position of a virtual luminous object of the augmented reality imagery on the partially reflective surface.

11. 10. The augmented reality system of claim 1, wherein the first scene, the second scene, or both, include one or more projectors, one or more liquid crystal displays, one or more lighting panels, or any combination thereof.

12. 1. An augmented reality system, comprising: a background scene including background scene lighting; an augmented reality scene including an augmented reality scene illuminator; a partially reflective surface positioned relative to the background scene, the augmented reality scene, and an audience member, the partially reflective surface facilitating viewing of the background scene through the partially reflective surface and facilitating reflecting the augmented reality scene toward the audience member as an augmented reality imagery; a sensor configured to detect lighting characteristics of the background scene, a position of a physical object in the background scene, or both, and generate data indicative of the lighting characteristics, the position of the physical object, or both; a correlation effects system including one or more processors and a memory accessible to said one or more processors; the memory storing instructions that, when executed by the one or more processors, and receiving the data and causing the one or more processors to adjust the augmented reality scene based on the data by commanding one or more actuators to reposition the partially reflective surface to present a virtual object that lights up in a position corresponding to the positioning of the physical object according to the lighting characteristics.

13. The augmented reality system of claim 12 , wherein the background scene comprises a theatre stage and the partially reflective surface is comprised of a partially mirrored foil.

14. The augmented reality system of claim 12 comprising a display within the background scene configured to mimic a reflection of the augmented reality imagery.

15. 13. The augmented reality system of claim 12, further comprising an additional sensor configured to detect additional lighting characteristics of the augmented reality scene and generate additional data indicative of the additional lighting characteristics, the correlation effects system configured to receive the additional data and adjust the lighting of the background scene based on the additional data.

16. The augmented reality system of claim 15 , wherein the augmented reality scene comprises a three-dimensional staging area.

17. The augmented reality system of claim 12 , wherein the augmented reality scene illumination includes one or more projectors, one or more liquid crystal displays, one or more lighting panels, or any combination thereof.

18. 1. An augmented reality system, comprising: a background scene including a three-dimensional stage area and background scene lighting configured to adjustably illuminate the three-dimensional stage area; an augmented reality scene including an augmented reality scene illuminator configured to provide an augmented reality image; a partially reflective surface positioned relative to the background scene, the augmented reality scene, and an audience member, the partially reflective surface facilitating viewing of the background scene through the partially reflective surface and facilitating reflecting the augmented reality imagery toward the audience member; a sensor configured to detect a characteristic of the background scene, the sensor also configured to generate data indicative of the characteristic; a correlation effects system including one or more processors and a memory accessible to said one or more processors; Equipped with the characteristics include a positioning of a physical object within the background scene; The memory stores instructions that, when executed by the one or more processors, and receiving the data and causing the one or more processors to adjust the augmented reality scene based on the data by commanding one or more actuators to reposition the partially reflective surface to present a virtual object at a position corresponding to the positioning of the physical object.

19. 20. The augmented reality system of claim 18, wherein the augmented reality scene comprises a display screen, the augmented reality scene illumination configures illumination of the display screen, the sensor is configured to generate the data indicative of lighting characteristics of the background scene, and the correlation effects system is configured to control the illumination of the display screen to provide characteristics of the augmented reality imagery that correlate to the data.

20. 20. The augmented reality system of claim 18, wherein the augmented reality scene illumination includes one or more projectors, one or more liquid crystal displays, one or more lighting panels, or any combination thereof.

21. 1. An augmented reality system, comprising: The first scene, a second scene presented by the projector; and a partially reflective surface positioned relative to the first scene, the projector, and an audience member, the partially reflective surface facilitating viewing of the first scene through the partially reflective surface and facilitating reflecting the second scene towards the audience member as an augmented reality imagery; receiving data indicative of a characteristic of the first scene; adjusting the reflection of the second scene by transitioning a relative positioning of the projector and the partially reflective surface based on the characteristic. A control device configured as above, An augmented reality system comprising:

22. The augmented reality system of claim 21 , wherein the characteristics include lighting characteristics of the first scene.

23. 22. The augmented reality system of claim 21, wherein the characteristics include lighting characteristics of the first scene, and the controller is configured to adjust lighting of the second scene based on the lighting characteristics of the first scene.

24. 22. The augmented reality system of claim 21, wherein the projector includes a light-emitting diode (LED) display, and the controller is configured to adjust color, contrast, brightness, or some combination thereof, of an image presented by the LED display based on lighting characteristics of the first scene.

25. 22. The augmented reality system of claim 21, wherein the control device is configured to receive the data from one or more sensors configured to detect the characteristics as a position of an object in the first scene, a lighting characteristic of the first scene, or both.

26. The augmented reality system of claim 21 , wherein the first scene comprises a three-dimensional stage setting.

27. 22. The augmented reality system of claim 21, wherein the controller is configured to adjust lighting of the first scene to cast shadows on the first scene based on positioning of a virtual light-emitting object in the augmented reality imagery.

28. 22. The augmented reality system of claim 21, comprising an image generator configured to mimic a reflection of the augmented reality imagery in the first scene by displaying an image of a virtual object presented by the augmented reality imagery on a surface positioned within the first scene.

29. 22. The augmented reality system of claim 21, comprising a first camera configured to detect a first illumination in the first scene and a second camera configured to detect a second illumination in the second scene, and the control device configured to adjust the augmented reality imagery based on the first illumination and the second illumination.

30. 22. The augmented reality system of claim 21, wherein the first scene includes one or more projectors, one or more liquid crystal displays, one or more lighting panels, or any combination thereof.

31. 22. The augmented reality system of claim 21 , wherein the controller is configured to command one or more actuators to reposition the partially reflective surface such that the augmented reality imagery is positioned relative to a detected position of an object in the first scene.

32. 1. A system comprising: Background scene and an augmented reality scene displayed via a projector; a partially reflective surface positioned relative to the background scene, the projector, and an audience member, the partially reflective surface facilitating viewing of the background scene through the partially reflective surface and facilitating reflecting the augmented reality scene toward the audience member as an augmented reality imagery; receiving data including lighting characteristics of the background scene; adjusting the augmented reality scene by transitioning a relative positioning between the projector and the partially reflective surface to present a virtual object in the augmented reality imagery that is illuminated according to the light characteristics. one or more processors configured to A system comprising:

33. The system of claim 32 , wherein the data includes a position of a physical object in the background scene.

34. 34. The system of claim 33, wherein the one or more processors adjust the augmented reality scene by transitioning the relative positioning between the projector and the partially reflective surface to present the virtual object of the augmented reality imagery at a position corresponding to a positioning of the physical object.

35. 33. The system of claim 32, wherein the background scene comprises one or more electronic displays, one or more projectors, one or more lighting panels, or any combination thereof.

36. 33. The system of claim 32, comprising a camera configured to detect the light characteristics of the background scene.

37. 33. The system of claim 32, wherein the partially reflective surface comprises a partially mirroring foil.

38. receiving, via one or more processors, a first data set including a background scene and augmented reality lighting of the augmented reality scene positioned relative to a partially reflective surface, the partially reflective surface configured to facilitate viewing of the background scene through the partially reflective surface; receiving, via the one or more processors, a second data set comprising background lighting of the background scene and a positioning of a physical object in the background scene; adjusting, via the one or more processors, the augmented reality scene by shifting a relative positioning between a projector of the augmented reality scene and the partially reflective surface to present a virtual object that lights up in a position corresponding to a positioning of the physical object according to the background lighting; A method comprising:

39. 40. The method of claim 38, comprising generating, via the one or more processors, an augmented reality imagery based on the augmented reality lighting and the background lighting to present the virtual object.

40. 40. The method of claim 38, comprising, via the one or more processors, adjusting the background lighting to cast shadows in the background scene based on positioning of the virtual object relative to the partially reflective surface.

Images