Correlated effects augmented reality system and method

Abstract

An augmented reality system includes a first scene and a second scene. Also, the augmented reality system includes a partially reflective surface positioned relative to the first scene, the second scene, and a viewer to facilitate viewing one of the first scene or the second scene through the partially reflective surface and to facilitate reflecting the other of the first scene or the second scene as an augmented reality image toward the viewer. A sensor of the augmented reality system is designed to detect and generate data indicative of a characteristic of the first scene, and a related effects system is operable to receive the data and adjust the second scene based on the data.

Description

Background Technology

[0001] Amusement park attractions (such as ride systems) can provide entertainment to customers in a variety of ways, including displaying augmented reality (AR) images that can be viewed by customers. For example, an amusement ride may include a display positioned adjacent to the customer and operating alone or in coordination with other features to output AR images for the customer to view. This can be done to create special effects, such as the illusion of a translucent illusion in a staged scene.

[0002] One technique used to provide such AR images is traditionally known as Pepper's Ghost. This technique for providing AR images is believed to have been developed in the 19th century. Pepper's Ghost utilizes the reflective properties of translucent or transparent materials (such as glass, plastic, or polyester foil) to virtually project images into a scene for viewers to see. For example, an angled glass block can be positioned in front of a stage, and an image can be projected from outside the viewer's line of sight toward the glass, and then partially reflected toward the viewer through the glass block. Thus, the viewer perceives the reflected image along with the scene presented behind the glass and in their line of sight. Depending on the lighting, this can give the reflected image an illusory appearance because light from behind the glass can still be seen through the reflected image. However, lighting techniques can be used to make the reflected image appear more three-dimensional by limiting competing light from the background. This type of AR has been used for many years and is currently used in many amusement park attractions. However, it is now recognized that viewers are becoming more sophisticated and able to discern the true nature of the illusion. Therefore, it is now recognized that technological improvements are needed to make the illusions more realistic and immersive.

[0003] This section aims to introduce the reader to various technical aspects that may relate to the present technology, which are described and / or claimed below. This discussion is intended to provide the reader with background information to facilitate a better understanding of the various aspects of this disclosure. Therefore, it should be understood that these statements are to be read in this context, and not as an admission of prior art. Summary of the Invention

[0004] Certain embodiments corresponding in scope to the disclosed subject matter are summarized below. These embodiments are not intended to limit the scope of this disclosure, but rather are intended only to provide a brief overview of some of the disclosed embodiments. In fact, this disclosure may include a wide variety of forms that may be similar to or different from the embodiments set forth below.

[0005] The embodiment includes an augmented reality system having a first scene and a second scene. A partially reflective surface is positioned relative to the first scene, the second scene, and the viewer to facilitate viewing one of the first or second scenes through the partially reflective surface and to facilitate reflecting the other scene as an augmented reality image toward the viewer. Sensors are configured to detect characteristics of the first scene and generate data indicating those characteristics. A related effects system is configured to receive the data and adjust the second scene based on it.

[0006] The embodiment includes an augmented reality system having a background scene including background scene lighting and an augmented reality scene including augmented reality scene lighting. A partially reflective surface is positioned relative to the background scene, the augmented reality scene, and the viewer to facilitate viewing the background scene through the partially reflective surface and to facilitate the reflection of the augmented reality scene as an augmented reality image toward the viewer. Sensors are configured to detect lighting characteristics of the background scene and generate data indicating these lighting characteristics. A related effects system is configured to receive this data and adjust the augmented reality scene lighting based on it.

[0007] The embodiment includes an augmented reality system having a background scene comprising a three-dimensional staged area and background scene lighting operable to adjustably illuminate the three-dimensional staged area. The augmented reality system also includes an augmented reality scene comprising augmented reality scene lighting operable to provide augmented reality imagery. A partially reflective surface is positioned relative to the background scene, the augmented reality scene, and the viewer to facilitate viewing the background scene through the partially reflective surface and to facilitate the reflection of the augmented reality imagery toward the viewer. A sensor is configured to detect characteristics of one of the scenes in the background scene or the augmented reality scene, wherein the sensor is further configured to generate data indicating the characteristics. A related effects system is configured to receive the data and, based on the data, adjust aspects of the other scene in the background scene or the augmented reality scene. Attached Figure Description

[0008] These and other features, aspects, and advantages of this disclosure will become better understood when the following detailed description is read with reference to the accompanying drawings, in which the same characters denote the same parts throughout the drawings, wherein:

[0009] Figure 1 It is a schematic representation of an augmented reality (AR) system arranged for viewing by an audience in accordance with aspects of this disclosure;

[0010] Figure 2 This is a schematic perspective view of an embodiment of an AR system according to embodiments of the present disclosure, wherein a projection source operates to adjust the illumination of the stage relative to a virtual luminous object, and adjusts the positioning of the virtual luminous object relative to a physical object;

[0011] Figure 3 This is a schematic perspective view of an embodiment of an AR system according to embodiments of the present disclosure, wherein an illumination system is arranged close to the boundary of a partially reflective surface and is operable to adjust the illumination of a stage relative to virtual luminous objects.

[0012] Figure 4 This is a side view schematic diagram of an AR system according to an embodiment of the present disclosure, wherein an image source operates to display or simulate the reflection of a virtual object onto objects in a background scene;

[0013] Figure 5 This is a side view schematic diagram of an AR system according to an embodiment of the present disclosure, wherein stage lights are operated to simulate the reflection of virtual images onto objects in a background scene;

[0014] Figure 6 This is a side view schematic diagram of an AR system according to an embodiment of the present disclosure, wherein an illumination array operates to simulate reflections of virtual images onto objects in a background scene; and

[0015] Figure 7 This is a schematic perspective view of an AR system according to an embodiment of the present disclosure, wherein the illumination of multiple three-dimensional scenes involved in the augmented reality display is monitored by an illumination detector to facilitate adjusted illumination of one or more of the multiple scenes. Detailed Implementation

[0016] One or more specific embodiments will be described below. For the purpose of providing a concise description of these embodiments, not all features of the actual implementation are described in the specification. It should be appreciated that, as in any engineering or design project, numerous implementation-specific decisions must be made in the development of any such implementation to achieve the developer's specific goals, which may vary depending on the implementation, such as compliance with system-related constraints and business-related constraints. Furthermore, it should be appreciated that such development efforts may be complex and time-consuming, but for those skilled in the art who benefit from this disclosure, such development efforts will be nothing more than routine tasks of design, fabrication, and manufacturing.

[0017] When describing elements of various embodiments of this disclosure, the articles “a,” “an,” and “the” are intended to mean the presence of one or more of the elements. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that additional elements may be present in addition to those listed. Furthermore, it should be understood that references to “an embodiment” or “an embodiment” in this disclosure are not intended to be construed as excluding the existence of additional embodiments that also incorporate the described features.

[0018] According to current embodiments, an augmented reality (AR) system may include an AR image source (e.g., a projector or luminous scene) that operates to project an AR image onto a partially reflective surface (e.g., glass, plastic, or polyester foil) positioned between a viewer (e.g., an observer) and a background scene (such as a stage, electronic display, screen, or set). The AR image source and background may include a combination of referenced features. The AR image may include any image from a variety of images projected onto a screen that reflects it toward the viewer so that it appears as if the AR image exists relative to the background scene. In some embodiments, the AR image may include characters (e.g., people), illusions, objects (e.g., tables), text, virtual luminous objects (e.g., flames), and so on. In embodiments, the partially reflective surface may include a translucent mirror that partially reflects the AR image from the source toward the viewer. In other words, the partially reflective surface reflects the AR scene as an AR image toward the viewer. The viewer may perceive the AR image reflected from the translucent mirror as overlapping with a background scene, which may include physical props, an electronic display (e.g., a projector screen or a liquid crystal display), or both. In this way, AR images can appear to viewers as if they are interacting with and / or positioned close to features of the background scene. To enhance this effect, AR images can include three-dimensional (3D) images, which can be described as two-dimensional images that appear three-dimensional when viewed through appropriate lenses (e.g., polarizing or tinted lenses for 3D glasses).

[0019] Transparent mirrors representing various other partially reflective surfaces (e.g., glass, mesh) can be positioned at an angle relative to the viewer and the AR image source, enabling an effect that can be described as the Pepper illusion effect. The Pepper illusion effect involves reflecting AR images so that the viewer can simultaneously view the AR image on the partially reflective surface along with features located on the opposite side of the surface. Specifically, the Pepper illusion effect may cause the AR image to appear superimposed on an object or image positioned behind the partially reflective surface. For example, a person presented as an AR image and reflected by a partially reflective surface can appear to be sitting in an actual chair, which is partially positioned on a stage behind the reflective surface relative to the viewer's viewpoint.

[0020] A scene behind a partially reflective surface that is directly viewed by the audience can be referred to as a background scene. A scene that provides AR images (which are viewed by the audience after being reflected from the partially reflective surface) can be referred to as an AR scene. Background scenes and AR scenes can include physical components (e.g., stage props, actors, structures) and / or electronic displays (e.g., projectors, liquid crystal displays, lighting panels). For example, a background screen can include a screen on which video is projected (e.g., a movie screen), while an AR scene can include luminous props and actors positioned to cause light to reflect off the partially reflective surface. In this example, the AR scene can be set up in a room beneath the stage, which is not directly viewed by the audience but is positioned to direct light toward the partially reflective surface so that it reflects toward the audience.

[0021] In another example, opposite or different combinations of features can be used for each scene in the background scene and the AR scene. Because the images provided by the background scene and the AR scene are combined to provide the Pepper illusion effect, the relative lighting of each scene affects the nature of the effect in relation to how it is perceived. For example, if the background scene is substantially brighter than the AR scene, the viewer may hardly perceive the AR image reflected towards them. However, if the background scene is substantially darker than the AR scene, the AR image may dominate, and the background scene may be invisible to the viewer. Furthermore, because the AR image is presented to the viewer for viewing through reflection, features of the displayed AR image (e.g., a moving character) will not illuminate the background scene in the way the viewer would expect actual features to do so. For example, an AR image of a particular feature (e.g., a person, a lamp, or a car) will not cause an actual reflector in the background scene to reflect that feature appropriately. Similarly, an AR image of a feature that would typically be a luminous feature (e.g., a lamp, a flash, a flame, or other luminous object) will not cause light and corresponding shadows to be generated in the background scene.

[0022] Current embodiments include a monitoring system that tracks illumination in one or both of the background scene and the AR scene to coordinate lighting effects and achieve the desired overlapping perception of the combined imagery from the background scene and the AR scene. Additionally, current embodiments include lighting features that simulate the expected lighting effects of features provided in the AR imagery, such that the background scene is affected in a way expected by the viewer, improving the immersive nature of the presentation. Current embodiments also include a tracking system that operates to track features in the background scene to provide the desired AR imagery associated with them. For example, a physical lantern can move around in the background scene and be tracked to provide location information for the lantern. This location information can be used to provide the AR imagery at a location on a partially reflective surface, relative to the lantern's location, such that the combined background scene and AR imagery make the lantern appear lit and glowing to the viewer. Specifically, luminous objects (such as flames) can be projected onto a partially reflective surface or screen at a location where the viewer views the luminous object as the physical location of the lantern is tracked. Moreover, for example, the background scene lighting can be adjusted to cast shadows as if caused by pseudo-lighting from the lantern.

[0023] Figure 1This is a schematic representation of an AR system 10 according to an embodiment of the present disclosure. The AR system 10 includes a background scene 12, an AR scene 14, and a partially reflective surface 16 (which may also be referred to as a partially reflective layer or screen 16) disposed therebetween. A viewer 18 is positioned such that an AR image 20, including light reflected from the AR scene 14, is reflected from the partially reflective surface 16 toward the viewer 18, as illustrated by the virtual image 21 in the dashed lines, so that the AR image 20 appears to be positioned within the background scene 12. Furthermore, the background scene 12 is positioned such that the viewer 18 can directly view the background scene 12 through the partially reflective surface 16, which is also partially transparent. The background scene 12 and the AR scene 14 can be defined by any combination of physical props, live actors, electronically generated images, etc. For example, the background scene 12 may include a stage with physical features (e.g., chairs, tables, and live actors), while the AR scene 14 may include a projector that directs the AR image 20 toward the partially reflective surface 16. In other embodiments, different combinations of physically characterized and electronically generated images can be used in each of the background scene 12 and the AR scene 14. The partially reflective surface 16 may include special foil, glass, plastic, partially reflective mirrors, etc., operable to allow a viewer 18 to both view through it and observe the image reflected from it under appropriate lighting conditions. The AR system 10 also includes a correlation effects system 22, which may be a controller designed to operate in conjunction with various sensors 24, 26 and lighting systems 28, 30 to provide the desired effects according to embodiments. Moreover, in embodiments according to this disclosure, the correlation effects system 22 may actually include sensors 24, 26 and lighting systems 28, 30. In some embodiments, only one sensor 24, 26 may be used to monitor one or both of the background scene 12 and the AR scene 14.

[0024] Specifically, the AR system 10 includes a correlation effects system 22 having features that allow the AR system 10 to correlate aspects of the AR scene 14 with aspects of the background scene 12 in a way that enhances the immersion of the viewer 18 in the AR illusion provided by the AR system 10. The AR system 10 may also include features such as actuators 19 that facilitate manipulation (e.g., repositioning) of the partially reflective surface 16 to achieve certain correlation results.

[0025] AR system 10 and / or associated effects system 22 may include one or more controllers 32, processors 34, and / or memory 36 to perform various functions (e.g., directing the operation of other system features). One or more memories 36 may include random access memory (RAM), read-only memory (ROM), rewritable non-volatile memory (such as flash memory), hard disk drive, optical disk, and / or other types of memory. One or more processors 34 of AR system 10 may include one or more general-purpose microprocessors, one or more application-specific integrated circuits (ASICs), and / or one or more field-programmable gate arrays (FPGAs). One or more controllers 36 may include programmable logic controllers (PLCs) or other computer-based controllers. These features (e.g., controllers 36, processors 34, and memory 32) may be components of associated effects system 22 or individual features. Furthermore, as will be understood, these features may be operated using stored instructions (e.g., code) that, when executed, initiate actions (e.g., dimming of lighting or manipulation of actuators).

[0026] In one example, AR system 10 can be operated to correlate lighting characteristics between AR scene 14 and background scene 12 to improve the coherence between directly observable aspects of AR image 20 and background scene 12. For example, it might be ideal to make adjustments so that AR image 20 has a tint similar to that of background scene 12. This can be achieved by observing lighting characteristics in AR scene 14 or background scene 12 via sensors 24, 26 and making corresponding lighting changes in another scene within AR scene 14 or background scene 12 via lighting systems 28, 30. In a particular system, one or two sensors 24, 26 may be included and / or used. Examples of lighting characteristics that can be detected and adjusted include intensity, position / orientation, brightness, color, temperature, contrast, and quality. The characteristics of one scene can be modified based on different characteristics of another scene. For example, correlation effects system 22 can be operated to adjust the brightness in background scene 12 based on the contrast in AR scene 14. Current embodiments can detect and adjust lighting in either a physical scene (e.g., a theater stage) or a virtual scene (e.g., a scene provided by an electronic display, such as a liquid crystal display or a projector). For example, in an embodiment where one or both of the background scene 12 and the AR scene 14 include an electronic display, sensors 24, 26 can detect display settings (e.g., color settings), and lighting systems 28, 30 can control the display settings based on the output from sensors 24, 26. In addition to changes in lighting, current embodiments can also adjust the physical positioning of the partially reflective surface 16. For example, the angle of the partially reflective surface 16 relative to the background scene 12, the AR scene 14, and the viewer 18 can be adjusted by actuator 19 based on instructions from the related effects system 22 to achieve a desired viewing result.

[0027] Figure 2 This is a schematic perspective view of an embodiment of an AR system 10 according to an embodiment of the present disclosure, wherein a projection source 60 operates to adjust the illumination of a stage 62 relative to a virtual luminous object 64 and to adjust the positioning of the virtual luminous object 64 relative to a physical object 66. Figure 2 In the illustrated embodiment, background scene 12 incorporates stage 62 and physical object 66, which is a moving background object. This physical object 66 can be any number of movable objects. For example, a moving object can be a robot figure, a torch, a sword, an actor, or any other trackable object. However, in the illustrated embodiment, the physical object is a prop lantern that does not actually emit light itself.

[0028] A tracking system 70 is included to facilitate tracking physical movement within the background scene 12. For example, the illustrated tracking system 70 includes sensors (e.g., a camera) 72, a processor 74, and a memory 76, which coordinate to track physical objects 66 and / or actors 80 in space. By tracking these physical aspects of the background scene 12 (e.g., physical objects and actors 80), the lighting system 60 can be appropriately adjusted to make the augmented reality presentation of the AR system 10 more immersive. For example, AR image 20 can be projected onto a partially reflective surface 16 via a projector 82 of the lighting system 60, such that the positioning of AR image 20 is related to the positioning of physical object 66 relative to the viewer's line of sight 84. The illustrated projector 82 represents a wide range of image generation devices that can be employed to achieve the desired effect. For example, projector 82 can represent a flat-screen television that generates an image of a flame, which is then reflected by the partially reflective surface 16, creating the illusion that the flame is actually positioned on or within physical object 66. In other embodiments, projector 82 may include a screen that receives images via front-projection or rear-projection. As with flat-screen televisions, the image presented on the screen may be reflected by the partially reflective surface 16 to create the desired illusion. Projector 82 may refer to any device that will operate to provide the desired image. Moreover, in some embodiments, projector 82 can be replaced by a physical characteristic (e.g., a lit candle) that allows it to be manipulated in space via actuators to provide reflections associated with the desired positioning of the illusory image relative to stage 62.

[0029] Considering the foregoing, the combined action of the tracking system 77 and the lighting system 60 allows the AR system 10 to make the physical object 66 (in the illustrated embodiment, a non-functional lantern prop) appear as if it were lit and emitting flames, since the AR image 20 represents flames and is properly positioned. Furthermore, the stage lights 86 of the lighting system 60 can be operated to project a shadow 88 associated with the positioning of the AR image 20. In the illustrated embodiment, for example, the shadow 88 can be projected by operating one or more specific luminaires 90 of the stage lights 86 to make it appear as if the AR image 20 is causing the shadow 88. For this purpose, a related effects system 22 (e.g., an automation controller) can be employed by taking input from the tracking system 70 and providing output to the lighting system 60. The related effects system 22 can also use known positioning information (e.g., a defined path for the physical object 66 and / or the AR image 20) to control the lighting system 60 to provide a relevant and immersive effect. For example, lantern props can be automated to be manipulated across stage 62, and the pattern across the stage can be coordinated with projector 82 to provide the desired effect. Such embodiments would improve efficiency and eliminate the processing time and / or costs associated with tracking physical features (e.g., physical object 66).

[0030] Because of the ability to detect or identify one or more features (e.g., shape, color, facial features, or RFID) by the tracking system 70, physical objects 66 and / or actors 80 can be tracked by the tracking system 70. For example, in one embodiment, the tracking system 70 is programmed to identify a specific shape of the physical object 66 and track the movement of the physical object 66, such that location data (e.g., current location) can be easily identified substantially in real time. To achieve this, the memory 76 and processor 74 can execute programming (e.g., object recognition or facial recognition programs) to interpret data and / or extrapolate data obtained from the self-sensor 72. Using the location data obtained by the tracking system, appropriate locations for use in controlling the lighting system 60 can be identified.

[0031] Figure 3 This is a schematic perspective view of an embodiment of the AR system 10, wherein stage lights 86 of the lighting system 60 are arranged close to the boundary 102 of the partially reflective surface 16 and are operable to adjust the illumination of the stage 62 of the background scene 12 relative to the AR image 20, which in the illustrated embodiment is presented as a virtual luminous object and specifically as a virtual flame. The audience 18 can perceive the virtual flame as existing in the center of the stage 62 and in front of the physical object 66, as generally indicated by the virtual image 21 in the dashed lines. Figure 3 The embodiments described herein are arranged in a manner consistent with the previous ones regarding... Figure 2The AR system 10 described in the illustration operates in a similar manner. That is, the AR image 20 (which is a virtual luminous object) is projected by the projector 82 onto the partially reflective surface 16 so that it appears positioned within the three-dimensional space of the background scene 12 and interacts with the background object 66 to generate shadows 88. In this embodiment, however, the stage light 86 is mounted on or near the edge of the partially reflective surface 16. This positioning allows various luminaires 90 of the stage light 86 (e.g., LED lights) to be activated based on the positioning of the AR image 20, causing a realistic effect on the background scene 12, which, in the illustrated embodiment, appears to be shadows 88 caused by a virtual flame represented as a virtual image 21.

[0032] As can be appreciated, if AR image 20 represents a flickering flame moving around partially reflective surface 16 to simulate a flickering flame moving around a three-dimensional scene, then stage light 86 can move around to cast different shadows based on the simulated positioning, and also flicker to induce appropriate shadow formation. In some embodiments, such as in Figure 3 In one embodiment, when the luminaire 90 itself is not moving, different luminaires positioned partially or completely around the partial reflective surface 16 can be activated to achieve a similar effect. Using such a technique and system, even observers in the audience 18 who have already seen many Pepper's illusions will be more immersed in the effect. It should be noted that the positioning of the various luminaires 90 can also be adjusted in combination with the activation of different luminaires 90 at different locations. For example, in the case where the various luminaires 90 are coupled to the boundary 102, the luminaires 90 can be moved by using actuator 27 to move the entire partial reflective surface 16 to achieve the desired viewing result. In other embodiments, a separate actuator can be used for each luminaire 90.

[0033] Figure 4 This is a side view of AR system 10, in which a projection source 112 (e.g., a movie projector, television screen, projection screen) operates to display or simulate reflections by displaying an image 116 of a virtual object 118 provided by AR image 20 onto a surface 124 in a background scene 12. In the illustrated embodiment, surface 124 includes a pseudo-reflector positioned on stage 62. In other embodiments, surface 124 may represent other objects, such as a piece of glossy furniture, a glass window, a set of plates, a metal panel, etc. Specifically, in Figure 4In the illustrated embodiment, the background scene 12 of the AR system 10 includes a pseudo-reflector as a surface 124, which is actually a projection surface used to present the image 116 to the viewer. Since the virtual object 118 will only appear to be positioned as the virtual image 21, it will not actually cause reflection in the real reflectors (or other glossy surfaces) in the background scene 12. Therefore, the current embodiment achieves the appearance of such reflection and enhances the viewer's immersion by simulating actual reflection.

[0034] A pseudo-reflector is essentially a prop that simulates the reflective properties of a mirror by displaying an image received from a projection source 112, which in the illustrated embodiment is positioned in front of surface 124. In other embodiments, back projection may be used. In still other embodiments, the projection source 112 and surface 124 may be combined features, such as a flat-screen television. To make the background scene 12 appear realistic and enhance the immersion of the viewer 18 in the presentation, the associated effects system 22 coordinates the operation of the projection source 112 and the projector 82. For example, the movement of a virtual object 118 provided by the projector 82 is coordinated with an image 116 provided by the projection source to provide the illusion that surface 124 is reflecting the virtual object 118. In some embodiments, this may include modifying data from the virtual object 118 to provide image 116. For example, depending on the simulated surface (e.g., slightly glossy paint on furniture), image 116 may need to appear blurred. It should be noted that the projection source 112 and the projector 82 can represent any number of images to provide features. For example, any of these features can represent a display screen, an illuminated stage, a traditional projector, and so on.

[0035] Figure 5This is a side view schematic diagram of an AR system 10 according to an embodiment of the present disclosure, wherein a stage light 86 operates to simulate the reflection of a virtual image 21 onto objects in a background scene 12. In the illustrated embodiment, a physically reflecting object 152 (e.g., a luminous ball, an hourglass, glossy paint, a piece of glossy furniture) is positioned in the background scene 12. In the AR scene 14, a physically illuminating object 154 (e.g., a projector, a television screen, a candle, an illuminated object) is positioned such that the image of the physically illuminating object 154 is reflected from a partially reflective surface 16 toward the viewer 18 in such a way that it creates the illusion that the physically illuminating object 154 is actually present in the background scene 12 and positioned as the virtual image 21. A sensor 24 (e.g., a camera) operates to detect illumination conditions that would exist on the surface of the physically reflecting object 152 if the physically illuminating object 154 is actually positioned as the virtual image 21 in the background scene 12. In an embodiment, this can be based on the following: if the virtual image 21 actually exists in the background scene 12, then the sensor 24 observes the physical lighting object 154 from the position corresponding to the physical reflective object 152. For example, the sensor can be positioned at a 45-degree angle relative to the physical lighting object 154 in the AR scene, because the shiny rubber ball is positioned at a 45-degree angle relative to the position where the virtual image 21 appears in the background scene 12. Moreover, such information can be computed using modeling techniques, and the approximation can be sufficient to provide the desired illusion. Such techniques can be applied when the virtual image 21 is based on an image displayed on a screen (e.g., a television screen). Thus, the data obtained from the sensor 24 can then be used to generate light from the stage light 85, which will be projected or reflected into the background scene 12 in an appropriate manner. For example, in Figure 6 In the illustrated embodiment, a pair of central stage lights 85 have been selected as active based on data that produces the desired reflective effect. Since the resulting lighting effect will match both the position and quality of the light that would strike the physical reflective object 152 if positioned relative to the virtual image 21 as perceived, the physical reflective object 152 will respond to the projected light in an accurate and reliable manner.

[0036] Figure 6 This is a side view schematic diagram of an AR system 10 according to an embodiment of the present disclosure, wherein an illumination array 162 (e.g., a planar array of collimated light, a light field display, a laser panel) operates to simulate the reflection of a virtual image 21 onto objects in a background scene 12. Figure 6 AR system 10 with Figure 5 The embodiments illustrated herein are established in a similar manner. However, Figure 6The stage light 86 in the illustrated embodiment includes an illumination array 162. The illumination array 162 can include any of a variety of focused illumination systems capable of directing a light spot to a specific location without causing the associated light cone to substantially expand and cause blurring, etc. Data obtained from sensor 24 is used to correlate with... Figure 5 In a similar manner, the illumination array 162 can direct light in a desired pattern toward the back of the partially reflective surface 16, so that the light is properly reflected into the background scene 12. For example, in an embodiment where the illumination array 162 includes a light field display, specific rays can be directed to converge at a perceived location of the virtual image 21 (based on data from sensor 24 indicating the perceived location), and then diverge from there in a manner similar to what would happen if the virtual image 21 were actually present. This can facilitate a highly accurate illusion of specular reflection of the virtual image 21 on the physically reflective object 152.

[0037] Figure 7 This is a schematic perspective view of an AR system 10 according to an embodiment of the present disclosure, wherein the illumination of multiple three-dimensional scenes involved in the augmented reality display is monitored by an illumination detector to facilitate adjusted lighting of one or more of the multiple scenes. In the illustrated embodiment, both background scene 12 and AR scene 14 are three-dimensional staged areas. Background scene 12 is illuminated by background scene lighting 202, and this illumination is monitored by background scene sensor 204 (e.g., a camera). The AR scene is illuminated by AR scene lighting 212, and this illumination is monitored by AR scene sensor 214. Props 218 in AR scene 14 reflect light from AR scene lighting 212 toward a partially reflective surface 16, which is then directed toward a viewer 18. The combined illumination of background scene 12 and AR scene 14 can be controlled by a related effects system 22, which, in the illustrated embodiment, wirelessly communicates with lighting system 60 and related sensors 204, 214.

[0038] In an embodiment, the related effects system 22 can control the lighting system 60 so that the AR image 20 appears semi-transparent to the viewer 18, which can provide an illusion effect to the AR image 20. Such control by the related effects system 22 can be based on sensor data from the background scene sensor 204 and / or the AR scene sensor 214. Based on the sensor data from the background scene sensor 204 and / or the AR scene sensor 214, lighting adjustments to the background scene lighting 202 and the AR scene lighting 212 can also be performed by the related effects system 22 to make other adjustments to the viewing experience. For example, the AR image 20 can be made to appear more three-dimensional, or various adjustments can be made to better correlate the background scene 12 and the AR scene 14 (e.g., harmonize shading, contrast). It should be noted that although... Figure 5 The illustration describes a specific embodiment where background scene 12 and AR scene 14 are provided by a 3D stage set, but multiple different features can be combined to form background scene 12 and AR scene 14. For example, projector 82 may be part of lighting system 60 and operate to facilitate the provision of additional AR images 20. As another example, display 222 (e.g., an LCD screen or projection screen) may facilitate the provision of background scene 12 in conjunction with physical objects 66 of stage 62. According to the current embodiment, each of these features (e.g., display 222, projector 82, lighting 202, 212) may be coordinated by associated effects system 22 to provide an immersive augmented reality presentation to the audience by coordinating effects to provide the desired image result.

[0039] Various aspects of this disclosure are conveyed through Figure 1-5 This is illustrated with its corresponding description. For example, Figure 4 An example of an AR system 10 that provides pseudo-reflections of AR image 20 in a background scene 12 is provided, while Figure 5 Examples of specific arrangements of three-dimensional staged areas as background scene 12 and AR scene 12 are provided, and those scenes are used to provide coordinated lighting in AR rendering. These are specific aspects of embodiments provided to convey the broader features covered by this disclosure and that can be combined in various ways to achieve different and combined results. For example, pseudo-reflections (such as, regarding) Figure 5 The described pseudo-reflection can be coordinated with various implementations of this disclosure (such as, together with) Figure 3 (Lighting arrangement). In fact, this disclosure covers all combinations of the disclosed features of AR system 10.

[0040] While only certain features of this disclosure have been described and illustrated herein, many modifications and alterations will occur to those skilled in the art. Therefore, it will be understood that the appended claims are intended to cover all such modifications and alterations that fall within the true spirit of this disclosure.

[0041] The techniques proposed and claimed herein are referenced and applied to substantial objects and specific examples of practical nature, which arguably improve upon the art and are therefore not abstract, intangible, or purely theoretical. Furthermore, if any claim appended to this specification contains one or more elements designated as “component for [implementing]…[function]” or “step for [implementing]…[function]”, such elements are intended to be interpreted according to 35 USC 112(f). However, for any claim containing elements designated in any other manner, such elements are intended not to be interpreted according to 35 USC 112(f).

Claims

1. An augmented reality system, comprising: Scene 1; Second scenario; A partially reflective surface, positioned relative to the first scene, the second scene, and the viewer, to facilitate viewing the first scene and to facilitate the reflection of the second scene as an augmented reality image toward the viewer. A sensor configured to detect at least one characteristic of the first scene and generate data indicating the at least one characteristic, wherein the at least one characteristic includes the location of a physical object in the first scene; and A related effects system includes one or more processors and a memory accessible to the one or more processors, wherein the memory stores instructions that, when executed by the one or more processors, cause the one or more processors to: The data is received, and the second scene is adjusted based on the data by instructing one or more actuators to reposition the partially reflective surface, such that the augmented reality image is positioned relative to the positioning of the physical object.

2. The augmented reality system according to claim 1, wherein, The first scene includes a first lighting, the second scene includes a second lighting, the at least one characteristic includes a lighting characteristic, and the related effects system is configured to adjust the second lighting based on the lighting characteristic of the first scene.

3. The augmented reality system according to claim 2, wherein, The first scene is a background scene, the first lighting is background scene lighting, the second scene is an augmented reality scene, and the second lighting is augmented reality lighting.

4. The augmented reality system according to claim 2, wherein, The first scene is an augmented reality scene, the first lighting is augmented reality lighting, the second scene is a background scene, and the second lighting is background scene lighting.

5. The augmented reality system according to claim 1, wherein, The second scene includes an image generator, and the related effects system is configured to adjust the color, contrast, or a combination thereof of an image provided by the image generator based on the data, wherein the data indicates the lighting of the first scene.

6. The augmented reality system of claim 1, further comprising an image generator configured to simulate reflections of the augmented reality image in the first scene based on the data.

7. The augmented reality system according to claim 1, wherein, The first scene includes a three-dimensional stage set, and the sensor includes a camera configured to detect the lighting characteristics of the three-dimensional stage set.

8. The augmented reality system according to claim 1, wherein, The first scenario includes an electronic display.

9. The augmented reality system according to claim 1, wherein, The sensor is one of a plurality of sensors configured to monitor one or both of the first scene and the second scene for motion and lighting characteristics.

10. The augmented reality system according to claim 1, wherein, The related effects system is configured to adjust the lighting of the first scene based on the position of the virtual luminous object of the augmented reality image on the partially reflective surface, so as to create shadows in the first scene.

11. The augmented reality system according to claim 1, wherein, The first scene, the second scene, or both include one or more projectors, one or more liquid crystal displays, one or more lighting panels, or any combination thereof.

12. An augmented reality system, comprising: Background scene, which includes background scene lighting; Augmented reality scenes, including augmented reality scene lighting; A partially reflective surface, positioned relative to the background scene, the augmented reality scene, and the viewer, to facilitate viewing the background scene through the partially reflective surface and to facilitate the reflection of the augmented reality scene as an augmented reality image toward the viewer. A sensor configured to detect illumination characteristics of the background scene, the location of a physical object in the background scene, or both, and to generate data indicating the illumination characteristics, the location of the physical object, or both; and A related effects system includes one or more processors and a memory accessible to the one or more processors, wherein the memory stores instructions that, when executed by the one or more processors, cause the one or more processors to receive the data and adjust the augmented reality scene based on the data by directing one or more actuators to reposition the partially reflective surface to provide virtual objects that are illuminated according to the lighting characteristics and in positions corresponding to the positioning of the physical objects.

13. The augmented reality system according to claim 12, wherein, The background scene includes a theater stage, and the partially reflective surface includes a partially mirrored foil.

14. The augmented reality system of claim 12, comprising a display in the background scene, the display being configured to simulate reflections of the augmented reality image.

15. The augmented reality system of claim 12, further comprising an additional sensor configured to detect additional lighting characteristics of the augmented reality scene and generate additional data indicative of the additional lighting characteristics, wherein, The related effects system is configured to receive the additional data and adjust the background scene lighting based on the additional data.

16. The augmented reality system according to claim 15, wherein, The augmented reality scene includes a three-dimensional staged area.

17. The augmented reality system according to claim 12, wherein, The augmented reality scene lighting includes one or more projectors, one or more liquid crystal displays, one or more lighting panels, or any combination thereof.

18. An augmented reality system, comprising: Background scene, which includes a three-dimensional staged area and background scene lighting, wherein the background scene lighting is configured to adjustably illuminate the three-dimensional staged area; Augmented reality scene, including augmented reality scene lighting configured to provide augmented reality imagery; A partially reflective surface, positioned relative to the background scene, the augmented reality scene, and the viewer, to facilitate viewing the background scene through the partially reflective surface and to facilitate the reflection of the augmented reality image toward the viewer; A sensor configured to detect characteristics of the background scene, wherein the sensor is further configured to generate data indicative of the characteristics, and wherein the characteristics include the location of a physical object within the background scene; and A related effects system includes one or more processors and a memory accessible to the one or more processors, wherein the memory stores instructions that, when executed by the one or more processors, cause the one or more processors to receive the data and adjust the augmented reality scene based on the data by directing one or more actuators to reposition the partially reflective surface to provide a virtual object at a location corresponding to the positioning of the physical object.

19. The augmented reality system according to claim 18, wherein, The augmented reality scene includes a display screen, the augmented reality scene lighting includes the lighting of the display screen, the sensor is configured to generate the data indicating the lighting characteristics of the background scene, and the correlation effects system is configured to control the lighting of the display screen to provide characteristics of the augmented reality image associated with the data.

20. The augmented reality system according to claim 18, wherein, The augmented reality scene lighting includes one or more projectors, one or more liquid crystal displays, one or more lighting panels, or any combination thereof.

21. An augmented reality system, comprising: Scene 1; The second scene is presented by a projector; A partially reflective surface, positioned relative to the first scene, the projector, and the viewer, to facilitate viewing the first scene through the partially reflective surface and to facilitate the reflection of the second scene as an augmented reality image toward the viewer; as well as The controller is configured as follows: Receive data indicating the characteristics of the first scenario; and The reflection of the second scene is adjusted based on the characteristics by changing the relative positioning of the projector and the partially reflective surface.

22. The augmented reality system according to claim 21, wherein, The characteristics include the lighting characteristics of the first scene.

23. The augmented reality system according to claim 21, wherein, The features include the lighting features of the first scene, and the controller is configured to adjust the lighting of the second scene based on the lighting features of the first scene.

24. The augmented reality system according to claim 21, wherein, The projector includes an electronic display, and the controller is configured to adjust the color, contrast, brightness, or any combination thereof of the image provided by the electronic display based on the lighting characteristics of the first scene.

25. The augmented reality system according to claim 21, wherein, The controller is configured to receive the data from one or more sensors, the one or more sensors being configured to detect the characteristic as the position of an object in the first scene, the lighting characteristics of the first scene, or both.

26. The augmented reality system according to claim 21, wherein, The first scene includes a three-dimensional stage set.

27. The augmented reality system according to claim 21, wherein, The controller is configured to adjust the lighting of the first scene to create shadows in the first scene based on the positioning of virtual luminous objects in the augmented reality image.

28. The augmented reality system of claim 21, further comprising an image generator configured to simulate reflections of the augmented reality image in the first scene.

29. The augmented reality system of claim 21, comprising a first camera configured to detect a first illumination in the first scene and a second camera configured to detect a second illumination in the second scene, wherein, The controller is configured to adjust the augmented reality imagery based on the first lighting and the second lighting.

30. The augmented reality system according to claim 21, wherein, The first scene includes one or more projectors, one or more liquid crystal displays, one or more lighting panels, or any combination thereof.

31. The augmented reality system according to claim 21, wherein, The controller is configured to direct one or more actuators to reposition the partial reflective surface so that the augmented reality image is located relative to the detected position of the object in the first scene.

32. A system comprising: Background scene; Augmented reality scenes are displayed via projector; A partially reflective surface, positioned relative to the background scene, the projector, and the viewer, to facilitate viewing the background scene through the partially reflective surface and to facilitate the reflection of the augmented reality scene as an augmented reality image toward the viewer; as well as One or more processors configured to: Receive data including the lighting characteristics of the background scene; as well as The augmented reality scene is adjusted by changing the relative positioning between the projector and the partially reflective surface to provide virtual objects of the augmented reality image illuminated according to the lighting characteristics.

33. The system according to claim 32, wherein, The data includes the location of physical objects within the background scene.

34. The system according to claim 33, wherein, The one or more processors are configured to adjust the augmented reality scene by causing the relative positioning transformation between the projector and the partially reflective surface to provide the virtual object of the augmented reality image in a position corresponding to the positioning of the physical object.

35. The system according to claim 32, wherein, The background scene includes one or more electronic displays, one or more projectors, one or more lighting panels, or any combination thereof.

36. The system of claim 32, further comprising a camera configured to detect the lighting characteristics of the background scene.

37. The system according to claim 32, wherein, The partially reflective surface includes a portion of mirror foil.

38. A method comprising: The system receives, via one or more processors, data including the illumination characteristics of augmented reality lighting for an augmented reality scene positioned relative to a background scene and partially reflective surfaces, the illumination characteristics of background lighting for the background scene, and the positioning of physical objects within the background scene; as well as By changing the relative positioning between the projector of the augmented reality scene and the partially reflective surface, the augmented reality scene is adjusted via the one or more processors to provide virtual objects that are illuminated according to the background lighting and in positions corresponding to the positioning of the physical objects.

39. The method of claim 38, further comprising generating an augmented reality image via the one or more processors based on the augmented reality lighting and the background lighting to provide the virtual object.

40. The method of claim 38, further comprising adjusting the background lighting via the one or more processors based on the position of the virtual object relative to the partially reflective surface to generate shadows in the background scene.

Images