Multi-sensory object renderer

Abstract

Some methods for controlling a set of controllable actuators may involve obtaining actuator data for the set of controllable actuators and receiving object-based sensory data including a set of sensory objects. Some methods may involve rendering the object-based sensory data to produce actuator control signals, wherein the rendering is based at least in part on the actuator data. Some methods may involve obtaining playback environment data. The rendering may be based, at least in part, on the playback environment data. Some methods may involve providing, by the control system, the actuator control signals to one or more controllable actuators of the set of controllable actuators. The set of controllable actuators may include one or more light fixtures, one or more haptic devices, one or more air flow control devices, or combinations thereof.

Description

MULTI-SENSORY OBJECT RENDERER CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority from U.S. Provisional Application No. 63 / 669,233 filed July 10, 2024, U.S. Provisional Application No.63 / 514,106 filed July 17, 2023, and U.S. Provisional Application No.63 / 514,095 filed July 17, 2023, each of which is hereby incorporate by reference in their entireties. TECHNICAL FIELD

[0002] The present disclosure relates to providing multi-sensory (MS) experiences, and is more specifically directed to aspects of an MS renderer. BACKGROUND

[0003] Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims in this application and are not admitted as prior art by inclusion in this section.

[0004] Media content delivery has generally focused on audio and screen-based visual experiences. There has been limited delivery of multi-sensory content due to the bespoke nature of actuation. Luminaires, for example, are used extensively as an expression of art and function for concerts. However, each installation is designed specifically for a unique set of luminaires. Delivering a lighting design beyond the set of fixtures the system was designed for is generally not feasible. Other systems that attempt to deliver light experiences more broadly simply do so by extending the screen visuals algorithmically, but are not specifically authored. Haptics content is designed for a specific haptics apparatus. If another device, such as a game controller, mobile phone or even a different brand of haptics device is used, there has been no way to translate the creative intent of content to the different actuators. SUMMARY

[0005] At least some aspects of the present disclosure may be implemented via methods, such as audio processing methods. In some instances, the methods may be implemented, at least in part, by a control system such as those disclosed herein. Some such methods may involve obtaining, by a control system, actuator data for the set of one or more controllableactuators. Some methods may involve receiving, by the control system, object-based sensory data including a set of one or more sensory objects. Some methods may involve rendering, by the control system, the object-based sensory data to produce one or more actuator control signals. The rendering may be based at least in part on the actuator data. Some methods may involve providing, by the control system, the one or more actuator control signals to one or more controllable actuators of the set of controllable actuators. In some examples, the set of one or more controllable actuators may include one or more light fixtures, one or more haptic devices, one or more air flow control devices, or combinations thereof.

[0006] According to some examples, the object-based sensory data may include sensory object metadata. In some examples, the sensory object metadata may include sensory object position metadata. According to some examples, the sensory object metadata may include sensory object size metadata.

[0007] Some methods may involve integrating the actuator data with the object-based sensory data. The integrating may involve finding one or more closest controllable actuators for each sensory object.

[0008] In some examples, the actuator data may include an actuator map. According to some examples, rendering the object-based sensory data to produce one or more actuator control signals may involve projecting the set of one or more sensory objects using the actuator map. In some examples, projecting the set of one or more sensory objects using the actuator map may produce an actuator activation matrix as, or representing, the one or more actuator control signals. According to some examples, the actuator map may be, or may include, a light fixture map. The light fixture map may, for example, be an allocentric light fixture map that is based on playback environment spatial coordinates or an egocentric light fixture map that is based spatial coordinates relative to an intended viewing location.

[0009] Some methods may involve obtaining playback environment data. In some such examples, the rendering may be based, at least in part, on the playback environment data.

[0010] In some examples, the object-based sensory data may include sensory object priority metadata. Some such methods may involve determining that two or more sensory objects map to a single actuator and rendering the object-based sensory data of the two or more sensory objects to one or more actuator control signals for the single actuator based, at least in part, on the sensory object priority metadata.

[0011] According to some examples, the rendering may be based, at least in part, on one or more renderer configuration parameters. In some examples, the one or more rendererconfiguration parameters may include a velocity priority parameter that assigns a higher priority to moving sensory objects than to stationary sensory objects. According to some examples, the one or more renderer configuration parameters may include a change priority parameter that assigns a higher priority to changing sensory objects than to static sensory objects.

[0012] In some examples, the one or more sensory objects may include one or more light objects. In some examples, the one or more light objects may include color information. According to some examples, the one or more renderer configuration parameters may include a color change priority parameter that assigns a higher priority to color-changing light objects than to color-static light objects. In some examples, the one or more light objects may include color saturation information. According to some examples, the one or more renderer configuration parameters may include a color saturation priority parameter that assigns a higher priority to light objects having a higher color saturation than to light objects having a lower color saturation.

[0013] According to some examples, the rendering may be based, at least in part, on one or more renderer configuration modes. According to some such examples, the one or more renderer configuration modes may include a low actuator count mode, a content type mode, a color mixing mode, a single-sensory-object-to-single-actuator mode, a mode that alters illuminance of a light object according to distance between a light object and a light fixture, or combinations thereof.

[0014] In some examples, the rendering may be based, at least in part, on data corresponding to non-controllable actuators that are not controllable by the actuator control signals.

[0015] According to some examples, the sensory objects may include one or more light objects. In some such examples, the one or more light objects may include color information. In some examples, the rendering may involve implementing a color mixing and prioritization method when multiple light objects are being represented by a single light fixture.

[0016] In some examples, the sensory objects may include light objects and light object metadata. According to some examples, the light object metadata may include light object position data, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradient metadata, light object priority metadata, or combinations thereof.

[0017] According to some examples, the light object metadata may include light object layer metadata corresponding to two or more layers. The two or more layers may, for example,include an ambient layer, a dynamic layer, a custom layer, an overlay layer, a direct light object layer, an indirect light object layer, or combinations thereof.

[0018] In some examples, the rendering may involve intra-layer mixing, inter-layer blending, or both. According to some examples, the rendering may involve applying one or more light activation laws. In some examples, the rendering may involve creating a slice room effect, creating ambient fill, or both. Creating the slice room effect may involve producing wave fronts that propagate throughout a playback environment.

[0019] Some methods may involve obtaining user position data. The rendering may be based, at least in part, on the user position data.

[0020] Some or all of the operations, functions and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more computer-readable non-transitory media. Such non-transitory media may include one or more memory devices such as those described herein, including but not limited to one or more random access memory (RAM) devices, read-only memory (ROM) devices, etc. Accordingly, some innovative aspects of the subject matter described in this disclosure can be implemented in one or more computer-readable non-transitory media having software stored thereon.

[0021] At least some aspects of the present disclosure may be implemented via apparatus. For example, one or more devices may be capable of performing, at least in part, the methods disclosed herein. In some implementations, an apparatus may include an interface system and a control system. The control system may include one or more general purpose single- or multi-chip processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or combinations thereof. The control system may be configured to perform some or all of the disclosed methods.

[0022] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, the drawings, and the claims. Note that the relative dimensions of the following figures may not be drawn to scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Disclosed embodiments now be described, by way of example only, with reference to the accompanying drawings.

[0024] Figure 1A is a block diagram that shows examples of components of an apparatus capable of implementing various aspects of this disclosure.

[0025] Figure 1B shows example elements of an endpoint.

[0026] Figure 2 shows examples of actuator elements.

[0027] Figure 3 shows example elements of a system for the creation and playback of multi- sensory (MS) experiences.

[0028] Figure 4 shows example elements of a multi-sensory (MS) renderer.

[0029] Figure 5 shows example elements of another system for the creation and playback of MS experiences.

[0030] Figure 6A shows an example light map for a table lamp.

[0031] Figure 6B shows an example of an egocentric light map.

[0032] Figure 7 shows elements of a lightscape renderer according to some examples.

[0033] Figure 8 shows an example of a slice room effect.

[0034] Figure 9 shows an ambient fill example.

[0035] Figure 10 shows examples of alpha compositing responses

[0036] Figure 11 shows an example of a 3D ambient fill result.

[0037] Figures 12A, 12B, 12C, 12D, 12E and 12F show additional examples of ambient fill responses.

[0038] Figure 13 shows an example of a graphical user interface (GUI) that may be presented by a display device of the lightscape creation tool of Figure 5

[0039] Figure 14 shows another example of a graphical user interface (GUI) that may be presented by a display device of the lightscape creation tool of Figure 5.

[0040] Figure 15 is a flow diagram that outlines one example of a method that may be performed by an apparatus or system such as those disclosed herein.

[0041] Figure 16 is a flow diagram that outlines one example of a method that may be performed by an apparatus or system such as those disclosed herein. DETAILED DESCRIPTION

[0042] Described herein are techniques related to providing multi-sensory media content. In the following description, for purposes of explanation, numerous examples and specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be evident, however, to one skilled in the art that the present disclosure as defined bythe claims may include some or all of the features in these examples alone or in combination with other features described below, and may further include modifications and equivalents of the features and concepts described herein.

[0043] In the following description, various methods, processes and procedures are detailed. Although particular steps may be described in a certain order, such order is mainly for convenience and clarity. A particular step may be repeated more than once, may occur before or after other steps (even if those steps are otherwise described in another order), and may occur in parallel with other steps. A second step is required to follow a first step only when the first step must be completed before the second step is begun. Such a situation will be specifically pointed out when not clear from the context.

[0044] In this document, the terms “and”, “or” and “and / or” are used. Such terms are to be read as having an inclusive meaning. For example, “A and B” may mean at least the following: “both A and B”, “at least both A and B”. As another example, “A or B” may mean at least the following: “at least A”, “at least B”, “both A and B”, “at least both A and B”. As another example, “A and / or B” may mean at least the following: “A and B”, “A or B”. When an exclusive-or is intended, such will be specifically noted (e.g., “either A or B”, “at most one of A and B”).

[0045] This document describes various processing functions that are associated with structures such as blocks, elements, components, circuits, etc. In general, these structures may be implemented by one or more processors controlled by one or more computer programs.

[0046] As noted above, media content delivery has generally been focused on audio and video experiences. There has been limited delivery of multi-sensory (MS) content due to the customized nature of actuation.

[0047] This application describes methods for extending the creative palette for content creators, allowing spatial, MS experiences to be created and delivered at scale. Some such methods involve the introduction of new layers of abstraction, in order to allow authored MS experiences to be delivered to different endpoints, with different types of fixtures or actuators. As used herein, the term “endpoint” is synonymous with “playback environment” or simply “environment,” meaning an environment that includes one or more actuators that may be used to provide an MS experience. Such endpoints may include a room, such as theliving room of a home, a car, a cinema, a night club or other venue, etc. Some disclosed methods involve the creation, delivery and / or rendering of object-based sensory data, which may include sensory objects and corresponding sensory metadata. This abstraction allows creative intent to be implemented in an object-based format that does not require prior knowledge of the specific controller actuation, thereby enabling greater flexibility and scalability of fixtures and actuators across endpoints. An MS experience provided via object-based sensory data may be referred to herein as a “flexibly-scaled MS experience.”

[0048] Acronyms MS – multisensory MSIE – MS Immersive Experience AR – Augmented Reality VR – Virtual Reality PC – personal computer

[0049] Figure 1A is a block diagram that shows examples of components of an apparatus capable of implementing various aspects of this disclosure. As with other figures provided herein, the types and numbers of elements shown in Figure 1A are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. According to some examples, the apparatus 101 may be, or may include, a device that is configured for performing at least some of the methods disclosed herein, such as a smart audio device, a laptop computer, a cellular telephone, a tablet device, a smart home hub, etc. In some such implementations the apparatus 101 may be, or may include, a server that is configured for performing at least some of the methods disclosed herein.

[0050] In this example, the apparatus 101 includes at least an interface system 105 and a control system 110. In some implementations, the control system 110 may be configured for performing, at least in part, the methods disclosed herein. In some examples, the control system 110 may be configured for obtaining, via the interface system 105, actuator data for a set of controllable actuators. The set of controllable actuators may, for example, be specific to a particular playback environment. According to some examples, the control system 110 may be configured for obtaining, via the interface system 105, object-based sensory data including a set of sensory objects. In some examples, the object-based sensory data may include object-based sensory metadata corresponding to some or all of the sensory objects.The encoded the object-based sensory data may correspond to sensory effects such as lighting, haptics, airflow, one or more positional actuators, or combinations thereof, to be provided by a plurality of sensory actuators in an environment.

[0051] According to some examples, the control system 110 may be configured for implementing a multi-sensory (MS) renderer. Accordingly, in some examples, the control system 110 may be configured for rendering the object-based sensory data to produce actuator control signals, wherein the rendering is based at least in part on the actuator data. The MS renderer also may be referred to herein as a “sensory renderer,” because in some instances the MS renderer may be rendering only one type of MS data, such as object-based lighting data. According to some examples, the control system 110 may be configured for sending the actuator control signals to one or more controllable actuators of the set of controllable actuators.

[0052] According to some examples, the object-based sensory metadata may include sensory spatial metadata indicating at least a spatial position for rendering the object-based sensory metadata within the environment, an area for rendering the object-based sensory metadata within the environment, or combinations thereof. In some implementations, the object-based sensory metadata does not correspond to any particular sensory actuator in the environment. In some examples, the object-based sensory metadata may include abstracted sensory reproduction information allowing the sensory renderer to reproduce authored sensory effects, which also may be referred to herein as intended sensory effects, via various sensory actuator types, via various numbers of sensory actuators and from various sensory actuator positions in the environment.

[0053] In some examples, the control system 110 may be configured for obtaining, via the interface system 105, local context information. The local context information may include local time of day information, local weather information, local human behavior information, local user input, information regarding one or more viewer preferences, information regarding presence or absence of one or more viewers, ambient light information, viewing environment information, local viewer location information, local device usage information, local viewer activity information, or combinations thereof. In some such examples, the rendering process may be based, at least in part, on the local context information.

[0054] In some examples, the content bitstream also may include encoded audio objects synchronized with the encoded object-based sensory metadata. The audio objects mayinclude audio signals and corresponding audio object metadata. According to some examples, the audio objects may include audio signals and corresponding audio object metadata. The audio object metadata may include at least audio object spatial metadata indicating an audio object spatial position for rendering the audio signals within the environment. In some examples, the MS renderer also may be configured for rendering the audio objects.

[0055] The interface system 105 may include one or more network interfaces and / or one or more external device interfaces (such as one or more universal serial bus (USB) interfaces). According to some implementations, the interface system 105 may include one or more wireless interfaces. The interface system 105 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system and / or a gesture sensor system. In some examples, the interface system 105 may include one or more interfaces between the control system 110 and a memory system, such as the optional memory system 115 shown in Figure 1A. However, the control system 110 may include a memory system in some instances.

[0056] The control system 110 may, for example, include a general purpose single- or multi- chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.

[0057] In some implementations, the control system 110 may reside in more than one device. For example, a portion of the control system 110 may reside in a device within an environment (such as a laptop computer, a tablet computer, a smart audio device, etc.) and another portion of the control system 110 may reside in a device that is outside the environment, such as a server. In other examples, a portion of the control system 110 may reside in a device within an environment and another portion of the control system 110 may reside in one or more other devices of the environment.

[0058] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. The one or more non-transitory media may, for example, reside in the optional memory system 115 shown in Figure 1A and / or in the control system110. Accordingly, various innovative aspects of the subject matter described in this disclosure can be implemented in one or more non-transitory media having software stored thereon. The software may, for example, include instructions for controlling at least one device to process audio data. The software may, for example, be executable by one or more components of a control system such as the control system 110 of Figure 1A.

[0059] In some examples, the apparatus 101 may include the optional microphone system 120 shown in Figure 1A. The optional microphone system 120 may include one or more microphones. In some implementations, one or more of the microphones may be part of, or associated with, another device, such as a speaker of the speaker system, a smart audio device, etc.

[0060] According to some implementations, the apparatus 101 may include the optional actuator system 125 shown in Figure 1A. The optional actuator system 125 may include one or more loudspeakers, one or more haptic devices, one or more light fixtures, also referred to herein as luminaires, one or more fans or other air-moving devices, one or more display devices, including but not limited to one or more televisions, one or more positional actuators, one or more other types of devices for providing a MS experience, or combinations thereof. The term “light fixture” as used herein refers generally to any actuator that is configured to provide light. The term “light fixture” encompasses various types of light sources, including individual light sources such as light bulbs, groups of light sources such as light strips, light panels such as light-emitting diode (LED) panels, projectors, display devices such as television (TV) screens, etc. A “light fixture” may be moveable, and therefore the word “fixture” in this context does not mean that a light fixture is necessarily in a fixed position in space. The term “positional actuators” as used herein refers generally to devices that are configured to change a position or orientation of a person or object, such as motion simulator seats. Loudspeakers may sometimes be referred to herein as “speakers.” In some implementations, the optional actuator system 125 may include a display system including one or more displays, such as one or more light-emitting diode (LED) displays, one or more organic light-emitting diode (OLED) displays, etc. In some examples wherein the apparatus 101 includes a display system, the optional sensor system 130 may include a touch sensor system and / or a gesture sensor system proximate one or more displays of the display system. According to some such implementations, the control system 110 may be configured for controlling the display system to present agraphical user interface (GUI), such as a GUI related to implementing one of the methods disclosed herein.

[0061] In some implementations, the apparatus 101 may include the optional sensor system 130 shown in Figure 1A. The optional sensor system 130 may include a touch sensor system, a gesture sensor system, one or more cameras, etc.

[0062] This application describes methods for rendering and delivering a flexibly scaled multi-sensory (MS) immersive experience (MSIE) to different playback environments, which also may be referred to herein as endpoints. Such endpoints may include a room, such as the living room of a home, a car, a cinema, a night club or other venue, an AR / VR headset, a PC, a mobile device, etc.

[0063] Figure 1B shows example elements of an endpoint. In this example, the endpoint is a living room 1001 containing multiple actuators 008, some furniture 1010 and a person 1000—also referred to herein as a user—who will consume a flexibly-scaled MS experience. Actuators 008 are devices capable of altering the environment 1001 that the user 1000 is in. Actuators 008 may include one or more haptic devices, one or more light fixtures, also referred to herein as luminaires, one or more fans or other air-moving devices, one or more display devices, including but not limited to one or more televisions, one or more positional actuators, one or more other types of devices for providing a MS experience, or combinations thereof.

[0064] The number of actuators 008, the arrangement of actuators 008 and the capabilities of actuators 008 in the space 1001 may vary significantly between different endpoint types. For example, the number, arrangement and capabilities of actuators 008 in a car will generally be different from the number, arrangement and capabilities of actuators 008 in a living room, a night club, etc. In many implementations, the number, arrangement and / or capabilities of actuators 008 may vary significantly between different instances of the same type, e.g., between a small living room with 2 actuators 008 and a large living room with 16 actuators 008. The present disclosure describes various methods for creating and delivering flexibly-scaled MSIEs to these non-homogenous endpoints.

[0065] Figure 2 shows examples of actuator elements. In this example, the actuator is a luminaire 1100, which includes a network module 1101, a control module 1102 and a light emitter 1103. According to this example, the light emitter 1103 includes one or more light-emitting devices, such as light-emitting diodes, which are configured to emit light into an environment in which the luminaire 1100 resides. In this example, the network module 1101 is configured to provide network connectivity to one or more other devices in the space, such as a device that sends commands to control the emission of light by the luminaire 1100. According to this example, the network module 1101 is an instance of the interface system 105 of Figure 1A. In this example, the control module 1102 is configured to receive signals via the network module 1101 and to control the light emitter 1103 accordingly. According to this example, the control module 1102 is an instance of the control system 110 of Figure 1A.

[0066] Other examples of actuators also may include a network module 1101 and a control module 1102, but may include other types of actuating elements. Some such actuators may include one or more haptic devices, one or more fans or other air-moving devices, one or more positional actuators, one or more loudspeakers, one or more display devices, etc.

[0067] Figure 3 shows example elements of a system for the creation and playback of multi- sensory (MS) experiences. As with other figures provided herein, the types and numbers of elements shown in Figure 3 are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. According to some examples, system 300 may be, or may include, one or more devices configured for performing at least some of the methods disclosed herein. In some examples, system 300 may include one or more instances of the control system 110 of Figure 1A that are configured for performing at least some of the methods disclosed herein.

[0068] According to the examples in the present disclosure, creating and providing an object-based MS Immersive Experience (MSIE) approach involves the application of a suite of technologies for creation, delivery and rendering of object-based sensory data, which may include sensory objects and corresponding sensory metadata, to the actuators 008. Some examples are described in the following paragraphs.

[0069] Object-Based Representation: In various disclosed implementations, multi-sensory (MS) effects are represented using what may be referred to herein as multi-sensory (MS) objects, or simply as “sensory objects.” According to some such implementations, properties such as layer-type and priority may be assigned to and associated with attached to each sensory object, enabling content creators’ intent to be represented in the rendered experiences. Detailed examples of sensory object properties are described below.

[0070] In this example, system 300 includes a content creation tool 000 that is configured for designing multi-sensory (MS) immersive content and for outputting object-based sensory data 005, either separately or in conjunction with corresponding audio data 011 and / or video data 012, depending on the particular implementation. The object-based sensory data 005 may include time stamp information, as well as information indicating the type of sensory object, the sensory object properties, etc. In this example, the object-based sensory data 005 is not “channel-based” data that corresponds to one or more particular sensory actuators in a playback environment, but instead is generalized for a wide range of playback environments with a wide range of actuator types, numbers of actuators, etc. In some examples, the object-based sensory data 005 may include object-based light data, object-based haptic data, object-based air flow data, or object-based positional actuator data, object-based olfactory data, object-based smoke data, object based data for one or more other types of sensor effects, or combinations thereof. According to some examples, the object-based sensory data 005 may include sensory objects and corresponding sensory metadata. For example, if the object-based sensory data 005 includes object-based light data, the object-based light data may include light object position metadata, light object color metadata, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradient metadata, light object priority metadata, light object layer metadata, or combinations thereof. In some examples, the object-based sensory data 005 may include time data, such as time stamp information. Although the content creation tool 000 is shown providing a stream of object-based sensory data 005 to the experience player 002 in this example, in alternative examples the content creation tool 000 may produce object-based sensory data 005 that is stored for subsequent use. Examples of graphical user interfaces for a light-object-based content creation tool are described below. Examples of MS Object Properties

[0071] Following is a non-exhaustive list of possible properties of MS objects: • Priority; • Layer; • Mixing Mode; • Persistence; • Effect; and • Spatial Panning Law.

[0072] Effect As used herein, the “effect” of a MS object is a synonym for the type of MS object. An “effect” is, or indicates, the sensory effect that the MS object is providing. If an MS object is a light object, its effect will involve providing direct or indirect light. If an MS object is a haptic object, its effect will involve providing some type of haptic feedback. If an MS object is an air flow object, its effect will involve providing some type of air flow. As described in more detail below, some examples involve other “effect” categories.

[0073] Persistence Some MS objects may contain a persistence property in their metadata. For example, as an moveable MS object moves around in a scene, the moveable MS object may persist for some period of time at locations that the moveable MS object passes through. That period of time may be indicated by persistence metadata. In some implementations, the MS renderer is responsible for constructing and maintaining the persistence state.

[0074] Layers According to some examples, individual MS objects may be assigned to “layers,” in which MS objects are grouped together according to one or more shared characteristics. For example, layers may group MS objects together according to their intended effect or type, which may include but are not limited to the following: - Mood / Ambience - Informational - Punctuational / Attention Alternatively, or additionally, in some examples, layers may be used to group MS objects together according to shared properties, which may include but are not limited to the following: - Color - Intensity - Size - Shape - Position - Region in space

[0075] PriorityIn some examples, MS objects may have a priority property that enables the renderer to determine which object(s) should take priority in an environment in which MS objects are contending for limited actuators. For example, if multiple light objects overlap with a single light fixture at a time during which all of the light objects are scheduled to be rendered, a renderer may refer to the priority of each light object in order to determine which light object(s) will be rendered. In some examples, priority may be defined between layers or within layers. According to some examples, priority may be linked to specific properties such as intensity. In some examples, priority may be defined temporally: for example, the most recent MS object to be rendered may take precedence over MS objects that have been rendered earlier. According to some examples, priority may be used to specify MS objects or layers that should be rendered regardless of the limitations of a particular actuator system in a playback environment.

[0076] Spatial Panning Laws Spatial panning laws may define a MS object’s movement across a space, how a MS object affects actuators as it moves between them, etc.

[0077] Mixing Mode The mixing mode may specify how multiple objects are multiplexed onto a single actuator. In some examples, mixing modes may include one or more of the following: - Max mode: select the MS object which activates an actuator the most; - Mix mode: mix in some or all the objects according to a rule set, for example by summing activation levels, taking the average of activation levels, mixing color according to activation level or priority level, etc.; - MaxNmix: mix in the top N MS objects (by activation level), according to a rule set.

[0078] According to some examples, more general metadata for an entire multi-sensory content file, instead of (or in addition to) per-object metadata may be defined. For example, MS content files may include metadata such as trim passes or mastering environment.

[0079] Trim Controls

[0080] What are referred to in the context of Dolby Vision™ as “trim controls” may act as guidance on how to modulate the default rendering algorithm for specific environments orconditions at the endpoint. Trim controls may specify ranges and / or default values for various properties, including saturation, tone detail, gamma, etc. For example, there may be automotive trim controls, which provide specific defaults and / or rule sets for rendering in automotive environments, for example guidance that includes only objects of a certain priority or layer. Other examples may provide trim controls for environments with limited, complex or sparse multisensory actuators.

[0081] Mastering Environment A single piece of multisensory content may include metadata on the properties of the mastering environment such as room size, reflectivity and ambient bias lighting level. The specific properties may differ depending on the desired endpoint actuators. Mastering environment information can aid in providing reference points for rendering in a playback environment.

[0082] MS Object Renderer: Various disclosed implementations provide a renderer that is configured render MS effects to actuators in a playback environment. According to this example, system 300 includes a MS renderer 001 that is configured to render object-based sensory data 005 to actuator control signals 310, based at least in part on environment and actuator data 004. In this example, the MS renderer 001 is configured to output the actuator control signals 310 to MS controllers 003, which are configured to control the actuators 008. In some examples, the MS renderer 001 may be configured to receive light objects and object-based lighting metadata indicating an intended lighting environment, as well as lighting information regarding a local lighting environment. The lighting information is one general type of environment and actuator data 004, and may include one or more characteristics of one or more controllable light sources in the local lighting environment. In some examples, the MS renderer 001 may be configured to determine a drive level for each of the one or more controllable light sources that approximates the intended lighting environment. According to some examples, the MS renderer 001 (or one of the MS controllers 003) may be configured to output the drive level to at least one of the controllable light sources. Some alternative examples may include a separate renderer for each type of actuator 008, such as one renderer for light fixtures, another renderer for haptic devices, another renderer for air flow devices, etc. In other implementations, a single renderer may be configured as a MS renderer and as an audio renderer and / or as a video renderer. In some implementations, the MS renderer 001 may be configured to adapt tochanging conditions. Some examples of MS renderer 001 implementations are described in more detail below.

[0083] The environment and actuator data 004 may include what are referred to herein as “room descriptors” that describe actuator locations (e.g., according to an x,y,z coordinate system or a spherical coordinate system). In some examples, the environment and actuator data 004 may indicate actuator orientation and / or placement properties (e.g., directional and north-facing, omnidirectional, occlusion information, etc.). According to some examples, the environment and actuator data 004 may indicate actuator orientation and / or placement properties according to a 3x3 matrix, in which three elements (for example, the elements of the first row) represent spatial position (x,y,z), three other elements (for example, the elements of the second row) represent orientation (roll, pitch, yaw), and three other elements (for example, the elements of the third row) indicate a scale or size (sx, sy, sz). In some examples, the environment and actuator data 004 may include device descriptors that describe the actuator properties relevant to the MS renderer 001, such as intensity range and color gamut of a light fixture, the air flow speed range and direction(s) for an air-moving device, etc.

[0084] In this example, system 300 includes an experience player 002 that is configured to receive object-based sensory data 005’, audio data 011’ and video data 012’, and to provide object-based sensory data 005 to the MS renderer 001, to provide audio data 011 to the audio renderer 006 and to provide the video data 012 to the video renderer 007. In this example, the reference numbers for the object-based sensory data 005’, audio data 011’ and video data 012’ received by the experience player 002 include primes (‘), in order to suggest that the data may in some instances be encoded. Likewise, the object-based sensory data 005, audio data 011 and video data 012 output by the experience player 002 do not include primes, in order to suggest that the data may in some instances have been decoded by the experience player 002. According to some examples, the experience player 002 may be a media player, a game engine or personal computer or mobile device, or a component integrated in an television, DVD player, sound bar, set top box, or a service provider media device such as a Chromecast, Apple TV device, or Amazon Fire TV. In some examples, the experience player 002 may be configured to receive encoded object-based sensory data 005’ along with encoded audio data 011’ and / or encoded video data 012’. In some such examples, the encoded object-based sensory data 005’ may be received as part of the same bitstream with the encoded audio data 011’ and / or the encoded video data 012’. Some examples aredescribed in more detail below. According to some examples, the experience player 002 may be configured to extract the object-based sensory data 005’ from the content bitstream and to provide decoded object-based sensory data 005 to the MS renderer 001, to provide decoded audio data 011 to the audio renderer 006 and to provide decoded video data 012 to the video renderer 007. In some examples, time stamp information in the object-based sensory data 005; may be used—for example, by the experience player 102, the MS renderer 001, the audio renderer 106, the video renderer 107, or all of them—to synchronize effects relating to the object-based sensory data 005’ with the audio data 111’ and / or the video data 112’, which may also include time stamp information.

[0085] According to this example, system 300 includes MS controllers 003 that are configured to communicate with a variety of actuator types using application program interfaces (APIs) or one or more similar interfaces. Generally speaking, each actuator will require a specific type of control signal to produce the desired output from the renderer. According to this example, the MS controllers 003 are configured to map outputs from the MS renderer 001 to control signals for each actuator. For example, a Philips Hue™ light bulb receives control information in a particular format to turn the light on, with a particular saturation, brightness and hue, and a digital representation of the desired drive level. In some alternative examples, the MS renderer 001 also may be configured to implement some or all of the MS controllers 003. For example, the MS renderer 001 also may be configured to implement one or more lighting-based APIs but not haptic-based APIs, or vice versa.

[0086] In some examples, room descriptors also may describe the size and orientation of the playback environment itself, to establish a relative or absolute coordinate system to which all objects are positioned. For example, in a living room a display screen may be regarded as the front, in some instances the front and center, and the floor and ceiling may be regarded as the vertical bounds. In some such examples, the room descriptors also may also indicate bounds corresponding with the left, right, front, and rear, walls relative to the front position. According to some examples, the room descriptor also may be provided in terms of a matrix, such as a 3x3 matrix. This room descriptor information is useful in describing the physical dimensions of the playback environment, for example in physical units of distance such as meters. In some such examples, sensory object locations, sensory object sizes, and sensory object orientations may be described in units that are relative to the room size, for example in a range from -1 to 1. Room descriptors may also describe a preferred viewing position, in some instances according to a matrix.

[0087] The types, numbers and arrangements of the actuators 008 will generally vary according to the particular implementation. In some examples, actuators 008 may include lights and / or light strips (also referred to herein as “luminaires”), vibrational motors, air flow generators, positional actuators, or combinations thereof.

[0088] Similarly, the types, numbers and arrangements of the loudspeakers 009 and the display devices 010 will generally vary according to the particular implementation. In the examples shown in Figure 3, audio data 011 and video data 012 are rendered by the audio renderer 006 and the video renderer 007 to the loudspeakers 009 and display devices 010, respectively.

[0089] As noted above, according to some implementations the system 300 may include one or more instances of the control system 110 of Figure 1A that are configured for performing at least some of the methods disclosed herein. In some such examples, one instance of the control system 110 may implement the content creation tool 000 and another instance of the control system 110 may implement the experience player 002. In some examples, one instance of the control system 110 may implement the audio renderer 006, the video renderer 007, the multi-sensory renderer 001, or combinations thereof. According to some examples, an instance of the control system 110 that is configured to implement the experience player 002 may also be configured to implement the audio renderer 006, the video renderer 007, the multi-sensory renderer 001, or combinations thereof.

[0090] Figure 4 shows example elements of a multi-sensory (MS) renderer. As with other figures provided herein, the types and numbers of elements shown in Figure 4 are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. According to this example, the MS renderer 001 is an instance of the MS renderer 001 that is described with reference to Figure 3. In some examples, the MS renderer 001 may be implemented by one or more instances of the control system 110 of Figure 1A.

[0091] According to this example, Figure 4 includes the following elements: • 004: Environment and actuator data, which may be as described with reference to Figure 3; • 005: Object-based sensory data 005, which may be as described with reference to Figure 3;• 423 an actuator map (AM) that indicates the locations of at least the controllable actuators 008 in a particular playback environment; • 450 a projection module configured to project the MS objects of the object-based sensory data 005 based, at least in part, on the AM 423. The MS objects may also be referred to herein as “sensory objects,” because in some instances only one type of sensory object—such as only haptic objects or only light objects—may be present in the object-based sensory data 005. In this example, the projection module 450 is configured to project the MS objects based, at least in part, on sensory object metadata, which may include at least sensory object location metadata and sensory object size metadata; • 440 an actuator activation matrix (AAM) that is output by the projection module 450 according to this example. The AAM may, for example, indicate sensory objects, if any, that are currently encompassing a volume within the playback environment corresponding to one or more corresponding actuators. For example, the AAM may indicate whether the light object location metadata and light object size metadata of a light object indicate that a particular light fixture is within a volume of the playback environment corresponding to the location and size of the light object; • 451 a mixing module configured to convert the AAM 440 into actuator control signals, based at least in part on the environment and actuator data and the renderer configuration data; • 452 optional renderer configuration data, which may include information regarding one or more settings for the MS renderer 001 such as settings to indicate the desired dynamism, mode, etc. In some examples, the renderer configuration data 452 may be changed automatically using context-aware systems such as those described in more detail below; and • 310: actuator control signals, which may be as described with reference to Figure 3. In some examples, the actuator control signals 310 may be sent to individual actuators 008, whereas in other examples the actuator control signals 310 may be sent to MS controllers 003, which may be configured to send appropriate control signals to various types of actuators 008.

[0092] According to some examples, the AAM 440 is a matrix describing “how much” a sensory object projects itself onto each actuator according to the actuator map 423. In some examples, the AAM 440 may be a real matrix of size NOby NA, where NOrepresents thenumber of sensor objects and NA represents the number of controllable actuators in the environment. In this example, the mixing module 451 is configured to produce the actuator control signals 310 based at least in part on the AAM 440 and the environment and actuator data 004. In some examples, the mixing module 451 may be configured to produce the actuator control signals 310 based at least in part on the optional renderer configuration data 452. According to some examples, the mixing module 451 may be configured to produce the actuator control signals 310 based at least in part on sensory object metadata—which may be received as part of the object-based sensory data 005, as shown in Figure 4—such as mixing and panning laws associated with at least one sensory object.

[0093] In some examples, the mixing module 451 may be configured to produce the actuator control signals 310 based at least in part on one or more of the following: 1. Thresholding elements of the AAM 440; 2. Taking the maximum of a particular column of the AAM 440—in other words, taking the sensory object that activated a particular actuator the most as the output; 3. Taking any combination of the top N and performing at least one of the following: o Mixing the objects together in the actuator channel; o Pushing objects to adjacent channels.

[0094] In some embodiments, the projection module 450 may be configured to generate a sensory object image using the sensory object’s spatial coordinates—for example, x,y,z coordinates—and the sensory object’s size to produce In(x,y,z), where Inrepresents the sensory object image for the nthsensor object. Then, in some examples, the projection module 450 may be configured to compute, for every column (actuator index) of the AAM 440, the nthrow (object index), by taking the inner product of this object image and the actuator map corresponding to that actuator. The object image, actuator map and dot product may be produced and performed by the projection module 450 in any spatial domain that is convenient including, without limitation, polar, cylindrical or rectangular coordinate systems.

[0095] Some implementations may involve implementing what may be referred to herein as “repellers,” which may be used to avoid potentially undesirable sensory effects, such as lighting effects, which may be caused when sensory objects are positioned in one or more areas of a playback environment. In some such examples, repeller data may be included with the environment and actuator data 004 and the AM 423, and therefore may be part of the information that is input to the projection module 450. In some such examples, the spatialcoordinate of the MS objects will be augmented when projecting them onto the AM 423 to produce the AAM 440. Multi-Sensory Rendering Synchronization

[0096] Object-based MS rendering involves different modalities being rendered flexibly to the endpoint / playback environment. Endpoints have differing capabilities according to various factors, including but not limited to the following: • The number of actuators, • The modalities of those actuators (e.g., light fixture vs. air flow control device vs. haptic device); • The types of those actuators (e.g., a white smart light vs. a RGB smart light, or a haptic vest vs. a haptic seat cushion) and • The location / layout of those actuators.

[0097] In order to render object-based sensory content to any endpoint, some processing of the object signals, e.g. intensities, colors, patterns etc., will generally need to be done. The processing of each modality’s signal path should not alter the relative phase of certain features within the object signals. For example, suppose that a lightning strike is presented in both the haptics and lightscape modalities. The signal processing chain for the corresponding actuator control signals should not result in a time delay of either type of sensory object signal—haptic or light—sufficient to alter the perceived synchronization of the two modalities. The level of required synchronization may depend on various factors, such as whether the experience is interactive and what other modalities are involved in the experience. Maximum time difference values may, for example, range from approximately 10ms to 100ms, depending on the particular context. HAPTICS Rendering of object-based haptics content

[0098] Object-based haptics content conveys sensory aspects of the scene through an abstract sensory representation rather than a channel-based scheme only. For example, instead of defining haptics content as a single-channel time-dependent amplitude signal only, that is in turn played out of a particular haptics actuator such as a vibro-tactile motor in a vest the user wears, object-based haptics content may be defined by the sensations that it isintended to convey. More specifically, in one example, we may have a haptic object representing a collision haptic sensory effect. Associated with this object is: • The haptic object’s spatial location; • The spatial direction / vector of the haptic effect; • The intensity of the haptic effect; • Haptic spatial and temporal frequency data; and • A time-dependent amplitude signal.

[0099] According to some examples, a haptic object of this type may be created automatically in an interactive experience such as a video game, e.g. in a car racing game when another car hits a player’s car from behind. In this example, the MS renderer will determine how to render the spatial modality of this effect to the set of haptic actuators in the endpoint. In some examples, the renderer does this according to information about the following: • The type(s) of haptic devices available, e.g., haptic vest vs. haptic glove vs. haptic seat cushion vs. haptic controller; • The locale of each haptic device with respect to the user(s) (some haptic devices may not be coupled to the user(s), e.g., a floor- or seat-mounted shaker); • The type of actuation each haptic device provides, e.g. kinesthetic vs. vibro-tactile; • The on- and off-set delay of each haptic device (in other words, how fast each haptic device can turn on and off); • The dynamic response of each haptic device (how much the amplitude can vary); • The time-frequency response of each haptic device (what time-frequencies the haptic device can provide); • The spatial distribution of addressable actuators within each haptic device: for example, a haptic vest may have dozens of addressable haptics actuators distributed over the user’s torso; and • The time-response of any haptic sensors used to render closed-loop haptic effects (e.g., an active force-feedback kinesthetic haptic device.

[0100] These attributes of the haptics modality of the endpoint will inform the render how best to render a particular haptic effect. Consider the car crash effect example again. In this example, a player is wearing a haptic vest, a haptic arm band and haptic gloves. According to this example, a haptic shockwave effect is spatially located at the place where the car hascollided into the player. The shockwave vector is dictated by the relative velocity of the player’s car and the car that has hit the player. The spatial and temporal frequency spectra of the shockwave effect are authored according to the type of material the virtual cars are intended to be made of, amongst other virtual world properties. The renderer then renders this shockwave through the set of haptics devices in the endpoint, according to the shockwave vector and the physical location of the haptics devices relative to the user.

[0101] The signals sent to each specific actuator are preferably provided so that the sensory effect is congruent across all of the (potentially heterogenous) actuators available. For example, the renderer may not render very high frequencies to just one of the haptic actuators (e.g., the haptic arm band) due to capabilities lacking in other actuators. Otherwise, as the shockwave moves through the player’s body, because the haptic vest and haptic gloves the user is wearing do not have the capability to render such high frequencies, there would a degradation of the haptic effect perceived by the user as the wave moves through the vest, into the arm band and finally into the gloves.

[0102] Some types of abstract haptic effects include: • Shockwave effects, such as described above; • Barrier effects, such as haptic effects which are used to represent spatial limitations of a virtual world, for example in a video game. If there are kinesthetic actuators on input devices (e.g., force feedback on a steering wheel or joystick), either active or resistive, then rendering of such an effect can be done through the resistive force applied to the users input. If no such actuators are available in the endpoint then in some examples vibro-tactile feedback may be rendered that is congruent with the collision of the in-game avatar with a barrier; • Presence, for example to indicate the presence of a large object approaching the scene such as a train. This type of haptic effect may be rendered using a low time- frequency rumbling of some haptic devices’ actuators. This type of haptic effect may also be rendered through contact spatial feedback applied as pressure from air- cuffs; • User interface feedback, such as clicks from a virtual button. For example, this type of haptic effect may be rendered to the closest actuator on the body of the user that performed the click, for example haptic gloves that the user is wearing. Alternatively, or additionally, this type of haptic effect may also be rendered to a shaker coupled to the chair in which the user is sitting. This type of haptic effectmay, for example, be defined using time-dependent amplitude signals. However, such signals may be altered (modulated, frequency-shifted, etc.) in order to best suit the haptic device(s) that will be providing the haptic effect; • Movement. These haptic effects are designed so that the user perceives some form of motion. These haptic effects may be rendered by an actuator that actually moves the user, e.g. a moving platform / seat. In some examples, an actuator may provide a secondary modality (via video, for example) to enhance the motion being rendered; and • Triggered sequences. These haptic effects are characterized mainly by their time- dependent amplitude signals. Such signals may be rendered to multiple actuators and may be augmented when doing so. Such augmentations may include splitting a signal in either time or frequency across multiple actuators. Some examples may involve augmenting the signal itself so that the sum of the haptic actuator outputs does not match the original signal. Spatial and Non-Spatial Effects

[0103] Spatial effects are those which are constructed in a way that convey some spatial information of the multi-sensory scene being rendered. For example, if the playback environment is a room, a shockwave moving through the room would be rendered differently to each haptic device given its location within the room, according to the position and size of one or more haptic objects being rendered at a particular time.

[0104] Non-spatial effects may, in some examples, target particular locations on the user regardless of the user’s location or orientation. One example is a haptic device that provides a swelling vibration on the users back to indicate immediate danger. Another example is a haptic device that provides a sharp vibration to indicate an injury to a particular body area.

[0105] Some effects may be non-diegetic effects. Such effects are typically associated with user interface feedback, such as a haptic sensation to indicate the user completed a level or has clicked a button on a menu item. Non-diegetic effects may be either spatial or non- spatial.Haptic Device Type

[0106] Receiving information regarding the different types of haptics devices available at the endpoint enables the renderer to determine what kinds of perceived effects and rendering strategies are available to it. For example, local haptics device data indicating that the user is wearing both haptic gloves and a vibro-tactile vest—or at least local haptics device data indicating that that haptic gloves and a vibro-tactile vest are present in the playback environment—allows the renderer to render a congruent recoil effect across the two devices when a user shoots a gun in a virtual world. The actual actuator control signals sent to the haptic devices may be different than in the situation where only a single device is available. For example, if the user is only wearing a vest, the actuator control signals used to actuate the vest may differ with regard to the timing of the onset, the maximum amplitude, frequency and decay time of the actuator control signals, or combinations thereof. Location of the Devices

[0107] Knowledge of the location of the haptics devices across the endpoint enables the renderer to render spatial effects congruently. For example, knowledge of the location of the shaker motors in a lounge enables the renderer to produce actuator control signals to each of the shaker motors in the lounge in a way to convey spatial effects such as a shockwave propagating through the room. Additionally, knowledge of where wearable haptics devices, whilst implicit by their type, e.g. a glove is on the user’s hand, may also bd used by the renderer to convey spatial effects in addition to non-spatial effects. Types of Actuation Provided by Haptic Devices

[0108] Haptic devices can provide a range of different actuations and thus perceived sensations. These are typically classed in two basic categories: 1. vibro-tactile , e.g. vibrations; or 2. Kinesthetic, e.g., resistive or active force feedback.

[0109] Either category of actuations may be static or dynamic, where dynamic effects are altered in real time according to some sensor input. Examples include a touch screen rendering a texture using a vibro-tactile actuator and a position sensor measuring the user’s finger position(s).

[0110] Moreover, the physical construction of such actuators varies widely and affects many other attributes of the device. An example of this is the onset delay or time-frequency response that varies significantly across the following haptic device types: • Eccentric rotating mass; • Linear resonant actuator; • Piezoelectric actuator; and • Linear magnetic ram.

[0111] The renderer should be configured to account for the onset delay of a particular haptics device type when rendering signals to be actuated by the haptics devices in the endpoint. The On- and Off-Set Delays of the Haptic Devices

[0112] The onset delay of the haptic device refers to the delay between the time that an actuator control signal is sent to the device and the device’s physical response. The off-set delay refers to the delay between the time that an actuator control signal is sent to zero the output of the device and the time the device stops actuating. The Time-Frequency Response

[0113] The time-frequency response refers to the frequency range of the signal amplitude as a function of time that the haptic device can actuate at steady state. The Spatial-Frequency Response

[0114] The spatial-frequency response refers to the frequency range of the signal amplitude as a function of the spacing of actuators of a haptic device. Devices with closely-spaced actuators have higher spatial-frequency responses. Dynamic Range

[0115] Dynamic range refers to the differences between the minimum and maximum amplitude of the physical actuation. Characteristics of Sensors in Closed-Loop Haptics Devices

[0116] Some dynamic effects use sensors to update the actuation signal as a function of some observed state. The sampling frequencies, both temporal and spatial along with thenoise characteristics will limit the capability of the control loop updating the actuator providing the dynamic effect. AIRFLOW

[0117] Another modality that some multi-sensory immersive experiences (MSIE) may use is airflow. The airflow may, for example, be rendered congruently with one or more other modalities such as audio, video, light-effects and / or haptics. Rather than only specialized (e.g. channel-based) setups for 4D experiences in cinemas which may include “wind effects,” some airflow effects may be provided at other endpoints that may typically include airflow, such as a car or a living room. Rather than a channel-based system, the airflow sensory effects may be represented as an airflow object that may include properties such as: • Spatial location; • Direction of the intended airflow effect; • Intensity / airflow speed; and / or • Air temperature.

[0118] Some examples of air flow objects may be used to represent the movement of a bird flying past. To render to the airflow actuators at the endpoint, the MS renderer 001 may be provided with information regarding: • The type of airflow devices e.g. fan, air conditioning, heating; • The position of each airflow device relative to the user’s location, or relative to an expected of the user; • The capabilities of the airflow device, e.g., the airflow device’s ability to control direction, airflow and temperature; • The level of control of each actuator, e.g., airflow speed, temperature range; and • The response time of each actuator, e.g., how long does it take to reach a chosen speed. Some Examples of Airflow Use in Different Endpoints

[0119] In a vehicle such as a car, the object-based metadata can be used to create experiences such as: • Mimicking “chills down your spine” during a horror movie or gaming piece of content with airflow down the chair; • Simulating the movement of a bird flying past; and / or• Create a gentle breeze in a seascape.

[0120] In the small enclosed space of a typical vehicle, temperature changes may be possible to achieve over relatively shorter periods of time—as compared to temperature changes in a larger environment, such as a living room environment. In one example, the MS renderer 001 may cause an increasing air temperature as a player enters a “lava level” or other hot area during a game. Some examples may include other elements, such as confetti in the air vents to celebrate an event, such as the celebration of a goal made by the user’s your favorite football team.

[0121] In a living space or other room, airflow may be synchronized to the breathing rhythm of a guided meditation in one example. In another example, airflow may be synchronized to the intensity of a workout, with increased airflow or decreased temperature as intensity increases. In some examples, there may be relatively less control over spatial aspects during rendering. For example, many existing airflow actuators are optimized for heating and / or air conditioning rather than for providing spatially diverse sensory actuation. Combinations of Lights, Airflow and Haptics Car Examples

[0122] In some examples, there may be a user interface on the steering wheel or on a touchscreen near or in the dashboard. According to some examples, the following actuators may be present in the car: 1. Individually addressable lights, spatially distributed around the car as follows: o on the dashboard; o under the footwells; o on the doors; and o in the center console. 2. Individually controllable air conditioning / heating outlets distributed around the car as follows: o In the front dashboard; o Under the footwells; o In the center console facing the rear seats; o On the side pillars; o In the seats; ando Directed to the windscreens (for defogging). 3. Individually controllable seats with vibro-tractile haptics; and 4. Individually controllable floor mats with vibro-tactile haptics.

[0123] In this example, the modalities supported by these actuators include the following: • Lights across the individually addressable LEDs in the car, plus the indicator lights on the dash and steering wheel; • Air flow via the controllable air conditioning vents; • Haptics, including: o Steering wheel: tactile vibration feedback; o Dash touchscreen: tactile vibration feedback and texture rendering; and o Seats: tactile vibrations and movement.

[0124] In one example, a live music stream is being rendered to four users sitting in the front seats. In this example, the MS renderer 001 attempts to optimize the experience for multiple viewing positions. During the build-up before the artist has taken the stage and the previous acts have finished, the content contains: • Interlude music; • Low intensity lighting; and • Haptic content representing the moshing of the crowd.

[0125] In addition to the rendered audio and video stream, the light content contains ambient light objects that are moving slowly around the scene. These may be rendered using one of the ambient layer methods disclosed herein, for example such that there is no spatial priority given to any user’s perspective. In some examples, the haptic content may be spatially concentrated in the lower time-frequency spectrum and may be rendered only by the vibro- tactile motors in the floor mats.

[0126] According to this example, pyrotechnic events during the music stream correspond to multi-sensory-sensory content including: • Light objects that spatially correspond to the location of the pyrotechnics at the event; and • Haptic objects to reinforce the dynamism of the pyrotechnics via a shockwave effect.

[0127] In this example, the MS renderer 001 renders both the light objects and the haptic objects spatially. Light objects may, for example, be rendered in the car such that eachperson in the car perceives the light objects to come from the left if the pyrotechnics content is located at the left of the scene. In this example, only lights on the left of the car are actuated. Haptics may be rendered across both the seats and floor mats in a way that conveys directionality to each user individually.

[0128] At the end of the concert the pyrotechnics are present in the audio content and both pyrotechnics and confetti are present in the video content. In addition to rendering light objects and haptic objects corresponding to the pyrotechnics as above, the effect of the confetti firing may be rendered using the airflow modality. For example, the individually controllable air flow vents of the HVAC system may be pulsed. Living Room Examples

[0129] In this implementation, in addition to an audio / visual (AV) system that includes multiple loudspeakers and a television, the following actuators and related controls are available in the living room: • A haptics vest that the user—also referred to as a player—is wearing; • Haptics shakers mounted to the seat in which the player is sitting; • A (haptics) controllable smart watch; • Smart lights spatially distributed around the room; • A wireless controller; and • An addressable air-flow bar (AFB), which includes an array of individually controllable fans directed to the user (similar to HVAC vents in the front dashboard of a car).

[0130] In this example, the user is playing a first person shooter game and the game contains a scene in which a destructive hurricane moves through the level. As it does so, in-game objects are thrown around and some hit the player. Haptics objects rendered by the MS renderer 001 cause a shockwave effect to be provided through all of the haptics devices that the user can perceive. The actuator control signals sent to each device may be optimized according to the intensity of the impact of the in-game objects, the direction(s) of the impact and the capabilities and location of each actuator (as described earlier).

[0131] At a time before the user is struck by an in-game object, the multi-sensory content contains a haptic object corresponding to a non-spatial rumble, one or more airflow objects corresponding to directional airflow; and one or more light objects corresponding tolightning. The MS renderer 001 renders the non-spatial rumble to the haptics devices. The actuator control signals sent to each haptics device may be rendered such that the ensemble of actuator control signals across the haptics array is congruent in perceived onset time, intensity and frequency. In some examples, the frequency content of the actuator control signals sent to the smart watch may be low-pass filtered, so that they are congruent with the frequency-limited capability of the vest, which is proximate to the watch. The MS renderer 001 may render the one or more airflow objects to actuator control signals for the AFB such that the air flow in the room is congruent with the location and look direction of the player in the game, as well as the hurricane direction itself. Lightning may be rendered across all modalities as (1) a white flash across lights that are located in suitable locations, e.g., in or on the ceiling; and (2) an impulsive rumble in the user’s wearable haptics and seat shaker.

[0132] When the user is struck by an in-game object, a directional shockwave may be rendered to the haptics devices. In some examples, a corresponding airflow impulse may be rendered. According to some examples, a damage take effect, indicating the amount of damage caused to the player by being struck by the in-game object, may be rendered by the lights.

[0133] In some such examples, signals may be rendered spatially to the haptics devices such that a perceived shockwave moves across the player’s body and the room. The MS renderer 001 may provide such effects according to actuator location information indicating the haptics devices locations relative to one another. The MS renderer 001 may provide the shockwave vector and position according to the actuator location information in addition to actuator capability information. According to some examples, a non-directional air flow impulse may be rendered, e.g., all the air vents of the AFB may be turned up briefly to reinforce the haptic modality. In some examples, at the same time, a red vignette may be rendered to the light strip surrounding the TV, indicating to the player that the player took damage in the game.

[0134] Figure 5 shows example elements of another system for the creation and playback of MS experiences. As with other figures provided herein, the types and numbers of elements shown in Figure 5 are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. According to some examples, system 500 may be, or may include, one or more devices configured for performing at least some of the methods disclosed herein. In some examples, system 500may include one or more instances of the control system 110 of Figure 1A that are configured for performing at least some of the methods disclosed herein.

[0135] According to this example, the system shown in Figure 5 is an instance of the system shown in Figure 3. In this example, the system shown in Figure 5 is a “lightscape” embodiment in which video, audio and light effects are combined to create the MS experience.

[0136] In this example, system 500 includes a lightscape creation tool 100, which is an instance of the content creation tool 000 that is described with reference to Figure 3. The lightscape creation tool 100 is configured for designing and outputting object-based light data 505’, either separately or in conjunction with corresponding audio data 111’ and / or video data 112’, depending on the particular implementation. The object-based light data 505’ may include time stamp information, as well as information indicating light object properties, etc. In some instances, the time stamp information may be used to synchronize effects relating to the object-based light data 505’ with the audio data 111’ and / or the video data 112’, which also may include time stamp information.

[0137] In this example, the object-based light data 505’ includes light objects and corresponding light metadata. For example, the object-based light data may include light object position metadata, light object color metadata, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradient metadata, light object priority metadata, light object layer metadata, or combinations thereof. Although the content creation tool 100 is shown providing a stream of object-based light data 505’ to the experience player 102 in this example, in alternative examples the content creation tool 100 may produce object-based light data 505’ that is stored for subsequent use. Examples of graphical user interfaces for a light-object-based content creation tool are described below.

[0138] In this example, system 500 includes an experience player 102 that is configured to receive object-based light data 505’, audio data 111’ and video data 112’, and to provide object-based light data 505 to the lightscape renderer 501, to provide audio data 111 to the audio renderer 106 and to provide video data 112 to the video renderer 107. As noted elsewhere herein, the object-based light data 505, the audio data 111 and the video data 112 may include time stamp information that may be used to synchronize MS effects with audio and / or video effects. According to some examples, the experience player 102 may be amedia player, a game engine or personal computer or mobile device, or a component integrated in an television, DVD player, sound bar, set top box, or a service provider media device such as a Chromecast, Apple TV device, or Amazon Fire TV. In some examples, the experience player 002 may be configured to receive encoded object-based light data 505’ along with encoded audio data 111’ and / or encoded video data 112’, e.g., as part of the same bitstream with the encoded audio data 111’ and / or the encoded video data 112’. According to some examples, the experience player 102 may be configured to extract the object-based light data 505 from the content bitstream and to provide decoded object-based light data 505 to the lightscape renderer 501, to provide decoded audio data 111 to the audio renderer 106 and to provide decoded video data 112 to the video renderer 107. In some examples, the experience player 002 may be configured to allow control of configurable parameters in the lightscape renderer 501, such as immersion intensity. Some examples are described below.

[0139] As noted above, according to some implementations the system 500 may include one or more instances of the control system 110 of Figure 1A configured for performing at least some of the methods disclosed herein. In some such examples, one instance of the control system 110 may implement the lightscape creation tool 100 and another instance of the control system 110 may implement the experience player 002. In some examples, one instance of the control system 110 may implement the audio renderer 006, the video renderer 007, the lightscape renderer 501, or combinations thereof. According to some examples, an instance of the control system 110 that is configured to implement the experience player 002 may also be configured to implement the audio renderer 006, the video renderer 007, the lightscape renderer 501, or combinations thereof.

[0140] In some examples, room descriptors of the environment and light fixture data 104 may describe the size and orientation of the playback environment itself, to establish a relative or absolute coordinate system to which all objects are positioned. Room descriptor information may indicate or describe the physical dimensions of the playback environment, for example in physical units of distance such as meters. In some such examples, sensory object locations, sensory object sizes, and sensory object orientations may be described in units that are relative to the room size, for example in a range from -1 to 1. Room descriptors may also describe a preferred viewing position. For example, in a living room a display screen may be regarded as the front, in some instances the front and center, and the floor and ceiling may be regarded as the vertical bounds. In some such examples, the room descriptors also may also indicate bounds corresponding with the left, right, front, and rear,walls relative to the front position. According to some examples, at least some room descriptor information may be provided as a matrix. In some such examples the matrix may be a 3x3 matrix, with one row or column corresponding to one dimension of a three- dimensional space.

[0141] According to this example, system 500 includes a lightscape renderer 501 that is configured to render object-based light data 505 to light fixture control signals 515, based at least in part on environment and actuator data 104. In this example, the lightscape renderer 501 is configured to output the light fixture control signals 515 to light controllers 103, which are configured to control the light fixtures 108. The light fixtures 108 may include individual controllable light sources, groups of controllable light sources (such as controllable light strips), or combinations thereof. In some examples, the lightscape renderer 501 may be configured to manage various types of light object metadata layers, examples of which are provided herein. According to some examples, the lightscape renderer 501 may be configured to render actuator signals for light fixtures based, at least in part, on the perspective of a viewer. If the viewer is in a living room, that includes a television (TV) screen, the lightscape renderer 501 may, in some examples, be configured to render the actuator signals relative to the TV screen. However, in virtual reality (VR) use cases, the lightscape renderer 501 may be configured to render the actuator signals relative to the position and orientation of the user’s head. In some examples, the lightscape renderer 501 may receive input from the playback environment—such as light sensor data corresponding to ambient light, camera data corresponding to a person’s location or orientation, etc.—to augment the render.

[0142] In some examples, the lightscape renderer 501 is configured to receive object-based light data 505 that includes light objects and object-based lighting metadata indicating an intended lighting environment, as well as environment and light fixture data 104 corresponding to light fixtures 108 and other features of a local playback environment, which may include, but are not limited to, reflective surfaces, windows, non-controllable light sources, light-occluding features, etc. In this example, the local playback environment includes one or more loudspeakers 109 and one or more display devices 510.

[0143] According to some examples, the lightscape renderer 501 is configured to calculate how to excite various controllable light fixtures 108 based at least in part on the object-based light data 505 and the environment and light fixture data 104. The environment and lightfixture data 104 may, for example, indicate the geometric locations of the light fixtures 108 in the environment, light fixture type information, etc. In some examples, the lightscape renderer 501 may configured to determine which light fixtures will be actuated based, at least in part, on the position metadata and size metadata associated with each light object, e.g., by determining which light fixtures are within a volume of a playback environment corresponding to the light object’s position and size at a particular time indicated by light object time stamp information. In this example, the lightscape renderer 501 is configured to send light fixture control signals 515 to the light controller 103 based on the environment and light fixture data 104 and the object-based light data 505. The light fixture control signals 515 may be sent via one or more of various transmission mechanisms, application program interfaces (APIs) and protocols. The protocols may, for example, include Hue API, LIFX API, DMX, Wi-Fi, Zigbee, Matter, Thread, Bluetooth Mesh, or other protocols.

[0144] In some examples, the lightscape renderer 501 may be configured to determine a drive level for each of the one or more controllable light sources that approximates a lighting environment intended by the author(s) of the object-based light data 505. According to some examples, the lightscape renderer 501 may be configured to output the drive level to at least one of the controllable light sources.

[0145] According to some examples, the lightscape renderer 501 may be configured to collapse one or more parts of the lighting fixture map according to the content metadata, user input (choosing a mode), limitations and / or configuration of the light fixtures, other factors, or combinations thereof. For example, the lightscape renderer 501 may be configured to render the same control signals to two or more different lights of a playback environment. In some such examples, two or more lights may be located close to one another. For example, two or more lights may be different lights of the same actuator, e.g., may be different bulbs within the same lamp. Rather than compute a very slightly different control signal for each light bulb, the lightscape renderer 501 may be configured to reduce the computational overhead, increase rendering speed, etc., by render the same control signals to two or more different, but closely-spaced, lights.

[0146] In some examples, the lightscape renderer 501 may be configured to spatially upmix the object-based light data 505. For example, if the object-based light data 505 was produced for a single plane, such as a horizontal plane, in some instances the lightscape renderer 501 may be configured to project light objects of the object-based light data 505 onto an upperhemispherical surface (e.g., above an actual or expected position of the user’s head) in order to enhance the experience.

[0147] According to some examples, the lightscape renderer 501 may be configured to apply one or more thresholds, such as one or more spatial thresholds, one or more luminosity thresholds, etc., when rendering actuator control signals to light actuators of a playback environment. Such thresholds may, in some instances, prevent some light objects from causing the activation of some light fixtures.

[0148] In some implementations, the lightscape renderer 501 may be configured to adapt to changing conditions. Some examples of lightscape renderer 501 implementations are described in more detail below.

[0149] Light objects may be used for various purposes, such as to set the ambience of the room, to give spatial information about characters or objects, to enhance special effects, to create a greater sense of interaction and immersion, to shift viewer attention, to punctuate the content, etc. Some such purposes may be expressed, at least in part, by a content creator according to sensory object metadata types and / or properties that are generally applicable to various types of sensory objects—such as object metadata indicating a sensory object’s location and size.

[0150] For example, the priority of sensory objects, including but not limited to light objects, may be indicated by sensory object priority metadata. In some such examples, sensory object priority metadata is taken into account when multiple sensor objects map to the same fixture(s) in a playback environment at the same time. Such priority may be indicated by light priority metadata. In some examples, priority may not need to be indicated via metadata. For example, the MS renderer 001 may give priority to sensory objects—including but not limited to light objects—that are moving over sensory objects that are stationary.

[0151] A light object may, depending on its location and size and the locations of light fixtures within a playback environment—potentially cause the excitation of multiple lights. In some examples, when the size of a light object encompasses multiple lights, the renderer may apply one or more thresholds—such as one or more spatial thresholds or one or more luminosity thresholds—to gate objects from activating some encompassed lights.Examples of Using a Lighting Map

[0152] In some implementations a lighting map, which is an instance of the of the actuator map (AM) that includes a description of lighting in a playback environment, may be provided to the lightscape renderer 501. In some such examples, the environment and light fixture data shown in Figure 5 may include the lighting map. According to some examples, the lighting map may be allocentric, e.g., indicating absolute spatial coordinate-based light fall-off, whereas in other examples the lighting map may be egocentric, e.g., a light projection mapped onto a sphere at an intended viewing position and orientation. In the case of a sphere, the lighting map may, in some examples, be projected onto a two-dimensional (2D) surface, e.g., in order to utilize 2D image textures in processing. In any case, the lighting map should indicate the capabilities and the lighting setup of the playback environment, such as a room. In some embodiments the lighting map may not directly relate to physical room characteristics, for example if certain user preference-based adjustments have been made.

[0153] In some examples, there may be one lighting map per light fixture, or per light, in a playback environment. According to some examples, the intensity of light indicated by the light map may be inversely correlated to the distance to the center of the light, or may be approximately (e.g., within plus or minus 5%, within plus or minus 10%, within plus or minus 15%, within plus or minus 20%, etc.) inversely correlated to the distance to the center of the light. The intensity values of the light map may indicate the strength or impact of the light object onto the light fixture. For example, as a light object approaches a lightbulb, the lightscape renderer 501 may be configured to determine that the lightbulb intensity will increase as the distance between the light object and the lightbulb decreases. The lightscape renderer 501 may be configured to determine the rate of this transition based, at least in part, on the intensity of light indicated by the light map.

[0154] Figure 6A shows an example light map for a table lamp. The area 602 indicates the location of the lamp, mapped onto polar coordinates from the main viewer position. The lightscape renderer 501 is, in some examples, configured to map lightscape objects into a common rendering space, which may be allocentric or egocentric. Figure 6B shows an example of an egocentric light map. In this example, Figure 6B shows mapping for a spot light object onto a sphere at an intended viewing position and orientation. In Figures 6A and6B, darker areas are indicated by dots that are relatively closer together, whereas brighter areas are indicated by dots that are relatively farther apart.

[0155] Inside this common rendering space, in some examples the lightscape renderer 501 may be configured to use a dot product multiplication between a light object and the light map for each light to compute a light activation metric, e.g., as follows:

[0156] In the foregoing equation, Y represents the light activation metric, LM represents the lighting map and Obj represents the map of a light object. The light activation metric indicates the relative light intensity for the actuator control signal output by the lightscape renderer 501 based on the overlap between the light object and the spread of light from the light fixture. In some examples, the lightscape renderer 501 may use the maximum or closest distance, or other geometric metrics, from the light object to the light fixture as part of the determination of light intensity. In some implementations, instead of computing the light activation metric, the lightscape renderer 501 may refer to a look-up-table to determine the light activation metric.

[0157] The lightscape renderer 501 may repeat one of the foregoing procedures for determining the light activation metric for all light objects and all controllable lights of the playback environment. Thresholding for light objects that produce a very low impact on light fixtures may be helpful to reduce complexity. For example, if the effect of a light object would cause an activation of less than a threshold percent of light fixture activation— such as less than 10%, less than 5%, etc.—the lightscape renderer 501 may disregard the effect of that light object.

[0158] The lightscape renderer 501 may then use the resultant light activation matrix Y, along with various other properties such as the chosen panning law (either indicated by light object metadata or renderer configuration) or the priority of the light object, to determine which objects get rendered by which lights and how. Rendering lights-objects into light fixture control signals may involve: • Altering the luminance of a light-object as a function of the distance it is from the light fixture; • Mixing the colors of multiple light-objects that are simultaneously (multiplexed) rendered by a single light fixture; or• Altering either of the above based on the light object priority. Some detailed examples are disclosed herein. RENDERING PARAMETERS

[0159] In addition to the information carried by the light object metadata, the rendering of light-objects can be a function of the settings or parameters of the lightscape renderer 501 itself. These may include: • Velocity priority – when this parameter is set, light objects that are moving are given a higher priority than those which are not. Having the velocity priority parameter set enhances the dynamism of the rendered scene; • Color priority – light-objects with higher saturation values will take priority; • Activation threshold – the minimum light activation, Y, that must be achieved in order to activate a light-fixture; • Accessibility – certain colors may be chosen over others to best represent the experience for colorblind users. Certain flash rates may be avoided for those with photo-sensitivities. RENDERING CONFIGURATION (MODES)

[0160] In addition to the information carried by the light object metadata, the lightscape renderer 501 may, in some implementations, be configured according to different modes. As used herein, the term “mode” is different from “parameter” in the sense that modes may, for example, involve completely different signal paths, whereas parameters may simply parameterize these signal paths. For example, one mode may involve the projection of all light objects onto a lighting map before determining how / what to render to the light-fixtures, while another mode may only snap the highest-priority lights to the nearest light fixtures. Modes may include: • Modes to support low light-fixture count. In these modes, the rendering parameters and the light object metadata are utilized in order to determine which subset of light- objects are to be rendered and in what manner. Here, the “manner” refers to the trade- off between the spatial, color, temporal fidelity of the most prominent light-objects in the scene; • Modes to support different content types, such as music vs. gaming;• Modes in which multiple light objects may be rendered by a single light fixture (or a single light) with color mixing; • Modes in which only a single light object can be rendered by a single light fixture (or a single light); • Modes in which the luminance of the light object is altered as a function of the geometric – or otherwise – distance between the light object and light fixture. COLOR MIXING AND PRIORITIZATION

[0161] Some implementations of the lightscape renderer 501 may implement one or more color mixing methods, prioritization methods, or combinations thereof. For example, when there are multiple light objects that simultaneously influence the same light fixture(s), the lightscape renderer 501 may implement a color prioritization algorithm. In the simplest embodiment, only one light object influences the light fixture. The particular light object that will affect the light fixture may, in some examples, be determined by one of the following criteria: (1) the light object that is closest to the light fixture, which may be referred to as “snap-to-color”; (2) the light object that has the highest percentage of its impact on that light fixture, (3) the light object with the highest priority pre-defined, (4) the light object that is the brightest. In each of these conditions, only the reference color (potentially at a diminished brightness) may be shown in some instances.

[0162] In some instances it may be desirable to mix lighting. For example, when mimicking the physical characteristics of having multiple colored light fixtures with a single light fixture, modeling the physics of light mixing can make rendering more realistic. This is best done in a physics based, perceptually uniform, color space such as XYZ. In some examples, light mixing may be a linear process within a color space. According to some examples, when there are multiple light objects that simultaneously influence the same light fixture(s), light mixing may involve mixing color and adding intensity.

[0163] According to some examples, the lightscape renderer 501 may be configured to calculate the light mixing of two light objects as follows:

[0164] In this example, the mixing occurs in the XYZ color space. In the foregoing expression, ^^^^ ^^^^ ^^^^^^^^ ^^^^ ^^^^represents the result of light mixing, ^^^^ ^^^^ ^^^^1represents the color of a first light object, α represents a constant that indicates the weighting of first light object’s color,^^^^ ^^^^ ^^^^2represents the color of a second light object and β represents a constant that indicates the weighting of second light object’s color. The alpha and beta values may, for example, correspond to the amount of light intensity falloff due to the distance from each light object to the light fixture. The alpha and beta values may, for example, be extracted from the lighting map or from some other, potentially geometric, model.

[0165] In cases where physical modeling may be too computationally expensive, faster methods will allow for approximations. In one embodiment, the lightscape renderer 501 may be configured to calculate the light mixing of two light objects in the HSV color space, as follows: ^^^^ ^^^^ ^^^ ^^^^^ ^^^^ ^^^^  =  [^^^^ ∗ ^^^^ ^^^^1  +   ^^^^ ∗ ^^^^ ^^^^2,     ^^^^1 + ^^^^2]In the foregoing expression, H represents hue and S represents saturation, ^^^^ ^^^^1represents the color of a first light object, α represents a constant that indicates the weighting of first light object’s color, V1 represents the intensity of the first light object, ^^^^ ^^^^2represents the color of a second light object, β represents a constant that indicates the weighting of second light object’s color and V2represents the intensity of the second light object. In this example, the hue (H) and the saturation (S) are scaled by alpha and beta based on the amount they contribute at the rendered light fixture location. Then the intensities (V) are added to model the addition of light sources.

[0166] A typical lightscape scene may have anywhere from a few light objects up to a few dozen light objects that are active at any given time. A content creator will generally want to have control over the way that these light objects interact with one another within the lightscape renderer 501 as they are rendered onto the light fixtures in the endpoint. This may be summarized as the creator wanting to control (1) the relative priority of light objects and (2) the way in which properties of light objects can and cannot be mixed together. The latter is the larger departure from object-based audio rendering procedures, because the lightscape renderer 501 normally cannot simply mix the effects of multiple light objects to compute actuator control signals for one actuator. When there are a large number of light objects (even just 3), in combination with the projection of color from the object spatial domain onto the light-fixtures which introduces some warping, the color resulting from simply mixing the effects of multiple light objects would generally not represent the creative intent or preserve the fidelity of the scene. Furthermore, we are restricted to the limited output capacity of the light-fixtures in the endpoint.

[0167] The foregoing issues highlight the importance of providing the content creator with some control over the actions of the lightscape renderer 501. Various disclosed examples provide the content creator with the ability to define the light objects layer, the priority on that layer, to control how light objects are mixed on a layer, how layers are mixed, or combinations thereof.

[0168] Figure 7 shows elements of a lightscape renderer according to some examples. As with other figures provided herein, the types and numbers of elements shown in Figure 7 are merely provided by way of example. Other implementations may include more, fewer and / or different types and numbers of elements. According to this example, the lightscape renderer 501 is an instance of the lightscape renderer 501 that is described with reference to Figure 5. In some examples, the lightscape renderer 501 may be implemented by one or more instances of the control system 110 of Figure 1A.

[0169] According to this example, the lightscape renderer 501 includes the following elements: • 705: Light objects, which are instances of the object-based sensory data 005 disclosed herein; • 004: Environment and actuator data; • 723: A lighting map (LM), which is an instance of the of the actuator map (AM) that includes a description of lighting in a playback environment; • 750: A projection module, which is an instance of the projection module 450 of Figure 4 and is configured to projects the light objects using the LM 723; • 740: A light activation matrix (LAM), which is an instance of the actuator activation matrix (AAM) 440 and is the output of the projection module 750; • 751: A mixing module 751, which is an instance of the mixing module 451 of Figure 4 and is configured to convert the LAM 740 into actuator commands 741; • 741: The actuator commands 741, which in this example are sent directly to the light fixtures—which are instances of the actuators 008—to control them, but which may in other examples be sent to light controller APIs 103, which will send corresponding control signals to the light fixtures 008; • 752: Renderer configuration data, which may include settings such as the desired dynamism and mode. In some disclosed context-aware examples, the renderer configuration data 752 may be changed automatically;• 702: An intra-layer mixing module, which is configured to mix light objects of the same layer according to a mixing law; • 710: Light activation vectors, in this example a light activation vector 710 for every layer, containing the activation values and mixed colors of every light fixture; and • 703: An inter-layer blending module, which is configured to blend the light activation vectors 710 together to obtain the rendered actuator commands 741.

[0170] According to some examples, the light (actuator) activation matrix (LAM) 750 is a real matrix of size ^^^^^^^^by ^^^^^^^^, which is denoted as A in the following equation:In the foregoing equation, ai,j represents the activation value of the ithlight object on the jthactuator. The actuators are lights in this example. The ithrow of the A matrix contains all of the lights activated by the ithlight object. The jthcolumn of the A matrix represents all of the light objects activating the jthlight fixture. This is a useful intermediate data product, as the lightscape renderer 501 has not yet performed any mixing of light objects, which results in information loss. This has various potential benefits, including the possibility of optimizing the scene by analyzing A and then warping the scene (see examples below).

[0171] In some examples, there may be an A matrix for every layer being processed by the lightscape renderer 501. Thus, the lightscape renderer 501 can define a tensor of size ^^^^^^^^by ^^^^^^^^by ^^^^^^^^(dimension ordering is arbitrary), where ^^^^^^^^represents the number of layers in the lightscape renderer 501. In this document A may refer to either the tensor or the matrix form. The context will inform the reader which it is.

[0172] According to some examples, the intra-layer mixing module 702 is configured to mix all of the light objects on a given layer. This process collapses the A matrix (for that layer) into a light activation vector 710 v, of size 1 by ^^^^^^^^. The A matrix contains the activation values, not the color values. We can express the intra-layer mixing module 702 in its general form as a function which produces the light activation vector v (710), as follows: ^^^^ = ^^^1^( ^^^^)

[0173] However, in this example the intra-layer mixing process also produces a vector c for every layer in the matrix containing the color mixing result for that layer, so we may alter the equation above as follows: ^^^^, ^^^^ = ^^^1^( ^^^^, ^^^^^^^^)

[0174] In the above equation,now outputs both the activation vector, v, and the intralayer mixed colors c as a function of A and ^^^^^^^^, which is a vector of length ^^^^^^^^containing all of the objects’ colors.

[0175] The interlayer blending process takes all ^^^^^^^^v and c vectors packed into v’ and c’ matrices of size ^^^^^^^^by ^^^^^^^^and outputs a single vector o of length ^^^^^^^^containing the colors for each actuator. This can be written generally as: ^^^^ = ^^^^1( ^^^^′, ^^^^′)

[0176] Examples of ^^^1^() are given in the Example Object Mixing Laws section of this disclosure. Examples of ^^^^1() are given in the Example Layer Blending Laws section of this disclosure. Example Light Activation Laws

[0177] Light activation laws can range from simple geometric projections and distances to complex responses that involve precomputing and storing in a look up table (LUT). An example of the latter is the lighting map. In some examples, the LUT provides an activation value that may be based on the objects size, position, layer, velocity and potentially other parameters. This activation value may not represent how much the actual light or light fixture projects light onto the playback environment. In some examples, the activation value may be optimized to provide a sparse A matrix (which simplifies the mixing and potential object prioritization problems) and is based on such a (measured or simulated) projection.

[0178] Simple geometric-based activation laws may be applied in different reference frames, e.g., an allocentric or an egocentric reference frame. Such activation laws may be applied using various coordinate systems, for example rectangular, spherical or cylindrical coordinate systems.

[0179] Following is an example of an allocentric rectangular activation law: ALGORITHM 1For i=0; i< ^^^^^^^^; i ++ do; / / loop over all objects For j=0; j< ^^^^^^^^; j ++ do / / loop over all light-fixtures A[i,j] = 0 For i=0; i< ^^^^^^^^; i ++ do; / / loop over all objects For j=0; j< ^^^^^^^^; j ++ do / / loop over all light-fixtures ^^^^ =� ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^^^^^− ^^^^^^^^�2If d < ^^^^^^^^^^^^ ^^^^ ^^^^ ^^^^A[i,j] = 1 ElseAlgorithm 1 may, for example, be implemented by a control system that is configured to provide an instance of the projection module 750 of Figure 7. In Algorithm 1, - ^^^^^^^^represents the ithobject; - ^^^^^^^^represents the jthlight-fixture; - d represents the Euclidean distance between the light and object; - ^^^^^^^^^^^^ ^^^^ ^^^^ ^^^^represents the size of the object in the lightscape metadata; - ^^^^^^^^ ^^^^ ^^^^ ^^^^ℎ ^^^^ ^^^^^^^^feather radius of the object in the lightscape metadata; and - ε represents a small number, e.g.10^-10 for regularization.

[0180] An implementation of an egocentric spherical activation law may be substantially like the above Euclidean implementation, but one in which the coordinates are first transformed so that the user position and orientation define the origin (the light and object positions are references from this position and orientation) and the position and size are now angular. Distances and activation functions that can be used in place of d include, but are not limited to, the Euclidean distance, p-norm distance, cosine distance, logistic function, gaussian activation function, and rectified linear activation.

[0181] Additional variations to activations laws may include: • warping the coordinates of the objects to account for multiple user perspectives (widening the sweet spot); and / or• using arbitrary reference frames, e.g., rotated frames to account for a TV / screen that is not placed orthogonally within the playback environment. Example Object Mixing Laws

[0182] A simple implementation ofwould be to simply sum over all ^^^^^^^^rows of A to produce v and using these to weight ^^^^^^^^to accumulate c. More specifically, for the jth light, v and c may be calculated as follows:

[0183] Some implementations may involve performing a normalization across the columns of A before performing the summation. This normalization may be linear, e.g.,:Or nonlinear, for example a SoftMax function:

[0184] However, these normalizations result in the sum of the columns becoming unity, which may not be desirable when performing inter-layer mixing. Some examples involve augmenting the SoftMax function to place it back onto the same range, so that the sum of the columns are equal after the non-linear normalization (which is now a scaling, not a norm), e.g., as follows:

[0185] The above-described scaling and / or normalization may be performed before, in some instances, only the top N object elements are contributing to the jthlight. Normalization and scaling are motivated by the fact that it may not be desirable to saturate the color c[j]. If we consider an example involving the RGB color model on a range of [0, 1], then the mixinglaws above can produce results that exceed 1. In such cases, clipping may be performed in order to send valid RGB codewords to the light fixtures. However, clipping introduces chromaticity errors and, in the worst case when all 3 RGB channels are saturated, the output is white. This is not desirable, which is a further motivator for only taking the top N (for example, N=2) when performing the summations above.

[0186] In some examples, the lightscape renderer 501 may be configured for screen mixing. According to some examples, screen mixing may be implemented as follows: ALGORITHM 2 ^^^^ ^^^^ ^^^^ ^^^^ = 0; ^^^^ < ^^^^^^^^; ^^^^ + +; do / / loop over all light fixtures: ^^^^[^^^^]= 0^^^^[^^^^]= 0^^^^ ^^^^ ^^^^ ^^^^ = 0; ^^^^ < ^^^^^^^^; ^^^^ + +; ^^^^ ^^^^; / / loop over all objects ^^^^[ ^^^^]+= 1 − (1 − ^^^^[ ^^^^, ^^^^] ^^^^^^^^[ ^^^^]) (1 – ^^^^[ ^^^^]) ^^^^[ ^^^^]+= ^^^^[ ^^^^, ^^^^]

[0187] According to some examples, the lightscape renderer 501 may be configured for screen mixing with or without activation weighting as exemplified with the screen mixing above. Example Layer Blending Laws

[0188] In some implementations, there is no fundamental difference between the mixing that happens in the intra-layer process and the blending that happens in the inter-layer process. Blending is typically used to refer to the process of combing multiple layers in image processing and computer graphics. Thus, to avoid confusion and to help delineate the intra- and inter-layer processes, the terms “mixing” and “blending” are used herein.

[0189] For blending layers, in one example we may use alpha compositing, specifically A over B alpha compositing. Recall that blending layers is the process of producing the output vector o from the matrices v’ and c’: ^^^^ = ^^^^1( ^^^^′, ^^^^′) In the foregoing equation, v’ and c’ are of size ^^^^^^^^by ^^^^^^^^, for example as follows:^^^^ ^^^^In the foregoing v’ equation, ^^^^^^^^, ^^^^represents the net activation value of the light objects mixed into the ithlight-fixture on the jthlayer. These light objects are mixed to produce the ^^^^^^^^, ^^^^color.

[0190] In some examples in which the lightscape renderer 501 uses multiple layers, the order of the layers may imply some semantics or priority. For example, in some instances there may be an ambient layer, a spatial layer and an overlay layer. In some such examples, the lightscape renderer 501 may render these layers in order, in other words the ambient layer may be rendered first, then the spatial layer, then the overlay layer. According to some such examples, the lightscape renderer 501 may implement methods such as alpha compositing to render these layers and may use the net activation values, v, as a proxy for the alpha values to blend these layers.

[0191] Following is an example of using A over B alpha compositing to implement the function ^^^^1() when using an RGB color model. ALGORITHM 3 ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^= [0,0,0] / / RGB value for black For ^^^^ = 0; ^^^^ < ^^^^^^^^; ^^^^ + +; do / / loop over all light-fixtures ^^^^[ ^^^^] = ^^^^[ ^^^^, 0] ^^^^^^^^ ^^^^ ^^^^= ^^^^[ ^^^^, 0] For ^^^^ = 1; ^^^^ < ^^^^^^^^; ^^^^ + +; do / / loop over layers ^^^^^^^^= ^^^^[ ^^^^, ^^^^ − 1]^^^^^^^^= ^^^^[ ^^^^, ^^^^]break; / / move on to next light fixture ^^^^0= ^^^^^^^^+ ^^^^^^^^(1 − ^^^^^^^^) If ^^^^0<= 0 ^^^^[ ^^^^] = ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^elseIn Algorithm 3, the layers are ordered so that ^^^^ = 0 has the highest priority and ^^^^ = ^^^^^^^^− 1 has the lowest. Feather Distance A feather distance is a distance applied to the sensory objects such that spatial smoothing occurs. It is used to create smooth transitions as objects move around and activate / deactivate actuators. For example, if an object has size = 0.1 and a feather distance (sometimes also called feather size in our docs) of 0.2, then an example activation law would produce an activation value A of:^^^^ =� ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^^^^^− ^^^^^^^^�2Abstract Objects and Effects

[0192] The disclosed light objects need not be limited to discrete objects having a position, a size, a color and perhaps a few other parameters such as a feather distance. Instead, light objects may be considered to be abstract sources of light. Light objects may providepowerful data expansion within the lightscape renderer 501, which may reduce the data rate required to transport the experience, reduce the workload of content creators and simplify the control of complex behavior. Slice Room Effect Example

[0193] For example, an effect could be used to produce wave fronts that propagate throughout the playback environment. Instead of a content creator manually defining multiple wave fronts and constructing a tightly coupled loop, in some examples the content creator can simply select an abstract light object effect, which we call a “slice room” light object in this example. According to some examples, this abstract light object may be parameterized as follows: - p : position, which is the effective phase center of the wavefronts; - ^^^^ : size, determines the area in the endpoint of which this effect is applied; - ^^^^ : density, the number of wavefronts in each dimension of the playback environment; - ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^: frequency, the rate at which these wavefronts propagate throughout the playback environment; - ^^^^ ∶ duty cycle, the proportion of a single period in the wavefront which is considered active; - Color: the color renderer to the active section; and - Alternative color: the color rendered to the non-active section.

[0194] In some instances, a slice room effect may be implemented using a simple 3- dimensional oscillator to determine the spatial locations which are either active or not active. The position of a particular location in the endpoint may be represented as follows: ^^^^ = [ ^^^^, ^^^^, ^^^^] The phase at this location can be expressed as:In the foregoing equation, ^^^^^^^^represents the phase component due to the position of the object and spatial location being evaluated. This phase component may be implemented using the density D as follows:^^^^^^^^= ‖ ^^^^ − ^^^^ ‖ ⊕ ^^^^ In the foregoing equation, ⊕ represents the Hadamard product (element-wise multiplication).

[0195] The phase due to the temporal oscillator running at frequency ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^may be represented thusly: ^^^^^^^^= ^^^^^^^^ ^^^^ ^^^^ ^^^^ ^^^^^^^^ The second term of the foregoing equation determines the phase shift caused by the spatial offset between the phase center, p, of the oscillator and the spatial location currently being evaluated, q. Then ϕ may be compared with the duty cycle, ^^^^, to determine if q is considered active, e.g., as follows: ^^^^ = �1 ^^^^ ^^^^ ^^^^ < ^^^^0^^^^ ^^^^ ^^^^ ^^^^

[0196] Figure 8 shows an example of a slice room effect. In this example, the stippled areas indicate areas of active spatial coordinates and the non-stippled areas indicate areas of inactive spatial coordinates. Due to the periodic nature of the slice room effect and the ability to parameterize the oscillator, in an alternative example the non-stippled areas could indicate active spatial coordinates and the stippled areas could indicate inactive spatial coordinates. According to some examples, the light object may move to various locations in the playback environment over time, causing a modulation effect in the rendered scene. The other parameters of the light object may, in some examples, be modulated to further enrich the MS experience. Ambient Fill

[0197] In some implementations of the lightscape renderer 501, layers may be implemented with certain prescribed properties or semantics. For example, on the ambient layer, light objects may be rendered in a “fill” mode in which multiple objects on the ambient layer fill it entirely amongst themselves. In some examples, empty / black spaces may be defined on the ambient layer by placing null / black objects.

[0198] Figure 9 shows an ambient fill example. According to this example, different types of fill correspond to different colors, as noted below. In this example, a 2D illustration of ambient fill is shown, in which:- There are 4 objects on the ambient layer: o a null(black) object 905 at x,y = (50, 10) with size 1, within the area 910, which corresponds to the empty or black areas 910a and 910b; o red object 915 at x,y = (10, 30) with size 0.1, which is within the red area 920; o a green object 925 at x,y = (80,50) with size 0.5, which is with the green area 930; and o a blue object 935 at x,y = (50, 80) with size 1, which is with the blue area 940. In this example, a user is located at (50, 50).

[0199] Figure 9 shows the results when the spatial coordinate (x,y) is assigned a color based on minimizing the following ratio, r: ^^^^ =^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^^^^^ ^^^^ ^^^^ ^^^^ In some examples, at each x,y location (or any coordinate in any coordinate system), the distance is computed based on the coordinate system type, which may be rectangular or spherical for example where the origin may be defined such that a allocentric or egocentric system is defined. An egocentric spherical coordinate system was used to produce Figure 9. Thus, the distance and size in the equation above are angular: the distance is the angular distance between an x,y cell and the object, while the size is the angle the object subtends at its location relative to the user (as we are in an egocentric reference system).

[0200] If the user position u is defined as ^^^^ = [xu, yu] , we can define a unit vector from the user to any spatial location q = [x,y] as follows:

[0201] Similarly, the unit vector from the user to light object i may be defined as follows:, where ^^^^^^^^represents the position of the light object. Then, using a simple cosine distance, one can determine at any spatial location the distance to every object as follows:

[0202] The apparent size of each light object viewed from the user position may be expressed as follows:, where ^^^^^^^^represents the absolute size of the ithlight object and ^^^^^^′^^is the apparent size of that light object. The ratio rifor the ithlight object described earlier may be expressed as follows:

[0203] The color assigned to a particular spatial location associated with the object j that minimizes this ratio riacross all objects may be expressed as follows:Mixing may be performed by finding the second-lowest ratio ^^^^2, e.g., by performing the argmin again without considering the ^^^^1element. The ratio I of these two ratios is given by:

[0204] The ratio I can be used to determine the contribution for each of two objects being mixed. In this example, we are implementing a ‘fill’ functionality. Thus, at any given spatial location or light fixture, the contribution of all the light objects, in our case we have specified only two light objects, to that spatial location will be normalized.

[0205] Some examples may implement a spatial gating function in which only one light object is contributing to a lightscape whenever the ratio ^^^^ is sufficiently large. However, this will introduce potentially sharp discontinutities in the spatial domain on the ambient layer. As a result, in some examples, alpha compositing may be used on the ambient layer itself to mix the top two light objects contributing to any particular spatial location. To implementsuch some such techniques, one needs to determine the values of ^^^^^^^^and ^^^^^^^^, and to select a type of alpha compositing. Unlike the blending of multiple layers within the renderer, the order in which the objects are mixed is generally not predetermined. It is very challenging to prescribe a mixing order that would be consistent across the entirety of the spatial domain.

[0206] Therefore, some example, involve using the mixing order provided by ^^^^1and ^^^^2. This mixing order will generally change between the and ^^th^^2objects passing through the inflection point where the ratios are equal. As a result, using A over B alpha compositing produces a sharp, discontinuous step in the output due to the swapping of which light object is being considered ‘over’ the other.

[0207] Figure 10 shows examples of alpha compositing responses. In Figure 10, compositing responses are shown on the vertical axis and position is shown on the horizontal axis. The position indicated on the horizontal axis indicates the position of a light object with respect to a first light fixture located at position zero (0) and a second light fixture located at position one (1). Figure 10 shows the Alpha_1 and Alpha_2 compositing responses rather than Alpha_A or Alpha_B compositing responses. This is for clarity, because a continual plot (e.g., the blue Alpha_1 plot) is associated with a single light object, which is sensible given our implementation. Looking at the plots of Alpha_1 and Alpha_2 for the A over B implementation, one may observe a sharp discontinuity at the object boundary (position = 0.5), because this is the transition from light object 1 being associated with Alpha_A to being associated with Alpha_B as position increases. Note that this discontinuity is not present in the A atop B functions. As a result, for spatial blending, A atop B alpha (for example) compositing is preferred to A over B compositing.

[0208] It is desirable to provide some control over the sharpness in the transition between any two light objects and to ensure that the ambient layer does not simply become a wash of all the colors of the objects on the layer. In one example of a method for accomplishing these goals, the alpha components may be determined as follows:^^^^− ^^^^ ^^^^ ^^^^2^^^^^^^^=^^^^− ^^^^ ^^^^ ^^^^1 + ^^^^− ^^^^ ^^^^ ^^^^2In the foregoing expressions: - ^^^^^^^^1and ^^^^^^^^2represent the ^^^^1and ^^^^2th object ratios; and- K represents a factor that allows control of the sharpness of the transition between any two colors. This factor K may, in some implementations, be provided as light object metadata. In some examples, factor K may be provided by user personalizing the scene.

[0209] The formulation above is effectively a SoftMax activation function, thus:

[0210] We can then utilize A atop B compositing. For example, for the jthlight fixture, then q is the location of that light-fixture and the lightscape renderer 501 may find the I_1 and I_2 th objects and compute their alpha values as described above. The lightscape renderer 501 can then determine the mixed color value for that light fixture on the ambient layer as follows: ^^^^[ ^^^^] = ^^^^ ^^^^ ^^^^ ^^^^[ ^^^^1] + (1 − ^^^^ ^^^^) ^^^^ ^^^^[ ^^^^2]=^^^^ ^^^^ ^^^^ ^^^^[ ^^^^1] + ^^^^ ^^^^ ^^^^ ^^^^[ ^^^^2]Some examples may set ^^^^[^^^^]= 1 for compatibility with the layer blending mechanisms described earlier in this document.

[0211] Figure 11 shows an example of a 3D ambient fill result. In this example, the ambient fill is displayed in polar coordinates using the same scenario described at the start of this section. In this example, elevation is shown on the vertical axis and azimuth is shown on the horizontal axis. According to this example, the user is located on the z plane as are all of the light objects. This layout is equivalent to that shown in Figure 12D. In the example shown in Figure 11, the area 1110 is black, the area 1120 is red, the areas 1130a and 1130b are green and the areas 1140 are blue.

[0212] Figures 12A, 12B, 12C, 12D, 12E and 12F show additional examples of ambient fill responses. In these examples, the areas 1210 are black, the areas 1220 are red, the areas 1230 are green and the areas 1240 are blue. According to these examples, black light objects 1205, red light objects 1215, green light objects 1225 and blue light objects 1235 are in the same positions as the black light object 905, the red light object 915, the green light object 925 and the blue light object 935, respectively, shown in Figure 9. These examples illustrate, in a two dimensional spatial domain, a range of ambient fill responses corresponding to a range of objective cost functions (the columns) for different reference systems (the rows). The row that includes Figures 12A, 12C and 12E shows examples ofambient fill responses according to allocentric Euclidean distance. The row that includes Figures 12B, 12D and 12F shows examples of ambient fill responses according to egocentric distances in the spherical domain. The response in Figure 12D is equivalent, although without alpha compositing, to the response shown in Figure 11. A cost function that was applied to generate the examples shown in each column is shown above Figures 12A, 12C and 12E. The egocentric spherical responses are suited to layouts where light-fixtures are distributed along and near the walls and ceiling while the allocentric rectangular responses are suited better to densely populated endpoints where the distance of a light-object from the user may be better represented in the endpoint. Object Priority

[0213] Sensory objects, including but not limited to light objects, can have a priority level associated with them, for example according to sensory object metadata. The ability to assign a sensory object priority level provides a control mechanism to the content creator. Alternatively, or additionally, in some implementations a renderer may assign an implicit or explicit priority, for example to the layer associated with a sensory object, to control which sensory objects take priority during the sensory object rendering process. In some examples, priority may be represented abstractly as a real vector of length ^^^^^^^^, which is denoted as P in the following equation:

[0214] In the foregoing equation, ^^^^^^^^represents the priority associated with the jthobject. In some examples, priority may be “broadcast” to have the same dimensionality as A, for example as follows:

[0215] Expressing priority with the same dimensionality as A allows instances of the MS renderer 001 to compute a priority-weighted activation matrix conveniently which may, in some embodiments, be the elementwise product of A and P’. For example, the lightscape renderer 501 may be configured to compute a priority-weighted light activation matrix 740 as the elementwise product of A and P’. In other embodiments the priority-weightedactivation matrix may be the element-wise product after thresholding A or P’, or both. Alternatively, instead of thresholding the MS renderer 001 may be configured to apply a linear or non-linear transformation function to A or P’, or to both. Alternatively the MS renderer 001 may or may not be configured to combine A and P’, and may be configured to continue with the rendering process by using A and P’ independently. When the MS renderer 001 is configured to combine A and P’, the MS renderer 001 produces the priority- weighted activation matrix A’. The MS renderer 001 may, in such instances, be configured or configurable to alterand / or ^^^^1() (our intra- and inter-layer mixing laws respectively) to accommodate for the content creator’s intent associated with the priority of the sensory object. For example, a simple modification of the above intra-layer blending function would yield: ^^^^, ^^^^ = ^^^1^ ( ^^^^′, ^^^^^^^^)

[0216] In the foregoing expression and in the light object context, A’ being used in place of A may result in light objects having a higher priority being mixed into light fixtures when using A may have resulted in the light objects being mixed in with so little activation function the light objects were either thresholded out or contributing so little to the color c that the light objects were not perceivable in the rendered actuator signals. If the lightscape renderer 501 does not combine A and P’ to compute A’, the intra-layer blending function may be expressed as follows: ^^^^, ^^^^ = ^^^1^( ^^^^, ^^^^′, ^^^^^^^^)

[0217] Some examples may use the form given above and only mix objects into the jthlight fixture that gave non-zero light activation values having the highest priority level of all light objects on the ithcolumn. In such examples, the lightscape renderer 501 may be configured to mix all of the light objects into the jthlight fixture that satisfy both of the following conditions:

[0218] Some examples may involve warping the scene in order to allow for light objects with high priority that do not have a significant activation level in A for any of the light- fixtures. Light objects with lower priority that may or may not have significant activation levels in A may be overwritten by way of a higher-priority light object. In such cases, using the distance law described earlier in Algorithm 1 may not be suitable, because it causessaturation whenever the distance between a light object and a light fixture is smaller than the size of the object. In some such instances, the lightscape renderer 501 may be configured to utilize one or more non-saturating functions, such as the Gaussian activation:

[0219] In some examples, a matrix D of non-saturating functions may be constructed for convenience, e.g., as follows:

[0220] The matrix D is useful as a metric of the spatial warping penalty for each light object to each light fixture: as the distance between the two increases, so “warping” of the location of a light object increases. If this distance is used in the intra-layer mixing function, the function may be expressed as follows: ^^^^, ^^^^ = ^^^1^( ^^^^, ^^^^′, ^^^^, ^^^^^^^^)

[0221] In some examples, general optimisation techniques may be used for) in order to solve for v and c.

[0222] Figure 13 shows an example of a graphical user interface (GUI) that may be presented by a display device of the lightscape creation tool of Figure 5. As with other figures provided herein, the types and numbers of elements shown in Figure 13 are merely provided by way of example. Other GUIs presented by a lightscape creation tool may include more, fewer and / or different types and numbers of elements. According to some examples, the GUI 1300 may be presented on a display device according to commands from an instance of the control system 110 of Figure 1A that is configured for implementing the lightscape creation tool 100 of Figure 5.

[0223] In this example, a user may interact with the GUI 1300 in order to create light objects and to assign light object properties, which may be associated with the light object as metadata. According to this example, a user is selecting properties of the light object 1330. In this example, the GUI 1300 shows the light object 1330 in a three-dimensional space 1331, the latter of which represents a playback environment. Element 1334 shows acoordinate system of the three-dimensional space 1331. Accordingly, in this example the light object 1330 and the three-dimensional space 1331 are being viewed from the upper left.

[0224] A user may interact with the GUI 1300 in order to select a position and a size of the light object 1330. In some examples, a user may select a position of the light object 1330 by dragging the light object 1330 to a desired position within the three-dimensional space 1331, for example by touching a touch screen, using a cursor, etc. According to some examples, a user may select a size of the light object 1330 by selecting the size of the circle (or other shape) that is shown on the GUI 1300 to indicate the outline of the light object 130. In some such examples, a user may decrease the size of the light object 1330 via a two-fingered pinch of the outline of the light object 130, may increase the size of the light object 1330 via a two-fingered expansion, etc.

[0225] Specifying the position and size of an MS object within an abstracted three- dimensional space, such as the three-dimensional space 1331 of GUI 1300, allows a content creator to generalize the position and extent of the corresponding MS effects without prior knowledge of the particular playback environment in which the MS effects will be provided. This is an advantage of the MS object-oriented approach of various disclosed implementations. For example, the GUI 1300 allows a content creator to specify the position and size of the light object 1330 within the three-dimensional space 1331, thereby allowing the content creator to generalize the position and extent of the corresponding light effects, without prior knowledge of the particular size of any particular playback environment in which the light effects will be provided, without prior knowledge of the number, type and positions of light fixtures, etc., within the playback environment in which the light effects will be provided, etc. The light fixtures that will potentially be actuated responsive to the presence of the light object 1330 at a particular time will be those within a volume of the playback environment corresponding to the position and size / extent of the light object 1330.

[0226] According to this example, a user may interact with the color circle 1335 of the GUI 1300 in order to select the color of the current light object and may interact with the slider 1336 in order to select the brightness of the current light object. These and other selectable properties of the light object 1330 are displayed in area 1332 of the GUI 1300. According to this example, the properties of the light object 1330 that may be selected via the GUI 1300 also include intensity, diffusivity, “feathering,” whether or not the light object is hidden,saturation, priority and layer. Light object layers and priority will be described in more detail below. Generally speaking, light object layers may be used to group light objects into categories such as “ambient,” “dynamic,” etc. Light object priority may be assigned by a content creator and used by a renderer to determine, for example, which light object(s) will be presented when two or more light objects are simultaneously active and are simultaneously encompassing an area that includes the same light fixture.

[0227] Area 1340 of the GUI 1300 indicates time information corresponding to each of a plurality of light objects that are being created via the lightscape creation tool. In this example, light objects are listed on the left side of the area 1340, along a vertical axis, and time is shown along a horizontal axis. In this example, four-second time intervals are delineated by vertical lines. Here, time information for each light object is shown as isolated or connected diamond symbols or lines along a series of horizontal rows, each of which corresponds to one of the light objects indicated on the left side of the area 1340. The line 1333, for example, indicates that light object 3 will be displayed starting between 39 and 40 seconds and will be continuously displayed until almost 1 minute and 6 seconds. The diamond symbols to the right of the line 1333 indicate that light object 3 will be displayed discontinuously for the next few seconds.

[0228] Figure 14 shows another example of a graphical user interface (GUI) that may be presented by a display device of the lightscape creation tool of Figure 5. As with other figures provided herein, the types and numbers of elements shown in Figure 14 are merely provided by way of example. Other GUIs presented by a lightscape creation tool may include more, fewer and / or different types and numbers of elements. According to some examples, the GUI 1400 may be presented on a display device according to commands from an instance of the control system 110 of Figure 1A that is configured for implementing the lightscape creation tool 100 of Figure 5.

[0229] In this example, the GUI 1400 represents an instant in time during which light fixtures in an actual playback environment are being controlled according to light objects that have been created by an implementation of the lightscape creation tool 100 of Figure 5. An image of the playback environment is shown in area 1405 of the GUI 1400. Various light fixtures 1408 and a television 1415 are shown in the playback environment of area 1405. The particular instant in time is shown by vertical line 1442 of area 1440. At this time, the vertical line 1442 intersects with horizontal lines 1444a, 1444b, 1444c, and 1444d,indicating that the light being provided in the corresponding light objects 1, 4, 5 and 7 are being played back. Area 1432 indicates light object properties.

[0230] One may observe that at the instant in time that is depicted in Figure 14, the left side of the playback environment shown in area 1405 is being illuminated by blue light. This corresponds, at least in part, to the effect of the light object 1430 shown within the three- dimensional space 1431.

[0231] According to this example, video data and audio data are also being played back in the audio environment, and the playback of rendered light objects is being synchronized with playback of the video data and audio data. In this example, an image of the played- back video is shown in area 1410 of the GUI 1400. The video may, for example, be played back by the television 1415.

[0232] In some examples, a user may be able to interact with the GUI 1400 in order to adjust light object properties, add or delete light objects, etc. For example, a user may cause the playback to be paused in order to adjust light object properties. In some alternative examples, a user may need to revert to a GUI such as the GUI 1300 of Figure 13 in order to adjust light object properties, add or delete light objects, etc.

[0233] In the example described with reference to Figure 14, although the GUI 1400 was being presented a display device corresponding to the lightscape creation tool 100 of Figure 5, light objects, audio and video were being rendered in an actual, real-world environment. Accordingly, in some implementations the example described with reference to Figure 14 may also involve at least some of the “downstream” rendering and playback functionality that can be provided by other blocks of Figure 5, including but not limited to that of the lightscape renderer 501, the light controller APIs 103—which may in some instances be implemented by the same device that implements the lightscape renderer 501—the light fixtures 108, the audio renderer 106, the loudspeakers 109, the video renderer 107 and the display device(s) 510. In some such examples, the processes described with reference to Figure 14 also may involve functionality of the experience player 102 of Figure 5.

[0234] In some alternative implementations, the example described with reference to Figure 14 may also involve at least some of the “downstream” rendering and playback functionality that can be provided by other blocks of Figure 3, including but not limited to that of the MS renderer 001, the MS controller APIs 003—which may in some instances be implementedby the same device that implements the MS renderer 001—the light fixtures 008, the audio renderer 006, the loudspeakers 009, the video renderer 007 and the display device(s) 010. In some such examples, the processes described with reference to Figure 14 also may involve functionality of the experience player 002 of Figure 3.

[0235] As noted elsewhere herein, some types of metadata and / or priority indications are specific to light objects. For example, priority may be given to light objects that indicate changing colors over light objects that indicate static colors. Following are descriptions of some additional metadata aspects that are specific to light objects. Lightscape Metadata Layers

[0236] As noted elsewhere in this disclosure, when authoring a lightscape for media content, it can be useful to identify at least two different methods (or layers) to author for. These layers may be used during the process of rendering the authored light objects according to the available and controllable light fixtures in a playback environment. These layers can aid in capturing artistic intent and can allow for flexibility in restrictions in a playback environment, for example due to the number of light fixtures or light occlusions, so that the primary intent of the author(s) may still be rendered but can be scaled or otherwise modified.

[0237] In some examples, direct lighting and indirect lighting may be assigned to different lighting metadata layers.

[0238] Direct light objects

[0239] Light objects in a direct light object layer, also referred to herein as “direct light objects,” are light objects representing light that is directly visible to the content creator or end user. Examples of direct light objects could include light fixtures in the scene, the sun, the moon, headlights from a car approaching, lightning during a storm, traffic lights, etc. Direct light objects also may be used to represent light sources that are part of a scene but are typically, or temporarily, not visible in the associated video content, for example because they are outside the video frame or because they move outside the video frame. In some examples, direct light objects may be used to enhance or augment auditory events, such as explosions, visually guiding moving object trajectories outside the video frame, etc. The use of direct light objects is typically of a dynamic nature. For example, the associated metadata such as intensity, color, saturation, and position will often change as a function of time within a scene of the media content.

[0240] Indirect light objects

[0241] Light objects in an indirect light object layer, also referred to herein as “indirect light objects,” are light objects representing the effect of indirect light. For example, indirect light objects may be used to represent the effect of light radiated by fixtures that is observed when the light is reflected by one or more surfaces. Some examples of the using indirect light objects include change the observed color of the walls, ceiling or floor of the environment into a color that matches the content, such as green colors for a forest scene, or blue colors for sky or water. Indirect light objects also may be used to set the scene and the mood of the environment in a similar way as is achieved by color grading video content, but in a more immersive way. For example, science-fiction movies often use very specific (blue or greenish) video color grading palettes to reinforce the sense of being in outer space. Flash- back scenes often use reduced saturation, muted colors or sepia color overlays in the video content to intensify the effect of a change in the time line. All these effects can be replicated, or approximated, outside the video frame by adjusting the light control signals accordingly. Light effects corresponding to indirect light objects are often more stationary within a scene, and are typically less localized and less dynamic, than light effects corresponding to indirect light objects. Layer Abstraction

[0242] Some examples involve a further abstraction of the direct and indirect light object layers into layers that include aspects of both. In some such examples, these layers may include one or more ambient layers, one or more dynamic layers, one or more custom layers, one or more overlay layers, or combinations thereof. These layers may, in some examples, be used for, or may correspond with, linear or event-based triggers in content. Ambient Layer(s)

[0243] Similar to indirect lighting, an ambient layer can be used to set the mood and tone in a space through washes of color on surfaces in the playback environment. An ambient layer may be used as a base layer on which to build lighting scenes. In some examples, an ambient layer may be represented via light objects that cover relatively large areas. In other examples, an ambient layer may be represented via light objects that cover relatively small areas, for example with one or more images. According to some examples, an ambient layer may be divided into zones. In some such examples, particular light effects always occupy acertain region of space. For example, the walls, ceiling and floor in an authoring or playback environment each may be considered separate ambient layer zones. Dynamic Layer(s)

[0244] In some implementations, a dynamic layer may be used to represent spatial and temporal variation of MS objects, such as light objects. Within a dynamic layer, individual MS objects may also have priority so that, for example, one light object may have preference over another light object in being presented via a light fixture. Within a dynamic layer, individual MS objects may, in some examples, be linked to other objects, such as to audio objects (from spatial audio) or to 3D world MS objects. Custom Layer(s)

[0245] In some examples, a custom layer can be used to design light sequences that can be freely assigned to light fixtures for functional purposes. These sequences may not be spatial in nature, but instead may provide further information to the user. For example, in a game a light strip may be assigned to show the player’s remaining life. Overlay Layer(s)

[0246] According to some examples, an overlay layer can be used to present persistent lights that have continuous priority. An overlay layer may, for example, be used to create a “watermark” over all other elements in a lighting scene. Authoring and distribution of lightscape layer data Authoring of Direct and Indirect Light Objects

[0247] In some examples, direct light objects may be authored by determining or setting light source position, intensity, hue, saturation and spatial extent as a function of time for one or more light objects. In some such examples, this authoring process may create corresponding metadata that can be distributed, with the direct light objects, alongside audio and / or video content of a content presentation. Ideally, direct light objects are rendered to direct light sources.

[0248] Indirect light effects may, in some examples, be authored as a dedicated group or class within the lightscape metadata content, focusing more on overall color and ambiance rather than dynamic effects. Indirect light effects may also be defined by intensity, hue,saturation, or combinations thereof as a function of time, but would typically be associated with a substantial area of the lightscape rendering environment. Indirect light effects are ideally (but not necessarily) rendered to indirect light sources, when available. Rendering of Lightscape Layer Data

[0249] The layer attributes and their metadata may be rendered by means of a lightscape renderer, such as the lightscape renderer 101 of Figure 5. In some examples, the lightscape renderer may be configured to send light fixture control signals 515 to light fixtures of a playback environment. In some examples, the lightscape renderer 101 may be configured to output the light fixture control signals 515 to light controllers 103, which are configured to control the light fixtures 108. According to some examples, the lightscape renderer uses the environment and light fixture data 104 to determine the capabilities and spatial positions of each light fixture. In some examples, the priority of layers may be a determining factor in what is ultimately rendered to a light fixture. Rendering Direct and Indirect Light Objects

[0250] In some implementations, the environment and light fixture data 104 received by a lightscape renderer include data regarding whether the fixture is directly visible from a viewing position or is an indirect light source. In some examples, if no indirect light fixtures are available, the indirect light data may be sent to direct light fixtures instead, potentially with a reduced brightness.

[0251] Direct light objects are preferably rendered to visible light fixtures such as ceiling downlights, lights fixed to a wall, table lamps, etc. Indirect light metadata is ideally targeting light fixtures that are not directly visible, such as LED strips that light up walls, ceilings, shelves, furniture, and spot lights that light up walls or ceilings. If no such indirect lights are available, the indirect light metadata can be used to control direct lights instead. In some such examples, a lightscape renderer may cause direct light object metadata and indirect light object metadata to be superimposed when rendering to light fixtures that function as both indirect and direct light sources.

[0252] Figure 15 is a flow diagram that outlines one example of a method that may be performed by an apparatus or system such as those disclosed herein. The blocks of method 1500, like other methods described herein, are not necessarily performed in the order indicated. In some implementation, one or more of the blocks of method 1500 may beperformed concurrently. Moreover, some implementations of method 1500 may include more or fewer blocks than shown and / or described. The blocks of method 1500 may be performed by one or more devices, which may be (or may include) one or more instances of control system such as the control system 110 that is shown in Figure 1A and described above. For example, at least some aspects of method 1500 may be performed by an instance of the control system 110 that is configured to implement the multi-sensory renderer of Figure 3. Some aspects of method 1500 may be performed by an instance of the control system 110 that is configured to implement the lightscape renderer 501 of Figure 5.

[0253] In this example, block 1505 involves obtaining, by a control system, actuator data for a set of controllable actuators. In some examples, the set of controllable actuators may include one or more light fixtures, one or more haptic devices, one or more air flow control devices, or combinations thereof.

[0254] According to this example, block 1510 involves receiving, by the control system, object-based sensory data including a set of sensory objects. In some instances, the object- based sensory data may include sensory object metadata, which also may be referred to herein as sensory object metadata.

[0255] In some examples, the sensory object metadata may include sensory spatial metadata, such as sensory object position metadata indicating a spatial position for rendering the object-based sensory data within the environment, sensory object size metadata indicating an area or volume for rendering the object-based sensory data within the environment, or both. In some instances, the object-based sensory data may include one or more other types of sensory object metadata.

[0256] According to some examples, the object-based sensory data does not correspond to particular sensory actuators in the environment. For example, as described with reference to Figures 13 and 14, the sensory objects of the object-based sensory data may correspond with a portion of a three-dimensional area that represents a playback environment. The actual playback environment in which the sensory objects will be rendered does not need to be known, and generally will not be known, at the time that the sensory objects are authored. Accordingly, the object-based sensory data includes abstracted sensory reproduction information—in these example, the sensory objects and corresponding sensory metadata— allowing the sensory renderer to reproduce authored sensory effects via various sensoryactuator types, via various numbers of sensory actuators and from various sensory actuator positions in the environment.

[0257] In this example, block 1515 involves rendering, by the control system, the object- based sensory data to produce actuator control signals. According to this example, the rendering is based at least in part on the actuator data. In some examples, method 1500 also may involve obtaining user position data. According to some such examples, the rendering may be based, at least in part, on the user position data. In some examples, the audio object metadata may include at least audio object spatial metadata indicating an audio object spatial position for rendering the audio signals within the environment. In some examples, the rendering may be based, at least in part, on data corresponding to non-controllable actuators that are not controllable by the actuator control signals.

[0258] According to this example, block 1520 involves providing, by the control system, the actuator control signals to one or more controllable actuators of the set of controllable actuators. According to some such examples, method 1500 also may involve receiving, by the sensory renderer, the object-based sensory data and receiving, by the sensory renderer, environment descriptor data corresponding to the playback environment. In some such examples, method 1500 also may involve receiving, by the sensory renderer, actuator descriptor data corresponding to properties of the sensory actuators in the environment. In some examples, the environment descriptor data and the actuator descriptor data may be, or may be included in, the environment and actuator data 004 that is described with reference to Figure 3. In some examples, the MS controller APIs 003 that are shown in Figure 3 may be implemented via the MS renderer 001 and actuator-specific signals may be provided to the actuators 008 by the MS renderer 001. In some alternative examples, the MS renderer 001 may provide actuator control signals 310 to the MS controller APIs 003 and the MS controller APIs 003 may provide actuator-specific control signals to the actuators 008. In some examples, method 1500 also may involve providing, by the sensory actuators in the environment, the sensory effects.

[0259] In some examples, method 1500 may involve integrating the actuator data with the object-based sensory data. In some such examples, the integrating may involve finding one or more closest controllable actuators for each sensory object. According to some examples, the actuator data may include an actuator map. In some such examples, rendering the object-based sensory data to produce actuator control signals may involve projecting the setof sensory objects using the actuator map. According to some examples, projecting the set of sensory objects using the actuator map may produce an actuator activation matrix as the actuator control signals. In some examples, the actuator map may include a light fixture map. In some such examples, the light fixture map may be an allocentric light fixture map that is based on playback environment spatial coordinates or an egocentric light fixture map that is based spatial coordinates relative to an intended viewing location.

[0260] In some implementations, the object-based sensory data may include sensory object priority metadata. In some such examples, method 1500 may involve determining that two or more sensory objects map to a single actuator and rendering the object-based sensory data of the two or more sensory objects to actuator control signals for the single actuator based, at least in part, on the sensory object priority metadata.

[0261] According to some examples, the rendering may be based, at least in part, on one or more renderer configuration parameters. In some examples, the one or more renderer configuration parameters may include a velocity priority parameter that assigns a higher priority to moving sensory objects than to stationary sensory objects. According to some examples,the one or more renderer configuration parameters may include a change priority parameter that assigns a higher priority to changing sensory objects than to static sensory objects.

[0262] In some examples, the sensory objects may include light objects. In some such examples, the light objects may include color information. According to some such examples, the color information may include color change information. In some such examples, the one or more renderer configuration parameters may include a color change priority parameter that assigns a higher priority to color-changing light objects than to color- static light objects. In some examples, the light objects may include color saturation information. In some such examples, the one or more renderer configuration parameters may include a color saturation priority parameter that assigns a higher priority to light objects having a higher color saturation than to light objects having a lower color saturation.

[0263] According to some examples, the rendering may be based, at least in part, on one or more renderer configuration modes. The one or more renderer configuration modes may include a low actuator count mode, a content type mode, a color mixing mode, a single- sensory-object-to-single-actuator mode, a mode that alters illuminance of a light object according to distance between a light object and a light fixture, or combinations thereof. Insome examples, the rendering may involve implementing a color mixing and prioritization method when multiple light objects are being represented by a single light fixture.

[0264] In some examples in which the sensory objects include light objects, the light objects may have associated light object metadata. The light object metadata may include light object position data, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradient metadata, light object priority metadata, or combinations thereof. In some examples, the light object metadata may include light object layer metadata corresponding to two or more layers. According to some examples, the two or more layers may include an ambient layer, a dynamic layer, a custom layer, an overlay layer, a direct light object layer, an indirect light object layer, or combinations thereof. In some such examples, the rendering may involve intra-layer mixing, inter-layer blending, or both. According to some examples, the rendering may involve applying one or more light activation laws. In some examples, the rendering may involve creating a slice room effect, creating ambient fill, or both, and wherein creating the slice room effect involves produce wave fronts that propagate throughout a playback environment.

[0265] In some examples, method 1500 may involve receiving, by an audio renderer, audio objects and receiving, by the audio renderer, loudspeaker data corresponding to loudspeakers in the environment. According to some such examples, method 1500 also may involve providing, by the audio renderer, loudspeaker control signals for controlling the loudspeakers in the environment to play back audio corresponding to the audio objects and synchronized with the sensory effects. The synchronization may, for example, be based on time information that is included in or with the sensory objects and the audio object, such as time stamps. In some examples, method 1500 also may involve playing back, by the loudspeakers in the environment, the audio corresponding to the audio objects.

[0266] According to some examples, method 1500 may involve receiving, by a video renderer, video data synchronized with the audio objects and the object-based sensory data. According to some such examples, method 1500 also may involve providing, by the video renderer, video control signals for controlling one or more display devices in the environment to present images corresponding to the video control signals and synchronized with the audio objects and the sensory effects. In some examples, method 1500 also mayinvolve presenting, by the images on one or more display devices in the environment. The images may correspond to the video control signals. Context-Based Rendering Examples

[0267] This section describes implementations of the multi-sensory renderer 001 that are configured for rendering based, at least in part, on local context information. This type of rendering may be referred to herein as “context-based rendering” or as “context-aware rendering.” The “context” is, or includes, local context information regarding the local playback environment. For example, one aspect of the local context information may be the time of day in a region that includes the local playback environment, the weather in an area that includes the local playback environment, etc. Alternatively, or additionally, the context may be, or may include, information regarding one or more people in the local playback environment, such as the apparent level of engagement with played-back content.

[0268] Information to allow the multi-sensory renderer 001 to provide context-based rendering may, in some examples, be provided explicitly. In some examples, such explicit information may be, or may include, user input for managing one or more aspects of the rendering process, such as information regarding the overall immersion level and / or interactivity level. According to some examples, such explicit information may be, or may include, input from a device or system that has access to information regarding one or more aspects of the local context information, from a device or system that is configured to learn local context information by analyzing sensor data, patterns of user behavior, etc.

[0269] Managing the immersion and / or interactivity of the sensory experience can be achieved by changing the way the multi-sensory renderer 001 manages temporal, frequency, intensity, input, spatial effects (e.g. color or vibration) dimensions, or combinations thereof. In some examples, the multi-sensory renderer 001 may be configured to manage the immersion and / or interactivity of the sensory experience automatically, for example by applying a low-pass filter to one or more of those dimensions. Alternatively, or additionally, the multi-sensory renderer 001 may be configured to manage the immersion and / or interactivity of the sensory experience according to sensory object metadata indicating artistic intent and received with content that includes object-based sensory data.

[0270] In some examples, one or more relatively more dynamic or relatively more spatial layers of sensory content may be excluded and only an ambient layer may be used when auser prefers a less immersive experience. According to some examples, one or more relatively more dynamic or relatively more spatial layers of sensory content may be reduced in intensity. These actions may be taken with regard to any combination of audio, visual and sensory experiences.

[0271] Accordingly, a context-aware MS renderer 001 may be configured to render object- based sensory data 005 to actuator control signals 310 based at least in part on local context information, which may include one or more of the following: • Local time of day. For example, if it is late at night in the current location the MS renderer 001 may avoid rendering extremely bright light. In some examples, the MS renderer 001 may avoid light content with a strong blue component during the last hour or so before a viewer’s bedtime, because this may hinder sleep. If it is daytime in the current location, the MS renderer 001—or another device—may be configured to control automated blinds to reduce the ambient light from outdoors, depending on the content and experience; • Local weather information as determined by an Internet weather forecast, by live internet weather observations from a nearby weather station, by local weather information supplied by a LAN, WLAN or Bluetooth connected weather station onsite, etc. For example, if it is very sunny outside the playback environment, high- intensity lighting may be required to overcome the light leaking into a viewing room through the windows. If it is very cloudy, it may be dark in the room so less light intensity may be required; • Observations of human behavior, which may in some examples include observations from multiple days, weeks or months. For example, based historical information from a home’s security system it may be possible to determine that at the present time it is highly likely that only one person is present in a living room and that the person is likely to be watching a movie, a television series, etc., on a TV The context-aware MS renderer 001 may determine that this a time to augment the TV- viewing experience using sensory effects that include the living room’s lighting system, because it is likely that no one else in the house is trying to do something different in the living room at the same time; • Explicit input regarding mode switches, such as an explicit instruction—for example, received via user input from a person in the playback environment—to not augment aTV-viewing experience because another member of the household is trying to do their homework in the same room; • Information indicating the presence of a particular person in the playback environment. For example, a person may have indicated preferences as to one or more types of sensory experiences. In some examples, the person may be a photosensitive or colorblind viewer for whom the lighting experience should be toned down or otherwise personalized. The presence of a particular viewer may be determined by a Bluetooth or Wi-Fi beacon from a phone or smart watch, by talker identification using a microphone, by face identification, etc.; • Information—such as light sensor information—indicating ambient light from internal (within the playback environment) sources, from external (e.g., outdoor) sources, or both. For example, if a homes has smart lights in the living room, but the kitchen is just to the side of the living room and has a separate light setup, the kitchen lights may interfere with the light from controllable light fixtures in the living room. In some such examples, a context-aware MS renderer 001 may be configured to adapt the rendering for controllable light fixtures in the living room due to the light from the kitchen lights. For example, MS renderer 001 may be configured to cause light fixtures near the kitchen to be relatively brighter than those farther from the kitchen in order to compensate for the kitchen lights. Such compensatory techniques may be particularly relevant if the MS renderer 001 is mixing colors. For example, if the kitchen light is somewhat orange, but a white light was desired, the MS renderer 001 may make the side lights slightly greenish so that in a user’s peripheral vision the colors mix to white; • Information indicating the current context of one or more people in the playback environment. For example, if the current context information indicates that a user is driving a car, the MS renderer 001 may cause only lower-level / less immersive sensory objects / types to be shown in order to prevent driver distraction. In contrast, if the same person is sitting in the back seat whilst the vehicle is stationary, the MS renderer 001 may cause all metadata layers may be used. In still another example, if the context-aware MS renderer 001 receives information—such as sensor information or user input—indicating one or more people are watching TV and no other nearby person is trying to do anything constructive, such as housework or homework, the context-aware MS renderer 001 may determine that this a time forrelatively more immersive sensory content playback corresponding to the content provided via the TV, whereas if no one seems to be watching TV, the context-aware MS renderer 001 may determine that this a time for relatively more ambient, or completely ambient, sensory content playback corresponding to the content provided via the TV; • Environment context information, including but not limited to information derived from the environment and actuator data 004. Environment context information may include information about what devices are currently in use. In some examples, the context-aware MS renderer 001 may be configured to provide a designated egocentric view when a display screen (with visuals) is in use, may provide a more allocentric or ambient view if only audio and lighting or lighting alone are used. If the local playback environment is a vehicle environment, the local environment context information may include information regarding whether the vehicle is parked or moving. For example, if the context-aware MS renderer 001 receives information indicating that the vehicle is parked, the context-aware MS renderer 001 may determine that this a time for relatively more immersive sensory content playback corresponding to the content provided via an audio system of the vehicle, via a video system of the vehicle, or both; and • Information indicating the location of one or more people within the playback environment, such as information indicating that the only person in a moving vehicle is in the driver’s seat, information indicating that the only people in a parked vehicle are in the back seat, information indicating that one or more people are on a sofa that faces a TV, etc. Alternatively, or additionally, some examples may involve optimizing the rendering of lights if created for an egocentric view. This may be particularly relevant for effects in front vs. behind a user’s location. For example, light effects intended to be provided to the rear of the user may rely on reflections or may be less likely to be rendered, depending on fixture availability.

[0272] As noted elsewhere herein, the sensory object metadata may also contain information to assist the renderer to deliver artistic intent for various contexts or user selectable levels of immersion. For example, the sensory object metadata may include a “mood” object type to denote that the role of the object is to set an ambience layer. In some examples, the sensory object metadata may include a “dynamic” object type that may be used to signal to the renderer that the role of the object is to bring change and movement. In different contexts,the context-aware MS renderer 001 may use this information to render different types of sensory experiences. For example, in an “immersive” context in which one or more people want to have a completely immersive sensory experience—for example, in a living room of a home—the full gamut of mood and dynamic sensory objects may be used to render actuator signals for the sensory experience. If, however, an “ambient” context is selected by the user or the system, then the context-aware MS renderer 001 may use all the mood objects, but only a subset of the dynamic objects or none at all. According to some implementations, this immersion control may be continuous from “immersion off” to “fully immersive,” for example within a range from zero to ten, from zero to one hundred, etc. In some examples, different contexts may be sensed, categorized and programmed to correspond to various immersion levels. Immersion level may change the sensory objects used, the intensity or amplitude of sensory actuator playback, the number of actuators used, etc. For example, the immersion level may change the light objects used, the brightness of the lights, the number of light fixtures used, etc. The following table provides examples of immersion levels, contexts and sensory object usage by the context-aware MS renderer 001. Examples of Immersion managementAdditional Lightscape Examples

[0273] Figure 16 is a flow diagram that outlines one example of a method that may be performed by an apparatus or system such as those disclosed herein. The blocks of method 1600, like other methods described herein, are not necessarily performed in the order indicated. In some implementation, one or more of the blocks of method 1600 may be performed concurrently. Moreover, some implementations of method 1600 may include more or fewer blocks than shown and / or described. The blocks of method 1600 may be performed by one or more devices, which may be (or may include) one or more instances of control system such as the control system 110 that is shown in Figure 1A and described above. For example, at least some aspects of method 1600 may be performed by an instance of the control system 110 that is configured to implement the multi-sensory renderer of Figure 3. Some aspects of method 1600 may be performed by an instance of the control system 110 that is configured to implement the lightscape renderer 501 of Figure 5.

[0274] In this example, block 1605 involves obtaining, by a control system, actuator data for a set of controllable actuators. In some examples, the set of controllable actuators may include one or more light fixtures, one or more haptic devices, one or more air flow control devices, or combinations thereof.

[0275] In the example shown in Figure 16, block 1607 involves obtaining, by the control system, local context information. The local context information may, for example, be one or more of the types of local context information that is disclosed herein. In some examples, the local context information may be, or may include, local time of day information, local weather information, local human behavior information, local user input, information regarding one or more viewer preferences, information regarding presence or absence of one or more viewers, local ambient light information, local viewing environment information, local viewer location information, local device usage information, local viewer activity information, or combinations thereof.

[0276] According to this example, block 1610 involves receiving, by the control system, object-based sensory data including a set of sensory objects. In some instances, the object- based sensory data may include sensory object metadata, which also may be referred to herein as sensory object metadata.

[0277] In some examples, the sensory object metadata may include sensory spatial metadata, such as sensory object position metadata indicating a spatial position for rendering the object-based sensory data within the environment, sensory object size metadata indicating an area or volume for rendering the object-based sensory data within the environment, or both. In some instances, the object-based sensory data may include one or more other types of sensory object metadata.

[0278] According to some examples, the object-based sensory data does not correspond to particular sensory actuators in the environment. For example, as described with reference to Figures 13 and 14, the sensory objects of the object-based sensory data may correspond with a portion of a three-dimensional area that represents a playback environment. The actual playback environment in which the sensory objects will be rendered does not need to be known, and generally will not be known, at the time that the sensory objects are authored. Accordingly, the object-based sensory data includes abstracted sensory reproduction information—in these example, the sensory objects and corresponding sensory metadata— allowing the sensory renderer to reproduce authored sensory effects via various sensory actuator types, via various numbers of sensory actuators and from various sensory actuator positions in the environment.

[0279] In this example, block 1615 involves rendering, by the control system, the object- based sensory data to produce actuator control signals. According to this example, the rendering is based at least in part on the actuator data and the local context information. In some examples, method 1600 also may involve obtaining user position data. According to some such examples, the rendering may be based, at least in part, on the user position data. In some examples, the audio object metadata may include at least audio object spatial metadata indicating an audio object spatial position for rendering the audio signals within the environment. In some examples, the rendering may be based, at least in part, on data corresponding to non-controllable actuators that are not controllable by the actuator control signals.

[0280] According to this example, block 1620 involves providing, by the control system, the actuator control signals to one or more controllable actuators of the set of controllable actuators. According to some such examples, method 1600 also may involve receiving, by the sensory renderer, the object-based sensory data and receiving, by the sensory renderer, environment descriptor data corresponding to the playback environment. In some suchexamples, method 1600 also may involve receiving, by the sensory renderer, actuator descriptor data corresponding to properties of the sensory actuators in the environment. In some examples, the environment descriptor data and the actuator descriptor data may be, or may be included in, the environment and actuator data 004 that is described with reference to Figure 3. In some examples, the MS controller APIs 003 that are shown in Figure 3 may be implemented via the MS renderer 001 and actuator-specific signals may be provided to the actuators 008 by the MS renderer 001. In some alternative examples, the MS renderer 001 may provide actuator control signals 310 to the MS controller APIs 003 and the MS controller APIs 003 may provide actuator-specific control signals to the actuators 008. In some examples, method 1600 also may involve providing, by the sensory actuators in the environment, the sensory effects.

[0281] According to some examples in which the local context information includes local time of day information, the set of controllable actuators may include a set of controllable light fixtures, the actuator control signals may include light fixture control signals. In some such examples, the rendering may involve controlling brightness indicated by the light fixture control signals, controlling color indicated by the light fixture control signals, controlling one or more automated window shades, or combinations thereof, based at least in part on the local time of day information.

[0282] In some examples in which the local context information includes local weather information, the rendering may involve controlling brightness indicated by light fixture control signals, controlling one or more automated window shades, or both, based at least in part on the local weather information.

[0283] According to some examples in which the local context information includes local human behavior information, the rendering may be based at least in part on the local human behavior information.

[0284] In some examples, the local context information may include local user input. According to some such examples, the local user input may include one or more explicit mode control indications.

[0285] According to some examples, the local context information may include local person immersion information. In some such examples, the immersive context mode may be enabled when the local context information indicates a high level of local person immersion.

[0286] In some examples, method 1600 may involve obtaining, by the control system, viewing position data corresponding with a user position and location. In some such examples, the rendering may be based, at least in part, on the viewing position data.

[0287] According to some examples, the object-based sensory data may include sensory object priority metadata. In some such examples, the rendering may be based, at least in part, on the sensory object priority metadata.

[0288] In some examples, method 1600 may involve receiving, by the control system, one or more renderer configuration parameters. According to some examples, the rendering may be based, at least in part, on the one or more renderer configuration parameters.

[0289] In some examples, the object-based sensory data may include object-based light data including a set of light objects. In some such examples, the one or more renderer configuration parameters may correspond with an ambient context mode that causes rendering of all mood light objects and rendering of only a subset of dynamic light objects or rendering of no dynamic light objects. According to some examples, the local context information may include local viewer immersion information. In some such examples, the ambient context mode may be enabled when the local context information indicates a low level of local viewer immersion. In some examples, the object-based light data may include light object position metadata, light object color metadata, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradient metadata, light object priority metadata, light object layer metadata, light object mood metadata, light object dynamism metadata, or combinations thereof. In some examples, the light object metadata may include light object layer metadata corresponding to two or more layers. According to some examples, the two or more layers may include an ambient layer, a dynamic layer, a custom layer, an overlay layer, a direct light object layer, an indirect light object layer, or combinations thereof. In some such examples, the rendering may involve intra-layer mixing, inter-layer blending, or both. According to some examples, the rendering may involve applying one or more light activation laws. In some examples, the rendering may involve creating a slice room effect, creating ambient fill, or both, and wherein creating the slice room effect involves produce wave fronts that propagate throughout a playback environment. According to some examples in which the rendering is based, at least in part, on one or more renderer configuration parameters, the one or more renderer configuration parameters may correspond with an immersive context mode thatcauses rendering of full light intensity, full color gamut, dynamic light objects, or combinations thereof..

[0290] In some examples, method 1600 may involve integrating the actuator data with the object-based sensory data. In some such examples, the integrating may involve finding one or more closest controllable actuators for each sensory object. According to some examples, the actuator data may include an actuator map. In some such examples, rendering the object-based sensory data to produce actuator control signals may involve projecting the set of sensory objects using the actuator map. According to some examples, projecting the set of sensory objects using the actuator map may produce an actuator activation matrix as the actuator control signals. In some examples, the actuator map may include a light fixture map. In some such examples, the light fixture map may be an allocentric light fixture map that is based on playback environment spatial coordinates or an egocentric light fixture map that is based spatial coordinates relative to an intended viewing location.

[0291] In some implementations, the object-based sensory data may include sensory object priority metadata. In some such examples, method 1600 may involve determining that two or more sensory objects map to a single actuator and rendering the object-based sensory data of the two or more sensory objects to actuator control signals for the single actuator based, at least in part, on the sensory object priority metadata.

[0292] According to some examples, the rendering may be based, at least in part, on one or more renderer configuration parameters. In some examples, the one or more renderer configuration parameters may include a velocity priority parameter that assigns a higher priority to moving sensory objects than to stationary sensory objects. According to some examples,the one or more renderer configuration parameters may include a change priority parameter that assigns a higher priority to changing sensory objects than to static sensory objects.

[0293] As noted elsewhere herein, in some examples the sensory objects may include light objects. In some such examples, the light objects may include color information. According to some such examples, the color information may include color change information. In some such examples, the one or more renderer configuration parameters may include a color change priority parameter that assigns a higher priority to color-changing light objects than to color-static light objects. In some examples, the light objects may include color saturation information. In some such examples, the one or more renderer configuration parametersmay include a color saturation priority parameter that assigns a higher priority to light objects having a higher color saturation than to light objects having a lower color saturation.

[0294] According to some examples, the rendering may be based, at least in part, on one or more renderer configuration modes. The one or more renderer configuration modes may include a low actuator count mode, a content type mode, a color mixing mode, a single- sensory-object-to-single-actuator mode, a mode that alters illuminance of a light object according to distance between a light object and a light fixture, or combinations thereof. In some examples, the rendering may involve implementing a color mixing and prioritization method when multiple light objects are being represented by a single light fixture.

[0295] In some examples, method 1600 may involve receiving, by an audio renderer, audio objects and receiving, by the audio renderer, loudspeaker data corresponding to loudspeakers in the environment. According to some such examples, method 1600 also may involve providing, by the audio renderer, loudspeaker control signals for controlling the loudspeakers in the environment to play back audio corresponding to the audio objects and synchronized with the sensory effects. The synchronization may, for example, be based on time information that is included in or with the sensory objects and the audio object, such as time stamps. In some examples, method 1600 also may involve playing back, by the loudspeakers in the environment, the audio corresponding to the audio objects.

[0296] According to some examples, method 1600 may involve receiving, by a video renderer, video data synchronized with the audio objects and the object-based sensory data. According to some such examples, method 1600 also may involve providing, by the video renderer, video control signals for controlling one or more display devices in the environment to present images corresponding to the video control signals and synchronized with the audio objects and the sensory effects. In some examples, method 1600 also may involve presenting, by the images on one or more display devices in the environment. The images may correspond to the video control signals.

[0297] Various features and aspects will be appreciated from the following enumerated example embodiments (“EEEs”): EEE 1. A method for controlling a set of controllable actuators, the method comprising: obtaining, by a control system, actuator data for the set of controllable actuators; obtaining, by the control system, local context information;receiving, by the control system, object-based sensory data including a set of sensory objects; rendering, by the control system, the object-based sensory data to produce actuator control signals, wherein the rendering is based at least in part on the actuator data and the local context information; and providing, by the control system, the actuator control signals to one or more controllable actuators of the set of controllable actuators. EEE 2. The method of EEE 1, wherein the set of controllable actuators includes one or more light fixtures, one or more haptic devices, one or more air flow control devices, or combinations thereof. EEE 3. The method of EEE 1 or EEE 2, further comprising obtaining user position data, wherein the rendering is based, at least in part, on the user position data. EEE 4. The method of any one of EEEs 1–3, wherein the local context information comprises local time of day information, wherein the set of controllable actuators includes a set of controllable light fixtures, wherein the actuator control signals include light fixture control signals and wherein the rendering involves controlling brightness indicated by the light fixture control signals, controlling color indicated by the light fixture control signals, controlling one or more automated window shades, or combinations thereof, based at least in part on the local time of day information. EEE 5. The method of EEE 4, wherein the local context information comprises local weather information and wherein the rendering involves controlling brightness indicated by light fixture control signals, controlling one or more automated window shades, or both, based at least in part on the local weather information. EEE 6. The method of any one of EEEs 1–5, wherein the local context information comprises local human behavior information and wherein the rendering is based at least in part on the local human behavior information. EEE 7. The method of any one of EEEs 1–6, wherein the local context information comprises local user input. EEE 8. The method of EEE 7, wherein the local user input comprises one or more explicit mode control indications.EEE 9. The method of any one of EEEs 1–8, wherein the local context information comprises information regarding one or more viewer preferences, information regarding presence or absence of one or more viewers, or combinations thereof. EEE 10. The method of any one of EEEs 1–9, wherein the local context information comprises ambient light information. EEE 11. The method of any one of EEEs 1–10, wherein the local context information comprises viewing environment information, local viewer location information, local device usage information, local viewer activity information, or combinations thereof. EEE 12. The method of any one of EEEs 1–11, wherein the object-based sensory data comprises object-based light data including a set of light objects and wherein the object- based light data includes light object position metadata, light object color metadata, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradient metadata, light object priority metadata, light object layer metadata, light object mood metadata, light object dynamism metadata, or combinations thereof. EEE 13. The method of any one of EEEs 1–12, further comprising obtaining viewing position data, the viewing position data corresponding with a user position and location, wherein the rendering is based, at least in part on the viewing position data. EEE 14. The method of any one of EEEs 1–11, wherein the object-based sensory data includes sensory object priority metadata. EEE 15. The method of any one of EEEs 1–14, wherein the rendering is based, at least in part, on one or more renderer configuration parameters. EEE 16. The method of EEE 15, wherein the object-based sensory data comprises object-based light data including a set of light objects and wherein the one or more renderer configuration parameters correspond with an immersive context mode that causes rendering of full light intensity, full color gamut, dynamic light objects, or combinations thereof. EEE 17. The method of EEE 16, wherein the local context information includes local viewer immersion information and wherein the immersive context mode is enabled when the local context information indicates a high level of local viewer immersion.EEE 18. The method of any one of EEEs 15–17, wherein the object-based sensory data comprises object-based light data including a set of light objects and wherein the one or more renderer configuration parameters correspond with an ambient context mode that causes rendering of all mood light objects and rendering of only a subset of dynamic light objects or rendering of no dynamic light objects. EEE 19. The method of EEE 18, wherein the local context information includes local viewer immersion information and wherein the ambient context mode is enabled when the local context information indicates a low level of local viewer immersion. EEE 20. An apparatus configured to implement the method of any one of EEEs 1–19. EEE 21. A system configured to implement the method of any one of EEEs 1–19. EEE 22. One or more non-transitory, computer-readable media having instructions stored thereon for controlling one or more devices to implement the method of any one of EEEs 1–19.

[0298] The above description illustrates various embodiments of the present disclosure along with examples of how aspects of the present disclosure may be implemented. The above examples and embodiments should not be deemed to be the only embodiments, and are presented to illustrate the flexibility and advantages of the present disclosure as defined by the following claims. Based on the above disclosure and the following claims, other arrangements, embodiments, implementations and equivalents will be evident to those skilled in the art and may be employed without departing from the spirit and scope of the disclosure as defined by the claims.

Claims

CLAIMS What Is Claimed Is:

1. A method for controlling a set of one or more controllable actuators, the method comprising: obtaining, by a control system, actuator data for the set of one or more controllable actuators; receiving, by the control system, object-based sensory data including a set of one or more sensory objects; rendering, by the control system, the object-based sensory data to produce one or more actuator control signals, wherein the rendering is based at least in part on the actuator data; and providing, by the control system, the one or more actuator control signals to one or more controllable actuators of the set of controllable actuators.

2. The method of claim 1, wherein the set of one or more controllable actuators includes one or more light fixtures, one or more haptic devices, one or more air flow control devices, or combinations thereof.

3. The method of claim 1 or claim 2, further comprising obtaining user position data, wherein the rendering is based, at least in part, on the user position data.

4. The method of any one of claims 1–3, wherein the object-based sensory data includes sensory object metadata.

5. The method of claim 4, wherein the sensory object metadata includes at least sensory object position metadata.

6. The method of claim 4 or claim 5, wherein the sensory object metadata includes sensory object size metadata.

7. The method of any one of claims 1–6, further comprising integrating the actuator data with the object-based sensory data, wherein the integrating involves finding one or more closest controllable actuators for each sensory object.

8. The method of any one of claims 1–7, wherein the actuator data includes an actuator map.

9. The method of claim 8, wherein rendering the object-based sensory data to produce one or more actuator control signals involves projecting the set of one or more sensory objects using the actuator map.

10. The method of claim 9, wherein projecting the set of one or more sensory objects using the actuator map produces an actuator activation matrix as the one or more actuator control signals.

11. The method of any one of claims 8–10, wherein the actuator map comprises a light fixture map and wherein the light fixture map is an allocentric light fixture map that is based on playback environment spatial coordinates or an egocentric light fixture map that is based spatial coordinates relative to an intended viewing location.

12. The method of any one of claims 1–11, further comprising obtaining playback environment data, wherein the rendering is based, at least in part, on the playback environment data.

13. The method of any one of claims 1–10, wherein the object-based sensory data includes sensory object priority metadata.

14. The method of claim 13, further comprising: determining that two or more sensory objects map to a single actuator; and rendering the object-based sensory data of the two or more sensory objects to one or more actuator control signals for the single actuator based, at least in part, on the sensory object priority metadata.

15. The method of any one of claims 1–14, wherein the rendering is based, at least in part, on one or more renderer configuration parameters.

16. The method of claim 15, wherein the one or more renderer configuration parameters include a velocity priority parameter that assigns a higher priority to moving sensory objects than to stationary sensory objects.

17. The method of claim 15 or claim 16, wherein the one or more renderer configuration parameters include a change priority parameter that assigns a higher priority to changing sensory objects than to static sensory objects.

18. The method of any one of claims 15–17, wherein the one or more sensory objects include one or more light objects, wherein the one or more light objects include color information and wherein the one or more renderer configuration parameters include a color change priority parameter that assigns a higher priority to color-changing light objects than to color-static light objects.

19. The method of claim 18, wherein the one or more light objects include color saturation information and wherein the one or more renderer configuration parameters include a color saturation priority parameter that assigns a higher priority to light objects having a higher color saturation than to light objects having a lower color saturation.

20. The method of any one of claims 1–19, wherein the rendering is based, at least in part, on one or more renderer configuration modes.

21. The method of claim 20, wherein the one or more renderer configuration modes include a low actuator count mode, a content type mode, a color mixing mode, a single- sensory-object-to-single-actuator mode, a mode that alters illuminance of a light object according to distance between a light object and a light fixture, or combinations thereof.

22. The method of any one of claims 1–21, wherein the rendering is based, at least in part, on data corresponding to non-controllable actuators that are not controllable by the actuator control signals.

23. The method of any one of claims 1–22, wherein the sensory objects include one or more light objects, wherein the one or more light objects include color information and wherein the rendering involves implementing a color mixing and prioritization method when multiple light objects are being represented by a single light fixture.

24. The method of any one of claims 1–22, wherein the sensory objects include light objects and light object metadata.

25. The method of claim 24, wherein the light object metadata comprises light object position data.

26. The method of claim 24 or claim 25, wherein the light object metadata comprises light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradient metadata, light object priority metadata, or combinations thereof.

27. The method of any one of claims 24–26, wherein the light object metadata comprises light object layer metadata corresponding to two or more layers.

28. The method of claim 27, wherein the two or more layers include an ambient layer, a dynamic layer, a custom layer, an overlay layer, a direct light object layer, an indirect light object layer, or combinations thereof.

29. The method of claim 27 or claim 28, wherein the rendering comprises intra-layer mixing, inter-layer blending, or both.

30. The method of any one of claims 24–29, wherein the rendering comprises applying one or more light activation laws.

31. The method of any one of claims 24–30, wherein the rendering comprises creating a slice room effect, creating ambient fill, or both, and wherein creating the slice room effect comprises producing wave fronts that propagate throughout a playback environment.

32. An apparatus configured to implement the method of any one of claims 1–31.

33. A system configured to implement the method of any one of claims 1–31.

34. One or more non-transitory, computer-readable media having instructions stored thereon for controlling one or more devices to implement the method of any one of claims 1–31.

Images