Multisensory Object Renderer

Abstract

Several methods for controlling a set of controllable actuators may include acquiring actuator data for the set of controllable actuators and receiving object-based sensory data, which includes a set of sensory objects. Some methods may include rendering the object-based sensory data to generate actuator control signals, which are at least partially based on the actuator data. Some methods may include acquiring regenerative environment data. Rendering may be at least partially based on the regenerative environment data. Some methods may include providing actuator control signals to one or more controllable actuators of the set of controllable actuators via a control system. The set of controllable actuators may include one or more luminaires, one or more sensory devices, one or more airflow control devices, or a combination thereof.

Description

[Technical Field]

[0001] [Related applications] This application claims priority from U.S. Provisional Application No. 63 / 669,233 filed on 10 July 2024, U.S. Provisional Application No. 63 / 514,106 filed on 17 July 2023, and U.S. Provisional Application No. 63 / 514,095 filed on 17 July 2023, each of which is incorporated in whole by reference into this specification.

[0002] [Technical field] This disclosure relates to providing multi-sensory (MS) experiences, and more specifically to aspects of MS renderers. [Background technology]

[0003] Unless otherwise specified, the methods described in this chapter are not prior art to the claims of this application, and their inclusion in this chapter does not make them recognized as prior art.

[0004] Media content delivery has generally focused on audio and screen-based visual experiences. The delivery of multisensory content is limited due to the specific nature of activation. For example, lighting fixtures are widely used as features for artistic expression and concerts. However, each installation is specifically designed for a particular set of lighting fixtures. It is generally not feasible for a system to provide lighting designs beyond the set of lighting fixtures for which it was designed. Other systems attempting to provide broader light experiences do so easily by algorithmically extending screen visuals, but this has not been specifically documented. Haptic content is designed for specific haptic devices. When other devices are used, such as game controllers, mobile phones, or haptic devices of different brands, there has been no way to translate the creative intent of the content to different actuators. [Overview of the project]

[0005] At least some aspects of this disclosure may be implemented by methods such as audio processing methods. In some examples, the methods can be implemented at least in part by a control system such as those disclosed herein. Some such methods may include the control system obtaining actuator data for one or more sets of controllable actuators. Some methods may include the control system receiving object-based sensory data, which includes one or more sets of sensory objects. Some methods may include the control system rendering the object-based sensory data to generate one or more actuator control signals. The rendering may be at least in part based on the actuator data. Some methods may include the control system providing one or more actuator control signals to one or more controllable actuators of a set of controllable actuators. In some examples, the set of one or more controllable actuators may include one or more luminaires, one or more tactile devices, one or more airflow control devices, or a combination thereof.

[0006] According to some examples, object-based sensory data may include sensory object metadata. In some examples, sensory object metadata may include sensory object location metadata. According to some examples, sensory object metadata may include sensory object size metadata.

[0007] Several approaches may involve integrating actuator data with object-based sensory data. This integration may involve finding one or more nearest controllable actuators for each sensory object.

[0008] In some examples, the actuator data may include an actuator map. According to some examples, rendering object-based sensory data to generate one or more actuator control signals may include projecting a set of one or more sensory objects using the actuator map. In some examples, projecting the set of one or more sensory objects using the actuator map may generate an actuator activation matrix as, or that represents, the one or more actuator control signals. According to some examples, the actuator map may be, or may include, a lighting fixture map. The lighting fixture map may be, for example, an allocentric lighting fixture map based on playback environment space coordinates, or an egocentric lighting fixture map based on spatial coordinates associated with an intended viewing position.

[0009] Some methods may include obtaining playback environment data. In some such examples, the rendering may be based at least in part on the playback environment data.

[0010] In some examples, the object-based sensory data may include sensory object priority metadata. Some such methods include determining that two or more sensory objects map to a single actuator, rendering, based at least in part on the sensory object priority metadata, the object-based sensory data of the two or more sensory objects into one or more actuator control signals for the single actuator, further comprising.

[0011] According to some examples, rendering may be based at least in part on one or more renderer configuration parameters. In some examples, the one or more renderer configuration parameters may include a velocity priority parameter that assigns a higher priority to moving sensory objects than to stationary sensory objects. According to some examples, the one or more renderer configuration parameters may include a change priority parameter that assigns a higher priority to changing sensory objects than to static sensory objects.

[0012] In some examples, the one or more sensory objects may include one or more light objects. In some examples, the one or more light objects may include color information. According to some examples, the one or more renderer configuration parameters may include a color change priority parameter that assigns a higher priority to light objects whose color changes than to light objects whose color does not change. In some examples, the one or more light objects may include saturation information. According to some examples, the one or more renderer configuration parameters may include a saturation priority parameter that assigns a higher priority to light objects having higher saturation than to light objects having lower saturation.

[0013] According to some examples, rendering may be based at least in part on one or more renderer configuration modes. According to some such examples, the one or more renderer configuration modes may include a low actuator count mode, a content type mode, a color mixing mode, a single sensory object to single actuator mode, a mode for changing the brightness of a light object according to the distance between the light object and a lighting fixture, or a combination thereof.

[0014] In some examples, rendering may be based at least in part on data corresponding to uncontrollable actuators that are not controllable by actuator control signals.

[0015] In some examples, a sensory object may contain one or more light objects. In some examples, one or more light objects may contain color information. In some examples, rendering may include implementing color mixing and prioritization methods when multiple light objects are represented by a single luminaire.

[0016] In some examples, a sensory object may include a light object and light object metadata. According to some examples, the light object metadata may include light object position data, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradient metadata, light object priority metadata, or a combination thereof.

[0017] According to several examples, light object metadata may include light object layer metadata corresponding to two or more layers. These two or more layers may include, for example, an environment layer, a dynamic layer, a custom layer, an overlay layer, a direct light object layer, an indirect light object layer, or a combination thereof.

[0018] In some examples, rendering may involve mixing within layers, blending between layers, or both. According to some examples, rendering may involve applying one or more photoactivation laws. In some examples, rendering may involve creating a slice room effect, creating an environment fill, or both. Creating a slice room effect may involve generating wavefronts that propagate throughout the entire reproduction environment.

[0019] Several methods may involve obtaining user location data. Rendering may be based at least partially on user location data.

[0020] Some or all of the operations, functions, and / or methods described in this specification may be executed by one or more devices in accordance with instructions (e.g., software) stored on one or more computer-readable non-temporary media. Such non-temporary media may include, but are not limited to, one or more memory devices as described in this specification, including one or more RAM (random access memory) devices, ROM (read-only memory) devices, etc. Accordingly, various novel aspects of the subject matter described in this disclosure may be implemented via one or more computer-readable non-temporary media on which software is stored.

[0021] At least some aspects of this disclosure may be implemented via devices. For example, one or more devices may be capable of performing at least partially the methods disclosed in this specification. In some implementations, the devices may include an interface system and a control system. The control system may include at least one of a general-purpose single or multiple chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic elements, discrete gates or transistor logic, discrete hardware components, or a combination thereof. The control system may be configured to perform some or all of the disclosed methods.

[0022] Details of one or more embodiments of the subject matter described in this specification are given in the accompanying drawings and the following description. Other features, embodiments, and advantages will become apparent from the specification, drawings, and claims. Note that the relative dimensions in the following figures may not be drawn to scale. [Brief explanation of the drawing]

[0023] The embodiments disclosed herein are described here merely as examples, with reference to the accompanying drawings.

[0024] [Figure 1A]This block diagram shows examples of components for devices that can implement various aspects of this disclosure.

[0025] [Figure 1B] An example of an endpoint is shown.

[0026] [Figure 2] An example of an actuator element is shown.

[0027] [Figure 3] This document illustrates exemplary elements of a system for creating and recreating multisensory (MS) experiences.

[0028] [Figure 4] This shows an example of a multisensory (MS) renderer.

[0029] [Figure 5] This demonstrates exemplary elements of another system for creating and recreating MS experiences.

[0030] [Figure 6A] An example light map of a table lamp is shown.

[0031] [Figure 6B] An example of a self-centered optical map is shown.

[0032] [Figure 7] The elements of a lightscape renderer are shown with several examples.

[0033] [Figure 8] An example of the slice room effect is shown.

[0034] [Figure 9] An example of environmental filling is shown.

[0035] [Figure 10] An example of an alpha synthesis response is shown.

[0036] [Figure 11] An example of 3D environment filling results is shown.

[0037] [Figure 12] An example of an additional environmental filling response is shown.

[0038] [Figure 13] Figure 5 shows an example of a graphical user interface (GUI) that may be presented by the display device of the lightscape creation tool.

[0039] [Figure 14] Figure 5 shows another example of a graphical user interface (GUI) that can be presented by the display device of the lightscape creation tool.

[0040] [Figure 15] This flowchart outlines an example of a method that can be performed by an apparatus or system as disclosed in this specification.

[0041] [Figure 16] This flowchart outlines an example of a method that can be performed by an apparatus or system as disclosed in this specification. [Modes for carrying out the invention]

[0042] This specification describes a technology relating to the provision of multisensory media content. Throughout the following detailed description, numerous examples and specific details are described for illustrative purposes to provide a complete understanding of the invention. However, it will be apparent to those skilled in the art that this disclosure, as defined by the claims, may include some or all of the features in these examples, either alone or in combination with other features described below, and may also include modifications and equivalents of the features and concepts described in this specification.

[0043] The following description details various methods, processes, and procedures. Certain steps may be described in a specific order, but such order is primarily for convenience and clarity. A particular step may be repeated more than once, may occur before or after other steps (even if those steps are described in a different order), and may occur concurrently with other steps. A second step must follow a first step only if the first step must be completed before the second step begins. Such situations will be specifically noted when they are not clear from the context.

[0044] In this specification, the terms “and,” “or,” and “and / or” are used. Such terms should be interpreted as having an inclusive meaning. For example, “A and B” may mean at least: “both A and B,” or “at least both A and B.” As another example, “A or B” may mean at least: “at least A,” “at least B,” “both A and B,” or “at least both A and B.” As yet another example, “A and / or B” may mean at least: “A and B,” or “A or B.” Particular attention should be paid to such things when an exclusive OR is intended (e.g., “either A or B,” or “at most one of A and B”).

[0045] This specification describes various processing functions associated with structures such as blocks, elements, components, circuits, etc. Generally, these structures may be implemented by one or more processors controlled by one or more computer programs.

[0046] As mentioned above, media content delivery has generally focused on audio and video experiences. The delivery of multi-sensory (MS) content is limited due to its customized activation nature.

[0047] This application describes methods for extending the creative palette for content creators and enabling the creation and delivery of spatial MS experiences at scale. Some such methods involve introducing a new layer of abstraction to enable the created MS experience to be delivered to different endpoints using different types of instruments or actuators. As used in this specification, the term “endpoint” is synonymous with “playback environment” or simply “environment” and means an environment containing one or more actuators that may be used to deliver an MS experience. Such endpoints may include rooms such as a living room in a house, a car, a movie theater, a nightclub, or other venue. Some of the disclosed methods involve creating, delivering, and / or rendering object-based sensory data, which may include sensory objects and corresponding sensory metadata. This abstraction enables the implementation of creative intent in an object-based format that does not require prior knowledge of specific controller activations, thereby enabling greater flexibility and scalability of instruments and actuators across endpoints. An MS experience delivered via object-based sensory data may be referred to in this specification as a “flexibly scaled MS experience.”

[0048] Glossary MS (multisensory): Multiple senses MSIE (MS Immersive Experience): MS Immersive Experience AR (Augmented Reality): Augmented Reality VR (Virtual Reality): Virtual Reality PC (personal computer): Personal computer

[0049] Figure 1A is a block diagram showing examples of components of a device that can implement various aspects of the present disclosure. As with other figures provided in this specification, the number and types of elements shown in Figure 1A are merely examples. Other implementations may include more, fewer, and / or different types and numbers of elements. In some examples, device 101 may be, or include, a device configured to perform at least some of the methods disclosed in this specification, such as a smart audio device, a laptop computer, a cellular phone, a tablet device, a smart home hub, etc. In some such implementations, device 101 may be, or include, a server configured to perform at least some of the methods disclosed in this specification.

[0050] In this example, the device 101 includes at least an interface system 105 and a control system 110. In some implementations, the control system 110 may be configured to perform at least partially the methods disclosed in this specification. In some examples, the control system 110 may be configured to acquire actuator data for a set of controllable actuators via the interface system 105. The set of controllable actuators may be, for example, specific to a particular regeneration environment. According to some examples, the control system 110 may be configured to acquire object-based sensory data, including a set of sensory objects, via the interface system 105. In some examples, the object-based sensory data may include object-based sensory metadata corresponding to some or all of the sensory objects. The encoded object-based sensory data may correspond to sensory effects, such as lighting, touch, airflow, one or more positional actuators, or a combination thereof, provided by multiple sensory actuators in the environment.

[0051] In some examples, the control system 110 may be configured to implement a multisensory (MS) renderer. Thus, in some examples, the control system 110 may be configured to render object-based sensory data to generate actuator control signals, the rendering being at least partially based on actuator data. The MS renderer may also be referred to as a “sensory renderer” in this specification, as in some examples the MS renderer may render only one type of MS data, such as object-based lighting data. In some examples, the control system 110 may be configured to send actuator control signals to one or more controllable actuators of a set of controllable actuators.

[0052] In some examples, object-based sensory metadata may include sensory spatial metadata indicating at least a spatial location for rendering object-based sensory metadata in an environment, an area for rendering object-based sensory metadata in an environment, or a combination thereof. In some implementations, object-based sensory metadata does not correspond to any particular sensory actuator in an environment. In some examples, object-based sensory metadata may include abstract sensory reproduction information that enables a sensory renderer to reproduce authored sensory effects, which may also be called intended sensory effects in this specification, from various sensory actuator locations in an environment, via various sensory actuator types and via various numbers of sensory actuators.

[0053] In some examples, the control system 110 may be configured to acquire local context information via the interface system 105. Local context information may include local time information, local weather information, local human behavior information, local user input, information about one or more viewer preferences, information about the presence or absence of one or more viewers, ambient light information, viewing environment information, local viewer location information, local device usage information, local viewer activity information, or a combination thereof. In some such examples, the rendering process may be based at least partially on the local context information.

[0054] In some examples, the content bitstream may also include encoded audio objects synchronized with encoded object-based sensory metadata. The audio objects may include an audio signal and corresponding audio object metadata. According to some examples, the audio objects may include an audio signal and corresponding audio object metadata. The audio object metadata may include, at a minimum, audio object spatial metadata indicating the audio object spatial location for rendering the audio signal within the environment. In some examples, the MS renderer may also be configured to render the audio objects.

[0055] The interface system 105 may include one or more network interfaces and / or one or more external device interfaces (e.g., one or more USB (universal serial bus) interfaces). According to some implementations, the interface system 105 may include one or more wireless interfaces. The interface system 105 may include one or more devices that implement a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. In some examples, the interface system 105 may include one or more interfaces between the control system 110 and a memory system, for example, an arbitrary memory system 115 shown in Figure 1A. However, in some cases, the control system 110 may include a memory system.

[0056] The control system 110 may include, for example, a general-purpose single or multiple chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic elements, discrete gates or transistor logic, and / or discrete hardware components.

[0057] In some implementations, the functionality of the control system 110 may reside in more than one device. For example, part of the control system 110 may reside in a device within the environment (such as a laptop computer, tablet computer, or smart audio device), while another part of the control system 110 may reside in a device outside the environment, such as a server. In other examples, part of the control system 110 may reside in a device within the environment, while another part of the control system 110 may reside in one or more other devices within the environment.

[0058] Some or all of the methods described herein may be executed by one or more devices in accordance with instructions (e.g., software) stored on one or more non-temporary media. Such non-temporary media may include, but are not limited to, random-access memory devices, read-only memory devices, and other memory devices as described herein. One or more non-temporary media may reside, for example, in an arbitrary memory system 115 and / or control system 110 shown in Figure 1A. Accordingly, various novel aspects of the subject matter described herein may be implemented on one or more non-temporary media storing software. The software may include, for example, instructions for controlling at least one device to process audio data. The software may be executable by one or more components of a control system, such as the control system 110 in Figure 1A.

[0059] In some examples, device 101 may include an optional microphone system 120, as shown in Figure 1A. The optional microphone system 120 may include one or more microphones. In some implementations, one or more microphones may be part of or associated with another device, such as a speaker in a speaker system, a smart audio device, etc.

[0060] In some implementations, the device 101 may include an optional actuator system 125 as shown in Figure 1A. The optional actuator system 125 may include one or more loudspeakers, one or more tactile devices, one or more illuminators (also referred to as lighting in this specification), one or more fans or other air-moving devices, one or more display devices including but not limited to one or more televisions, one or more position actuators, one or more other types of devices for providing an MS experience, or a combination thereof. As used in this specification, the term “illuminator” generally refers to any actuator configured to provide light. The term “illuminator” encompasses a variety of light sources, including individual light sources such as light bulbs, groups of light sources such as tape lights, light panels such as light-emitting diode (LED) panels, and display devices such as projectors and television (TV) screens. “Illuminators” may be movable, and therefore the term “device” in this context does not necessarily mean that illuminators are in a fixed position in space. As used in this specification, the term “position actuator” generally refers to a device configured to change the position or orientation of a person or object, such as a motion simulator seat. A loudspeaker may be referred to as a “speaker” in this specification. In some implementations, an optional actuator system 125 may include a display system comprising one or more displays, such as one or more light-emitting diode (LED) displays, one or more organic light-emitting diode (OLED) displays, etc. In some examples where the device 101 includes a display system, an optional sensor system 130 may include a touch sensor system and / or a gesture sensor system adjacent to one or more displays of the display system. According to some such implementations, a control system 110 may be configured to control the display system to present a graphical user interface (GUI), such as a GUI, relating to implementing one of the methods disclosed in this specification.

[0061] In some examples, the device 101 may include an optional sensor system 130, as shown in Figure 1A. The optional sensor system 130 may include a touch sensor system, a gesture sensor system, one or more cameras, and so on.

[0062] This application describes a method for rendering a flexibly scaled multi-sensory (MS) immersive experience (MSIE) and delivering it to different playback environments, which may also be referred to as endpoints in this specification. Such endpoints may include rooms such as a living room at home, a car, a movie theater, a nightclub, or other venue, as well as AR / VR headsets, PCs, and mobile devices.

[0063] Figure 1B shows exemplary elements of an endpoint. In this example, the endpoint is a living room 1001 containing multiple actuators 008, several pieces of furniture 1010, and a person 1000 (also referred to as a user in this specification) who will consume a flexibly scaled MS experience. The actuators 008 are devices that can change the environment 1001 in which the user 1000 is located. The actuators 008 may include one or more haptic devices, one or more lighting fixtures (also referred to as lighting in this specification), one or more fans or other air-moving devices, one or more display devices including but not limited to one or more televisions, one or more position actuators, one or more other types of devices for providing the MS experience, or a combination thereof.

[0064] The number of actuators 008 within space 1001, their arrangement, and their capabilities can vary significantly between different endpoint types. For example, the number, arrangement, and capabilities of actuators 008 in a car are generally different from those in a living room, nightclub, etc. In many implementations, the number, arrangement, and / or capabilities of actuators 008 can vary significantly between different instances of the same type, for example, between a small living room with two actuators 008 and a large living room with sixteen actuators 008. This invention describes various methods for creating a flexibly scaled MSIE and delivering it to these non-homogeneous endpoints.

[0065] Figure 2 shows an example of an actuator element. In this example, the actuator is a light 1100 including a network module 1101, a control module 1102, and a light emitter 1103. According to this example, the light emitter 1103 includes one or more light-emitting devices, such as light-emitting diodes, configured to emit light into the environment in which the light 1100 is present. In this example, the network module 1101 is configured to provide network connectivity to one or more other devices in space, such as a device that transmits commands to control the emission of light by the light 1100. According to this example, the network module 1101 is an example of the interface system 105 in Figure 1A. In this example, the control module 1102 is configured to receive signals via the network module 1101 and control the light emitter 1103 accordingly. According to this example, the control module 1102 is an example of the control system 110 in Figure 1A.

[0066] Other examples of actuators may include a network module 1101 and a control module 1102, but may also include other types of activation elements. Some such actuators may include one or more tactile devices, one or more fans or other air transport devices, one or more position actuators, one or more loudspeakers, one or more display devices, and so on.

[0067] Figure 3 shows exemplary elements of a system for creating and reproducing multisensory (MS) experiences. As with other figures provided in this specification, the number and types of elements shown in Figure 3 are merely examples. Other implementations may include more, fewer, and / or different types and numbers of elements. In some examples, system 300 may be, or include, one or more devices configured to perform at least some of the methods disclosed in this specification. In some examples, system 300 may include one or more examples of the control system 110 of Figure 1A configured to perform at least some of the methods disclosed in this specification.

[0068] According to examples in this disclosure, creating and providing an object-based MS Immersive Experience (MSIE) approach involves applying a set of techniques for creating, delivering, and rendering object-based sensory data, which may include sensory objects and corresponding sensory metadata, to actuator 008. Several examples are described in the following paragraphs.

[0069] Object-Based Representation: In some disclosed implementations, multisensory (MS) effects are represented using multisensory (MS) objects, which may be referred to in this specification as simply “sensory objects.” According to some such implementations, properties such as layer type and priority may be assigned to, associated with, and attached to each sensory object, enabling the content creator’s intent to be expressed in the rendered experience. Detailed examples of sensory object properties are described below.

[0070] In this example, system 300 includes a content creation tool 000 configured to design multisensory (MS) immersive content and output object-based sensory data 005 separately from or in association with corresponding audio data 011 and / or video data 012, depending on the specific implementation. The object-based sensory data 005 may include timestamp information, as well as information indicating the type of sensory object, sensory object characteristics, etc. In this example, the object-based sensory data 005 is not “channel-based” data corresponding to one or more specific sensory actuators in the playback environment, but rather generalized to a wide range of playback environments having a wide range of actuator types, a number of actuators, etc. In some examples, the object-based sensory data 005 may include object-based optical data, object-based tactile data, object-based airflow data, or object-based position actuator data, object-based olfactory data, object-based smoke data, object-based data for one or more other types of sensory effects, or a combination thereof. According to some examples, the object-based sensory data 005 may include sensory objects and corresponding sensory metadata. For example, if object-based sensory data 005 includes object-based optical data, the object-based optical data may include optical object position metadata, optical object color metadata, optical object size metadata, optical object intensity metadata, optical object shape metadata, optical object diffusion metadata, optical object gradient metadata, optical object priority metadata, optical object layer metadata, or a combination thereof. In some examples, object-based sensory data 005 may include temporal data such as timestamp information. In this example, the content creation tool 000 is shown to provide a stream of object-based sensory data 005 to the experience player 002, but in alternative examples, the content creation tool 000 may generate object-based sensory data 005 that is stored for later use.An example of a graphical user interface for a light object-based content creation tool is described below.

[0071] Examples of MS object properties The following is a non-exhaustive list of possible properties of an MS object. priority; layer; Mixing mode; Durability; effect; Spatial panning rules;

[0072] effect: As used in this specification, “effect” of an MS object is a synonym for the type of MS object. “Effect” is a sensory effect that the MS object provides or demonstrates. If the MS object is a light object, its effect includes providing direct or indirect light. If the MS object is a tactile object, its effect includes providing some kind of tactile feedback. If the MS object is an airflow object, its effect includes providing a certain type of airflow. Some examples include other “effect” categories, as will be described in more detail below.

[0073] Durability: Some MS objects can include persistence properties within their metadata. For example, if a movable MS object moves around the scene, the movable MS object can persist for a certain period of time at the locations it passes through. This period can be indicated by persistence metadata. In some implementations, the MS renderer is responsible for constructing and maintaining the persistence state.

[0074] layer: In some examples, individual MS objects may be assigned to a “layer” in which MS objects are grouped together according to one or more shared characteristics. For example, a layer can group MS objects together according to their intended effect or type, which may include, but is not limited to, the following: Mood / Environment information Punctuation / Notes. Alternatively, or additionally, in some examples, layers may be used to group MS objects together according to shared properties, which may include, but are not limited to, the following: color strength size shape position A region within space.

[0075] priority In some examples, MS objects may have priority properties that allow the renderer to determine which object should have priority in an environment where MS objects are competing for a limited number of actuators. For example, if multiple light objects overlap with a single light fixture at a time when all light objects are scheduled to be rendered, the renderer may refer to the priority of each light object to determine which light object is rendered. In some examples, priority may be defined between or within layers. According to some examples, priority may be linked to specific properties such as intensity. In some examples, priority may be defined temporally, for example, the most recent MS object to be rendered may have priority over previously rendered MS objects. According to some examples, priority may be used to specify which MS objects or layers should be rendered regardless of the limitations of a particular actuator system in the playback environment.

[0076] Spatial panning rules: Spatial panning laws can define the movement of MS objects across space, and how MS objects affect actuators when moving between them.

[0077] Mixing mode: The mixing mode can specify how multiple objects are multiplexed onto a single actuator. In some examples, the mixing mode may include one or more of the following: Max Mode: Selects the MS object that will activate the actuator the most. Mixing mode: Mixes some or all of an object according to a rule set, for example by summing the activation levels, averaging the activation levels, or mixing colors according to the activation level or priority level. MaxNmix: Mixes the top N MS objects (by activation level) according to a set of rules.

[0078] According to some examples, instead of (or in addition to) object-specific metadata, more general metadata for the entire multisensory content file can be defined. For example, an MS content file could include metadata such as trim paths or mastering environments.

[0079] Trim control:

[0080] In the context of Dolby Vision®, what is called “trim control” can function as guidance on how to modulate the default rendering algorithm for a particular environment or condition at an endpoint. Trim control can specify ranges and / or default values ​​for various characteristics, including saturation, tonal detail, and gamma. For example, there may be automotive trim control that provides guidance for rendering in an automotive environment, such as including only objects of a specific priority or layer. Other examples may include providing trim control for environments with limited, complex, or sparse multisensory actuators.

[0081] Mastering environment: A single multisensory content can include metadata about the characteristics of the mastering environment, such as room size, reflectivity, and environmental bias lighting levels. Specific characteristics may vary depending on the desired endpoint actuator. Mastering environment information can help provide a reference point for rendering in the playback environment.

[0082] MS Object Renderer: Various disclosed implementations provide a renderer configured to render MS effects to actuators in a playback environment. In this example, system 300 includes an MS renderer 001 configured to render object-based sensory data 005 to actuator control signals 310, at least in part based on environment and actuator data 004. In this example, the MS renderer 001 is configured to output the actuator control signals 310 to an MS controller 003, which is configured to control actuators 008. In some examples, the MS renderer 001 may be configured to receive light objects, object-based lighting metadata indicating an intended lighting environment, and lighting information about a local lighting environment. The lighting information may be one general type of environment and actuator data 004 and may include one or more characteristics of one or more controllable light sources in the local lighting environment. In some examples, the MS renderer 001 may be configured to determine the drive level of each of one or more controllable light sources that approximate an intended lighting environment. In some examples, the MS renderer 001 (or one of the MS controllers 003) may be configured to output a drive level to at least one of the controllable light sources. Some alternatives may include separate renderers for each type of actuator 008, such as one renderer for lighting fixtures, another for tactile devices, and another for airflow devices. In other implementations, a single renderer may be configured as an MS renderer, as well as an audio renderer and / or video renderer. In some implementations, the MS renderer 001 may be configured to adapt to changing conditions. Several examples of implementations of the MS renderer 001 are described in more detail below.

[0083] The environment and actuator data 004 may include what is referred to in this specification as a “room descriptor” that describes the actuator position (e.g., according to an x, y, z coordinate system or a spherical coordinate system). In some examples, the environment and actuator data 004 may indicate the orientation and / or placement characteristics of the actuator (e.g., direction and north-facing, omnidirectional, occlusion information, etc.). According to some examples, the environment and actuator data 004 may indicate the orientation and / or placement characteristics of the actuator according to a 3x3 matrix, where three elements (e.g., elements in the first row) represent the spatial position (x, y, z), three other elements (e.g., elements in the second row) represent the orientation (roll, pitch, yaw), and three other elements (e.g., elements in the third row) represent the scale or size (sx, sy, sz). In some examples, the environment and actuator data 004 may include device descriptors that describe actuator characteristics related to the MS renderer 001, such as the intensity range and color gamut of a luminaire, and the airflow velocity range and direction of an air transport device.

[0084] In this example, system 300 includes an experience player 002 configured to receive object-based sensory data 005', audio data 011', and video data 012', and to provide the object-based sensory data 005 to the MS renderer 001, the audio data 011 to the audio renderer 006, and the video data 012 to the video renderer 007. In this example, the reference numbers of the object-based sensory data 005', audio data 011', and video data 012' received by the experience player 002 include a prime (') to suggest that the data may be encoded in some cases. Similarly, the object-based sensory data 005, audio data 011, and video data 012 output by the experience player 002 do not include a prime to suggest that the data may be decoded by the experience player 002 in some cases. In some examples, the experience player 002 may be a component incorporated into a media player, game engine, personal computer or mobile device, or a television, DVD player, soundbar, set-top box, or a service provider's media device such as Chromecast, Apple TV, or Amazon Fire TV. In some examples, the experience player 002 may be configured to receive encoded object-based sensory data 005' along with encoded audio data 011' and / or encoded video data 012'. In some such examples, the encoded object-based sensory data 005' may be received as part of the same bitstream as the encoded audio data 011' and / or encoded video data 012'. Several examples are described in more detail below. In some examples, the experience player 002 may be configured to extract object-based sensory data 005' from the content bitstream, provide the decoded object-based sensory data 005 to the MS renderer 001, provide the decoded audio data 011 to the audio renderer 006, and provide the decoded video data 012 to the video renderer 007.In some examples, timestamp information within object-based sensory data 005' may be used, for example, by the experience player 102, MS renderer 001, audio renderer 106, video renderer 107, or all of them, to synchronize the effects associated with object-based sensory data 005' with audio data 111' and / or video data 112', which may also contain timestamp information.

[0085] In this example, system 300 includes an MS controller 003 configured to communicate with various actuator types using an application program interface (API) or one or more similar interfaces. Generally speaking, each actuator requires a specific type of control signal to produce a desired output from a renderer. In this example, the MS controller 003 is configured to map the output from the MS renderer 001 to the control signals of each actuator. For example, a Philips Hue® light bulb receives control information in a specific format for illumination, in a digital representation of a specific saturation, brightness, and hue, as well as a desired drive level. In some alternative examples, the MS renderer 001 may also be configured to implement some or all of the MS controller 003. For example, the MS renderer 001 may also be configured to implement one or more lighting-based APIs but not haptic-based APIs, or vice versa.

[0086] In some examples, a room descriptor may also describe the size and orientation of the playback environment itself in order to establish a relative or absolute coordinate system in which all objects are positioned. For example, in a living room, the display screen may be considered front, and possibly front and center, and the floor and ceiling may be considered vertical boundaries. In some such examples, a room descriptor may also indicate boundaries corresponding to the left, right, front, and rear walls relative to the front position. According to some examples, a room descriptor may also be provided in terms of a matrix, such as a 3x3 matrix. This room descriptor information is useful for describing the physical dimensions of the playback environment in units of physical distance, such as meters. In some such examples, the position, size, and orientation of sensory objects may be described in units relative to the size of the room, for example, in the range of -1 to 1. In some cases, a room descriptor may also describe a preferred viewing position according to a matrix.

[0087] The type, number, and arrangement of actuators 008 generally vary depending on the specific implementation. In some examples, actuators 008 may include lights and / or tape lights (also referred to as “lighting” in this specification), vibration motors, airflow generators, position actuators, or combinations thereof.

[0088] Similarly, the type, number, and arrangement of the loudspeakers 009 and display devices 010 generally vary according to the specific implementation. In the example shown in Figure 3, audio data 011 and video data 012 are rendered to the loudspeakers 009 and display devices 010, respectively, by the audio renderer 006 and video renderer 007.

[0089] As described above, according to several implementations, system 300 may include one or more examples of the control system 110 of Figure 1A configured to perform at least some of the methods disclosed in this specification. In some such examples, one example of the control system 110 may implement a content creation tool 000, another example of the control system 110 may implement an experience player 002. In some examples, one example of the control system 110 may implement an audio renderer 006, a video renderer 007, a multisensory renderer 001, or a combination thereof. According to some examples, an example of the control system 110 configured to implement the experience player 002 may be configured to implement an audio renderer 006, a video renderer 007, a multisensory renderer 001, or a combination thereof.

[0090] Figure 4 shows exemplary elements of a multisensory (MS) renderer. As with other figures provided in this specification, the number and types of elements shown in Figure 4 are merely examples. Other implementations may include more, fewer, and / or different types and numbers of elements. In this example, MS renderer 001 is an example of MS renderer 001 described with reference to Figure 3. In some examples, MS renderer 001 may be implemented by one or more examples of the control system 110 in Figure 1A.

[0091] According to this example, Figure 4 includes the following elements: 004: Environmental and actuator data, which can be explained with reference to Figure 3. 005: Object-based sensory data 005, which can be explained with reference to Figure 3. 423: Actuator map (AM) showing the positions of at least controllable actuators 008 in a specific playback environment. A projection module configured to project MS objects of object-based sensory data 005, at least in part, based on 450:AM423. MS objects may also be referred to as “sensory objects” in this specification, because in some cases only one type of sensory object may exist in object-based sensory data 005, such as only tactile objects or only luminous objects. In this example, the projection module 450 is configured to project MS objects, at least in part, based on sensory object metadata. Sensory object metadata may include at least sensory object position metadata and sensory object size metadata. 440: An actuator activation matrix (AAM) output by projection module 450 according to this example. The AAM may indicate, for example, if there are sensory objects currently containing volumes in the regeneration environment corresponding to one or more corresponding actuators. For example, the AAM may indicate whether the light object position metadata and light object size metadata of a light object indicate that a particular luminaire is in a volume in the regeneration environment corresponding to the position and size of the light object. 451: A mixing module configured to convert AAM440 into actuator control signals based at least partially on environmental and actuator data and renderer configuration data. 452: Optional renderer configuration data which may contain information about one or more settings for MS renderer 001, such as settings to indicate desired dynamics, modes, etc. In some examples, renderer configuration data 452 may be automatically modified using a context-aware system, as described in more detail below. 310: Actuator control signals, which may be described with reference to Figure 3. In some examples, the actuator control signals 310 may be transmitted to individual actuators 008, and in other examples, the actuator control signals 310 may be transmitted to an MS controller 003 which can be configured to transmit appropriate control signals to various types of actuators 008.

[0092] In some examples, AAM440 is a matrix that describes "how much" a sensory object projects itself onto each actuator, according to actuator map 423. In some examples, AAM440 is of size N O ×N A It can be a real matrix, where N O This represents the number of sensory objects, N A This represents the number of controllable actuators in the environment. In this example, the mixing module 451 is configured to generate actuator control signals 310 based at least partially on the AAM 440 and the environment and actuator data 004. In some examples, the mixing module 451 may be configured to generate actuator control signals 310 based at least partially on arbitrary renderer configuration data 452. According to some examples, the mixing module 451 may be configured to generate actuator control signals 310 based at least partially on sensory object metadata that may be received as part of object-based sensory data 005, such as mixing and panning laws associated with at least one sensory object, as shown in Figure 4.

[0093] In some examples, the mixing module 451 may be configured to generate actuator control signals 310 based at least partially on one or more of the following: 1. Threshold elements of AAM440; 2. To take the maximum value of a specific column in the AAM440, in other words, to take the sensory object that activated a particular actuator the most as the output; 3. Take any combination of the top N elements and perform at least one of the following: • Mix objects together within an actuator channel; • Push the object to an adjacent channel.

[0094] In some embodiments, the projection module 450 generates a sensory object image using the spatial coordinates of the sensory object, e.g., x, y, z coordinates, and the size of the sensory object, l n It may be configured to generate (x,y,z), l n This represents the sensory object image of the nth sensory object. Then, in some examples, the projection module 450 may be configured to calculate the nth row (object index) by taking the dot product of this object image and the actuator map corresponding to that actuator for all columns (actuator indices) of the AAM 440. The object image, actuator map, and dot product can be generated and performed by the projection module 450 in any convenient spatial region, including but not limited to polar coordinates, cylindrical coordinates, or Cartesian coordinates.

[0095] Some implementations may include implementing what may be referred to in this specification as “repellers,” which may be used to avoid potentially undesirable sensory effects, such as lighting effects, that may be caused when sensory objects are placed in one or more areas of the reproduction environment. In some such examples, repeller data may be included together with environment and actuator data 004 and AM423, and therefore may be part of the information input to the projection module 450. In some such examples, the spatial coordinates of MS objects are expanded when projecting them onto AM423 to generate AAM440.

[0096] Multisensory rendering synchronization Object-based MS rendering includes different modalities that render flexibly to endpoints / playback environments. Endpoints have different capabilities according to various factors, including, but not limited to, the following: Number of actuators; The modalities of those actuators (e.g., lighting fixtures, airflow control devices, tactile devices); The types of actuators (e.g., white smart lights and RGB smart lights, or tactile vests and tactile seat cushions); The position / layout of these actuators.

[0097] To render object-based sensory content to any endpoint, some processing of the object signals, such as intensity, color, or pattern, is generally required. The processing of the signal paths for each modality should not alter the relative phase of any particular feature within the object signal. For example, suppose lightning is presented in both the tactile and lightscape modalities. The signal processing chain for the corresponding actuator control signals should not introduce a time delay in either type of sensory object signal (tactile or luminous) sufficient to alter the perceived synchronization of the two modalities. The required level of synchronization can depend on various factors, including whether the experience is interactive and which other modalities are involved. The maximum time difference can range, for example, from approximately 10ms to 100ms, depending on the specific context.

[0098] Touch Rendering of object-based haptic content Object-based haptic content conveys the sensory aspects of a scene not only through channel-based methods but also through abstract sensory representations. For example, instead of defining haptic content solely as a single-channel time-dependent amplitude signal played from a specific haptic actuator, such as a vibrating haptic motor in a vest worn by the user, object-based haptic content may be defined by the sensation it is intended to convey. More specifically, one example might have a haptic object representing a collision haptic sensation effect. This object might be associated with the following: Spatial position of tactile objects; Spatial direction / vector of tactile effects; The intensity of the tactile effect; Haptic space and time-frequency data; Time-dependent amplitude signal.

[0099] In some examples, this type of haptic object can be automatically created in interactive experiences such as video games, for example, in a car racing game when another car crashes into the player's car from behind. In this example, the MS renderer determines how to render the spatial modality of this effect for a set of haptic actuators within the endpoint. In some examples, the renderer does this according to the following information: Types of tactile devices available, such as tactile vests, tactile gloves, tactile seat cushions, and tactile controllers; Location of each haptic device relative to the user (some haptic devices may not be attached to the user, for example, shakers mounted on the floor or seat); The type of activation provided by each tactile device, e.g., kinesthetic, vibratory touch; The on and offset delays of each haptic device (in other words, how quickly each haptic device can be turned on and off); The dynamic response of each tactile device (how much the amplitude can change); The time-frequency response of each tactile device (the time-frequency range that the tactile device can provide); Spatial distribution of addressable actuators within each haptic device: For example, a haptic vest may have dozens of addressable haptic actuators distributed across the user's torso; The time response of any tactile sensor (e.g., an active force feedback kinesthetic tactile device) used to render a closed-loop tactile effect.

[0100] These attributes of the endpoint's haptic modality inform the renderer how best to render a particular haptic effect. Again, consider the example of a car crash effect. In this example, the player is wearing a haptic vest, haptic armbands, and haptic gloves. In this example, the haptic shock wave effect is spatially positioned at the location where the car crashes into the player. The shock wave vector is determined by the relative velocity between the player's car and the car that hit the player. The spatial and temporal frequency spectrum of the shock wave effect is authored according to the type of material the virtual car is intended to be made of, among other virtual world properties. The renderer then renders this shock wave through the set of haptic devices in the endpoint, according to the shock wave vector and the physical position of the haptic devices relative to the user.

[0101] The signals transmitted to each specific actuator are preferably provided so that the sensory effects match across all (potentially heterogeneous) actuators for which sensory effects are available. For example, a renderer may not render very high frequencies to only one of the tactile actuators (e.g., a tactile armband) because other actuators lack the capability to do so. Otherwise, as a shock wave travels through the player's body, the tactile vest and gloves worn by the user may not have the capability to render such high frequencies, so the tactile effect perceived by the user may be degraded as the wave travels through the vest into the armband and finally into the gloves.

[0102] Some types of abstract tactile effects include: The shock wave effect described above; For example, barrier effects such as haptic effects used in video games to represent the spatial limitations of a virtual world. If the input device has active or resistive kinesthetic actuators (e.g., force feedback on a steering wheel or joystick), such effects can be rendered via the resistive force applied to the user input. If such actuators are not available at the endpoint, in some examples, vibrational haptic feedback corresponding to a collision between the in-game avatar and the barrier may be rendered; This indicates the presence of a large object approaching the scene, such as a train. This type of tactile effect can be rendered using low-time-frequency vibrations of actuators in some tactile devices. This type of tactile effect may also be rendered through contact space feedback applied as pressure from an air cuff; User interface feedback, such as clicks from virtual buttons. For example, this type of tactile effect may be rendered to the nearest actuator on the user's body that performed the click, such as a tactile glove worn by the user. Alternatively or additionally, this type of tactile effect may be rendered to a shaker coupled to the chair the user is sitting in. This type of tactile effect may be defined, for example, using a time-dependent amplitude signal. However, such a signal may be modified (modulated, frequency shifted, etc.) to best suit the tactile device providing the tactile effect; Movement. These haptic effects are designed to make the user perceive some form of movement. These haptic effects may be rendered by actuators that actually move the user, e.g., a moving platform / seat. In some examples, the actuators may provide secondary modalities (e.g., via video) to enhance the rendered movement; Triggered sequence. These tactile effects are primarily characterized by their time-dependent amplitude signals. Such signals may be rendered to multiple actuators and, when so, may be augmented. Such augmentation may involve splitting the signal across multiple actuators in either time or frequency. Some examples may involve augmenting the signal itself so that the sum of the tactile actuator outputs does not match the original signal.

[0103] Spatial and non-spatial effects Spatial effects are effects configured to convey some spatial information in a rendered multisensory scene. For example, if the playback environment is a room, shock waves traveling through the room will be rendered differently for each haptic device according to their location within the room, depending on the position and size of one or more haptic objects being rendered at a given time.

[0104] Non-spatial effects, in some examples, can target specific locations on the user regardless of their position or orientation. One example is a tactile device that provides an inflating vibration to the user's back to indicate immediate danger. Another example is a tactile device that provides a sharp vibration to indicate injury to a specific area of ​​the body.

[0105] Some effects can be non-diegetic. Such effects are typically associated with user interface feedback, such as haptic feedback indicating that a user has completed a level or clicked a button on a menu item. Non-diegetic effects can be either spatial or non-spatial.

[0106] Tactile device type By receiving information about different types of haptic devices available at the endpoint, the renderer can determine what types of perceived effects and rendering strategies are available to it. For example, local haptic device data indicating that the user is wearing both haptic gloves and a vibrating haptic vest, or at least that both haptic gloves and a vibrating haptic vest are present in the playback environment, allows the renderer to render a combined recoil effect across both devices when the user fires a gun in the virtual world. The actual actuator control signals sent to the haptic devices may differ from those in a situation where only a single device is available. For example, if the user is wearing only the vest, the actuator control signals used to activate the vest may differ in terms of the timing of the actuator control signal's initiation, maximum amplitude, frequency, and decay time, or a combination thereof.

[0107] Location of the device Knowledge of the location of haptic devices across endpoints allows the renderer to render spatial effects in a consistent manner. For example, knowledge of the location of shaker motors in a lounge allows the renderer to generate actuator control signals to each of the shaker motors in the lounge so as to convey spatial effects, such as shock waves propagating through the room. Furthermore, knowledge of the location of wearable haptic devices may be used by the renderer to convey spatial effects in addition to non-spatial effects, although this is implicit due to their type, for example, a glove being on the user's hand.

[0108] Types of activation provided by tactile devices Tactile devices can provide a variety of different activations, and therefore perceived sensations. These typically fall into two basic categories: 1. Vibratory tactile sensation, e.g., vibration; or 2. Kinesthetic sense, such as resistance or active force feedback.

[0109] Any category of activation can be static or dynamic, and dynamic effects are modified in real time according to several sensor inputs. Examples include touchscreens that render textures using vibratory tactile actuators, and position sensors that measure the position of the user's fingers.

[0110] Furthermore, the physical structure of such actuators can vary significantly and affect many other attributes of the device. An example of this is the onset delay or time-frequency response, which varies considerably across the following types of tactile devices: Eccentric rotating mass; Linear resonant actuator; Piezoelectric actuator; and Linear magnetic ram.

[0111] The renderer should be configured to account for the onset delay of a particular haptic device type when rendering signals activated by haptic devices within an endpoint.

[0112] Start and end delays of tactile devices The onset delay of a haptic device refers to the delay between the time the actuator control signal is sent to the device and the device's physical response. The termination delay refers to the delay between the time the actuator control signal is sent to zero out the device's output and the time the device stops activating.

[0113] Time-frequency response The time-frequency response refers to the frequency range of the signal amplitude as a function of time over which a haptic device can be activated in a steady state.

[0114] Spatial-frequency response The spatial-frequency response refers to the frequency range of the signal amplitude as a function of the spacing between the actuators of a tactile device. Devices with actuators positioned closer together have a higher spatial-frequency response.

[0115] Dynamic range Dynamic range refers to the difference between the minimum and maximum amplitudes of physical activation.

[0116] Sensor characteristics in closed-loop tactile devices Several dynamic effects are activated as a function of several observed states using sensors. The signal is updated. The temporal and spatial sampling frequencies, along with noise characteristics, limit the control loop's ability to update the actuator, which provides a dynamic effect.

[0117] air current Another modality that some multisensory immersive experiences (MSIEs) may utilize is airflow. Airflow may be rendered in conjunction with one or more other modalities, such as audio, video, lighting effects, and / or tactile sensations. Not only may “wind effects” be included in dedicated (e.g., channel-based) setups for 4D experiences in cinemas, but some airflow effects may also be provided in other endpoints that typically involve airflow, such as cars or living rooms. Rather than channel-based systems, airflow sensory effects may be represented as airflow objects that may include the following characteristics: Spatial location; Intended airflow effect direction: Intensity / airflow velocity; and / or temperature.

[0118] Some examples of airflow objects can be used to represent the movement of a flying bird. To render to an airflow actuator at the endpoint, MS Renderer 001 may provide the following information: Types of airflow devices, e.g., fans, air conditioners, heaters; The position of each airflow device relative to the user's position or the user's expectations; The capabilities of an airflow device, for example, the ability of an airflow device to control direction, airflow, and temperature; Control levels for each actuator, e.g., airflow velocity, temperature range; and The response time of each actuator, for example, how long it takes to reach the selected speed.

[0119] Several examples of airflow usage at different endpoints In vehicles such as automobiles, object-based metadata can be used to create experiences like the following: Using the airflow from a chair to mimic the "spine-chilling" feeling during content in horror movies or games; To simulate the movement of a flying bird; and / or To create a gentle breeze over the ocean view.

[0120] In the small, enclosed space of a typical vehicle, temperature changes can be achieved over relatively short periods compared to temperature changes in larger environments such as a living room. For example, MS Renderer 001 may raise the temperature when the player enters a "lava level" or other hot area during gameplay. Some examples may include other elements such as confetti in the vents to celebrate events such as a goal scored by the user's favorite soccer team.

[0121] In living spaces or other rooms, in one example, airflow may be synchronized with the breathing rhythm of guided meditation. In another example, airflow may be synchronized with the intensity of training, with the airflow increasing or the temperature decreasing as the intensity increases. In some examples, there may be relatively little control over the spatial aspects during rendering. For example, many existing airflow actuators are optimized for heating and / or air conditioning rather than to provide spatially diverse sensory activation.

[0122] A combination of light, airflow, and tactile sensations. Car example In some examples, the user interface may be located on the steering wheel, near the dashboard, or on a touchscreen within the dashboard. According to some examples, the following actuators may be present in the vehicle: 1. Individually addressable lighting is spatially distributed around the vehicle as follows: On the dashboard; Below the space under your feet; Above the door; and Inside the center console. 2. Individually controllable air conditioning / heating vents are positioned around the vehicle as follows: Inside the front dashboard; Below the space under your feet; Inside the center console facing the rear seats; Top of the side pillar; Inside the seats; and It is aimed at the windshield (to remove fog). 3. Individually controllable seats with vibration-tactile feedback; and 4. Individually controllable floor mats with vibration-tactile feedback.

[0123] In this example, the modalities supported by these actuators include: In addition to individually addressable LEDs inside the vehicle, the lights extend to the dashboard and steering wheel indicator lights; Airflow through controllable air conditioning vents; The following are examples of touch: • Handle: Tactile vibration feedback; • Dash touchscreen: Haptic vibration feedback and texture rendering; • Seats: Tactile vibration and movement.

[0124] In one example, a live music stream is being rendered for four users seated in the front row. In this example, MS Renderer 001 attempts to optimize the experience for multiple viewing positions. During the build-up period before the artist appears on stage, the previous performance has ended and the content includes: Interlude music; Low-intensity lighting; and Haptic content representing a mosh pit in a crowd.

[0125] In addition to the rendered audio and video streams, the lighting content includes ambient light objects that move slowly throughout the scene. These may be rendered using one of the environment layering methods disclosed in this specification, for example, so that no spatial priority is given to any particular user's viewpoint. In some examples, the haptic content may be spatially concentrated in a lower time-frequency spectrum and may be rendered only by vibrating haptic motors in a floor mat.

[0126] According to this example, a fireworks event in a music stream corresponds to multisensory content, including: Light objects that spatially correspond to the position of fireworks in an event; and A tactile object designed to enhance the dynamism of fireworks through a shockwave effect.

[0127] In this example, MS Renderer 001 spatially renders both light and haptic objects. For example, if fireworks content is located on the left side of the scene, the light object can be rendered inside the car so that each person inside the car perceives the light object as coming from the left. In this example, only the left-side light of the car is activated. Haptics can be rendered across both the seats and floor mats to provide individual directional information to each user.

[0128] At the end of the concert, fireworks are present in the audio content, and both fireworks and confetti are present in the video content. In addition to rendering light and tactile objects corresponding to the fireworks as described above, the effect of confetti launching can be rendered using airflow modalities. For example, individually controllable airflow outlets in an HVAC system may pulsate.

[0129] Living room example In this implementation, in addition to an audio / visual (AV) system including multiple loudspeakers and a television, the following actuators and associated controls are available in the living room: A tactile vest worn by the user (also called the player); A tactile shaker attached to the seat where the player sits; (Haptic) controllable smartwatch; Smart lighting spatially distributed throughout the room; Wireless controller; and An addressable airflow bar (AFB) (similar to the HVAC vents on a car's front dashboard) containing an array of individually controllable fans directed towards the user.

[0130] In this example, the user is playing a first-person shooting game, and the game includes a scene in which a destructive hurricane moves through the level. In this scene, in-game objects are thrown around, some of which hit the player. The tactile objects rendered by MS Renderer 001 provide a shockwave effect through all tactile devices that the user can perceive. The actuator control signals sent to each device may be optimized according to the intensity and direction of the impact of the in-game object, as well as the capabilities and position of each actuator (as described above).

[0131] Prior to the user colliding with an in-game object, the multisensory content includes a haptic object corresponding to a non-spatial rumble, one or more airflow objects corresponding to directional airflow, and one or more light objects corresponding to lightning. MS renderer 001 renders the non-spatial rumble to the haptic device. Actuator control signals sent to each haptic device may be rendered such that the set of actuator control signals across the haptic array matches in perceived onset time, intensity, and frequency. In some examples, the frequency components of the actuator control signals sent to a smartwatch may be low-pass filtered so that they match the frequency limiting capability of the best adjacent to the watch. MS renderer 001 may render one or more airflow objects into the actuator control signals for AFB such that the airflow in the room matches the player's position and line of sight in the game, as well as the direction of the hurricane itself. Lightning may be rendered across all modalities as (1) a white flash across lighting located in a suitable location, such as in or above the ceiling, and (2) a shocking rumble in the user's wearable haptic and seat shaker.

[0132] When an in-game object collides with a user, a directional shock wave may be rendered to the haptic device. In some examples, a corresponding airflow impulse may be rendered. In some examples, a damage trigger effect indicating the amount of damage inflicted on the player by the collision of an in-game object may be rendered by lighting.

[0133] In some such examples, the signal may be spatially rendered to the haptic device so that the perceived shock wave travels across the player's body and the room. The MS renderer 001 can provide such an effect according to actuator position information indicating the positions of the haptic devices relative to each other. In addition to actuator capability information, the MS renderer 001 may provide shock wave vectors and positions according to actuator position information. According to some examples, non-directional airflow impulses may be rendered, for example, all the vents of the AFB may be temporarily opened to enhance the haptic modality. In some examples, a red vignette may be rendered simultaneously on tape lights surrounding the TV to indicate to the player that the player has taken damage in the game.

[0134] Figure 5 shows exemplary elements of another system for creating and reproducing MS experiences. As with other figures provided in this specification, the number and types of elements shown in Figure 5 are merely examples. Other implementations may include more, fewer, and / or different types and numbers of elements. In some examples, system 500 may be, or include, one or more devices configured to perform at least some of the methods disclosed in this specification. In some examples, system 500 may include one or more examples of the control system 110 of Figure 1A configured to perform at least some of the methods disclosed in this specification.

[0135] In this example, the system shown in Figure 5 is an example of the system shown in Figure 3. In this example, the system shown in Figure 5 is an embodiment of a "lightscape" in which video, audio, and light effects are combined to create an MS experience.

[0136] In this example, system 500 includes a lightscape creation tool 100, which is an example of a content creation tool 000 as described with reference to Figure 3. Depending on the specific implementation, the lightscape creation tool 100 is configured to design and output object-based optical data 505', either separately from or in relation to the corresponding audio data 111' and / or video data 112'. The object-based optical data 505' may include timestamp information, as well as information indicating optical object characteristics, etc. In some examples, timestamp information may be used to synchronize the effects associated with the object-based optical data 505' with the audio data 111' and / or video data 112', which may also include timestamp information.

[0137] In this example, the object-based optical data 505' includes optical objects and their corresponding optical metadata. For example, the object-based optical data may include optical object position metadata, optical object color metadata, optical object size metadata, optical object intensity metadata, optical object shape metadata, optical object diffusion metadata, optical object gradient metadata, optical object priority metadata, optical object layer metadata, or a combination thereof. In this example, the content creation tool 100 is shown to provide a stream of object-based optical data 505' to the experience player 102, but in an alternative example, the content creation tool 100 may generate object-based optical data 505' that is stored for later use. An example of a graphical user interface for an optical object-based content creation tool is described below.

[0138] In this example, system 500 includes an experience player 102 configured to receive object-based optical data 505', audio data 111', and video data 112', and to provide the object-based optical data 505 to the MS renderer 501, the audio data 111 to the audio renderer 106, and the video data 112 to the video renderer 107. As described elsewhere in this specification, the object-based optical data 505, audio data 111, and video data 112 may include timestamp information that can be used to synchronize the MS effects with the audio and / or video effects. According to some examples, the experience player 102 may be a component incorporated into a media player, game engine, personal computer or mobile device, or a television, DVD player, soundbar, set-top box, or a service provider's media device such as Chromecast, Apple TV, or Amazon Fire TV. In some examples, the experience player 002 may be configured to receive the encoded object-based optical data 505' together with the encoded audio data 111' and / or encoded video data 112' as part of the same bitstream, for example, the encoded audio data 111' and / or encoded video data 112'. According to some examples, the experience player 102 may be configured to extract the object-based optical data 505 from the content bitstream, provide the decoded object-based optical data 505 to the lightscape renderer 501, provide the decoded audio data 111 to the audio renderer 106, and provide the decoded video data 112 to the video renderer 107. In some examples, the experience player 002 may be configured to allow control of configurable parameters within the lightscape renderer 501, such as immersion intensity. Several examples are described below.

[0139] As described above, according to several implementations, system 500 may include one or more examples of the control system 110 of Figure 1A configured to perform at least some of the methods disclosed in this specification. In some such examples, one example of the control system 110 may implement the lightscape creation tool 100, another example of the control system 110 may implement the experience player 002. In some examples, one example of the control system 110 may implement the audio renderer 006, the video renderer 007, the lightscape renderer 501, or a combination thereof. According to some examples, an example of the control system 110 configured to implement the experience player 002 may be configured to implement the audio renderer 006, the video renderer 007, the lightscape renderer 501, or a combination thereof.

[0140] In some examples, the room descriptor and lighting fixture data 104 of the environment may describe the size and orientation of the reproduction environment itself in order to establish a relative or absolute coordinate system in which all objects are placed. The room descriptor information may indicate or describe the physical dimensions of the reproduction environment in physical units of distance, such as meters. In some such examples, the position, size, and orientation of sensory objects may be described in units relative to the size of the room, for example, in the range of -1 to 1. The room descriptor may also describe the desired viewing position. For example, in a living room, the display screen may be considered front, and possibly front and center, and the floor and ceiling may be considered vertical boundaries. In some such examples, the room descriptor may also indicate boundaries corresponding to the left, right, front, and rear walls relative to the front position. According to some examples, at least some of the room descriptor information may be provided as a matrix. In some such examples, the matrix may be a 3x3 matrix, where one row or column corresponds to one dimension of three-dimensional space.

[0141] In this example, system 500 includes a lightscape renderer 501 configured to render object-based light data 505 into luminaire control signals 515, at least partially based on environment and actuator data 104. In this example, the lightscape renderer 501 is configured to output luminaire control signals 515 to a lighting controller 103 configured to control luminaires 108. Luminaires 108 may include individual controllable light sources, groups of controllable light sources (such as controllable tape lights), or combinations thereof. In some examples, the lightscape renderer 501 may be configured to manage various types of light object metadata layers, examples of which are provided in this specification. According to some examples, the lightscape renderer 501 may be configured to render actuator signals for luminaires, at least partially based on the viewer's viewpoint. If the viewer is in a living room including a television (TV) screen, the lightscape renderer 501 may, in some examples, be configured to render actuator signals to the TV screen. However, in virtual reality (VR) use cases, the lightscape renderer 501 may be configured to render actuator signals for the position and orientation of the user's head. In some examples, the lightscape renderer 501 may receive inputs from the playback environment, such as light sensor data corresponding to ambient light and camera data corresponding to the position or orientation of a person, in order to extend the rendering.

[0142] In some examples, the lightscape renderer 501 is configured to receive object-based lighting data 505, which includes object-based lighting metadata indicating light objects and the intended lighting environment, as well as environment and lighting data 104 corresponding to lighting fixtures 108 and other features of the local playback environment, which may include, but are not limited to, reflective surfaces, windows, uncontrollable light sources, and light occlusion features. In this example, the local playback environment includes one or more loudspeakers 109 and one or more display devices 510.

[0143] In some examples, the lightscape renderer 501 is configured to calculate how to excite various controllable luminaires 108 based at least partially on object-based optical data 505 and environment and luminaire data 104. The environment and luminaire data 104 may, for example, indicate the geometric location of the luminaires 108 in the environment, luminaire type information, etc. In some examples, the lightscape renderer 501 may be configured to determine which luminaires are activated based at least partially on location metadata and size metadata associated with each optical object, for example, by determining which luminaires are in the volume of the regenerated environment corresponding to the location and size of the optical object at a particular time indicated by optical object timestamp information. In this example, the lightscape renderer 501 is configured to send a luminaire control signal 515 to the optical controller 103 based on the environment and luminaire data 104 and object-based optical data 505. The luminaire control signal 515 may be sent via one or more of various transmission mechanisms, application programming interfaces (APIs), and protocols. The protocols may include, for example, Hue API, LIFX API, DMX, Wi-Fi, Zigbee, Matter, Thread, Bluetooth Mesh, or other protocols.

[0144] In some examples, the lightscape renderer 501 may be configured to determine the drive level of each of one or more controllable light sources that approximate the lighting environment intended by the creator of the object-based light data 505. According to some examples, the lightscape renderer 501 may be configured to output a drive level to at least one of the controllable light sources.

[0145] In some examples, the lightscape renderer 501 may be configured to reduce one or more parts of the luminaire map according to content metadata, user input (mode selection), luminaire limitations and / or configuration, other factors, or a combination thereof. For example, the lightscape renderer 501 may be configured to render the same control signal to two or more different lights in the playback environment. In some such examples, the two or more lights may be placed close to each other. For example, the two or more lights may be different lights of the same actuator, or different bulbs in the same lamp, for example. The lightscape renderer 501 may be configured to reduce computational overhead and increase rendering speed by rendering the same control signal to two or more different but close-proximity lights, rather than calculating very slightly different control signals for each bulb.

[0146] In some examples, the lightscape renderer 501 may be configured to spatially upmix object-based illuminating data 505. For example, if object-based illuminating data 505 is generated for a single plane, such as a horizontal plane, the lightscape renderer 501 may be configured to project the illuminating objects of the object-based illuminating data 505 onto an upper hemisphere (e.g., above the actual or expected position of the user's head) to enhance the experience.

[0147] In some examples, the lightscape renderer 501 may be configured to apply one or more thresholds, such as one or more spatial thresholds and one or more luminous intensity thresholds, when rendering actuator control signals to optical actuators in the regeneration environment. Such thresholds may, in some examples, prevent some light objects from causing the activation of some luminaires.

[0148] In some implementations, the lightscape renderer 501 can be configured to adapt to changing conditions. Several examples of implementations of the lightscape renderer 501 are described in more detail below.

[0149] Light objects can be used to set the atmosphere of a room, provide spatial information about a character or object, enhance special effects, create greater interaction and immersion, shift the viewer's attention, or puncture content. Some of these purposes may be expressed by the content creator, at least in part, according to sensory object metadata types and / or characteristics that are generally applicable to various types of sensory objects, such as object metadata indicating the location and size of the sensory object.

[0150] For example, the priority of sensory objects, including but not limited to light objects, may be indicated by sensory object priority metadata. In some such examples, sensory object priority metadata is considered when multiple sensory objects are simultaneously mapped to the same instrument in the playback environment. Such priority may be indicated by light priority metadata. In some examples, priority may not need to be indicated via metadata. For example, MS renderer 001 may prioritize moving sensory objects, including but not limited to light objects, over stationary sensory objects.

[0151] A light object can potentially trigger multiple light excitations depending on its position and size, as well as the position of luminaires in the rendering environment. In some cases, when the size of a light object encompasses multiple lights, the renderer may apply one or more thresholds, such as one or more spatial thresholds or one or more luminous intensity thresholds, to control the object from activating some of the encompassed lights.

[0152] Example of using a lighting map In some implementations, a lighting map, which is an instance of an actuator map (AM) containing a description of the lighting in the playback environment, may be provided to the lightscape renderer 501. In some such examples, the environment and lighting fixture data shown in Figure 5 may include a lighting map. In some examples, the lighting map may be other-centered, for example, showing light attenuation based on absolute spatial coordinates, while in other examples, the lighting map may be egocentered, for example, a light projection mapped onto a sphere at the intended viewing position and orientation. In the case of a sphere, the lighting map may, in some examples, be projected onto a 2D surface to utilize a two-dimensional (2D) image texture in processing. In any case, the lighting map should indicate the capabilities and lighting settings of the playback environment, such as a room. In some embodiments, the lighting map may not be directly related to the physical characteristics of the room, for example, if adjustments have been made based on the preferences of a particular user.

[0153] In some examples, a lighting map may exist for each luminaire or for each light source in the playback environment. According to some examples, the intensity of light shown by the light map may be inversely correlated with the distance to the center of the light, or approximately inversely correlated with the distance to the center of the light (e.g., within plus or minus 5%, within plus or minus 10%, within plus or minus 15%, within plus or minus 20%). The intensity values ​​of the light map may indicate the intensity or influence of a light object on a luminaire. For example, as a light object approaches a light bulb, the lightscape renderer 501 may be configured to determine that the bulb intensity increases as the distance between the light object and the bulb decreases. The lightscape renderer 501 may be configured to determine the rate of this transition based at least in part on the intensity of light shown by the light map.

[0154] Figure 6A shows an exemplary light map of a table lamp. Region 602 shows the lamp's position mapped onto polar coordinates from the primary viewer's position. In some examples, the lightscape renderer 501 is configured to map lightscape objects to a common rendering space that can be other-centered or self-centered. Figure 6B shows an example of a self-centered light map. In this example, Figure 6B shows the mapping of a spot light object onto a sphere at the intended viewing position and orientation. In Figures 6A and 6B, darker areas are indicated by dots that are relatively close to each other, while brighter areas are indicated by dots that are relatively far apart.

[0155] Within this common rendering space, in some examples, the lightscape renderer 501 may be configured to calculate the light activation metric for each light using a dot product multiplication between the light object and the light map, for example, as follows:

number

[0156] In the above equation, Y represents the photoactivation metric, LM represents the illumination map, and Obj represents the map of the light object. The photoactivation metric indicates the relative light intensity of the actuator control signal output by the lightscape renderer 501 based on the overlap between the light object and the light spread from the luminaire. In some examples, the lightscape renderer 501 may use the maximum or minimum distance from the light object to the luminaire, or other geometric metrics, as part of determining the light intensity. In some implementations, the lightscape renderer 501 may determine the photoactivation metric by referring to a lookup table instead of calculating it.

[0157] The lightscape renderer 501 may repeat one of the above steps to determine the photoactivation metric for all light objects and all controllable lighting in the regenerated environment. Thresholding for light objects that have a very low impact on luminaires may help reduce complexity. For example, if the effect of a light object causes activation below a threshold percentage for luminaire activation, such as less than 10% or less than 5%, the lightscape renderer 501 may ignore the effect of that light object.

[0158] The lightscape renderer 501 can then use the resulting light activation matrix Y, along with various other properties such as selected panning rules or light object priorities (indicated by either the light object metadata or the renderer configuration), to determine which objects are rendered and how by which lighting. Rendering light objects to illuminator control signals may include: Change the brightness of a light object as a function of the distance from the light fixture; Mixing the colors of multiple light objects rendered simultaneously (multiplexed) by a single light fixture; or Modify one of the above based on the priority of the light object. Several detailed examples are disclosed in this specification.

[0159] Rendering parameters In addition to the information carried by the light object metadata, the rendering of light objects may also be a function of the settings or parameters of the lightscape renderer 501 itself. This may include: Speed ​​Priority - When this parameter is set, moving light objects are given higher priority than stationary light objects. Setting the Speed ​​Priority parameter improves the dynamism of the rendered scene. Color Priority - Light objects with high saturation values ​​are given priority. Activation threshold - the minimum photoactivation Y that must be achieved to activate a luminaire. Accessibility - Certain colors may be prioritized over others to best represent the experience of colorblind users. Certain flash speeds may be avoided for people with photosensitivity.

[0160] Rendering configuration (mode) In addition to the information carried by the light object metadata, the lightscape renderer 501 may be configured according to different modes in some implementations. The term “mode” as used in this specification is different from “parameter,” in that a mode may involve, for example, entirely different signal paths, while a parameter may simply parameterize these signal paths. For example, one mode may involve projecting all light objects onto the lighting map before determining how / what to render on the luminaires, while another mode may simply snap the highest-priority lights to the nearest luminaires. Modes may include: Modes that support a small number of lighting fixtures. In these modes, rendering parameters and light object metadata are used to determine which subset of light objects should be rendered and how they should be rendered. Here, "how" refers to the trade-offs between the spatial, chromatic, and temporal fidelity of the most prominent light objects in the scene; A mode that supports different content types such as music and games; A mode in which multiple light objects can be rendered by a single light source (or single light) using color mixing; A mode that allows rendering only a single light object using a single light source (or single light source); A mode in which the brightness of a light object is changed as a function of the geometric distance or other distance between the light object and the luminaire.

[0161] Color mixing and prioritization Several implementations of the lightscape renderer 501 may implement one or more color mixing methods, prioritization methods, or combinations thereof. For example, if there are multiple light objects that simultaneously affect the same luminaire, the lightscape renderer 501 may implement a color prioritization algorithm. In the simplest embodiment, only one light object affects the luminaire. In some examples, the specific light object affecting the luminaire may be determined by one of the following criteria: (1) the light object closest to the luminaire, which may be called the "snap-to-color" light object; (2) the light object that has the highest percentage influence on that luminaire; (3) the light object with the highest predefined priority; or (4) the brightest light object. In each of these conditions, in some examples, only a reference color (potentially with reduced luminance) may be shown.

[0162] In some cases, mixing lighting may be desirable. For example, when simulating the physical properties of a multi-colored light fixture with a single light fixture, modeling the physical properties of light mixing can make the rendering more realistic. This is best done in a physics-based, perceptually uniform color space such as XYZ. In some cases, light mixing can be a linear process within a color space. According to some examples, when there are multiple light objects affecting the same light fixture simultaneously, light mixing may involve mixing colors and adding intensity.

[0163] According to some examples, the lightscape renderer 501 may be configured to calculate the light mixing of two light objects as follows:

number

[0164] In this example, mixing is performed in the XYZ color space. Here, XYZ newThe alpha and beta values ​​represent the result of light mixing, where XYZ1 represents the color of the first light object, α represents a constant representing the weighting of the color of the first light object, XYZ2 represents the color of the second light object, and β represents a constant representing the weighting of the color of the second light object. The alpha and beta values ​​may correspond, for example, to the amount of decrease in light intensity due to the distance from each light object to the luminaire. The alpha and beta values ​​may be extracted, for example, from a lighting map or from some other potentially geometric model.

[0165] When physical modeling is computationally too expensive, faster methods allow for approximation. In one embodiment, the lightscape renderer 501 may be configured to calculate the light mixing of two light objects in the HSV color space as follows:

number

[0166] A typical lightscape scene can have anywhere from a few light objects to dozens of light objects that are active at any given time. Content creators generally want to control how these light objects interact with each other within the lightscape renderer 501 when rendered on luminaires in the endpoint. This essentially means that the creator may want to control (1) the relative priority of the light objects, and (2) how the properties of the light objects can and cannot be mixed together. The latter is a significant deviation from object-based audio rendering procedures, as the lightscape renderer 501 typically cannot simply mix the effects of multiple light objects to compute the actuator control signal for a single actuator. When there are many light objects (even just three), the resulting colors from simply mixing the effects of multiple light objects, combined with the projection of color from the object space region onto the luminaire, introduce some distortion, and generally do not represent the creative intent or preserve the fidelity of the scene. Furthermore, it is limited by the limited output capacity of the luminaires in the endpoint.

[0167] The aforementioned issues highlight the importance of providing content creators with some control over the operation of the lightscape renderer 501. Various disclosed examples provide content creators with the ability to define layers of light objects, the priority on those layers, and the ability to control how light objects are mixed on the layers, how the layers are mixed, or a combination thereof.

[0168] Figure 7 shows elements of a litescape renderer in several examples. As with other figures provided in this specification, the number and types of elements shown in Figure 7 are merely examples. Other implementations may include more, fewer, and / or different types and numbers of elements. In this example, litescape renderer 501 is an example of litescape renderer 501 described with reference to Figure 5. In several examples, litescape renderer 501 may be implemented by one or more examples of control system 110 in Figure 1A.

[0169] According to this example, the lightscape renderer 501 includes the following elements: 705: A light object, which is an instance of object-based sensory data 005 disclosed in this specification; 004: Environmental and actuator data; 723: Lighting map (LM), which is an instance of actuator map (AM) containing a description of lighting in the playback environment; 750: An example of projection module 450 in Figure 4, a projection module configured to project light objects using an LM723; 740: An example of actuator activation matrix (AAM) 440, which is the light activation matrix (LAM) output of projection module 750; 751: An example of the mixing module 451 in Figure 4, the mixing module 751 configured to convert LAM740 into actuator command 741; 741: Actuator command 741, in this example, is sent directly to the luminaires, which are example actuators 008, to control them, but in other examples, it may be sent to the lighting controller API 103, which then sends the corresponding control signal to the luminaires 008; 752: This is renderer configuration data, which may include settings such as desired dynamics and modes. In some disclosed context-aware examples, renderer configuration data 752 may be automatically modified; 702: an in-layer mixing module configured to mix light objects of the same layer in accordance with a mixing law; 710: photoactivation vectors, in this example, the photoactivation vectors 710 of all layers including activation values of all lighting fixtures and mixed colors; 703: an inter-layer blending module configured to blend the photoactivation vectors 710 together to obtain a rendered actuator command 741.

[0170] According to some examples, a light activation matrix (LAM) 750 has a size N o ×N A real matrix, which is denoted as A in the following formula:

Formula

[0171] In some examples, there may be one matrix A for all layers being processed by the lightscape renderer 501. Accordingly, the lightscape renderer 501 can define a tensor of size N o ×N A ×N l (the dimension order is arbitrary), where N lThis represents the number of layers in the lightscape renderer 501. In this specification, A can represent either a tensor or a matrix. From the context, the reader will understand what it is.

[0172] In some examples, the in-layer mixing module 702 is configured to mix all light objects on a given layer. This process involves creating a matrix A (of that layer) of size 1 × N. A It is compressed into a photoactivation vector 710v. Matrix A contains the activation value but not the color value. The in-layer mixing module 702 can be expressed in its general form as a function that generates the photoactivation vector v(710), as follows:

number

[0173] However, in this example, the in-layer mixing process also generates a vector c of all layers in the matrix containing the color mixing results of that layer, so the above equation can be modified as follows:

number

[0174] In the above equation, f1() is the activation vector v, and A and c o It outputs both the mixed color c within the layer as a function of c o This includes all the colors of the object, with a length of N. o It is a vector.

[0175] The inter-layer blending process is size N A ×N l Take all N1 v and c vectors packed into the v' and c' matrices, and include the color of each actuator, length N A Output a single vector o. This can generally be written as follows:

number

[0176] An example of f1() is given in the section on exemplary object mixing laws of this disclosure. An example of g1() is given in the section on example layer blending laws of this disclosure.

[0177] Exemplary Photoactivation Law Light activation laws can range from simple geometric projections and distances to complex responses involving pre-calculation and storage in lookup tables (LUTs). An example of the latter is lighting maps. In some examples, the LUT provides activation values ​​that may be based on the size, position, layer, velocity, and potentially other parameters of an object. These activation values ​​may not represent how much light the actual light or luminaire projects onto the reproducible environment. In some examples, the activation values ​​may be optimized to provide a sparse matrix A (which simplifies mixing and potential object prioritization problems) based on such (measured or simulated) projections.

[0178] Simple geometric activation laws can be applied to different reference frames, such as other-centered or ego-centered reference frames. Such activation laws can be applied using various coordinate systems, such as Cartesian, spherical, or cylindrical coordinate systems.

[0179] The following is an example of an other-centered orthogonal activation law:

number

[0180] An implementation of the self-centered spherical activation law may be substantially similar to the Euclidean implementation described above, but the coordinates are first transformed such that the user's position and orientation define the origin (the positions of illumination and objects are referenced from this position and orientation), and position and size here have angles. Distances and activation functions that can be used instead of d include, but are not limited to, the Euclidean distance, p-norm distance, cosine distance, logistic function, Gaussian activation function, and normalized linear activation.

[0181] Additional variations on the activation law may include: Warping the coordinates of an object to consider multiple user perspectives (expanding the sweet spot); and / or Use an arbitrary reference frame, for example, a rotated frame to account for TVs / screens that are not orthogonally positioned within the playback environment.

[0182] Exemplary object mixing rules A simple implementation of f1() is for all N in A o Simply sum across the rows to generate v, and use these to c o We accumulate c by weighting it. More specifically, for the j-th light, v and c can be calculated as follows:

number

[0183] Some implementations may include performing normalization across the column A before performing the addition. This normalization may be linear, for example:

number

number

[0184] However, these normalizations result in the column sums becoming 1, which can be undesirable when performing cross-layer mixing. Some examples involve extending the Softmax function to return to the same range, resulting in the column sums being equal after non-linear normalization (which here is scaling, not norming), for example, as follows:

number

[0185] The scaling and / or normalization described above may, in some examples, be performed before only the top N object elements contribute to the j-th light. Normalization and scaling are motivated by the fact that it may not be desirable to saturate the color c[j]. Given an example involving an RGB color model in the range [0,1], the above mixing rule can produce a result greater than 1. In such cases, clipping may be performed to send a valid RGB codeword to the luminaire. However, clipping introduces a chromaticity error, and in the worst case, where all three RGB channels are saturated, the output is white. This is undesirable and provides further motivation to take only the top N (e.g., N=2) when performing the above addition.

[0186] In some examples, the lightscape renderer 501 can be configured for scene mixing. According to some examples, screen mixing can be performed as follows: [Math.]]

[0187] According to some examples, the lightscape renderer 501 may be configured for screen mixing with or without activation weighting as exemplified in the above screen mixing.

[0188] Exemplary layer blending rules In some implementations, there is no fundamental difference between mixing performed in an intra-layer process and blending performed in an inter-layer process. Blending is typically used to refer to the process of combining multiple layers in image processing and computer graphics. Therefore, to avoid confusion and help describe intra-layer and inter-layer processes, the terms "mixing" and "blending" are used herein.

[0189] To blend layers, in one example, alpha compositing, specifically A-over-B alpha compositing, may be used. Recall that blending layers is the process of generating an output vector o from matrices v' and c': [Math.]] In the above formula, v' and c' are of size N A ×N l as follows: [Math.]] In the above formula for v', v i,j represents the net activation value of light objects mixed to the i-th luminaire on the j-th layer. These light objects are mixed to generate c i,j color.

[0190] In some examples where the Lightscape Renderer 501 uses multiple layers, the order of the layers may imply some semantics or priority. For example, in some examples, there may be an environment layer, a spatial layer, and an overlay layer. In some such examples, the Lightscape Renderer 501 may render these layers in order, in other words, the environment layer may be rendered first, then the spatial layer, and then the overlay layer. According to some such examples, the Lightscape Renderer 501 may implement a method such as alpha blending to render these layers, and may use a net activation value v as a proxy for the alpha value to blend these layers.

[0191] The following is an example of using A over B alpha blending to execute the function g1() when using the RGB color model.

number

number

[0192] Abstract objects and effects The disclosed light objects do not need to be limited to individual objects with several other parameters such as position, size, color, and possibly feather distance. Instead, light objects can be thought of as abstract light sources. Light objects can provide powerful data augmentation within the lightscape renderer 501, thereby reducing the data rate required to transport the experience, lowering the workload of content creators, and simplifying the control of complex behavior.

[0193] Examples of the effects of a slice room For example, an effect can be used to generate a wavefront that propagates throughout the entire playback environment. Instead of the content creator manually defining multiple wavefronts and building tightly coupled loops, in some examples the content creator can simply select an abstract light object effect, which in this example we'll call a 'slice room' light object. According to some examples, this abstract light object can be parameterized as follows: p: The position that is the effective phase center of the wavefront; s: The size that determines the area within the endpoint to which this effect applies; D: Density, which is the number of wavefronts in each dimension of the regeneration environment; f slice :Frequency is the speed at which these wavefronts propagate through the regeneration environment: d: Duty cycle, which is the proportion of a single period in the wavefront considered active; Color: Color renderer for the active section; Alternative color: The color rendered for inactive sections.

[0194] In some examples, the slice room effect can be implemented using a simple 3D oscillator to determine whether a spatial position is active or inactive. The position of a particular location at an endpoint can be expressed as follows:

number

Num

Num

Num

[0195] The phase of a time oscillator operating at frequency f slice can be expressed as follows:

Num

Num

[0196] Figure 8 shows an example of the Slice Room effect. In this example, the dotted areas represent areas of active spatial coordinates, and the non-dotted areas represent areas of inactive spatial coordinates. Due to the periodic nature of the Slice Room effect and the ability to parameterize the oscillator, in alternative examples, the non-dotted areas could represent active spatial coordinates, and the dotted areas could represent inactive spatial coordinates. In some examples, a light object can move to various positions within the playback environment over time, causing a modulation effect within the rendered scene. Other parameters of the light object can be modulated in some examples to further enrich the MS experience.

[0197] Ambient Fill In some implementations of the Lightscape Renderer 501, layers can be implemented using certain defined properties or semantics. For example, on the environment layer, light objects may be rendered in a "fill" mode where multiple objects on the environment layer completely fill it among themselves. In some examples, blank / black spaces may be defined on the environment layer by placing null / black objects.

[0198] Figure 9 shows an example of environmental filling. According to this example, different types of filling correspond to different colors, as shown below. This example shows a 2D diagram of environmental filling. The environment layer contains the following four objects: A null (black) object 905 of size 1 located at x,y=(50,10) corresponds to empty or black regions 910a and 910b within region 910; Within the red region 920, there is a red object 915 with size 0.1 located at x,y=(10,30); Green object 925 with size 0.5 located at x,y=(80,50) within the green region 930; and Blue object 935, size 1, is located at x,y=(50,80) within the blue region 940. In this example, the user is located at (50,50).

[0199] Figure 9 shows the results when spatial coordinates (x,y) are assigned colors based on minimizing the following ratio r:

number

[0200] If we define the user position u as follows:

number

number

[0201] Similarly, a unit vector from the user to the light object i may be defined as follows:

number

number

[0202] The apparent size of each light object as seen from the user's position can be expressed as follows:

number

number

[0203] This ratio applies to all objects. i The color assigned to a specific spatial position associated with object j that minimizes can be represented as follows:

number

number

[0204] Ratio I can be used to determine the contribution of each of the two objects being mixed. In this example, a "filling" function is being implemented. Thus, at any given spatial position or luminaire, the contributions of all light objects (in this invention, only two light objects are specified) to that spatial position are normalized.

[0205] In some examples, when the ratio I is sufficiently large, it is possible to implement a spatial control function in which only one light object contributes to the lightscape. However, this will introduce a potentially sharp discontinuity in the spatial domain on the environment layer. As a result, in some examples, alpha synthesis can be used on the environment layer itself to mix the top two light objects contributing to any particular spatial position. To implement some such techniques, alpha a and α b It is necessary to determine the order in which objects are mixed and to select the type of alpha blending. Unlike blending multiple layers within a renderer, the order in which objects are mixed is generally not predetermined. It is extremely difficult to define a mixing order that is consistent across the entire spatial domain.

[0206] Therefore, some examples involve using the mixing order provided by I1 and I2. This mixing order generally changes between the I1 and I2 objects as they pass through inflection points where their ratios are equal. As a result, using A over B alpha blending produces sharp discontinuous steps in the output due to swapping of which light object is considered "over" the other light object.

[0207] Figure 10 shows an example of an alpha composite response. In Figure 10, the composite response is shown on the vertical axis and the position is shown on the horizontal axis. The position shown on the horizontal axis indicates the position of the light object relative to the first luminaire located at position zero (0) and the second luminaire located at position 1 (1). Figure 10 shows the Alpha_1 and Alpha_2 composite response, not the Alpha_A or Alpha_B composite response. This is for clarity, as a continuous plot (e.g., the blue Alpha_1 plot) is associated with a single light object. This makes sense when considering the implementation of the present invention. Looking at the plots of Alpha_1 and Alpha_2 for the A over B implementation, a sharp discontinuity can be observed at the object boundary (position = 0.5). This is because, as the position increases, there is a transition from the association of light object 1 with Alpha_A to the association with Alpha_B. Note that this discontinuity does not exist in the A atop B function. As a result, in the case of spatial blending, A atop B (for example) synthesis is preferable to A over B synthesis.

[0208] It is desirable to provide some control over the sharpness in the transition between any two light objects and to ensure that the environment layer does not simply become a wash of all the colors of the objects on the layer. In one example of a method to achieve these goals, the alpha component may be determined as follows:

number

[0209] The above equation is, in effect, the Softmax activation function, and therefore αa +α b = 1.

[0210] Next, A-to-B compositing can be used. For example, for the j-th luminaire, q is the position of the luminaire, and the lightscape renderer 501 can find the I_1 and I_2 objects and calculate their alpha values ​​as described above. The lightscape renderer 501 can then determine the mixed color values ​​for that luminaire on the environment layer as follows.

number

[0211] Figure 11 shows an example of a 3D environment filling result. In this example, the environment filling is displayed in polar coordinates using the same scenario as described at the beginning of this section. In this example, the elevation angle is shown on the vertical axis and the azimuth angle is shown on the horizontal axis. According to this example, the user is positioned on the z-plane as well as all light objects. This layout is equivalent to the layout shown in Figure 12(D). In the example shown in Figure 11, region 1110 is black, region 1120 is red, regions 1130a and 1130b are green, and region 1140 is blue.

[0212] Figures 12(A), (B), (C), (D), (E), and (F) show additional examples of environment-filling responses. In these examples, region 1210 is black, region 1220 is red, region 1230 is green, and region 1240 is blue. According to these examples, the black light object 1205, red light object 1215, green light object 1225, and blue light object 1235 are in the same positions as the black light object 905, red light object 915, green light object 925, and blue light object 935 shown in Figure 9, respectively. These examples show the range of environment-filling responses corresponding to the range of objective cost functions (columns) for different reference systems (rows) in a two-dimensional spatial domain. The rows including (A), (C), and (E) in Figure 12 show examples of environment-filling responses according to other-centered Euclidean distance. The rows containing Figures 12(B), (D), and (F) show examples of environment-filling responses according to self-centered distance in the spherical region. The response in Figure 12(D) is equivalent to the response shown in Figure 11, although it lacks alpha synthesis. The cost functions applied to generate the examples shown in each column are shown above Figures 12(A), 12(C), and 12(E). The self-centered spherical response is suitable for layouts where luminaires are distributed along and near walls and ceilings, while the other-centered rectangular response is better suited to densely populated endpoints where the distance of light objects from the user can be better represented at the endpoint.

[0213] Object Priority Sensory objects, including but not limited to light objects, may have priority levels associated with them, for example, according to sensory object metadata. The ability to assign sensory object priority levels provides content creators with a control mechanism. Alternatively or additionally, in some implementations, the renderer may, for example, implicitly or explicitly assign priorities to layers associated with sensory objects to control which sensory objects are prioritized during the sensory object rendering process. In some examples, the priority is of length N. oIt can be abstractly represented as a real vector, which is shown as P in the following equation:

number

[0214] In the above formula, p j This represents the priority associated with the j-th object. In some examples, the priority can be "broadcast" to have the same number of dimensions as A, for example, as follows:

number

[0215] By representing priority with the same number of dimensions as A, instances of MS Renderer 001 can conveniently compute a priority-weighted activation matrix, which in some embodiments may be an element-wise product of A and P'. For example, Lightscape Renderer 501 may be configured to compute a priority-weighted photoactivation matrix 740 as an element-wise product of A and P'. In other embodiments, the priority-weighted activation matrix may be an element-wise product of A or P' or both after thresholding. Alternatively, instead of thresholding, MS Renderer 001 may be configured to apply a linear or nonlinear transformation function to A or P' or both. Alternatively, MS Renderer 001 may or may not be configured to combine A and P', and may be configured to continue the rendering process by using A and P' independently. When MS Renderer 001 is configured to combine A and P', MS Renderer 001 generates a priority-weighted activation matrix A'. MS Renderer 001 may be configured or configurable to modify f1() and / or g1() (the intra-layer and inter-layer mixing rules of the present invention, respectively) to adapt to the content creator's intent associated with the priority of sensory objects in such cases. For example, a simple modification of the above intra-layer blending function results in:

number

[0216] In the aforementioned formula and in the context of light objects, if using A results in the light object being mixed with a very small activation function, either because the light object is thresholded or because it contributed so little to color c that it was imperceptible in the rendered actuator signal, then using A' instead of A may result in the light object with higher priority being mixed into the luminaire. If the lightscape renderer 501 does not combine A and P' to compute A', the in-layer blending function may be expressed as follows:

number

[0217] Several examples may use the format given above to mix objects only into the j-th luminaire, which has a non-zero light activation value that has the highest priority level among all light objects on the i-th column. In such examples, the lightscape renderer 501 may be configured to mix all light objects into the j-th luminaire that satisfies both of the following conditions:

number

[0218] Some examples may involve warping the scene to allow high-priority light objects that do not have a significant activation level in A for any of the luminaires. Lower-priority light objects, which may or may not have a significant activation level in A, may be overridden by higher-priority light objects. In such cases, using the distance law described above in algorithm 1 may not be appropriate, as it will cause saturation whenever the distance between the light object and the luminaire is less than the size of the object. In some such cases, the lightscape renderer 501 may be configured to utilize one or more non-saturating functions, such as Gaussian activation.

number

[0219] In some examples, the matrix D of the non-saturated function can be constructed for convenience, for example, as follows:

number

[0220] Matrix D is useful as a metric for the spatial warping penalty of each light object relative to each luminaire, where the "warping" of the light object's position increases as the distance between the two increases. When this distance is used in an in-layer mixing function, the function can be expressed as follows:

number

[0221] In some cases, general optimization techniques for f1() may be used to solve for v and c.

[0222] Figure 13 shows an example of a graphical user interface (GUI) that may be presented by the display device of the lightscape creation tool of Figure 5. As with other figures provided in this specification, the number and types of elements shown in Figure 13 are merely examples. Other GUIs presented by the lightscape creation tool may have more, fewer, and / or different element types and numbers. According to some examples, GUI 1300 may be presented on the display device in accordance with commands from an instance of the control system 110 of Figure 1A configured to implement the lightscape creation tool 100 of Figure 5.

[0223] In this example, the user can interact with GUI 1300 to create a light object and assign light object properties that can be associated with the light object as metadata. In this example, the user is selecting properties for light object 1330. In this example, GUI 1300 displays light object 1330 in 3D space 1331, the latter representing the playback environment. Element 1334 indicates the coordinate system of 3D space 1331. Thus, in this example, light object 1330 and 3D space 1331 are viewed from the top left.

[0224] The user may interact with the GUI 1300 to select the position and size of the light object 1330. In some examples, the user may select the position of the light object 1330 by dragging it to a desired position in three-dimensional space 1331, for example by touching a touchscreen or using a cursor. In some examples, the user may select the size of the light object 1330 by selecting the size of a circle (or other shape) shown on the GUI 1300 to indicate the outline of the light object 1330. In some such examples, the user may reduce the size of the light object 1330 by pinching its outline with two fingers, and increase its size by spreading it with two fingers, and so on.

[0225] By specifying the location and size of MS objects in an abstracted 3D space such as the 3D space 1331 of GUI1300, content creators can generalize the location and range of corresponding MS effects without prior knowledge of the specific playback environment in which the MS effect is provided. This is an advantage of MS object-oriented approaches in various disclosed implementations. For example, GUI1300 allows content creators to specify the location and size of a light object 1330 in the 3D space 1331, thereby enabling content creators to generalize the location and range of corresponding light effects without prior knowledge of the specific size of any particular playback environment in which the light effect is provided, or without prior knowledge of the number, type, and location of luminaires within the playback environment in which the light effect is provided. Luminaires that are potentially activated in response to the presence of the light object 1330 at a particular time are luminaires in the volume of the playback environment corresponding to the location and size / range of the light object 1330.

[0226] In this example, the user may interact with the color circle 1335 of GUI 1300 to select the color of the current light object, and may interact with the slider 1336 to select the brightness of the current light object. These and other selectable properties of the light object 1330 are displayed in area 1332 of GUI 1300. In this example, the properties of the light object 1330 that can be selected via GUI 1300 also include intensity, diffuseness, "feathering", whether the light object is hidden or not, saturation, priority, and layer. Light object layers and priority are described in more detail below. Generally speaking, light object layers can be used to group light objects into categories such as "environment" and "dynamic". Light object priority is assigned by the content creator and can be used by the renderer to determine, for example, which light object is presented when two or more light objects are active simultaneously and simultaneously contain areas with the same luminaire.

[0227] Area 1340 of GUI1300 displays the time information corresponding to each of the multiple light objects created via the Lightscape Creation Tool. In this example, the light objects are listed on the left side of area 1340 along the vertical axis, and the time is shown along the horizontal axis. In this example, a 4-second time interval is drawn by a vertical line. Here, the time information for each light object is shown as an isolated or connected diamond symbol or line along a series of horizontal rows, and each diamond symbol or line corresponds to one of the light objects shown on the left side of area 1340. For example, line 1333 indicates that light object 3 begins to appear between 39 and 40 seconds and is displayed continuously until approximately 1 minute and 6 seconds. The diamond symbol to the right of line 1333 indicates that light object 3 is displayed discontinuously for the next few seconds.

[0228] Figure 14 shows another example of a graphical user interface (GUI) that may be presented by the display device of the lightscape creation tool of Figure 5. As with other figures provided in this specification, the number and types of elements shown in Figure 14 are merely examples. Other GUIs presented by the lightscape creation tool may have more, fewer, and / or different element types and numbers. According to some examples, GUI 1400 may be presented on the display device in accordance with commands from an instance of the control system 110 of Figure 1A configured to implement the lightscape creation tool 100 of Figure 5.

[0229] In this example, GUI1400 represents a moment in which lighting fixtures in an actual playback environment are controlled according to light objects created by one implementation of the lightscape creation tool 100 in Figure 5. An image of the playback environment is shown in area 1405 of GUI1400. Various lighting fixtures 1408 and a television 1415 are shown in the playback environment in area 1405. A specific moment is indicated by a vertical line 1442 in area 1440. At this time, the vertical line 1442 intersects with horizontal lines 1444a, 1444b, 1444c, and 1444d, indicating that the light supplied to the corresponding light objects 1, 4, 5, and 7 is being reproduced. Area 1432 shows the characteristics of the light objects.

[0230] At the point shown in Figure 14, it can be observed that the left side of the regeneration environment shown in region 1405 is illuminated by blue light. This corresponds, at least in part, to the effect of the light object 1430 shown in three-dimensional space 1431.

[0231] In this example, video and audio data are also played in the audio environment, and the playback of the rendered light object is synchronized with the playback of the video and audio data. In this example, the image of the played video is shown in area 1410 of GUI 1400. The video may be played by, for example, a television 1415.

[0232] In some examples, the user may be able to interact with GUI1400 to adjust the properties of optical objects, add or delete optical objects, etc. For example, the user may pause playback to adjust the properties of optical objects. In some alternative examples, the user may need to return to a GUI such as GUI1300 in Figure 13 to adjust the properties of optical objects, add or delete optical objects, etc.

[0233] In the example illustrated with reference to Figure 14, the GUI 1400 was presented on a display device corresponding to the lightscape creation tool 100 in Figure 5, while the light objects, audio, and video were rendered in an actual real-world environment. Thus, in some implementations, the example illustrated with reference to Figure 14 may also include, but are not limited to, the functions of the lightscape renderer 501, the light controller API 103 which may be implemented in some cases by the same device implementing the lightscape renderer 501, the lighting fixture 108, the audio renderer 106, the loudspeaker 109, the video renderer 107, and the display device 510, as well as at least some of the “downstream” rendering and playback functions that may be provided by other blocks in Figure 5. In some such examples, the process illustrated with reference to Figure 14 may also include the functions of the experience player 102 in Figure 5.

[0234] In some alternative implementations, the example described with reference to Figure 14 may also include, but are not limited to, the functionality of the MS Renderer 001, the MS Controller API 003 (which in some cases may be implemented by the same device implementing the MS Renderer 001), the Lighting Fixture 008, the Audio Renderer 006, the Loudspeaker 009, the Video Renderer 007, and the Display Device 010, as well as at least some of the "downstream" rendering and playback functions that may be provided by other blocks in Figure 3. In some such examples, the process described with reference to Figure 14 may also include the functionality of the Experience Player 002 in Figure 3.

[0235] As described elsewhere in this specification, certain types of metadata and / or priority indications are specific to light objects. For example, a light object showing a changing color may have priority over a light object showing a static color. The following describes some additional metadata characteristics specific to light objects.

[0236] Lightscape Metadata Layer As described elsewhere in this disclosure, when authoring a lightscape for media content, it may be useful to identify at least two different methods (or layers) of authoring. These layers may be used in the process of rendering light objects authored according to luminaires available and controllable in the playback environment. These layers can help capture artistic intent and allow for flexibility in limitations in the playback environment, such as the number of luminaires or light occlusion, so that the author's primary intent can still be rendered, but is scalable or otherwise modified.

[0237] In some examples, direct and indirect lighting may be assigned to different lighting metadata layers.

[0238] Direct light object

[0239] Light objects within a direct light object layer, also referred to as “direct light objects” in this specification, are light objects that represent light directly visible to the content creator or end user. Examples of direct light objects may include luminaires in a scene, the sun, the moon, headlights from an approaching car, lightning during a storm, and traffic lights. Direct light objects may also be used to represent light sources that are part of the scene but are typically or temporarily invisible within the associated video content, for example, because they are outside the video frame or moving outside the video frame. In some examples, direct light objects may be used to enhance or amplify auditory events such as explosions, or to visually guide the trajectory of moving objects outside the video frame. The use of direct light objects is typically of dynamic nature. For example, associated metadata such as intensity, color, saturation, and position often change as a function of time within the scene of the media content.

[0240] Indirect light object

[0241] Light objects in the indirect light object layer, also referred to as “indirect light objects” in this specification, are light objects that represent the effect of indirect light. For example, indirect light objects may be used to represent the effect of light emitted by a luminaire, which is observed when light is reflected by one or more surfaces. Some examples of using indirect light objects include changing the observed color of the walls, ceiling, or floor of an environment to a color that matches the content, such as green for a forest scene or blue for the sky or water. Indirect light objects may also be used to set the atmosphere of a scene and environment in a similar, but more immersive, way to how video content is achieved by color grading. For example, science fiction films often use very specific (blue or greenish) video color grading palettes to enhance the sense of being in outer space. Flashback scenes often enhance the effect of timeline changes in video content by desaturating, decolorizing, or overlaying sepia tones. All of these effects can be reproduced or approximated outside the video frame by appropriately adjusting the light control signals. Lighting effects corresponding to indirectly lit objects tend to be more static within the scene, and are generally less localized and less dynamic than other lighting effects corresponding to indirectly lit objects.

[0242] Layer abstraction Some examples involve further abstracting direct and indirect light object layers into layers that encompass both aspects. In some such examples, these layers may include one or more environment layers, one or more dynamic layers, one or more custom layers, one or more overlay layers, or a combination thereof. In some examples, these layers may be used for or correspond to linear or event-based triggers in the content.

[0243] Environment layer Similar to indirect lighting, environment layers can be used to set the atmosphere and tone in a space through a wash of surface colors in the recreated environment. Environment layers may also be used as a base layer for constructing a lighting scene. In some examples, environment layers may be represented through light objects that cover a relatively large area. In other examples, environment layers may be represented through light objects that cover a relatively small area, for example, using one or more images. According to some examples, environment layers may be divided into zones. In some such examples, a particular light effect always occupies a specific area of ​​space. For example, the walls, ceiling, and floor in the authoring or recreated environment may each be considered separate environment layer zones.

[0244] Dynamic Layer In some implementations, dynamic layers may be used to represent the spatial and temporal variations of MS objects, such as light objects. Within a dynamic layer, individual MS objects may have priorities, for example, one light object may take precedence over another when presented through a light fixture. Within a dynamic layer, individual MS objects may, in some examples, be linked to other objects, such as audio objects (from spatial audio) or 3D world MS objects.

[0245] Custom Layer In some cases, custom layers can be used to design lighting sequences that can be freely assigned to lighting fixtures for functional purposes. These sequences may not be spatial in nature, but instead can provide additional information to the user. For example, in a game, tape lights could be assigned to indicate the player's remaining lifespan.

[0246] Overlay Layer As some examples show, overlay layers can be used to present persistent light with a continuous priority. Overlay layers may also be used, for example, to create a "watermark" on top of all other elements in a lighting scene.

[0247] Authoring and distribution of lightscape layer data Authoring of direct and indirect light objects In some examples, direct light objects can be authored by determining or setting the light source position, intensity, hue, saturation, and spatial range as a function of time for one or more light objects. In some such examples, this authoring process can create corresponding metadata that can be delivered along with the direct light object, along with the audio and / or video content of the content presentation. Ideally, direct light objects are rendered to a direct light source.

[0248] In some cases, indirect lighting effects may be authored as a dedicated group or class within the lightscape metadata content, focusing on overall color and atmosphere rather than dynamic effects. Indirect lighting effects can also be defined by intensity, hue, saturation, or a combination thereof as a function of time, but are typically associated with a significant portion of the lightscape rendering environment. Indirect lighting effects, where available, are ideally (but not necessarily) rendered with indirect lighting.

[0249] Rendering lightscape layer data Layer attributes and their metadata can be rendered by a lightscape renderer, such as the lightscape renderer 101 in Figure 5. In some examples, the lightscape renderer may be configured to send luminaire control signals 515 to luminaires in the replay environment. In some examples, the lightscape renderer 101 may be configured to output luminaire control signals 515 to a lighting controller 103 configured to control luminaires 108. According to some examples, the lightscape renderer uses environment and luminaire data 104 to determine the capabilities and spatial location of each luminaire. In some examples, layer priority may be a determinant of what is ultimately rendered to the luminaires.

[0250] Rendering of direct and indirect light objects In some implementations, the environment and illuminator data 104 received by the lightscape renderer includes data on whether the fixtures are directly visible from the viewing position or are indirect light sources. In some examples, if indirect lighting fixtures are not available, the indirect light data may be sent to direct lighting fixtures instead, with potentially reduced brightness.

[0251] Direct light objects are preferably rendered as visible light fixtures such as ceiling downlights, wall-mounted lights, table lamps, etc. Indirect light metadata ideally covers light fixtures that are not directly visible, such as LED strips illuminating walls, ceilings, shelves, furniture, and spotlights illuminating walls or ceilings. If such indirect light is not available, indirect light metadata can be used to control direct light instead. In some such examples, the lightscape renderer may overlay direct light object metadata and indirect light object metadata when rendering to a light fixture that functions as both an indirect and direct light source.

[0252] Figure 15 is a flowchart outlining an example of a method that may be performed by an apparatus or system as disclosed in this specification. The blocks of Method 1500, as with other methods described in this specification, are not necessarily performed in the order shown. In some implementations, one or more blocks of Method 1500 may be performed simultaneously. Furthermore, some implementations of Method 1500 may include more or fewer blocks than those shown and / or described. The blocks of Method 1500 may be performed by one or more devices which may be (or include) one or more instances of a control system, such as the control system 110 shown in Figure 1A and described above. For example, at least some aspects of Method 1500 may be performed by an instance of the control system 110 configured to implement the multisensory renderer of Figure 3. At least some aspects of Method 1500 may be performed by an instance of the control system 110 configured to implement the lightscape renderer 501 of Figure 5.

[0253] In this example, block 1505 includes the control system obtaining actuator data for a set of controllable actuators. In some examples, the set of controllable actuators may include one or more lighting fixtures, one or more sensory devices, one or more airflow control devices, or a combination thereof.

[0254] In this example, block 1510 includes receiving object-based sensory data, which includes a set of sensory objects, by a control system. In some examples, the object-based sensory data may include sensory object metadata, which may also be referred to as sensory object metadata in this specification.

[0255] In some examples, sensory object metadata may include sensory object location metadata indicating the spatial location for rendering object-based sensory data within an environment, sensory object size metadata indicating the region or volume for rendering object-based sensory data within an environment, or both. In some examples, object-based sensory data may include one or more other types of sensory object metadata.

[0256] In some examples, object-based sensory data does not correspond to specific sensory actuators in an environment. For example, as illustrated with reference to Figures 13 and 14, sensory objects in object-based sensory data may correspond to a portion of a three-dimensional region representing the reproduction environment. The actual reproduction environment in which the sensory object is rendered does not need to be known at the time the sensory object is authored, and is generally unknown. Therefore, object-based sensory data includes abstracted sensory reproduction information, and in these examples, includes sensory objects and corresponding sensory metadata that enable a sensory renderer to reproduce authored sensory effects from various sensory actuator locations in the environment, via various sensory actuator types and a varying number of sensory actuators.

[0257] In this example, block 1515 includes the control system rendering object-based sensory data to generate actuator control signals. According to this example, the rendering is at least partially based on actuator data. In some examples, method 1500 may also include obtaining user position data. According to some such examples, the rendering may be at least partially based on user position data. In some examples, audio object metadata may include at least audio object spatial metadata indicating the audio object spatial position for rendering audio signals in the environment. In some examples, the rendering may be at least partially based on data corresponding to uncontrollable actuators that are not controllable by actuator control signals.

[0258] In this example, block 1520 may include, by a control system, providing actuator control signals to one or more controllable actuators from a set of controllable actuators. In some such examples, method 1500 may also include, by a sensory renderer, receiving object-based sensory data and by a sensory renderer, receiving environment descriptor data corresponding to the replayed environment. In some such examples, method 1500 may also include, by a sensory renderer, receiving actuator descriptor data corresponding to the characteristics of sensory actuators in the environment. In some examples, the environment descriptor data and actuator descriptor data may be or be included in the environment and actuator data 004 described with reference to Figure 3. In some examples, the MS controller API 003 shown in Figure 3 may be implemented via an MS renderer 001, and actuator-specific signals may be provided to actuator 008 by the MS renderer 001. In some alternative examples, the MS renderer 001 may provide actuator control signals 310 to the MS controller API 003, and the MS controller API 003 may provide actuator-specific control signals to actuator 008. In some examples, method 1500 may also include providing sensory effects by sensory actuators in the environment.

[0259] In some examples, method 1500 may include integrating actuator data with object-based sensory data. In some such examples, integrating may include finding one or more nearest controllable actuators for each sensory object. According to some examples, the actuator data may include an actuator map. In some such examples, rendering object-based sensory data to generate actuator control signals may include projecting a set of sensory objects using an actuator map. According to some examples, projecting a set of sensory objects using an actuator map may generate an actuator activation matrix as the actuator control signal. In some examples, the actuator map may include a luminaire map. In some such examples, the luminaire map may be an other-centered luminaire map based on playback environment spatial coordinates, or an ego-centered luminaire map based on spatial coordinates related to the intended viewing position.

[0260] In some implementations, object-based sensory data may include sensory object priority metadata. In some such examples, method 1500 may include determining that two or more sensory objects map to a single actuator, and rendering the object-based sensory data of the two or more sensory objects into actuator control signals for the single actuator, at least in part based on the sensory object priority metadata.

[0261] In some examples, rendering may be based at least partially on one or more renderer configuration parameters. In some examples, one or more renderer configuration parameters may include a speed priority parameter that assigns higher priority to moving sensory objects than to stationary sensory objects. In some examples, one or more renderer configuration parameters may include a change priority parameter that assigns higher priority to changing sensory objects than to static sensory objects.

[0262] In some examples, a sensory object may include a light object. In some such examples, a light object may include color information. According to some such examples, the color information may include color change information. In some such examples, one or more renderer configuration parameters may include a color change priority parameter that assigns a higher priority to color-changing light objects than to light objects that do not change color. In some examples, a light object may include saturation information. In some such examples, one or more renderer configuration parameters may include a saturation priority parameter that assigns a higher priority to light objects with higher saturation than to light objects with lower saturation.

[0263] In some examples, rendering may be based at least partially on one or more renderer configuration modes. These may include a low actuator count mode, a content type mode, a color mixing mode, a single-sensory object versus single-actuator mode, a mode that modifies the brightness of light objects according to the distance between the light object and the luminaire, or a combination thereof. In some examples, rendering may include implementing color mixing and prioritization methods when multiple light objects are represented by a single luminaire.

[0264] In some examples where a sensory object includes a light object, the light object may have associated light object metadata. This light object metadata may include light object position data, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradient metadata, light object priority metadata, or a combination thereof. In some examples, the light object metadata may include light object layer metadata corresponding to two or more layers. According to some examples, the two or more layers may include an environment layer, a dynamic layer, a custom layer, an overlay layer, a direct light object layer, an indirect light object layer, or a combination thereof. In some such examples, rendering may involve in-layer mixing, inter-layer blending, or both. According to some examples, rendering may include applying one or more photoactivation laws. In some examples, rendering may include creating a slice room effect, creating environment filling, or both, and creating a slice room effect involves generating wavefronts that propagate throughout the entire reproduction environment.

[0265] In some examples, method 1500 may include, by an audio renderer, receiving audio objects and by an audio renderer, receiving loudspeaker data corresponding to loudspeakers in the environment. According to some such examples, method 1500 may also include, by an audio renderer, providing loudspeaker control signals to control loudspeakers in the environment to play audio corresponding to audio objects and synchronized with sensory effects. Synchronization may be based on time information contained in or having in sensory objects and audio objects, such as timestamps. In some examples, method 1500 may also include, by a loudspeaker in the environment, playing audio corresponding to audio objects.

[0266] In some examples, Method 1500 may include receiving video data synchronized with audio objects and object-based sensory data by a video renderer. In some such examples, Method 1500 may also include providing video control signals by the video renderer to control one or more display devices in the environment to present images that correspond to video control signals and are synchronized with audio objects and sensory effects. In some examples, Method 1500 may also include presenting images on one or more display devices in the environment. The images may correspond to video control signals.

[0267] Examples of context-based rendering This section describes an implementation of a multisensory renderer 001 configured to render at least in part based on local context information. This type of rendering may be referred to in this specification as “context-based rendering” or “context-aware rendering.” “Context” is or includes local context information relating to the local playback environment. For example, one aspect of local context information may be the time of day in the region containing the local playback environment, or the weather in the area containing the local playback environment. Alternatively or additionally, the context may also include or include information about one or more people in the local playback environment, such as the apparent level of engagement with the played content.

[0268] Information that enables the multisensory renderer 001 to provide context-based rendering may be explicitly provided in some examples. In some examples, such explicit information may be or include user input for managing one or more aspects of the rendering process, such as information about the overall immersion level and / or interactivity level. According to some examples, such explicit information may be or include input from a device or system that has access to information about one or more aspects of local context information, input from a device or system configured to learn local context information by analyzing sensor data, user behavior patterns, etc.

[0269] Managing the immersion and / or interactivity of the sensory experience can be achieved by changing how the multisensory renderer 001 manages the time, frequency, intensity, input, spatial effect (e.g., color or vibration) dimensions, or combinations thereof. In some examples, the multisensory renderer 001 may be configured to automatically manage the immersion and / or interactivity of the sensory experience, for example, by applying a low-pass filter to one or more of those dimensions. Alternatively, or additionally, the multisensory renderer 001 may be configured to manage the immersion and / or interactivity of the sensory experience according to sensory object metadata received with content that indicates artistic intent and includes object-based sensory data.

[0270] In some cases, when users prefer a less immersive experience, one or more relatively dynamic or relatively spatial layers of sensory content may be excluded, leaving only the environmental layer. In some cases, the intensity of one or more relatively dynamic or relatively spatial layers of sensory content may be reduced. These measures may be taken with respect to any combination of auditory, visual, and sensory experiences.

[0271] Therefore, the context-aware MS renderer 001 may be configured to render object-based sensory data 005 to the actuator control signal 310 based at least in part on local context information which may include one or more of the following: Local time. For example, if it is late at night in the current location, MS Renderer 001 may avoid rendering extremely bright light. In some cases, MS Renderer 001 may avoid light content with a strong blue component during the last hour or so before the viewer goes to bed, because this could disrupt sleep. If it is daytime in the current location, MS Renderer 001 or another device may be configured to control automatic blinds to reduce ambient light from outside, depending on the content and experience. Local weather information is determined by internet weather forecasts, live internet weather observations from nearby weather stations, and local weather information supplied by on-site weather stations connected via LAN, WLAN, or Bluetooth®. For example, if the outside of the playback environment is very sunny, high-intensity lighting may be required to overcome the light leaking into the viewing room through the windows. If it is very cloudy, the room may be dark, and therefore less light intensity may be required. In some cases, this may involve observations of human behavior over multiple days, weeks, or months. For example, based on historical information from a home security system, it may be possible to determine that at this moment, there is a high probability that only one person is in the living room, and that person is watching a movie, TV series, etc., on TV. Context-aware MS renderer 001 may determine that this is a time to enhance the TV viewing experience using sensory effects, including the living room lighting system, because it is likely that no one else in the house is doing anything different in the living room at the same time. Explicit input regarding mode switching, such as an explicit command received via user input from a person in the playback environment, to avoid extending the TV viewing experience, for example, if another member of the household is doing homework in the same room. Information indicating the presence of a specific person within the playback environment. For example, a person may have preferences regarding one or more types of sensory experiences. In some cases, a person may be a photosensitive or colorblind viewer, for whom the lighting experience should be toned down or otherwise personalized. The presence of a specific viewer may be determined by Bluetooth® or Wi-Fi beacons from a phone or smartwatch, by speaker identification using a microphone, by facial recognition, etc. Information such as light sensor information that indicates ambient light from internal (in the playback environment) light sources, external (e.g., outdoors) light sources, or both. For example, if a house has smart lights in the living room, but the kitchen is right next to the living room and has separate lighting fixtures, the kitchen lights may interfere with the light from the controllable luminaires in the living room. In some such cases, the context-aware MS renderer 001 may be configured to adapt the rendering for the controllable luminaires in the living room due to the light from the kitchen lights. For example, the MS renderer 001 may be configured to make luminaires near the kitchen relatively brighter than luminaires further away from the kitchen in order to compensate for the kitchen lights. Such compensation techniques may be particularly relevant if the MS renderer 001 is mixing colors. For example, if the kitchen lights are somewhat orange, but white light is desired, the MS renderer 001 may make the side lights slightly greenish so that the colors mix to white in the user's peripheral field of view. Information indicating the current context of one or more people in the playback environment. For example, if the current context information indicates that the user is driving a car, the MS renderer 001 may show only lower-level / less-immersive sensory objects / types to prevent driver distraction. In contrast, if the same person is sitting in the back seat while the vehicle is stationary, the MS renderer 001 may allow all metadata layers to be used. In yet another example, if the context-aware MS renderer 001 receives information such as sensor information or user input indicating that one or more people are watching TV and no one else nearby is nearby doing something constructive such as housework or homework, the context-aware MS renderer 001 may determine that this is time for relatively more immersive sensory content playback corresponding to content delivered via TV. On the other hand, if it appears that no one is watching TV, the context-aware MS renderer 001 may determine that this is time for relatively more environmental, or entirely environmental, sensory content playback corresponding to content delivered via TV. Environmental context information, including but not limited to information derived from environmental and actuator data 004. Environmental context information may include information about which devices are currently in use. In some examples, the context-aware MS renderer 001 may be configured to provide a specified egocentric view when a display screen (with visuals) is in use, and to provide a more other-centered or environmental view when only audio and lighting or only lighting is used. If the local playback environment is a vehicle environment, the local environment context information may include information about whether the vehicle is parked or moving. For example, if the context-aware MS renderer 001 receives information indicating that the vehicle is parked, the context-aware MS renderer 001 may determine that this is time for a relatively more immersive sensory content playback corresponding to content provided via the vehicle's audio system, via the vehicle's video system, or both, and Information indicating the location of one or more people in a playback environment, such as information indicating that the only person in a moving car is in the driver's seat, information indicating that the only person in a parked car is in the back seat, or information indicating that one or more people are on a sofa facing a TV. Alternatively or additionally, some examples may include optimizing the rendering of light when created for a self-centered view. This may be particularly relevant to the front versus rear effect of the user's position. For example, a light effect intended to be provided behind the user may depend on reflection or may be less likely to be rendered, depending on the availability of the fixture.

[0272] As described elsewhere in this specification, the sensory object metadata may also include information to help the renderer provide artistic intent for various contexts or user-selectable levels of immersion. For example, the sensory object metadata may include “mood” object types to indicate that the role of an object is to set the environment layer. In some examples, the sensory object metadata may include “dynamic” object types that can be used to signal to the renderer that the role of an object is to bring about change and movement. In different contexts, the context-aware MS renderer 001 may use this information to render different types of sensory experiences. For example, in an “immersive” situation where one or more people want to have a fully immersive sensory experience, such as in a living room of a house, the full range of mood and dynamic sensory objects may be used to render actuator signals for the sensory experience. However, if the “environment” context is selected by the user or system, the context-aware MS renderer 001 may use all mood objects, but only a subset of dynamic objects, or none at all. According to some implementations, this immersion control can be continuous, for example, from "immersion off" to "fully immersed," within a range such as 0 to 10 or 0 to 100. In some examples, different contexts can be sensed, categorized, and programmed to correspond to various levels of immersion. The level of immersion can vary depending on the sensory objects used, the intensity or amplitude of sensory actuator playback, the number of actuators used, etc. For example, the level of immersion can vary depending on the light objects used, the brightness of the light, the number of luminaires used, etc. The following table provides examples of immersion levels, contexts, and sensory object usage by the context-aware MS renderer 001. [Table 1]

[0273] Examples of additional lightscapes Figure 16 is a flowchart outlining an example of a method that may be performed by an apparatus or system as disclosed in this specification. The blocks of Method 1600, as with other methods described in this specification, are not necessarily performed in the order shown. In some implementations, one or more blocks of Method 1600 may be performed simultaneously. Furthermore, some implementations of Method 1600 may include more or fewer blocks than those shown and / or described. The blocks of Method 1600 may be performed by one or more devices which may be (or include) one or more instances of a control system, such as the control system 110 shown in Figure 1A and described above. For example, at least some aspects of Method 1600 may be performed by an instance of the control system 110 configured to implement the multisensory renderer of Figure 3. At least some aspects of Method 1600 may be performed by an instance of the control system 110 configured to implement the lightscape renderer 501 of Figure 5.

[0274] In this example, block 1605 includes the control system obtaining actuator data for a set of controllable actuators. In some examples, the set of controllable actuators may include one or more lighting fixtures, one or more sensory devices, one or more airflow control devices, or a combination thereof.

[0275] In the example shown in Figure 16, block 1607 includes the acquisition of local context information by the control system. The local context information may be, for example, one or more of the types of local context information disclosed in this specification. In some examples, the local context information may be, or include, local time information, local weather information, local human activity information, local user input, information about one or more viewer preferences, information about the presence or absence of one or more viewers, local ambient light information, local viewing environment information, local viewer location information, local device usage information, local viewer activity information, or a combination thereof.

[0276] In this example, block 1610 includes receiving object-based sensory data, which includes a set of sensory objects, by a control system. In some examples, the object-based sensory data may include sensory object metadata, which may also be referred to as sensory object metadata in this specification.

[0277] In some examples, sensory object metadata may include sensory object location metadata indicating the spatial location for rendering object-based sensory data within an environment, sensory object size metadata indicating the region or volume for rendering object-based sensory data within an environment, or both. In some examples, object-based sensory data may include one or more other types of sensory object metadata.

[0278] In some examples, object-based sensory data does not correspond to specific sensory actuators in an environment. For example, as illustrated with reference to Figures 13 and 14, sensory objects in object-based sensory data may correspond to a portion of a three-dimensional region representing the reproduction environment. The actual reproduction environment in which the sensory object is rendered does not need to be known at the time the sensory object is authored, and is generally unknown. Therefore, object-based sensory data includes abstracted sensory reproduction information, and in these examples, includes sensory objects and corresponding sensory metadata that enable a sensory renderer to reproduce authored sensory effects from various sensory actuator locations in the environment, via various sensory actuator types and a varying number of sensory actuators.

[0279] In this example, block 1615 includes the control system rendering object-based sensory data to generate actuator control signals. According to this example, the rendering is at least partially based on actuator data and local context information. In some examples, method 1600 may also include obtaining user position data. According to some such examples, the rendering may be at least partially based on user position data. In some examples, audio object metadata may include at least audio object spatial metadata indicating the audio object spatial position for rendering audio signals in the environment. In some examples, the rendering may be at least partially based on data corresponding to uncontrollable actuators that are not controllable by actuator control signals.

[0280] In this example, block 1620 may include the control system providing actuator control signals to one or more controllable actuators from a set of controllable actuators. In some such examples, method 1600 may also include the sensory renderer receiving object-based sensory data and the sensory renderer receiving environment descriptor data corresponding to the replayed environment. In some such examples, method 1600 may also include the sensory renderer receiving actuator descriptor data corresponding to the characteristics of sensory actuators in the environment. In some examples, the environment descriptor data and actuator descriptor data may be or be included in the environment and actuator data 004 described with reference to Figure 3. In some examples, the MS controller API 003 shown in Figure 3 may be implemented via the MS renderer 001, and actuator-specific signals may be provided to actuator 008 by the MS renderer 001. In some alternative examples, the MS renderer 001 may provide actuator control signals 310 to the MS controller API 003, and the MS controller API 003 may provide actuator-specific control signals to actuator 008. In some examples, method 1600 may also include providing sensory effects by sensory actuators in the environment.

[0281] In some examples where local context information includes local time information, a set of controllable actuators may include a set of controllable luminaires, and actuator control signals may include luminaire control signals. In some such examples, rendering may include controlling brightness indicated by luminaire control signals, controlling color indicated by luminaire control signals, controlling one or more automatic window shades, or a combination thereof, at least partially based on local time information.

[0282] In some examples where local context information includes local weather information, rendering may include, at least partially, controlling the brightness indicated by lighting fixture control signals, controlling one or more automatic window shades, or both, based on local weather information.

[0283] As some examples show, local context information includes local human behavior information, so rendering can be based at least partially on local human behavior information.

[0284] In some examples, local context information may include local user input. According to some such examples, local user input may include one or more explicit mode control instructions.

[0285] According to some examples, local context information may include local character immersion information. In some such examples, immersive context mode may be enabled when local context information indicates a high level of local character immersion.

[0286] In some examples, method 1600 may include obtaining viewing position data corresponding to the user's position and location by a control system. In some such examples, rendering may be based at least in part on the viewing position data.

[0287] According to some examples, object-based sensory data may include sensory object priority metadata. In some such examples, rendering may be based at least partially on sensory object priority metadata.

[0288] In some examples, method 1600 may include receiving one or more renderer configuration parameters by a control system. According to some examples, rendering may be based at least in part on one or more renderer configuration parameters.

[0289] In some examples, object-based sensory data may include object-based light data, which includes a set of light objects. In some such examples, one or more renderer configuration parameters may correspond to an environment context mode that causes rendering of all mood light objects and only a subset of dynamic light objects, or does not cause rendering of dynamic light objects. According to some examples, local context information may include local person immersion information. In some such examples, an environment context mode may be enabled when local context information indicates a low level of local viewer immersion. In some examples, object-based light data may include light object position metadata, light object color metadata, light object size metadata, light object intensity metadata, light object shape metadata, light object diffuse metadata, light object gradient metadata, light object priority metadata, light object layer metadata, light object mood metadata, light object dynamism metadata, or a combination thereof. In some examples, light object metadata may include light object layer metadata corresponding to two or more layers. According to some examples, two or more layers may include an environment layer, a dynamic layer, a custom layer, an overlay layer, a direct light object layer, an indirect light object layer, or a combination thereof. In some such examples, rendering may involve mixing within layers, blending between layers, or both. In some examples, rendering may involve applying one or more photoactivation laws. In some examples, rendering may involve creating a slice room effect, creating an environment fill, or both, and creating a slice room effect involves generating wavefronts that propagate throughout the entire reproduction environment. In some examples, two or more layers may include an environment layer, a dynamic layer, a custom layer, an overlay layer, a direct light object layer, an indirect light object layer, or a combination thereof.

[0290] In some examples, method 1600 may include integrating actuator data with object-based sensory data. In some such examples, integrating may include finding one or more nearest controllable actuators for each sensory object. According to some examples, the actuator data may include an actuator map. In some such examples, rendering object-based sensory data to generate actuator control signals may include projecting a set of sensory objects using an actuator map. According to some examples, projecting a set of sensory objects using an actuator map may generate an actuator activation matrix as the actuator control signal. In some examples, the actuator map may include a luminaire map. In some such examples, the luminaire map may be an other-centered luminaire map based on the spatial coordinates of the playback environment, or an ego-centered luminaire map based on spatial coordinates related to the intended viewing position.

[0291] In some implementations, object-based sensory data may include sensory object priority metadata. In some such examples, Method 1600 may include determining that two or more sensory objects map to a single actuator, and rendering the object-based sensory data of the two or more sensory objects into actuator control signals for the single actuator, at least in part based on the sensory object priority metadata.

[0292] In some examples, rendering may be based at least partially on one or more renderer configuration parameters. In some examples, one or more renderer configuration parameters may include a speed priority parameter that assigns higher priority to moving sensory objects than to stationary sensory objects. In some examples, one or more renderer configuration parameters may include a change priority parameter that assigns higher priority to changing sensory objects than to static sensory objects.

[0293] As described elsewhere in this specification, in some examples, a sensory object may include a light object. In some such examples, a light object may include color information. According to some such examples, the color information may include color change information. In some such examples, one or more renderer configuration parameters may include a color change priority parameter that assigns a higher priority to color-changing light objects than to color-non-changing light objects. In some examples, a light object may include saturation information. In some such examples, one or more renderer configuration parameters may include a saturation priority parameter that assigns a higher priority to light objects with higher saturation than to light objects with lower saturation.

[0294] In some examples, rendering may be based at least partially on one or more renderer configuration modes. These may include a low actuator count mode, a content type mode, a color mixing mode, a single-sensory object versus single-actuator mode, a mode that modifies the brightness of light objects according to the distance between the light object and the luminaire, or a combination thereof. In some examples, rendering may include implementing color mixing and prioritization methods when multiple light objects are represented by a single luminaire.

[0295] In some examples, method 1600 may include, by an audio renderer, receiving audio objects and by an audio renderer, receiving loudspeaker data corresponding to loudspeakers in the environment. According to some such examples, method 1600 may also include, by an audio renderer, providing loudspeaker control signals to control loudspeakers in the environment to play audio corresponding to audio objects and synchronized with sensory effects. Synchronization may be based on time information contained in or having in sensory objects and audio objects, such as timestamps. In some examples, method 1600 may also include, by a loudspeaker in the environment, playing audio corresponding to audio objects.

[0296] In some examples, Method 1600 may include receiving video data synchronized with audio objects and object-based sensory data by a video renderer. In some such examples, Method 1600 may also include providing video control signals by the video renderer to control one or more display devices in the environment to present images that correspond to video control signals and are synchronized with audio objects and sensory effects. In some examples, Method 1600 may also include presenting images on one or more display devices in the environment. The images may correspond to video control signals.

[0297] Various features and embodiments are evident from the following enumerated example embodiments (EEE).

[0298] (EEE1) A method for controlling a set of controllable actuators, the method being The control system acquires actuator data for the set of controllable actuators, The aforementioned control system acquires local context information, The control system receives object-based sensory data, including a set of sensory objects. The control system renders the object-based sensory data and generates actuator control signals, wherein the rendering is at least partially based on the actuator data and the local context information. The control system provides the actuator control signal to one or more controllable actuators of the set of controllable actuators, A method that includes this.

[0299] (EEE2) The method according to EEE1, wherein the set of controllable actuators includes one or more lighting fixtures, one or more tactile devices, one or more airflow control devices, or a combination thereof.

[0300] (EEE3) The method according to EEE1 or 2, further comprising obtaining user location data, wherein the rendering is at least partially based on the user location data.

[0301] (EEE4) The method according to any one of EEE1 to 3, wherein the local context information includes local time information, the set of controllable actuators includes a set of controllable luminaires, the actuator control signal includes a luminaire control signal, and the rendering includes, at least in part, controlling the brightness indicated by the luminaire control signal, controlling the color indicated by the luminaire control signal, controlling one or more automatic window shades, or a combination thereof, based on the local time information.

[0302] (EEE5) The method according to EEE4, wherein the local context information includes local weather information, and the rendering includes, at least in part, controlling brightness indicated by a lighting fixture control signal, controlling one or more automatic window shades, or both, based on the local weather information.

[0303] (EEE6) The local context information includes local human behavior information, and the rendering is based at least in part on the local human behavior information, according to the method of any one of EEE1 to 5.

[0304] (EEE7) The local context information includes local user input, as described in any one of EEE1 to 6.

[0305] (EEE8) The method according to EEE7, wherein the local user input includes one or more explicit mode control instructions.

[0306] (EEE9) The local context information is the method described in any one of EEE1 to 8, which includes information about one or more viewer preferences, information about the presence or absence of one or more viewers, or a combination thereof.

[0307] (EEE10) The local context information includes ambient light information, as described in any one of EEE1 to 9.

[0308] (EEE11) The method according to any one of EEE1 to 10, wherein the local context information includes viewing environment information, local viewer location information, local device usage information, local viewer activity information, or a combination thereof.

[0309] (EEE12) The method according to any one of EEE1 to 11, wherein object-based sensory data includes object-based optical data comprising a set of optical objects, wherein the object-based optical data includes optical object position metadata, optical object color metadata, optical object size metadata, optical object intensity metadata, optical object shape metadata, optical object diffusion metadata, optical object gradient metadata, optical object priority metadata, optical object layer metadata, optical object mood metadata, optical object dynamism metadata, or a combination thereof.

[0310] (EEE13) The method according to any one of EEE1 to 12, further comprising obtaining viewing location data, wherein the viewing location data corresponds to the user's location and place, and the rendering is at least partially based on the viewing location data.

[0311] (EEE14) The object-based sensory data includes sensory object priority metadata, as described in any one of EEE1 to 11.

[0312] (EEE15) The rendering described above is the method described in any one of EEE1 to 14, which is at least partially based on one or more renderer configuration parameters.

[0313] (EEE16) The method according to EEE15, wherein the object-based sensory data includes object-based light data, which includes a set of light objects, and the one or more renderer configuration parameters correspond to an immersive context mode that causes rendering of full light intensity, full color gamut, dynamic light objects, or a combination thereof.

[0314] (EEE17) The method according to EEE16, wherein the local context information includes local viewer immersion information, and the immersive context mode is enabled when the local context information indicates a high level of local viewer immersion.

[0315] (EEE18) The method according to any one of EEE15-17, wherein the object-based sensory data includes object-based light data including a set of light objects, and the one or more renderer configuration parameters correspond to an environment context mode that causes rendering of all mood light objects and only a subset of dynamic light objects, or does not cause rendering of dynamic light objects.

[0316] (EEE19) The method according to EEE18, wherein the local context information includes local viewer immersion information, and the environment context mode is enabled when the local context information indicates a low level of local viewer immersion.

[0317] (EEE20) Equipment configured to perform the method described in any one of the EEE1-19 clauses.

[0318] (EEE21) A system configured to implement the method described in any one of the EEE1-19 items.

[0319] (EEE22) One or more non-temporary computer-readable media containing instructions for controlling one or more devices to perform the actions described in any one of the EEE1-19 clauses.

[0320] The above description illustrates various embodiments of the Disclosure, along with examples of how the aspects of the Disclosure may be implemented. The above examples and embodiments should not be considered as sole embodiments, but are provided to illustrate the flexibility and advantages of the Disclosure as defined by the following claims. Based on the above disclosure and the following claims, other configurations, embodiments, implementations and equivalents will be apparent to those skilled in the art and may be used without departing from the spirit and scope of the Disclosure as defined by the claims.

Claims

1. A method for controlling a set of one or more controllable actuators, wherein the method is The control system acquires actuator data for one or more sets of controllable actuators, The control system receives object-based sensory data, which includes a set of one or more sensory objects. The control system renders the object-based sensory data to generate one or more actuator control signals, wherein the rendering is at least partially based on the actuator data. The control system provides the control signals of one or more actuators to one or more controllable actuators of the set of controllable actuators, A method that includes this.

2. The method according to claim 1, wherein the set of one or more controllable actuators includes one or more lighting fixtures, one or more tactile devices, one or more airflow control devices, or a combination thereof.

3. The method according to claim 1 or 2, further comprising obtaining user location data, wherein the rendering is at least partially based on the user location data.

4. The method according to any one of claims 1 to 3, wherein the object-based sensory data includes sensory object metadata.

5. The method according to claim 4, wherein the sensory object metadata includes at least sensory object location metadata.

6. The method according to claim 4 or 5, wherein the sensory object metadata is sensory object size metadata.

7. The method according to any one of claims 1 to 6, further comprising integrating the actuator data with the object-based sensory data, wherein the integrating includes finding one or more nearest controllable actuators for each sensory object.

8. The method according to any one of claims 1 to 7, wherein the actuator data includes an actuator map.

9. The method according to claim 8, wherein rendering the object-based sensory data to generate one or more actuator control signals includes projecting the set of one or more sensory objects using the actuator map.

10. The method according to claim 9, wherein projecting the set of one or more sensory objects using the actuator map generates an actuator activation matrix as the control signal for one or more actuators.

11. The method according to any one of claims 8 to 10, wherein the actuator map includes a lighting fixture map, and the lighting fixture map is an other-centered lighting fixture map based on spatial coordinates of the playback environment, or a self-centered lighting fixture map based on spatial coordinates related to the intended viewing position.

12. The method according to any one of claims 1 to 11, further comprising obtaining playback environment data, wherein the rendering is at least partially based on the playback environment data.

13. The method according to any one of claims 1 to 10, wherein the object-based sensory data includes sensory object priority metadata.

14. The decision to map two or more sensory objects to a single actuator, Based at least partially on the sensory object priority metadata, the object-based sensory data of the two or more sensory objects is rendered into one or more actuator control signals for the single actuator. The method according to claim 13, further comprising:

15. The rendering according to any one of claims 1 to 14, wherein the rendering is at least partially based on one or more renderer configuration parameters.

16. The method according to claim 15, wherein one or more renderer configuration parameters include a speed priority parameter that assigns a higher priority to moving sensory objects than to stationary sensory objects.

17. The method according to claim 15 or 16, wherein one or more renderer configuration parameters include a change priority parameter that assigns a higher priority to changing sensory objects than to static sensory objects.

18. The method according to any one of claims 15 to 17, wherein the one or more sensory objects include one or more light objects, the one or more light objects include color information, and the one or more renderer configuration parameters include a color change priority parameter that assigns a higher priority to a color-changing light object than to a color-non-changing light object.

19. The method according to claim 18, wherein the one or more light objects include saturation information, and the one or more renderer configuration parameters include a saturation priority parameter that assigns a higher priority to light objects having a higher saturation than to light objects having a lower saturation.

20. The rendering according to any one of claims 1 to 19, wherein the rendering is at least partially based on one or more renderer configuration modes.

21. The method according to claim 20, wherein the one or more renderer configuration modes include a low actuator count mode, a content type mode, a color mixing mode, a single-sensory object versus single-actuator mode, a mode that modifies the brightness of a light object according to the distance between the light object and the luminaire, or a combination thereof.

22. The method according to any one of claims 1 to 21, wherein the rendering is at least partially based on data corresponding to an uncontrollable actuator that is not controllable by the actuator control signal.

23. The method according to any one of claims 1 to 22, wherein the sensory object includes one or more light objects, the one or more light objects include color information, and the rendering includes performing a color mixing and prioritization method when the multiple light objects are represented by a single luminaire.

24. The method according to any one of claims 1 to 22, wherein the sensory object includes a light object and light object metadata.

25. The method according to claim 24, wherein the optical object metadata includes optical object position data.

26. The method according to claim 24 or 25, wherein the optical object metadata includes optical object size metadata, optical object intensity metadata, optical object shape metadata, optical object diffusion metadata, optical object gradient metadata, optical object priority metadata, or a combination thereof.

27. The method according to any one of claims 24 to 26, wherein the optical object metadata includes optical object layer metadata corresponding to two or more layers.

28. The method according to claim 27, wherein the two or more layers include an environment layer, a dynamic layer, a custom layer, an overlay layer, a direct light object layer, an indirect light object layer, or a combination thereof.

29. The method according to claim 27 or 28, wherein the rendering includes mixing within layers, blending between layers, or both.

30. The method according to any one of claims 24 to 29, wherein the rendering includes applying one or more photoactivation laws.

31. The method according to any one of claims 24 to 30, wherein the rendering includes creating a slice room effect, creating an environment fill, or both, and the creation of the slice room effect includes generating a wavefront that propagates throughout the entire regeneration environment.

32. An apparatus configured to carry out the method described in any one of claims 1 to 31.

33. A system configured to carry out the method described in any one of claims 1 to 31.

34. One or more non-temporary computer-readable media storing instructions for controlling one or more devices to carry out the method according to any one of claims 1 to 31.

Images