Multi-sensory object renderer

Abstract

Some methods for controlling a set of controllable actuators may involve obtaining actuator data for the set of controllable actuators and receiving object-based sensory data including a set of sensory objects. Some methods may involve rendering object-based sensory data to generate actuator control signals, where the rendering is based at least in part on actuator data. Some methods may involve obtaining playback environment data. The rendering may be based at least in part on the playback environment data. Some methods may involve providing, by a control system, an actuator control signal to one or more controllable actuators of a set of controllable actuators. The set of controllable actuators may include one or more light fixtures, one or more haptic devices, one or more airflow control devices, or a combination thereof.

Description

Cross-references to related applications

[0001] This application claims priority to U.S. Provisional Application No. 63 / 669,233, filed July 10, 2024; U.S. Provisional Application No. 63 / 514,106, filed July 17, 2023; and U.S. Provisional Application No. 63 / 514,095, filed July 17, 2023, each of which is incorporated herein by reference in its entirety. Technical Field

[0002] This disclosure relates to providing a multi-sensory (MS) experience, and more specifically to aspects of the MS renderer. Background Technology

[0003] Unless otherwise indicated herein, the methods described in this section are not prior art to the claims of this application and are not acknowledged as prior art by virtue of their inclusion in this section.

[0004] Media content delivery typically focuses on audio and screen-based visual experiences. The delivery of multi-sensory content has been limited due to the customized nature of actuation. For example, lighting is widely used as an artistic and functional expression in concerts. However, each installation is specifically designed for a particular set of lighting fixtures. Delivering lighting design outside of that set of fixtures targeted by the system design is generally not feasible. Other systems attempting to provide a broader light experience merely extend screen visuals through algorithms, but are not specifically created. Haptic content is designed for specific haptic devices. If another device (such as a game controller, mobile phone, or even a different brand of haptic device) is used, the creative intent of the content cannot be translated to a different actuator. Summary of the Invention

[0005] At least some aspects of this disclosure can be implemented via methods such as audio processing methods. In some cases, these methods can be implemented at least in part by control systems such as those disclosed herein. Some such methods may involve obtaining actuator data of the set of one or more controllable actuators by the control system. Some methods may involve receiving object-based sensory data comprising a set of one or more sensory objects by the control system. Some methods may involve rendering the object-based sensory data by the control system to generate one or more actuator control signals. This rendering may be at least in part based on the actuator data. Some methods may involve providing the one or more actuator control signals by the control system to one or more controllable actuators in the set of controllable actuators. In some examples, the set of one or more controllable actuators may include one or more luminaires, one or more haptic devices, one or more airflow control devices, or combinations thereof.

[0006] In some examples, the object-based sensory data may include sensory object metadata. In some examples, the sensory object metadata may include sensory object location metadata. In some examples, the sensory object metadata may include sensory object size metadata.

[0007] Some methods may involve integrating the actuator data with the object-based sensory data. This integration may involve finding one or more nearest controllable actuators for each sensory object.

[0008] In some examples, the actuator data may include an actuator map. According to some examples, rendering the object-based sensory data to generate one or more actuator control signals may involve projecting a set of one or more sensory objects using the actuator map. In some examples, projecting the set of one or more sensory objects using the actuator map may generate an actuator activation matrix as or representing the one or more actuator control signals. According to some examples, the actuator map may be or may include a lighting map. For example, a lighting map may be an egocentric lighting map based on spatial coordinates of the playback environment or an egocentric lighting map based on spatial coordinates relative to the intended viewing location.

[0009] Some methods may involve obtaining playback environment data. In some such examples, the rendering may be based at least in part on this playback environment data.

[0010] In some examples, the object-based sensory data may include sensory object priority metadata. Some such methods may involve determining that two or more sensory objects are mapped to a single actuator, and rendering the object-based sensory data of the two or more sensory objects into one or more actuator control signals for the single actuator, at least in part based on the sensory object priority metadata.

[0011] According to some examples, the rendering may be at least partially based on one or more renderer configuration parameters. In some examples, these one or more renderer configuration parameters may include a velocity priority parameter that assigns a higher priority to moving sensory objects than to stationary sensory objects. According to some examples, these one or more renderer configuration parameters may include a change priority parameter that assigns a higher priority to changing sensory objects than to static sensory objects.

[0012] In some examples, the one or more sensory objects may include one or more light objects. In some examples, the one or more light objects may include color information. According to some examples, the one or more renderer configuration parameters may include a color variation priority parameter that assigns a higher priority to color-varying light objects than color-static light objects. In some examples, the one or more light objects may include color saturation information. According to some examples, the one or more renderer configuration parameters may include a color saturation priority parameter that assigns a higher priority to light objects with higher color saturation than light objects with lower color saturation.

[0013] According to some examples, the rendering may be at least partially based on one or more renderer configuration modes. According to some such examples, the one or more renderer configuration modes may include a low actuator count mode, a content type mode, a color blending mode, a single-sensory object to single-actuator mode, a mode that changes the illuminance of a light object based on the distance between the light object and the luminaire, or a combination thereof.

[0014] In some examples, the rendering may be based at least in part on data corresponding to uncontrollable actuators that cannot be controlled by these actuator control signals.

[0015] According to some examples, the sensory object may include one or more light objects. In some such examples, the one or more light objects may include color information. In some examples, the rendering may involve implementing color blending and priority sorting methods when multiple light objects are represented by a single light fixture.

[0016] In some examples, these sensory objects may include light objects and light object metadata. According to some examples, the light object metadata may include light object location data, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradation metadata, light object priority metadata, or a combination thereof.

[0017] According to some examples, the light object metadata may include light object layer metadata corresponding to two or more layers. For example, the two or more layers may include an environment layer, a dynamic layer, a custom layer, an overlay layer, a direct light object layer, an indirect light object layer, or a combination thereof.

[0018] In some examples, the rendering may involve intra-layer blending, inter-layer blending, or both. According to some examples, the rendering may involve applying one or more light activation laws. In some examples, the rendering may involve creating a sliced ​​room effect, creating ambient fill, or both. Creating the sliced ​​room effect may involve generating a wavefront that propagates throughout the playback environment.

[0019] Some methods may involve obtaining user location data. The rendering can then be based, at least in part, on this user location data.

[0020] Some or all of the operations, functions, and / or methods described herein can be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory computer-readable media. Such non-transitory media may include one or more memory devices as described herein, including but not limited to one or more random access memory (RAM) devices, read-only memory (ROM) devices, etc. Therefore, some innovative aspects of the subject matter described in this disclosure can be implemented in one or more computer-readable non-transitory media on which software is stored.

[0021] At least some aspects of this disclosure can be implemented via apparatus. For example, one or more devices may be able to perform at least partially the methods disclosed herein. In some embodiments, the apparatus may include an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or combinations thereof. The control system may be configured to perform some or all of the disclosed methods.

[0022] Details of one or more embodiments of the subject matter described in this specification are set forth in the following figures and description. Other features, aspects, and advantages will become apparent from the description, figures, and claims. Note that the relative dimensions in the following figures may not be drawn to scale. Attached Figure Description

[0023] The disclosed embodiments will now be described by way of example only with reference to the accompanying drawings.

[0024] Figure 1A This is a block diagram illustrating examples of components of an apparatus capable of implementing various aspects of this disclosure.

[0025] Figure 1B Example elements of the endpoint are shown.

[0026] Figure 2 An example of an actuator element is shown.

[0027] Figure 3 Example components of a system for creating and playing multi-sensory (MS) experiences are shown.

[0028] Figure 4 Example components of a multi-sensory (MS) renderer are shown.

[0029] Figure 5Example components of another system for creating and playing MS experiences are shown.

[0030] Figure 6A An example light map of a table lamp is shown.

[0031] Figure 6B An example of an egocentric light map is shown.

[0032] Figure 7 The components of a scene renderer are shown based on some examples.

[0033] Figure 8 An example of a sliced ​​room effect is shown.

[0034] Figure 9 An example of environment filling is shown.

[0035] Figure 10 An example of the alpha synthesis response is shown.

[0036] Figure 11 An example of the 3D environment filling result is shown.

[0037] Figure 12A , Figure 12B , Figure 12C , Figure 12D , Figure 12E and Figure 12F Additional examples of environmental filling responses are shown.

[0038] Figure 13 It shows that it can be made by Figure 5 An example of a graphical user interface (GUI) presented on a display device for a scene creation tool.

[0039] Figure 14 It shows that it can be made by Figure 5 Another example of a graphical user interface (GUI) presented on a display device for a scene creation tool.

[0040] Figure 15 It is a flowchart outlining an example of a method that can be performed by an apparatus or system such as the apparatus or system disclosed herein.

[0041] Figure 16 It is a flowchart outlining an example of a method that can be performed by an apparatus or system such as the apparatus or system disclosed herein. Detailed Implementation

[0042] This document describes techniques related to providing multi-sensory media content. In the following description, numerous examples and specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that this disclosure, as defined by the claims, may include some or all of these examples, either alone or in combination with other features described below, and may further include modifications and equivalents of the features and concepts described herein.

[0043] The following description details various methods, processes, and procedures. While specific steps may be described in a particular order, this order is primarily for convenience and clarity. A particular step may be performed more than once, may occur before or after other steps (even if these steps are described in a different order), and may occur in parallel with other steps. A second step is only necessary if the first step must be completed before the second step can begin. This will be specifically indicated when it is unclear from the context.

[0044] In this document, the terms “and,” “or,” and “and / or” are used. These terms should be understood to have inclusive meanings. For example, “A and B” can at least mean: “both A and B,” or “at least both A and B.” As another example, “A or B” can at least mean: “at least A,” “at least B,” “both A and B,” or “at least both A and B.” As yet another example, “A and / or B” can at least mean: “A and B,” or “A or B.” When XOR is intended to be used, it will be specifically indicated (e.g., “either A or B,” or “at most one of A and B”).

[0045] This document describes the various processing functions associated with structures such as blocks, components, parts, and circuits. Typically, these structures can be implemented by one or more processors controlled by one or more computer programs.

[0046] As mentioned above, media content delivery typically focuses on audio and video experiences. Due to the personalized nature of actuation, the delivery of multi-sensory (MS) content is limited.

[0047] This application describes methods for expanding the creative palette of content creators, allowing for the creation and delivery of spatial MS experiences at scale. Some of these methods involve introducing new layers of abstraction to deliver the created MS experience to different endpoints using different types of lighting fixtures or actuators. As used herein, the term "endpoint" is synonymous with "playback environment" or simply "environment," referring to an environment that includes one or more actuators that can be used to deliver the MS experience. Such endpoints can include rooms (such as the living room in a home), cars, movie theaters, nightclubs, or other locations. Some disclosed methods involve creating, delivering, and / or rendering object-based sensory data, which can include sensory objects and corresponding sensory metadata. This abstraction enables creative intent to be implemented in an object-based format without prior knowledge of specific controller actuations, thereby achieving greater flexibility and scalability across endpoints for lighting fixtures and actuators. In this document, the MS experience delivered via object-based sensory data may be referred to as a "flexibly extended MS experience."

[0048] acronym MS - Multisensory MSIE – MS Immersive Experience AR - Augmented Reality VR - Virtual Reality PC — Personal Computer Figure 1A This is a block diagram illustrating examples of components of an apparatus capable of implementing various aspects of this disclosure. As with the other figures provided herein, Figure 1A The types and quantities of elements shown are provided as examples only. Other embodiments may include more, fewer, and / or different types and quantities of elements. According to some examples, device 101 may be or may include a device configured to perform at least some of the methods disclosed herein, such as a smart audio device, laptop computer, cellular phone, tablet device, smart home hub, etc. In some such embodiments, device 101 may be or may include a server configured to perform at least some of the methods disclosed herein.

[0049] In this example, apparatus 101 includes at least an interface system 105 and a control system 110. In some embodiments, the control system 110 may be configured to perform at least partially the methods disclosed herein. In some examples, the control system 110 may be configured to obtain actuator data of a set of controllable actuators via the interface system 105. The set of controllable actuators may be, for example, specific to a particular playback environment. According to some examples, the control system 110 may be configured to obtain object-based sensory data, including a set of sensory objects, via the interface system 105. In some examples, the object-based sensory data may include object-based sensory metadata corresponding to some or all of the sensory objects. The encoded object-based sensory data may correspond to sensory effects to be provided by multiple sensory actuators in the environment, such as lighting, touch, airflow, one or more position actuators, or combinations thereof.

[0050] According to some examples, the control system 110 can be configured to implement a multi-sensory (MS) renderer. Thus, in some examples, the control system 110 can be configured to render object-based sensory data to generate actuator control signals, wherein the rendering is at least partially based on the actuator data. The MS renderer may also be referred to herein as a "sensory renderer" because in some cases, the MS renderer may render only one type of MS data, such as object-based lighting data. According to some examples, the control system 110 can be configured to send actuator control signals to one or more controllable actuators in a set of controllable actuators.

[0051] According to some examples, object-based sensory metadata may include sensory spatial metadata, which at least indicates the spatial location used to render the object-based sensory metadata within the environment, the region used to render the object-based sensory metadata within the environment, or a combination thereof. In some implementations, the object-based sensory metadata does not correspond to any specific sensory actuator in the environment. In some examples, object-based sensory metadata may include abstract sensory reproduction information, thereby allowing a sensory renderer to reproduce created sensory effects via various sensory actuator types, via various numbers of sensory actuators, and from various sensory actuator locations in the environment; these sensory effects may also be referred to herein as expected sensory effects.

[0052] In some examples, the control system 110 may be configured to obtain local context information via the interface system 105. Local context information may include local time information, local weather information, local human behavior information, local user input, information about the preferences of one or more viewers, information about the presence or absence of one or more viewers, ambient light information, viewing environment information, local viewer location information, local device usage information, local viewer activity information, or combinations thereof. In some such examples, the rendering process may be at least partially based on the local context information.

[0053] In some examples, the content bitstream may also include encoded audio objects synchronized with encoded object-based sensory metadata. The audio object may include an audio signal and corresponding audio object metadata. According to some examples, the audio object may include an audio signal and corresponding audio object metadata. The audio object metadata may at least include audio object spatial metadata indicating the audio object's spatial location for rendering the audio signal within the environment. In some examples, the MS renderer may also be configured to render the audio object.

[0054] Interface system 105 may include one or more network interfaces and / or one or more external device interfaces (such as one or more Universal Serial Bus (USB) interfaces). According to some implementations, interface system 105 may include one or more wireless interfaces. Interface system 105 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. In some examples, interface system 105 may include a control system 110 and a memory system (such as...). Figure 1A The control system 110 may include one or more interfaces between the optional memory system 115 shown in the diagram. However, in some cases, the control system 110 may include a memory system.

[0055] For example, the control system 110 may include a general-purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.

[0056] In some implementations, the control system 110 may reside in more than one device. For example, a portion of the control system 110 may reside in a device within the environment (such as a laptop computer, tablet computer, smart audio device, etc.), and another portion of the control system 110 may reside in a device outside the environment (such as a server). In other examples, a portion of the control system 110 may reside in a device within the environment, and another portion of the control system 110 may reside in one or more other devices within the environment.

[0057] Some or all of the methods described herein can be executed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices as described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. One or more non-transitory media may, for example, reside on... Figure 1A In the optional memory system 115 and / or control system 110 shown. Therefore, various innovative aspects of the subject matter described in this disclosure can be implemented in one or more non-transitory media on which software is stored. For example, the software may include instructions for controlling at least one device to process audio data. For example, the software may be provided by, for example, Figure 1A The control system 110 and other control system components perform the operation.

[0058] In some examples, device 101 may include Figure 1A The optional microphone system 120 is shown in the figure. The optional microphone system 120 may include one or more microphones. In some embodiments, the one or more microphones may be part of or associated with another device, such as a speaker in a speaker system, a smart audio device, etc.

[0059] According to some embodiments, device 101 may include Figure 1AThe optional actuator system 125 is shown in the figure. The optional actuator system 125 may include one or more loudspeakers, one or more haptic devices, one or more luminaires (also referred to herein as illuminators), one or more fans or other airflow devices, one or more display devices (including, but not limited to, one or more televisions), one or more position actuators, one or more other types of devices for providing an MS experience, or combinations thereof. As used herein, the term "luminaire" generally refers to any actuator configured to provide light. The term "luminaire" encompasses various types of light sources, including individual light sources (such as light bulbs), light source groups (such as light strips), light panels (such as light-emitting diode (LED) panels), projectors, display devices (such as television (TV) screens), etc. A "light fixture" can be movable, therefore, in this context, the word "fixture" does not mean that the fixture must be in a fixed position in space. As used herein, the term "position actuator" generally refers to a device configured to change the position or orientation of a person or object, such as a motion simulator seat. A loudspeaker may sometimes be referred to herein as a "speaker". In some embodiments, the optional actuator system 125 may include a display system comprising one or more displays, such as one or more light-emitting diode (LED) displays, one or more organic light-emitting diode (OLED) displays, etc. In some examples where device 101 includes a display system, the optional sensor system 130 may include a touch sensor system and / or gesture sensor system proximate to one or more displays of the display system. According to some such embodiments, the control system 110 may be configured to control the display system to present a graphical user interface (GUI), such as a GUI associated with implementing one of the methods disclosed herein.

[0060] In some embodiments, device 101 may include Figure 1A The optional sensor system 130 shown in the figure may include a touch sensor system, a gesture sensor system, one or more cameras, etc.

[0061] This application describes a method for rendering a flexible, expandable multi-sensory (MS) immersive experience (MSIE) and delivering it to different playback environments (which may also be referred to herein as endpoints). Such endpoints may include rooms (such as a living room in a home), cars, cinemas, nightclubs or other venues, AR / VR headsets, PCs, mobile devices, etc.

[0062] Figure 1BExample elements of an endpoint are shown. In this example, the endpoint is a living room 1001, which contains multiple actuators 008, some furniture 1010, and a person 1000 (also referred to herein as a user) who will flexibly extend the MS experience. Actuators 008 are devices capable of altering the environment 1001 in which the user 1000 is located. Actuators 008 may include one or more haptic devices, one or more lamps (also referred to herein as illuminators), one or more fans or other airflow devices, one or more display devices (including, but not limited to, one or more televisions), one or more position actuators, one or more other types of devices for providing the MS experience, or combinations thereof.

[0063] The number, arrangement, and capability of the actuators 008 in space 1001 can vary significantly between different endpoint types. For example, the number, arrangement, and capability of actuators 008 in a car typically differ from those in a living room, nightclub, etc. In many embodiments, the number, arrangement, and / or capability of actuators 008 can also vary significantly between different instances of the same type (e.g., between a small living room with 2 actuators 008 and a large living room with 16 actuators 008). This disclosure describes various methods for creating flexible, expandable MSIEs and extending them to these heterogeneous endpoints.

[0064] Figure 2 An example of actuator elements is shown. In this example, the actuator is an illuminator 1100, which includes a network module 1101, a control module 1102, and a light emitter 1103. According to this example, the light emitter 1103 includes one or more light-emitting devices, such as light-emitting diodes, configured to emit light into the environment where the illuminator 1100 resides. In this example, the network module 1101 is configured to provide network connectivity to one or more other devices in space, such as devices that send commands to control the illuminator 1100 to emit light. According to this example, the network module 1101 is... Figure 1A An example of an interface system 105. In this example, control module 1102 is configured to receive signals via network module 1101 and control optical transmitter 1103 accordingly. According to this example, control module 1102 is Figure 1A An example of a control system 110.

[0065] Other examples of actuators may include network module 1101 and control module 1102, but may include other types of actuation elements. Some such actuators may include one or more tactile devices, one or more fans or other airflow devices, one or more position actuators, one or more loudspeakers, one or more display devices, etc.

[0066] Figure 3 Example components of a system for creating and playing multisensory (MS) experiences are shown. Similar to other figures provided in this article, Figure 3 The types and quantities of elements shown are provided by way of example only. Other implementations may include more, fewer, and / or different types and quantities of elements. According to some examples, system 300 may be or may include one or more devices configured to perform at least some of the methods disclosed herein. In some examples, system 300 may include devices configured to perform at least some of the methods disclosed herein. Figure 1A One or more instances of the control system 110.

[0067] According to the examples in this disclosure, the method for creating and delivering an object-based MS Immersive Experience (MSIE) involves applying a set of techniques for creating, delivering, and rendering object-based sensory data (which may include sensory objects and corresponding sensory metadata) to actuator 008. Some examples are described in the following paragraphs.

[0068] Object-based representation: In various disclosed implementations, multi-sensory (MS) effects are represented using content that can be referred to herein as multi-sensory (MS) objects or simply "sensory objects". According to some such implementations, attributes such as layer type and priority can be assigned to, associated with, and attached to each sensory object, thereby enabling the content creator's intent to be represented in the rendered experience. Detailed examples of sensory object attributes are described below.

[0069] In this example, system 300 includes a content creation tool 000 configured to design multi-sensory (MS) immersive content and to output object-based sensory data 005, individually or in combination with corresponding audio data 011 and / or video data 012, depending on a specific implementation. The object-based sensory data 005 may include timestamp information and information indicating the type of sensory object, sensory object attributes, etc. In this example, the object-based sensory data 005 is not “channel-based” data corresponding to one or more specific sensory actuators in the playback environment, but rather generalized to multiple playback environments with multiple actuator types, multiple actuators, etc. In some examples, the object-based sensory data 005 may include object-based light data, object-based tactile data, object-based airflow data, or object-based position actuator data, object-based olfactory data, object-based smoke data, object-based data for one or more other types of sensor effects, or combinations thereof. According to some examples, the object-based sensory data 005 may include sensory objects and corresponding sensory metadata. For example, if object-based sensory data 005 includes object-based light data, then the object-based light data may include light object location metadata, light object color metadata, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradation metadata, light object priority metadata, light object layer metadata, or a combination thereof. In some examples, object-based sensory data 005 may include time data, such as timestamp information. Although in this example, content creation tool 000 is shown as providing a stream of object-based sensory data 005 to experience player 002, in alternative examples, content creation tool 000 may generate object-based sensory data 005 that is stored for later use. An example of a graphical user interface for a light-based object-based content creation tool is described below.

[0070] Examples of MS object properties The following is a non-exhaustive list of possible properties of an MS object: • Priority; •layer; • Hybrid mode; •Persistence; • Effects; and • Spatial sound image law.

[0071] Effect As used herein, the term "effect" for an MS object is a synonym for the MS object type. An "effect" is, or indicates, the sensory effect provided by the MS object. If the MS object is a light object, its effect will involve providing direct or indirect light. If the MS object is a tactile object, its effect will involve providing some type of tactile feedback. If the MS object is an airflow object, its effect will involve providing some type of airflow. Some examples involve other "effect" categories, as described in more detail below.

[0072] Persistence Some MS objects may contain persistent properties in their metadata. For example, when a movable MS object moves around the scene, it can persist for a period of time at the locations it has passed through. This period of time can be indicated by persistent metadata. In some implementations, the MS renderer is responsible for building and maintaining persistent state.

[0073] layer Based on some examples, individual MS objects can be assigned to "layers" where MS objects are grouped together based on one or more shared characteristics. For example, layers can be grouped together based on the expected effects or types of MS objects, which may include, but are not limited to, the following: - Atmosphere / Environment -Informative - Emphasis / Attention Alternatively or additionally, in some examples, layers can be used to group MS objects together based on shared properties, which may include, but are not limited to, the following: -color -strength -size -shape -Location -A zone in space Priority In some examples, MS objects can have a priority attribute, which allows the renderer to determine which(s) should have priority in an environment where MS objects compete for limited actuators. For example, if multiple light objects overlap with a single light fixture when all light objects are scheduled to be rendered, the renderer can refer to the priority of each light object to determine which(s) to render. In some examples, priority can be defined between or within layers. According to some examples, priority can be associated with specific attributes such as intensity. In some examples, priority can be defined by time: for example, the most recently rendered MS object may take precedence over previously rendered MS objects. According to some examples, priority can be used to specify MS objects or layers that should be rendered regardless of the limitations of a particular actuator system in the playback environment.

[0074] Spatial sound image law The spatial acoustic-image law can define how MS objects move in space and how MS objects affect actuators when they move between actuators.

[0075] Hybrid mode Blend patterns specify how multiple objects can be reused on a single actuator. In some examples, blend patterns may include one or more of the following: -Maximum Mode: Selects the MS object that activates the actuator the most times; - Blend mode: Blend some or all objects according to a set of rules, such as by summing activation levels, taking the average of activation levels, or blending colors according to activation level or priority level; -MaxNmix: Mix the first N MS objects according to the rule set (by activation level).

[0076] Based on some examples, instead of (or in addition to) per-object metadata, more general metadata can be defined for the entire multi-sensory content file. For example, an MS content file may include metadata such as the trimming process or the mastering environment.

[0077] Repair and control In the context of Dolby Vision™, a feature called "Trim Controls" can serve as guidance on how to adjust the default rendering algorithm for specific environments or conditions at endpoints. Trim Controls can specify ranges and / or default values ​​for various attributes, including saturation, tonal detail, gamma, etc. For example, automotive trim controls can exist that provide specific default values ​​and / or sets of rules for rendering in automotive environments, such as guidance on including only objects of a specific priority or layer. Other examples include trim controls for environments with limited, complex, or sparse multi-sensory actuators.

[0078] Mastering environment A single multi-sensory content item can include metadata about attributes of the mastering environment, such as room size, reflectivity, and ambient bias lighting levels. Specific attributes can vary depending on the desired endpoint actuator. Mastering environment information can help provide reference points for rendering within the playback environment.

[0079] MS Object Renderer: Various disclosed embodiments provide a renderer configured to render MS effects to actuators in a playback environment. According to this example, system 300 includes an MS renderer 001 configured to render object-based sensory data 005 to actuator control signals 310, at least in part based on environment and actuator data 004. In this example, MS renderer 001 is configured to output the actuator control signals 310 to MS controllers 003 configured to control actuators 008. In some examples, MS renderer 001 may be configured to receive light objects and object-based lighting metadata indicating a desired lighting environment, as well as lighting information about the local lighting environment. The lighting information is a generic type of environment and actuator data 004 and may include one or more characteristics of one or more controllable light sources in the local lighting environment. In some examples, MS renderer 001 may be configured to determine the approximate drive level of each of the one or more controllable light sources in relation to the desired lighting environment. According to some examples, the MS renderer 001 (or one of the MS controllers 003) can be configured to output drive levels to at least one of the controllable light sources. Some alternative examples may include separate renderers for each type of actuator 008, such as one renderer for a luminaire, another for a haptic device, another for an airflow device, etc. In other embodiments, a single renderer may be configured as an MS renderer and an audio renderer and / or a video renderer. In some embodiments, the MS renderer 001 may be configured to adapt to changing conditions. Some examples of implementations of the MS renderer 001 are described in more detail below.

[0080] The environment and actuator data 004 may include content referred to herein as a “room descriptor” that describes the actuator’s location (e.g., according to an x, y, z coordinate system or a spherical coordinate system). In some examples, the environment and actuator data 004 may indicate actuator orientation and / or placement attributes (e.g., orientation and north-facing, omnidirectional, occlusion information, etc.). According to some examples, the environment and actuator data 004 may indicate actuator orientation and / or placement attributes according to a 3 × 3 matrix, in which three elements (e.g., elements in the first row) represent spatial location (x, y, z), three other elements (e.g., elements in the second row) represent orientation (roll, pitch, yaw), and three other elements (e.g., elements in the third row) indicate scale or size (sx, sy, sz). In some examples, the environment and actuator data 004 may include a device descriptor describing actuator attributes associated with the MS renderer 001, such as the intensity range and color gamut of a luminaire, the airflow velocity range and(s) direction of an airflow device, etc.

[0081] In this example, system 300 includes an experience player 002 configured to receive object-based sensory data 005', audio data 011', and video data 012', and to provide the object-based sensory data 005' to an MS renderer 001, the audio data 011' to an audio renderer 006, and the video data 012' to a video renderer 007. In this example, the reference numerals for the object-based sensory data 005', audio data 011', and video data 012' received by the experience player 002 include an apostrophe (') to indicate that the data may be encoded in some cases. Similarly, the object-based sensory data 005', audio data 011', and video data 012' output by the experience player 002 do not include an apostrophe to indicate that the data may have been decoded by the experience player 002 in some cases. According to some examples, the experience player 002 can be a media player, game engine, or a component integrated into a television, DVD player, soundbar, set-top box, or service provider media device (such as Chromecast, Apple TV, or Amazon Fire TV). In some examples, the experience player 002 can be configured to receive encoded object-based sensory data 005' as well as encoded audio data 011' and / or encoded video data 012'. In some such examples, the encoded object-based sensory data 005' can be received as part of the same bitstream as the encoded audio data 011' and / or encoded video data 012'. Some examples are described in more detail below. According to some examples, the experience player 002 can be configured to extract object-based sensory data 005' from the content bitstream and provide decoded object-based sensory data 005' to the MS renderer 001, decoded audio data 011 to the audio renderer 006, and decoded video data 012 to the video renderer 007. In some examples, the timestamp information in the object-based sensory data 005 can be used, for example, by the experience player 102, MS renderer 001, audio renderer 106, video renderer 107, or all of them, to synchronize the effects associated with the object-based sensory data 005' with the audio data 111' and / or the video data 112', which may also include timestamp information.

[0082] According to this example, system 300 includes MS controllers 003 configured to communicate with various actuator types using an application programming interface (API) or one or more similar interfaces. Generally, each actuator will require a specific type of control signal to produce the desired output from the renderer. According to this example, MS controller 003 is configured to map the output from MS renderer 001 to the control signals for each actuator. For example, a Philips Hue™ bulb receives control information in a specific format to turn on a digital representation of the light with a specific saturation, brightness, and hue, as well as a desired drive level. In some alternative examples, MS renderer 001 may also be configured to implement some or all of the MS controllers 003. For example, MS renderer 001 may also be configured to implement one or more lighting-based APIs but not haptic-based APIs, and vice versa.

[0083] In some examples, the room descriptor can also describe the size and orientation of the playback environment itself to establish a relative or absolute coordinate system to which all objects are positioned. For example, in a living room, the display screen can be considered as the front, or in some cases, as the center of the front, and the floor and ceiling can be considered as vertical boundaries. In some such examples, the room descriptor can also indicate the boundaries corresponding to the left, right, front, and back walls relative to the front position. According to some examples, the room descriptor can also be provided in the form of a matrix (such as a 3 × 3 matrix). This room descriptor information is used to describe the physical dimensions of the playback environment (e.g., expressed in physical distance units such as meters). In some such examples, sensory object positioning, sensory object size, and sensory object orientation can be described in units relative to the room size, such as in the range of -1 to 1. In some cases, the room descriptor can also describe the preferred viewing position based on the matrix.

[0084] The type, number, and arrangement of actuators 008 will generally vary depending on the specific implementation. In some examples, actuators 008 may include lamps and / or light strips (also referred to herein as “illuminators”), vibration motors, airflow generators, position actuators, or combinations thereof.

[0085] Similarly, the type, quantity, and arrangement of the loudspeaker 009 and display device 010 will generally vary depending on the specific implementation. Figure 3 In the example shown, audio data 011 and video data 012 are rendered by audio renderer 006 and video renderer 007 to loudspeaker 009 and display device 010, respectively.

[0086] As described above, according to some embodiments, system 300 may include methods configured to perform at least some of the methods disclosed herein. Figure 1AThe control system 110 may be one or more instances. In some such examples, one instance of the control system 110 may implement a content creation tool 000, and another instance of the control system 110 may implement an experience player 002. In some examples, one instance of the control system 110 may implement an audio renderer 006, a video renderer 007, a multi-sensory renderer 001, or a combination thereof. According to some examples, an instance of the control system 110 configured to implement the experience player 002 may also be configured to implement an audio renderer 006, a video renderer 007, a multi-sensory renderer 001, or a combination thereof.

[0087] Figure 4 Example components of a multi-sensory (MS) renderer are shown. Similar to other figures provided in this article, Figure 4 The types and quantities of components shown are provided as examples only. Other implementations may include more, fewer, and / or different types and quantities of components. Based on this example, MS Renderer 001 is a reference. Figure 3 The described instance is MS Renderer 001. In some examples, MS Renderer 001 can be... Figure 1A The control system 110 is implemented using one or more instances of it.

[0088] Based on this example, Figure 4 Includes the following elements: •004: Environmental and actuator data, which can be referenced. Figure 3 Describe it; •005: Object-based sensory data 005, which can be referenced Figure 3 Describe it; •423 Actuator Mapping (AM) indicates the location of at least controllable actuator 008 in a specific playback environment; • Projection module 450 is configured to project an MS object based at least partially on object-based sensory data 005, using AM 423. The MS object may also be referred to as a "sensory object" in this document because in some cases, only one type of sensory object may exist in the object-based sensory data 005 (e.g., only tactile objects or only light objects). In this example, projection module 450 is configured to project the MS object based at least partially on sensory object metadata, wherein the sensory object metadata may include at least sensory object positioning metadata and sensory object size metadata; • 440 Actuator Activation Matrix (AAM), output by projection module 450 according to this example. For example, the AAM can indicate whether there are sensory objects (if any) currently contained within the playback environment that correspond to one or more corresponding actuators. For example, the AAM can indicate whether the light object positioning metadata and light object size metadata of a light object indicate that a particular luminaire is located within a volume of the playback environment corresponding to the positioning and size of that light object; • The 451 hybrid module is configured to convert AAM 440 into actuator control signals based at least in part on environmental and actuator data and renderer configuration data; • 452 Optional renderer configuration data, which may include information about one or more settings for the MS renderer 001, such as settings indicating desired motion, mode, etc. In some examples, renderer configuration data 452 may be changed automatically using a context-aware system (as described in more detail below); and •310: Actuator control signal, which can be referenced. Figure 3 The following description is provided. In some examples, the actuator control signal 310 can be sent to the individual actuators 008; while in other examples, the actuator control signal 310 can be sent to the MS controller 003, which can be configured to send appropriate control signals to various types of actuators 008.

[0089] According to some examples, AAM 440 is a matrix that describes, according to actuator map 423, how much of the sensory object itself is projected onto each actuator. In some examples, AAM 440 can be of size N. O × N A A real matrix, where N O This represents the number of sensor objects in the environment, and N A This indicates the number of controllable actuators in the environment. In this example, the hybrid module 451 is configured to generate actuator control signals 310 at least in part based on AAM 440 and environmental and actuator data 004. In some examples, the hybrid module 451 may be configured to generate actuator control signals 310 at least in part based on optional renderer configuration data 452. According to some examples, the hybrid module 451 may be configured to be at least in part based on sensory object metadata (such as... Figure 4 As shown, the sensory object metadata can be received as part of object-based sensory data 005 to generate actuator control signals 310, such as mixing and audio-visual laws associated with at least one sensory object.

[0090] In some examples, the hybrid module 451 may be configured to generate the actuator control signal 310 based at least in part on one or more of the following: 1. Threshold element of AAM 440; 2. Take the maximum value of a specific column of AAM 440—in other words, take the sensory object that activates a specific actuator the most times as the output; 3. Take any combination of the first N elements and perform at least one of the following operations: o Mixes objects into actuator channels; o Pushes the object to an adjacent channel.

[0091] In some embodiments, the projection module 450 can be configured to generate a sensory object image using the spatial coordinates (e.g., x, y, z coordinates) and the size of the sensory object to produce an I n (x,y,z), where I n Indicates the first n A sensor object image of a sensor object. Then, in some examples, the projection module 450 can be configured to calculate the first [value] of each column (actuator index) of AAM 440 by taking the inner product of the object image and the actuator mapping corresponding to the actuator. n Row (object index). Projection module 450 can generate and perform object images, actuator mappings, and dot products in any convenient spatial domain, including but not limited to polar coordinates, cylindrical coordinates, or Cartesian coordinates.

[0092] Some implementations may involve implementing what may be referred to herein as "repulsion," which can be used to avoid potentially undesirable sensory effects, such as lighting effects that might occur when a sensory object is located in one or more areas of the playback environment. In some such examples, repulsion data may be included in the environmental and actuator data 004 and AM 423, and therefore may be part of the information input to the projection module 450. In some such examples, when the MS object is projected onto AM 423 to produce AAM 440, the spatial coordinates of the MS object are augmented.

[0093] Multi-sensory rendering synchronization Object-based MS rendering involves flexibly rendering different modalities to endpoints / playback environments. Endpoints have different capabilities depending on various factors, including but not limited to the following: • Number of actuators • Modalities of these actuators (e.g., lighting and airflow control devices and tactile devices); • The types of these actuators (e.g., white smart lights versus RGB smart lights, or haptic vests versus haptic cushions) and • The positioning / layout of these actuators.

[0094] To render object-based sensory content to any endpoint, some processing of the object signals (e.g., intensity, color, pattern, etc.) is typically required. The processing of the signal path for each modality should not alter the relative phase of any feature within the object signal. For example, suppose lightning is presented in both tactile and visual modalities. The signal processing chain corresponding to the actuator control signals should not cause any type of sensory object signal (tactile or visual) to introduce a time delay sufficient to alter the perceptual synchronicity between the two modalities. The required level of synchronization may depend on various factors, such as whether the experience is interactive and what other modalities are involved. Depending on the specific context, the maximum time difference can range, for example, from approximately 10 ms to 100 ms.

[0095] Touch Rendering of object-based haptic content Object-based haptic content conveys the sensory aspects of a scene through abstract sensory representations rather than channel-based schemes. For example, instead of simply defining haptic content as a single-channel time-correlated amplitude signal played from a specific haptic actuator (such as a vibrating haptic motor) on a vest worn by the user, object-based haptic content can be defined by the sensation it is intended to convey. More specifically, in one example, there could be a haptic object representing the sensory effect of impact. Associated with this object are: • Spatial positioning of tactile objects; • Spatial direction / vector of tactile effect; • The intensity of the tactile effect; • Tactile spatial and temporal frequency data; and • Time-dependent amplitude signal.

[0096] Based on some examples, this type of haptic object can be automatically created in interactive experiences such as video games, for example, in a racing game when another car crashes into the player's car from behind. In this example, the MS renderer will determine how to render the spatial modality of this effect to a set of haptic actuators in the endpoints. In some examples, the renderer does this based on information about: • Various types of tactile devices are available, such as tactile vests and tactile gloves, tactile cushions and tactile controllers; • The location of each haptic device relative to (multiple) users (some haptic devices may not be coupled to (multiple) users, for example, a vibrator mounted on the floor or a seat); • Each haptic device offers a type of actuation, such as kinematic and vibratory haptic feedback; • Start-up and stop delay for each haptic device (in other words, the speed at which each haptic device can be turned on and off); • The dynamic response of each haptic device (how much the amplitude can change); • The time-frequency response of each haptic device (the time-frequency response that a haptic device can provide); • Spatial distribution of addressable actuators within each haptic device: For example, a haptic vest could have dozens of addressable haptic actuators distributed across the user's torso; and • The time response of any haptic sensor used to render closed-loop haptic effects (e.g., active force feedback kinetic haptic devices).

[0097] These properties of the haptic modality at the endpoints inform the renderer how best to render a particular haptic effect. Consider again the car crash effect example. In this example, the player wears a haptic vest, haptic armbands, and haptic gloves. According to this example, the haptic shockwave effect is spatially located at the point where the car hits the player. The shockwave vector is determined by the relative velocity of the player's car and the car that hits the player. The spatial and temporal spectrum of the shockwave effect is created based on the type of materials the virtual car is expected to use, as well as other virtual world properties. The renderer then renders the shockwave using a set of haptic devices at the endpoints, based on the shockwave vector and the physical location of the haptic devices relative to the user.

[0098] The signals sent to each specific actuator are preferably provided in such a way that the sensory effect is consistent across all available (potentially heterogeneous) actuators. For example, due to the lack of capability of other actuators, the renderer may not render very high frequencies only to one of the haptic actuators (e.g., a haptic armband). Otherwise, when a shockwave moves through the player's body, the haptic effect perceived by the user will decrease as the wave moves through the vest, into the armband, and finally into the gloves because the haptic vest and gloves worn by the user are not capable of rendering such high frequencies.

[0099] Some types of abstract tactile effects include: • Shockwave effect, as described above; • Barrier effects, such as haptic effects used to represent spatial constraints in virtual worlds (e.g., in video games). If an active or resistive kinetic actuator is present on the input device (e.g., force feedback on a steering wheel or joystick), this effect can be rendered by applying resistance to the user's input. If no such actuator is available at the endpoint, in some examples, vibratory haptic feedback consistent with the collision of an in-game avatar with a barrier can be rendered; • Presence, such as indicating the presence of a large object (like a train) approaching a scene. This type of haptic effect can be rendered using the low-frequency rumble of certain haptic device actuators. It can also be rendered using pressure applied by an air bladder to provide tactile spatial feedback. • User interface feedback, such as a click from a virtual button. For example, this type of haptic effect can be rendered to the nearest actuator on the user's body that performed the click, such as a haptic glove worn by the user. Alternatively or additionally, this type of haptic effect can also be rendered to a vibrator coupled to the chair the user is sitting on. This type of haptic effect can be defined, for example, using a time-dependent amplitude signal. However, this signal can be modified (modulated, frequency-shifted, etc.) to best suit the haptic device(s) that will provide the haptic effect; • Motion Sensation. These haptic effects are designed to make the user perceive some form of motion. These haptic effects can be rendered by actuators on an actual moving user (e.g., a moving platform / seat). In some examples, the actuators can provide auxiliary modalities (e.g., via video) to enhance the motion being rendered; and • Trigger Sequence. These haptic effects are primarily characterized by their time-dependent amplitude signals. This signal can be rendered across multiple actuators and, in doing so, can be amplified. This amplification can include splitting the signal across multiple actuators in time or frequency. Some examples may involve amplifying the signal itself so that the sum of the haptic actuator outputs does not match the original signal.

[0100] Spatial effects and non-spatial effects Spatial effects are spatial effects constructed in a way that conveys certain spatial information about the multi-sensory scene being rendered. For example, if the playback environment is a room, then a shockwave moving through the room will be rendered differently to each haptic device, depending on the position and size of one or more haptic objects being rendered at a particular time, based on the location of each haptic device within the room.

[0101] In some examples, non-spatial effects can be targeted at specific parts of a user's body, regardless of the user's position or orientation. One example is a haptic device providing enhanced vibrations to a user's back to indicate immediate danger. Another example is a haptic device providing strong vibrations to indicate injury to a specific area of ​​the body.

[0102] Some effects can be non-narrative effects. These effects are often associated with user interface feedback, such as the tactile sensation used to indicate that a user has completed a level or clicked a button on a menu item. Non-narrative effects can be spatial or non-spatial.

[0103] Types of tactile devices Receiving information about the different types of haptic devices available at the endpoints allows the renderer to determine which types of sensory effects and rendering strategies it can use. For example, local haptic device data instructing the user to wear both a haptic glove and a vibrating haptic vest (or at least local haptic device data indicating the presence of the haptic glove and vibrating haptic vest in the playback environment) allows the renderer to render a consistent recoil effect on both devices when the user fires a gun in the virtual world. The actual actuator control signals sent to the haptic devices may differ from cases where only a single device is available. For example, if the user is only wearing the vest, the actuator control signals used to actuate the vest may differ in terms of the actuator control signal's activation timing, maximum amplitude, frequency, decay time, or combinations thereof.

[0104] Equipment positioning Understanding the location of haptic devices at endpoints allows renderers to consistently render spatial effects. For example, knowing the location of vibrating motors in a lounge allows the renderer to send actuator control signals to each vibrating motor in the lounge in a way that conveys spatial effects (such as shock waves propagating within the room). Additionally, although the location of wearable haptic devices is implied by their type (e.g., gloves on a user's hand), the renderer can also use knowledge of the location of these wearable haptic devices to convey both spatial and non-spatial effects.

[0105] Types of actuation provided by tactile devices Haptic devices can provide a range of different actuations, and thus provide a perceived sensation. These are generally divided into two basic categories: 1. Vibrational tactile sensation, such as vibration; or 2. Kinesthetic feedback, such as resistance feedback or propulsion feedback.

[0106] Actuation of any type can be static or dynamic, with dynamic effects changing in real time based on some sensor inputs. Examples include touchscreens that use vibration haptic actuators to render textures and position sensors that measure the position of the user's (multiple) fingers(s).

[0107] Furthermore, the physical construction of such actuators varies considerably and affects many other properties of the device. An example of this is the significant differences in activation delay or time-frequency response across the following types of haptic devices: •Eccentric rotating mass; • Linear resonant actuator; • Piezoelectric actuators; and • Linear magnetic ram.

[0108] The renderer should be configured to take into account the startup latency of a specific haptic device type when rendering signals that will be actuated by a haptic device in an endpoint.

[0109] Start-up and shutdown delay of tactile devices The start-up delay of a haptic device refers to the delay between the time when the actuator control signal is sent to the device and the time when the device physically responds. The stop delay refers to the delay between the time when the actuator control signal is sent to bring the device's output to zero and the time when the device stops actuating.

[0110] Time-frequency response Time-frequency response refers to the frequency range in which a haptic device can be actuated in a steady state, with the signal amplitude as a function of time.

[0111] Spatial frequency response Spatial frequency response refers to the frequency range in which the signal amplitude is a function of the spacing between the actuators of a tactile device. Devices with closely spaced actuators have a higher spatial frequency response.

[0112] Dynamic range Dynamic range refers to the difference between the minimum and maximum amplitude of a physical actuation.

[0113] Characteristics of sensors in closed-loop haptic devices Some dynamic effects use sensors to update actuation signals based on certain observed states. The sampling frequency of time and space, as well as noise characteristics, will limit the ability to update the control loop of the actuator providing the dynamic effect.

[0114] airflow Another modality that some multi-sensory immersive experiences (MSIEs) can use is airflow. Airflow can be rendered consistently with one or more other modalities, such as audio, video, lighting effects, and / or haptics. Unlike dedicated (e.g., channel-based) setups designed solely for 4D experiences in theaters (which may include "wind effects"), some airflow effects can also be provided at other endpoints that typically include airflow, such as a car or living room. Unlike channel-based systems, airflow sensory effects can be represented as airflow objects, which can include properties such as: • Spatial positioning; • The direction of the expected airflow effect; • Intensity / airflow velocity; and / or • Air temperature.

[0115] Some examples of airflow objects can be used to represent the movement of a bird flying by. To render the airflow actuator at the endpoint, information about the following can be provided to MS Renderer 001: • Types of airflow equipment, such as fans, air conditioners, and heaters; • The position of each airflow device relative to the user's location or the user's expected location; • The capabilities of airflow devices, such as their ability to control direction, airflow, and temperature; • The control level for each actuator, such as airflow speed and temperature range; and • The response time of each actuator, for example, how long it takes to reach a selected speed.

[0116] Examples of airflow usage at different endpoints In vehicles, such as cars, object-based metadata can be used to create experiences such as the following: • During scenes in horror movies or games, mimic the feeling of "chills down your spine" by using airflow down a chair; • Simulate the movement of a bird flying by; and / or • Create a gentle breeze in the sea view.

[0117] Within the small, enclosed space of a typical vehicle, temperature changes can likely be achieved over a relatively short period compared to temperature variations in a larger environment, such as a living room. In one example, MS Renderer 001 could raise the air temperature when a player enters a "lava level" or other hot area during gameplay. Some examples could include other elements, such as confetti in vents, to celebrate an event, such as a goal scored by a user's favorite football team.

[0118] In one example, airflow in a living space or other room can be synchronized with the breathing rhythm of guided meditation. In another example, airflow can be synchronized with the intensity of exercise, increasing airflow or decreasing temperature as intensity increases. In some examples, spatial control during rendering may be relatively limited. For instance, many existing airflow actuators are optimized for heating and / or air conditioning, rather than for providing sensory actuation that offers spatial diversity.

[0119] A combination of light, airflow, and touch Car example In some examples, a user interface may be present on the steering wheel or on a touchscreen near or within the dashboard. According to some examples, the following actuators may be present in a car: 1. Individually addressable lights are spatially distributed throughout the vehicle in the following manner: o on the dashboard; o is below the footrest space; o on the door; and o is located within the central control console.

[0120] 2. Individually controllable air conditioning / heating vents are distributed throughout the vehicle as follows: o is in the front dashboard; o is below the footrest space; o is located in the center console, facing the rear seats; o is on the side pillar; o in the seat; and o. Guide windshield (for defogging).

[0121] 3. A individually controllable seat with vibrating tactile feedback; and 4. Individually controllable floor mats with vibrating tactile feedback.

[0122] In this example, the modes supported by these actuators include the following: • Lights with individually addressable LEDs throughout the car, as well as indicator lights on the dashboard and steering wheel; • Airflow via controlled air conditioning vents; •Touch, including: o Steering wheel: Tactile vibration feedback; o Dashboard touchscreen: haptic feedback and texture rendering; and o Seat: Touch, vibration, and movement.

[0123] In one example, a live music stream is rendered to four users seated in the front row. In this example, MS Renderer 001 attempts to optimize the experience for multiple viewing positions. During construction, before the artists take the stage and after the previous performances have concluded, the content includes: • Interlude music; •Low-intensity lighting; and • Represents the tactile content of a crowd colliding.

[0124] In addition to the rendered audio and video streams, light content also includes ambient light objects that move slowly within the scene. These ambient light objects can be rendered using one of the environment layer methods disclosed herein, for example, by not assigning spatial priority to any user's viewpoint. In some examples, haptic content can be spatially focused in a lower temporal frequency spectrum and can be rendered solely by vibrating haptic motors in the mat.

[0125] Based on this example, a fireworks event during a music stream corresponds to multi-sensory content including the following: • The light object that spatially corresponds to the location of the fireworks in the event; and • A tactile object that enhances the dynamic feel of fireworks through shockwave effects.

[0126] In this example, MS Renderer 001 renders both light and tactile objects spatially. For instance, light objects can be rendered in a car so that if the fireworks content is on the left side of the scene, everyone in the car will perceive the light object as coming from the left. In this example, only the lights on the left side of the car are actuated. Tactile objects can be rendered on both seats and floor mats in a way that conveys directionality to each user separately.

[0127] At the end of the concert, fireworks will appear in the audio content, and fireworks and confetti will also appear in the video content. In addition to rendering the light and tactile objects corresponding to the fireworks as described above, airflow modalities can be used to render the effect of confetti spray. For example, individually controllable airflow vents in an HVAC system can be pulsed.

[0128] Living room example In this embodiment, in addition to an audio / visual (AV) system including multiple loudspeakers and a television, the following actuators and related controls are available in the living room: • A haptic vest worn by the user (also known as the player); • A haptic vibrator installed on the seat where the player is sitting; • (Haptic) controllable smartwatch; • Smart lights distributed throughout the room; • Wireless controller; and • Addressable airflow bar (AFB), which includes an array of individually controllable fans directed at the user (similar to HVAC vents in a car's dashboard).

[0129] In this example, the user is playing a first-person shooter game that includes a scene where a destructive hurricane moves through the level. As the hurricane moves, in-game objects are thrown around, and some objects hit the player. Haptic objects rendered by MS Renderer 001 enable the delivery of shockwave effects across all haptic devices the user can perceive. The actuator control signals sent to each device can be optimized based on the impact intensity of the in-game objects, the direction(s) of the impact, and the capabilities and positioning (as previously described) of each actuator.

[0130] Before the user is hit by an in-game object, the multisensory content includes tactile objects corresponding to non-spatial rumble, one or more airflow objects corresponding to directional airflow, and one or more light objects corresponding to lightning. The MS renderer 001 renders the non-spatial rumble to the haptic devices. The actuator control signals sent to each haptic device can be rendered such that the set of actuator control signals across the entire haptic array is consistent in the timing, intensity, and frequency of the perceived rumble. In some examples, the frequency content of the actuator control signals sent to the smartwatch can be low-pass filtered to match the limited frequency capabilities of the vest near the watch. The MS renderer 001 can render one or more airflow objects as actuator control signals for AFB, such that the airflow in the room is consistent with the player's position and line of sight in the game, as well as the direction of the hurricane itself. Lightning can be rendered in all modalities as (1) a white flash produced by a light source located in a suitable position (e.g., in or on the ceiling); and (2) a pulsed rumble in the user's wearable haptic and seat vibrator.

[0131] When a user is hit by an in-game object, a directional shockwave can be rendered to the haptic device. In some examples, a corresponding airflow pulse can be rendered. According to some examples, a damage absorption effect can be rendered by a light, indicating the amount of damage the player takes from being hit by an in-game object.

[0132] In some such examples, the signal can be spatially rendered to the haptic devices, causing the perceived shockwave to move across the player's body and within the room. The MS Renderer 001 can provide this effect based on actuator positioning information that indicates the haptic devices' positioning relative to each other. In addition to actuator capability information, the MS Renderer 001 can also provide the shockwave vector and position based on the actuator positioning information. According to some examples, non-directional airflow pulses can be rendered; for example, all AFB vents can be briefly enlarged to enhance the haptic modality. In some examples, a red halo can be rendered onto the light strip around the TV simultaneously to indicate to the player that they have taken damage in the game.

[0133] Figure 5 This illustrates example components of another system used for creating and playing MS experiences. Similar to the other diagrams provided in this article, Figure 5 The types and quantities of elements shown are provided by way of example only. Other implementations may include more, fewer, and / or different types and quantities of elements. According to some examples, system 500 may be or may include one or more devices configured to perform at least some of the methods disclosed herein. In some examples, system 500 may include devices configured to perform at least some of the methods disclosed herein. Figure 1A One or more instances of the control system 110.

[0134] Based on this example, Figure 5 The system shown is Figure 3 An example of the system shown. In this example, Figure 5 The system shown is an example of "Light and Shadow," in which video, audio, and lighting effects are combined to create an MS experience.

[0135] In this example, system 500 includes a scene creation tool 100, which is a reference... Figure 3 An example of the described content creation tool 000. The lighting creation tool 100 is configured to design and output object-based lighting data 505', which, depending on a specific implementation, may be used alone or in combination with corresponding audio data 111' and / or video data 112'. The object-based lighting data 505' may include timestamp information and information indicating attributes of the lighting object, etc. In some cases, timestamp information may be used to synchronize effects associated with the object-based lighting data 505' with the audio data 111' and / or video data 112', where timestamp information may also be included.

[0136] In this example, object-based light data 505' includes light objects and corresponding light metadata. For example, object-based light data may include light object location metadata, light object color metadata, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradation metadata, light object priority metadata, light object layer metadata, or a combination thereof. Although in this example, content creation tool 100 is shown as providing a stream of object-based light data 505' to experience player 102, in alternative examples, content creation tool 100 may generate object-based light data 505' that is stored for later use. An example of a graphical user interface for a light object-based content creation tool is described below.

[0137] In this example, system 500 includes an experience player 102 configured to receive object-based light data 505', audio data 111', and video data 112', and to provide the object-based light data 505 to a lighting renderer 501, the audio data 111 to an audio renderer 106, and the video data 112 to a video renderer 107. As described elsewhere herein, the object-based light data 505, audio data 111, and video data 112 may include timestamp information that can be used to synchronize MS effects with audio and / or video effects. According to some examples, experience player 102 may be a media player, a game engine, or a component integrated into a television, DVD player, soundbar, set-top box, or service provider media device (such as Chromecast, Apple TV, or Amazon Fire TV). In some examples, experience player 002 can be configured to receive encoded object-based light data 505' and encoded audio data 111' and / or encoded video data 112', for example, as part of the same bitstream as the encoded audio data 111' and / or encoded video data 112'. According to some examples, experience player 102 can be configured to extract object-based light data 505 from the content bitstream and provide decoded object-based light data 505 to a lighting renderer 501, decoded audio data 111 to an audio renderer 106, and decoded video data 112 to a video renderer 107. In some examples, experience player 002 can be configured to allow control over configurable parameters in the lighting renderer 501, such as immersion intensity. Some examples are described below.

[0138] As described above, according to some embodiments, system 500 may include methods configured to perform at least some of the methods disclosed herein. Figure 1A One or more instances of the control system 110. In some such examples, one instance of the control system 110 may implement the scene creation tool 100, and another instance of the control system 110 may implement the experience player 002. In some examples, one instance of the control system 110 may implement an audio renderer 006, a video renderer 007, a scene renderer 501, or a combination thereof. According to some examples, an instance of the control system 110 configured to implement the experience player 002 may also be configured to implement an audio renderer 006, a video renderer 007, a scene renderer 501, or a combination thereof.

[0139] In some examples, the room descriptor of the environment and lighting data 104 can describe the size and orientation of the playback environment itself to establish a relative or absolute coordinate system to which all objects are located. The room descriptor information can indicate or describe the physical dimensions of the playback environment (e.g., expressed in physical distance units such as meters). In some such examples, sensory object positioning, sensory object size, and sensory object orientation can be described in units relative to the room size, such as in the range of -1 to 1. The room descriptor can also describe the preferred viewing position. For example, in a living room, the display screen can be considered as being in front, or in some cases as being centered in front, and the floor and ceiling can be considered as vertical boundaries. In some such examples, the room descriptor can also indicate the boundaries corresponding to the left, right, front, and back walls relative to the in front position. According to some examples, at least some of the room descriptor information can be provided in matrix form. In some such examples, the matrix can be a 3 × 3 matrix, where a row or column corresponds to one dimension of three-dimensional space.

[0140] According to this example, system 500 includes a lighting renderer 501 configured to render object-based light data 505 into luminaire control signals 515, at least in part based on environment and actuator data 104. In this example, lighting renderer 501 is configured to output the luminaire control signals 515 to lamp controllers 103 configured to control luminaires 108. Luminaires 108 may include a single controllable light source, a group of controllable light sources (such as controllable light strips), or a combination thereof. In some examples, lighting renderer 501 may be configured to manage various types of light object metadata layers, examples of which are provided herein. According to some examples, lighting renderer 501 may be configured to render the luminaire actuator signals at least in part based on the viewer's perspective. If the viewer is in a living room that includes a television (TV) screen, in some examples, lighting renderer 501 may be configured to render the actuator signals relative to the TV screen. However, in virtual reality (VR) use cases, the lighting renderer 501 can be configured to render actuator signals relative to the position and orientation of the user's head. In some examples, the lighting renderer 501 can receive input from the playback environment (such as light sensor data corresponding to ambient light, camera data corresponding to the person's position or orientation, etc.) to enhance the rendering effect.

[0141] In some examples, the lighting renderer 501 is configured to receive object-based lighting data 505, which includes light objects and object-based lighting metadata indicating the expected lighting environment, as well as environment and lighting data 104 corresponding to other features of the luminaires 108 and the local playback environment. These features may include, but are not limited to, reflective surfaces, windows, uncontrollable light sources, shading features, etc. In this example, the local playback environment includes one or more speakers 109 and one or more display devices 510.

[0142] According to some examples, the lighting renderer 501 is configured to calculate, at least in part, how to activate various controllable luminaires 108 based on object-based lighting data 505 and environment and luminaire data 104. For example, environment and luminaire data 104 may indicate the geometric location of the luminaires 108 in the environment, luminaire type information, etc. In some examples, the lighting renderer 501 may be configured to determine, at least in part, which luminaires will be activated based on position metadata and size metadata associated with each light object, for example, by determining which luminaires are located within a volume of the playback environment, corresponding to the position and size of the light object at a specific time indicated by light object timestamp information. In this example, the lighting renderer 501 is configured to send luminaire control signals 515 to the luminaire controller 103 based on environment and luminaire data 104 and object-based lighting data 505. The luminaire control signals 515 may be sent via one or more transmission mechanisms, application programming interfaces (APIs), and protocols. For example, these protocols may include Hue API, LIFX API, DMX, Wi-Fi, Zigbee, Matter, Thread, Bluetooth Mesh, or other protocols.

[0143] In some examples, the lighting renderer 501 can be configured to determine the drive level of the lighting environment expected by the authors of the object-based light data 505 for each of one or more controllable light sources. According to some examples, the lighting renderer 501 can be configured to output a drive level to at least one of the controllable light sources.

[0144] According to some examples, the lighting renderer 501 can be configured to collapse one or more portions of a lighting map based on content metadata, user input (selection mode), lighting fixture limitations and / or configurations, other factors, or combinations thereof. For example, the lighting renderer 501 can be configured to render the same control signals to two or more different lights in the playback environment. In some such examples, the two or more lights can be positioned close to each other. For example, the two or more lights can be different lights with the same actuator, such as different bulbs in the same light bulb. Instead of calculating slightly different control signals for each bulb, the lighting renderer 501 can be configured to reduce computational overhead, improve rendering speed, etc., by rendering the same control signals to two or more different but closely spaced lights.

[0145] In some examples, the lighting renderer 501 can be configured to spatially upmix the object-based light data 505. For example, if the object-based light data 505 is generated for a single plane (such as a horizontal plane), in some cases, the lighting renderer 501 can be configured to project the light objects of the object-based light data 505 onto an upper hemisphere surface (e.g., above the user's actual or intended head position) to enhance the experience.

[0146] Based on some examples, the lighting renderer 501 can be configured to apply one or more thresholds, such as one or more spatial thresholds, one or more brightness thresholds, etc., when rendering actuator control signals to the light actuators of the playback environment. In some cases, such thresholds may prevent some light objects from activating some lights.

[0147] In some implementations, the lighting renderer 501 can be configured to adapt to changing conditions. Some examples of implementations of the lighting renderer 501 are described in more detail below.

[0148] Light objects can be used for a variety of purposes, such as creating room atmosphere, providing spatial information about people or objects, enhancing special effects, creating greater interactivity and immersion, diverting the viewer's attention, emphasizing content, and so on. Content creators may use sensory object metadata types and / or attributes (typically applicable to various types of sensory objects) to express some of these purposes, such as object metadata indicating the location and size of a sensory object.

[0149] For example, the priority of sensor objects (including but not limited to light objects) can be indicated via sensor object priority metadata. In some such examples, sensor object priority metadata is considered when multiple sensor objects are simultaneously mapped to the same light fixture in the playback environment. This priority can be indicated via light priority metadata. In some examples, priority may not need to be indicated via metadata. For example, MS Renderer 001 might prioritize moving sensor objects (including but not limited to light objects) over stationary sensor objects.

[0150] A light object may trigger the activation of multiple lights, depending on its location and size, as well as the position of the light fixture within the playback environment. In some examples, when the size of a light object contains multiple lights, the renderer may apply one or more thresholds (such as one or more spatial thresholds or one or more brightness thresholds) to prevent the object from activating some of the contained lights.

[0151] Example of using lighting mapping In some implementations, a lighting map (an instance of an actuator map (AM) that includes a description of lighting in the playback environment) may be provided to the scene renderer 501. In some such examples, Figure 5 The environmental and lighting data shown may include a lighting map. According to some examples, the lighting map may be heterocentric, for example, indicating light attenuation based on absolute spatial coordinates; while in other examples, the lighting map may be egocentric, for example, projecting light onto a sphere representing the intended viewing position and orientation. In the case of a sphere, in some examples, the lighting map may be projected onto a two-dimensional (2D) surface, for example, to use a 2D image texture in processing. In any case, the lighting map should indicate the capabilities of the playback environment (e.g., a room) and the lighting settings. In some embodiments, the lighting map may not be directly related to the physical room characteristics, for example, if certain adjustments based on user preferences have been made.

[0152] In some examples, each luminaire or light in the playback environment may have a lighting map. According to some examples, the intensity of the light indicated by the lighting map may be inversely correlated with the distance to the center of the light, or may be approximately inversely correlated with the distance to the center of the light (e.g., within ±5%, ±10%, ±15%, ±20%, etc.). The intensity value of the lighting map can indicate the intensity or effect of a light object on a luminaire. For example, when a light object approaches a light bulb, the lighting renderer 501 can be configured to determine that the intensity of the light bulb will increase as the distance between the light object and the light bulb decreases. The lighting renderer 501 can be configured to determine the rate of this transition based at least in part on the light intensity indicated by the lighting map.

[0153] Figure 6AAn example light mapping for a table lamp is shown. The location of the indicator light in area 602 is mapped from the main viewer's position to polar coordinates. In some examples, the lightscape renderer 501 is configured to map lightscape objects into a common render space, which can be either other-centered or egocentric. Figure 6B An example of an egocentric light map is shown. In this example, Figure 6B This illustrates mapping a spotlight object onto a sphere at the intended viewing position and orientation. Figure 6A and Figure 6B In the diagram, darker areas are indicated by points that are relatively closer, while brighter areas are indicated by points that are relatively farther away.

[0154] In this general rendering space, in some examples, the lightscape renderer 501 can be configured to calculate the light activation index for each light using a dot product multiplication between the light object and the light map, for example, as shown below:

[0155] In the aforementioned equation, Y represents the light activation index, LM represents the illumination map, and Obj represents the mapping of the light object. The light activation index indicates the relative light intensity of the actuator control signal output by the lighting renderer 501 based on the overlap between the light object and the light diffusion from the luminaire. In some examples, the lighting renderer 501 may use the maximum or nearest distance from the light object to the luminaire, or other geometric measures, as part of determining the light intensity. In some implementations, the lighting renderer 501 does not calculate the light activation index but may instead determine it by referring to a lookup table.

[0156] The lighting renderer 501 can repeat one of the above processes to determine the light activation metrics for all light objects and all controllable lights in the playback environment. Thresholding light objects that have a minimal impact on the lights can help reduce complexity. For example, if the effect of a light object would cause the light activation to fall below a certain threshold percentage (such as below 10%, below 5%, etc.), the lighting renderer 501 might ignore the effect of that light object.

[0157] Then, the lighting renderer 501 can use the generated light activation matrix Y, along with various other attributes such as the selected panning rule (indicated by the light object metadata or renderer configuration) or the priority of the light objects, to determine which objects are rendered by which lights and how. Rendering light objects into lighting control signals can involve: • Adjust the brightness of the light source based on the distance between the light source and the light fixture; • Mix the colors of multiple light objects rendered simultaneously (multiplexed) by a single light fixture; or • Change any of the above options based on the light object priority.

[0158] This article provides some detailed examples.

[0159] Rendering parameters In addition to the information carried by the light object's metadata, the rendering of a light object can be a function of the settings or parameters of the light renderer 501 itself. These settings or parameters may include: • Speed ​​Priority - When this parameter is set, moving light objects have higher priority than stationary objects. Having a speed priority parameter set enhances the dynamism of the rendered scene; • Color Priority - Light objects with higher saturation values ​​will be given priority; • Activation threshold - The minimum light activation Y that must be achieved to activate the luminaire; • Accessibility - Certain colors can be selected instead of others to best represent the experience for colorblind users. For light-sensitive users, certain flash rates can be avoided.

[0160] Rendering configuration (mode) In some implementations, in addition to the information carried by the light object metadata, the lighting renderer 501 can be configured according to different modes. As used herein, the term "mode" differs from "parameter" because a mode can, for example, involve entirely different signal paths, while a parameter can simply parameterize those signal paths. For example, one mode might involve casting all light objects onto a lighting map before determining how / what to render to the lights, while another mode might align only the highest priority lights to the nearest lights. Modes may include: • Supports modes with low light counts. In these modes, rendering parameters and light object metadata are used to determine which subset of light objects to render and how to render them. Here, "how" refers to the trade-off between spatial, color, and temporal fidelity of the most prominent light objects in the scene; • Supports different content types (such as music and games); • Multiple light objects can be rendered using a pattern that employs color mixing by a single light fixture (or a single light); • A single light fixture (or a single lamp) can only render a single light object; • A mode that changes the brightness of a light object based on the geometric or other distance between the light object and the luminaire.

[0161] Color blending and priority sorting Some implementations of the lighting renderer 501 may implement one or more color blending methods, priority sorting methods, or combinations thereof. For example, when there are multiple light objects that simultaneously affect the same luminaire, the lighting renderer 501 may implement a color priority sorting algorithm. In the simplest embodiment, only one light object affects the luminaire. In some examples, the specific light object that will affect the luminaire can be determined by one of the following criteria: (1) the light object that is closest to the luminaire (which may be referred to as the "aligned color"); (2) the light object that has the highest percentage of influence on the luminaire; (3) the predefined highest priority light object; (4) the brightest light object. Under each of these conditions, in some cases, only the reference color may be shown (potentially with reduced brightness).

[0162] In some cases, mixed lighting can be desirable. For example, modeling the physical properties of light mixing can make rendering more realistic when mimicking the physical characteristics of multiple colored lights with a single luminaire. This is best done in a physics-based, perceptually uniform color space (such as XYZ). In some examples, light mixing can be a linear process within the color space. According to some examples, when there are multiple light objects simultaneously affecting the same luminaire, light mixing can involve mixing colors and increasing intensity.

[0163] Based on some examples, the Lightscape Renderer 501 can be configured to calculate the light blending of two light objects as follows:

[0164] In this example, blending occurs in the XYZ color space. In the aforementioned expression, This indicates the result of light mixing. Let represent the color of the first light object, and α represent a weighted constant indicating the color of the first light object. Let represent the color of the second light object, and β represent a weighted constant indicating the color of the second light object. The alpha and beta values ​​can, for example, correspond to the amount of light intensity attenuation caused by the distance from each light object to the luminaire. The alpha and beta values ​​can, for example, be extracted from the lighting map or from some other underlying geometric model.

[0165] Where physical modeling may be computationally too expensive, a faster method will allow for approximations. In one embodiment, the lighting renderer 501 can be configured to calculate the light blending of two light objects in the HSV color space as follows:

[0166] In the aforementioned expression, H represents hue and S represents saturation. Let represent the color of the first light object, α represent a weighted constant indicating the color of the first light object, and V1 represent the intensity of the first light object. Let represent the color of the second light object, β represent a weighted constant indicating the color of the second light object, and V2 represent the intensity of the second light object. In this example, hue (H) and saturation (S) are scaled by alpha and beta based on their contribution at the rendered light fixture location. Intensity (V) is then added to model the addition of the light source.

[0167] A typical lighting scene can have anywhere from a few to dozens of light objects active at any given time. Content creators will typically want to control how these light objects interact with each other within the lighting renderer 501, as these light objects are rendered onto lights in the endpoints. This can be summarized as the creator wanting to control (1) the relative priority of the light objects and (2) how the characteristics of the light objects can and cannot be mixed together. The latter deviates significantly from object-based audio rendering processes, as the lighting renderer 501 typically cannot simply mix the effects of multiple light objects to calculate the actuator control signal for a single actuator. When there are a large number of light objects (even just 3), combined with color projection from the object space domain onto the lights (which introduces some distortion), the colors produced by simply mixing the effects of multiple light objects often do not represent the creative intent or maintain the fidelity of the scene. Furthermore, there is the limitation of the finite output capacity of the lights in the endpoints.

[0168] The aforementioned issues highlight the importance of providing content creators with some control over the actions of the lighting renderer 501. Various publicly available examples provide content creators with the ability to define light object layers, priorities on those layers, and control how light objects blend on layers, and how layers or combinations thereof are blended.

[0169] Figure 7 The diagram shows components of a scene renderer based on some examples. Similar to the other diagrams provided in this article, Figure 7 The types and quantities of components shown are provided as examples only. Other implementations may include more, fewer, and / or different types and quantities of components. According to this example, the scene renderer 501 is a reference. Figure 5 An instance of the described light and shadow renderer 501. In some examples, the light and shadow renderer 501 can be... Figure 1A The control system 110 is implemented using one or more instances of it.

[0170] According to this example, the lighting renderer 501 includes the following elements: •705: Light object, which is an instance of object-based sensory data 005 disclosed herein; •004: Environmental and actuator data; •723: Lighting Map (LM), which is an instance of Actuator Map (AM) that includes a description of the lighting in the playback environment; •750: Projection module, this projection module is Figure 4 An instance of the projection module 450 is configured to project light objects using the LM723; • 740: Light Activation Matrix (LAM), which is an instance of Actuator Activation Matrix (AAM) 440 and is the output of Projection Module 750; • 751: Hybrid Module 751, which is Figure 4 An instance of the hybrid module 451 is configured to translate LAM 740 into actuator commands 741; • 741: Actuator command 741. In this example, these actuator commands are sent directly to the luminaires (which are instances of actuator 008) to control them. However, in other examples, these actuator commands can be sent to the lamp controller API 103, which will send corresponding control signals to the luminaires 008. • 752: Renderer configuration data, which may include settings such as desired motion and mode. In some publicly available context-aware examples, renderer configuration data 752 can change automatically; •702: In-layer blending module, which is configured to blend light objects in the same layer according to blending rules; • 710: Light activation vector, in this example, one light activation vector 710 per layer, containing the activation value and blended color for each luminaire; and • 703: Interlayer blending module, which is configured to blend light activation vectors 710 together to obtain rendered actuator commands 741.

[0171] Based on some examples, the optical (actuator) activation matrix (LAM) 750 is of size [missing information]. × A real matrix is ​​represented by the following equation:

[0172] In the aforementioned equation, a i,j This represents the activation value of the i-th light object on the j-th actuator. In this example, the actuator is a light. The i-th row of matrix A contains all the lights activated by the i-th light object. The j-th column of matrix A represents all the light objects that activate the j-th light. This is a useful intermediate data product because the scene renderer 501 has not yet performed any blending of the light objects, resulting in information loss. This has various potential benefits, including the possibility of optimizing the scene by analyzing A and then distorting the scene (see the example below).

[0173] In some examples, a matrix A may exist for each layer processed by the lighting renderer 501. Therefore, the lighting renderer 501 can be defined with a size of... × × A tensor (the order of dimensions is arbitrary), where, This indicates the layer number in the lighting renderer 501. In this document, A can refer to a tensor or matrix. The context will tell the reader what it is.

[0174] According to some examples, the intra-layer blending module 702 is configured to blend all light objects on a given layer. This process collapses matrix A (for that layer) into a 1 × [missing information] matrix. The light activation vector is 710v. Matrix A contains activation values, not color values. The intra-layer blending module 702 can be represented in its general form as a function that generates the light activation vector v(710), as follows:

[0175] However, in this example, the intra-layer blending process also generates a vector c for each layer in a matrix containing the color blending result of that layer, so the above equation can be modified as follows:

[0176] In the above equations, Now, based on A and (It is the length of the colors of all objects) The output is the activation vector v and the color c of the layer blending.

[0177] The interlayer blending process will Pack the vectors v and c into a size of ... × In matrices v' and c', and the output contains the color of each actuator with a length of [length missing]. A single vector o. This can usually be written as:

[0178] The example object mixing rules are given in the section on the disclosure. Examples are provided in the example layer blending rule section of this disclosure. Examples.

[0179] Example of light activation law Light activation rules range from simple geometric projections to complex responses involving pre-computed and stored in lookup tables (LUTs). An example of the latter is lighting mapping. In some examples, the LUT provides activation values ​​that can be based on object size, position, layer, velocity, and potentially other parameters. These activation values ​​may not represent the amount of light that an actual lamp or luminaire projects onto the playback environment. In some examples, the activation values ​​can be optimized to provide a sparse matrix A (which simplifies blending and potential object prioritization problems) based on this (measured or simulated) projection.

[0180] Simple geometry-based activation rules can be applied to different reference frames, such as heterocentric or egocentric reference frames. Various coordinate systems (e.g., rectangular, spherical, or cylindrical) can be used to apply these activation rules.

[0181] Here is an example of the non-self-centered rectangle activation rule: Algorithm 1

[0182]

[0183] Algorithm 1 can be implemented, for example, by a control system configured to provide... Figure 7 An example of the projection module 750. In Algorithm 1, - Represents the i-th object; - This represents the j-th lamp; -d indicates the Euclidean distance between the light and the object; - Indicates the size of objects in the scene metadata; - This represents the feather radius of an object in the scene metadata; and - Represents small numbers, such as 10^-10 used for regularization.

[0184] The implementation of the egocentric spherical mirror activation law can be substantially similar to the Euclidean implementation described above, but it first transforms the coordinates so that the user's position and orientation define the origin (the light and object positions are reference values ​​relative to this position and orientation), and the position and size are now angular values. This can be replaced with... d The distances and activation functions used include, but are not limited to, Euclidean distance, p-norm distance, cosine distance, logistic function, Gaussian activation function, and modified linear activation.

[0185] Other variations of the activation rule may include: • Distort the object's coordinates to account for multiple user perspectives (widening the optimal position); and / or • Use any frame of reference (e.g., a rotating frame of reference) to interpret TVs / screens that are not orthogonally placed within the playback environment.

[0186] Example Object Mixing Rules A simple implementation would be to simply apply all of A Sum the rows to produce v, and use these rows to pair The weighted sum is c. More specifically, for the j-th light, v and c can be calculated as follows:

[0187]

[0188] Some implementations may involve performing normalization across the columns of A before performing the summation. This normalization can be linear, for example:

[0189] Or non-linear, such as the SoftMax function:

[0190] However, these normalizations cause the column sums to become uniform, which may be undesirable when performing inter-layer blending. Some examples involve expanding the SoftMax function to bring it back to the same range so that the column sums are equal after non-linear normalization (which is now scaling, not normalization), for example, as follows:

[0191] In some cases, the scaling and / or normalization described above can be performed before the first N object elements contribute to the j-th light. The motivation for normalization and scaling is that saturation of color c[j] may not be desired. If we consider an example involving the RGB color model in the range [0, 1], the above mixing rule can produce results greater than 1. In this case, clipping can be performed to send valid RGB codewords to the luminaire. However, clipping introduces chromaticity errors, and in the worst case, the output is white when all 3 RGB channels are saturated. This is undesirable, which is a further incentive to take only the first N (e.g., N=2) when performing the above summation.

[0192] In some examples, the lighting renderer 501 can be configured for screen blending. According to some examples, screen blending can be implemented as follows: Algorithm 2

[0193] According to some examples, the scene renderer 501 can be configured to perform screen blending with or without activation weighting, as illustrated in the screen blending example above.

[0194] Example layer blending rules In some implementations, there is no fundamental difference between mixing that occurs during an intra-layer process and blending that occurs during an inter-layer process. Blending is generally used to refer to the process of combining multiple layers in image processing and computer graphics. Therefore, to avoid confusion and to help describe intra-layer and inter-layer processes, the terms "mixing" and "blending" are used herein.

[0195] For blending layers, in one example, alpha synthesis can be used, specifically A over B alpha synthesis. Recall that a blending layer is the process of generating an output vector o from matrices v' and c':

[0196] In the aforementioned equation, the magnitudes of v' and c' are... × For example, as follows:

[0197]

[0198] In the aforementioned equation v', This represents the net activation value of the light object blended into the i-th luminaire on the j-th layer. These light objects blend to produce color. .

[0199] In some examples where the lighting renderer 501 uses multiple layers, the order of the layers can imply some semantics or priority. For example, in some cases, there may be an environment layer, a spatial layer, and an overlay layer. In some such examples, the lighting renderer 501 may render these layers sequentially; in other words, it may render the environment layer first, then the spatial layer, and then the overlay layer. According to some such examples, the lighting renderer 501 may implement methods such as alpha compositing to render these layers, and may use the net activation value v as a proxy for the alpha value to blend these layers.

[0200] The following describes how to implement the function using A over B alpha synthesis when using the RGB color model. Examples.

[0201] Algorithm 3

[0202]

[0203] In Algorithm 3, each layer is sorted such that... Having the highest priority and It has the lowest priority.

[0204] Feathering distance Feather distance is the distance applied to a sensory object to create spatial smoothness. It is used to create smooth transitions as the object moves and to activate / deactivate actuators. For example, if the object's size = 0.1 and the feather distance (sometimes referred to as feather size in documentation) is 0.2, the example activation rule will produce the following activation value A:

[0205]

[0206] Abstract objects and effects The exposed light objects are not limited to discrete objects with position, size, color, and possibly some other parameters (such as feather distance). Instead, light objects can be considered as abstract light sources. Light objects can provide powerful data extensions within the Lightscape Renderer 501, which can reduce the data rate required for delivering a better experience, reduce the workload for content creators, and simplify control over complex behaviors.

[0207] Sliced ​​Room Effect Example For example, effects can be used to generate wavefronts that propagate throughout the playback environment. Instead of content creators manually defining multiple wavefronts and building tightly coupled loops, in some examples, content creators can simply choose an abstract light object effect, referred to in this example as the "slice room" light object. According to some examples, this abstract light object can be parameterized as follows: -p: Position, which is the effective phase center of the wavefront; - Size: Determines the area within the endpoints to which the effect is applied; - Density: The number of wavefronts in each dimension of the playback environment; - Frequency: the rate at which these wavefronts propagate throughout the playback environment; - Duty cycle, considered the proportion of a single cycle of activity in the wavefront; -Color: Color renderer to the active part; and - Alternate Color: The color rendered to inactive areas.

[0208] In some cases, a simple 3D oscillator can be used to implement a sliced ​​room effect to determine the spatial location of active or inactive areas. The location of a specific point in the endpoint can be represented as follows:

[0209] The phase at this location can be represented as:

[0210] In the aforementioned equation, This represents the phase component resulting from the object's position and the spatial location being evaluated. This phase component can be implemented using density D as follows:

[0211] In the aforementioned equation, This represents the Hadamard product (element-by-element multiplication).

[0212] Because the time oscillator operates at a frequency The phase generated by the operation can therefore be expressed as:

[0213] The second term of the aforementioned equation determines the phase shift caused by the spatial offset between the oscillator's phase center p and the currently evaluated spatial location q. Then, Can be compared with duty cycle A comparison is made to determine whether q is considered active, for example, as shown below:

[0214] Figure 8 An example of the sliced ​​room effect is shown. In this example, the dotted area indicates the region of active space coordinates, and the non-dotted area indicates the region of inactive space coordinates. Due to the periodicity of the sliced ​​room effect and the ability to parameterize the oscillator, in alternative examples, the non-dotted area can indicate active space coordinates, and the dotted area can indicate inactive space coordinates. According to some examples, light objects can move to various locations in the playback environment over time, thus causing modulation effects in the rendered scene. In some examples, other parameters of the light objects can be modulated to further enrich the MS experience.

[0215] Environmental filling In some implementations of the lighting renderer 501, layers can be implemented with certain defined properties or semantics. For example, on the environment layer, light objects can be rendered in a "fill" mode, in which multiple objects on the environment layer completely fill the space between them. In some examples, empty / black spaces can be defined on the environment layer by placing empty / black objects.

[0216] Figure 9An example of ambient fill is shown. According to this example, different types of fill correspond to different colors, as described below. In this example, a 2D illustration of ambient fill is shown, where: - There are 4 objects in the environment layer: o is a zero (black) object 905 of size 1 located at x,y = (50, 10) within region 910, which corresponds to the empty or black regions 910a and 910b; The red object 915 of size 0.1 at x,y = (10, 30) is within the red region 920; A green object 925 of size 0.5 located at x,y = (80,50) is within the green region 930; and A blue object 935 of size 1 located at x,y = (50, 80) is within the blue area 940.

[0217] In this example, the user is located at (50, 50).

[0218] Figure 9 It shows that when based on minimizing the following ratio r The result of assigning color to spatial coordinates (x, y):

[0219] In some examples, at each x,y location (or any coordinate in any coordinate system), the distance is calculated based on the coordinate system type, which can be rectangular or spherical, for example, where the origin can be defined such that it defines an egocentric or heterocentric system. An egocentric spherical coordinate system is used to generate... Figure 9 Therefore, the distance and magnitude in the above equations are angular values: the distance is the angular distance between the x, y unit and the object, while the magnitude is the angle the object faces relative to its position relative to the user (because it is in an egocentric frame of reference).

[0220] If the user location u is defined , A unit vector q = [x, y] can be defined from the user to any spatial location, as shown below:

[0221] Similarly, the unit vector from the user to the light object i can be defined as follows:

[0222] in, This represents the position of the light object. Then, using simple cosine distance, the distance to each object can be determined from any spatial location, as shown below:

[0223] The apparent size of each light object as observed from the user's location can be represented as follows:

[0224] in, Let represent the absolute size of the i-th light object, and This is the apparent size of the light object. The ratio of the i-th light object described earlier. r i It can be represented as follows:

[0225] The ratio is assigned to a specific spatial location associated with object j. r i The color that minimizes across all objects can be represented as follows:

[0226] The mixing can be achieved by finding the second lowest ratio. To do, for example, by not considering In the case of elements, perform argmin again. The ratio of these two ratios. I It is given by the following formula:

[0227] ratio I This can be used to determine the contribution of each of the two objects being blended. In this example, the 'fill' function is being implemented. Therefore, at any given spatial location or luminaire, the contribution of all light objects (in our case, we only specified two light objects) to that spatial location is normalized.

[0228] Some examples can implement spatial gating functions, where, whenever the ratio When large enough, only one light object contributes to the lighting. However, this introduces potentially significant discontinuities in the spatial domain on the environment layer. Therefore, in some examples, alpha composition can be used on the environment layer itself to blend the first two light objects that contribute to any particular spatial positioning. To implement some of these techniques, it is necessary to determine... and The value is selected, and an alpha composition type is chosen. Unlike blending multiple layers within a renderer, the blending order of objects is usually not predetermined. Specifying a consistent blending order across the entire spatial domain is very challenging.

[0229] Therefore, some examples involve using and The provided mixing order. This mixing order will typically be the first one after the inflection point where the ratios are equal. and the Changes occur between the objects. Therefore, using A over B alpha synthesis will produce a significant discontinuity in the step size of the output due to considering which light object is swapped with another light object.

[0230] Figure 10 An example of the alpha synthesis response is shown. Figure 10 In the diagram, the composite response is shown on the vertical axis and the position is shown on the horizontal axis. The position indicated on the horizontal axis indicates the position of the light object relative to the first luminaire located at position zero (0) and the second luminaire located at position one (1). Figure 10 The synthesized responses of Alpha_1 and Alpha_2 are shown instead of those of Alpha_A or Alpha_B. This is for clarity, as a continuous plot (e.g., the blue Alpha_1 plot) is associated with a single light object, which is reasonable in this implementation. Looking at the Alpha_1 and Alpha_2 plots of the A over B implementation, a noticeable discontinuity can be observed at the object boundary (position = 0.5), as this is the transition of light object 1 from being associated with Alpha_A to being associated with Alpha_B as position increases. Note that this discontinuity is not present in the A atop B function. Therefore, for spatial blending, A atop B alpha (e.g.) synthesis is superior to A over B synthesis.

[0231] The goal is to provide some control over the sharpness of the transition between any two light objects and to ensure that the environment layer doesn't simply become a thin layer of all the colors of the objects on it. In one example of a method used to achieve these goals, the alpha component can be determined as follows:

[0232]

[0233] In the aforementioned expression: - and Indicates the first and The ratio of objects; and -K represents a factor that allows control over the sharpness of the transition between any two colors. In some implementations, this factor K can be provided as metadata about the light object. In some examples, the factor K can be provided through user personalization of the scene.

[0234] The above formula is effectively the SoftMax activation function, therefore:

[0235] Then, A atop B compositing can be used. For example, for the j-th lamp, q is the position of that lamp, and the scene renderer 501 can find the I_1 and I_2 objects and calculate their alpha values ​​as described above. The scene renderer 501 can then determine the blend color value of that lamp on the environment layer as follows:

[0236]

[0237] Some examples can be set To ensure compatibility with the layer blending mechanism described earlier in this document.

[0238] Figure 11 An example of a 3D environment fill result is shown. In this example, the same scene described at the beginning of this section is used, and the environment fill is displayed in polar coordinates. In this example, the elevation angle is shown on the vertical axis, and the azimuth angle is shown on the horizontal axis. According to this example, the user, along with all light objects, is positioned on the z-plane. This layout is equivalent to... Figure 12D The layout shown. In Figure 11 In the example shown, area 1110 is black, area 1120 is red, areas 1130a and 1130b are green, and area 1140 is blue.

[0239] Figure 12A , Figure 12B , Figure 12C , Figure 12D , Figure 12E and Figure 12F Additional examples of ambient fill responses are shown. In these examples, region 1210 is black, region 1220 is red, region 1230 is green, and region 1240 is blue. Based on these examples, black light object 1205, red light object 1215, green light object 1225, and blue light object 1235 are respectively... Figure 9 The black light object 905, red light object 915, green light object 925, and blue light object 935 shown are in the same position. These examples illustrate a series of environment-filled responses in a two-dimensional spatial domain corresponding to the range of the target cost function (columns) for different reference frames (rows). Figure 12A , Figure 12C and Figure 12E The lines illustrate an example of an environment-filled response based on the Euclidean distance from the heterocentric center. (Including...) Figure 12B , Figure 12D and Figure 12FThe row shows an example of an environment-filled response based on the egocentric distance in the spherical domain. Figure 12D The response and Figure 11 The responses shown are comparable, although there is no alpha synthesis. Figure 12A , Figure 12C and Figure 12E The cost function applied to generate the examples shown in each column is shown above. The egocentric spherical response is suitable for layouts where luminaires are distributed along and close to walls and ceilings, while the heterocentric rectangular response is better suited for densely filled endpoints where the distance of the light object from the user can be better represented at the endpoint.

[0240] Object priority Sensory objects (including, but not limited to, light objects) can, for example, have an associated priority level based on sensory object metadata. The ability to assign sensory object priority levels provides content creators with a control mechanism. Alternatively or additionally, in some implementations, the renderer can, for example, assign implicit or explicit priorities to layers associated with sensory objects to control which sensory objects are prioritized during the sensory object rendering process. In some examples, priority can be abstractly represented as a length of... The real vector is represented by P in the following equation:

[0241] In the above equation, This represents the priority associated with the j-th object. In some examples, priorities can be "broadcast" to have the same dimension as A, such as the following:

[0242] Expressing priority with the same dimension as A allows instances of MS renderer 001 to conveniently compute a priority-weighted activation matrix, which in some embodiments may be an element-wise product of A and P'. For example, lighting renderer 501 may be configured to compute a priority-weighted lighting activation matrix 740 as an element-wise product of A and P'. In other embodiments, the priority-weighted activation matrix may be an element-wise product of thresholded A or P' or both. Alternatively, instead of thresholding, MS renderer 001 may be configured to apply a linear or non-linear transformation function to A or P' or both. Alternatively, MS renderer 001 may be configured or may not be configured to combine A and P' and may be configured to continue the rendering process by using A and P' independently. When MS renderer 001 is configured to combine A and P', MS renderer 001 produces a priority-weighted activation matrix A'. In this case, MS renderer 001 may be configured or may be configured to change and / or (These are the intra-layer blending rules and inter-layer blending rules, respectively) to suit the intentions of content creators who prioritize sensory objects. For example, a simple modification to the above intra-layer blending function would produce:

[0243] In the aforementioned expression, and within the context of light objects, using A' instead of A might result in higher-priority light objects being blended into the light fixture when using A. It might also result in light objects being blended together with so few activation functions that they are either thresholded or contribute so little to color c that they are imperceptible in the rendered actuator signal. If the lighting renderer 501 does not combine A and P' to calculate A', then the intra-layer blending function can be expressed as follows:

[0244] Some examples can use the form given above, and only objects with the highest priority level among all light objects in the i-th column whose given non-zero light activation value is blended into the j-th luminaire. In such an example, the lighting renderer 501 can be configured to blend all light objects that satisfy the following two conditions into the j-th luminaire:

[0245] Some examples may involve distorting the scene to achieve light objects with high priority that do not have a significant activation level in A for any light fixture. Lower priority light objects (which may or may not have a significant activation level in A) can be overlaid by higher priority light objects. In this case, using the distance rule described earlier in Algorithm 1 may be inappropriate because it will cause saturation whenever the distance between the light object and the light fixture is less than the size of the object. In some such cases, the lighting renderer 501 can be configured to utilize one or more non-saturating functions, such as Gaussian activation:

[0246] In some examples, for convenience, the matrix D of the unsaturated function can be constructed, as follows:

[0247] Matrix D can be used as a measure of the spatial distortion penalty from each light object to each luminaire: the "distortion" of the light object's positioning increases as the distance between them increases. If this distance is used in the intra-layer blending function, the function can be expressed as follows:

[0248] In some examples, it is possible to... Use general optimization techniques to solve for v and c.

[0249] Figure 13 It shows that it can be made by Figure 5 This is an example of a graphical user interface (GUI) displayed on a device for a scene creation tool. Like the other figures provided in this article, Figure 13 The types and quantities of components shown are provided as examples only. Other GUIs rendered by the scene creation tool may include more, fewer, and / or different types and quantities of components. Based on some examples, GUI 1300 can be customized based on... Figure 1A Commands for an instance of the control system 110 are presented on a display device; this control system is configured to implement... Figure 5 Scene creation tool 100.

[0250] In this example, the user can interact with GUI 1300 to create light objects and assign light object properties, which can be associated with the light object as metadata. According to this example, the user is selecting properties for light object 1330. In this example, GUI 1300 displays light object 1330 in three-dimensional space 1331, which represents the playback environment. Element 1334 illustrates the coordinate system of three-dimensional space 1331. Therefore, in this example, light object 1330 and three-dimensional space 1331 are viewed from the top left corner.

[0251] Users can interact with the GUI 1300 to select the position and size of the light object 1330. In some examples, users can select the position of the light object 1330 by dragging it to the desired location within the three-dimensional space 1331 (e.g., by touching a touchscreen, using a cursor, etc.). According to some examples, users can select the size of the light object 1330 by selecting the size of a circle (or other shape) displayed on the GUI 1300 to indicate the outline of the light object 130. In some such examples, users can decrease the size of the light object 1330 by pinching its outline with two fingers, increase its size by spreading their fingers, and so on.

[0252] Specifying the position and size of a light object within an abstract three-dimensional space (such as 3D space 1331 of GUI 1300) allows content creators to generalize the position and extent of corresponding light effects without prior knowledge of the specific playback environment that will provide the light effect. This is an advantage of the object-oriented approach in various disclosed implementations. For example, GUI 1300 allows content creators to specify the position and size of a light object 1330 within 3D space 1331, thereby allowing content creators to generalize the position and extent of corresponding light effects without prior knowledge of the specific size of any particular playback environment that will provide the light effect, the number, type, and position of lights within the playback environment that will provide the light effect, etc. Lights that will potentially be actuated at a specific time in response to the presence of light object 1330 will be lights within the volume of the playback environment corresponding to the position and size / extension of light object 1330.

[0253] According to this example, a user can interact with the color wheel 1335 of GUI 1300 to select the color of the current light object, and can interact with the slider 1336 to select the brightness of the current light object. These and other selectable properties of the light object 1330 are displayed in area 1332 of GUI 1300. According to this example, the properties of the light object 1330 that can be selected via GUI 1300 also include intensity, diffusivity, "feathering", whether the light object is hidden or not, saturation, priority, and layer. Light object layers and priorities will be described in more detail below. Generally, light object layers can be used to group light objects into categories such as "ambience" and "dynamic". Light object priorities can be assigned by the content creator and used by the renderer to determine, for example, which light object(s) will be rendered when two or more light objects are active simultaneously and simultaneously surround an area including the same light fixture.

[0254] Area 1340 of GUI 1300 indicates the corresponding time information for each of the multiple light objects created via the lightscape creation tool. In this example, the light objects are listed along the vertical axis on the left side of area 1340, and the time is shown along the horizontal axis. In this example, the four-second time interval is depicted by a vertical line. Here, the time information for each light object is shown as isolated or connected diamond symbols or lines along a series of horizontal rows, each of which corresponds to one of the light objects indicated on the left side of area 1340. For example, line 1333 indicates that light object 3 will begin to appear between 39 and 40 seconds and will appear continuously until almost 1 minute and 6 seconds. The diamond symbol to the right of line 1333 indicates that light object 3 will appear discontinuously over the next few seconds.

[0255] Figure 14 It shows that it can be made by Figure 5Another example of a graphical user interface (GUI) presented on a display device for a scene creation tool. Similar to the other figures provided in this article, Figure 14 The types and quantities of components shown are provided as examples only. Other GUIs rendered by the scene creation tool may include more, fewer, and / or different types and quantities of components. Based on some examples, GUI 1400 can be customized based on... Figure 1A Commands for an instance of the control system 110 are presented on a display device; this control system is configured to implement... Figure 5 Scene creation tool 100.

[0256] In this example, GUI 1400 indicates that it has been passed Figure 5 The light creation tool 100 creates light objects to control the timing of lighting fixtures in the actual playback environment. An image of the playback environment is shown in area 1405 of the GUI 1400. Various lighting fixtures 1408 and a television 1415 are shown in the playback environment of area 1405. Specific moments are indicated by the vertical line 1442 in area 1440. At this time, the vertical line 1442 intersects with the horizontal lines 1444a, 1444b, 1444c, and 1444d, indicating that the light provided in the corresponding light objects 1, 4, 5, and 7 is being played. Area 1432 indicates the characteristics of the light objects.

[0257] It can be observed that, Figure 14 At the moment depicted, the left side of the playback environment shown in area 1405 is illuminated by blue light. This corresponds at least in part to the effect of the light object 1430 shown in three-dimensional space 1431.

[0258] According to this example, video and audio data are also played in an audio environment, and the playback of the rendered light object is synchronized with the playback of the video and audio data. In this example, an image of the video being played is shown in area 1410 of GUI 1400. The video can be played, for example, by a television 1415.

[0259] In some examples, users may be able to interact with GUI 1400 to adjust light object properties, add or delete light objects, etc. For example, a user can pause playback to adjust light object properties. In some alternative examples, users may need to return to the GUI (e.g., Figure 13 The GUI 1300 allows users to adjust light object properties, add or delete light objects, etc.

[0260] Reference Figure 14 In the described example, although GUI 1400 is presented as... Figure 5The scene creation tool 100 corresponds to a display device, but renders light objects, audio, and video in a real-world environment. Therefore, in some implementations, reference is made to... Figure 14 The examples described may also involve those that can be derived from... Figure 5 The other boxes provide at least some of the "downstream" rendering and playback capabilities, including but not limited to a lightscape renderer 501, a light controller API 103 (in some cases, these light controller APIs may be implemented by the same device that implements the lightscape renderer 501), a light fixture 108, an audio renderer 106, a loudspeaker 109, a video renderer 107, and (multiple) display devices 510. In some such examples, refer to Figure 14 The description process can also involve Figure 5 Experience the features of Player 102.

[0261] In some alternative implementations, reference Figure 14 The examples described may also involve those that can be derived from... Figure 3 The other boxes provide at least some of the "downstream" rendering and playback functions, including but not limited to MS renderer 001, MS controller API 003 (in some cases, these rendering and playback functions may be implemented by the same device that implements MS renderer 001), lighting fixtures 008, audio renderer 006, loudspeakers 009, video renderer 007, and (multiple) display devices 010. In some such examples, refer to Figure 14 The process of description can also involve Figure 3 Experience the features of Player 002.

[0262] As described elsewhere in this document, certain types of metadata and / or priority indicators are specific to light objects. For example, a light object indicating a change in color can be given priority compared to a light object indicating a static color. The following is a description of some additional metadata aspects specific to light objects.

[0263] Scene metadata layer As described elsewhere in this disclosure, when creating lighting for media content, it may be useful to identify at least two different methods (or layers) to be created. These layers can be used during the process of rendering the created light objects based on available and controllable lighting fixtures in the playback environment. These layers can help capture artistic intent and allow flexibility in the constraints of the playback environment (e.g., due to the number of lighting fixtures or light occlusions) so that the main intent of (multiple) authors can still be rendered, but can be scaled or otherwise modified.

[0264] In some examples, direct lighting and indirect lighting can be assigned to different lighting metadata layers.

[0265] direct light object Light objects in the direct light object layer (also referred to as "direct light objects" in this document) are light objects that represent light directly visible to the content creator or end user. Examples of direct light objects can include lamps in a scene, the sun, the moon, headlights from an approaching car, lightning during a storm, traffic lights, etc. Direct light objects can also be used to represent light sources that are part of a scene but are typically or temporarily invisible in the associated video content, for example, because they are outside a video frame or because they have moved outside a video frame. In some examples, direct light objects can be used to amplify or enhance auditory events, such as explosions, or to visually guide the trajectory of moving objects outside a video frame. The use of direct light objects is often dynamic. For example, associated metadata such as intensity, color, saturation, and position will typically change over time within the scene of the media content.

[0266] Indirect light object Light objects in the Indirect Light Object layer (also referred to as “indirect light objects” in this document) are light objects that represent the effects of indirect lighting. For example, an indirect light object can be used to represent the effect of light radiated by a luminaire as observed when light is reflected by one or more surfaces. Some examples of using indirect light objects include changing the observed color of the walls, ceiling, or floor of an environment to a color that matches the content, such as green for a forest scene, or blue for the sky or water. Indirect light objects can also be used to set the atmosphere of a scene and environment in a similar way to that achieved through color-coded video content, but in a more immersive way. For example, science fiction films often use very specific (blue or green) video color-coded palettes to enhance the feeling of being in outer space. Flashback scenes often use reduced saturation, matte, or sepia overlays in the video content to enhance the effect of timeline changes. All of these effects can be replicated or approximated outside of video frames by adjusting the light control signals accordingly. Compared to the lighting effects corresponding to indirect light objects, the lighting effects corresponding to indirect light objects are generally more static within a scene and are generally less localized and less dynamic.

[0267] Layer abstraction Some examples involve further abstracting the direct light object layer and the indirect light object layer into layers that include aspects of both. In some such examples, these layers may include one or more environment layers, one or more dynamic layers, one or more custom layers, one or more overlay layers, or combinations thereof. In some examples, these layers may be used for, or correspond to, linear or event-based triggering within content.

[0268] (Multiple) environmental layers Similar to indirect lighting, an environment layer can be used to set the atmosphere and tone of a space by playing thin layers of color on surfaces within the environment. An environment layer can serve as a base layer upon which to build a lighting scene. In some examples, an environment layer can be represented by a light object covering a relatively large area. In other examples, an environment layer can be represented by a light object covering a relatively small area (e.g., using one or more images). Depending on the example, an environment layer can be divided into zones. In some such examples, a particular lighting effect always occupies a specific spatial area. For example, the walls, ceiling, and floor in the creation or playback environment can each be considered a separate environment layer zone.

[0269] (Multiple) dynamic layers In some implementations, a dynamic layer can be used to represent the spatial and temporal changes of MS objects (such as light objects). Within the dynamic layer, individual MS objects can also have priorities, such that, for example, one light object may take precedence over another when rendered by a light fixture. Within the dynamic layer, in some examples, individual MS objects can be linked to other objects, such as linking to audio objects or 3D world MS objects (from spatial audio).

[0270] (Multiple) Customized Layers In some examples, custom layers can be used to design light sequences that can be freely assigned to lights for functional purposes. These sequences may not be spatial in nature, but rather provide further information to the user. For example, in a game, light strips could be assigned to display the player's remaining health.

[0271] (Multiple) overlays Based on some examples, overlays can be used to render persistent light with sequential priority. Overlays can also be used, for example, to create "watermarks" on all other elements in a lighting scene.

[0272] Creation and distribution of light layer data Create direct and indirect light objects In some examples, direct light objects can be created by determining or setting the light source position, intensity, hue, saturation, and spatial extent of one or more light objects as a function of time. In some such examples, the creation process can create corresponding metadata, which can be distributed along with the direct light object and the audio and / or video content used for content rendering. Ideally, the direct light object is rendered as a direct light source.

[0273] In some examples, indirect lighting effects can be created as a dedicated group or category within the scene's metadata content, allowing for a greater focus on overall color and environment rather than dynamic effects. Indirect lighting effects can also be defined by intensity, hue, saturation, or a combination thereof as a function of time, but are typically associated with a large area of ​​the scene's rendering environment. Indirect lighting effects are ideally (but not necessarily) rendered to indirect light sources (if available).

[0274] Rendering of light layer data Layer attributes and their metadata can be obtained using a lighting renderer (such as...) Figure 5 A lighting renderer 101 is used for rendering. In some examples, the lighting renderer can be configured to send lighting control signals 515 to the lighting fixtures in the playback environment. In some examples, the lighting renderer 101 can be configured to output the lighting control signals 515 to lighting controllers 103, which are configured to control lighting fixtures 108. According to some examples, the lighting renderer uses environment and lighting fixture data 104 to determine the capabilities and spatial location of each lighting fixture. In some examples, layer priority can be a determining factor in the final content rendered to the lighting fixtures.

[0275] Rendering direct and indirect light objects In some implementations, the environmental and lighting data 104 received by the lighting renderer includes data about whether the lighting fixture is a direct or indirect light source from the viewing position. In some examples, if no indirect lighting fixture is available, the indirect light data may instead be sent to the direct lighting fixture, potentially with reduced brightness.

[0276] Direct light objects are preferably rendered to visible light fixtures, such as ceiling downlights, wall-mounted lights, table lamps, etc. Indirect light metadata is ideally targeted at light fixtures that are not directly visible, such as LED light strips illuminating walls, ceilings, shelves, and furniture, and spotlights illuminating walls or ceilings. If no such indirect light is available, indirect light metadata can be used instead to control direct light. In some such examples, the lighting renderer can overlay direct and indirect light object metadata when rendering to light fixtures that function as both indirect and direct light sources.

[0277] Figure 15This is a flowchart outlining an example of a method that can be performed by a device or system such as the apparatus or system disclosed herein. As with other methods described herein, the blocks of method 1500 need not be performed in the indicated order. In some embodiments, one or more blocks of method 1500 may be performed simultaneously. Furthermore, some embodiments of method 1500 may include more or fewer blocks than those shown and / or described. The blocks of method 1500 may be performed by one or more devices, which may be (or may include) a control system (such as those described above). Figure 1A One or more instances of the control system 110 shown and described. For example, at least some aspects of method 1500 can be implemented by a system configured to perform... Figure 3 The multi-sensory renderer's control system 110 is executed in an instance. Some aspects of method 1500 can be configured to implement... Figure 5 The instance execution of the control system 110 of the scene renderer 501.

[0278] In this example, box 1505 relates to obtaining actuator data for a set of controllable actuators by a control system. In some examples, the set of controllable actuators may include one or more luminaires, one or more haptic devices, one or more airflow control devices, or combinations thereof.

[0279] According to this example, box 1510 relates to receiving object-based sensory data, including a set of sensory objects, by a control system. In some cases, the object-based sensory data may include sensory object metadata, which may also be referred to herein as sensory object metadata.

[0280] In some examples, sensory object metadata may include sensory spatial metadata, such as sensory object location metadata indicating the spatial location used to render object-based sensory data within the environment, sensory object size metadata indicating the region or volume used to render object-based sensory data within the environment, or both. In some cases, object-based sensory data may include one or more other types of sensory object metadata.

[0281] Based on some examples, object-based sensory data does not correspond to specific sensory actuators in the environment. For example, see references... Figure 13 and Figure 14As described, a sensory object based on object-based sensory data can correspond to a portion of a three-dimensional region representing a playback environment. When creating a sensory object, the actual playback environment in which the sensory object will be rendered does not need to be known, and typically is not. Therefore, object-based sensory data includes abstract sensory reproduction information—in these examples, the sensory object and its corresponding sensory metadata—allowing the sensory renderer to reproduce the created sensory effects via various sensory actuator types, via various numbers of sensory actuators, and from various sensory actuator locations in the environment.

[0282] In this example, block 1515 relates to rendering the object-based sensory data by a control system to generate actuator control signals. According to this example, the rendering is at least partially based on the actuator data. In some examples, method 1500 may also involve obtaining user location data. According to some such examples, the rendering may be at least partially based on the user location data. In some examples, the audio object metadata may at least include audio object spatial metadata indicating the spatial location of the audio object used to render audio signals within the environment. In some examples, the rendering may be at least partially based on data corresponding to uncontrollable actuators that cannot be controlled by these actuator control signals.

[0283] According to this example, block 1520 relates to providing actuator control signals from a control system to one or more controllable actuators in a set of controllable actuators. According to some such examples, method 1500 may also involve receiving object-based sensory data and environment descriptor data corresponding to the playback environment from a sensory renderer. In some such examples, method 1500 may also involve receiving actuator descriptor data corresponding to characteristics of sensory actuators in the environment from a sensory renderer. In some examples, the environment descriptor data and actuator descriptor data may be or may be included in a reference... Figure 3 The environment and actuator data are described in data 004. In some examples, Figure 3 The MS controller API 003 shown can be implemented via MS renderer 001, and actuator-specific signals can be provided to actuator 008 by MS renderer 001. In some alternative examples, MS renderer 001 can provide actuator control signal 310 to MS controller API 003, and MS controller API 003 can provide actuator-specific control signals to actuator 008. In some examples, method 1500 may also involve providing sensory effects by sensory actuators in the environment.

[0284] In some examples, method 1500 may involve integrating the actuator data with the object-based sensory data. In some such examples, the integration may involve finding one or more nearest controllable actuators for each sensory object. According to some examples, the actuator data may include an actuator map. In some such examples, rendering the object-based sensory data to generate actuator control signals may involve projecting the set of sensory objects using the actuator map. According to some examples, projecting the set of sensory objects using the actuator map may generate actuator activation matrices as these actuator control signals. In some examples, the actuator map may include a lighting map. In some such examples, the lighting map may be an egocentric lighting map based on spatial coordinates of the playback environment or an egocentric lighting map based on spatial coordinates relative to the intended viewing location.

[0285] In some implementations, the object-based sensory data may include sensory object priority metadata. In some such examples, method 1500 may involve determining that two or more sensory objects are mapped to a single actuator, and rendering the object-based sensory data of the two or more sensory objects as actuator control signals for the single actuator, at least in part based on the sensory object priority metadata.

[0286] According to some examples, the rendering may be at least partially based on one or more renderer configuration parameters. In some examples, these one or more renderer configuration parameters may include a velocity priority parameter that assigns a higher priority to moving sensory objects than to stationary sensory objects. According to some examples, these one or more renderer configuration parameters may include a change priority parameter that assigns a higher priority to changing sensory objects than to static sensory objects.

[0287] In some examples, these sensory objects may include light objects. In some such examples, these light objects may include color information. According to some such examples, the color information may include color variation information. In some such examples, the one or more renderer configuration parameters may include a color variation priority parameter that assigns a higher priority to color-varying light objects than to color-static light objects. In some examples, these light objects may include color saturation information. In some such examples, the one or more renderer configuration parameters may include a color saturation priority parameter that assigns a higher priority to light objects with higher color saturation than to light objects with lower color saturation.

[0288] According to some examples, the rendering may be at least partially based on one or more renderer configuration modes. These one or more renderer configuration modes may include a low actuator count mode, a content type mode, a color blending mode, a single-sensory object to single-actuator mode, a mode that changes the illuminance of a light object based on the distance between the light object and the luminaire, or a combination thereof. In some examples, the rendering may involve implementing color blending and priority ordering methods when multiple light objects are represented by a single luminaire.

[0289] In some examples where the sensory object includes a light object, the light object may have associated light object metadata. This light object metadata may include light object position data, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradation metadata, light object priority metadata, or a combination thereof. In some examples, the light object metadata may include light object layer metadata corresponding to two or more layers. According to some examples, the two or more layers may include an environment layer, a dynamic layer, a custom layer, an overlay layer, a direct light object layer, an indirect light object layer, or a combination thereof. In some such examples, the rendering may involve intra-layer blending, inter-layer blending, or both. According to some examples, the rendering may involve applying one or more light activation laws. In some examples, the rendering may involve creating a sliced ​​room effect, creating ambient fill, or both, wherein creating the sliced ​​room effect involves generating a wavefront that propagates throughout the playback environment.

[0290] In some examples, method 1500 may involve receiving an audio object by an audio renderer and receiving loudspeaker data corresponding to a loudspeaker in the environment by the audio renderer. According to some such examples, method 1500 may also involve the audio renderer providing loudspeaker control signals for controlling a loudspeaker in the environment to play audio corresponding to the audio object and synchronized with a sensory effect. Synchronization may be based, for example, on time information included in or along with the sensory object and audio object, such as a timestamp. In some examples, method 1500 may also involve a loudspeaker in the environment playing audio corresponding to the audio object.

[0291] According to some examples, method 1500 may involve receiving video data synchronized with audio objects and object-based sensory data by a video renderer. According to some such examples, method 1500 may also involve providing video control signals by the video renderer for controlling one or more display devices in the environment to render images corresponding to the video control signals and synchronized with audio objects and sensory effects. In some examples, method 1500 may also involve rendering images on one or more display devices in the environment. The images may correspond to the video control signals.

[0292] Context-based rendering example This section describes an implementation of a multi-sensory renderer 001 configured to render at least in part based on local context information. This type of rendering may be referred to herein as "context-based rendering" or "context-aware rendering." "Context" is or includes local context information about the local playback environment. For example, one aspect of local context information may be the time in the region of the local playback environment, the weather in the region of the local playback environment, etc. Alternatively or additionally, context may be or may include information about one or more people in the local playback environment, such as their apparent level of engagement with the playback content.

[0293] In some examples, information that allows the multi-sensory renderer 001 to provide context-based rendering may be explicitly provided. In some examples, this explicit information may be or may include user input for managing one or more aspects of the rendering process, such as information about the overall level of immersion and / or interactivity. According to some examples, this explicit information may be or may include input from a device or system capable of accessing one or more aspects of local context information, or input from a device or system configured to learn local context information by analyzing sensor data, user behavior patterns, etc.

[0294] Managing the immersion and / or interactivity of a sensory experience can be achieved by altering how the multisensory renderer 001 manages dimensions of time, frequency, intensity, input, spatial effects (e.g., color or vibration), or combinations thereof. In some examples, the multisensory renderer 001 can be configured to automatically manage the immersion and / or interactivity of a sensory experience, for example, by applying a low-pass filter to one or more of these dimensions. Alternatively or additionally, the multisensory renderer 001 can be configured to manage the immersion and / or interactivity of a sensory experience based on sensory object metadata received along with content including object-based sensory data that indicates artistic intent.

[0295] In some examples, when users prefer a less immersive experience, one or more layers of sensory content that are relatively more dynamic or spatial can be excluded, and only the environmental layer can be used. According to some examples, the intensity of one or more layers of sensory content that are relatively more dynamic or spatial can be reduced. These actions can be taken for any combination of audio, visual, and sensory experiences.

[0296] Therefore, the context-aware MS renderer 001 can be configured to render object-based sensory data 005 to the actuator control signal 310 based at least in part on local context information, which may include one or more of the following: • Local time. For example, if the current location is late at night, MS Renderer 001 may avoid rendering extremely bright light. In some examples, MS Renderer 001 may avoid light content with a strong blue component during the last hour or so before the viewer goes to sleep, as this can interfere with sleep. If the current location is daytime, MS Renderer 001 or another device can be configured to control automatic blinds to reduce ambient light from outside, depending on the content and experience; • Local weather information determined by internet weather forecasts, real-time internet weather observations from nearby weather stations, or local weather information provided by on-site weather stations connected via LAN, WLAN, or Bluetooth. For example, if the outdoor environment is very sunny, high-intensity lighting may be needed to overcome the light seeping into the viewing room from the windows. If the weather is very overcast, the room may be darker, and therefore lower light intensity may be required. • Observations of human behavior, in some examples, could include observations over days, weeks, or months. For instance, based on historical information from a home security system, it might be determined that there is likely only one person in the living room at present, and that person is probably watching a movie, TV show, etc. A context-aware MS renderer 001 could determine that this is the time to use sensory effects, including those from the living room lighting system, to enhance the TV viewing experience, because there is likely no one else in the house trying to do anything else in the living room at the same time; • Explicit input regarding mode switching, such as explicit instructions—e.g., instructions received via user input from people in the playback environment—should not enhance the TV viewing experience, as another member of the household is trying to do their homework in the same room. • Information indicating the presence of a specific person in the playback environment. For example, a person may have indicated a preference for one or more types of sensory experiences. In some examples, this person might be a light-sensitive viewer or a colorblind viewer whose lighting experience should be dimmed or otherwise personalized. The presence of a specific viewer can be determined via Bluetooth or Wi-Fi beacons from a phone or smartwatch, speaker identification using a microphone, facial recognition, etc. • Information indicating ambient light from internal (within the playback environment), external (e.g., outdoor) sources, or both (such as light sensor information). For example, if a home has smart lights in the living room, but the kitchen is adjacent to the living room and has a separate lighting setup, the kitchen light might interfere with the light from the controllable lights in the living room. In some such examples, the context-aware MS Renderer 001 can be configured to adjust the rendering of the controllable lights in the living room based on the light from the kitchen light. For example, MS Renderer 001 could be configured to make lights near the kitchen relatively brighter than those further away from the kitchen to compensate for the kitchen light. This compensation technique might be particularly relevant if MS Renderer 001 is mixing colors. For example, if the kitchen light is somewhat orange-toned but white light is desired, MS Renderer 001 could make the side lights slightly greener so that the color appears white in the user's peripheral vision. • Information indicating the current context of one or more people in the playback environment. For example, if the current context information indicates that the user is driving a car, the MS renderer 001 can make it so that only lower-level / less immersive sensory objects / types are displayed to prevent driver distraction. In contrast, if the same person is sitting in the back seat while the vehicle is stationary, the MS renderer 001 can make it so that all metadata layers are available. In yet another example, if the context-aware MS renderer 001 receives information (such as sensor information or user input) indicating that one or more people are watching TV and no one else nearby is trying to do anything constructive (such as housework or homework), the context-aware MS renderer 001 can determine that this is a time for relatively more immersive sensory content playback (corresponding to content provided via TV), while if it appears that no one is watching TV, the context-aware MS renderer 001 can determine that this is a time for relatively more environmental, or fully environmental, sensory content playback (corresponding to content provided via TV); • Environmental context information, including but not limited to information obtained from environmental and actuator data 004. Environmental context information may include information about which devices are currently being used. In some examples, the context-aware MS renderer 001 may be configured to provide a specified egocentric view when using a display screen (with visual effects), and a more heterocentric or environmental view if only audio and lighting are used, or lighting alone. If the local playback environment is a vehicle environment, the local environmental context information may include information about whether the vehicle is parked or moving. For example, if the context-aware MS renderer 001 receives information indicating that the vehicle is parked, the context-aware MS renderer 001 may determine that this is the time for playback of relatively more immersive sensory content corresponding to content provided via the vehicle's audio system, video system, or both; and • Information indicating the location of one or more people within the playback environment, such as information indicating the only person in a moving vehicle is in the driver's seat, information indicating the only person in a parked vehicle is in the back seat, information indicating one or more people are on a sofa facing the TV, etc. Alternatively or additionally, some examples may involve optimizing the rendering of light (if created for an egocentric view). This can be particularly relevant to the effects in front of and behind the user. For example, lighting effects intended to provide behind the user might rely on reflections or might be unlikely to be rendered, depending on the availability of the lighting fixtures.

[0297] As described elsewhere in this document, sensory object metadata may also contain information to assist the renderer in conveying artistic intent for various contexts or user-selectable levels of immersion. For example, sensory object metadata may include an “ambience” object type to indicate that the object’s role is to set an environmental layer. In some examples, sensory object metadata may include a “dynamic” object type, which can be used to signal to the renderer that the object’s role is to bring about change and movement. Context-aware MS renderer 001 can use this information to render different types of sensory experiences in different contexts. For example, in an “immersive” context where one or more people want a fully immersive sensory experience (e.g., in a living room at home), the full color gamut of ambient and dynamic sensory objects can be used to render actuator signals to achieve the sensory experience. However, if the user or system selects an “environment” context, the context-aware MS renderer 001 can use all ambient objects but only a subset of dynamic objects or none at all. According to some implementations, this immersion control can be continuous, ranging from “off immersion” to “fully immersive,” for example, in a range from zero to ten, from zero to one hundred, etc. In some examples, different contexts can be sensed, classified, and programmed to correspond to various immersion levels. Immersion levels can vary by the sensory objects used, the intensity or amplitude of the sensory actuators played, the number of actuators used, and so on. For example, immersion levels can vary by the light objects used, the brightness of the light, the number of lights used, and so on. The table below provides examples of immersion levels, contexts, and sensory object usage in the context-aware MS Renderer 001.

[0298] Examples of immersion management

[0299] Additional scene examples Figure 16This is a flowchart outlining an example of a method that can be performed by a device or system such as the apparatus or system disclosed herein. As with other methods described herein, the blocks of method 1600 need not be performed in the indicated order. In some embodiments, one or more blocks of method 1600 may be performed simultaneously. Furthermore, some embodiments of method 1600 may include more or fewer blocks than those shown and / or described. The blocks of method 1600 may be performed by one or more devices, which may be (or may include) a control system (such as those described above). Figure 1A One or more instances of the control system 110 shown and described. For example, at least some aspects of method 1600 can be implemented by a system configured to perform... Figure 3 The multi-sensory renderer's control system 110 is executed in an instance. Some aspects of method 1600 can be configured to implement... Figure 5 The instance execution of the control system 110 of the scene renderer 501.

[0300] In this example, box 1605 relates to obtaining actuator data for a set of controllable actuators by a control system. In some examples, the set of controllable actuators may include one or more luminaires, one or more haptic devices, one or more airflow control devices, or combinations thereof.

[0301] exist Figure 16 In the example shown, box 1607 relates to obtaining local context information by the control system. Local context information can be, for example, one or more types of local context information disclosed herein. In some examples, local context information can be or may include local time information, local weather information, local human behavior information, local user input, information about the preferences of one or more viewers, information about the presence or absence of one or more viewers, local ambient light information, local viewing environment information, local viewer location information, local device usage information, local viewer activity information, or combinations thereof.

[0302] According to this example, box 1610 relates to receiving object-based sensory data, including a set of sensory objects, by a control system. In some cases, the object-based sensory data may include sensory object metadata, which may also be referred to herein as sensory object metadata.

[0303] In some examples, sensory object metadata may include sensory spatial metadata, such as sensory object location metadata indicating the spatial location used to render object-based sensory data within the environment, sensory object size metadata indicating the region or volume used to render object-based sensory data within the environment, or both. In some cases, object-based sensory data may include one or more other types of sensory object metadata.

[0304] Based on some examples, object-based sensory data does not correspond to specific sensory actuators in the environment. For example, see references... Figure 13 and Figure 14 As described, a sensory object based on object-based sensory data can correspond to a portion of a three-dimensional region representing a playback environment. When creating a sensory object, the actual playback environment in which the sensory object will be rendered does not need to be known, and typically is not. Therefore, object-based sensory data includes abstract sensory reproduction information—in these examples, the sensory object and its corresponding sensory metadata—allowing the sensory renderer to reproduce the created sensory effects via various sensory actuator types, via various numbers of sensory actuators, and from various sensory actuator locations in the environment.

[0305] In this example, box 1615 relates to the rendering of the object-based sensory data by the control system to generate actuator control signals. According to this example, the rendering is at least partially based on actuator data and local context information. In some examples, method 1600 may also involve obtaining user location data. According to some such examples, the rendering may be at least partially based on user location data. In some examples, the audio object metadata may at least include audio object spatial metadata indicating the spatial location of the audio object used to render the audio signal within the environment. In some examples, the rendering may be at least partially based on data corresponding to uncontrollable actuators that cannot be controlled by these actuator control signals.

[0306] According to this example, block 1620 relates to providing actuator control signals from a control system to one or more controllable actuators in a set of controllable actuators. According to some such examples, method 1600 may also involve receiving object-based sensory data and environment descriptor data corresponding to the playback environment from a sensory renderer. In some such examples, method 1600 may also involve receiving actuator descriptor data corresponding to characteristics of sensory actuators in the environment from a sensory renderer. In some examples, the environment descriptor data and actuator descriptor data may be, or may be included in, a reference. Figure 3 The environment and actuator data are described in data 004. In some examples, Figure 3 The MS controller API 003 shown can be implemented via MS renderer 001, and actuator-specific signals can be provided to actuator 008 by MS renderer 001. In some alternative examples, MS renderer 001 can provide actuator control signal 310 to MS controller API 003, and MS controller API 003 can provide actuator-specific control signals to actuator 008. In some examples, method 1600 may also involve providing sensory effects by sensory actuators in the environment.

[0307] Based on local context information, including local time information, among other examples, the set of controllable actuators may include a set of controllable luminaires, and the actuator control signals may include luminaire control signals. In some such examples, rendering may involve controlling brightness indicated by the luminaire control signals, controlling color indicated by the luminaire control signals, and controlling one or more automatic curtains, or combinations thereof, based at least in part on local time information.

[0308] In some examples where local context information includes local weather information, rendering may involve controlling brightness indicated by lighting control signals, controlling one or more automatic curtains, or both, based at least in part on local weather information.

[0309] Rendering can be based, at least in part, on local human behavior information, depending on some examples of local contextual information, including local human behavior information.

[0310] In some examples, local context information may include local user input. According to some such examples, local user input may include one or more explicit modal control indications.

[0311] Based on some examples, local context information may include local person immersion information. In some such examples, an immersive context mode may be enabled when the local context information indicates a high level of local person immersion.

[0312] In some examples, method 1600 may involve obtaining viewing position data corresponding to the user's location and position from the control system. In some such examples, rendering may be based at least in part on the viewing position data.

[0313] Based on some examples, object-based sensory data can include sensory object priority metadata. In some such examples, rendering can be based at least in part on sensory object priority metadata.

[0314] In some examples, method 1600 may involve receiving one or more renderer configuration parameters by a control system. According to some examples, the rendering may be at least partially based on one or more renderer configuration parameters.

[0315] In some examples, object-based sensory data may include object-based lighting data containing a set of light objects. In some such examples, the one or more renderer configuration parameters may correspond to an environment context mode that renders all ambient light objects and renders only a subset of dynamic light objects or none of the dynamic light objects. According to some examples, local context information may include local viewer immersion information. In some such examples, an environment context mode may be enabled when the local context information indicates a low level of local viewer immersion. In some examples, object-based lighting data may include light object position metadata, light object color metadata, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradient metadata, light object priority metadata, light object layer metadata, light object ambient metadata, light object motion metadata, or a combination thereof. In some examples, the light object metadata may include light object layer metadata corresponding to two or more layers. According to some examples, the two or more layers may include an environment layer, a motion layer, a custom layer, an overlay layer, a direct light object layer, an indirect light object layer, or a combination thereof. In some such examples, the rendering may involve intra-layer blending, inter-layer blending, or both. According to some examples, the rendering may involve applying one or more light activation laws. In some examples, the rendering may involve creating a sliced ​​room effect, creating an environment fill, or both, wherein creating the sliced ​​room effect involves generating a wavefront that propagates throughout the playback environment. According to some examples, the rendering is at least partially based on one or more renderer configuration parameters that may correspond to an immersive context mode that enables rendering of full light intensity, full color gamut, dynamic light objects, or combinations thereof.

[0316] In some examples, method 1600 may involve integrating the actuator data with the object-based sensory data. In some such examples, the integration may involve finding one or more nearest controllable actuators for each sensory object. According to some examples, the actuator data may include an actuator map. In some such examples, rendering the object-based sensory data to generate actuator control signals may involve projecting the set of sensory objects using the actuator map. According to some examples, projecting the set of sensory objects using the actuator map may generate actuator activation matrices as these actuator control signals. In some examples, the actuator map may include a lighting map. In some such examples, the lighting map may be an egocentric lighting map based on spatial coordinates of the playback environment or an egocentric lighting map based on spatial coordinates relative to the intended viewing location.

[0317] In some implementations, the object-based sensory data may include sensory object priority metadata. In some such examples, method 1600 may involve determining that two or more sensory objects are mapped to a single actuator, and rendering the object-based sensory data of the two or more sensory objects as actuator control signals for the single actuator, at least in part based on the sensory object priority metadata.

[0318] According to some examples, the rendering may be at least partially based on one or more renderer configuration parameters. In some examples, these one or more renderer configuration parameters may include a velocity priority parameter that assigns a higher priority to moving sensory objects than to stationary sensory objects. According to some examples, these one or more renderer configuration parameters may include a change priority parameter that assigns a higher priority to changing sensory objects than to static sensory objects.

[0319] As described elsewhere in this document, in some examples, these sensory objects may include light objects. In some such examples, these light objects may include color information. According to some such examples, this color information may include color variation information. In some such examples, the one or more renderer configuration parameters may include a color variation priority parameter that assigns a higher priority to color-varying light objects than to color-static light objects. In some examples, these light objects may include color saturation information. In some such examples, the one or more renderer configuration parameters may include a color saturation priority parameter that assigns a higher priority to light objects with higher color saturation than to light objects with lower color saturation.

[0320] According to some examples, the rendering may be at least partially based on one or more renderer configuration modes. These one or more renderer configuration modes may include a low actuator count mode, a content type mode, a color blending mode, a single-sensory object to single-actuator mode, a mode that changes the illuminance of a light object based on the distance between the light object and the luminaire, or a combination thereof. In some examples, the rendering may involve implementing color blending and priority ordering methods when multiple light objects are represented by a single luminaire.

[0321] In some examples, method 1600 may involve receiving an audio object by an audio renderer and receiving loudspeaker data corresponding to a loudspeaker in the environment by the audio renderer. According to some such examples, method 1600 may also involve the audio renderer providing loudspeaker control signals for controlling a loudspeaker in the environment to play audio corresponding to the audio object and synchronized with a sensory effect. Synchronization may be based, for example, on time information included in or with the sensory object and audio object, such as a timestamp. In some examples, method 1600 may also involve a loudspeaker in the environment playing audio corresponding to the audio object.

[0322] According to some examples, method 1600 may involve receiving video data synchronized with audio objects and object-based sensory data by a video renderer. According to some such examples, method 1600 may also involve providing video control signals by the video renderer for controlling one or more display devices in the environment to render images corresponding to the video control signals and synchronized with audio objects and sensory effects. In some examples, method 1600 may also involve rendering images on one or more display devices in the environment. The images may correspond to the video control signals.

[0323] The various features and aspects will be understood from the following enumerated example embodiments (EEE): EEE 1. A method for controlling a collection of controllable actuators, the method comprising: The control system obtains actuator data for the set of controllable actuators; The local context information is obtained by the control system; The control system receives object-based sensory data, which includes a set of sensory objects. The control system renders the object-based sensory data to generate actuator control signals, wherein the rendering is based at least in part on the actuator data and the local context information; and The control system provides actuator control signals to one or more of the controllable actuators in the set of controllable actuators.

[0324] EEE 2. The method as described in EEE 1, wherein the collection of controllable actuators includes one or more luminaires, one or more tactile devices, one or more airflow control devices, or combinations thereof.

[0325] EEE 3. The method as described in EEE 1 or EEE 2, further comprising obtaining user location data, wherein the rendering is based at least in part on the user location data.

[0326] EEE 4. The method of any one of EEE 1 to 3, wherein the local context information includes local time information, wherein the set of controllable actuators includes a set of controllable luminaires, wherein the actuator control signal includes a luminaire control signal, and wherein the rendering involves controlling the brightness indicated by the luminaire control signal, controlling the color indicated by the luminaire control signal, and controlling one or more automatic curtains, or a combination thereof, based at least in part on the local time information.

[0327] EEE 5. The method as described in EEE 4, wherein the local context information includes local weather information, and wherein the rendering involves controlling brightness indicated by a lighting control signal, controlling one or more automatic curtains, or both, based at least in part on the local weather information.

[0328] EEE 6. The method of any one of EEE 1 to 5, wherein the local context information includes local human behavior information, and wherein the rendering is based at least in part on the local human behavior information.

[0329] EEE 7. The method of any one of EEE 1 to 6, wherein the local context information includes local user input.

[0330] EEE 8. The method as described in EEE 7, wherein the local user input includes one or more explicit mode control indications.

[0331] EEE 9. The method of any one of EEE 1 to 8, wherein the local context information includes information about the preferences of one or more viewers, information about the presence or absence of one or more viewers, or a combination thereof.

[0332] EEE 10. The method of any one of EEE 1 to 9, wherein the local context information includes ambient light information.

[0333] EEE 11. The method of any one of EEE 1 to 10, wherein the local context information includes viewing environment information, local viewer location information, local device usage information, local viewer activity information, or a combination thereof.

[0334] EEE 12. The method of any one of EEE 1 to 11, wherein the object-based sensory data comprises object-based light data containing a set of light objects, and wherein the object-based light data comprises light object location metadata, light object color metadata, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradation metadata, light object priority metadata, light object layer metadata, light object atmosphere metadata, light object motion metadata, or a combination thereof.

[0335] EEE 13. The method of any one of EEE 1 to 12, further comprising obtaining viewing location data corresponding to a user location and positioning, wherein the rendering is based at least in part on the viewing location data.

[0336] EEE 14. The method of any one of EEE 1 to 11, wherein the object-based sensory data includes sensory object priority metadata.

[0337] EEE 15. The method of any one of EEE 1 to 14, wherein the rendering is based at least in part on one or more renderer configuration parameters.

[0338] EEE 16. The method as described in EEE 15, wherein the object-based sensory data includes object-based light data comprising a set of light objects, and wherein the one or more renderer configuration parameters correspond to an immersive context mode that enables rendering of full light intensity, full color gamut, dynamic light objects, or a combination thereof.

[0339] EEE 17. The method as described in EEE 16, wherein the local context information includes local viewer immersion information, and wherein the immersive context mode is enabled when the local context information indicates a high level of local viewer immersion.

[0340] EEE 18. The method of any one of EEE 15 to 17, wherein the object-based sensory data comprises object-based light data containing a set of light objects, and wherein the one or more renderer configuration parameters correspond to an environment context mode that causes all ambient light objects to be rendered and only a subset of dynamic light objects to be rendered or no dynamic light objects to be rendered.

[0341] EEE 19. The method as described in EEE 18, wherein the local context information includes local viewer immersion information, and wherein the environmental context mode is enabled when the local context information indicates a low level of local viewer immersion.

[0342] EEE 20. An apparatus configured to perform the method as described in any one of EEE 1 to 19.

[0343] EEE 21. A system configured to implement the method as described in any one of EEE 1 to 19.

[0344] EEE 22. One or more non-transitory computer-readable media having instructions stored thereon for controlling one or more devices to perform a method as described in any one of EEE 1 to 19.

[0345] The foregoing description illustrates various embodiments of this disclosure and examples of how aspects of this disclosure may be implemented. The foregoing examples and embodiments should not be considered as limited embodiments, but are presented to illustrate the flexibility and advantages of this disclosure as defined by the appended claims. Other arrangements, embodiments, implementations, and equivalents will be apparent to those skilled in the art based on the foregoing disclosure and the appended claims, and may be employed without departing from the spirit and scope of this disclosure as defined by the claims.

Claims

1. A method for controlling a collection of one or more controllable actuators, the method comprising: The control system obtains actuator data for the set of one or more controllable actuators; The control system receives object-based sensory data, which includes a set of one or more sensory objects. The control system renders the object-based sensory data to generate one or more actuator control signals, wherein the rendering is at least partially based on the actuator data; as well as The control system provides one or more actuator control signals to one or more of the set of controllable actuators.

2. The method as described in claim 1, wherein, The collection of one or more controllable actuators includes one or more luminaires, one or more tactile devices, one or more airflow control devices, or combinations thereof.

3. The method of claim 1 or claim 2, further comprising obtaining user location data, wherein, The rendering is based at least in part on the user location data.

4. The method according to any one of claims 1 to 3, wherein, The object-based sensory data includes sensory object metadata.

5. The method of claim 4, wherein, The sensory object metadata includes at least sensory object location metadata.

6. The method of claim 4 or claim 5, wherein, The sensory object metadata includes at least sensory object size metadata.

7. The method of any one of claims 1 to 6, further comprising integrating the actuator data with the object-based sensory data, wherein, The integration involves finding one or more nearest controllable actuators for each sensory object.

8. The method according to any one of claims 1 to 7, wherein, The actuator data includes actuator mapping.

9. The method of claim 8, wherein, Rendering the object-based sensory data to generate one or more actuator control signals involves using the actuators to map a set of the one or more sensory objects.

10. The method of claim 9, wherein, The actuator map is used to project a set of one or more sensory objects to generate an actuator activation matrix as a control signal for the one or more actuators.

11. The method according to any one of claims 8 to 10, wherein, The actuator mapping includes a luminaire mapping, wherein the luminaire mapping is either an egocentric luminaire mapping based on spatial coordinates of the playback environment or an egocentric luminaire mapping based on spatial coordinates relative to the intended viewing location.

12. The method of any one of claims 1 to 11, further comprising obtaining playback environment data, wherein, The rendering is based at least in part on the playback environment data.

13. The method according to any one of claims 1 to 10, wherein, The object-based sensory data includes sensory object priority metadata.

14. The method of claim 13, further comprising: Determine whether two or more sensory objects are mapped to a single actuator; as well as The object-based sensory data of the two or more sensory objects are rendered into one or more actuator control signals for the single actuator, based at least in part on the sensory object priority metadata.

15. The method according to any one of claims 1 to 14, wherein, The rendering is based, at least in part, on one or more renderer configuration parameters.

16. The method of claim 15, wherein, The one or more renderer configuration parameters include a speed priority parameter, which assigns a higher priority to moving sensory objects than to stationary sensory objects.

17. The method of claim 15 or claim 16, wherein, The one or more renderer configuration parameters include a change priority parameter, which assigns a higher priority to changing sensory objects than to static sensory objects.

18. The method according to any one of claims 15 to 17, wherein, The one or more sensory objects include one or more light objects, wherein the one or more light objects include color information, and wherein the one or more renderer configuration parameters include a color change priority parameter, which assigns a higher priority to color-changing light objects than to color-static light objects.

19. The method of claim 18, wherein, The one or more light objects include color saturation information, and wherein the one or more renderer configuration parameters include a color saturation priority parameter, which assigns a higher priority to light objects with higher color saturation than to light objects with lower color saturation.

20. The method according to any one of claims 1 to 19, wherein, The rendering is based, at least in part, on one or more renderer configuration modes.

21. The method of claim 20, wherein, The one or more renderer configuration modes include low actuator count mode, content type mode, color blending mode, single-sensory object to single-actuator mode, mode that changes the illuminance of a light object based on the distance between the light object and the luminaire, or combinations thereof.

22. The method according to any one of claims 1 to 21, wherein, The rendering is based, at least in part, on data corresponding to uncontrollable actuators that cannot be controlled by the actuator control signal.

23. The method according to any one of claims 1 to 22, wherein, The sensory object includes one or more light objects, wherein the one or more light objects include color information, and wherein the rendering involves implementing a color blending and priority sorting method when multiple light objects are represented by a single luminaire.

24. The method according to any one of claims 1 to 22, wherein, The sensory objects include light objects and light object metadata.

25. The method of claim 24, wherein, The optical object metadata includes optical object location data.

26. The method of claim 24 or claim 25, wherein, The light object metadata includes light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradient metadata, light object priority metadata, or a combination thereof.

27. The method of any one of claims 24 to 26, wherein, The optical object metadata includes optical object layer metadata corresponding to two or more layers.

28. The method of claim 27, wherein, The two or more layers include an environment layer, a dynamic layer, a customization layer, an overlay layer, a direct light object layer, an indirect light object layer, or a combination thereof.

29. The method of claim 27 or claim 28, wherein, The rendering includes intra-layer blending, inter-layer blending, or both.

30. The method according to any one of claims 24 to 29, wherein, The rendering process involves applying one or more light activation rules.

31. The method according to any one of claims 24 to 30, wherein, The rendering includes creating a slice room effect, creating an environment fill, or both, and wherein creating the slice room effect includes generating a wavefront that propagates throughout the playback environment.

32. An apparatus configured to perform the method as described in any one of claims 1 to 31.

33. A system configured to implement the method as described in any one of claims 1 to 31.

34. One or more non-transitory computer-readable media having instructions stored thereon for controlling one or more devices to perform the method as described in any one of claims 1 to 31.

Images