Multi-sensory (MS) spatial mapping and characterization for MS rendering

Abstract

Some methods may involve obtaining controllable actuator positioning information for a set of one or more controllable actuators in an environment, the controllable actuators including one or more light fixtures, one or more tactile sensors, one or more airflow control devices, or a combination thereof. Some methods may involve obtaining controllable actuator capability information for each controllable actuator in the set of controllable actuators, and obtaining environment information corresponding to the environment. Some methods may involve generating an actuator-room response (ARR) based at least in part on the controllable actuator positioning information, the controllable actuator capability information, and the environment information, the actuator-room response summarizing a response of the environment to controllable actuator activation. Some methods may involve modifying the ARR to generate an actuator map (AM). The modification may involve regularizing, filling gaps in the ARR, reducing one or more overlapping volumes of the environment affected by multiple actuator responses, or a combination thereof.

Description

Cross-reference to related applications

[0001] This application claims priority to U.S. Provisional Application No. 63 / 669,236, filed July 10, 2024, and U.S. Provisional Application No. 63 / 514,105, filed July 17, 2023, each of which is incorporated herein by reference in its entirety. Technical Field

[0002] This disclosure relates to providing a multi-sensory (MS) experience, and more specifically to mapping and characterizing actuators and actuator deployment environments. Background Technology

[0003] Unless otherwise indicated herein, the methods described in this section are not prior art to the claims of this application and are not acknowledged as prior art by virtue of their inclusion in this section.

[0004] Media content delivery typically focuses on audio and screen-based visual experiences. The delivery of multi-sensory content has been limited due to the customized nature of actuation. For example, lighting is widely used as an artistic and functional expression in concerts. However, each installation is specifically designed for a particular set of lighting fixtures. Delivering lighting design outside of that set of fixtures targeted by the system design is generally not feasible. Other systems attempting to provide a broader light experience merely extend screen visuals through algorithms, but are not specifically created. Haptic content is designed for specific haptic devices. If another device (such as a game controller, mobile phone, or even a different brand of haptic device) is used, the creative intent of the content cannot be translated to a different actuator. Summary of the Invention

[0005] At least some aspects of this disclosure can be implemented via methods such as audio processing methods. In some cases, these methods can be implemented at least in part by control systems such as those disclosed herein.

[0006] Some methods may involve obtaining controllable actuator positioning information of a set of one or more controllable actuators in an environment by a control system. The one or more controllable actuators may include one or more luminaires, one or more tactile sensors, one or more airflow control devices, or combinations thereof. Some methods may involve obtaining controllable actuator capability information of each controllable actuator in the set of one or more controllable actuators by the control system. Some methods may involve obtaining environmental information corresponding to the environment by the control system. Some methods may involve generating an actuator-room response (ARR) by the control system based at least in part on the controllable actuator positioning information, the controllable actuator capability information, and the environmental information, the ARR summarizing the environment's response to one or more activations of the set of one or more controllable actuators.

[0007] In some examples, the environmental information may include at least one or more locations, one or more orientations, and one or more dimensions of one or more environmental features. In some examples, the one or more environmental features may include one or more structural elements of the environment. In some examples, the one or more structural elements may include one or more walls, ceilings, floors, or combinations thereof. In some examples, the one or more environmental features may include one or more furniture items. In some examples, the environmental information may include furniture location information. According to some examples, the environmental information may include environmental feature color information, environmental feature reflectivity information, or combinations thereof.

[0008] In some examples, obtaining the controllable actuator positioning information may involve the control system receiving camera data from one or more cameras. In some examples, the camera data may include one or more images of the set of one or more controllable actuators. In some examples, obtaining the controllable actuator positioning information may involve the control system determining the controllable actuator positioning information based at least in part on the camera data. Some methods may involve the control system determining controllable actuator actuation direction information based at least in part on the camera data. In some examples, the controllable actuator actuation direction information may include light direction information, air movement direction information, or both.

[0009] According to some examples, the camera data may include one or more optical images, one or more depth images, or a combination thereof. Some methods may involve obtaining inertial measurement data by the control system. In some examples, the inertial measurement data may include gyroscope data, accelerometer data, or both. In some examples, the controllable actuator positioning information may be determined based on a simultaneous localization and mapping (SLAM) process, at least in part, based on the camera data and the inertial measurement data.

[0010] In some examples, the one or more controllable actuators may include one or more controllable luminaires. Some methods may involve the control system sending one or more calibration signals to the one or more controllable luminaires while the one or more cameras are acquiring camera data. The one or more calibration signals may modulate the light intensity, light color, or both of the one or more controllable luminaires. Obtaining the controllable actuator capability information and obtaining the environmental information may be based at least in part on the camera data and the inertial measurement data. In some examples, the one or more calibration signals may be, or may include, a sequence having bounded cross-correlation within a set.

[0011] According to some examples, the one or more calibration signals can control the one or more controllable luminaires to emit light according to codes or patterns that enable the control system to distinguish one or more lighting effects produced by each of the one or more controllable luminaires. Some methods may involve color matching of the light emitted by two or more of the controllable luminaires.

[0012] According to some examples, obtaining the controllable actuator capability information may involve a lamp characterization process that determines the characteristics of one or more lights in the environment. Some methods may involve a light localization and segmentation process that is at least partially based on the camera data. The lamp characterization process may be at least partially based on the light localization and segmentation process. In some examples, the light localization and segmentation process may involve analyzing the pixels of the camera data and determining which pixels correspond to the individual lights in the environment.

[0013] In some examples, obtaining this environmental information may involve an environmental characterization process that determines characteristic color information and characteristic reflectivity information of the environment. According to some examples, obtaining the controllable actuator capability information and obtaining this environmental information may involve an iterative process that determines one or more characteristics of a lamp within the environment, determines characteristic color information of the environment, and determines characteristic reflectivity information of the environment.

[0014] Some methods may involve modifying the ARR by the control system to produce an actuator map (AM). According to some examples, this modification may include regularization, filling gaps in the ARR by increasing one or more volumes of the environment affected by actuator responses, reducing one or more overlapping volumes of the environment affected by multiple actuator responses, or combinations thereof. In some examples, the AM may indicate the effect of each controllable actuator on the environment. According to some examples, the AM may be mathematically applicable to inversion. In some examples, the AM may allow object-based sensory data to be rendered as actuator commands.

[0015] Some methods may involve the control system identifying one or more ill-posed spatial coordinates of the AM. Some such methods may involve the control system generating one or more repulsion functions configured to deviate the trajectory of the sensory object from the one or more ill-posed spatial coordinates. Some such methods may involve storing or providing the one or more repulsion functions along with the AM.

[0016] In some examples, the ARR can be represented as a matrix, and the AM can be represented as a modified version of that matrix. According to some examples, the matrix may include values ​​corresponding to one or more actuator indices, one or more execution commands, and spatial coordinates.

[0017] In some examples, obtaining the controllable actuator positioning information may involve receiving controllable actuator capability information via an interface system. According to some examples, this interface system may be or may include a user interface system.

[0018] Some or all of the operations, functions, and / or methods described herein can be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory computer-readable media. Such non-transitory media may include one or more memory devices as described herein, including but not limited to one or more random access memory (RAM) devices, read-only memory (ROM) devices, etc. Therefore, some innovative aspects of the subject matter described in this disclosure can be implemented in one or more computer-readable non-transitory media on which software is stored.

[0019] At least some aspects of this disclosure can be implemented via apparatus. For example, one or more devices may be able to perform at least partially the methods disclosed herein. In some embodiments, the apparatus may include an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or combinations thereof. The control system may be configured to perform some or all of the disclosed methods.

[0020] Details of one or more embodiments of the subject matter described in this specification are set forth in the following figures and description. Other features, aspects, and advantages will become apparent from the description, figures, and claims. Note that the relative dimensions in the following figures may not be drawn to scale. Attached Figure Description

[0021] The disclosed embodiments will now be described by way of example only with reference to the accompanying drawings.

[0022] Figure 1A This is a block diagram illustrating examples of components of an apparatus capable of implementing various aspects of this disclosure.

[0023] Figure 1B Example elements of the endpoint are shown.

[0024] Figure 2 An example of an actuator element is shown.

[0025] Figure 3 Example components of a system for creating and playing multi-sensory (MS) experiences are shown.

[0026] Figure 4A Example components of a multi-sensory (MS) renderer are shown.

[0027] Figure 4B It is a flowchart outlining an example of a method that can be performed by an apparatus or system such as the apparatus or system disclosed herein.

[0028] Figure 5 Example components of another system for creating and playing MS experiences are shown.

[0029] Figure 6 The light room response (LRR) of a single luminaire is depicted using egocentric polar coordinates.

[0030] Figure 7 An example element of a light mapping system is shown.

[0031] Figure 8 An example of color volume is shown.

[0032] Figure 9 Alternative methods for generating LRR are described.

[0033] Figure 10 It shows what can be provided Figure 9 Examples of simulator information.

[0034] Figure 11A and Figure 11B An example of the enhanced LRR is shown.

[0035] Figure 12 It is a flowchart outlining an example of a method that can be performed by an apparatus or system such as the apparatus or system disclosed herein. Detailed Implementation

[0036] This document describes techniques related to providing multi-sensory media content. In the following description, numerous examples and specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that this disclosure, as defined by the claims, may include some or all of these examples, either alone or in combination with other features described below, and may further include modifications and equivalents of the features and concepts described herein.

[0037] The following description details various methods, processes, and procedures. While specific steps may be described in a particular order, this order is primarily for convenience and clarity. A particular step may be performed more than once, may occur before or after other steps (even if these steps are described in a different order), and may occur in parallel with other steps. A second step is only necessary if the first step must be completed before the second step can begin. This will be specifically indicated when it is unclear from the context.

[0038] In this document, the terms “and,” “or,” and “and / or” are used. These terms should be understood to have inclusive meanings. For example, “A and B” can at least mean: “both A and B,” or “at least both A and B.” As another example, “A or B” can at least mean: “at least A,” “at least B,” “both A and B,” or “at least both A and B.” As yet another example, “A and / or B” can at least mean: “A and B,” or “A or B.” When XOR is intended to be used, it will be specifically indicated (e.g., “either A or B,” or “at most one of A and B”).

[0039] This document describes the various processing functions associated with structures such as blocks, components, parts, and circuits. Typically, these structures can be implemented by one or more processors controlled by one or more computer programs.

[0040] As mentioned above, media content delivery typically focuses on audio and video experiences. Due to the personalized nature of actuation, the delivery of multi-sensory (MS) content has always been limited.

[0041] This application describes methods for expanding the creative palette of content creators, allowing for the creation and delivery of spatial MS experiences at scale. Some of these methods involve introducing new layers of abstraction to deliver the created MS experience to different endpoints using different types of lighting or actuators. As used herein, the term "endpoint" is synonymous with "playback environment" or simply "environment," referring to an environment that includes one or more actuators that can be used to deliver the MS experience. Such endpoints can include rooms (such as the living room in a home), cars, movie theaters, nightclubs, or other locations. Some disclosed methods involve creating, delivering, and / or rendering object-based sensory data, which can include sensory objects and corresponding sensory metadata. This abstraction enables the implementation of creative intent in an object-based format without prior knowledge of the specific controller type, number, and layout of any particular playback environment or "endpoint," thus achieving greater flexibility and scalability. In this document, the MS experience delivered via object-based sensory data can be referred to as a "flexibly expanded MS experience."

[0042] acronym MS - Multisensory MSIE – MS Immersive Experience AR - Augmented Reality VR - Virtual Reality PC — Personal Computer Figure 1A This is a block diagram illustrating examples of components of an apparatus capable of implementing various aspects of this disclosure. As with the other figures provided herein, Figure 1A The types and quantities of elements shown are provided as examples only. Other embodiments may include more, fewer, and / or different types and quantities of elements. According to some examples, device 101 may be or may include a device configured to perform at least some of the methods disclosed herein, such as a smart audio device, laptop computer, cellular phone, tablet device, smart home hub, etc. In some such embodiments, device 101 may be or may include a server configured to perform at least some of the methods disclosed herein.

[0043] In this example, device 101 includes at least an interface system 105 and a control system 110. In some embodiments, control system 110 may be configured to at least partially perform the methods disclosed herein. According to some examples, control system 110 may be configured to determine environmental and actuator data, such as those referenced herein. Figure 3 , Figure 4A and Figure 5 The described environment and actuator data 004.

[0044] In some examples, the control system 110 may be configured to obtain (e.g., via interface system 105) controllable actuator positioning information of a set of controllable actuators in an environment, in this example, a playback environment. For example, the playback environment could be a room in a house, a vehicle, etc. According to this example, the controllable actuators include one or more lighting fixtures, one or more tactile sensors, one or more airflow control devices, or combinations thereof. According to some examples, the control system 110 may be configured to obtain controllable actuator actuation direction information, such as light direction information, airflow direction information, etc.

[0045] According to some examples, the control system 110 can be configured to obtain controllable actuator capability information for each controllable actuator in a set of controllable actuators. Each controllable actuator may require a specific type and / or level of control to produce a desired output. For example, a Philips Hue™ bulb may need to receive control information in a specific format to turn on a lamp with a digital representation of a specific saturation, brightness, and hue, as well as a desired drive level. Therefore, the controllable actuator capability information may indicate the type of controllable actuator, the required control signal format (if any), the drive level required to obtain a specific actuator response, etc. In some examples, obtaining this controllable actuator capability information may involve a lamp characterization process that determines the characteristics of the controllable lamp within the environment.

[0046] In some examples, the control system 110 may be configured to obtain environmental information corresponding to the environment, in other words, environmental information corresponding to the environment in which the controllable actuator is located. This environmental information may include the location, orientation, and / or size of one or more environmental features. These one or more environmental features may include one or more structural elements of the environment, such as one or more walls, ceilings, floors, or combinations thereof. In some examples, the environmental information may include environmental feature color information, environmental feature reflectivity information, or combinations thereof. These one or more environmental features may include one or more furniture items. In some such examples, the environmental information may include furniture location information.

[0047] This paper discloses various methods for obtaining controllable actuator positioning information, controllable actuator capability information, and environmental information. Such methods include automated methods, manual / user input-based methods, and "hybrid" methods that are partially automated and partially manual.

[0048] According to some examples, the control system 110 can be configured to generate an actuator-room response (ARR) that summarizes the environment’s response to the activation of a set of controllable actuators.

[0049] The matrix representation of ARR is generally ill-posed for inversion. For example, according to some matrix representations of ARR, if there is no actuator effect at a particular location in the playback environment, the values ​​of some columns or rows of the ARR matrix can be zero.

[0050] In some examples, the control system 110 can be configured to modify the ARR (e.g., modify the matrix representation of the ARR) to produce an actuator map (AM) indicating the effect of each controllable actuator on the environment. This AM can allow object-based sensory data to be rendered as actuator commands. According to some examples, the matrix representation of the AM is mathematically suitable for inversion. Modifying the ARR can involve regularization, filling gaps in the ARR by increasing one or more volumes of the environment affected by actuator responses (e.g., eliminating zero values ​​in the ARR matrix), reducing one or more overlapping volumes of the environment affected by multiple actuator responses, or combinations thereof.

[0051] Interface system 105 may include one or more network interfaces and / or one or more external device interfaces (such as one or more Universal Serial Bus (USB) interfaces). According to some implementations, interface system 105 may include one or more wireless interfaces. Interface system 105 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. In some examples, interface system 105 may include a control system 110 and a memory system (such as...). Figure 1A The control system 110 may include one or more interfaces between the optional memory system 115 shown in the diagram. However, in some cases, the control system 110 may include a memory system.

[0052] For example, the control system 110 may include a general-purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.

[0053] In some implementations, the control system 110 may reside in more than one device. For example, a portion of the control system 110 may reside in a device within the environment (such as a laptop computer, tablet computer, smart audio device, etc.), and another portion of the control system 110 may reside in a device outside the environment (such as a server). In other examples, a portion of the control system 110 may reside in a device within the environment, and another portion of the control system 110 may reside in one or more other devices within the environment.

[0054] Some or all of the methods described herein can be executed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices as described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. One or more non-transitory media may, for example, reside on... Figure 1A In the optional memory system 115 and / or control system 110 shown. Therefore, various innovative aspects of the subject matter described in this disclosure can be implemented in one or more non-transitory media on which software is stored. For example, the software may include instructions for controlling at least one device to process audio data. For example, the software may be provided by, for example, Figure 1A The control system 110 and other control system components perform the operation.

[0055] In some examples, device 101 may include Figure 1A The optional microphone system 120 is shown in the figure. The optional microphone system 120 may include one or more microphones. In some embodiments, the one or more microphones may be part of or associated with another device, such as a speaker in a speaker system, a smart audio device, etc.

[0056] According to some embodiments, device 101 may include Figure 1AThe optional actuator system 125 is shown in the figure. The optional actuator system 125 may include one or more loudspeakers, one or more haptic devices, one or more luminaires (also referred to herein as illuminators), one or more fans or other airflow devices, one or more display devices (including, but not limited to, one or more televisions), one or more position actuators, one or more other types of devices for providing an MS experience, or combinations thereof. As used herein, the term "luminaire" generally refers to any actuator configured to provide light. The term "luminaire" encompasses various types of light sources, including individual light sources (such as light bulbs), light source groups (such as light strips), light panels (such as light-emitting diode (LED) panels), projectors, display devices (such as television (TV) screens), etc. A "light fixture" can be movable, therefore, in this context, the word "fixture" does not mean that the fixture must be in a fixed position in space. As used herein, the term "position actuator" generally refers to a device configured to change the position or orientation of a person or object, such as a motion simulator seat. A loudspeaker may sometimes be referred to herein as a "speaker". In some embodiments, the optional actuator system 125 may include a display system comprising one or more displays, such as one or more light-emitting diode (LED) displays, one or more organic light-emitting diode (OLED) displays, etc. In some examples where device 101 includes a display system, the optional sensor system 130 may include a touch sensor system and / or gesture sensor system proximate to one or more displays of the display system. According to some such embodiments, the control system 110 may be configured to control the display system to present a graphical user interface (GUI), such as a GUI associated with implementing one of the methods disclosed herein.

[0057] In some embodiments, device 101 may include Figure 1A The optional sensor system 130 shown in the figure may include a touch sensor system, a gesture sensor system, one or more cameras, etc.

[0058] Figure 1BExample elements of an endpoint are shown. In this example, the endpoint is a living room 1001, which contains multiple actuators 008, some furniture 1010, and a person 1000 (also referred to herein as a user) who will flexibly extend the MS experience. Actuators 008 are devices capable of altering the environment 1001 in which the user 1000 is located. Actuators 008 may include one or more haptic devices, one or more lamps (also referred to herein as illuminators), one or more fans or other airflow devices, one or more display devices (including, but not limited to, one or more televisions), one or more position actuators, one or more other types of devices for providing the MS experience, or combinations thereof.

[0059] The number, arrangement, and capability of the actuators 008 in space 1001 can vary significantly between different endpoint types. For example, the number, arrangement, and capability of actuators 008 in a car or other vehicle typically differ from those in a living room, nightclub, terrace, etc. In many embodiments, the number, arrangement, and / or capability of actuators 008 can also vary significantly between different instances of the same type (e.g., between a small living room with 2 actuators 008 and a large living room with 16 actuators 008). This disclosure describes various methods for creating flexible, expandable MSIEs and extending them to these heterogeneous endpoints.

[0060] Figure 2 An example of actuator elements is shown. In this example, the actuator is an illuminator 1100, which includes a network module 1101, a control module 1102, and a light emitter 1103. The terms "illuminator" and "luminaire" may be used synonymously herein. According to this example, the light emitter 1103 includes one or more light-emitting devices, such as light-emitting diodes, configured to emit light into the environment where the illuminator 1100 resides. In this example, the network module 1101 is configured to provide network connectivity to one or more other devices in space, such as devices that send commands to control the illuminator 1100 to emit light. According to this example, the network module 1101 is... Figure 1A An example of an interface system 105. In this example, control module 1102 is configured to receive signals via network module 1101 and control optical transmitter 1103 accordingly. According to this example, control module 1102 is Figure 1A An example of a control system 110.

[0061] Other examples of actuators may include network module 1101 and control module 1102, but may include other types of actuation elements. Some such actuators may include one or more tactile devices, one or more fans or other airflow devices, one or more position actuators, etc.

[0062] Figure 3 Example components of a system for creating and playing multisensory (MS) experiences are shown. Similar to other figures provided in this article, Figure 3 The types and quantities of elements shown are provided by way of example only. Other implementations may include more, fewer, and / or different types and quantities of elements. According to some examples, system 300 may be or may include one or more devices configured to perform at least some of the methods disclosed herein. In some examples, system 300 may include devices configured to perform at least some of the methods disclosed herein. Figure 1A One or more instances of the control system 110.

[0063] According to the examples in this disclosure, the method for creating and delivering an object-based MS Immersive Experience (MSIE) involves applying a set of techniques for creating, delivering, and rendering object-based sensory data, which may include sensory objects and corresponding sensory metadata, to actuator 008. Some examples are described in the following paragraphs.

[0064] Object-based representation: In various disclosed implementations, multi-sensory (MS) effects are represented using content that can be referred to herein as multi-sensory (MS) objects or simply "sensory objects". According to some such implementations, attributes such as layer type and priority can be assigned to, associated with, and attached to each sensory object, thereby enabling the content creator's intent to be represented in the rendered experience. Detailed examples of sensory object attributes are described below.

[0065] In this example, system 300 includes a content creation tool 000 configured to design multi-sensory (MS) immersive content and to output object-based sensory data 005, individually or in combination with corresponding audio data 011 and / or video data 012, depending on a specific implementation. The object-based sensory data 005 may include timestamp information and information indicating the type of sensory object, sensory object attributes, etc. In this example, the object-based sensory data 005 is not “channel-based” data corresponding to one or more specific sensory actuators in the playback environment, but rather generalized to multiple playback environments with multiple actuator types, multiple actuators, etc. In some examples, the object-based sensory data 005 may include object-based light data, object-based tactile data, object-based airflow data, or object-based position actuator data, object-based olfactory data, object-based smoke data, object-based data for one or more other types of sensor effects, or combinations thereof. According to some examples, the object-based sensory data 005 may include sensory objects and corresponding sensory metadata. For example, if object-based sensory data 005 includes object-based light data, then the object-based light data may include light object location metadata, light object color metadata, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradation metadata, light object priority metadata, light object layer metadata, or a combination thereof. In some examples, object-based sensory data 005 may include time data, such as timestamp information. Although in this example, content creation tool 000 is shown as providing a stream of object-based sensory data 005 to experience player 002, in alternative examples, content creation tool 000 may generate object-based sensory data 005 that is stored for later use. An example of a graphical user interface for a light-based object-based content creation tool is described below.

[0066] Examples of MS object properties The following is a non-exhaustive list of possible properties of an MS object: • Priority; •layer; • Hybrid mode; •Persistence; • Effects; and • Spatial sound and image laws.

[0067] Effect As used herein, the term "effect" for an MS object is a synonym for the MS object type. An "effect" is, or indicates, the sensory effect provided by the MS object. If the MS object is a light object, its effect will involve providing direct or indirect light. If the MS object is a tactile object, its effect will involve providing some type of tactile feedback. If the MS object is an airflow object, its effect will involve providing some type of airflow. Some examples involve other "effect" categories, as described in more detail below.

[0068] Persistence Some MS objects may contain persistent properties in their metadata. For example, when a movable MS object moves around the scene, it can persist for a period of time at the locations it has passed through. This period of time can be indicated by persistent metadata. In some implementations, the MS renderer is responsible for building and maintaining persistent state.

[0069] layer Based on some examples, individual MS objects can be assigned to "layers" where MS objects are grouped together based on one or more shared characteristics. For example, layers can be grouped together based on the expected effects or types of MS objects, which may include, but are not limited to, the following: - Atmosphere / Environment -Informative - Emphasis / Attention Alternatively or additionally, in some examples, layers can be used to group MS objects together based on shared properties, which may include, but are not limited to, the following: -color -strength -size -shape -Location -A zone in space Priority In some examples, MS objects can have a priority attribute, which allows the renderer to determine which(s) should have priority in an environment where MS objects compete for limited actuators. For example, if the volume corresponding to multiple light objects overlaps with a single light fixture when all light objects are scheduled to be rendered, the renderer can refer to the priority of each light object to determine which(s) to render. In some examples, priority can be defined between or within layers. According to some examples, priority can be associated with specific attributes such as intensity. In some examples, priority can be defined by time: for example, the most recently rendered MS object may take precedence over previously rendered MS objects. According to some examples, priority can be used to specify MS objects or layers that should be rendered regardless of the limitations of a particular actuator system in the playback environment.

[0070] Spatial sound image law The spatial acoustic-image law can define how MS objects move in space and how MS objects affect actuators when they move between actuators.

[0071] Hybrid mode Blend patterns specify how multiple objects can be reused on a single actuator. In some examples, blend patterns may include one or more of the following: -Maximum Mode: Selects the MS object that activates the actuator the most times; - Blend mode: Blend some or all objects according to a set of rules, such as by summing activation levels, taking the average of activation levels, or blending colors according to activation level or priority level; -MaxNmix: Mix the first N MS objects according to the rule set (by activation level).

[0072] Based on some examples, instead of (or in addition to) per-object metadata, more general metadata can be defined for the entire multi-sensory content file. For example, an MS content file may include metadata such as the trimming process or the mastering environment.

[0073] Repair and control In the context of Dolby Vision™, a feature called "Trim Controls" can serve as guidance on how to adjust the default rendering algorithm for specific environments or conditions at endpoints. Trim Controls can specify ranges and / or default values ​​for various attributes, including saturation, tonal detail, gamma, etc. For example, automotive trim controls can exist that provide specific default values ​​and / or sets of rules for rendering in automotive environments, such as guidance on including only objects of a specific priority or layer. Other examples include trim controls for environments with limited, complex, or sparse multi-sensory actuators.

[0074] Mastering environment A single multi-sensory content item can include metadata about attributes of the mastering environment, such as room size, reflectivity, and ambient bias lighting levels. Specific attributes can vary depending on the desired endpoint actuator. Mastering environment information can help provide reference points for rendering within the playback environment.

[0075] MS Object Renderer: Various disclosed embodiments provide a renderer configured to render MS effects to actuators in a playback environment. According to this example, system 300 includes an MS renderer 001 configured to render object-based sensory data 005 to actuator control signals 310, at least in part based on environment and actuator data 004. In this example, MS renderer 001 is configured to output the actuator control signals 310 to MS controllers 003 configured to control actuators 008. In some examples, MS renderer 001 may be configured to receive light objects and object-based lighting metadata indicating a desired lighting environment, as well as lighting information about the local lighting environment. The lighting information is a generic type of environment and actuator data 004 and may include one or more characteristics of one or more controllable light sources in the local lighting environment. In some examples, MS renderer 001 may be configured to determine the approximate drive level of each of the one or more controllable light sources in relation to the desired lighting environment. According to some examples, the MS renderer 001 (or one of the MS controllers 003) can be configured to output drive levels to at least one of the controllable light sources. Some alternative examples may include separate renderers for each type of actuator 008, such as one renderer for a luminaire, another for a haptic device, another for an airflow device, etc. In other embodiments, a single renderer may be configured as an MS renderer and an audio renderer and / or a video renderer. In some embodiments, the MS renderer 001 may be configured to adapt to changing conditions. Some examples of implementations of the MS renderer 001 are described in more detail below.

[0076] The environment and actuator data 004 may include content referred to herein as a “room descriptor” that describes the actuator’s location (e.g., according to an x, y, z coordinate system or a spherical coordinate system). In some examples, the environment and actuator data 004 may indicate actuator orientation and / or placement attributes (e.g., orientation and north-facing, omnidirectional, occlusion information, etc.). According to some examples, the environment and actuator data 004 may indicate actuator orientation and / or placement attributes according to a 3 × 3 matrix, in which three elements (e.g., elements in the first row) represent spatial location (x, y, z), three other elements (e.g., elements in the second row) represent orientation (roll, pitch, yaw), and three other elements (e.g., elements in the third row) indicate scale or size (sx, sy, sz). In some examples, the environment and actuator data 004 may include a device descriptor describing actuator attributes associated with the MS renderer 001, such as the intensity range and color gamut of a luminaire, the airflow velocity range and(s) direction of an airflow device, etc. The following section provides additional details about environmental and actuator data 004, including how environmental and actuator data 004 can be obtained.

[0077] In this example, system 300 includes an experience player 002 configured to receive object-based sensory data 005', audio data 011', and video data 012', and to provide the object-based sensory data 005' to an MS renderer 001, the audio data 011' to an audio renderer 006, and the video data 012' to a video renderer 007. In this example, the reference numerals for the object-based sensory data 005', audio data 011', and video data 012' received by the experience player 002 include an apostrophe (') to indicate that the data may be encoded in some cases. Similarly, the object-based sensory data 005', audio data 011', and video data 012' output by the experience player 002 do not include an apostrophe to indicate that the data may have been decoded by the experience player 002 in some cases. According to some examples, the experience player 002 can be a media player, game engine, or component integrated into a television, DVD player, soundbar, set-top box, or service provider media device (such as Chromecast, Apple TV, or Amazon Fire TV). In some examples, the experience player 002 can be configured to receive encoded object-based sensory data 005' as well as encoded audio data 011' and / or encoded video data 012'. In some such examples, the encoded object-based sensory data 005' can be received as part of the same bitstream as the encoded audio data 011' and / or encoded video data 012'. Some examples are described in more detail below. According to some examples, the experience player 002 can be configured to extract object-based sensory data 005 from a content bitstream and provide decoded object-based sensory data 005 to an MS renderer 001, decoded audio data 011 to an audio renderer 006, and decoded video data 012 to a video renderer 007. In some examples, the timestamp information in the object-based sensory data 005 may be used, for example, by the experience player 102, MS renderer 001, audio renderer 106, video renderer 107, or all of them, to synchronize the effects associated with the object-based sensory data 005 with the audio data 111 and / or the video data 112, which may also include timestamp information.

[0078] According to this example, system 300 includes MS controllers 003 configured to communicate with various actuator types using an application programming interface (API) or one or more similar interfaces. Generally, each actuator will require a specific type of control signal to produce the desired output from the renderer. According to this example, MS controller 003 is configured to map the output from MS renderer 001 to the control signals for each actuator. For example, a Philips Hue™ bulb receives control information in a specific format to turn on a digital representation of the light with a specific saturation, brightness, and hue, as well as a desired drive level. In some alternative examples, MS renderer 001 may also be configured to implement some or all of the MS controllers 003. For example, MS renderer 001 may also be configured to implement one or more lighting-based APIs but not haptic-based APIs, and vice versa.

[0079] In some examples, the room descriptor can also describe the size and orientation of the playback environment itself to establish a relative or absolute coordinate system to which all objects are positioned. For example, in a living room, the display screen can be considered as the front, or in some cases, as the center of the front, and the floor and ceiling can be considered as vertical boundaries. In some such examples, the room descriptor can also indicate the boundaries corresponding to the left, right, front, and back walls relative to the front position. According to some examples, the room descriptor can also be provided in the form of a matrix (such as a 3 × 3 matrix). This room descriptor information is used to describe the physical dimensions of the playback environment (e.g., expressed in physical distance units such as meters). In some such examples, sensory object positioning, sensory object size, and sensory object orientation can be described in units relative to the room size, such as in the range of -1 to 1. In some cases, the room descriptor can also describe the preferred viewing position based on the matrix.

[0080] The type, number, and arrangement of actuators 008 will generally vary depending on the specific implementation. In some examples, actuators 008 may include lamps and / or light strips (also referred to herein as “illuminators”), vibration motors, airflow generators, position actuators, or combinations thereof.

[0081] Similarly, the type, quantity, and arrangement of the loudspeaker 009 and display device 010 will generally vary depending on the specific implementation. Figure 3 In the example shown, audio data 011 and video data 012 are rendered by audio renderer 006 and video renderer 007 to loudspeaker 009 and display device 010, respectively.

[0082] As described above, according to some embodiments, system 300 may include methods configured to perform at least some of the methods disclosed herein. Figure 1AThe control system 110 may be one or more instances. In some such examples, one instance of the control system 110 may implement a content creation tool 000, and another instance of the control system 110 may implement an experience player 002. In some examples, one instance of the control system 110 may implement an audio renderer 006, a video renderer 007, a multi-sensory renderer 001, or a combination thereof. According to some examples, an instance of the control system 110 configured to implement the experience player 002 may also be configured to implement an audio renderer 006, a video renderer 007, a multi-sensory renderer 001, or a combination thereof.

[0083] Figure 4A Example components of a multi-sensory (MS) renderer are shown. Similar to other figures provided in this article, Figure 4A The types and quantities of components shown are provided as examples only. Other implementations may include more, fewer, and / or different types and quantities of components. MS Renderer 001 is a reference in this example. Figure 3 The described instance is MS Renderer 001. In some examples, MS Renderer 001 can be... Figure 1A The control system 110 is implemented using one or more instances of it.

[0084] Based on this example, Figure 4A Includes the following elements: •004: Environmental and actuator data, which can be referenced. Figure 3 Describe it; •005: Object-based sensory data 005, which can be referenced Figure 3 Describe it; •423 Actuator Mapping (AM) indicates the location of at least controllable actuator 008 in a specific playback environment; • Projection module 450 is configured to project an MS object based at least partially on object-based sensory data 005, using AM 423. The MS object may also be referred to as a "sensory object" in this document because in some cases, only one type of sensory object may exist in the object-based sensory data 005 (e.g., only tactile objects or only light objects). In this example, projection module 450 is configured to project the MS object based at least partially on sensory object metadata, wherein the sensory object metadata may include at least sensory object positioning metadata and sensory object size metadata; • 440 Actuator Activation Matrix (AAM), output by projection module 450 according to this example. For example, the AAM can indicate whether there are sensory objects (if any) currently contained within the playback environment that correspond to one or more corresponding actuators. For example, the AAM can indicate whether the light object positioning metadata and light object size metadata of a light object indicate that a particular luminaire is located within a volume of the playback environment corresponding to the positioning and size of that light object; • The 451 hybrid module is configured to convert AAM 440 into actuator control signals based at least in part on environmental and actuator data and renderer configuration data; • 452 Optional renderer configuration data, which may include information about one or more settings for the MS renderer 001, such as settings indicating desired motion, mode, etc. In some examples, renderer configuration data 452 may be changed automatically using a context-aware system (as described in more detail below); and •310: Actuator control signal, which can be referenced. Figure 3 The following description is provided. In some examples, the actuator control signal 310 can be sent to the individual actuators 008; while in other examples, the actuator control signal 310 can be sent to the MS controller 003, which can be configured to send appropriate control signals to various types of actuators 008.

[0085] According to some examples, AAM 440 is a matrix that describes, according to actuator map 423, how much of the sensory object itself is projected onto each actuator. In some examples, AAM 440 can be of size N. O × N A A real matrix, where N O This represents the number of sensor objects in the environment, and N A This indicates the number of controllable actuators in the environment. In this example, the hybrid module 451 is configured to generate actuator control signals 310 at least in part based on AAM 440 and environmental and actuator data 004. In some examples, the hybrid module 451 may be configured to generate actuator control signals 310 at least in part based on optional renderer configuration data 452. According to some examples, the hybrid module 451 may be configured to be at least in part based on sensory object metadata (such as... Figure 4A As shown, the sensory object metadata can be received as part of object-based sensory data 005 to generate actuator control signals 310, such as mixing and audio-visual laws associated with at least one sensory object.

[0086] In some examples, the hybrid module 451 may be configured to generate the actuator control signal 310 based at least in part on one or more of the following: 1. Threshold element of AAM 440; 2. Take the maximum value of a specific column of AAM 440—in other words, take the sensory object that activates a specific actuator the most times as the output; 3. Take any combination of the first N elements and perform at least one of the following operations: o Mixes objects in the actuator channel; o Pushes the object to an adjacent channel.

[0087] In some embodiments, the projection module 450 can be configured to generate a sensory object image using the spatial coordinates (e.g., x, y, z coordinates) and the size of the sensory object to produce an I n (x,y,z), where I n Indicates the first n A sensor object image of a sensor object. Then, in some examples, the projection module 450 can be configured to calculate the first [value] of each column (actuator index) of AAM 440 by taking the inner product of the object image and the actuator mapping corresponding to the actuator. n Row (object index). Projection module 450 can generate and perform object images, actuator mappings, and dot products in any convenient spatial domain, including but not limited to polar coordinates, cylindrical coordinates, or Cartesian coordinates.

[0088] Some implementations may involve implementing what may be referred to herein as "repulsion," which can be used to avoid potentially undesirable sensory effects, such as lighting effects that might occur when a sensory object is located in one or more areas of the playback environment. In some such examples, repulsion data may be included in the environmental and actuator data 004 and AM 423, and therefore may be part of the information input to the projection module 450. In some such examples, when the MS object is projected onto AM423 to produce AAM 440, the spatial coordinates of the MS object are augmented.

[0089] Multi-sensory rendering synchronization Object-based MS rendering involves flexibly rendering different modalities to endpoints / playback environments. Endpoints have different capabilities depending on various factors, including but not limited to the following: • Number of actuators • Modalities of these actuators (e.g., lighting and airflow control devices and tactile devices); • The types of these actuators (e.g., white smart lights versus RGB smart lights, or haptic vests versus haptic cushions) and • The positioning / layout of these actuators.

[0090] To render object-based sensory content to any endpoint, some processing of the object signals (e.g., intensity, color, pattern, etc.) is typically required. The processing of the signal path for each modality should not alter the relative phase of any feature within the object signal. For example, suppose lightning is presented in both tactile and visual modalities. The signal processing chain corresponding to the actuator control signals should not cause any type of sensory object signal (tactile or visual) to introduce a time delay sufficient to alter the perceptual synchronicity between the two modalities. The required level of synchronization may depend on various factors, such as whether the experience is interactive and what other modalities are involved. Depending on the specific context, the maximum time difference can range, for example, from approximately 10 ms to 100 ms.

[0091] Touch Rendering of object-based haptic content Object-based haptic content conveys the sensory aspects of a scene through abstract sensory representations rather than channel-based schemes. For example, instead of simply defining haptic content as a single-channel time-correlated amplitude signal played from a specific haptic actuator (such as a vibrating haptic motor) on a vest worn by the user, object-based haptic content can be defined by the sensation it is intended to convey. More specifically, in one example, there could be a haptic object representing the sensory effect of impact. Associated with this object are: • Spatial positioning of tactile objects; • Spatial direction / vector of tactile effect; • The intensity of the tactile effect; • Tactile spatial and temporal frequency data; and • Time-dependent amplitude signal.

[0092] Based on some examples, this type of haptic object can be automatically created in interactive experiences such as video games, for example, in a racing game when another car crashes into the player's car from behind. In this example, the MS renderer will determine how to render the spatial modality of this effect to a set of haptic actuators in the endpoints. In some examples, the renderer does this based on information about: • Various types of tactile devices are available, such as tactile vests and tactile gloves, tactile cushions and tactile controllers; • The location of each haptic device relative to (multiple) users (some haptic devices may not be coupled to (multiple) users, for example, a vibrator mounted on the floor or a seat); • Each haptic device offers different types of actuation, such as kinematic and vibratory haptics; • Start-up and stop delay for each haptic device (in other words, the speed at which each haptic device can be turned on and off); • The dynamic response of each haptic device (how much the amplitude can change); • The time-frequency response of each haptic device (the time-frequency response that a haptic device can provide); • Spatial distribution of addressable actuators within each haptic device: For example, a haptic vest could have dozens of addressable haptic actuators distributed across the user's torso; and • The time response of any haptic sensor used to render closed-loop haptic effects (e.g., active force feedback kinetic haptic devices).

[0093] These properties of the haptic modality at the endpoints inform the renderer how best to render a particular haptic effect. Consider again the car crash effect example. In this example, the player wears a haptic vest, haptic armbands, and haptic gloves. According to this example, the haptic shockwave effect is spatially located at the point where the car hits the player. The shockwave vector is determined by the relative velocity of the player's car and the car that hits the player. The spatial and temporal spectrum of the shockwave effect is created based on the type of materials the virtual car is expected to use, as well as other virtual world properties. The renderer then renders the shockwave using a set of haptic devices at the endpoints, based on the shockwave vector and the physical location of the haptic devices relative to the user.

[0094] The signals sent to each specific actuator are preferably provided in such a way that the sensory effect is consistent across all available (potentially heterogeneous) actuators. For example, due to the lack of capability of other actuators, the renderer may not render very high frequencies only to one of the haptic actuators (e.g., a haptic armband). Otherwise, when a shockwave moves through the player's body, the haptic effect perceived by the user will decrease as the wave moves through the vest, into the armband, and finally into the gloves because the haptic vest and gloves worn by the user are not capable of rendering such high frequencies.

[0095] Some types of abstract tactile effects include: • Shockwave effect, as described above; • Barrier effects, such as haptic effects used to represent spatial constraints in virtual worlds (e.g., in video games). If an active or resistive kinetic actuator is present on the input device (e.g., force feedback on a steering wheel or joystick), this effect can be rendered by applying resistance to the user's input. If no such actuator is available at the endpoint, in some examples, vibratory haptic feedback consistent with the collision of an in-game avatar with a barrier can be rendered; • Presence, such as indicating the presence of a large object (like a train) approaching a scene. This type of haptic effect can be rendered using the low-frequency rumble of certain haptic device actuators. It can also be rendered using pressure applied by an air bladder to provide tactile spatial feedback. • User interface feedback, such as a click from a virtual button. For example, this type of haptic effect can be rendered to the nearest actuator on the user's body that performed the click, such as a haptic glove worn by the user. Alternatively or additionally, this type of haptic effect can also be rendered to a vibrator coupled to the chair the user is sitting on. This type of haptic effect can be defined, for example, using a time-dependent amplitude signal. However, this signal can be modified (modulated, frequency-shifted, etc.) to best suit the haptic device(s) that will provide the haptic effect; • Motion Sensation. These haptic effects are designed to make the user perceive some form of motion. These haptic effects can be rendered by actuators on an actual moving user (e.g., a moving platform / seat). In some examples, the actuators can provide auxiliary modalities (e.g., via video) to enhance the motion being rendered; and • Trigger Sequence. These haptic effects are primarily characterized by their time-dependent amplitude signals. This signal can be rendered across multiple actuators and, in doing so, can be amplified. This amplification can include splitting the signal across multiple actuators in time or frequency. Some examples may involve amplifying the signal itself so that the sum of the haptic actuator outputs does not match the original signal.

[0096] Spatial effects and non-spatial effects Spatial effects are spatial effects constructed in a way that conveys certain spatial information about the multi-sensory scene being rendered. For example, if the playback environment is a room, then a shockwave moving through the room will be rendered differently to each haptic device, depending on the position and size of one or more haptic objects being rendered at a particular time, based on the location of each haptic device within the room.

[0097] In some examples, non-spatial effects can be targeted at specific parts of a user's body, regardless of the user's position or orientation. One example is a haptic device providing enhanced vibrations to a user's back to indicate immediate danger. Another example is a haptic device providing strong vibrations to indicate injury to a specific area of ​​the body.

[0098] Some effects can be non-narrative effects. These effects are often associated with user interface feedback, such as the tactile sensation used to indicate that a user has completed a level or clicked a button on a menu item. Non-narrative effects can be spatial or non-spatial.

[0099] Types of tactile devices Receiving information about the different types of haptic devices available at the endpoints allows the renderer to determine which types of sensory effects and rendering strategies it can use. For example, local haptic device data instructing the user to wear both a haptic glove and a vibrating haptic vest (or at least local haptic device data indicating the presence of the haptic glove and vibrating haptic vest in the playback environment) allows the renderer to render a consistent recoil effect on both devices when the user fires a gun in the virtual world. The actual actuator control signals sent to the haptic devices may differ from cases where only a single device is available. For example, if the user is only wearing the vest, the actuator control signals used to actuate the vest may differ in terms of the actuator control signal's activation timing, maximum amplitude, frequency, decay time, or combinations thereof.

[0100] Equipment positioning Understanding the location of haptic devices at endpoints allows renderers to consistently render spatial effects. For example, knowing the location of vibrating motors in a lounge allows the renderer to send actuator control signals to each vibrating motor in the lounge in a way that conveys spatial effects (such as shock waves propagating within the room). Additionally, although the location of wearable haptic devices is implied by their type (e.g., gloves on a user's hand), the renderer can also use knowledge of the location of these wearable haptic devices to convey both spatial and non-spatial effects.

[0101] Types of actuation provided by tactile devices Haptic devices can provide a range of different actuations, and thus provide a perceived sensation. These are generally divided into two basic categories: 1. Vibrational tactile sensation, such as vibration; or 2. Kinesthetic feedback, such as resistance feedback or propulsion feedback.

[0102] Actuation of any type can be static or dynamic, with dynamic effects changing in real time based on some sensor inputs. Examples include touchscreens that use vibration haptic actuators to render textures and position sensors that measure the position of the user's (multiple) fingers(s).

[0103] Furthermore, the physical construction of these actuators varies considerably and affects many other properties of the device. An example of this is the significant differences in activation delay or time-frequency response across the following types of haptic devices: •Eccentric rotating mass; • Linear resonant actuator; • Piezoelectric actuators; and • Linear magnetic ram.

[0104] The renderer should be configured to take into account the startup latency of a specific haptic device type when rendering signals that will be actuated by a haptic device in an endpoint.

[0105] Start-up and shutdown delay of tactile devices The start-up delay of a haptic device refers to the delay between the time when the actuator control signal is sent to the device and the time when the device physically responds. The stop delay refers to the delay between the time when the actuator control signal is sent to bring the device's output to zero and the time when the device stops actuating.

[0106] Time-frequency response Time-frequency response refers to the frequency range in which a haptic device can be actuated in a steady state, with the signal amplitude as a function of time.

[0107] Spatial frequency response Spatial frequency response refers to the frequency range in which the signal amplitude is a function of the spacing between the actuators of a tactile device. Devices with closely spaced actuators have a higher spatial frequency response.

[0108] Dynamic range Dynamic range refers to the difference between the minimum and maximum amplitude of a physical actuation.

[0109] Characteristics of sensors in closed-loop haptic devices Some dynamic effects use sensors to update actuation signals based on certain observed states. The sampling frequency of time and space, as well as noise characteristics, will limit the ability to update the control loop of the actuator providing the dynamic effect.

[0110] airflow Another modality that some multi-sensory immersive experiences (MSIEs) can use is airflow. Airflow can be rendered consistently with one or more other modalities, such as audio, video, lighting effects, and / or haptics. Unlike dedicated (e.g., channel-based) setups designed solely for 4D experiences in theaters (which may include "wind effects"), some airflow effects can also be provided at other endpoints that typically include airflow, such as a car or living room. Unlike channel-based systems, airflow sensory effects can be represented as airflow objects, which can include properties such as: • Spatial positioning; • The direction of the expected airflow effect; • Intensity / airflow velocity; and / or • Air temperature.

[0111] Some examples of airflow objects can be used to represent the movement of a bird flying by. To render the airflow actuator at the endpoint, information about the following can be provided to MS Renderer 001: • Types of airflow equipment, such as fans, air conditioners, and heaters; • The position of each airflow device relative to the user's location or the user's expected location; • The capabilities of airflow devices, such as their ability to control direction, airflow, and temperature; • The control level for each actuator, such as airflow speed and temperature range; and • The response time of each actuator, for example, how long it takes to reach a selected speed.

[0112] Examples of airflow usage at different endpoints In vehicles, such as cars, object-based metadata can be used to create experiences such as the following: • During scenes in horror movies or games, mimic the feeling of "chills down your spine" by using airflow down a chair; • Simulate the movement of a bird flying by; and / or • Create a gentle breeze in the sea view.

[0113] Within the small, enclosed space of a typical vehicle, temperature changes can likely be achieved over a relatively short period compared to temperature variations in a larger environment, such as a living room. In one example, MS Renderer 001 could raise the air temperature when a player enters a "lava level" or other hot area during gameplay. Some examples could include other elements, such as confetti in vents, to celebrate an event, such as a goal scored by a user's favorite football team.

[0114] In one example, airflow in a living space or other room can be synchronized with the breathing rhythm of guided meditation. In another example, airflow can be synchronized with the intensity of exercise, increasing airflow or decreasing temperature as intensity increases. In some examples, spatial control during rendering may be relatively limited. For instance, many existing airflow actuators are optimized for heating and / or air conditioning, rather than for providing sensory actuation that offers spatial diversity.

[0115] A combination of light, airflow, and touch Car example The following examples are described with reference to automobiles, but are applicable to other vehicles such as trucks and vans. In some examples, a user interface may be present on the steering wheel or on a touchscreen near or within the dashboard. According to some examples, the following actuators may be present in a car: 1. Individually addressable lights are spatially distributed throughout the vehicle in the following manner: o on the dashboard; o is below the footrest space; o on the door; and o is located within the central control console.

[0116] 2. Individually controllable air conditioning / heating vents are distributed throughout the vehicle as follows: o is in the front dashboard; o is below the footrest space; o is located in the center console, facing the rear seats; o is on the side pillar; o in the seat; and o. Guide windshield (for defogging).

[0117] 3. A individually controllable seat with vibrating tactile feedback; and 4. Individually controllable floor mats with vibrating tactile feedback.

[0118] In this example, the modes supported by these actuators include the following: • Lights with individually addressable LEDs throughout the car, as well as indicator lights on the dashboard and steering wheel; • Airflow via controlled air conditioning vents; •Touch, including: o Steering wheel: Tactile vibration feedback; o Dashboard touchscreen: haptic feedback and texture rendering; and o Seat: Touch, vibration, and movement.

[0119] In one example, a live music stream is rendered to four users seated in the front row. In this example, MS Renderer 001 attempts to optimize the experience for multiple viewing positions. During construction, before the artists take the stage and after the previous performances have concluded, the content includes: • Interlude music; •Low-intensity lighting; and • Represents the tactile content of a crowd colliding.

[0120] In addition to the rendered audio and video streams, light content also includes ambient light objects that move slowly within the scene. These ambient light objects can be rendered using one of the environment layer methods disclosed herein, for example, by not assigning spatial priority to any user's viewpoint. In some examples, haptic content can be spatially focused in a lower temporal frequency spectrum and can be rendered solely by vibrating haptic motors in the mat.

[0121] Based on this example, a fireworks event during a music stream corresponds to multi-sensory content including the following: • The light object that spatially corresponds to the location of the fireworks in the event; and • A tactile object that enhances the dynamic feel of fireworks through shockwave effects.

[0122] In this example, MS Renderer 001 renders both light and tactile objects spatially. For instance, light objects can be rendered in a car so that if the fireworks content is on the left side of the scene, everyone in the car will perceive the light object as coming from the left. In this example, only the lights on the left side of the car are actuated. Tactile objects can be rendered on both seats and floor mats in a way that conveys directionality to each user separately.

[0123] At the end of the concert, fireworks will appear in the audio content, and fireworks and confetti will also appear in the video content. In addition to rendering the light and tactile objects corresponding to the fireworks as described above, airflow modalities can be used to render the effect of confetti spray. For example, individually controllable airflow vents in an HVAC system can be pulsed.

[0124] Living room example In this embodiment, in addition to an audio / visual (AV) system including multiple loudspeakers and a television, the following actuators and related controls are available in the living room: • A haptic vest worn by the user (also known as the player); • A haptic vibrator installed on the seat where the player is sitting; • (Haptic) controllable smartwatch; • Smart lights distributed throughout the room; • Wireless controller; and • Addressable airflow bar (AFB), which includes an array of individually controllable fans directed at the user (similar to HVAC vents in a car's dashboard).

[0125] In this example, the user is playing a first-person shooter game that includes a scene where a destructive hurricane moves through the level. As the hurricane moves, in-game objects are thrown around, and some objects hit the player. Haptic objects rendered by MS Renderer 001 enable the delivery of shockwave effects across all haptic devices the user can perceive. The actuator control signals sent to each device can be optimized based on the impact intensity of the in-game objects, the direction(s) of the impact, and the capabilities and positioning (as previously described) of each actuator.

[0126] Before the user is hit by an in-game object, the multisensory content includes tactile objects corresponding to non-spatial rumble, one or more airflow objects corresponding to directional airflow, and one or more light objects corresponding to lightning. The MS renderer 001 renders the non-spatial rumble to the haptic devices. The actuator control signals sent to each haptic device can be rendered such that the set of actuator control signals across the entire haptic array is consistent in the timing, intensity, and frequency of the perceived rumble. In some examples, the frequency content of the actuator control signals sent to the smartwatch can be low-pass filtered to match the limited frequency capabilities of the vest near the watch. The MS renderer 001 can render one or more airflow objects as actuator control signals for AFB, such that the airflow in the room is consistent with the player's position and line of sight in the game, as well as the direction of the hurricane itself. Lightning can be rendered in all modalities as (1) a white flash produced by a light source located in a suitable position (e.g., in or on the ceiling); and (2) a pulsed rumble in the user's wearable haptic and seat vibrator.

[0127] When a user is hit by an in-game object, a directional shockwave can be rendered to the haptic device. In some examples, a corresponding airflow pulse can be rendered. According to some examples, a damage absorption effect can be rendered by a light, indicating the amount of damage the player takes from being hit by an in-game object.

[0128] In some such examples, the signal can be spatially rendered to the haptic devices, causing the perceived shockwave to move across the player's body and within the room. The MS Renderer 001 can provide this effect based on actuator positioning information that indicates the haptic devices' positioning relative to each other. In addition to actuator capability information, the MS Renderer 001 can also provide the shockwave vector and position based on the actuator positioning information. According to some examples, non-directional airflow pulses can be rendered; for example, all AFB vents can be briefly enlarged to enhance the haptic modality. In some examples, a red halo can be rendered onto the light strip around the TV simultaneously to indicate to the player that they have taken damage in the game.

[0129] Figure 4B This is a flowchart outlining an example of a method that can be performed by a device or system such as the apparatus or system disclosed herein. As with other methods described herein, the blocks of method 400 need not be performed in the indicated order. In some embodiments, one or more blocks of method 400 may be performed simultaneously. Furthermore, some embodiments of method 400 may include more or fewer blocks than those shown and / or described. The blocks of method 400 may be performed by one or more devices, which may be (or may include) a control system (such as those described above). Figure 1AOne or more instances of the control system 110 shown and described.

[0130] In this example, box 405 relates to estimating (e.g., by control system 110) the actuator-room response (ARR), which summarizes the environment's response to activation of a set of controllable actuators in the environment. For example, the playback environment could be a room in a house, a vehicle, etc. In some examples, the set of controllable actuators could include one or more lighting fixtures, one or more haptic devices, one or more airflow control devices, or combinations thereof. In some examples, control system 110 can be configured to obtain controllable actuator positioning information for the set of controllable actuators in the environment. According to some examples, control system 110 can be configured to obtain controllable actuator capability information for each controllable actuator in the set of controllable actuators. For example, controllable actuator capability information could indicate the type of controllable actuator, the required control signal format (if any), the drive level required to obtain a particular actuator response, etc.

[0131] In some examples, the control system 110 may be configured to obtain environmental information corresponding to the environment, in other words, environmental information corresponding to the environment in which the controllable actuator is located. This environmental information may include the location, orientation, and / or size of one or more environmental features. These one or more environmental features may include one or more structural elements of the environment, such as one or more walls, ceilings, floors, or combinations thereof. In some examples, the environmental information may include environmental feature color information, environmental feature reflectivity information, or combinations thereof. These one or more environmental features may include one or more furniture items. In some such examples, the environmental information may include furniture location information.

[0132] This paper discloses various methods for obtaining controllable actuator positioning information, controllable actuator capability information, and environmental information. Such methods include automated methods, manual / user input-based methods, and "hybrid" methods that are partially automated and partially manual.

[0133] The process of estimating ARR and the way ARR is represented may differ depending on the specific implementation. In some examples, ARR can be represented as R(j, x, y, z, a, t), where: x represents the x-coordinate of the environment, which in this example is the room; y represents the y-coordinate of the room; z represents the z-coordinate of the room; 'a' represents the command to be executed, and it can be a vector, such as a color. t represents time; and J represents the actuator index.

[0134] For the steady-state response of all actuated actuators (in other words, ignoring t), that is, for all j, a is the same. For spatially overlapping actuators, ARR is not suitable for inversion due to its low rank. According to some examples, R(j,x,y,z) can be reformulated as a matrix R(j, X), where X represents the spatial coordinates (x,y,z) converted to planar indices, since x,y,z are orthogonal.

[0135] R is the projection of the actuator's effect on the environment (in this example, the room). Assume vector A represents all the actuators within the room. If we can invert R, we can obtain the mapping Ra. -1 This mapping projects the room effector onto vector A. In other words, the actuator control signal can be obtained from a desired room effect, which can be represented as object-based sensory data. The ability to obtain the actuator control signal from object-based sensory data indicating the desired room effect is desirable for object-based MS rendering.

[0136] However, the "raw" or unmodified version of R is often ill-posed (mathematically inapplicable) for inversion. One reason is that, in many cases, there may be "dead zones" where no actuator affects a particular X-coordinate. Therefore, a corresponding number of columns or rows may have zero values. In some cases, there may be correlations between columns or rows.

[0137] According to this example, box 410 relates to modifying the ARR to produce an actuator map (AM). According to some examples, the matrix representation of the AM is mathematically suitable for inversion. Modifying the ARR may involve regularization, filling gaps in the ARR by increasing one or more volumes of the environment affected by the actuator response (e.g., eliminating zero values ​​in the ARR matrix), reducing one or more overlapping volumes of the environment affected by multiple actuator responses, or a combination thereof.

[0138] This AM can allow object-based sensory data to be rendered as actuator commands. According to some implementations, AM can be used for the following purposes: This enables the creation of object-based sensory data (MS) in a reasonable, endpoint-independent manner. In other words, content creators do not need to direct MS content to specific channels corresponding to individual actuators. Instead, content creators can create MS objects and place them within a general representation of the playback environment. Supports MS renderer 001 to project spatial MS objects onto the actuator array in the environment.

[0139] A sensory object O can be represented using coordinate positioning (e.g., (x, y, z)) or an equivalent flattened coordinate representation X. Attributes of the sensory object can include its positioning and size, which allows the sensory object to be projected onto a range of X within the playback environment / endpoint, corresponding to its positioning and size. At any given time, actuators located within the endpoint volume corresponding to the sensory object size are actuation candidates for that time. Therefore, object-based representations of multisensory effects can be readily represented as effects on the endpoint room. These considerations not only facilitate the estimation of ARR but also the adjustment / enhancement of ARR to better suit inversion AM. AM can be used to... n Each sensory object is mapped to a vector A representing all actuators, for example, as shown below: .

[0140] In some examples, modifying ARR to AM in box 410 can make reversibility more feasible by performing one or both of the following operations: Support is achieved on X by artificially expanding the actuator's response to fill the gap; or Where feasible, reduce the overlap between actuators so that the response of a point object (an infinitely small object) is an impulse of form A.

[0141] Even after modification, the actuator mapping AM may still suffer from rank insufficiency (ignoring zero activations). In other words, the volume corresponding to multiple sensory objects may contain one or more of the same actuators in the environment at a given time. In addition to the rest of the environment and actuator data 004, the MS renderer 001 can also be configured to use other data (such as sensory object priority metadata) to determine the actuator control signal 310.

[0142] Multiple modes In some examples, when arranging the simultaneous rendering of two or more modalities of multi-sensory effects (such as tactile and light), the ARR and / or AM corresponding to each modality can be evaluated and / or modified. Models that link the effects of the two sensory modes should be employed to ensure they are suitable for the joint rendering of multi-modal sensory effects. The ARR and / or AM of multiple modalities may not overlap across all endpoints and may not mutually support each other (areas that can be activated by both modalities) because different devices are responsible for activating different modalities. This mutually supporting overlap (or lack thereof) can occur in any of the multiple domains, including the frequency and spatial domains. Furthermore, the transient or dynamic response of the actuators and the corresponding physical characteristics under different modalities should also be considered. For example, due to the time lag between sending the airflow actuation signal and achieving the desired airflow, there may be a time offset between the ARR of a lighting effect and the ARR of an airflow effect.

[0143] Figure 5 This illustrates example components of another system used for creating and playing MS experiences. Similar to the other diagrams provided in this article, Figure 5 The types and quantities of elements shown are provided by way of example only. Other implementations may include more, fewer, and / or different types and quantities of elements. According to some examples, system 500 may be or may include one or more devices configured to perform at least some of the methods disclosed herein. In some examples, system 500 may include devices configured to perform at least some of the methods disclosed herein. Figure 1A One or more instances of the control system 110.

[0144] Based on this example, Figure 5 The system shown is Figure 3 An example of the system shown. In this example, Figure 5 The system shown is an example of "Light and Shadow," in which video, audio, and lighting effects are combined to create an MS experience.

[0145] In this example, system 500 includes a scene creation tool 100, which is a reference... Figure 3 An example of the described content creation tool 000. The lighting creation tool 100 is configured to design and output object-based lighting data 505', which, depending on a specific implementation, is output individually or in combination with corresponding audio data 111' and / or video data 112'. The object-based lighting data 505' may include timestamp information and information indicating the attributes of the lighting object, etc. In some cases, timestamp information may be used to synchronize effects associated with the object-based lighting data 505' with the audio data 111' and / or video data 112', where timestamp information may also be included.

[0146] In this example, object-based light data 505' includes light objects and corresponding light metadata. For example, object-based light data may include light object location metadata, light object color metadata, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradation metadata, light object priority metadata, light object layer metadata, or a combination thereof. Although in this example, content creation tool 100 is shown as providing a stream of object-based light data 505' to experience player 102, in alternative examples, content creation tool 100 may generate object-based light data 505' that is stored for later use. An example of a graphical user interface for a light object-based content creation tool is described below.

[0147] In this example, system 500 includes an experience player 102 configured to receive object-based light data 505', audio data 111', and video data 112', and to provide the object-based light data 505 to a lighting renderer 501, the audio data 111 to an audio renderer 106, and the video data 112 to a video renderer 107. As described elsewhere herein, the object-based light data 505, audio data 111, and video data 112 may include timestamp information that can be used to synchronize MS effects with audio and / or video effects. According to some examples, experience player 102 may be a media player, a game engine, or a component integrated into a television, DVD player, soundbar, set-top box, or service provider media device (such as Chromecast, Apple TV, or Amazon Fire TV). In some examples, experience player 002 can be configured to receive encoded object-based light data 505' and encoded audio data 111' and / or encoded video data 112', for example, as part of the same bitstream as the encoded audio data 111' and / or encoded video data 112'. According to some examples, experience player 102 can be configured to extract object-based light data 505 from the content bitstream and provide decoded object-based light data 505 to a lighting renderer 501, decoded audio data 111 to an audio renderer 106, and decoded video data 112 to a video renderer 107. In some examples, experience player 002 can be configured to allow control over configurable parameters in the lighting renderer 501, such as immersion intensity. Some examples are described below.

[0148] As described above, according to some embodiments, system 500 may include methods configured to perform at least some of the methods disclosed herein. Figure 1A One or more instances of the control system 110. In some such examples, one instance of the control system 110 may implement the scene creation tool 100, and another instance of the control system 110 may implement the experience player 002. In some examples, one instance of the control system 110 may implement an audio renderer 006, a video renderer 007, a scene renderer 501, or a combination thereof. According to some examples, an instance of the control system 110 configured to implement the experience player 002 may also be configured to implement an audio renderer 006, a video renderer 007, a scene renderer 501, or a combination thereof.

[0149] In some examples, the room descriptor of the environment and lighting data 104 can describe the size and orientation of the playback environment itself to establish a relative or absolute coordinate system to which all objects are located. The room descriptor information can indicate or describe the physical dimensions of the playback environment (e.g., expressed in physical distance units such as meters). In some such examples, sensory object positioning, sensory object size, and sensory object orientation can be described in units relative to the room size, such as in the range of -1 to 1. The room descriptor can also describe the preferred viewing position. For example, in a living room, the display screen can be considered as being in front, or in some cases as being centered in front, and the floor and ceiling can be considered as vertical boundaries. In some such examples, the room descriptor can also indicate the boundaries corresponding to the left, right, front, and back walls relative to the in front position. According to some examples, at least some of the room descriptor information can be provided in matrix form. In some such examples, the matrix can be a 3 × 3 matrix, where a row or column corresponds to one dimension of three-dimensional space.

[0150] According to this example, system 500 includes a lighting renderer 501 configured to render object-based light data 505 into luminaire control signals 515, at least in part, based on environment and actuator data 104. In this example, lighting renderer 501 is configured to output the luminaire control signals 515 to lamp controllers 103, which are configured to control luminaires 108. Luminaires 108 may include a single controllable light source, a group of controllable light sources (such as controllable light strips), or a combination thereof. In some examples, lighting renderer 501 may be configured to manage various types of light object metadata layers, examples of which are provided herein. According to some examples, lighting renderer 501 may be configured to render the luminaire actuator signals at least in part based on the viewer's perspective. If the viewer is in a living room that includes a television (TV) screen, in some examples, lighting renderer 501 may be configured to render the actuator signals relative to the TV screen. However, in virtual reality (VR) use cases, the lighting renderer 501 can be configured to render actuator signals relative to the position and orientation of the user's head. In some examples, the lighting renderer 501 can receive input from the playback environment (such as light sensor data corresponding to ambient light, camera data corresponding to the person's position or orientation, etc.) to enhance the rendering effect.

[0151] In some examples, the lighting renderer 501 is configured to receive object-based lighting data 505, which includes light objects and object-based lighting metadata indicating the expected lighting environment, as well as environment and lighting data 104 corresponding to other features of the luminaires 108 and the local playback environment. These features may include, but are not limited to, reflective surfaces, windows, uncontrollable light sources, shading features, etc. In this example, the local playback environment includes one or more speakers 109 and one or more display devices 510.

[0152] According to some examples, the lighting renderer 501 is configured to calculate, at least in part, how to activate various controllable luminaires 108 based on object-based lighting data 505 and environment and luminaire data 104. For example, environment and luminaire data 104 may indicate the geometric location of the luminaires 108 in the environment, luminaire type information, etc. In some examples, the lighting renderer 501 may be configured to determine, at least in part, which luminaires will be activated based on position metadata and size metadata associated with each light object, for example, by determining which luminaires are located within a volume of the playback environment, corresponding to the position and size of the light object at a specific time indicated by light object timestamp information. In this example, the lighting renderer 501 is configured to send luminaire control signals 515 to the luminaire controller 103 based on environment and luminaire data 104 and object-based lighting data 505. The luminaire control signals 515 may be sent via one or more transmission mechanisms, application programming interfaces (APIs), and protocols. For example, these protocols may include Hue API, LIFX API, DMX, Wi-Fi, Zigbee, Matter, Thread, Bluetooth Mesh, or other protocols.

[0153] In some examples, the lighting renderer 501 can be configured to determine the drive level of the lighting environment expected by the authors of the object-based light data 505 for each of one or more controllable light sources. According to some examples, the lighting renderer 501 can be configured to output a drive level to at least one of the controllable light sources.

[0154] According to some examples, the lighting renderer 501 can be configured to collapse one or more portions of a lighting map based on content metadata, user input (selection mode), lighting fixture limitations and / or configurations, other factors, or combinations thereof. For example, the lighting renderer 501 can be configured to render the same control signals to two or more different lights in the playback environment. In some such examples, the two or more lights can be positioned close to each other. For example, the two or more lights can be different lights with the same actuator, such as different bulbs in the same light bulb. Instead of calculating slightly different control signals for each bulb, the lighting renderer 501 can be configured to reduce computational overhead, improve rendering speed, etc., by rendering the same control signals to two or more different but closely spaced lights.

[0155] In some examples, the lighting renderer 501 can be configured to spatially upmix the object-based light data 505. For example, if the object-based light data 505 is generated for a single plane (such as a horizontal plane), in some cases, the lighting renderer 501 can be configured to project the light objects of the object-based light data 505 onto an upper hemisphere surface (e.g., above the user's actual or intended head position) to enhance the experience.

[0156] Based on some examples, the lighting renderer 501 can be configured to apply one or more thresholds, such as one or more spatial thresholds, one or more brightness thresholds, etc., when rendering actuator control signals to the light actuators of the playback environment. In some cases, such thresholds may prevent some light objects from activating some lights.

[0157] Light objects can be used for a variety of purposes, such as creating room atmosphere, providing spatial information about people or objects, enhancing special effects, creating greater interactivity and immersion, diverting the viewer's attention, emphasizing content, and so on. Content creators may use sensory object metadata types and / or attributes (typically applicable to various types of sensory objects) to express some of these purposes, such as object metadata indicating the location and size of a sensory object.

[0158] For example, the priority of sensor objects (including but not limited to light objects) can be indicated via sensor object priority metadata. In some such examples, sensor object priority metadata is considered when multiple sensor objects are simultaneously mapped to the same light fixture in the playback environment. This priority can be indicated via light priority metadata. In some examples, priority may not need to be indicated via metadata. For example, MS Renderer 001 might prioritize moving sensor objects (including but not limited to light objects) over stationary sensor objects.

[0159] A light object may trigger the activation of multiple lights, depending on its location and size, as well as the position of the light fixture within the playback environment. In some examples, when the size of a light object contains multiple lights, the renderer may apply one or more thresholds (such as one or more spatial thresholds or one or more brightness thresholds) to prevent the object from activating some of the contained lights.

[0160] Example of using lighting mapping In some implementations, a lighting map (an instance of an actuator map (AM) that includes a description of lighting in the playback environment) may be provided to the scene renderer 501. In some such examples, Figure 5 The environmental and lighting data shown may include a lighting map. According to some examples, the lighting map may be heterocentric, for example, indicating light attenuation based on absolute spatial coordinates; while in other examples, the lighting map may be egocentric, for example, projecting light onto a sphere representing the intended viewing position and orientation. In the case of a sphere, in some examples, the lighting map may be projected onto a two-dimensional (2D) surface, for example, to use a 2D image texture in processing. In any case, the lighting map should indicate the capabilities of the playback environment (e.g., a room) and the lighting settings. In some embodiments, the lighting map may not be directly related to the physical room characteristics, for example, if certain adjustments based on user preferences have been made.

[0161] In some examples, each luminaire or light in the playback environment may have a lighting map. According to some examples, the intensity of the light indicated by the lighting map may be inversely correlated with the distance to the center of the light, or may be approximately inversely correlated with the distance to the center of the light (e.g., within ±5%, ±10%, ±15%, ±20%, etc.). The intensity value of the lighting map can indicate the intensity or effect of a light object on a luminaire. For example, when a light object approaches a light bulb, the lighting renderer 501 can be configured to determine that the intensity of the light bulb will increase as the distance between the light object and the light bulb decreases. The lighting renderer 501 can be configured to determine the rate of this transition based at least in part on the light intensity indicated by the lighting map.

[0162] In this general rendering space, in some examples, the lightscape renderer 501 can be configured to calculate the light activation index for each light using a dot product multiplication between the light object and the light map, for example, as shown below:

[0163] In the aforementioned equation, Y represents the light activation index, LM represents the illumination map, and Obj represents the mapping of the light object. The light activation index indicates the relative light intensity of the actuator control signal output by the lighting renderer 501 based on the overlap between the light object and the light diffusion from the luminaire. In some examples, the lighting renderer 501 may use the maximum or nearest distance from the light object to the luminaire, or other geometric measures, as part of determining the light intensity. In some implementations, the lighting renderer 501 does not calculate the light activation index but may instead determine it by referring to a lookup table.

[0164] The lighting renderer 501 can repeat one of the above processes to determine the light activation metrics for all light objects and all controllable lights in the playback environment. Thresholding light objects that have a minimal impact on the lights can help reduce complexity. For example, if the effect of a light object would cause the light activation to fall below a certain threshold percentage (such as below 10%, below 5%, etc.), the lighting renderer 501 might ignore the effect of that light object.

[0165] Then, the lighting renderer 501 can use the generated light activation matrix Y, along with various other attributes such as the selected panning rule (indicated by the light object metadata or renderer configuration) or the priority of the light objects, to determine which objects are rendered by which lights and how. Rendering light objects into lighting control signals can involve: • Adjust the brightness of the light source based on the distance between the light source and the light fixture; • Mix the colors of multiple light objects rendered simultaneously (multiplexed) by a single light fixture; or • Change any of the above options based on the light object priority.

[0166] Mapping illuminator actuator This section describes the apparatus, systems, and methods for mapping and characterizing luminaires and the environments in which they are deployed. The process typically involves obtaining luminaire information, such as luminaire location and capability information, and obtaining environmental information corresponding to the environment in which the luminaires are deployed. Luminaire information may also include light direction information, indicating the direction(s) in which each luminaire emits light within the environment. Environmental information may include the location, orientation, and size of one or more environmental features, such as structural elements, furniture, or objects within the environment. Environmental information may include environmental feature color information, environmental feature reflectivity information, or both.

[0167] In some examples, users can manually input luminaire and environmental information. According to some examples, devices or systems can estimate luminaire and environmental information, for example, by using camera data (such as video feeds) and inertial measurement data from sensing devices, thereby obtaining data corresponding to the effect produced when the luminaire is actuated to illuminate the environment (e.g., a room). Detailed examples are provided below.

[0168] Some examples involve obtaining the Light Room Response (LRR) based on luminaire information and environmental information, and determining the Light Mapping (LM) based on the LRR. LRR and LM are instances of ARR and AM, respectively. LRR describes the joint response of the luminaire and the environment in which it is deployed.

[0169] Figure 6 The light room response (LRR) of a single luminaire is depicted using egocentric polar coordinates. Figure 6 The image used was generated using first-order reflection simulation and manually input data. Figure 6 The x, y, and z coordinates in the figure are defined as follows: -The origin point is at the user's head; - The direction to the right of the user is a positive x value; -The direction behind the user is a positive y-value; and - The z-value is positive in the direction above the user.

[0170] from Figure 6 As can be seen, the light is positioned below and in front of the user's head, and slightly to the left of the user's head. From the user's perspective, the light is located on the floor, shining upwards along the left wall of the room. Figure 6 The illustration shows the lighting field of the lamp in the room.

[0171] Video-based analysis and light mapping generation Light Response Rate (LRR) indicates the response of each light in an environment such as a room. In some examples, LRR can be estimated by manually inputting the location, characteristics, room dimensions, etc., of each light. However, some publicly available devices, methods, and systems provide an automated and more accurate light mapping process involving direct observation of the actual light within the room using video-based analytics. In addition to a more detailed characterization, such methods allow for compensation for subtle differences between the actual manufactured light and the specifications provided in the datasheet.

[0172] Figure 7 An example element of a light mapping system is shown. Similar to the other figures provided herein, Figure 7The types and quantities of elements shown are provided by way of example only. Other implementations may include more, fewer, and / or different types and quantities of elements. According to some examples, system 700 may be or may include one or more devices configured to perform at least some of the methods disclosed herein. In some examples, system 700 may include devices configured to perform at least some of the methods disclosed herein. Figure 1A One or more instances of the control system 110.

[0173] According to this example, the light mapping system 700 includes the following elements: Module 701 - System controller, which in this example includes a state machine; 702 - Calibration Signal Generator: A module configured to generate calibration signals to control the illumination of lamps 750A, 750B and 750C to support the positioning, mapping and characterization process; 703 - Lamp controller module, configured to control lamps 750A, 750B and 750C according to signal 721 from calibration signal generator 702; 704 - A sensing device that provides observations as input to the light mapping system 700. In some examples, the sensing device may be a mobile phone; 705 - Optical positioning and segmentation module, which is configured to determine the positioning of lights 750A, 750B and 750C in both video feed 723 and room 730; 706 - Room characterization module, which is configured to estimate parameters of room 730, such as (multiple) wall colors, (multiple) ceiling colors, (multiple) floor colors, (multiple) furniture colors, surface reflectivity, etc. 707 - Lamp Characterization Module, which is configured to estimate parameters of lamps 750A, 750B and 750C, including luminous sensitivity and color matching; 708 - Analysis and fusion module, which is configured to fuse the estimated room parameters 727 and the estimated lamp parameters 728; 709 - A positioning module configured to determine the position and orientation of the sensing device 704 within room 730. In this example, the positioning module 709 is configured to perform a simultaneous localization and mapping (SLAM) process based at least in part on camera data (video feed 723 in this example) and inertial measurement data from the sensing device 704; 710 - A camera system, including one or more camera devices located on sensing device 704. The one or more camera devices may include one or more visible light cameras, one or more infrared light cameras, one or more stereo cameras, one or more lidar devices, etc. 712 - An auxiliary sensor, which is part of sensing device 704 and is physically coupled to camera system 710 in this example. In this example, auxiliary sensor 712 is configured to acquire inertial measurement data, which may include gyroscope data, accelerometer data, or both. 730 - Environment (in this example, a room) where numerous lights can be controlled to create a flexible lighting rendering system; and 750A, 750B, and 750C – Multiple controllable lights in room 730. Although this example shows three controllable lights in room 730, other examples may include different numbers of controllable lights, such as 5, 8, 10, 15, 20, or more.

[0174] Signal 720 - Control signals sent by lamp controller 703 to lamps 750A, 750B and 750C; 721 - A calibration signal sent by the calibration signal generator to the lamp controller 703 for controlling lamps 750A, 750B and 750C; 723 - Multiple video feeds generated by camera 710 of sensing device 704; 724 - Sensor feed(s) generated by sensor 712 of sensing device 704; 725 - Data indicating the estimated position and orientation of sensing device 704 within room 730; 726 - Estimated lamp location and its segmentation mask in video feed; Estimated room parameters for room 727 - room 730; 728 - Estimated lamp parameters for lamps 750A, 750B and 750C; 729 - Jointly estimated room and lighting parameters, which may include the estimation results coherently converged to the solution by the analysis and fusion module 708; and 730 - Command from system controller 701 to control the generation of calibration signal 721.

[0175] In this example, the process of representing the room and light is iterative. For example... Figure 7As shown, this iterative process is indicated by sending the jointly estimated room and lamp parameters 729 back to the room characterization module 706, the lamp characterization module 707, and the system controller 701. The light sensed by the camera is the product of a light source reflecting off (multiple) objects. For example, consider a pixel in the camera feed corresponding to a wall illuminated by one of the lamps during a certain stage of the calibration measurement process. In this case, the lamp and the wall each have an infinite number of possible illumination and reflection spectra, which can produce the color sensed by the camera. For content-aware consumption calibrated by the system, the control system does not need to estimate the spectral response because typical humans are trichromatic and only require three digits (even fewer for those with color vision deficiencies) to represent and reproduce a color. Since each primary color of the lamp needs to be characterized separately, the color sensed by the camera still has ambiguity introduced during the calibration process. For example, without any prior information about the wall color, the room characterization module 706 can initially estimate the wall color to be blue based on the blue light reflected from the wall illuminated by the lamp. Simultaneously, the lamp characterization module 707 can initially estimate that the lamp has a low emissivity, or the room characterization module can similarly estimate that the wall has a low reflectivity. However, the room characterization module 706 and the lamp characterization module 707 can update the wall color estimate and lamp emissivity based on the different colors of light subsequently reflected from the wall illuminated by a specific lamp. Over time, the control system can use observations of multiple parts of the room and multiple lamps to jointly estimate the room characteristics and lamp characteristics.

[0176] Optical calibration signal / encoded transmission According to some examples, the calibration signal 121 generated by the calibration signal generator 102 may be or may include a set of color signals that perform the following functions: 1. Illuminate the room to provide data for the positioning module 709 to determine the position and orientation of the sensing device 704 within the room 730, for example, according to a SLAM process; 2. Identify each light; 3. Position each light; and 4. Estimate the illuminance of each light in the room.

[0177] These features may have different requirements for the calibration signal. In some examples, these features may appear at different stages of the mapping process, and in others, these stages may correspond to the numerical order in the list above.

[0178] According to some examples, the calibration signal 121 can be configured to cause relatively low-frequency light modulation, for example, to change the color, intensity, etc. of each lamp every 100 milliseconds (ms), every 200 ms, every 300 ms, every 400 ms, every 500 ms, etc. In such examples, the calibration signal 121 can cause the lamps to modulate at a rate compatible with the sampling rate of the camera performing the calibration.

[0179] In some alternative examples, the lights can be controlled by a simple on / off switch, for example, by sequentially turning each light in the room on and off, so that each controllable light in the room can be detected and located. Subsequently, in some examples, the calibration signal 121 may be altered to help estimate the light response in the room.

[0180] According to some examples, the calibration signal 121 can be configured to modulate the lamps in a way that allows for unique identification of each lamp. For example, each calibration signal 121 can be configured to perform on / off keying based on a unique binary sequence. After each lamp in the video feed 723 is detected, the task of identifying each lamp becomes the task of identifying the binary sequence, for which conventional estimators exist. The choice of a unique code set used as the calibration signal modulator (amplitude in the case of on / off keying) may vary depending on the specific implementation. Some coding families, such as Gold codes, can provide the best separation. Gold codes have a length of 2^32. N - 1 Where N is the order of the gold code, which is greater than 4 and preferably odd. For N = 5, a separation of 12 dB can be obtained between any two calibration signals obtained from this coding series. The length of these sequences is 31, which will be a compatible length for mapping applications for cameras calibrated at 30 frames per second (FPS).

[0181] Wall color The color of the walls plays a significant role in the color output of a light source. For example, projecting blue light onto a red wall will produce a purplish hue.

[0182] Bidirectional reflectivity measurement (BRDF) In this context, "bidirectional" refers to a reflectance measurement taken over a range of angles (e.g., over 180 degrees), rather than in a single direction. Such measurements are useful because the reflection from some surfaces in a playback environment may differ from that from another side or direction. The reflectance of structural elements in the environment (such as walls, floors, ceilings, etc.) will determine the number and directionality, or "directivity," of light sources. Especially when the light source is a spotlight, the reflectance or diffuser of the walls will affect the diffusion of light and the shape of the impulse response.

[0183] Spectral response A typical RGB camera sensor can be used to measure the combination of direct light and reflected light from structural elements, such as walls. However, capturing the spectral power distribution of the light source and the spectral reflectance of the structural element is beneficial. The spectral power distribution (SPD) of the light source describes the power magnitude of each wavelength in the visible spectrum. The spectral reflectance (SR) of the structural element describes how much light of each wavelength is reflected back.

[0184] Each person reacts differently to different wavelengths of light; this is known as a person's color matching function (CMF). These CMF responses are directly related to people's perception of color. Some implementations can be configured to uniquely render a scene based on the individual's CMF response, for example, as shown below:

[0185] In the preceding equations, the directionality of the light mapping (LM) is driven by the BRDF of the structuring element. The spectral response is determined by the product of the SPD and SR. The final rendered response will correspond to the human CMF.

[0186] lamp representation Every light source has a set of colors it can produce. In most additive color systems, this set of colors can be broken down into hue and luminance. However, some light sources may have additional settings / bulbs with higher luminance, thus forming non-additive color systems. One goal of generating light maps is to determine the relationship between the drive signal and the light output. Therefore, a comprehensive characterization of the light source is desirable. The light output can be a simple XYZ, or a more unique SPD, as previously described. The Encoded Emission section describes a method for efficiently characterizing a light source. Characterizing the diffusion or directionality of the light source is also important. Spotlights, light strips, or bulbs have unique effects on the viewer and the room.

[0187] Equipment obstruction In some cases, light may be partially blocked by objects such as furniture or other lamps. When working in polar coordinates, lights on a straight line can be combined into a single light source. If one or more light occlusions or "dead zones" exist from the observer's perspective, such occlusions should be included in the LM (Light Model). For example, in the LM, a dead zone can be represented by a low or zero value of the coordinates corresponding to the dead zone. The renderer can use information about dead zones, for example, to better position light objects.

[0188] Device color matching When generating lighting, it's crucial that all lights and displays produce similar colors. Making a single light fixture too bright or with a inconsistent hue can degrade the sensory experience. In some cases, it may be easier and / or more beneficial to "embed" color and intensity calibration into the light map and / or the luminaire itself. In the latter case, it's possible to specify a drive value that produces a certain color, rather than specifying a color and its associated drive value. In the case of a very simple LM, which can only utilize a certain percentage of the light object, a complete characterization of the light source will allow the control system to match the colors.

[0189] Figure 8 An example of a color volume is shown. The color within a color volume is indicated by the fill density of a specific area, with the lowest fill density corresponding to red and the highest fill density corresponding to purple. Color key 801 indicates the fill density corresponding to red, orange, yellow, green, blue, indigo, and violet (ROYGBIV). To ensure accurate matching of all colors, some implementations involve defining the color volumes of all lights, reflective surfaces, etc., in a room, for example... Figure 8 As shown. The control system can determine the overlapping volume of all lights, reflectors, etc., in the room. Not all light sources produce colors outside this overlapping volume. The mapping itself can be mathematical or based on a three-dimensional (3D) lookup table (LUT) that converts driving values ​​into hue / intensity. For color matching, using a perceptually uniform and uncorrelated space (such as ICTCp) is helpful, as... Figure 8 As shown.

[0190] External light source In many cases, one or more uncontrollable light sources may exist in the environment. Whether these uncontrollable light sources include lights from other rooms or ambient light from windows, they affect the accurate light mapping process. In some examples, uncontrollable light sources can be characterized as a base environment layer independent of the impulse response. According to some examples, the light object renderer can compensate for uncontrollable light sources by altering the light object rendering, for example, by increasing the drive signal of the light object within the area affected by the uncontrollable light source. In some examples, a light map (LM) can be the “baseline” light map obtained with all controllable lights off. In some such examples, future LM map captures may negate the effects of the baseline light map.

[0191] Baseline Environment In some cases, it may be preferred to maintain an ambient light baseline level in the playback environment. This ambient light baseline level can be characterized in a similar way to that of external light sources. However, in some implementations, the renderer can be configured to at least maintain this ambient light baseline level. In some such examples, the LM can indicate the baselines of controllable and uncontrollable lights separately.

[0192] Manual input and mixed systems Figure 9 An alternative method for generating LRR is described. In this example, the LRR is generated without using video analytics. In this example, environment descriptor data 901 and light descriptor data 902 are provided to simulator 903. According to this example, simulator 903 is... Figure 1A An example implementation of the control system 110 is configured to estimate LRR 904 based on environmental descriptor data 901 and lamp descriptor data 902. In some examples, the simulation may consider only the diffusion component of light reflection. For example, using the Lambert model, positioning can be simulated as follows: LRR of the j-th lamp:

[0193] ,in: - Represents the anti-incidence vector; - The D subscript indicates the diffusion component; - This represents the intensity of the light emitted by the j-th lamp. - Indicates the position of the j-th light fixture; - N represents the normal to the surface at that location; and - α and β This parameter represents the effect of attenuation caused by changing the distance between the luminaire and the positioning p.

[0194] In addition to modeling diffuse reflection, some examples involve modeling the specular component of light reflection to obtain more accurate results, especially at the smoother ends of the surface. This can be achieved by modeling the specular component separately and then simply adding it to the diffuse component simulated using methods such as those described above. One way to simulate specular reflection is to use the Blinn model, for example, as shown below:

[0195] ,in: B represents the normalized angle bisector of the reflection vector and the anti-incidence vector, and ; - Represents the reflection vector; - Represents the normalized anti-incidence vector; - This represents the normalized reflection vector; - Φ represents a parameter describing the mirror-like (gloss) finish of the surface; and - U indicates the user's location.

[0196] whenever At that time, they should all Set to zero.

[0197] The above expressions can be combined to obtain the comprehensive simulation ARR, as shown below:

[0198] The model shown here is not based on a physically accurate model. If sufficiently accurate prior information about the room's surfaces and lighting fixtures were available, then a more accurate physically based rendering could be used.

[0199] Environmental descriptor data 901 includes content referred to herein as "environmental information," which relates to the playback environment, including controllable luminaires. For example, environmental descriptor data 901 may include the location, orientation, and size of one or more environmental features, color information of environmental features, reflectivity information of environmental features, or combinations thereof. According to this example, lamp descriptor data 902 includes at least information about controllable luminaires in the playback environment. For example, lamp descriptor data 902 may include luminaire positioning data, lamp parameter data (including but not limited to light color data), light directivity data (indicating the direction(s) of light provided by one or more lamps of the luminaire), etc. In some cases, at least a portion of lamp descriptor data 902 may be derived from published specifications regarding the capabilities, parameters, etc., of one or more commercially available lighting products. In some alternative examples, environmental descriptor data 901 may include luminaire positioning data, light directivity data, or both.

[0200] In some examples, environment descriptor data 901, lamp descriptor data 902, or both can be manually entered. For example, control system 110 can control a display (not shown) to provide a graphical user interface (GUI) through which a user can interact to enter environment descriptor data 901, lamp descriptor data 902, or both.

[0201] However, in some “hybrid” examples, at least a portion of the environment descriptor data 901, the lamp descriptor data 902, or both can be generated automatically. For example, in some implementations, the location, orientation, and size of one or more environmental features, the location of one or more lamps, etc., can be determined based on a simultaneous localization and mapping (SLAM) process, rather than being manually entered.

[0202] Figure 10 It shows what can be provided Figure 9 Examples of simulator information. Figure 10The positioning and illumination direction of the luminaires within environment 1000 are depicted. In these examples, a single light is depicted as a cone, with the apex of the cone indicating its positioning and the axis of the cone indicating its illumination direction. For example, downlight 1005a is located on the ceiling of environment 1000. The illumination direction of downlight 1005a is indicated by the direction of arrow 1004. In this example, environment 1000 includes ceiling downlight 1005, television (TV) gradient strip light 1010, and floor light 1015.

[0203] Enhance LRR to produce light mapping The LRR indicator's response within the environment. In some examples, the LRR can be enhanced so that the enhanced LRR can also map areas in the environment where light cannot reach, such as behind a TV or sofa, and use this as an activation mechanism. Such enhancement is an example that can be referred to in this paper as "filling the gap" in the LRR. Some examples may involve other modifications to the LRR. In this paper, the resulting enhanced LRR can be referred to as light mapping (LM).

[0204] Figure 11A and Figure 11B An example of the enhanced LRR is shown. Figure 11A and Figure 11B The enhanced LRR (which can be called LM) is shown, corresponding to the two parts of the TV gradient light strip. Figure 11A The LM 1110a is shown, produced by enhancing the TV gradient light strip LRR in the positive x-direction. Figure 11B The diagram illustrates an LM1110b produced by enhancing the TV gradient light strip LRR in the positive z-direction. In both examples, the enhanced LRR now includes a mapping from behind the TV. If a light object is placed behind the TV, using an LM such as LM 1110a or LM 1110b will allow the lighting renderer 101 to project that light object from the spatial domain onto the light, thereby exciting the entire TV gradient strip. Therefore, LM 1110a and LM 1110b provide a natural extension of the TV gradient light strip LRR, allowing the lighting renderer 101 to map the entire spatial domain back to the light in the environment 1100.

[0205] Regularization of light mapping and repulsion generation As described elsewhere in this document, in some examples, the lighting renderer 101 uses a reversed LM to project light objects in the spatial domain onto the lights of the playback environment. In some cases, even with enhanced or otherwise modified LRRs, the resulting LM may still include one or more ill-posed spatial coordinates. Some such examples may occur where two lights have strongly responsive and overlapping locations, but these lights also cover large, non-intersecting areas. If light objects are placed in overlapping areas, the lighting renderer 101 may cause a large portion of the environment to be illuminated, and if another light object is located in a large, non-intersecting area of ​​the lights, it may reduce the ability to render that other light object.

[0206] Some implementations involve mitigating the effects of ill-posed spatial coordinates. Some examples involve regularization techniques that can minimize such effects. Some such examples involve applying functions of the SoftMax type, which may only be applied to the overlapping regions in the preceding examples.

[0207] Alternatively or additionally, some examples involve implementing one or more repulsion functions configured to deflect the trajectory of a light object (or other sensory object) away from one or more ill-posed spatial coordinates. Such repulsion functions can be applied to the light object itself in the lighting renderer 101 to prevent the light object from entering regions that include ill-posed spatial coordinates. In some such examples, repulsion metadata can be appended to the LM, and the lighting renderer 101 can apply the corresponding repulsion function. According to some examples, the control system can be configured to determine regions that include ill-posed spatial coordinates and generate or select corresponding repulsion functions for regions where these ill-posed spatial coordinates occur. For example, the process of determining regions that include ill-posed spatial coordinates can involve analyzing synthetic LRR generated by simulation, or analyzing LRR generated by analyzing video feeds, for example, as referenced herein. Figure 7 As stated above.

[0208] Figure 12 This is a flowchart outlining an example of a method that can be performed by a device or system such as the apparatus or system disclosed herein. As with other methods described herein, the blocks of method 1200 need not be performed in the indicated order. In some embodiments, one or more blocks of method 1200 may be performed simultaneously. Furthermore, some embodiments of method 1200 may include more or fewer blocks than those shown and / or described. The blocks of method 1200 may be performed by one or more devices, which may be (or may include) a control system (such as those described above). Figure 1A One or more instances of the control system 110 shown and described.

[0209] In this example, box 1205 relates to obtaining controllable actuator positioning information for a set of controllable actuators in the environment by a control system. In this example, the set of controllable actuators includes one or more luminaires, one or more haptic devices, one or more airflow control devices, or combinations thereof.

[0210] According to this example, block 1210 relates to the controllable actuator capability information of each controllable actuator in the set of controllable actuators obtained by the control system. In some cases, the controllable actuator capability information may include actuation type (e.g., light, tactile, airflow, etc.). For luminaires, the controllable actuator capability information may indicate the light color(s), light intensity(s), etc.(s) that can be emitted. In some examples, method 1200 may involve obtaining controllable actuator actuation direction information. For luminaires, the controllable actuator direction information may indicate the directions(s) in which the luminaire is currently configured to transmit light.

[0211] In this example, box 1215 relates to environmental information corresponding to the environment obtained by the control system. In some cases, this environmental information may include environmental feature color information, environmental feature reflectivity information, or a combination thereof. According to some examples, the environmental information may at least include the location, orientation, and size of one or more environmental features. In some cases, the one or more environmental features may include one or more structural elements of the environment, such as one or more walls, ceilings, floors, or combinations thereof. In some examples, the one or more environmental features may include one or more furniture items. The environmental information may include furniture location information.

[0212] According to this example, block 1220 relates to the generation of an actuator-room response (ARR) by the control system based at least in part on the controllable actuator positioning information, the controllable actuator capability information, and the environmental information, which summarizes the environment's response to the activation of the set of controllable actuators.

[0213] In some examples, one or more of boxes 1205 may involve a manual input process. The manual input process may include references... Figure 9 One or more processes are described. In some such examples, obtaining the controllable actuator positioning information may involve receiving controllable actuator capability information via an interface system (such as a user interface system or a network system).

[0214] However, according to some examples, one or more of boxes 1205 may involve automated processes, such as those mentioned above. Figure 7One or more processes are described. For example, obtaining the controllable actuator positioning information in block 1205 may involve the control system receiving camera data from one or more cameras, and the control system determining the controllable actuator positioning information based at least in part on the camera data. The camera data may include images of a set of controllable actuators. The camera data may include optical images, depth images, or combinations thereof.

[0215] In some examples, method 1200 may involve the control system determining controllable actuator actuation direction information based at least in part on the camera data. This controllable actuator actuation direction information may include light direction information, air movement direction information, or both.

[0216] According to some examples, method 1200 may involve obtaining inertial measurement data by the control system. The inertial measurement data may include gyroscope data, accelerometer data, or both. In some examples, the controllable actuator positioning information may be determined based on a simultaneous localization and mapping (SLAM) process, at least in part, based on the camera data and the inertial measurement data.

[0217] In some examples, the controllable actuator may include a controllable luminaire. In some such examples, method 1200 may also involve a control system sending a calibration signal to the controllable luminaire while one or more cameras are acquiring camera data. The calibration signal may cause the controllable luminaire to modulate light intensity, light color, or both. In some such examples, obtaining the controllable actuator capability information and obtaining the environmental information may both be based at least in part on the camera data and the inertial measurement data. According to some examples, the calibration signal may control the controllable luminaire to emit light according to a code or pattern that enables the control system to distinguish the lighting effects produced by each of the controllable luminaires. In some examples, the calibration signal may be or may include a sequence having bounded cross-correlation within a set, such as a Gold sequence. According to some examples, method 1200 may involve color matching of the light emitted by two or more of the controllable luminaires.

[0218] In some examples, obtaining the controllable actuator capability information may involve a lamp characterization process, which may or may involve determining the characteristics of lamps in the environment. According to some such examples, method 1200 may involve a light localization and segmentation process based at least in part on the camera data. The lamp characterization process may be based at least in part on the light localization and segmentation process. For example, the light localization and segmentation process may involve analyzing the pixels of the camera data and determining which pixels correspond to individual lamps in the environment.

[0219] According to some examples, obtaining this environmental information may involve an environmental characterization process that determines characteristic color information and characteristic reflectivity information of the environment. In some examples, obtaining the controllable actuator capability information and obtaining this environmental information may involve an iterative process that determines the characteristics of the lights in the environment, determines characteristic color information of the environment, and determines characteristic reflectivity information of the environment.

[0220] In some examples, method 1200 may involve modifying the ARR by the control system to produce an actuator map (AM). This modification may include regularization, filling gaps in the ARR by increasing one or more volumes of the environment affected by actuator responses, reducing one or more overlapping volumes of the environment affected by multiple actuator responses, or a combination thereof. The AM may indicate the effect of each controllable actuator on the environment. The AM may be mathematically applicable to inversion. The AM may allow object-based sensory data to be rendered as actuator commands. In some examples, the ARR may be represented as a matrix, and the AM may be represented as a modified version of that matrix. The matrix may include values ​​corresponding to actuator indices, executed commands, and spatial coordinates.

[0221] According to some examples, method 1200 may involve the control system identifying one or more ill-posed spatial coordinates of the AM. Some such examples may involve the control system generating one or more repulsion functions configured to deviate the trajectory of the sensory object from the one or more ill-posed spatial coordinates. Some such examples may involve storing or providing the one or more repulsion functions along with the AM.

[0222] The foregoing description illustrates various embodiments of this disclosure and examples of how aspects of this disclosure may be implemented. The foregoing examples and embodiments should not be considered as limited embodiments, but are presented to illustrate the flexibility and advantages of this disclosure as defined by the appended claims. Other arrangements, embodiments, implementations, and equivalents will be apparent to those skilled in the art based on the foregoing disclosure and the appended claims, and may be employed without departing from the spirit and scope of this disclosure as defined by the claims.

Claims

1. A method comprising: The control system obtains controllable actuator positioning information of a set of one or more controllable actuators in the environment, wherein the one or more controllable actuators include one or more lamps, one or more tactile sensors, one or more airflow control devices, or combinations thereof; The control system obtains controllable actuator capability information for each controllable actuator in the set of one or more controllable actuators; The control system obtains environmental information corresponding to the environment. as well as The control system generates an actuator-room response (ARR) based at least in part on the controllable actuator positioning information, the controllable actuator capability information, and the environmental information. The actuator-room response summarizes the environment's response to one or more activations of the set of one or more controllable actuators.

2. The method as described in claim 1, wherein, The environmental information includes at least one or more locations, one or more orientations, and one or more dimensions of one or more environmental features, wherein the one or more environmental features include one or more structural elements of the environment.

3. The method as described in claim 2, wherein, The one or more structural elements include one or more walls, ceilings, floors, or combinations thereof.

4. The method as claimed in claim 2 or claim 3, wherein, The one or more environmental features include one or more furniture items, and the environmental information includes furniture location information.

5. The method according to any one of claims 2 to 4, wherein, The environmental information includes environmental characteristic color information, environmental characteristic reflectance information, or a combination thereof.

6. The method according to any one of claims 1 to 5, wherein, Obtaining the positioning information of the controllable actuator involves: The control system receives camera data from one or more cameras, the camera data including one or more images of the set of one or more controllable actuators; as well as The control system determines the positioning information of the controllable actuator based at least in part on the camera data.

7. The method of claim 6, further comprising determining actuation direction information of the controllable actuator by the control system based at least in part on the camera data, and wherein, The controllable actuator actuation direction information includes light direction information, air movement direction information, or both.

8. The method of claim 6 or claim 7, wherein, The camera data includes one or more optical images, one or more depth images, or a combination thereof.

9. The method of any one of claims 6 to 8, further comprising obtaining inertial measurement data by the control system, wherein, The inertial measurement data includes gyroscope data, accelerometer data, or both, and wherein the controllable actuator positioning information is determined based on a simultaneous localization and mapping (SLAM) process based at least in part on the camera data and the inertial measurement data.

10. The method according to any one of claims 1 to 9, wherein, The one or more controllable actuators include one or more controllable luminaires, and the method further includes the control system sending one or more calibration signals to the one or more controllable luminaires while the one or more cameras are acquiring the camera data, the one or more calibration signals causing the one or more controllable luminaires to modulate light intensity, light color, or both, and wherein obtaining the controllable actuator capability information and obtaining the environmental information are both based at least in part on the camera data and the inertial measurement data.

11. The method of claim 10, wherein, The one or more calibration signals control the one or more controllable luminaires to emit light according to codes or patterns that enable the control system to distinguish one or more lighting effects produced by each of the one or more controllable luminaires.

12. The method of claim 10 or claim 11, wherein, The one or more calibration signals comprise sequences that have bounded cross-correlation within the set.

13. The method of any one of claims 10 to 12, further comprising color matching of the light emitted by two or more of the controllable luminaires.

14. The method according to any one of claims 10 to 13, wherein, Obtaining the controllable actuator capability information involves a lamp characterization process that determines the characteristics of one or more lamps within the environment.

15. The method of claim 14, further comprising at least in part a light localization and segmentation process based on the camera data, wherein, The lamp characterization process is based at least in part on the light localization and segmentation process.

16. The method of claim 15, wherein, The light localization and segmentation process involves analyzing the pixels of the camera data and determining which pixels correspond to various lights in the environment.

17. The method according to any one of claims 10 to 16, wherein, Obtaining the environmental information involves an environmental characterization process that determines environmental characteristic color information and environmental characteristic reflectance information.

18. The method according to any one of claims 10 to 17, wherein, Obtaining the controllable actuator capability information and obtaining the environmental information involves an iterative process of determining one or more characteristics of the lights in the environment, determining environmental characteristic color information, and determining environmental characteristic reflectivity information.

19. The method according to any one of claims 1 to 18, wherein, Obtaining the controllable actuator positioning information involves receiving controllable actuator capability information via an interface system.

20. The method of claim 19, wherein, The interface system includes a user interface system.

21. The method of any one of claims 1 to 20, further comprising modifying the ARR by the control system to generate an actuator map (AM).

22. The method of claim 21, wherein, The modifications include regularization, filling gaps in the ARR by increasing one or more volumes of the environment affected by actuator responses, reducing one or more overlapping volumes of the environment affected by multiple actuator responses, or combinations thereof.

23. The method of claim 21 or claim 22, wherein, The AM indicates the effect of each controllable actuator on the environment.

24. The method according to any one of claims 21 to 23, wherein, The AM is mathematically applicable to inversion.

25. The method according to any one of claims 21 to 24, wherein, The AM allows object-based sensory data to be rendered as actuator commands.

26. The method of any one of claims 21 to 25, further comprising: The control system identifies one or more ill-posed spatial coordinates of the AM; The control system generates one or more repulsion functions, which are configured to cause the trajectory of the sensory object to deviate from the one or more ill-posed spatial coordinates. as well as The one or more rejection functions are stored or provided together with the AM.

27. The method according to any one of claims 21 to 26, wherein, The ARR is represented as a matrix, and the AM is represented as a modified version of the matrix.

28. The method of claim 27, wherein, The matrix includes values ​​corresponding to one or more actuator indices, one or more execution commands, and spatial coordinates.

29. An apparatus configured to perform the method as claimed in any one of claims 1 to 28.

30. A system configured to perform the method as claimed in any one of claims 1 to 28.

31. One or more non-transitory computer-readable media having instructions stored thereon for controlling one or more devices to perform the method as described in any one of claims 1 to 28.

Images