Multi-sensory (MS) spatial mapping and characterization for ms rendering

Abstract

Some methods may involve obtaining controllable actuator positioning information for a set of one or more controllable actuators in an environment, the controllable actuators including one or more light fixtures, one or more tactile sensors, one or more airflow control devices, or a combination thereof. Some methods may involve obtaining controllable actuator capability information for each controllable actuator in the set of controllable actuators, and obtaining environment information corresponding to the environment. Some methods may involve generating an actuator-room response (ARR) based at least in part on the controllable actuator positioning information, the controllable actuator capability information, and the environment information, the actuator-room response summarizing a response of the environment to controllable actuator activation. Some methods may involve modifying the ARR to generate an actuator map (AM). The modification may involve regularizing, filling gaps in the ARR, reducing one or more overlapping volumes of the environment affected by multiple actuator responses, or a combination thereof.

Description

(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) (19) World Intellectual Property Organization International Bureau (43) International Publication Date 23 January 2025 (23.01.2025) (10) International Publication Number WO 2025 / 019443 A1 WIPO PCT (51) International Patent Classification: G06F 3 / 16 (2006.01) H05B 47 / 105 (2020.01) G06F 3 / 01 (2006.01) (21) International Application Number: PCT / US2024 / 038079 (22) International Filing Date: 15 July 2024 (15.07.2024) (25) Filing Language: English 94103 (US). PYTLARZ, Jaclyn Anne; c / o Dolby Labo- ratories, Inc., 1275 Market Street, San Francisco, Califor- nia 94103 (US). GUNAWAN, David; c / o Dolby Labora- tories, Inc., 1275 Market Street, San Francisco, California 94103 (US). HO, Yin-Lee; c / o Dolby Laboratories, Inc., 1275 Market Street, San Francisco, California 94103 (US). DYONISIO, Carlos Eduardo Medaglia; Lot 3 Farm Rd, Angus, New South Wales 2765 (AU). ATKINS, Robin; c / o Dolby Laboratories, Inc., 1275Market Street, San Francis- co, California 94103 (US) (26) Publication Language: English (30) Priority Data: 63 / 514,105 63 / 669,236 17 July 2023 (17.07,2023) US (74) Agent: PURTILL, Elizabeth et al.; DOLBY LABORA- TORIES, INC., Intellectual Property Group, 1275 Market Street, San Francisco, California 94103 (US). 10 July 2024 (10.07.2024) US (71) Applicant: DOLBY LABORATORIES LICENSING CORPORATION [US / US]; 1275 Market Street, San Fran- cisco, California 94103 (US). (72) Inventors: SOUTHWELL, Benjamin; c / o Dolby Labora- tories, Inc., 1275 Market Street, San Francisco, California (81) Designated States (unless otherwise indicated, for every kind of national protection available): AE, AG, AL, AM, AO, AT, AU, AZ, BA, BB, BG, BH, BN, BR, BW., BY, BZ, CA, CH, CL, CN, CO. CR, CU, CV. CZ, DE, DJ, DK, DM. DO, DZ, EC, EE, EG, ES, FI, GB, GD, GE, GH, GM, GT, HN, HR, HU, ID, IL, IN, IQ, IR, IS, IT, JM, JO, JP, KE, KG, (54) Title: MULTI-SENSORY (MS) SPATIAL MAPPING AND CHARACTERIZATION FOR MSRENDERING 005 OBJECT-BASED SENSORY DATA PROJECTION MODULE 450 001 MULTI-SENSORY RENDERER ACTUATOR MAP (AM) 423 ACTUATOR ACTIVATION MATRIX (AAM) 440 Figure 4А ENVIRONMENT AND ACTUATOR DATA -004 MIXING MODULE 451 452 RENDERER CONFIGURATION DATA 310 ACTUATOR CONTROL SIGNALS (57) Abstract: Some methods may involve obtaining controllable actuator location information for a set of one or more controllable actuators in an environment, the controllable actuators including one or more light fixtures, one or more haptic sensors, one or more ☑ airflow control devices, or combinations thereof. Some methods may involve obtaining controllable actuator capability information ☑ for each controllable actuator of the set of controllable actuators and obtaining environment information corresponding to the environ- ment. Some methods may involve generating, based at least in part on the controllable actuator location information, the controllable actuator capability information, and the environmentinformation, an actuator-room response (ARR) that summarizes responses of the environment to controllable actuator activations. Some methods may involve modifying the ARR to produce an actuator map (AM). The modifying may involve regularization, filling gaps in the ARR, reducing one or more overlapping volumes of the environment affected by multiple actuator responses, or combinations thereof. W O 2 0 2 5 / 0 1 9 4 4 3 [Continued on next page] (84) WO 2025 / 019443 A1 KH, KN, KP, KR, KW, KZ, LA, LC, LK, LR, LS, LU, LY, MA, MD, MG, MK, MN, MU, MW, MX, MY, MZ, NA, NG, NI, NO, NZ, OM, PA, PE, PG, PH, PL, PT, QA, RO, RS, RU, RW, SA, SC, SD, SE, SG, SK, SL, ST, SV, SY, TH, TJ, TM, TN, TR, TT, TZ, UA, UG, US, UZ, VC, VN, WS, ZA, ZM, ZW. Designated States (unless otherwise indicated, for every kind of regional protection available): ARIPO (BW, CV, GH, GM, KE, LR, LS, MW, MZ, NA, RW, SC, SD, SL, ST, SZ, TZ, UG, ZM, ZW), Eurasian (AM, AZ, BY, KG, KZ, RU, TJ, TM), European (AL, AT, BE, BG, CH, CY,CZ, DE, DK, EE, ES, FI, FR, GB, GR, HR, HU, IE, IS, IT, LT, LU, LV, MC, ME, MK, MT, NL, NO, PL, PT, RO, RS, SE, SI, SK, SM, TR), OAPI (BF, BJ, CF, CG, CI, CM, GA, GN, GQ, GW, KM, ML, MR, NE, SN, TD, TG). Published: with international search report (Art. 21(3)) (19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202480059549.X (22) Application Date 2024.07.15 (30) Priority Data 63 / 514,105 2023.07.17 US 63 / 669,236 2024.07.10 US (85) PCT International Application Enters National Phase Date 2026.03.17 (86) PCT International Application Application Data PCT / US2024 / 038079 2024.07.15 (87) PCT International Application Publication Data WO2025 / 019443 EN 2025.01.23 (71) Applicant Dolby Laboratories Inc. Address California, USA (72) Inventors B. Southwell J.A. Paratz D. Gunnarwan YL. Ho C.E.M. Dionysio R. Atkins (74) Patent Agency Beijing Han Kun Law Offices 11602 Patent Attorneys Wei Xiaowei Wu Lili (51) Int.Cl. G06F 3 / 16 (2006.01) G06F 3 / 01(2006.01) H05B 47 / 105(2006.01) (54) Invention Title Multi-sensory (MS) Spatial Mapping and Characterization for MS Rendering (57) Abstract Some methods may involve obtaining controllable actuator positioning information of a set of one or more controllable actuators in an environment, including one or more luminaires, one or more tactile sensors, one or more airflow control devices, or combinations thereof. Some methods may involve obtaining controllable actuator capability information for each controllable actuator in the set of controllable actuators, and obtaining environmental information corresponding to the environment. Some methods may involve generating an actuator-room response (ARR) at least in part based on the controllable actuator positioning information, the controllable actuator capability information, and the environmental information, which summarizes the environment.The environment responds to the activation of a controllable actuator. Some methods may involve modifying the ARR to produce an actuator map (AM). This modification may involve regularization, filling gaps in the ARR, reducing one or more overlapping volumes of the environment affected by multiple actuator responses, or a combination thereof. Claims 3 pages, Description 33 pages, Drawings 13 pages, CN 121866536 A 2026.04.14 CN 1 21 86 65 36 A 1. A method comprising: obtaining controllable actuator positioning information of a set of one or more controllable actuators in an environment by a control system, said one or more controllable actuators including one or more luminaires, one or more tactile sensors, one or more airflow control devices or combinations thereof; obtaining controllable actuator capability information of each controllable actuator in the set of one or more controllable actuators by the control system; obtaining environmental information corresponding to said environment by the control system; and generating an actuator-room response (ARR) by the control system at least in part based on the controllable actuator positioning information, the controllable actuator capability information and the environmental information, said actuator-room response summarizing the response of said environment to one or more activations of said one or more controllable actuators. 2. The method of claim 1, wherein the environmental information includes at least one or more locations, one or more orientations, and one or more dimensions of one or more environmental features, and wherein the one or more environmental features include one or more structural elements of the environment. 3. The method of claim 2, wherein the one or more structural elements include one or more walls, ceilings, floors, or combinations thereof. 4. The method of claim 2 or claim 3, wherein the one or more environmental features include one or more furniture items, and wherein the environmental information includes furniture location information. 5. The method of any one of claims 2 to 4, wherein the environmental information includes environmental feature color information, environmental feature reflectivity information, or combinations thereof. 6. The method of any one of claims 1 to 5, wherein obtaining the controllable actuator positioning information involves: receiving camera data from one or more cameras, the camera data including one or more images of a set of one or more controllable actuators, by the control system; and determining the controllable actuator positioning information by the control system based at least in part on the camera data. 7. The method of claim 6, further comprising determining controllable actuator actuation direction information by the control system based at least in part on the camera data, wherein the controllable actuator actuation direction information includes light direction information, air movement direction information, or both. 8. The method of claim 6 or claim 7, wherein the camera data includes one or more optical images.9. The method of any one of claims 6 to 8, further comprising obtaining inertial measurement data by the control system, wherein the inertial measurement data includes gyroscope data, accelerometer data, or both, and wherein the controllable actuator positioning information is determined based on a simultaneous localization and mapping (SLAM) process based at least in part on the camera data and the inertial measurement data. 10. The method of any one of claims 1 to 9, wherein the one or more controllable actuators include one or more controllable luminaires, the method further comprising sending one or more calibration signals to the one or more controllable luminaires by the control system while the one or more cameras are acquiring the camera data, the one or more calibration signals causing the one or more controllable luminaires to modulate light intensity, light color, or both, and wherein obtaining the controllable actuator capability information and obtaining the environmental information are both based at least in part on the camera data and the inertial measurement data. 11. The method of claim 10, wherein the one or more calibration signals control the one or more controllable luminaires to emit light according to a code or pattern that enables the control system to distinguish one or more lighting effects produced by each of the one or more controllable luminaires. 12. The method of claim 10 or claim 11, wherein the one or more calibration signals comprise a sequence having bounded cross-correlation within a set. 13. The method of any one of claims 10 to 12, further comprising color matching of the light emitted by two or more of the controllable luminaires. 14. The method of any one of claims 10 to 13, wherein obtaining the controllable actuator capability information involves a lamp characterization process that determines the characteristics of one or more lamps within the environment. 15. The method of claim 14, further comprising a light localization and segmentation process at least partially based on the camera data, wherein the lamp characterization process is at least partially based on the light localization and segmentation process. 16. The method of claim 15, wherein the light localization and segmentation process involves analyzing pixels of the camera data and determining which pixels correspond to individual lamps in the environment. 17. The method of any one of claims 10 to 16, wherein obtaining the environmental information involves an environmental characterization process of determining environmental characteristic color information and determining environmental characteristic reflectivity information. 18. The method of any one of claims 10 to 17, wherein obtaining the controllable actuator capability information and obtaining the environmental information involves an iterative process of determining one or more characteristics of lamps within the environment, determining environmental characteristic color information, and determining environmental characteristic reflectivity information.19. The method of any one of claims 1 to 18, wherein obtaining the controllable actuator positioning information involves receiving controllable actuator capability information via an interface system. 20. The method of claim 19, wherein the interface system comprises a user interface system. 21. The method of any one of claims 1 to 20, further comprising modifying the ARR by the control system to generate an actuator mapping (AM). 22. The method of claim 21, wherein the modification comprises regularization, filling gaps in the ARR by increasing one or more volumes of the environment affected by actuator responses, reducing one or more overlapping volumes of the environment affected by multiple actuator responses, or a combination thereof. 23. The method of claim 21 or claim 22, wherein the AM indicates the effect of each controllable actuator on the environment. 24. The method of any one of claims 21 to 23, wherein the AM is mathematically adapted for inversion. 25. The method of any one of claims 21 to 24, wherein the AM allows object-based sensory data to be rendered as actuator commands. 26. The method of any one of claims 21 to 25, further comprising: identifying one or more ill-posed spatial coordinates of the AM by the control system; generating one or more repulsion functions by the control system, the one or more repulsion functions being configured to deviate the trajectory of the sensory object from the one or more ill-posed spatial coordinates; and storing or providing the one or more repulsion functions together with the AM. 27. The method of any one of claims 21 to 26, wherein the ARR is represented as a matrix, and the AM is represented as a modified version of the matrix. 28. The method of claim 27, wherein the matrix includes values ​​corresponding to one or more actuator indices, one or more execution commands, and spatial coordinates. Claims 2 / 3 Page 3 CN 121866536 A 29. An apparatus configured to perform the method of any one of claims 1 to 28. 30. A system configured to perform the method of any one of claims 1 to 28. 31. One or more non-transitory computer-readable media storing instructions thereon for controlling one or more devices to perform the method of any one of claims 1 to 28. Claims 3 / 3 Page 4 CN 121866536 A Multi-sensory (MS) spatial mapping and representation for MS rendering Cross-reference to related applications

[0001] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 669,236, filed July 10, 2024 and U.S. Provisional Application No. 63 / 514,105, filed July 17, 2023, each of which isThe entire text of the cited document is incorporated herein by reference. Technical Field

[0002] This disclosure relates to providing multi-sensory (MS) experiences, and more specifically to mapping and characterizing actuators and actuator deployment environments. Background Art

[0003] Unless otherwise indicated herein, the methods described in this section are not prior art to the claims of this application and are not admitted to be prior art by virtue of their inclusion in this section.

[0004] Media content delivery typically focuses on audio and screen-based visual experiences. The delivery of multi-sensory content has been limited due to the customized nature of actuation. For example, illuminators are widely used as an expression of the art and function of concerts. However, each device is specifically designed for a particular set of illuminators. Delivering the lighting design outside of that set of luminaires to which the system design is targeted is often not feasible. Other systems attempting to provide a broader light experience only extend screen visuals through algorithms, but are not specifically created. Haptic content is designed for a specific haptic device. If another device (such as a game controller, mobile phone, or even a different brand of haptic device) is used, it is not possible to translate the creative intent of the content to a different actuator.

[0005] At least some aspects of this disclosure can be implemented via methods such as audio processing methods. In some cases, these methods can be implemented at least in part by control systems such as those disclosed herein.

[0006] Some methods may involve obtaining controllable actuator positioning information of a set of one or more controllable actuators in an environment by a control system. The one or more controllable actuators may include one or more luminaires, one or more tactile sensors, one or more airflow control devices, or combinations thereof. Some methods may involve obtaining controllable actuator capability information of each controllable actuator in the set of one or more controllable actuators by the control system. Some methods may involve obtaining environmental information corresponding to the environment by the control system. Some methods may involve generating an actuator-room response (ARR) by the control system based at least in part on the controllable actuator positioning information, the controllable actuator capability information, and the environmental information, the actuator-room response summarizing the environment's response to one or more activations of the set of one or more controllable actuators.

[0007] In some examples, the environmental information may include at least one or more locations, one or more orientations, and one or more dimensions of one or more environmental features. In some examples, the one or more environmental features may include one or more structural elements of the environment. In some examples, the one or more structural elements may include one or more walls, ceilings, floors, or combinations thereof. In some examples, the one or more environmental features may include one or more furniture items. In some examples, the environmental information may include furniture location information. According to some examples, the environmental information specification is 1 / 33 pages 5 CN 121866536A may include environmental feature color information, environmental feature reflectivity information, or a combination thereof.

[0008] In some examples, obtaining the controllable actuator positioning information may involve the control system receiving camera data from one or more cameras. In some examples, the camera data may include one or more images of the set of one or more controllable actuators. In some examples, obtaining the controllable actuator positioning information may involve the control system determining the controllable actuator positioning information based at least in part on the camera data. Some methods may involve the control system determining controllable actuator actuation direction information based at least in part on the camera data. In some examples, the controllable actuator actuation direction information may include light direction information, air movement direction information, or both.

[0009] According to some examples, the camera data may include one or more optical images, one or more depth images, or a combination thereof. Some methods may involve the control system obtaining inertial measurement data. In some examples, the inertial measurement data may include gyroscope data, accelerometer data, or both. In some examples, the controllable actuator positioning information can be determined based on a Simultaneous Localization and Mapping (SLAM) process, at least in part, based on the camera data and the inertial measurement data.

[0010] In some examples, the one or more controllable actuators may include one or more controllable luminaires. Some methods may involve the control system sending one or more calibration signals to the one or more controllable luminaires while the one or more cameras are acquiring the camera data. The one or more calibration signals may modulate the light intensity, light color, or both of the one or more controllable luminaires. Obtaining the controllable actuator capability information and obtaining the environmental information may be based at least in part on the camera data and the inertial measurement data. In some examples, the one or more calibration signals may be or may include a sequence having bounded cross-correlation within a set.

[0011] According to some examples, the one or more calibration signals may control the one or more controllable luminaires to emit light according to a code or pattern that enables the control system to distinguish one or more lighting effects produced by each of the one or more controllable luminaires. Some methods may involve color matching of the light emitted by two or more of the controllable luminaires.

[0012] According to some examples, obtaining the controllable actuator capability information may involve a lamp characterization process that determines the characteristics of one or more lamps in the environment. Some methods may involve a light localization and segmentation process that is at least partially based on the camera data. The lamp characterization process may be at least partially based on the light localization and segmentation process. In some examples, the light localization and segmentation process may involve analyzing the pixels of the camera data and determining which pixels correspond to the individual lamps in the environment.

[0013] In some examples, obtaining the environment information may involve determining environmental characteristic color information and determining environmental features.Environmental characterization process of reflectivity information. According to some examples, obtaining the controllable actuator capability information and obtaining the environmental information may involve an iterative process of determining one or more characteristics of lamps in the environment, determining environmental characteristic color information, and determining environmental characteristic reflectivity information.

[0014] Some methods may involve modifying the ARR by the control system to produce an actuator map (AM). According to some examples, the modification may include regularization, filling gaps in the ARR by increasing one or more volumes of the environment affected by actuator responses, reducing one or more overlapping volumes of the environment affected by multiple actuator responses, or a combination thereof. In some examples, the AM may indicate the effect of each controllable actuator on the environment. According to some examples, the AM may be mathematically applicable to inversion. In some examples, the AM may allow object-based sensory data to be rendered as actuator commands.

[0015] Some methods may involve identifying one or more ill-posed spatial coordinates of the AM by the control system. Some such methods may involve generating one or more repulsion functions by the control system, the one or more repulsion functions being configured to deviate the trajectory of the sensory object from the one or more ill-posed spatial coordinates. Some such methods may be described in the specification on page 2 / 33 of CN 121866536 A, involving storing or providing the one or more exclusion functions together with the AM.

[0016] In some examples, the ARR may be represented as a matrix, and the AM may be represented as a modified version of the matrix. According to some examples, the matrix may include values ​​corresponding to one or more actuator indices, one or more execution commands, and spatial coordinates.

[0017] In some examples, obtaining the controllable actuator positioning information may involve receiving controllable actuator capability information via an interface system. According to some examples, the interface system may be or may include a user interface system.

[0018] Some or all of the operations, functions, and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory computer-readable media. Such non-transitory media may include one or more memory devices as described herein, including but not limited to one or more random access memory (RAM) devices, read-only memory (ROM) devices, etc. Therefore, some innovative aspects of the subject matter described in this disclosure may be implemented in one or more computer-readable non-transitory media on which software is stored.

[0019] At least some aspects of this disclosure can be implemented via apparatus. For example, one or more devices may be able to perform at least partially the methods disclosed herein. In some embodiments, the apparatus may include an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc.ASICs, field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or combinations thereof. The control system may be configured to perform some or all of the disclosed methods.

[0020] Details of one or more embodiments of the subject matter described in this specification are set forth in the following figures and description. Other features, aspects, and advantages will become apparent from the description, figures, and claims. Note that the relative dimensions of the following figures may not be drawn to scale. Brief Description of the Drawings

[0021] The disclosed embodiments will now be described by way of example only with reference to the accompanying drawings.

[0022] FIG1A is a block diagram illustrating examples of components of an apparatus capable of implementing various aspects of the present disclosure.

[0023] FIG1B illustrates example elements of an endpoint.

[0024] FIG2 illustrates an example of an actuator element.

[0025] FIG3 illustrates example elements of a system for creating and playing multi-sensory (MS) experiences.

[0026] FIG4A illustrates example elements of a multi-sensory (MS) renderer.

[0027] FIG4B is a flowchart outlining an example of a method that can be performed by a device or system such as the apparatus or system disclosed herein.

[0028] FIG5 shows example elements of another system for creating and playing MS experiences.

[0029] FIG6 depicts the light room response (LRR) of a single luminaire in egocentric polar coordinates.

[0030] FIG7 shows example elements of a light mapping system.

[0031] FIG8 shows an example of color volume.

[0032] FIG9 depicts an alternative method for generating LRR.

[0033] FIG10 shows an example of information that can be provided to the simulator of FIG9.

[0034] FIG11A and FIG11B show examples of enhanced LRR.

[0035] FIG12 is a flowchart outlining an example of a method that can be performed by a device or system such as the apparatus or system disclosed herein. Specification 3 / 33 pages 7 CN 121866536 A Detailed Description

[0036] This document describes techniques related to providing multi-sensory media content. In the following description, numerous examples and specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that this disclosure, as defined by the claims, may include some or all of these examples individually or in combination with other features described below, and may further include modifications and equivalents of the features and concepts described herein.

[0037] In the following description, various methods, processes, and procedures are described in detail. While particular steps may be described in a certain order, this order is primarily for convenience and clarity. Specific steps may be repeated more than once, and may be performed in other...Steps occur before or after (even if these steps are described in a different order) and can occur in parallel with other steps. A second step is only required after the first step if the first step must be completed before the second step can begin. This will be specifically indicated when it is unclear from the context.

[0038] In this document, the terms “and,” “or,” and “and / or” are used. Such terms should be understood to have inclusive meanings. For example, “A and B” can at least mean: “both A and B,” “at least both A and B.” As another example, “A or B” can at least mean: “at least A,” “at least B,” “both A and B,” “at least both A and B.” As another example, “A and / or B” can at least mean: “A and B,” “A or B.” When XOR is intended to be used, this will be specifically indicated (e.g., “either A or B,” “at most one of A and B”).

[0039] This document describes various processing functions associated with structures such as blocks, elements, components, circuits, etc. Typically, these structures can be implemented by one or more processors controlled by one or more computer programs.

[0040] As mentioned above, media content delivery typically focuses on audio and video experiences. The delivery of multi-sensory (MS) content has been limited due to the customized nature of actuation.

[0041] This application describes methods for expanding the creative palette of content creators, allowing for the creation and delivery of spatial MS experiences at scale. Some such methods involve introducing new layers of abstraction to deliver created MS experiences to different endpoints using different types of lighting or actuators. As used herein, the term "endpoint" is synonymous with "playback environment" or simply "environment," referring to an environment that includes one or more actuators that can be used to deliver the MS experience. Such endpoints can include rooms (such as the living room in a home), cars, movie theaters, nightclubs, or other locations. Some disclosed methods involve creating, delivering, and / or rendering object-based sensory data, which can include sensory objects and corresponding sensory metadata. This abstraction enables the implementation of creative intent in an object-based format without prior knowledge of the specific controller type, number, and layout of any particular playback environment or "endpoint," thus achieving greater flexibility and scalability. In this document, the MS experience delivered via object-based sensory data may be referred to as a "flexibly expanded MS experience."

[0042] Acronyms MS – Multisensory MSIE – MS Immersive Experience AR – Augmented Reality VR – Virtual Reality PC – Personal Computer Figure 1A is a block diagram illustrating examples of components of a device capable of implementing various aspects of the present disclosure. As with other figures provided herein, the types and quantities of elements shown in Figure 1A are provided by way of example only. Other embodiments may includeMore, fewer, and / or different types and numbers of elements. According to some examples, device 101 may be or may include a device configured to perform at least some of the methods disclosed herein, such as a smart audio device, a laptop computer, a cellular phone, a tablet device, a smart home hub, etc. In some such embodiments, device 101 may be or may include a server configured to perform at least some of the methods disclosed herein.

[0043] In this example, device 101 includes at least an interface system 105 and a control system 110. In some embodiments, control system 110 may be configured to perform at least partially the methods disclosed herein. According to some examples, control system 110 may be configured to determine environmental and actuator data, such as environmental and actuator data 004 described herein with reference to Figures 3, 4A, and 5.

[0044] In some examples, control system 110 may be configured to obtain (e.g., via interface system 105) controllable actuator positioning information of a set of controllable actuators in an environment, in this example, a playback environment. For example, the playback environment could be a room in a house, a vehicle, etc. According to this example, the controllable actuator includes one or more luminaires, one or more tactile sensors, one or more airflow control devices, or a combination thereof. According to some examples, the control system 110 can be configured to obtain controllable actuator actuation direction information, such as light direction information, air movement direction information, etc.

[0045] According to some examples, the control system 110 can be configured to obtain controllable actuator capability information for each controllable actuator in a set of controllable actuators. Each controllable actuator may require a specific type and / or level of control to produce the desired output. For example, a Philips Hue™ bulb may need to receive control information in a specific format to turn on a lamp with a digital representation of a specific saturation, brightness, and hue, as well as a desired drive level. Therefore, the controllable actuator capability information can indicate the type of controllable actuator, the required control signal format (if any), the drive level required to obtain a specific actuator response, etc. In some examples, obtaining this controllable actuator capability information may involve a lamp characterization process to determine the characteristics of the controllable lamp within the environment.

[0046] In some examples, the control system 110 may be configured to obtain environmental information corresponding to the environment, in other words, environmental information corresponding to the environment in which the controllable actuator is located. This environmental information may include the location, orientation, and / or size of one or more environmental features. These one or more environmental features may include one or more structural elements of the environment, such as one or more walls, ceilings, floors, or combinations thereof. In some examples, the environmental information may include...Environmental feature color information, environmental feature reflectivity information, or a combination thereof. The one or more environmental features may include one or more furniture items. In some such examples, the environmental information may include furniture location information.

[0047] Various methods for obtaining controllable actuator positioning information, controllable actuator capability information, and environmental information are disclosed herein. Such methods include automated methods, manual / user input-based methods, and a "hybrid" method that is partially automated and partially manual.

[0048] According to some examples, the control system 110 may be configured to generate an actuator-room response (ARR) that summarizes the environment's response to activation of a set of controllable actuators.

[0049] The matrix representation of the ARR is generally ill-posed for inversion. For example, according to some matrix representations of the ARR, if there is no actuator effect at a particular location in the playing environment, the values ​​of several columns or rows of the ARR matrix may be zero.

[0050] In some examples, the control system 110 may be configured to modify the ARR (e.g., modify the matrix representation of the ARR) to produce an actuator map (AM) indicating the effect of each controllable actuator on the environment. The AM can allow object-based sensory data to be rendered as actuator commands. According to some examples, the matrix representation of the AM is mathematically suitable for inversion. Modifying the ARR can involve regularization, filling gaps in the ARR by increasing one or more volumes of the environment affected by actuator responses (e.g., eliminating zero values ​​in the ARR matrix), reducing one or more overlapping volumes of the environment affected by multiple actuator responses, or a combination thereof.

[0051] Interface system 105 may include one or more network interfaces and / or one or more external device interfaces (such as one or more Universal Serial Bus (USB) interfaces). According to some embodiments, interface system 105 may include one or more wireless interfaces. Interface system 105 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, display systems, touch sensor systems, and / or gesture sensor systems. In some examples, interface system 105 may include one or more interfaces between control system 110 and memory systems (such as optional memory system 115 shown in FIG. 1A). However, in some cases, the control system 110 may include a memory system.

[0052] For example, the control system 110 may include a general-purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.

[0053] In some implementations, the control system 110 may reside in more than one device. For example, a portion of the control system 110 may reside in a device within the environment (such as a laptop computer, tablet computer, smart audio device, etc.), and another portion of the control system 110 may reside in a device outside the environment (such as a server). In other examples, a portion of the control system 110 may reside in a device within the environment, and another portion of the control system 110 may reside in one or more other devices within the environment.

[0054] Some or all of the methods described herein can be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices as described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. One or more non-transitory media may, for example, reside in the optional memory system 115 and / or control system 110 shown in FIG. 1A. Thus, various innovative aspects of the subject matter described herein can be implemented in one or more non-transitory media on which software is stored. For example, the software may include instructions for controlling at least one device to process audio data. For example, the software may be executed by one or more components of a control system, such as control system 110 as shown in FIG1A.

[0055] In some examples, device 101 may include the optional microphone system 120 shown in FIG1A. The optional microphone system 120 may include one or more microphones. In some embodiments, the one or more microphones may be part of or associated with another device, such as a speaker of a speaker system, a smart audio device, etc.

[0056] According to some embodiments, device 101 may include the optional actuator system 125 shown in FIG1A. The optional actuator system 125 may include one or more loudspeakers, one or more haptic devices, one or more luminaires (also referred to herein as illuminators), one or more fans or other airflow devices, one or more display devices (including but not limited to one or more televisions), one or more position actuators, one or more other types of devices for providing an MS experience, or combinations thereof. As used herein, the term "luminaire" generally refers to any actuator configured to provide light. The term "light fixture" encompasses various types of light sources, including individual light sources (such as light bulbs), light source arrays (such as LED strips), light panels (such as LED panels), projectors, display devices (such as television (TV) screens), and so on. A "light fixture" can be movable; therefore, in this context, the term "fixture" does not necessarily mean that the fixture must be in a fixed position in space. As used herein, the term "position actuator" generally refers to something configured to change the position of a person or object.Devices for positioning or orientation of an object, such as motion simulator seats. A loudspeaker may sometimes be referred to herein as a “speaker.” In some embodiments, the optional actuator system 125 may include a display system comprising one or more displays, such as one or more light-emitting diode (LED) displays, one or more organic light-emitting diode (OLED) displays, etc. In some examples where device 101 includes a display system, the optional sensor system 130 may include a touch sensor system and / or a gesture sensor system proximate to one or more displays of the display system. According to some such embodiments, the control system 110 may be configured to control the display system to present a graphical user interface (GUI), such as a GUI associated with implementing one of the methods disclosed herein.

[0057] In some embodiments, device 101 may include the optional sensor system 130 shown in FIG. 1A. Optional sensor system 130 may include a touch sensor system, a gesture sensor system, one or more cameras, etc.

[0058] FIG. 1B shows example elements of an endpoint. In this example, the endpoint is a living room 1001, which contains multiple actuators 008, some furniture 1010, and a person 1000 (also referred to herein as a user) who will flexibly extend the MS experience. Actuators 008 are devices capable of altering the environment 1001 in which the user 1000 is located. Actuators 008 may include one or more haptic devices, one or more lamps (also referred to herein as illuminators), one or more fans or other airflow devices, one or more display devices (including, but not limited to, one or more televisions), one or more position actuators, one or more other types of devices for providing the MS experience, or combinations thereof.

[0059] The number of actuators 008, the arrangement of actuators 008, and the capabilities of actuators 008 in space 1001 can vary significantly between different endpoint types. For example, the number, arrangement, and capabilities of actuators 008 in a car or other vehicle are typically different from those in a living room, nightclub, terrace, etc. In many implementations, the number, arrangement, and / or capability of actuators 008 may also differ significantly between different instances of the same type (e.g., between a small living room with 2 actuators 008 and a large living room with 16 actuators 008). This disclosure describes various methods for creating flexible, expandable MSIEs and extending them to these heterogeneous endpoints.

[0060] Figure 2 illustrates an example of actuator elements. In this example, the actuator is a illuminator 1100, which includes a network module 1101, a control module 1102, and a light emitter 1103. The terms “illuminator” and “light fixture” may be used synonymously herein.According to this example, the light emitter 1103 includes one or more light-emitting devices, such as light-emitting diodes, configured to emit light into the environment where the illuminator 1100 resides. In this example, the network module 1101 is configured to provide network connectivity to one or more other devices in the space, such as devices that send commands to control the illuminator 1100 to emit light. According to this example, the network module 1101 is an example of the interface system 105 of FIG. 1A. In this example, the control module 1102 is configured to receive signals via the network module 1101 and control the light emitter 1103 accordingly. According to this example, the control module 1102 is an example of the control system 110 of FIG. 1A.

[0061] Other examples of actuators may also include the network module 1101 and the control module 1102, but may include other types of actuation elements. Some such actuators may include one or more tactile devices, one or more fans or other airflow devices, one or more position actuators, etc.

[0062] FIG3 illustrates example elements of a system for creating and playing a multi-sensory (MS) experience. As with other figures provided herein, the types and numbers of elements shown in FIG3 are provided by way of example only. Other embodiments may include more, fewer, and / or different types and numbers of elements. According to some examples, system 300 may be or may include one or more devices configured to perform at least some of the methods disclosed herein. In some examples, system 300 may include one or more instances of the control system 110 of FIG1A configured to perform at least some of the methods disclosed herein.

[0063] According to examples in this disclosure, methods for creating and delivering object-based MS immersive experiences (MSIE) involve applying a set of techniques for creating, delivering, and rendering object-based sensory data, which may include sensory objects and corresponding sensory metadata, to actuator 008. Some examples are described in the following paragraphs.

[0064] Object-based representation: In various disclosed embodiments, multi-sensory (MS) effects are represented using content that may be referred to herein as multi-sensory (MS) objects or simply “sensory objects”. According to some such implementations, attributes such as layer type and priority can be assigned to, associated with, and attached to each sensory object, thereby enabling the content creator's intent to be represented in the rendered experience. Detailed examples of sensory object attributes are described below.

[0065] In this example, system 300 includes a content creation tool 000 configured to design multi-sensory (MS) immersive content and to individually or in combination with corresponding audio data 011 and / or specification page 7 / 33 11 CN 121866536 A, depending on a particular implementation.Video data 012 outputs object-based sensory data 005. Object-based sensory data 005 may include timestamp information and information indicating the type of sensory object, sensory object attributes, etc. In this example, object-based sensory data 005 is not “channel-based” data corresponding to one or more specific sensory actuators in the playback environment, but rather generalized to multiple playback environments with multiple actuator types, multiple actuators, etc. In some examples, object-based sensory data 005 may include object-based light data, object-based tactile data, object-based airflow data, or object-based position actuator data, object-based olfactory data, object-based smoke data, object-based data for one or more other types of sensor effects, or combinations thereof. According to some examples, object-based sensory data 005 may include sensory objects and corresponding sensory metadata. For example, if object-based sensory data 005 includes object-based light data, then the object-based light data may include light object position metadata, light object color metadata, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradation metadata, light object priority metadata, light object layer metadata, or combinations thereof. In some examples, object-based sensory data 005 may include time data, such as timestamp information. Although in this example, content creation tool 000 is shown as providing a stream of object-based sensory data 005 to experience player 002, in alternative examples, content creation tool 000 may generate object-based sensory data 005 that is stored for later use. An example of a graphical user interface for a light object-based content creation tool is described below.

[0066] Examples of MS Object Properties The following is a non-exhaustive list of possible properties of an MS object: • Priority; • Layer; • Blending mode; • Persistence; • Effect; and • Spatial audiovisual law.

[0067] Effect As used herein, “effect” for an MS object is a synonym for the MS object type. “Effect” is or indicates the sensory effect provided by the MS object. If the MS object is a light object, its effect will involve providing direct or indirect light. If the MS object is a tactile object, its effect will involve providing some type of tactile feedback. If the MS object is an airflow object, its effect will involve providing some type of airflow. As described in more detail below, some examples involve other “effects” categories.

[0068] Persistence Some MS objects may have persistent attributes in their metadata. For example, when a movable MS object moves around in the scene, the movable MS object may persist for a period of time at the location that the movable MS object passes through. This period of time may be indicated by persistent metadata. In some implementations, the MS renderer is responsible for building and maintaining persistent state.

[0069] Layers, according to some examples, may assign individual MS objects to a “layer” where MS objects are grouped together based on one or more shared characteristics. For example, layers may group MS objects together based on the expected effect or type, which may include, but is not limited to, the following: - Atmosphere / Environment - Informative - Emphasis / Attention Specification 8 / 33 Page 12 CN 121866536 A Alternatively or additionally, in some examples, layers may be used to group MS objects together based on shared attributes, which may include, but are not limited to, the following: - Color - Intensity - Size - Shape - Position - Spatial Priority In some examples, MS objects may have a priority attribute that allows the renderer to determine which(s) should have priority in an environment where MS objects compete for a limited actuator. For example, if the volume corresponding to multiple light objects overlaps with a single light fixture when all light objects are planned to be rendered, the renderer may refer to the priority of each light object to determine which(s) light objects will be rendered. In some examples, priorities may be defined between or within layers. According to some examples, priorities may be associated with specific attributes such as intensity. In some examples, priority can be defined by time: for example, the most recently rendered MS object may take precedence over the previously rendered MS object. According to some examples, priority can be used to specify which MS object or layer should be rendered regardless of the limitations of a particular actuator system in the playback environment.

[0070] Spatial Panning Laws Spatial panning laws can define how the movement of MS objects in space, how MS objects affect actuators when they move between actuators, etc.

[0071] Mixing Modes Mixing modes can specify how multiple objects are reused on a single actuator. In some examples, a mixing mode can include one or more of the following: - Max Mode: Select the MS object that is activated most times by the actuator; - Mixing Mode: Mix some or all objects according to a set of rules, such as by summing activation levels, taking the average of activation levels, mixing colors according to activation level or priority level, etc.; - MaxNmix: Mix the top N MS objects (by activation level) according to a set of rules.

[0072] According to some examples, instead of (or in addition to) per-object metadata, more general metadata for the entire multi-sensory content file can be defined. For example, MS content files may include metadata such as trimming processes or mastering environments.

[0073] The trimming control, referred to as "trim control" in the context of Dolby Vision™, can serve as guidance on how to adjust the default rendering algorithm for specific environments or conditions at endpoints. Trimming control can be applied to various attributes (including saturation, color, etc.).(Adjusting details, gamma values, etc.) specifies ranges and / or default values. For example, there may be car trim controls that provide specific default values ​​and / or rule sets for rendering in a car environment, such as guidance to include only objects of a specific priority or layer. Other examples may provide trim controls for environments with limited, complex, or sparse multi-sensory actuators.

[0074] A single multi-sensory content in a mastering environment may include metadata about attributes of the mastering environment, such as room size, reflectivity, and ambient bias lighting levels. Specific attributes may vary depending on the desired endpoint actuators. Mastering environment information can help provide reference points for rendering in the playback environment.

[0075] MS Object Renderer: Various disclosed embodiments provide a renderer configured to render effects from MS Specification Page 9 / 33, 13 CN 121866536 A to actuators in a playback environment. According to this example, system 300 includes an MS renderer 001 configured to render object-based sensory data 005 into actuator control signals 310, at least in part, based on environment and actuator data 004. In this example, MS renderer 001 is configured to output actuator control signals 310 to MS controllers 003, which are configured to control actuators 008. In some examples, MS renderer 001 may be configured to receive light objects and object-based lighting metadata indicating a desired lighting environment, as well as lighting information about the local lighting environment. The lighting information is a generic type of environment and actuator data 004 and may include one or more characteristics of one or more controllable light sources in the local lighting environment. In some examples, MS renderer 001 may be configured to determine a drive level for each of the one or more controllable light sources that approximates the desired lighting environment. According to some examples, MS renderer 001 (or one of the MS controllers 003) may be configured to output drive levels to at least one of the controllable light sources. Some alternative examples may include separate renderers for each type of actuator 008, such as one renderer for a luminaire, another renderer for a haptic device, another renderer for an airflow device, etc. In other embodiments, a single renderer may be configured as an MS renderer and an audio renderer and / or a video renderer. In some embodiments, the MS renderer 001 may be configured to adapt to changing conditions. Some examples of implementations of the MS renderer 001 are described in more detail below.

[0076] The environment and actuator data 004 may include content referred to herein as a “room descriptor” that describes the location of the actuator (e.g., according to an x, y, z coordinate system or a spherical coordinate system). In some examples, the environment and actuatorData 004 can indicate actuator orientation and / or placement attributes (e.g., directional and north-facing, omnidirectional, occlusion information, etc.). According to some examples, ambient and actuator data 004 can indicate actuator orientation and / or placement attributes according to a 3 × 3 matrix, in which three elements (e.g., elements in the first row) represent spatial location (x, y, z), three other elements (e.g., elements in the second row) represent orientation (roll, pitch, yaw), and three other elements (e.g., elements in the third row) indicate scale or size (sx, sy, sz). In some examples, ambient and actuator data 004 may include device descriptors describing actuator attributes associated with the MS renderer 001, such as the intensity range and color gamut of a luminaire, the airflow velocity range and(s) direction of an airflow device, etc. Additional details regarding ambient and actuator data 004, including how it can be obtained, are described below.

[0077] In this example, system 300 includes an experience player 002 configured to receive object-based sensory data 005', audio data 011', and video data 012', and to provide the object-based sensory data 005' to an MS renderer 001, the audio data 011' to an audio renderer 006, and the video data 012' to a video renderer 007. In this example, the reference numerals for the object-based sensory data 005', audio data 011', and video data 012' received by the experience player 002 include an apostrophe (') to indicate that the data may be encoded in some cases. Similarly, the object-based sensory data 005', audio data 011', and video data 012' output by the experience player 002 do not include an apostrophe to indicate that the data may have been decoded by the experience player 002 in some cases. According to some examples, the experience player 002 can be a media player, game engine, or component integrated into a television, DVD player, soundbar, set-top box, or service provider media device (such as Chromecast, Apple TV, or Amazon Fire TV). In some examples, the experience player 002 can be configured to receive encoded object-based sensory data 005' as well as encoded audio data 011' and / or encoded video data 012'. In some such examples, the encoded object-based sensory data 005' can be received as part of the same bitstream as the encoded audio data 011' and / or encoded video data 012'. Some examples are described in more detail below. According to some examples, the experience player 002 can be configured to extract object-based sensory data 005 from the content bitstream and then...Decoded object-based sensory data 005 is provided to MS renderer 001, decoded audio data 011 is provided to renderer 006 (audio specification page 10 / 33, 14 CN 121866536 A), and decoded video data 012 is provided to video renderer 007. In some examples, timestamp information in the object-based sensory data 005 may be used, for example, by experience player 102, MS renderer 001, audio renderer 106, video renderer 107, or all of them, to synchronize effects associated with the object-based sensory data 005 with audio data 111 and / or video data 112, which may also include timestamp information.

[0078] According to this example, system 300 includes MS controllers 003 configured to communicate with various actuator types using an application programming interface (API) or one or more similar interfaces. Generally, each actuator will require a specific type of control signal to produce the desired output from the renderer. According to this example, MS controller 003 is configured to map the output from MS renderer 001 to control signals for each actuator. For example, a Philips Hue™ bulb receives control information in a specific format to turn on a lamp with a digital representation of a specific saturation, brightness, and hue, as well as a desired drive level. In some alternative examples, MS renderer 001 may also be configured to implement some or all of the features in MS controller 003. For example, MS renderer 001 may also be configured to implement one or more lighting-based APIs but not haptic-based APIs, or vice versa.

[0079] In some examples, the room descriptor may also describe the size and orientation of the playback environment itself to establish a relative or absolute coordinate system to which all objects are positioned. For example, in a living room, a display screen may be considered in front, in some cases in the center of the front, and the floor and ceiling may be considered vertical boundaries. In some such examples, the room descriptor may also indicate the boundaries corresponding to the left, right, front, and back walls relative to the front position. According to some examples, the room descriptor may also be provided in the form of a matrix (e.g., a 3 × 3 matrix). The room descriptor information is used to describe the physical dimensions of the playback environment (e.g., in units of physical distance, such as meters). In some such examples, sensory object positioning, sensory object size, and sensory object orientation can be described in units relative to the room size, such as in the range of -1 to 1. In some cases, the room descriptor can also describe the preferred viewing position according to a matrix.

[0080] The type, number, and arrangement of actuators 008 will generally vary depending on the particular implementation. In some examples, actuators 008 may include lamps and / or light strips (also referred to herein as “illuminators”), vibration motors, airflow generators, position actuators, or combinations thereof.

[0081] Similarly, the type, number, and arrangement of the loudspeaker 009 and display device 010 will generally vary depending on the specific implementation. In the example shown in FIG3, audio data 011 and video data 012 are rendered to the loudspeaker 009 and display device 010 by audio renderer 006 and video renderer 007, respectively.

[0082] As described above, according to some implementations, system 300 may include one or more instances of the control system 110 of FIG1A configured to perform at least some of the methods disclosed herein. In some such examples, one instance of control system 110 may implement content creation tool 000, and another instance of control system 110 may implement experience player 002. In some examples, one instance of control system 110 may implement audio renderer 006, video renderer 007, multi-sensory renderer 001, or a combination thereof. According to some examples, an instance of control system 110 configured to implement experience player 002 may also be configured to implement audio renderer 006, video renderer 007, multi-sensory renderer 001, or a combination thereof.

[0083] FIG4A shows example elements of a multi-sensory (MS) renderer. As with other figures provided herein, the type and number of elements shown in FIG4A are provided by way of example only. Other implementations may include more, fewer, and / or different types and numbers of elements. According to this example, MS renderer 001 is an instance of MS renderer 001 described with reference to FIG3. In some examples, MS renderer 001 may be implemented by one or more instances of the control system 110 of FIG1A.

[0084] According to this example, FIG4A includes the following elements: • 004: Environmental and actuator data, which may be described with reference to FIG3; Specification 11 / 33 pages 15 CN 121866536 A • 005: Object-based sensory data 005, which may be described with reference to FIG3; • 423 Actuator Mapping (AM), indicating the positioning of at least controllable actuators 008 in a particular playback environment; • 450 Projection Module, configured to project MS objects based at least in part on the object-based sensory data 005 based on AM 423. In this document, the MS object can also be referred to as a "sensory object" because in some cases, the object-based sensory data 005 may contain only one type of sensory object (e.g., only tactile objects or only light objects). In this example, the projection module 450 is configured to project the MS object at least partially based on sensory object metadata, which may include at least sensory object positioning metadata and sensory object size metadata; • 440 Actuator Activation Matrix (AAM), output by the projection module 450 according to this example. For example, the AAM may indicate the sensory object (if any) currently present in the playback environment, containing a volume corresponding to one or more corresponding actuators.For example, AAM can indicate whether the light object positioning metadata and light object size metadata indicate that a particular luminaire is located within a volume in the playback environment corresponding to the positioning and size of the light object; • 451 Mixing module, configured to convert AAM 440 into actuator control signals based at least in part on environment and actuator data and renderer configuration data; • 452 Optional renderer configuration data, which may include information about one or more settings of MS renderer 001, such as settings indicating desired motion, mode, etc. In some examples, the renderer configuration data 452 may be changed automatically using a context-aware system (as described in more detail below); and • 310: Actuator control signals, which can be described with reference to FIG3. In some examples, actuator control signals 310 may be sent to individual actuators 008; while in other examples, actuator control signals 310 may be sent to MS controller 003, which may be configured to send appropriate control signals to various types of actuators 008.

[0085] According to some examples, AAM 440 is a matrix that describes, according to actuator mapping 423, how much of the sensory object itself is projected onto each actuator. In some examples, AAM 440 may be a real matrix of size NO × NA, where NO represents the number of sensory objects in the environment and NA represents the number of controllable actuators in the environment. In this example, mixing module 451 is configured to generate actuator control signals 310 at least in part based on AAM 440 and environmental and actuator data 004. In some examples, mixing module 451 may be configured to generate actuator control signals 310 at least in part based on optional renderer configuration data 452. According to some examples, mixing module 451 may be configured to generate actuator control signals 310 at least in part based on sensory object metadata (as shown in FIG. 4A, which may be received as part of object-based sensory data 005), such as mixing and imaging laws associated with at least one sensory object.

[0086] In some examples, the mixing module 451 may be configured to generate the actuator control signal 310 based at least in part on one or more of the following: 1. a threshold element of AAM 440; 2. taking the maximum value of a specific column of AAM 440—in other words, taking the sensory object that activates a specific actuator the most times as the output; 3. taking any combination of the first N elements and performing at least one of the following operations: o mixing the object in the actuator channel; o pushing the object to an adjacent channel.

[0087] In some embodiments, the projection module 450 may be configured to generate a sensory object image using the spatial coordinates (e.g., x, y, z coordinates) and the size of the sensory object to produce In(x, y, z), where In represents the nth sensor.The sensory object image of the actuator object. Then, in some examples, the projection module 450 can be configured to calculate the nth line (object index) of each column (actuator index) of AAM 440 by taking the inner product of the object image and the actuator mapping corresponding to the actuator. The projection module 450 can generate and perform the object image, actuator mapping and dot product in any convenient spatial domain, including but not limited to polar coordinates, cylindrical coordinates or rectangular coordinates.

[0088] Some embodiments may involve implementing what may be referred to herein as "rejection", which can be used to avoid potentially undesirable sensory effects, such as lighting effects that may occur when the sensory object is located in one or more areas of the playback environment. In some such examples, rejection data may be included in the environment and actuator data 004 and AM 423, and therefore may be part of the information input to the projection module 450. In some such examples, when the MS object is projected onto AM 423 to generate AAM 440, the spatial coordinates of the MS object will be enhanced.

[0089] Multisensory rendering synchronization based on object-based MS rendering involves flexibly rendering different modalities to endpoints / playback environments. Endpoints have different capabilities depending on various factors, including but not limited to: • the number of actuators, • the modality of these actuators (e.g., lighting fixtures and airflow control devices versus haptic devices); • the type of these actuators (e.g., white smart lights versus RGB smart lights, or haptic vests versus haptic cushions); and • the positioning / layout of these actuators.

[0090] In order to render object-based sensory content to any endpoint, some processing of the object signals (e.g., intensity, color, pattern, etc.) is typically required. The processing of the signal path for each modality should not change the relative phase of certain features within the object signal. For example, suppose lightning is presented in both haptic and light modalities. The signal processing chain corresponding to the actuator control signals should not cause any type of sensory object signal (haptic signal or light signal) to produce a time delay sufficient to alter the perceptual synchronicity of the two modalities. The required level of synchronization may depend on various factors, such as whether the experience is interactive and what other modalities are involved in the experience. Depending on the specific context, the maximum time difference can, for example, range from approximately 10 ms to 100 ms.

[0091] Rendering of haptic content based on objects: Object-based haptic content conveys the sensory aspects of a scene through abstract sensory representations rather than channel-based schemes. For example, instead of simply defining haptic content as a single-channel time-correlated amplitude signal, which is then played from a specific haptic actuator (such as a vibrating haptic motor) on a vest worn by the user, object-based...The tactile content can be defined by the sensation it is intended to convey. More specifically, in one example, there may be a tactile object representing a collision tactile sensory effect. Associated with this object are: • the spatial location of the tactile object; • the spatial direction / vector of the tactile effect; • the intensity of the tactile effect; • tactile spatial and temporal frequency data; and • a time-dependent amplitude signal.

[0092] According to some examples, this type of tactile object can be automatically created in interactive experiences such as video games, for example, in a racing game when another car hits the player's car from behind. In this example, the MS renderer will determine how to render the spatial modality of this effect to a set of tactile actuators at the endpoints. In some examples, the renderer performs this operation based on information about: • the (multiple) types of haptic devices available, such as haptic vests, haptic gloves, haptic cushions, and haptic controllers; • the location of each haptic device relative to (multiple) users (some haptic devices may not be coupled to (multiple) users, such as vibrators mounted on the floor or a seat); • the type of actuation provided by each haptic device, such as kinematic and vibratory haptics; • the start-up and stop delay of each haptic device (in other words, the speed at which each haptic device can be turned on and off); • ​​the dynamic response of each haptic device (how much the amplitude can vary); • the time-frequency response of each haptic device (the time frequency that the haptic device can provide); • the spatial distribution of addressable actuators within each haptic device: for example, a haptic vest may have dozens of addressable haptic actuators distributed across the user's torso; and • the temporal response of any haptic sensors used to render closed-loop haptic effects (e.g., active force feedback kinematic haptic devices).

[0093] These properties of the haptic modality of the endpoints will inform the renderer how best to render a particular haptic effect. Consider again the car crash effect example. In this example, the player wears a haptic vest, haptic armband, and haptic gloves. According to this example, the haptic shockwave effect is spatially located at the location where the car crashes into the player. The shockwave vector is determined by the relative velocity of the player's car and the car that crashes into the player. The spatial and temporal spectrum of the shockwave effect is created based on the type of material the virtual car is expected to use, as well as other virtual world properties. The renderer then renders the shockwave through a set of haptic devices in the endpoints based on the shockwave vector and the physical location of the haptic devices relative to the user.

[0094] The signals sent to each particular actuator are preferably provided in such a way that the sensory effect is consistent across all available (potentially heterogeneous) actuators. For example, the renderer may not render very high frequencies only to one of the haptic actuators (e.g., the haptic armband) due to the lack of capability of other actuators. Otherwise, when the shockwave moves through the playerWhen the user is on the body, the perceived tactile effect decreases as the wave moves through the vest, into the armband, and finally into the glove, because the tactile vest and tactile gloves worn by the user are not capable of rendering such high frequencies.

[0095] Some types of abstract tactile effects include: • Shock wave effects, as described above; • Barrier effects, such as tactile effects used to represent spatial constraints in a virtual world (e.g., in video games). If an active or resistive kinetic actuator (e.g., force feedback on a steering wheel or joystick) is present on the input device, this effect can be rendered by applying resistance to the user input. If no such actuator is available at the endpoint, in some examples, vibratory tactile feedback consistent with the collision of an in-game avatar with a barrier can be rendered; • Presence, such as indicating the presence of a large object (e.g., a train) approaching the scene. This type of tactile effect can be rendered using the low-frequency rumble of some tactile device actuators. This type of tactile effect can also be rendered as tactile spatial feedback by pressure applied by an air bladder; • User interface feedback, such as a click from a virtual button. For example, this type of haptic effect can be rendered to the nearest actuator on the user's body that performs the click, such as a haptic glove worn by the user. Alternatively or additionally, this type of haptic effect can also be rendered to a vibrator coupled to the chair the user is sitting on. This type of haptic effect can be defined, for example, using a time-dependent amplitude signal. However, this signal can be modified (modulated, frequency-shifted, etc.) to best suit the haptic device(s) that will provide the haptic effect; • Motion sensation. These haptic effects are designed to make the user perceive some form of motion. These haptic effects can be rendered by actuators that actually move the user (e.g., a moving platform / seat). In some examples, the actuator can provide an auxiliary modality (e.g., via video) to enhance the motion being rendered; and • Trigger sequences. These haptic effects are characterized primarily by their time-dependent amplitude signals. This signal can be rendered to multiple actuators and, in doing so, can be enhanced. This enhancement can include splitting the signal across multiple actuators in time or frequency. Some examples may involve amplifying the signal itself so that the sum of the haptic actuator outputs does not match the original signal.

[0096] Spatial Effects vs. Non-Spatial Effects Spatial effects are spatial effects constructed in a way that conveys certain spatial information about the multi-sensory scene being rendered. For example, if the playback environment is a room, then shockwaves moving across the room will be rendered differently to each haptic device, depending on the position and size of one or more haptic objects being rendered at a particular time, based on the location of each haptic device within the room.

[0097] In some examples, non-spatial effects can be targeted at specific parts of a user's body, regardless of the user's position or orientation. One example is a haptic device providing enhanced vibrations to a user's back to indicate immediate danger. Another example is a haptic device providing strong vibrations to indicate injury to a specific area of ​​the body.

[0098] Some effects can be non-narrative effects. Such effects are typically associated with user interface feedback, such as haptic sensations used to indicate that a user has completed a level or clicked a button on a menu item. Non-narrative effects can be spatial or non-spatial.

[0099] The haptic device type receives information about the different types of haptic devices available at the endpoint, allowing the renderer to determine which types of perceptual effects and rendering strategies it can use. For example, local haptic device data indicating that a user is wearing both a haptic glove and a vibrating haptic vest (or at least local haptic device data indicating that the haptic glove and vibrating haptic vest are present in the playback environment) allows the renderer to render a consistent recoil effect on both devices when the user fires a gun in the virtual world. The actual actuator control signals sent to the haptic devices may differ from those where only a single device is available. For example, if the user is only wearing a vest, the actuator control signal used to actuate the vest may vary for the start timing, maximum amplitude, frequency, and decay time of the actuator control signal, or combinations thereof.

[0100] Device positioning Understanding the positioning of the haptic device in the endpoints allows the renderer to consistently render spatial effects. For example, understanding the positioning of vibrating motors in a lounge allows the renderer to generate actuator control signals to each vibrating motor in the lounge in a manner that conveys spatial effects (such as shock waves propagating in the room). Additionally, although the position of wearable haptic devices is implied by their type (e.g., gloves on the user's hands), the renderer may also use the understanding of the position of these wearable haptic devices to convey both spatial and non-spatial effects.

[0101] Types of actuation provided by haptic devices Haptic devices can provide a range of different actuations and thus provide perceived sensations. These are generally divided into two basic categories: 1. Vibratory tactile sensation, such as vibration; or 2. Kinesthetic sensation, such as resistance feedback or active force feedback.

[0102] Actuation of any category can be static or dynamic, where the dynamic effect changes in real time based on some sensor input. Examples include touchscreens that use vibratory haptic actuators to render textures and position sensors that measure the position of the user's (multiple) fingers(s).

[0103] Furthermore, the physical construction of such actuators varies considerably and affects many other properties of the device. This is illustrated in the specification on page 15 / 33 of CN 121866536 A, where examples include start-up delays or time-frequency responses that differ significantly in the following types of haptic devices: • eccentric rotating masses; • linear resonant actuators; • piezoelectric actuators; and• Linear magnetic ram.

[0104] The renderer should be configured to take into account the startup delay of a specific haptic device type when rendering a signal to be actuated by a haptic device in an endpoint.

[0105] Startup and Stop Delay of Haptic Device The startup delay of a haptic device is the delay between the time when the actuator control signal is sent to the device and the physical response of the device. The stop delay is the delay between the time when the actuator control signal is sent to bring the output of the device to zero and the time when the device stops actuating.

[0106] Time Frequency Response The time frequency response is the frequency range in which the signal amplitude can be actuated in a steady state as a function of time.

[0107] Spatial Frequency Response The spatial frequency response is the frequency range in which the signal amplitude is a function of the spacing of the actuators of the haptic device. Devices with closely spaced actuators have a higher spatial frequency response.

[0108] Dynamic Range The dynamic range is the difference between the minimum and maximum amplitude of physical actuation.

[0109] Sensor Characteristics in Closed-Loop Haptic Devices Some dynamic effects use sensors to update the actuation signal based on certain observed states. The sampling frequency of time and space, as well as noise characteristics, will limit the ability to update the control loop of the actuator that provides the dynamic effect.

[0110] Airflow Another modality that some multi-sensory immersive experiences (MSIEs) can use is airflow. Airflow can be rendered in a consistent manner with one or more other modalities, such as audio, video, lighting effects, and / or haptics. Unlike dedicated (e.g., channel-based) settings designed only for 4D experiences in theaters (which may include “wind effects”), some airflow effects can also be provided at other endpoints that may typically include airflow, such as a car or living room. Unlike channel-based systems, airflow sensory effects can be represented as airflow objects, which may include the following attributes: • spatial location; • direction of the expected airflow effect; • intensity / airflow speed; and / or • air temperature.

[0111] Some examples of airflow objects can be used to represent the movement of a bird flying by. To render the airflow actuators at the endpoints, information about the following can be provided to the MS renderer 001: • The type of airflow device, such as a fan, air conditioner, or heater; • The position of each airflow device relative to the user's location or the user's expected location; • The capabilities of the airflow device, such as its ability to control direction, airflow, and temperature; • The control level of each actuator, such as airflow speed and temperature range; and • The response time of each actuator, such as how long it takes to reach a selected speed.

[0112] Some examples of airflow usage in different endpoints: In vehicles such as automobiles, object-based metadata can be used to create experiences such as:• To mimic a “chills down your spine” during scenes from horror movies or games; • To simulate the movement of birds flying by; and / or • To create a gentle breeze in a seascape.

[0113] Temperature changes can be achieved over a relatively short period of time within the small, enclosed space of a typical vehicle, compared to temperature changes in a larger environment such as a living room. In one example, MS Renderer 001 can raise the temperature when a player enters a “lava level” or other hot area during gameplay. Some examples may include other elements, such as confetti in vents, to celebrate an event, such as a goal scored by a user’s favorite football team.

[0114] In one example, airflow in a living space or other room can be synchronized with the breathing rhythm of guided meditation. In another example, airflow can be synchronized with the intensity of an exercise, increasing the airflow or decreasing the temperature as the intensity increases. In some examples, there may be relatively little control over spatial aspects during rendering. For example, many existing airflow actuators are optimized for heating and / or air conditioning, rather than for providing sensory actuation that diversifies the space.

[0115] Examples of Combined Light, Airflow, and Tactile Sensations in Automobiles The following examples are described with reference to automobiles, but are also applicable to other vehicles, such as trucks, vans, etc. In some examples, a user interface may be present on the steering wheel or on a touchscreen near or within the dashboard. According to some examples, the following actuators may be present in the automobile: 1. Individually addressable lights, spatially distributed throughout the automobile as follows: o on the dashboard; o below the footwell; o on the doors; and o within the center console.

[0116] 2. Individually controllable air conditioning / heating vents, distributed throughout the automobile as follows: o in the front dashboard; o below the footwell; o within the center console, facing the rear seats; o on the side pillars; o in the seats; and o guiding the windshield (for defogging).

[0117] 3. Individually controllable seats with vibrating tactile sensation; and 4. Individually controllable floor mats with vibrating tactile sensation.

[0118] In this example, the modalities supported by these actuators include the following: • lights with individually addressable LEDs throughout the vehicle, and indicator lights on the dashboard and steering wheel; • airflow via controllable air conditioning vents; • haptics, including: o steering wheel: haptic vibration feedback; o dashboard touchscreen: haptic vibration feedback and texture rendering; and o seats: haptic vibration and movement. Specification 17 / 33 pages 21 CN 121866536 A

[0119] In one example, a live music stream is rendered to four users sitting in the front seats. In this example, MS Renderer 001 attempts to optimize the experience for multiple viewing positions. During the construction period before the artists take the stage and after the first few performances have ended, the content includes: • interlude music;• Low-intensity lighting; and • Tactile content representing a crowd collision.

[0120] In addition to the rendered audio and video streams, the light content also includes ambient light objects that move slowly in the scene. These ambient light objects can be rendered using one of the environment layer methods disclosed herein, such that no spatial priority is given to any user's perspective. In some examples, the tactile content may be spatially focused in a lower temporal frequency spectrum and may be rendered solely by vibrating haptic motors in the floor mat.

[0121] According to this example, a fireworks event during a music stream corresponds to multi-sensory content including: • a light object spatially corresponding to the location of the fireworks in the event; and • a tactile object that enhances the dynamism of the fireworks via a shockwave effect.

[0122] In this example, the MS renderer 001 renders both the light object and the tactile object spatially. For example, the light object may be rendered in a car such that if the fireworks content is located on the left side of the scene, everyone in the car perceives the light object as coming from the left. In this example, only the lights on the left side of the car are actuated. Haptic effects can be rendered on both the seats and floor mats in a way that conveys directionality to each user separately.

[0123] At the end of the concert, fireworks will appear in the audio content, and fireworks and confetti will also appear in the video content. In addition to rendering light and tactile objects corresponding to the fireworks as described above, airflow modalities can also be used to render the effect of confetti spray. For example, individually controllable airflow vents of an HVAC system can be pulsed.

[0124] Living Room Example In this embodiment, in addition to an audio / visual (AV) system including multiple loudspeakers and a television, the following actuators and associated controls are available in the living room: • A haptic vest worn by the user (also known as the player); • A haptic vibrator mounted on the seat where the player sits; • A (haptic) controllable smartwatch; • Smart lights spatially distributed throughout the room; • A wireless controller; and • An addressable airflow bar (AFB) that includes an array of individually controllable fans pointing towards the user (similar to HVAC vents in a car dashboard).

[0125] In this example, the user is playing a first-person shooter game, and the game includes a scene where a destructive hurricane moves through the level. As the hurricane moves, in-game objects are thrown around, and some objects hit the player. Haptic objects rendered by MS Renderer 001 provide shockwave effects through all haptic devices that the user can perceive. The actuator control signals sent to each device can be optimized based on the impact intensity of the in-game objects, the direction(s) of the impact, and the capabilities and positioning (as described above) of each actuator.

[0126] In the time before the user is hit by an in-game object, the multisensory content includes haptic objects corresponding to non-spatial rumbles, one or more airflow objects corresponding to directional airflows, and one or more lights corresponding to lightning.Objects. MS renderer 001 renders non-spatial rumble to the haptic device. Actuator control signals sent to each haptic device (Page 18 / 33, CN 121866536 A) can be rendered such that the set of actuator control signals across the entire haptic array is consistent in the perceived arousal time, intensity, and frequency. In some examples, the frequency content of the actuator control signals sent to the smartwatch can be low-pass filtered to match the limited frequency capability of the vest near the watch. MS renderer 001 can render one or more airflow objects as actuator control signals for AFB such that the airflow in the room is consistent with the player's position and line of sight in the game, as well as the direction of the hurricane itself. Lightning can be rendered in all modalities as (1) a white flash produced on a light located in a suitable position (e.g., in or on the ceiling); and (2) a pulsed rumble in the user's wearable haptic and seat vibrator.

[0127] When the user is hit by an in-game object, a directional shock wave can be rendered to the haptic device. In some examples, a corresponding airflow pulse can be rendered. According to some examples, a damage-bearing effect can be rendered by lights, indicating the amount of damage a player suffers from being hit by an in-game object.

[0128] In some such examples, a signal can be spatially rendered to a haptic device, causing a perceived shockwave to move across the player's body and within the room. The MS renderer 001 can provide this effect based on actuator positioning information indicating the positioning of the haptic devices relative to each other. In addition to actuator capability information, the MS renderer 001 can also provide the shockwave vector and position based on the actuator positioning information. According to some examples, a non-directional airflow pulse can be rendered; for example, all vents of the AFB can be briefly enlarged to enhance the haptic modality. In some examples, a red halo can be rendered simultaneously onto a light strip around the TV to indicate to the player that the player has taken damage in the game.

[0129] Figure 4B is a flowchart outlining an example of a method that can be performed by a device or system such as those disclosed herein. As with other methods described herein, the blocks of method 400 need not be performed in the indicated order. In some embodiments, one or more blocks of method 400 may be performed simultaneously. Furthermore, some implementations of method 400 may include more or fewer blocks than those shown and / or described. The blocks of method 400 may be performed by one or more devices, which may be (or may include) one or more instances of a control system (such as control system 110 shown and described above in FIG. 1A).

[0130] In this example, block 405 relates to estimating (e.g., by control system 110) an actuator-room response (ARR) that summarizes the environment's response to activation of a set of controllable actuators in the environment. For example, the playback environment may...This could be a room in a house, a vehicle, etc. In some examples, the set of controllable actuators may include one or more light fixtures, one or more haptic devices, one or more airflow control devices, or combinations thereof. In some examples, the control system 110 may be configured to obtain controllable actuator positioning information of the set of controllable actuators in the environment. According to some examples, the control system 110 may be configured to obtain controllable actuator capability information for each controllable actuator in the set of controllable actuators. For example, the controllable actuator capability information may indicate the type of controllable actuator, the required control signal format (if any), the drive level required to obtain a particular actuator response, etc.

[0131] In some examples, the control system 110 may be configured to obtain environmental information corresponding to the environment, in other words, environmental information corresponding to the environment in which the controllable actuators are located. The environmental information may include the location, orientation, and / or size of one or more environmental features. The one or more environmental features may include one or more structural elements of the environment, such as one or more walls, ceilings, floors, or combinations thereof. In some examples, the environmental information may include environmental feature color information, environmental feature reflectivity information, or combinations thereof. The one or more environmental features may include one or more furniture items. In some such examples, the environmental information may include furniture location information.

[0132] This document discloses various methods for obtaining controllable actuator positioning information, controllable actuator capability information, and environmental information. Such methods include automated methods, manual / user input-based methods, and a "hybrid" method that is partially automated and partially manual. Specification 19 / 33 pages 23 CN 121866536 A

[0133] The process of estimating ARR and the way ARR is represented may vary depending on the specific implementation. In some examples, ARR may be represented as R(j, x, y, z, a, t), where: x represents the x-coordinate of the environment, which in this example is the room; y represents the y-coordinate of the room; z represents the z-coordinate of the room; a represents the command to be executed, which may be a vector, such as color; t represents time; and j represents the actuator index.

[0134] For the steady-state response of all actuated actuators (in other words, ignoring t), that is, for all j, a is the same. For spatially overlapping actuators, ARR is not suitable for inversion due to its low rank. According to some examples, R(j,x,y,z) can be reformulated as a matrix R(j, X), where X represents the spatial coordinates (x,y,z) converted to plane indices, since x,y,z are orthogonal.

[0135] R is the projection of the actuator's effect on the environment (in this example, a room). Assume vector A represents all actuators in the room. If R can be inverted, a mapping R⁻¹ can be obtained that projects the room effectors onto vector A.Above. In other words, actuator control signals can be obtained from a desired room effect, which can be represented as object-based sensory data. Being able to obtain actuator control signals from object-based sensory data indicating the desired room effect is desirable for object-based MS rendering.

[0136] However, the “raw” or unmodified version of R is generally not suitable (mathematically inapplicable) for inversion. One reason is that in many cases, there may be “dead zones” where no actuator affects a particular X coordinate. Thus, a corresponding number of columns or rows may have zero values. In some cases, there may be correlations between columns or rows.

[0137] According to this example, block 410 relates to modifying the ARR to produce an actuator map (AM). According to some examples, the matrix representation of the AM is mathematically suitable for inversion. Modifying the ARR may involve regularization, filling gaps in the ARR by increasing one or more volumes of the environment affected by actuator responses (e.g., eliminating zero values ​​in the ARR matrix), reducing one or more overlapping volumes of the environment affected by multiple actuator responses, or a combination thereof.

[0138] This AM can allow object-based sensory data to be rendered as actuator commands. According to some implementations, the AM can be used to: enable the creation of object-based sensory data 005 in a reasonable, endpoint-independent manner. In other words, content creators do not need to orient MS content to specific channels corresponding to individual actuators. Instead, content creators can create MS objects and spatially place them in a general representation of the playback environment; enable the MS renderer 001 to project spatial MS objects onto an actuator array in the environment.

[0139] Sensory objects O can be represented using coordinate positioning (e.g., (x, y, z)) or an equivalent flat coordinate representation X. The attributes of a sensory object can include sensory object positioning and sensory object size, which allows the sensory object to be projected into the X range of the playback environment / endpoint, which corresponds to the sensory object positioning and size. At any given time, actuators located within the endpoint volume corresponding to the sensory object size are actuation candidates at that time. Therefore, an object-based representation of a multisensory effect can be readily represented as an effect on the endpoint room. The above considerations not only prompt the estimation of ARR, but also prompt the adjustment / enhancement of ARR to AM, which is more suitable for inversion. AM can be used to map the nth sensory object to a vector A representing all actuators, for example, as follows:

[0140] In some examples, modifying ARR to AM in box 410 can make invertibility more feasible by doing one or both of the following: filling the gap by artificially expanding the actuator response to complete support on X; orWhere feasible, reduce the overlap between actuators so that the response of a point object (an object of infinitesimal size) is an impulse of the form A.

[0141] Even with modifications, the actuator mapping AM may still suffer from rank insufficiency (ignoring zero activations). In other words, the volume corresponding to multiple sensory objects may contain one or more of the same actuators in the environment at a given time. In addition to the rest of the environment and actuator data 004, the MS renderer 001 may also be configured to use other data (such as sensory object priority metadata) to determine the actuator control signal 310.

[0142] Multiple modalities In some examples, when scheduling the simultaneous rendering of two or more multi-sensory effects (such as touch and light), the ARR and / or AM corresponding to each modality may be evaluated and / or modified. A model that links the effects of two sensory modes should be adopted to ensure that they are suitable for the joint rendering of multi-modal sensory effects. The ARR and / or AM of multiple modalities may not overlap across the entire endpoint and may not support each other (areas that can be activated by both modalities) because different devices are responsible for activating different modalities. This mutually supporting overlap (or lack thereof) may occur in any of the multiple domains, including the frequency domain and the spatial domain. Furthermore, the transient or dynamic response of the actuator and the corresponding physical characteristics under different modes should also be considered. For example, due to the time lag between sending the airflow actuation signal and achieving the desired airflow, there may be a time offset between the ARR of the lighting effect and the ARR of the airflow effect.

[0143] Figure 5 illustrates example elements of another system for creating and playing an MS experience. As with other figures provided herein, the types and numbers of elements shown in Figure 5 are provided only as examples. Other implementations may include more, fewer, and / or different types and numbers of elements. According to some examples, system 500 may be or may include one or more devices configured to perform at least some of the methods disclosed herein. In some examples, system 500 may include one or more instances of the control system 110 of Figure 1A configured to perform at least some of the methods disclosed herein.

[0144] According to this example, the system shown in Figure 5 is an example of the system shown in Figure 3. In this example, the system shown in Figure 5 is a “scene” embodiment in which video, audio, and lighting effects are combined to create an MS experience.

[0145] In this example, system 500 includes a lighting creation tool 100, which is an example of the content creation tool 000 described with reference to FIG. 3. The lighting creation tool 100 is configured to design and output object-based lighting data 505', which is output individually or in combination with corresponding audio data 111' and / or video data 112' according to a particular implementation.The object-based light data 505' may include timestamp information and information indicating light object attributes, etc. In some cases, timestamp information may be used to synchronize effects associated with the object-based light data 505' with audio data 111' and / or video data 112', which may also include timestamp information.

[0146] In this example, the object-based light data 505' includes a light object and corresponding light metadata. For example, the object-based light data may include light object location metadata, light object color metadata, light object size metadata, light object intensity metadata, light object shape metadata, light object diffusion metadata, light object gradation metadata, light object priority metadata, light object layer metadata, or a combination thereof. Although in this example, the content creation tool 100 is shown as providing a stream of object-based light data 505' to the experience player 102, in alternative examples, the content creation tool 100 may generate object-based light data 505' that is stored for later use. An example of a graphical user interface for a light object-based content creation tool is described below.

[0147] In this example, system 500 includes an experience player 102 configured to receive light data 505', audio data 111', and video data 112' based on the object-based specification page 21 / 33, 25 CN 121866536 A, and to provide the object-based light data 505 to a lighting renderer 501, the audio data 111 to an audio renderer 106, and the video data 112 to a video renderer 107. As described elsewhere herein, the object-based light data 505, audio data 111, and video data 112 may include timestamp information that can be used to synchronize MS effects with audio and / or video effects. According to some examples, experience player 102 may be a media player, game engine, or component integrated into a television, DVD player, soundbar, set-top box, or service provider media device (such as Chromecast, Apple TV, or Amazon Fire TV). In some examples, experience player 002 can be configured to receive encoded object-based light data 505' and encoded audio data 111' and / or encoded video data 112', for example, as part of the same bitstream as the encoded audio data 111' and / or encoded video data 112'. According to some examples, experience player 102 can be configured to extract object-based light data 505 from the content bitstream and provide decoded object-based light data 505 to a lighting renderer 501, and provide decoded audio data 111 to an audio renderer 106, and...The decoded video data 112 is provided to the video renderer 107. In some examples, the experience player 002 may be configured to allow control of configurable parameters in the scene renderer 501, such as immersion intensity. Some examples are described below.

[0148] As described above, according to some embodiments, system 500 may include one or more instances of the control system 110 of FIG1A configured to perform at least some of the methods disclosed herein. In some such examples, one instance of control system 110 may implement scene creation tool 100, and another instance of control system 110 may implement experience player 002. In some examples, one instance of control system 110 may implement audio renderer 006, video renderer 007, scene renderer 501, or a combination thereof. According to some examples, an instance of control system 110 configured to implement experience player 002 may also be configured to implement audio renderer 006, video renderer 007, scene renderer 501, or a combination thereof.

[0149] In some examples, the room descriptor of the environment and lighting data 104 can describe the size and orientation of the playback environment itself to establish a relative or absolute coordinate system to which all objects are located. The room descriptor information can indicate or describe the physical dimensions of the playback environment (e.g., expressed in physical distance units, such as meters). In some such examples, sensory object positioning, sensory object size, and sensory object orientation can be described in units relative to the room size, such as in the range of -1 to 1. The room descriptor can also describe a preferred viewing position. For example, in a living room, the display screen can be considered as in front, in some cases as in the center of the front, and the floor and ceiling can be considered as vertical boundaries. In some such examples, the room descriptor can also indicate the boundaries corresponding to the left, right, front, and back walls relative to the in front position. According to some examples, at least some of the room descriptor information can be provided in matrix form. In some such examples, the matrix can be a 3 × 3 matrix, where a row or column corresponds to one dimension of three-dimensional space.

[0150] According to this example, system 500 includes a lighting renderer 501 configured to render object-based light data 505 into luminaire control signals 515 based at least in part on environment and actuator data 104. In this example, lighting renderer 501 is configured to output the luminaire control signals 515 to lamp controllers 103 configured to control luminaires 108. Luminaires 108 may include a single controllable light source, a group of controllable light sources (such as controllable light strips), or a combination thereof. In some examples, lighting renderer 501 may be configured to manage various types of light object metadata layers, examples of which are provided herein. According to some examples, lighting renderer 501 may be configured to render luminaires at least in part based on the viewer's perspective.The actuator signal. If the viewer is in a living room, which includes a television (TV) screen, in some examples, the lighting renderer 501 can be configured to render the actuator signal relative to the TV screen. However, in virtual reality (VR) use cases, the lighting renderer 501 can be configured to render the actuator signal relative to the position and orientation of the user's head. In some examples, the lighting renderer 501 can receive input from the playback environment (such as light sensor data corresponding to ambient light, camera data corresponding to the person's position or orientation, etc.) to enhance the rendering effect.

[0151] In some examples, the lighting renderer 501 is configured to receive object-based light data 505, which includes light objects and object-based lighting metadata indicating the expected lighting environment, as well as environment and lighting data 104 corresponding to luminaire 108 and other features of the local playback environment, which may include, but are not limited to, reflective surfaces, windows, uncontrollable light sources, shading features, etc. In this example, the local playback environment includes one or more speakers 109 and one or more display devices 510.

[0152] According to some examples, the lighting renderer 501 is configured to calculate, at least in part, how to activate various controllable luminaires 108 based on object-based light data 505 and environment and luminaire data 104. For example, environment and luminaire data 104 may indicate the geometric location of the luminaire 108 in the environment, luminaire type information, etc. In some examples, the lighting renderer 501 may be configured to determine, at least in part, which luminaires will be activated based on position metadata and size metadata associated with each light object, for example, by determining which luminaires are located within a volume of the playback environment, which corresponds to the position and size of the light object at a specific time indicated by light object timestamp information. In this example, the lighting renderer 501 is configured to send luminaire control signals 515 to the luminaire controller 103 based on environment and luminaire data 104 and object-based light data 505. The luminaire control signals 515 may be sent via one or more transmission mechanisms, application programming interfaces (APIs), and protocols. For example, these protocols may include Hue API, LIFX API, DMX, Wi-Fi, Zigbee, Matter, Thread, Bluetooth Mesh, or other protocols.

[0153] In some examples, the lighting renderer 501 may be configured to determine the drive level of the lighting environment expected by the author(s) of the object-based light data 505 for each of one or more controllable light sources. According to some examples, the lighting renderer 501 may be configured to output drive levels to at least one of the controllable light sources.

[0154] According to some examples, the lighting renderer 501 may be configured to, based on content metadata, user input (selection mode)...The lighting fixtures may be folded into one or more parts of the lighting fixture map by means of the fixture's configuration, limitations and / or configuration, other factors, or combinations thereof. For example, the lighting renderer 501 may be configured to render the same control signals to two or more different lights in the playback environment. In some such examples, the two or more lights may be positioned close to each other. For example, the two or more lights may be different lights with the same actuator, such as different bulbs in the same light bulb. Instead of calculating slightly different control signals for each bulb, the lighting renderer 501 may be configured to reduce computational overhead, increase rendering speed, etc., by rendering the same control signals to two or more different but closely spaced lights.

[0155] In some examples, the lighting renderer 501 may be configured to spatially blend the object-based light data 505. For example, if the object-based light data 505 is generated for a single plane (such as a horizontal plane), in some cases, the lighting renderer 501 may be configured to project the light objects of the object-based light data 505 onto an upper hemispherical surface (e.g., above the user's actual or intended head position) to enhance the experience.

[0156] According to some examples, the lighting renderer 501 may be configured to apply one or more thresholds, such as one or more spatial thresholds, one or more brightness thresholds, etc., when rendering actuator control signals to the light actuators of the playback environment. In some cases, such thresholds may prevent some light objects from activating some lights.

[0157] Light objects can be used for a variety of purposes, such as creating room atmosphere, providing spatial information about people or objects, enhancing special effects, creating greater interactivity and immersion, diverting viewer attention, emphasizing content, etc. Content creators may express some of these purposes (at least in part) based on sensor object metadata types and / or attributes (typically applicable to various types of sensor objects), such as object metadata indicating the location and size of sensor objects.

[0158] For example, the priority of sensor objects (including but not limited to light objects) can be indicated by sensor object priority metadata. In some such examples, sensor object priority metadata is considered when multiple sensor objects are simultaneously mapped to the same light fixture in the playback environment. This priority can be indicated by light priority metadata. In some examples on pages 23 / 33 of the specification, 27 CN 121866536 A, priority may not need to be indicated via metadata. For example, the MS renderer 001 may prioritize moving sensory objects (including but not limited to light objects) over stationary sensory objects.

[0159] A light object may cause multiple lights to be activated depending on its position and size, as well as the position of the light fixture in the playback environment. In some examples, when the size of a light object contains multiple lights, the renderer may apply one or more thresholds (e.g.,One or more spatial thresholds or one or more brightness thresholds) to prevent objects from activating some of the included lights.

[0160] Examples of using lighting maps In some implementations, a lighting map (which is an instance of an actuator map (AM) that includes a description of lighting in the playback environment) may be provided to the scene renderer 501. In some such examples, the environment and luminaire data shown in FIG5 may include a lighting map. According to some examples, the lighting map may be heterocentric, for example, indicating light attenuation based on absolute spatial coordinates; while in other examples, the lighting map may be egocentric, for example, projecting light onto a sphere of the intended viewing position and orientation. In the case of a sphere, in some examples, the lighting map may be projected onto a two-dimensional (2D) surface, for example, to use a 2D image texture in the processing. In any case, the lighting map should indicate the capabilities and lighting settings of the playback environment (e.g., a room). In some embodiments, the lighting map may not be directly related to the physical room characteristics, for example, if some user preference-based adjustments have been made.

[0161] In some examples, there may be a lighting map for each luminaire or each light in the playback environment. According to some examples, the intensity of the light indicated by the light map may be inversely correlated with the distance to the center of the lamp, or may be approximately inversely correlated with the distance to the center of the lamp (e.g., within ±5%, ±10%, ±15%, ±20%, etc.). The intensity value of the light map can indicate the intensity or effect of the light object on the lamp. For example, when the light object approaches the bulb, the lighting renderer 501 can be configured to determine that the intensity of the bulb will increase as the distance between the light object and the bulb decreases. The lighting renderer 501 can be configured to determine the rate of this transition based at least in part on the light intensity indicated by the light map.

[0162] In this general rendering space, in some examples, the lighting renderer 501 can be configured to calculate the light activation index for each lamp using a dot product multiplication between the light object and the light map, for example, as follows:

[0163] In the foregoing equation, Y represents the light activation index, LM represents the lighting map, and Obj represents the mapping of the light object. The light activation index indicates the relative light intensity of the actuator control signal output by the lighting renderer 501, based on the overlap between the light object and the light diffusion from the luminaire. In some examples, the lighting renderer 501 may use the maximum or nearest distance from the light object to the luminaire, or other geometric measures, as part of determining the light intensity. In some implementations, the lighting renderer 501 may not calculate the light activation index, but may instead refer to a lookup table to determine the light activation index.

[0164] The lighting renderer 501 may repeat one of the above processes to determine the light activation index for all light objects and all controllable lights in the playback environment. Thresholding light objects that have a minimal impact on the luminaires can help reduce complexity.For example, if the effect of a light object would cause the luminaire activation to be below a certain threshold percentage (e.g., below 10%, below 5%, etc.), the lighting renderer 501 might ignore the effect of the light object.

[0165] The lighting renderer 501 can then use the generated light activation matrix Y, as well as various other attributes, such as selected imaging rules (indicated by light object metadata or renderer configuration) or the priority of the light object, to determine which objects are rendered by which lights and how. Rendering a light object into a luminaire control signal may involve: • changing the brightness of the light object according to the distance between the light object and the luminaire; • mixing the colors of multiple light objects rendered simultaneously (multiplexed) by a single luminaire; or • changing any of the above options based on the light object priority.

[0166] Mapping Lighting Actuator Specification 24 / 33 pages 28 CN 121866536 A This section describes apparatus, systems, and methods for mapping and characterizing luminaires and the environment in which the luminaires are deployed. This process typically involves obtaining luminaire information, such as luminaire location and capability information, and obtaining environmental information corresponding to the environment in which the luminaires are deployed. Luminaire information may also include light direction information, which indicates the direction(s) in which each lamp emits light in the environment. Environmental information may include the location, orientation, and size of one or more environmental features, such as structural elements of the environment, furniture, etc. Environmental information may include environmental feature color information, environmental feature reflectivity information, or both.

[0167] In some examples, the user can manually input luminaire information and environmental information. According to some examples, a device or system may estimate luminaire information and environmental information, for example, by using camera data (e.g., video feeds) and inertial measurement data from sensing devices, to obtain data corresponding to the effect produced when the luminaires are actuated by illuminating the environment (e.g., a room). Detailed examples are provided below.

[0168] Some examples involve obtaining a light room response (LRR) based on luminaire information and environmental information, and determining a light map (LM) based on the LRR. LRR and LM are instances of ARR and AM, respectively. LRR describes the joint response of the lamp and the environment in which the lamp is deployed.

[0169] Figure 6 depicts the light room response (LRR) of a single luminaire in egocentric polar coordinates. The image on which Figure 6 is based was generated using first-order reflection simulation and manually input data. The x, y, and z coordinates in Figure 6 are defined as follows: - The origin is at the user's head; - Positive x-values ​​are in the direction to the user's right; - Positive y-values ​​are in the direction behind the user; and - Positive z-values ​​are in the direction above the user.

[0170] As can be seen from Figure 6, the lamp is located below and in front of the user's head, and is biased to the left of the user's head. From the user's perspective, the lamp is located on the floor and is shining upwards along the left wall of the room. Figure 6 shows the illumination field of the lamp in the room.

[0171] Video-based analysis and light mapping generationLRR indicates the response of each light in an environment (e.g., a room). In some examples, LRR can be estimated by manually inputting the location, characteristics, room dimensions, etc., of each light. However, some disclosed devices, methods, and systems provide an automated and more accurate light mapping process involving direct observation of the actual light in the room using video-based analysis. In addition to a more detailed characterization, such a method also allows for compensation for subtle differences between the actual manufactured light and the specifications provided on a datasheet.

[0172] Figure 7 illustrates example elements of a light mapping system. As with other figures provided herein, the types and numbers of elements shown in Figure 7 are provided by way of example only. Other implementations may include more, fewer, and / or different types and numbers of elements. According to some examples, system 700 may be or may include one or more devices configured to perform at least some of the methods disclosed herein. In some examples, system 700 may include one or more instances of the control system 110 of Figure 1A configured to perform at least some of the methods disclosed herein.

[0173] According to this example, the light mapping system 700 includes the following elements: Module 701 - a system controller, which in this example includes a state machine; 702 - a calibration signal generator: a module configured to generate calibration signals for controlling the illumination of lamps 750A, 750B, and 750C to support the positioning, mapping, and characterization process; 703 - a lamp controller module configured to control lamps 750A, 750B, and 750C according to a signal 721 from the calibration signal generator 702; 704 - a sensing device that provides observations as input to the light mapping system 700. In some examples, the sensing device can be a mobile phone; 705 - Optical positioning and segmentation module, configured to determine the positioning of lights 750A, 750B, and 750C in both the video feed 723 and the room 730; 706 - Room characterization module, configured to estimate parameters of the room 730, such as (multiple) wall colors, (multiple) ceiling colors, (multiple) floor colors, (multiple) furniture colors, surface reflectivity, etc.; 707 - Light characterization module, configured to estimate parameters of lights 750A, 750B, and 750C, including luminous sensitivity and color matching; 708 - Analysis and fusion module, configured to fuse the estimated room parameters 727 and the estimated light parameters 728; 709 - Positioning module, configured to determine the position and orientation of the sensing device 704 within the room 730. In this example, the localization module 709 is configured to perform a simultaneous localization and mapping (SLAM) process based at least in part on camera data (video feed 723 in this example) and inertial measurement data from sensing device 704; 710 -A camera system includes one or more camera devices located on sensing device 704. The one or more camera devices may include one or more visible light cameras, one or more infrared light cameras, one or more stereo cameras, one or more lidar devices, etc.; 712 – an auxiliary sensor, which is part of sensing device 704 and physically coupled to camera system 710 in this example. In this example, auxiliary sensor 712 is configured to acquire inertial measurement data, which may include gyroscope data, accelerometer data, or both; 730 – an environment (a room in this example) where a number of illuminators can be controlled to form a flexible lighting rendering system; and 750A, 750B, and 750C – multiple controllable lights within room 730. While this example includes three controllable lights in room 730, other examples may include different numbers of controllable lights, such as 5, 8, 10, 15, 20, or more.

[0174] Signal 720 - Control signal sent by lamp controller 703 to lamps 750A, 750B, and 750C; 721 - Calibration signal sent by calibration signal generator to lamp controller 703 for controlling lamps 750A, 750B, and 750C; 723 - Video feed(s) generated by camera 710 of sensing device 704; 724 - Sensor feed(s) generated by sensor 712 of sensing device 704; 725 - Data indicating the estimated position and orientation of sensing device 704 within room 730; 726 - Estimated lamp positioning and its segmentation mask in the video feed; 727 - Estimated room parameters of room 730; 728 - Estimated lamp parameters of lamps 750A, 750B, and 750C; 729 - The jointly estimated room and light parameters may include the estimation results of the analysis and fusion module 708 on their coherent convergence to the solution; and 730 - a command from the system controller 701 for controlling the generation of the calibration signal 721.

[0175] In this example, the room and light characterization process is iterative. As shown in FIG7, this iterative process is indicated by sending the jointly estimated room and light parameters 729 back to the room characterization module 706 and the light characterization module 707 and the system controller 701. The light sensed by the camera is the product of the light source reflecting off the object(s). For example, consider the pixels in the camera feed corresponding to the wall illuminated by one of the lamps during a certain stage of the calibration measurement process, on pages 26 / 33 of 30 CN 121866536 A. In this case, the lamp and the wall have an infinite number of possible illumination and reflection spectra, which can produce the color sensed by the camera.For content-aware consumption calibration, the control system does not need to estimate the spectral response because typical humans have trichromatic vision, requiring only three digits (even fewer for those with color vision deficiencies) to represent and reproduce a color. However, because each primary color of the luminaire needs to be characterized separately, the color sensed by the camera still introduces ambiguity during the calibration process. For example, without any prior wall color information, the room characterization module 706 might initially estimate the wall color as blue based on the blue light reflected from the wall illuminated by the luminaire. Simultaneously, the lamp characterization module 707 might initially estimate that the lamp's emissivity is low, or the room characterization module might similarly estimate that the wall's reflectivity is low. However, the room characterization module 706 and the lamp characterization module 707 can update the wall color estimate and luminaire emissivity based on the different colors of light subsequently reflected from the wall illuminated by a particular luminaire. Over time, the control system can use observations of multiple parts of the room and multiple luminaires to jointly estimate room and lamp characteristics.

[0176] Optical calibration signal / encoded transmission According to some examples, the calibration signal 121 generated by the calibration signal generator 102 may be or may include a set of color signals that perform the following functions: 1. Illuminating the room to provide data to the positioning module 709 to determine the position and orientation of the sensing device 704 within the room 730, for example, according to a SLAM process; 2. Identifying individual lamps; 3. Locating individual lamps; and 4. Estimating the illuminance of individual lamps in the room.

[0177] These functions may have different requirements for the calibration signal. In some examples, these functions may appear at different stages of the mapping process, and in some examples, these stages may correspond to the numerical order in the list above.

[0178] According to some examples, the calibration signal 121 may be configured to cause relatively low-frequency optical modulation, for example, to change the color, intensity, etc. of individual lamps every 100 milliseconds (ms), every 200 ms, every 300 ms, every 400 ms, every 500 ms, etc. In such an example, calibration signal 121 can cause the lamp to modulate at a rate compatible with the sampling rate of the camera performing the calibration.

[0179] In some alternative examples, the lamp can be controlled by simple on / off keying, for example, sequentially turning each lamp in the room on and off so that each controllable lamp in the room can be detected and located. Subsequently, in some examples, calibration signal 121 may be modified to help estimate the light response in the room.

[0180] According to some examples, calibration signal 121 can be configured to cause the lamp to modulate in a way that allows each lamp to be uniquely identified. For example, each calibration signal 121 can be configured to be on / off keyed according to a unique binary sequence. After each lamp in the video feed 723 is detected, the task of identifying each lamp becomes identifying the binary sequence.For this task, there are conventional estimators. The choice of the only code set used as the calibration signal modulator (amplitude in the case of on / off keying) may vary depending on the specific implementation. Some coding families, such as Gold codes, can provide the best separation. The length of Gold codes is 2N-1, where N is the order of the Gold code, which is greater than 4 and preferably odd. For N = 5, a separation of 12 dB can be obtained between any two calibration signals obtained from this coding family. The length of these sequences is 31, which will be a compatible length for mapping applications for cameras calibrated at 30 frames per second (FPS).

[0181] Wall Color The wall color plays an important role in the output color of the light source. For example, projecting blue light onto a red wall will produce a purple hue. Specification 27 / 33 pages 31 CN 121866536 A

[0182] Bidirectional Reflectance Measurement (BRDF) In this context, "bidirectional" means that the reflectance measurement is performed over a range of angles (e.g., more than 180 degrees), rather than in a single direction. Such measurements are useful because the reflection from one side or direction of some surfaces in the playback environment may differ from that from another side or direction. The reflectivity of structural elements in the environment (such as walls, floors, ceilings, etc.) will determine the number and directionality or "directivity" of the light source. In particular, when the light source is a spotlight, the reflectivity or diffuser of the walls will affect the diffusion of light and the shape of the impulse response.

[0183] Spectral response can be measured using a typical RGB camera sensor, which measures the combination of direct light and reflected light from structural elements (such as walls). However, capturing the spectral power distribution of the light source and the spectral reflectivity of the structural elements is beneficial. The spectral power distribution (SPD) of a light source describes the power magnitude of each wavelength in the visible spectrum. The spectral reflectivity (SR) of a structural element describes how much light of each wavelength is reflected back.

[0184] Each person responds differently to different wavelengths of light, which is called the human color matching function (CMF). These CMF responses are directly related to people's perception of color. Some implementations can be configured to render a scene uniquely based on the CMF response of a single individual, for example, as follows:

[0185] In the preceding equations, the directionality of the light map (LM) is driven by the BRDF of the structuring element. The spectral response is determined by the product of the SPD and SR. The final rendered response will correspond to the human CMF.

[0186] A lamp characterizes each lamp as having a set of colors it can produce. In most additive color systems, this set of colors can be decomposed into hue and luminance. However, some light sources may have additional settings / bulbs with higher luminance, thus forming a non-additive color system. One goal of generating the light map is to determine the relationship between the driving signal and the light output. Therefore,A comprehensive characterization of the light source is desirable. The light output can be a simple XYZ or a more unique SPD, as previously described. The Encoded Emissions section describes a method for efficiently characterizing lights. Characterizing the diffusion or directionality of the light source is also important. Spotlights, light strips, or bulbs can have unique effects on viewers and rooms.

[0187] Device Occlusion In some cases, light may be partially obscured by objects such as furniture or other lights. When working in a polar coordinate system, lights on a straight line can be merged into a single light source. If there are one or more light obstructions or “dead zones” from the viewer’s perspective, such obstructions should be included in the LM. For example, in the LM, a dead zone can be represented by a low or zero value of the coordinates corresponding to the dead zone. The renderer can use information about the dead zone, for example, to better position light objects.

[0188] Device Color Matching When generating a lighting scene, it is important to ensure that all lights and displays produce similar colors. Making a light fixture too bright or with a different hue can reduce the sensory experience. In some cases, it may be easier and / or more beneficial to “embed” color and intensity calibration into the light mapping and / or the luminaire itself. In the latter case, a drive value that produces a certain color can be specified, rather than specifying a color and an associated drive value. In the case of a very simple LM, it can only provide a certain percentage of the light object of specification 28 / 33 pages 32 CN 121866536 A for use, so a full characterization of the light source will allow the control system to match colors.

[0189] Figure 8 shows an example of a color volume. The color within a color volume is indicated by the fill density of a specific area, with the lowest fill density corresponding to red and the highest fill density corresponding to violet. Color key 801 indicates the fill density corresponding to red, orange, yellow, green, blue, indigo, and violet (ROYGBIV). To ensure accurate matching of all colors, some implementations involve defining the color volume of all lights, reflective surfaces, etc. in a room, as shown in Figure 8, for example. The control system can determine the overlapping volume of all lights, reflective surfaces, etc. in the room. Not all light sources can produce colors outside of this overlapping volume. The mapping itself can be mathematical or based on a three-dimensional (3D) lookup table (LUT) to convert driving values ​​into hue / intensity. For color matching, using a perceptually uniform and uncorrelated space (such as ICtCp) is helpful, as shown in Figure 8.

[0190] External Light Sources In many cases, there may be one or more uncontrollable light sources in the environment. Whether uncontrollable light sources include lights from other rooms or ambient light from windows, these lights affect the accurate light mapping process. In some examples, uncontrollable light sources can be characterized independently of the impulse response as the underlying environment layer. According to some examples, the light object renderer can compensate for uncontrollable light sources by changing the light object rendering, for example, by increasing the light pairs in the area affected by the uncontrollable light source.The driving signal of the image. In some examples, an LM can be a “baseline” light map obtained with all controllable lights off. In some such examples, future LM map captures may offset the effect of the baseline light map.

[0191] Baseline Environment In some cases, one may prefer to maintain the ambient light baseline level in the playback environment. This ambient light baseline level can be characterized in a manner similar to that of an external light source. However, in some implementations, the renderer can be configured to at least maintain this ambient light baseline level. In some such examples, the LM can indicate the baselines of controllable and uncontrollable lights, respectively.

[0192] Manual Input and Mixing System Figure 9 depicts an alternative method for generating LRR. In this example, LRR is generated without using video analysis. In this example, ambient descriptor data 901 and light descriptor data 902 are provided to simulator 903. According to this example, simulator 903 is implemented by an instance of control system 110 of Figure 1A and is configured to estimate LRR 904 based on ambient descriptor data 901 and light descriptor data 902. In some examples, the simulation may only consider the diffusion component of light reflection. For example, using the Lambert model, the LRR of the j-th luminaire at the location can be simulated as follows:

[0193] , where: - represents the back-incidence vector; - the subscript D indicates the diffusion component; - represents the intensity of the light emitted by the j-th luminaire; - represents the position of the j-th luminaire; - N represents the normal to the surface at the location; and - α and β represent parameters to change the attenuation effect caused by the distance between the luminaire and the location p.

[0194] In addition to modeling diffuse reflection, some examples also involve modeling the specular component of light reflection to obtain more accurate results, especially at the end of the specification on pages 29 / 33 of CN 121866536 A where the surface is smoother. This can be achieved by modeling the specular component separately and then simply adding it to the diffusion component simulated using, for example, the method described above. One way to simulate specular reflection is to use a Blinn model, for example, as shown below:

[0195] , where: B represents the normalized angle bisector of the reflection vector and the anti-incidence vector, and; - represents the reflection vector; - represents the normalized anti-incidence vector; - represents the normalized reflection vector; - Φ represents a parameter describing the specular (gloss) degree of the surface; and - U represents the user position.

[0196] At any time, should be set to zero.

[0197] The above expressions can be combined to obtain a comprehensive simulated ARR, as shown below:

[0198] The model illustrated here is not based on a physically accurate model. If sufficiently accurate prior information about the surfaces and lighting fixtures in the room is available, then a more accurate physically based rendering can be used.

[0199] The environmental descriptor data 901 includes content referred to herein as “environmental information,” which relates to the playback environment, including controllable luminaires. For example, environmental descriptor data 901 may include the location, orientation, and size of one or more environmental features, environmental feature color information, environmental feature reflectivity information, or a combination thereof. According to this example, lamp descriptor data 902 includes at least information about controllable luminaires in the playback environment. For example, lamp descriptor data 902 may include luminaire positioning data, lamp parameter data (including but not limited to light color data), light directivity data (indicating the direction(s) of light(s) provided by one or more lamps of the luminaire), etc. In some cases, at least a portion of the lamp descriptor data 902 may come from published specifications regarding the capabilities, parameters, etc., of one or more commercially available lighting products. In some alternative examples, environmental descriptor data 901 may include luminaire positioning data, light directivity data, or both.

[0200] In some examples, environmental descriptor data 901, lamp descriptor data 902, or both may be manually entered. For example, control system 110 may control a display (not shown) to provide a graphical user interface (GUI) through which a user can interact to input environment descriptor data 901, lamp descriptor data 902, or both.

[0201] However, in some “hybrid” examples, at least a portion of environment descriptor data 901, lamp descriptor data 902, or both may be automatically generated. For example, in some embodiments, the location, orientation, and size of one or more environmental features, the location of one or more luminaires, etc., may be determined based on a Simultaneous Localization and Mapping (SLAM) process, rather than being manually entered.

[0202] Figure 10 shows examples of information that may be provided to the simulator of Figure 9. Figure 10 depicts the location and illumination direction of luminaires within environment 1000. In these examples, a single lamp is depicted as a cone, with the apex of the cone indicating location and the axis of the cone indicating illumination direction. For example, downlight 1005a is located on the ceiling of environment 1000. The illumination direction of downlight 1005a is indicated by arrow 1004 on pages 30 / 33 of CN 121866536 A. In this example, environment 1000 includes ceiling downlight 1005, television (TV) gradient strip light 1010, and floor light 1015.

[0203] The LRR is enhanced to produce a light-mapping LRR indicator response within the environment. In some examples, the LRR may be enhanced so that the enhanced LRR can also map areas in the environment that light cannot reach, such as behind a TV or sofa, and this serves as an activation mechanism. Such enhancement is an example that may be referred to herein as “filling the gap” in the LRR. Some examples may involve other modifications to the LRR. In this document, the resulting enhanced LRR may be referred to as light mapping (LM).

[0204] Figures 11A and 11B illustrate examples of the enhanced LRR. Figures 11A and 11B show the enhanced LRR (which may be referred to as LM), corresponding to two portions of the TV gradient light strip. Figure 11A shows LM 1110a produced by enhancing the TV gradient light strip LRR in the positive x-direction, while Figure 11B shows LM 1110b produced by enhancing the TV gradient light strip LRR in the positive z-direction. In both examples, the enhanced LRR now includes a mapping from behind the TV. If a light object is placed behind the TV, using an LM such as LM 1110a or LM 1110b will allow the lighting renderer 101 to project that light object from the spatial domain onto the light, thereby activating the entire TV gradient strip. Therefore, LM 1110a and LM 1110b provide a natural extension of the TV gradient light strip LRR, allowing the lighting renderer 101 to map the entire spatial domain back to the light in the environment 1100.

[0205] As described elsewhere herein, light mapping and repulsion generation regularization, in some examples, the lightscape renderer 101 uses a reverse LM to project light objects in the spatial domain onto the lights of the playback environment. In some cases, even if the LRR is enhanced or otherwise modified, the resulting LM may still include one or more ill-posed spatial coordinates. Some such examples may occur in locations where two lights respond strongly and overlap, but these lights also cover large, non-intersecting areas. If light objects are placed in overlapping areas, the lightscape renderer 101 may cause a large portion of the environment to be illuminated, and if another light object is located in a large area where the lights do not intersect, it may reduce the ability to render that other light object.

[0206] Some implementations involve mitigating the effects caused by ill-posed spatial coordinates. Some examples involve regularization techniques that can minimize such effects. Some such examples involve applying functions of the SoftMax type, which may only be applied to overlapping areas as in the preceding examples.

[0207] Alternatively or additionally, some examples involve implementing one or more repulsion functions configured to deviate the trajectory of light objects (or other sensory objects) from one or more ill-posed spatial coordinates. Such a rejection function can be applied to the light object itself in the lighting renderer 101 to prevent the light object from entering regions that include ill-posed spatial coordinates. In some such examples, rejection metadata can be appended to the LM, and the lighting renderer 101 can apply the corresponding rejection function. According to some examples, the control system can be configured to determine regions that include ill-posed spatial coordinates and generate or select corresponding rejection functions for regions where these ill-posed spatial coordinates appear. For example, the process of determining regions that include ill-posed spatial coordinates can involve analyzing synthetic LRRs generated by simulation, or analyzing LRRs generated by analyzing video feeds, for example, as described herein with reference to Figure 7.

[0208] Figure 12 is a flowchart outlining an example of a method that can be performed by a device or system such as those disclosed herein. As with other methods described herein, the blocks of method 1200 need not be performed in the indicated order. In some embodiments, one or more blocks of method 1200 may be performed simultaneously. Furthermore, some embodiments of method 1200 may include more or fewer blocks than those shown and / or described. The blocks of method 1200 may be performed by one or more devices, which may be (or may include) one or more instances of a control system (such as control system 110 shown and described above in Figure 1A).

[0209] In this example, block 1205 relates to obtaining positioning information of a set of controllable actuators in the environment by the control system. In this example, the set of controllable actuators includes one or more luminaires, one or more tactile devices, one or more airflow control devices, or combinations thereof.

[0210] According to this example, block 1210 relates to the controllable actuator capability information of each controllable actuator in the set of controllable actuators obtained by the control system. In some cases, the controllable actuator capability information may include the actuation type (e.g., light, tactile, airflow, etc.). For a luminaire, the controllable actuator capability information may indicate the color, intensity, etc., of the light(s) that can be emitted. In some examples, method 1200 may relate to obtaining controllable actuator actuation direction information. For a luminaire, the controllable actuator direction information may indicate the direction(s) in which the luminaire is currently configured to transmit light.

[0211] In this example, block 1215 relates to the control system obtaining environmental information corresponding to the environment. In some cases, the environmental information may include environmental feature color information, environmental feature reflectivity information, or a combination thereof. According to some examples, the environmental information may at least include the location, orientation, and size of one or more environmental features. In some cases, the one or more environmental features may include one or more structural elements of the environment, such as one or more walls, ceilings, floors, or combinations thereof. In some examples, the one or more environmental features may include one or more furniture items. The environmental information may include furniture location information.

[0212] According to this example, block 1220 relates to the control system generating an actuator-room response (ARR) based at least in part on the controllable actuator positioning information, the controllable actuator capability information, and the environmental information, the actuator-room response summarizing the environment's response to activation of the set of controllable actuators.

[0213] In some examples, one or more of blocks 1205 may relate to a manual input process. The manual input process may include one or more processes described with reference to FIG. 9. In some such examples, the controllable actuator positioning information is obtained.This may involve receiving controllable actuator capability information via an interface system (such as a user interface system or a network system).

[0214] However, according to some examples, one or more of blocks 1205 may involve an automated process, such as one or more processes described with reference to FIG. 7. For example, obtaining the controllable actuator positioning information in block 1205 may involve the control system receiving camera data from one or more cameras, and the control system determining the controllable actuator positioning information based at least in part on the camera data. The camera data may include an image of a set of controllable actuators. The camera data may include optical images, depth images, or a combination thereof.

[0215] In some examples, method 1200 may involve the control system determining controllable actuator actuation direction information based at least in part on the camera data. The controllable actuator actuation direction information may include light direction information, air movement direction information, or both.

[0216] According to some examples, method 1200 may involve the control system obtaining inertial measurement data. The inertial measurement data may include gyroscope data, accelerometer data, or both. In some examples, the controllable actuator positioning information can be determined based on a Simultaneous Localization and Mapping (SLAM) process, at least in part, based on the camera data and the inertial measurement data.

[0217] In some examples, the controllable actuator may include a controllable luminaire. In some such examples, method 1200 may also involve sending a calibration signal from a control system to the controllable luminaire while one or more cameras are acquiring camera data. The calibration signal may modulate the light intensity, light color, or both of the controllable luminaire. In some such examples, obtaining the controllable actuator capability information and obtaining the environmental information may both be based at least in part on the camera data and the inertial measurement data. According to some examples, the calibration signal may control the controllable luminaire to emit light according to a code or pattern that enables the control system to distinguish the lighting effects produced by each of the controllable luminaires. In some examples, the calibration signal may be or may include a sequence having bounded cross-correlation within a set, such as a Gold sequence. According to some examples, method 1200 may involve color matching of the light emitted by two or more of the controllable luminaires.

[0218] In some examples, obtaining the controllable actuator capability information may involve a lamp characterization process, which may or may involve determining the characteristics of lamps in the environment. According to some such examples, method 1200 may involve a light localization and segmentation process based at least in part on the camera data. The lamp characterization process may be based at least in part on the light localization and segmentation process. For example, the light localization and segmentation process may involve analyzing the pixels of the camera data and determining which pixels correspond to individual lamps in the environment.

[0219] According to some examples, obtaining the environmental information may involve an environmental characterization process that determines environmental characteristic color information and environmental characteristic reflectivity information. In some examples, obtaining the controllable actuator capability information and obtaining the environmental information may involve an iterative process that determines the characteristics of the lights in the environment, determines environmental characteristic color information, and determines environmental characteristic reflectivity information.

[0220] In some examples, method 1200 may involve the control system modifying the ARR to produce an actuator map (AM). The modification may include regularization, filling gaps in the ARR by increasing one or more volumes of the environment affected by actuator responses, reducing one or more overlapping volumes of the environment affected by multiple actuator responses, or a combination thereof. The AM may indicate the effect of each controllable actuator on the environment. The AM may be mathematically applicable to inversion. The AM may allow object-based sensory data to be rendered as actuator commands. In some examples, the ARR may be represented as a matrix, and the AM may be represented as a modified version of the matrix. The matrix may include values ​​corresponding to actuator indices, executed commands, and spatial coordinates.

[0221] According to some examples, method 1200 may involve the control system identifying one or more ill-posed spatial coordinates of the AM. Some such examples may involve the control system generating one or more repulsion functions configured to deviate the trajectory of the sensory object from the one or more ill-posed spatial coordinates. Some such examples may involve storing or providing the one or more repulsion functions along with the AM.

[0222] The foregoing description illustrates various embodiments of the present disclosure and examples of how aspects of the present disclosure may be implemented. The foregoing examples and embodiments should not be considered as limited embodiments, but are presented to illustrate the flexibility and advantages of the present disclosure as defined by the appended claims. Other arrangements, embodiments, implementations, and equivalents will be apparent to those skilled in the art based on the foregoing disclosure and the appended claims, and may be employed without departing from the spirit and scope of the present disclosure as defined by the claims. Instruction manual page 33 / 33, page 37, CN 121866536 A, Figure 1A, Instruction manual drawing page 1 / 13, page 38, CN 121866536 A, Figure 1B, Figure 2, Instruction manual drawing page 2 / 13, page 39, CN 121866536 A, Figure 3, Instruction manual drawing page 3 / 13, page 40, CN 121866536 A, Figure 4A, Instruction manual drawing page 4 / 13, page 41, CN 121866536 A, Figure 4B, Instruction manual drawing page 5 / 13, page 42, CN 121866536 A, Figure 5, Instruction manual drawing page 6 / 13, page 43, CN 121866536 A, Figure 6, Instruction manual drawing.Page 7 / 13, 44 CN 121866536 A, Figure 7, Appendix Figure; Page 8 / 13, 45 CN 121866536 A, Figure 8, Appendix Figure; Page 9 / 13, 46 CN 121866536 A, Figure 9, Appendix Figure; Page 10 / 13, 47 CN 121866536 A, Figure 10, Appendix Figure; Page 11 / 13, 48 CN 121866536 A, Figure 11A, Figure 11B, Appendix Figure; Page 12 / 13, 49 CN 121866536 A, Figure 12, Appendix Figure; Page 13 / 13, 50 CN 121866536 A

Claims

1. A method comprising: The control system obtains controllable actuator positioning information of a set of one or more controllable actuators in the environment, wherein the one or more controllable actuators include one or more lamps, one or more tactile sensors, one or more airflow control devices, or combinations thereof; The control system obtains controllable actuator capability information for each controllable actuator in the set of one or more controllable actuators; The control system obtains environmental information corresponding to the environment. as well as The control system generates an actuator-room response (ARR) based at least in part on the controllable actuator positioning information, the controllable actuator capability information, and the environmental information. The actuator-room response summarizes the environment's response to one or more activations of the set of one or more controllable actuators.

2. The method as described in claim 1, wherein, The environmental information includes at least one or more locations, one or more orientations, and one or more dimensions of one or more environmental features, wherein the one or more environmental features include one or more structural elements of the environment.

3. The method as described in claim 2, wherein, The one or more structural elements include one or more walls, ceilings, floors, or combinations thereof.

4. The method as claimed in claim 2 or claim 3, wherein, The one or more environmental features include one or more furniture items, and the environmental information includes furniture location information.

5. The method according to any one of claims 2 to 4, wherein, The environmental information includes environmental characteristic color information, environmental characteristic reflectance information, or a combination thereof.

6. The method according to any one of claims 1 to 5, wherein, Obtaining the positioning information of the controllable actuator involves: The control system receives camera data from one or more cameras, the camera data including one or more images of the set of one or more controllable actuators; as well as The control system determines the positioning information of the controllable actuator based at least in part on the camera data.

7. The method of claim 6, further comprising determining actuation direction information of the controllable actuator by the control system based at least in part on the camera data, and wherein, The controllable actuator actuation direction information includes light direction information, air movement direction information, or both.

8. The method of claim 6 or claim 7, wherein, The camera data includes one or more optical images, one or more depth images, or a combination thereof.

9. The method of any one of claims 6 to 8, further comprising obtaining inertial measurement data by the control system, wherein, The inertial measurement data includes gyroscope data, accelerometer data, or both, and wherein the controllable actuator positioning information is determined based on a simultaneous localization and mapping (SLAM) process based at least in part on the camera data and the inertial measurement data.

10. The method according to any one of claims 1 to 9, wherein, The one or more controllable actuators include one or more controllable luminaires, and the method further includes the control system sending one or more calibration signals to the one or more controllable luminaires while the one or more cameras are acquiring the camera data, the one or more calibration signals causing the one or more controllable luminaires to modulate light intensity, light color, or both, and wherein obtaining the controllable actuator capability information and obtaining the environmental information are both based at least in part on the camera data and the inertial measurement data.

11. The method of claim 10, wherein, The one or more calibration signals control the one or more controllable luminaires to emit light according to codes or patterns that enable the control system to distinguish one or more lighting effects produced by each of the one or more controllable luminaires.

12. The method of claim 10 or claim 11, wherein, The one or more calibration signals comprise sequences that have bounded cross-correlation within the set.

13. The method of any one of claims 10 to 12, further comprising color matching of the light emitted by two or more of the controllable luminaires.

14. The method according to any one of claims 10 to 13, wherein, Obtaining the controllable actuator capability information involves a lamp characterization process that determines the characteristics of one or more lamps within the environment.

15. The method of claim 14, further comprising at least in part a light localization and segmentation process based on the camera data, wherein, The lamp characterization process is based at least in part on the light localization and segmentation process.

16. The method of claim 15, wherein, The light localization and segmentation process involves analyzing the pixels of the camera data and determining which pixels correspond to various lights in the environment.

17. The method according to any one of claims 10 to 16, wherein, Obtaining the environmental information involves an environmental characterization process that determines environmental characteristic color information and environmental characteristic reflectance information.

18. The method according to any one of claims 10 to 17, wherein, Obtaining the controllable actuator capability information and obtaining the environmental information involves an iterative process of determining one or more characteristics of the lights in the environment, determining environmental characteristic color information, and determining environmental characteristic reflectivity information.

19. The method according to any one of claims 1 to 18, wherein, Obtaining the controllable actuator positioning information involves receiving controllable actuator capability information via an interface system.

20. The method of claim 19, wherein, The interface system includes a user interface system.

21. The method of any one of claims 1 to 20, further comprising modifying the ARR by the control system to generate an actuator map (AM).

22. The method of claim 21, wherein, The modifications include regularization, filling gaps in the ARR by increasing one or more volumes of the environment affected by actuator responses, reducing one or more overlapping volumes of the environment affected by multiple actuator responses, or combinations thereof.

23. The method of claim 21 or claim 22, wherein, The AM indicates the effect of each controllable actuator on the environment.

24. The method according to any one of claims 21 to 23, wherein, The AM is mathematically applicable to inversion.

25. The method according to any one of claims 21 to 24, wherein, The AM allows object-based sensory data to be rendered as actuator commands.

26. The method of any one of claims 21 to 25, further comprising: The control system identifies one or more ill-posed spatial coordinates of the AM; The control system generates one or more repulsion functions, which are configured to cause the trajectory of the sensory object to deviate from the one or more ill-posed spatial coordinates. as well as The one or more rejection functions are stored or provided together with the AM.

27. The method according to any one of claims 21 to 26, wherein, The ARR is represented as a matrix, and the AM is represented as a modified version of the matrix.

28. The method of claim 27, wherein, The matrix includes values ​​corresponding to one or more actuator indices, one or more execution commands, and spatial coordinates.

29. An apparatus configured to perform the method as claimed in any one of claims 1 to 28.

30. A system configured to perform the method as claimed in any one of claims 1 to 28.

31. One or more non-transitory computer-readable media having instructions stored thereon for controlling one or more devices to perform the method as described in any one of claims 1 to 28.

Images