Conditionally adjusting light effects based on second audio channel content

Abstract

A system for controlling multiple lighting devices (11-15) to present a light effect accompanying audiovisual content (81) is configured to associate a first audio channel / object with a first lighting device (12) and not with a second lighting device (13) based on a first characteristic of a first audio channel / object (86). The first characteristic indicates the location of the audio source. The system is also configured to associate a second audio channel (87) with the first and second lighting devices. The system is further configured to determine whether a second audio content in the second audio channel meets predetermined criteria and to determine a first light effect based on the determined chromaticity. If one or more predetermined criteria are not met, the light intensity is based on the first audio content in the first audio channel / object; otherwise, the light intensity is based on the second audio content.

Description

Technical Field

[0001] The present invention relates to a system for controlling multiple lighting devices to present light effects accompanying audiovisual content, the audiovisual content including an audio portion and a video portion, the audio portion including multiple audio channels.

[0002] The present invention also relates to a method for controlling multiple lighting devices to present light effects accompanying audiovisual content, the audiovisual content including an audio portion and a video portion, the audio portion including multiple audio channels.

[0003] The present invention also relates to a computer program product that enables a computer system to execute this method. Background Technology

[0004] The experience of visual and auditory content can greatly benefit from dynamic lighting systems. Entertainment lighting systems, such as Hue Sync, can significantly alter the user's viewing experience by presenting light colors extracted from the scene in real time or programmed offline. Furthermore, accompanying audio can be considered, where, for example, the intensity of the audio can be used to modulate the presented lighting effects.

[0005] For example, US2010 / 265414 A1 discloses that a scene accompanied by high-intensity audio can be presented with a higher-intensity lighting effect than the same scene accompanied by low-intensity audio. US2010 / 265414A1 further discloses that video-based ambient lighting characteristics intended for display on the left side of a display can be combined with audio-based ambient lighting data involving the left channel, while video-based ambient lighting characteristics intended for display on the right side of a display can be combined with audio-based ambient lighting data involving the right channel. US 9763021 B1 discloses a system for visually displaying positional audio information using multiple light sources based on multi-channel audio information.

[0006] The drawback of US2010 / 265414 A1 is that the disclosed system does not utilize newer audio formats to create enhanced ambient lighting effects for both the video and audio portions of the audiovisual content. Summary of the Invention

[0007] The first objective of this invention is to provide a system capable of creating enhanced ambient lighting effects based on both the video and audio portions of audiovisual content.

[0008] A second objective of this invention is to provide a method for creating enhanced ambient lighting effects based on both the video and audio portions of audiovisual content.

[0009] In a first aspect of the invention, a system for controlling a plurality of lighting devices to present light effects accompanying the presentation of audiovisual content, the audiovisual content including an audio portion and a video portion, the audio portion including a plurality of audio channels, the system including at least one input interface, at least one transmitter, and at least one processor, the at least one processor being configured to: obtain the audiovisual content via the at least one input interface; determine a first characteristic of a first audio channel among the plurality of audio channels or an audio object contained in the audio portion, the first characteristic indicating an audio source location; and associate the first audio channel or the audio object with a first lighting device among the plurality of lighting devices based on the first characteristic, wherein the association is based on the audio source location relative to the location of the first lighting device.

[0010] At least one processor is further configured to: determine a second characteristic of a second audio channel among the plurality of audio channels; associate the second audio channel with the first lighting device and a second lighting device among the plurality of lighting devices based on the second characteristic, wherein the first audio channel is not associated with the second lighting device; determine whether second audio content in the second audio channel meets one or more predetermined criteria; determine at least chroma based on the video portion of the audiovisual content; determine a first lighting effect based on the determined chroma, wherein if the one or more predetermined criteria are not met, the light intensity of the first lighting effect is based on the first audio content in the first audio channel or the audio object, and if the one or more predetermined criteria are met, the light intensity of the first lighting effect is based on the second audio content in the second audio channel; and control the first lighting device to present the first lighting effect via the at least one transmitter.

[0011] More and more people are enjoying surround sound configurations at home, and most TV shows and movies today feature surround sound audio. By considering the semantic properties of audio channels and, optionally, audio objects when determining lighting effects, it is possible to determine what is happening in the audio channels and, optionally, audio objects, and thus achieve a more immersive entertainment experience.

[0012] This is not achieved by simply modulating the light effects using the intensity of the audio content, but rather by modulating the light effects using audio channels and, optionally, the semantics of the audio objects. For example, a low-frequency effects (subwoofer) channel might affect all connected lights, so that a deep bass effect (like a loud bang) can not only be heard but also seen throughout the entire entertainment area. This can be contrasted with audio effects on the left / right audio channels (also known as side channels), which can be used to modulate light effects only on lighting devices located on the left / right sides.

[0013] Using the system described above, a first audio channel or audio object can be mapped to a first lighting device based on the location of an audio source associated with the first audio channel relative to the location of the first lighting device, and a second audio channel can be mapped to multiple lighting devices including the first lighting device. The sound effects reproduced on the first audio channel or audio object can be more precisely located by the user compared to the sound effects reproduced on the second audio channel, for example, because the second audio channel is a low-frequency effect (subwoofer) channel.

[0014] By mapping a first audio channel or audio object to fewer lighting devices (e.g., a single lighting device) than a second audio channel, the light effect determined based on the first audio content on the first audio channel or audio object reflects the corresponding sound effect with a more specific location, while the light effect determined based on the second audio content on the second audio channel reflects the corresponding sound effect with a less specific location.

[0015] The chromaticity (and optionally the entire color) of the light effect is determined based on at least the video portion of the audiovisual content, and the light intensity of the light effect is determined based on at least the audio portion of the audiovisual content, wherein the light intensity is based on the second audio content in the second audio channel at certain moments (e.g., in the case of a loud event) to obtain the optimal light experience. Optionally, the brightness of the color is also determined based on the audio portion of the audiovisual content.

[0016] The second characteristic may not indicate the location of the audio source. For example, the second characteristic could indicate whether the second audio channel is a low-frequency effects channel. Alternatively, for example, the first characteristic could be determined by the audio object, and the second characteristic could indicate the desired speaker location for the second audio channel.

[0017] The at least one processor can be configured to further determine the light intensity of the first light effect based on the first audio content in the first audio channel or the audio object if one or more predetermined criteria are met. By always determining the light intensity based on the first audio content in the first audio channel, a less intense light experience can be obtained, which is preferred by some users. For example, the highest light intensity can only be achieved if a loud event exists in both the first and second audio channels. By always determining the light intensity based on the first audio content in the audio object, the audio object is emphasized in the light effect.

[0018] The at least one processor can be configured to further determine the light intensity of the first lighting effect based on the video portion of the audiovisual content. This can be used to ensure that the light intensity matches not only the audio portion but also the video portion. The user can configure whether the light intensity of the lighting effect should be determined based on the video portion of the audiovisual content.

[0019] The at least one processor can be configured to determine whether the second audio content in the second audio channel meets the one or more predetermined criteria by determining whether the audio intensity of the second audio content exceeds a threshold. Therefore, if a loud event (e.g., a loud explosion) exists in the second audio channel, the light intensity can be determined based on the second content in the second audio channel.

[0020] The at least one processor can be configured to select a spatial region in the current frame of the video portion depending on whether one or more predetermined criteria are met, and to determine at least the chroma from the selected spatial region only. While the chroma (or overall color) of the lighting effect is preferably determined based on the video portion of the audiovisual content, the audio portion may still have some influence on the chroma (or overall color). For example, if a loud event is detected in the low-frequency effects channel, the color of the lighting effect for the lighting device located on the upper left side can be extracted from the center region of the video frame, and otherwise the color of the lighting effect for the lighting device located on the upper left side can be extracted from the left-side region of the video frame.

[0021] The first characteristic of the first audio channel can be determined, and the at least one processor can be configured to: determine whether the audio intensity of the first audio content exceeds a threshold; select a spatial region in the current frame of the video portion based on whether the audio intensity of the first audio content exceeds the threshold; and determine at least the chroma from the selected spatial region only. Which spatial region of the video portion to extract the chroma (or the entire color) from may depend not only on the second audio content in the second audio channel but also on the first audio content in the first audio channel. For example, when a loud event is detected in the second audio channel, the light intensity for the lighting effects of all lighting devices can be determined based on the second audio content in the second audio channel; and if a loud event is also detected in the first audio channel, the color of the lighting effects for the lighting devices located on the upper left side can be extracted from the central region of the video frame, and otherwise the color of the lighting effects for the lighting devices located on the upper left side can be extracted from the left region of the video frame.

[0022] The at least one processor can be configured to determine one or more speaker signals for a loudspeaker based on the audio portion of the audiovisual content. The at least one processor can be configured to determine the light intensity of the first light effect based on the one or more speaker signals. Instead of determining the light intensity of the first light effect directly based on the audio portion of the audiovisual content, the light intensity can be determined based on one or more speaker signals. This may be beneficial if the user's audio system cannot reconstruct the specified audio source location in the content sufficiently close, or if the user's audio system enhances the specified audio effect in the audiovisual content.

[0023] As an example of the latter, there exist audio upmixing algorithms that create pseudo-channels for traditional content that does not include those channels (e.g., Dolby Surround without height channels). An example of such an upmixing algorithm is DTS Virtual:X. Other audio analysis steps can also be performed based on one or more speaker signals, such as determining first and second characteristics and / or determining whether second audio content in a second audio channel meets one or more predetermined criteria.

[0024] Alternatively, the at least one processor can be configured to further determine the light intensity of the first light effect based on information about available speakers and / or information about the 3D audio virtualization used. This may be beneficial if the user's audio system is unable to reconstruct the location of the specified audio source in the content close enough.

[0025] In a second aspect of the invention, a method for controlling a plurality of lighting devices to present light effects accompanying the presentation of audiovisual content, the audiovisual content including an audio portion and a video portion, the audio portion including a plurality of audio channels, the method comprising: obtaining the audiovisual content; determining a first characteristic of a first audio channel among the plurality of audio channels or an audio object contained in the audio portion, the first characteristic indicating an audio source location; and associating the first audio channel or the audio object with a first lighting device among the plurality of lighting devices based on the first characteristic, wherein the association is based on the audio source location relative to the location of the first lighting device.

[0026] The method further includes: determining a second characteristic of a second audio channel among the plurality of audio channels; associating the second audio channel with a first lighting device and a second lighting device among the plurality of lighting devices based on the second characteristic, wherein the first audio channel is not associated with a second lighting device; determining whether second audio content in the second audio channel meets one or more predetermined criteria; determining at least chroma based on the video portion of the audiovisual content; determining a first lighting effect based on the determined chroma, wherein if the one or more predetermined criteria are not met, the light intensity of the first lighting effect is based on the first audio content in the first audio channel, and if the one or more predetermined criteria are met, the light intensity of the first lighting effect is based on the second audio content in the second audio channel; and controlling the first lighting device to present the first lighting effect. The method can be executed by software running on a programmable device. This software can be provided as a computer program product.

[0027] In addition, a computer program for implementing the methods described herein is provided, as well as a non-transitory computer-readable storage medium for storing the computer program. The computer program may be downloaded or uploaded to an existing device, for example, or stored during the manufacture of these systems.

[0028] A non-transitory computer-readable storage medium stores at least one portion of software code that, when executed or processed by a computer, is configured to perform executable operations for controlling a plurality of lighting devices to present light effects accompanying audiovisual content, the audiovisual content including an audio portion and a video portion, the audio portion including a plurality of audio channels.

[0029] The operable operations include: obtaining the audiovisual content; determining a first characteristic of a first audio channel among the plurality of audio channels or an audio object contained in the audio portion, the first characteristic indicating the location of an audio source; and associating the first audio channel or the audio object with a first lighting device among the plurality of lighting devices based on the first characteristic, wherein the association is based on the location of the audio source relative to the location of the first lighting device.

[0030] The executable operations further include: determining a second characteristic of a second audio channel among the plurality of audio channels; associating the second audio channel with the first lighting device and a second lighting device among the plurality of lighting devices based on the second characteristic, wherein the first audio channel is not associated with the second lighting device; determining whether second audio content in the second audio channel meets one or more predetermined criteria; determining at least chroma based on the video portion of the audiovisual content; determining a first lighting effect based on the determined chroma, wherein if the one or more predetermined criteria are not met, the light intensity of the first lighting effect is based on the first audio content in the first audio channel, and if the one or more predetermined criteria are met, the light intensity of the first lighting effect is based on the second audio content in the second audio channel; and controlling the first lighting device to present the first lighting effect.

[0031] As those skilled in the art will appreciate, aspects of the present invention can be embodied as devices, methods, or computer program products. Therefore, aspects of the present invention can take the form of entirely hardware embodiments, entirely software embodiments (including firmware, resident software, microcode, etc.), or embodiments combining software and hardware aspects, which are generally referred to herein as “circuit,” “module,” or “system.” The functionality described in this disclosure can be implemented as algorithms executed by a computer’s processor / microprocessor. Furthermore, aspects of the present invention can take the form of computer program products embodied in one or more computer-readable media having computer-readable program code embodied thereon (e.g., stored thereon).

[0032] Any combination of one or more computer-readable media can be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any suitable combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the context of this invention, a computer-readable storage medium can be any tangible medium that can contain or store a program used by or in conjunction with an instruction execution system, apparatus, or device.

[0033] Computer-readable signal media may include propagated data signals having computer-readable program code embodied therein (e.g., in baseband or as part of a carrier wave). Such propagated signals may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium that is not a computer-readable storage medium and may convey, propagate, or deliver a program used by or in conjunction with an instruction execution system, apparatus, or device.

[0034] The program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic, cable, RF, or any suitable combination thereof. The computer program code for carrying out the operations of various aspects of the invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Java™, Smalltalk, or C++) and conventional procedural programming languages ​​(such as the "C" programming language or similar programming languages). The program code may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet provided by an Internet service provider).

[0035] The various aspects of the invention are described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, particularly a microprocessor or central processing unit (CPU), to produce a machine such that the instructions, executable via the processor of the computer, other programmable data processing apparatus, or other device, create means for implementing the functions / actions specified in the flowchart illustrations and / or one or more block diagram blocks.

[0036] These computer program instructions may also be stored in a computer-readable medium that can instruct a computer, other programmable data processing apparatus, or other device to operate in a particular manner, such that the instructions stored in the computer-readable medium produce an article of writing, which includes instructions that implement functions / actions specified in flowcharts and / or one or more block diagrams.

[0037] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide for implementing the functions / actions specified in the flowchart and / or one or more block diagram boxes.

[0038] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which includes one or more executable instructions for implementing a specified logical function(s). It should also be noted that in some alternative implementations, the functions described in the blocks may not appear in the order shown in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or sometimes these blocks may be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a system based on dedicated hardware, or a combination of dedicated hardware and computer instructions, that performs the specified function or action. Attached Figure Description

[0039] Referring to the accompanying drawings, these and other aspects of the invention will be clear and further illustrated by way of example, in which:

[0040] Figure 1 This is a block diagram of the first embodiment of the system;

[0041] Figure 2 This is a block diagram of a second embodiment of the system;

[0042] Figure 3 This is a flowchart of the first embodiment of the method;

[0043] Figure 4 This is a flowchart of the second embodiment of the method;

[0044] Figure 5 This is a flowchart of the third embodiment of the method;

[0045] Figure 6 An example of a room where five entertainment lights have already been installed is shown.

[0046] Figure 7 This is a flowchart of the fourth embodiment of the method;

[0047] Figure 8 This is a flowchart of the fifth embodiment of the method; and

[0048] Figure 9 This shows the effect when the second audio channel is loud and the first audio channel is not loud. Figure 8 An example of a method to control a light;

[0049] Figure 10 This shows the effect when both the second and first audio channels are loud. Figure 8 An example of a method to control a light;

[0050] Figure 11 This is a flowchart of the sixth embodiment of the method; and

[0051] Figure 12 This is a block diagram of an exemplary data processing system for performing the methods of the present invention.

[0052] Corresponding elements in the accompanying drawings are represented by the same reference numerals. Detailed Implementation

[0053] Figure 1 A first embodiment of a system for controlling multiple lighting devices to produce light effects accompanying audiovisual content is shown: an HDMI module 1. The audiovisual content includes an audio component and a video component. The audio component includes multiple audio channels. For example, the HDMI module 1 could be a Hue Play HDMI Sync Box. Figure 1 In the example, HDMI module 1 controls five lighting devices 11-15.

[0054] exist Figure 1 In this example, HDMI module 1 can control lighting devices 11-15 via bridge 19. For example, bridge 19 could be a Hue bridge. Bridge 19 communicates with lighting devices 11-15, for example, using Zigbee technology. HDMI module 1 is connected to wireless LAN access point 21, for example, via Wi-Fi. Bridge 19 is also connected to wireless LAN access point 21, for example, via Wi-Fi or Ethernet.

[0055] Alternatively or additionally, HDMI module 1 may be able to communicate directly with bridge 19, for example, using Zigbee technology, and / or may be able to communicate with bridge 19 via the Internet / cloud. Alternatively or additionally, HDMI module 1 may be able to control lighting devices 11-15 without a bridge (e.g., directly via Wi-Fi, Bluetooth, or Zigbee, or via the Internet / cloud).

[0056] Wireless LAN access point 21 is connected to the Internet 25. Media server 27 is also connected to the Internet 25. For example, media server 27 may be a server for video-on-demand services such as Netflix, Amazon Prime Video, Hulu, HBO Max, Paramount+, Peacock, Disney+, or Apple TV+. HDMI module 1 is connected via HDMI to display device 23 and local media receivers 31 and 32. Local media receivers 31 and 32 may include one or more streaming media devices or content generation devices (such as Apple TV, Microsoft Xbox, and / or Sony PlayStation), and / or one or more cable or satellite TV receivers. Display device 23 is connected to audio system 35, for example, via HDMI ARC. Audio system 35 is connected to speakers 36.

[0057] In an alternative embodiment, the system for controlling multiple lighting devices to produce light effects accompanying the presentation of audiovisual content is a display device. In this alternative embodiment, the HDMI module logic may be built into the display device. Media receivers 31 and 32 may then also be included in the display device (e.g., a smart TV).

[0058] HDMI module 1 includes a receiver 3, a transmitter 4, a processor 5, and a memory 7. The processor 5 is configured to acquire audiovisual content from media receivers 31 or 32 via the receiver 3, determine a first characteristic of a first audio channel or an audio object contained in an audio portion among a plurality of audio channels, and associate the first audio channel or audio object with a first lighting device among lighting devices 11-15 based on the first characteristic. The first characteristic indicates the location of the audio source, and the association is based on the location of the audio source relative to the location of the first lighting device.

[0059] The processor 5 is further configured to determine a second characteristic of a second audio channel among a plurality of audio channels, and to associate the second audio channel with the first lighting device and the second lighting device among lighting devices 11-15 based on the second characteristic. The first audio channel is not associated with the second lighting device.

[0060] The processor 5 is further configured to determine whether the second audio content in the second audio channel meets one or more predetermined criteria, determine at least the chromaticity based on the video portion of the audiovisual content, determine a first lighting effect based on the determined chromaticity, and control a first lighting device via the transmitter 4 to present the first lighting effect.

[0061] If one or more predetermined criteria are not met, the light intensity of the first light effect is based on the first audio content in the first audio channel or in the audio object, and if one or more predetermined criteria are met, the light intensity of the first light effect is based on the second audio content in the second audio channel.

[0062] exist Figure 1 In the embodiment of the HDMI module 1 shown, the HDMI module 1 includes a processor 5. In alternative embodiments, the HDMI module 1 includes multiple processors. The processor 5 of the HDMI module 1 may be a general-purpose processor (e.g., ARM-based) or a dedicated processor. The processor 5 of the HDMI module 1 may run an operating system such as Unix. The memory 7 may include one or more memory cells. For example, the memory 7 may include solid-state memory.

[0063] For example, receiver 3 and transmitter 4 can communicate with bridge 19 using one or more wired or wireless communication technologies (such as Zigbee), and can communicate with display device 23 and local media receivers 31 and 32 using one or more wired or wireless communication technologies (such as HDMI). In alternative embodiments, multiple receivers and / or multiple transmitters are used instead of a single receiver and a single transmitter. Figure 1 In the illustrated embodiment, a separate receiver and a separate transmitter are used. In an alternative embodiment, receiver 3 and transmitter 4 are combined into a transceiver. HDMI module 1 may include other components typically used in network devices, such as a power connector. The invention can be implemented using a computer program running on one or more processors.

[0064] exist Figure 1 In one embodiment, the system of the present invention is an HDMI module. In an alternative embodiment, the system may be another device, such as a mobile device, laptop, personal computer, bridge, media presentation device, streaming media device, or internet server. Figure 1 In one embodiment, the system of the present invention includes a single device. In an alternative embodiment, the system includes multiple devices.

[0065] Figure 2 A second embodiment of a system for controlling multiple lighting devices to produce light effects accompanying audiovisual content is shown: a mobile device 51. The mobile device 51 can be, for example, a smartphone or tablet. The lighting devices 11-15 can be controlled by the mobile device 51 via a bridge 19. The mobile device 51 is connected, for example, via Wi-Fi to a wireless LAN access point 21.

[0066] Mobile device 51 includes a receiver 53, a transmitter 54, a processor 55, a memory 57, and a display 59. The video portion is preferably displayed on display device 23, but may also be displayed on the display 59 of mobile device 51. In the former case, the audio portion may be presented, for example, on display device 23 or an audio system connected to display device 23. Figure 2 (Not shown in the middle)

[0067] The processor 55 is configured to acquire audiovisual content via the receiver 53, determine a first characteristic of a first audio channel among a plurality of audio channels or an audio object contained in an audio portion, and associate the first audio channel or audio object with a first lighting device among the lighting devices 11-15 based on the first characteristic. The first characteristic indicates the location of the audio source, and the association is based on the location of the audio source relative to the location of the first lighting device.

[0068] The processor 55 is further configured to determine a second characteristic of a second audio channel among a plurality of audio channels, and to associate the second audio channel with the first lighting device and the second lighting device among lighting devices 11-15 based on the second characteristic. The first audio channel is not associated with the second lighting device.

[0069] The processor 55 is further configured to determine whether the second audio content in the second audio channel meets one or more predetermined criteria, determine at least the chromaticity based on the video portion of the audiovisual content, determine a first lighting effect based on the determined chromaticity, and control a first lighting device via the transmitter 54 to present the first lighting effect.

[0070] If one or more predetermined criteria are not met, the light intensity of the first light effect is based on the first audio content in the first audio channel or in the audio object, and if one or more predetermined criteria are met, the light intensity of the first light effect is based on the second audio content in the second audio channel.

[0071] exist Figure 2 In the embodiment of the mobile device 51 shown, the mobile device 51 includes a processor 55. In alternative embodiments, the mobile device 51 includes multiple processors. The processor 55 of the mobile device 51 may be a general-purpose processor (e.g., from ARM or Qualcomm) or a dedicated processor. The processor 55 of the mobile device 51 may run an operating system such as Android or iOS. For example, the display 59 may be a touchscreen display. The display 59 may include, for example, an LCD or OLED display panel. The memory 57 may include one or more memory cells. For example, the memory 57 may include solid-state memory.

[0072] For example, receiver 53 and transmitter 54 can use one or more wireless communication technologies (such as Wi-Fi (IEEE 802.11)) to communicate with wireless LAN access point 21. In alternative embodiments, multiple receivers and / or multiple transmitters are used instead of a single receiver and a single transmitter. Figure 2 In the illustrated embodiment, a separate receiver and a separate transmitter are used. In an alternative embodiment, receiver 53 and transmitter 54 are combined into a transceiver. Mobile device 51 may further include a camera (not shown). For example, the camera may include a CMOS or CCD sensor. Mobile device 51 may include other components typically used in mobile devices, such as a battery and power connector. The invention can be implemented using a computer program running on one or more processors.

[0073] exist Figure 2 In one embodiment, lighting devices 11-15 are controlled via bridge 19. In an alternative embodiment, one or more of lighting devices 11-15 are controlled without a bridge (e.g., directly via Bluetooth). If lighting devices 11-15 are controlled without a bridge, wireless LAN access point 21 may not be necessary. Mobile devices may connect to the Internet 25 via a mobile communication network (e.g., 5G) instead of via wireless LAN access point 21.

[0074] Figure 3 The first embodiment of a method for controlling multiple lighting devices to present light effects accompanying audiovisual content is shown. The audiovisual content includes an audio component and a video component. The audio component includes multiple audio channels. The method can be, for example, […]. Figure 1 HDMI module 1 or Figure 2 The mobile device 51 is used to execute this.

[0075] Step 101 involves acquiring the audiovisual content. Steps 103 and 107 are performed after step 101. Step 103 involves determining a first characteristic of an audio object included in a first audio channel or audio segment among multiple audio channels. The first characteristic indicates the location of the audio source. In many audio formats, most audio channels are associated with the desired speaker locations in a room (e.g., front left, front right, center). Some audio formats (such as Dolby Atmos and DTS:X) support the use of audio objects. Audio objects are typically associated with the location of that audio object in virtual 3D space.

[0076] Step 105 includes obtaining the position of the first lighting device, such as its x / y / z position. This can be done manually, but can also be done automatically, for example, via RF sensing. Step 105 further includes associating a first audio channel or audio object with the first lighting device among a plurality of lighting devices based on a first characteristic determined in step 103, wherein the association is based on the location of the audio source relative to the position of the first lighting device.

[0077] Step 107 includes determining a second characteristic of the second audio channel among a plurality of audio channels. Step 109 includes associating the second audio channel with a first lighting device and a second lighting device among a plurality of lighting devices based on the second characteristic determined in step 107. The first audio channel is not associated with a second lighting device. For example, a low-frequency effects (LFE) channel may be associated with all lighting devices in the room, or a left audio channel (at listener level or height level) may be associated with a plurality of lighting devices on the left side of the room.

[0078] exist Figure 3 In one embodiment, steps 103 and 105 are performed at least partially in parallel with steps 107 and 109. In an alternative embodiment, step 107 is performed after step 105, or step 103 is performed after step 109.

[0079] After steps 105 and 109 are completed, step 111 is performed. Step 111 includes determining whether the second audio content in the second audio channel meets one or more predetermined criteria. For example, step 111 may include determining whether the audio intensity of the second audio content exceeds a threshold. Next, step 113 includes determining at least the chroma (and optionally the entire color) based on the video portion of the audiovisual content.

[0080] Step 115 includes determining a first lighting effect based on the determined chromaticity and based on the light intensity. If one or more predetermined criteria are not met, the light intensity of the first lighting effect is based on a first audio content in a first audio channel. If one or more predetermined criteria are met, the light intensity of the first lighting effect is based on a second audio content in a second audio channel.

[0081] Additionally, the light intensity of the first light effect may depend on the distance between the speaker (or, for example, the position of an audio object presented using multiple speakers) and the lighting device presenting the first light effect. In this case, for example, if two lighting devices are located on the upper left, but one is farther from the left channel speaker(s), the adjustment for the farther lighting device may be less than the adjustment for the closer lighting device. Step 117 includes controlling the first lighting device to present the first light effect determined in step 115.

[0082] Figure 4The diagram illustrates a second embodiment of a method for controlling multiple lighting devices to present light effects accompanying audiovisual content. The audiovisual content includes an audio component and a video component. The audio component includes multiple audio channels. This method can be implemented using, for example... Figure 1 HDMI module 1 or Figure 2 The mobile device 51 is used to execute this.

[0083] Step 101 involves acquiring the audiovisual content. In step 121, a mapping from audio channels to lighting equipment is determined. First, the characteristics of each audio channel are determined. Some audio channels are associated with an audio source location, and in this case, the determined characteristics indicate that audio source location. For example, the left front channel in the Dolby Digital encoded audio section is associated with the desired left front speaker location. However, not all audio channels are associated with an audio source location. An example is the LFE (subwoofer) channel.

[0084] Step 121 involves determining the location of all lighting devices among the multiple lighting fixtures, such as their x / y / z positions. This can be done manually, but can also be done automatically, for example, via RF sensing. The characteristics determined for the LFE audio channel (also referred to as the second audio channel in this embodiment) indicate that it is an LFE channel and do not indicate the location of the audio source, because humans cannot locate the source of low-frequency sounds. Therefore, in step 121, the LFE audio channel is associated with all lighting devices among the multiple lighting fixtures.

[0085] Other audio channels (also referred to as the first audio channel in this embodiment) are associated with lighting devices based on the location of the audio source associated with the respective audio channel and the location of the corresponding lighting device. For example, the left front audio channel may be associated with the left front lighting device. The type and capability of the lighting device may affect how the mapping between audio channels and lighting devices is performed. Furthermore, the type and capability of the lighting device may also affect how the luminance and chromaticity for that lighting device are determined in step 123. For example, a point light source may be treated differently from a linear light source (such as a light strip).

[0086] Step 111 includes determining whether the second audio content in the second audio channel (i.e., the LFE audio channel) meets one or more predetermined criteria. Figure 4 In one embodiment, step 111 includes determining whether the audio intensity of the second audio content exceeds a threshold.

[0087] Next, in step 123, lighting effects are determined for multiple lighting devices. The chromaticity of each lighting effect is determined based on the video portion of the audiovisual content. Furthermore, the light intensity is determined for each lighting effect. The chromaticity is extracted from a specific spatial region of the video frame in the video portion. In this embodiment, this spatial region depends on the position of the lighting devices. For example, the chromaticity of the lighting effect to be presented by the lighting device on the left side is extracted from the region on the upper left of the video frame, and the chromaticity of the lighting effect to be presented by the lighting device on the right side is extracted from the region on the upper right of the video frame.

[0088] exist Figure 4 In one embodiment, the light intensity of the light effect is based solely on the audio portion of the audiovisual content. In an alternative embodiment, the light intensity of the light effect is also based on the video portion of the audiovisual content. For example, the intensity can be extracted from the same spatial region from which chroma is extracted, and this intensity can then be adjusted based on the audio portion. The adjusted intensity is then used as the light intensity of the light effect.

[0089] If it is determined in step 111 that the audio intensity in the second audio channel (i.e., the LFE audio channel) does not exceed a threshold, the light intensity of the lighting effect of the lighting device is determined based on the first audio content in the first audio channel associated with the lighting device. For example, the light intensity of the left front lighting device is then determined based on the audio content in the left front audio channel.

[0090] If, in step 111, it is determined that the audio intensity in the second audio channel (i.e., the LFE audio channel) exceeds a threshold, then the light intensity of each light effect for each lighting device is determined based on the second audio content in the second audio channel. Figure 4 In one embodiment, the light intensity is based solely on the second audio content. In an alternative embodiment, the light intensity of the lighting effect of a lighting device is determined based on both the first audio content in a first audio channel associated with a lighting device and the second audio content in a second audio channel (i.e., the LFE audio channel).

[0091] Next, step 125 includes controlling the lighting device to present the light effect determined in step 123. Step 111 is repeated after step 125, and then the method proceeds as follows: Figure 4 The procedure is as shown. Since the characteristics of the audio channels generally do not change during the viewing or listening content, step 121 is not repeated in this embodiment (for the same viewing or listening content). Figure 4 In one embodiment, the audiovisual content is fully acquired before the lighting effects are determined. In an alternative embodiment, the audiovisual content may be streamed and thus partially acquired. In this alternative embodiment, step 121 may be performed, for example, after a first portion of the audiovisual content has been acquired.

[0092] Figure 5The image illustrates a third embodiment of a method for controlling multiple lighting devices to produce light effects accompanying audiovisual content. The audiovisual content includes an audio component and a video component. The audio component includes multiple audio channels. The method can be implemented using, for example... Figure 1 HDMI module 1 or Figure 2 The mobile device 51 is used to execute this.

[0093] Step 101 includes obtaining audiovisual content. In step 141, a mapping from audio channels to lighting devices is determined. Figure 5 Step 141 is similar to Figure 4 Step 121, except that associating the LFE audio channel with all of the multiple lighting fixtures is optional. For example, the LFE audio channel may not be associated with any lighting fixture. If the LFE audio channel is not associated with any of the multiple lighting fixtures, then the LFE audio channel is not considered a second audio channel.

[0094] If the LFE audio channel is not associated with all of the multiple lighting fixtures, then one or more other audio channels are associated with the multiple lighting fixtures. These one or more other audio channels are then considered second audio channels. In this case, one or more second characteristics indicating the corresponding desired speaker location are determined for the one or more second audio channels.

[0095] For example, the left front audio channel can be associated with two lighting fixtures on the left side of the room. When the audio section includes both a left front audio channel and a left surround audio channel, the left front audio channel can be mapped to the left front lighting fixture and the left surround audio channel can be mapped to the front and rear lighting fixtures, or both audio channels can be mapped to the two lighting fixtures. The same principle applies to the right audio channel, and when the audio section includes a rear audio channel and / or a high-frequency channel. The left and right audio channels are preferably not mapped to the same lighting fixture.

[0096] Alternatively, if the LFE audio channel is associated with all of the lighting devices in a plurality of lighting devices, the LFE audio channel and one or more of the other audio channels mentioned above can be regarded as a second audio channel.

[0097] In step 143, a mapping from the audio object to the lighting equipment is determined. For example, the audio object might represent an airplane flying from left to right and might be mapped to different lighting equipment depending on its position. A first characteristic of the audio object indicating the current location of the audio source is determined.

[0098] Step 111 includes determining whether the second audio content in the second audio channel meets one or more predetermined criteria. If there is more than one second audio channel, this can be done for each second audio channel. Figure 5In one embodiment, step 111 includes determining whether the audio intensity of the second audio content exceeds a threshold.

[0099] Next, in step 145, the lighting effects for the multiple lighting devices are determined. (See also: Regarding...) Figure 4 As described in step 123, the chromaticity of each lighting effect is determined based on the video portion of the audiovisual content. Furthermore, the light intensity is determined for each lighting effect.

[0100] exist Figure 5 In this embodiment, the light intensity of the lighting effect is determined (in step 145) based on both the audio and video portions of the audiovisual content. The intensity is extracted from the same spatial region from which chromaticity is extracted, and then adjusted based on the audio portion. The adjusted intensity is then used as the light intensity of the lighting effect.

[0101] If the second audio channel is not (only) an LFE audio channel, then in step 145, it is determined which corresponding second audio channel is already associated with the corresponding lighting device (if any). If the lighting device is not associated with a second audio channel in step 141, and the audio object is not associated with a lighting device in step 143, then the light intensity is not adjusted. If the lighting device is not associated with a second audio channel in step 141, and the audio object is associated with a lighting device in step 143, then the light intensity is adjusted based solely on the first audio content in the audio object.

[0102] If the lighting device is associated with the second audio channel in step 141, and it is determined in step 111 that the audio intensity in the second audio channel does not exceed a threshold, then the light intensity for the lighting effect of the lighting device is not adjusted based on the second audio content in the second audio channel. If the lighting device is associated with an audio object in step 143, then the light intensity is adjusted based on the first audio content in the audio object.

[0103] If the lighting device is associated with the second audio channel in step 141, and it is determined in step 111 that the audio intensity in the second audio channel exceeds a threshold, then the light intensity used for the lighting effect of the lighting device is adjusted based on the second audio content in that second audio channel. In this case, if the lighting device is already associated with an audio object, then... Figure 5 In one embodiment, the light intensity is further adjusted based on the first audio content in the audio object. In an alternative embodiment, the light intensity is adjusted only based on the second audio content in the second audio channel.

[0104] Optionally, step 145 includes further determining the light intensity of the light effect based on information about the available speakers and / or information about the 3D audio virtualization used. For example, if the user only has front speakers and a center speaker, and his audio system does not support 3D audio virtualization, it may be best not to adjust the light intensity of the light effect presented on the lighting equipment at the rear of the room based on the first audio channel or the audio content of the audio object whose audio source is located at the rear of the room, as this would create a conflict between the presented light effect and the presented audio.

[0105] about Figure 4 Step 125 is executed after step 145. Step 143 is repeated after step 125, and thereafter the method proceeds as described. Figure 5 The procedure is as shown. Because the characteristics of audio channels typically remain unchanged during audiovisual content viewing, unlike the characteristics of audio objects, step 141 is not repeated in this embodiment. Figure 5 In one embodiment, the audiovisual content is fully acquired before the lighting effects are determined. In an alternative embodiment, the audiovisual content may be streamed and thus partially acquired. In this alternative embodiment, step 141 may be performed, for example, after a first portion of the audiovisual content has been acquired.

[0106] Figure 6 An example of a room 71 in which five entertainment lighting fixtures 11-15 are already installed is shown. Lighting fixture 11 is installed behind display device 23. Lighting fixture 12 is installed to the left of display device 23. Lighting fixture 13 is installed to the right of display device 23. Lighting fixtures 11-13 are installed at the front of the room. Lighting fixtures 14-15 are installed at the rear of the room. Lighting fixture 14 is installed to the left of sofa 73. Lighting fixture 15 is installed to the right of sofa 73.

[0107] Video content 81 includes a video portion 84 and an audio portion 83. In this example, the audio portion 83 includes six audio channels (5.1 audio channels to be exact): left surround channel, left front channel 86, center channel, right front channel, right surround channel, and low-frequency effects channel 87. In alternative examples, the audio portion 83 may include more or fewer than six audio channels. The audio portion further includes two audio objects: a first audio object 88 and a second audio object 89. In practice, an audio portion including audio objects will include more than two audio objects.

[0108] In the first usage example, using Figure 4The method involves mapping the left surround channel, left front channel 86, center channel, right front channel, and right surround channel to lighting devices 14, 12, 11, 13, and 15, respectively. These audio channels are considered the first audio channels. Additionally, the low-frequency effects channel 87 is associated with all lighting devices 11-15. The low-frequency effects channel 87 is considered the second audio channel. When a loud effect is present on the low-frequency effects channel 87, the light intensity of the light effect presented on the lighting devices 11-15 is relatively high.

[0109] In the second usage example, using Figure 5 The method is used to render audio object 88 at virtual source location 78. As in the first use case, low-frequency effects channel 87 is associated with all lighting devices 11-15. Low-frequency effects channel 87 is considered a second audio channel. When a loud effect is present on low-frequency effects channel 87, the light intensity of the light effects rendered on lighting devices 11-15 is relatively high.

[0110] In this situation, the light effect presented by the lighting device closest to the virtual source position 78 (i.e., lighting device 14) may be even higher than the light effect presented by other lighting devices. When there is no loud effect on the low-frequency effects channel 87, only the light intensity of the light effect presented by the lighting device closest to the virtual source position 78 (and not the light intensity of the light effect presented by other lighting devices) is relatively high.

[0111] In the third usage example, using Figure 5 The method involves rendering an audio object 88 at a virtual source location 78. The left surround channel and left front channel 86 are combined, and the combined left channel is associated with both lighting devices 12 and 14. Furthermore, the right surround channel and right front channel are combined, and the combined right channel is associated with both lighting devices 13 and 15. These combined audio channels are considered as second audio channels. Alternatively, surround channels may be absent, and the left and right front channels may then be considered as second audio channels.

[0112] When there is a loud effect on the left audio channel of the combination, the light intensity of the light effects presented on lighting devices 12 and 14 is relatively high. In this case, the light effect presented by the lighting device closest to the virtual source position 78 (i.e., lighting device 14) may even be higher than the light effect presented by the other lighting device (i.e., lighting device 12). When there is no loud effect on the left audio channel of the combination, only the light intensity of the light effect presented by the lighting device closest to the virtual source position 78 (i.e., lighting device 14) (and not the light intensity of the light effect presented by lighting device 12) is relatively high.

[0113] Figure 7The document illustrates a fourth embodiment of a method for controlling multiple lighting devices to produce light effects accompanying audiovisual content. The fourth embodiment is... Figure 3 An extension of the first embodiment. In Figure 7 In the embodiments, Figure 3 Step 113 is implemented by step 163, and step 161 is performed between steps 111 and 163.

[0114] Step 161 includes selecting a spatial region in the current frame of the video portion based on whether one or more predetermined criteria, as determined in step 111, are met. Step 163 includes extracting chroma from the spatial region selected in step 161 (only). If intensity is also extracted from the video portion, such as regarding... Figure 5 The intensity is extracted only from the selected spatial region.

[0115] As an example, when the loudness of the LFE channel does not exceed a threshold, a spatial region on the left side of the video frame is selected for the left front lighting device. When the loudness of the LFE channel exceeds a threshold, a spatial region in the center of the video frame is selected for the left front lighting device.

[0116] Figure 8 The fifth embodiment of a method for controlling multiple lighting devices to present light effects accompanying audiovisual content is shown. The fifth embodiment is... Figure 3 An extension of the first embodiment. In Figure 8 In some embodiments, similar to in Figure 7 In the embodiments, Figure 3 Step 113 is implemented by step 163. Furthermore, in Figure 8 In one embodiment, step 111 is implemented by step 181, and steps 183 and 185 are performed between steps 181 and 163.

[0117] Step 181 includes determining whether the audio intensity of the second audio content in the second audio channel exceeds a threshold. Step 183 includes determining whether the audio intensity of the first audio content in the first audio channel exceeds an additional threshold, which may be the same as the first threshold.

[0118] Step 185 includes selecting a spatial region in the current frame of the video portion, depending on whether the audio intensity of the first audio content, as determined in step 183, exceeds the additional threshold, and optionally also depending on whether the audio intensity of the second audio content, as determined in step 181, exceeds the threshold. Step 163 includes extracting chroma from the spatial region selected in step 185 (only).

[0119] use Figure 8The method involves a relatively high light intensity for the light effect (of all lighting devices) if a loud event is present on the LFE audio channel, and the chromaticity of the light effect presented on a particular lighting device depends on the presence of a loud event on the first audio channel associated with that lighting device. For example, the loudness of the first audio channel can control whether the chromaticity of the light effect is taken from a portion of the screen assigned to it (e.g., the left side) or from the center of the screen. This is in... Figure 9 and Figure 10 As shown in the image. Optionally, the louder the sound effect, the more colors are taken from the center of the screen.

[0120] Figure 9 This shows the effect when the second audio channel is loud and (one or more) the first audio channel is not loud. Figure 8 An example of a lighting device controlled by a method. In this case, the light intensity of the light effect presented by lighting devices 11-13 is relatively high, and the chromaticity to be used for the light effect presented by lighting devices 11, 12 and 13 is extracted from spatial regions 96, 95 and 97, respectively.

[0121] Figure 10 This shows the effect when the second audio channel and (one or more) the first audio channel are both loud. Figure 8 An example of a lighting device controlled by a method. In this case, the light intensity of the light effect presented by lighting devices 11-13 is also relatively high. However, the chromaticity to be used for the light effect presented by lighting devices 11, 12 and 13 is extracted only from spatial region 96.

[0122] Figure 11 The document illustrates a sixth embodiment of a method for controlling multiple lighting devices to produce light effects accompanying audiovisual content. The sixth embodiment is... Figure 3 An extension of the first embodiment. In Figure 11 In the embodiments, Figure 3 Step 103 is implemented by step 203. Figure 3 Step 105 is implemented by step 205, and step 201 is executed after step 201 and before steps 203 and 205. Furthermore, step 111 is implemented by step 207, and step 115 is implemented by step 209.

[0123] Step 201 includes determining one or more speaker signals for one or more loudspeakers based on the audio portion of the audiovisual content obtained in step 101. In step 203, a first characteristic of a first audio channel or audio object is determined based on the one or more speaker signals determined in step 201. In this case, the first characteristic indicates the speaker location associated with the first audio channel or audio object.

[0124] In many cases, the audio source location specified by the audio section is the same as the presented audio source location. However, there are some exceptions, including:

[0125] 1) There is no speaker associated with a specific audio channel in the user's audio system, and that specific audio channel is presented on different speakers when available. For example, some audio systems may create speaker signals for height speakers based on (e.g., as included in the 7.1. audio format) the rear audio channel (e.g., in the 5.1.2. audio system).

[0126] 2) The user's audio system does not use 3D audio virtualization technology. In this case, adjusting the light intensity of the lighting fixture closest to the speaker presenting the audio object may be better than adjusting the light intensity of the lighting fixture closest to the location of the specified audio object in the audio section.

[0127] exist Figure 11 In this embodiment, for consistency, in step 205, a second characteristic is further determined based on one or more speaker signals, and in step 207, it is further determined based on one or more speaker signals whether the audio content in the second audio channel meets one or more predetermined criteria. In step 209, the light intensity of the first light effect is determined based on one or more speaker signals.

[0128] Figure 7 , Figure 8 and Figure 11 The embodiments have been described Figure 3 An extension of. Figure 4 and Figure 5 The embodiments can be extended in a similar manner.

[0129] Figure 12 The description indicates that the procedure can be performed as shown in the reference. Figures 3 to 5 , Figures 7 to 8 ,and Figure 11 A block diagram of an exemplary data processing system for the described method.

[0130] like Figure 12 As shown, the data processing system 300 may include at least one processor 302 coupled to a memory element 304 via a system bus 306. Thus, the data processing system can store program code within the memory element 304. Furthermore, the processor 302 can execute program code accessed from the memory element 304 via the system bus 306. In one aspect, the data processing system may be implemented as a computer suitable for storing and / or executing program code. However, it should be understood that the data processing system 300 may be implemented in the form of any system including a processor and memory capable of performing the functions described herein.

[0131] Memory element 304 may include one or more physical memory devices, such as, for example, local memory 308 and one or more mass storage devices 310. Local memory may refer to random access memory or (multiple) other non-persistent memory devices generally used during the actual execution of program code. Mass storage devices may be implemented as hard disk drives or other persistent data storage devices. Processing system 300 may also include one or more cache memories (not shown) that provide temporary storage for at least some program code to reduce the number of times program code must be retrieved from mass storage device 310 during execution. For example, if processing system 300 is part of a cloud computing platform, processing system 300 may also be able to use memory elements of another processing system.

[0132] Optionally, the input / output (I / O) devices depicted as input device 312 and output device 314 can be coupled to the data processing system. Examples of input devices may include, but are not limited to, a keyboard, a pointing device such as a mouse, a microphone (e.g., for sound and / or speech recognition), etc. Examples of output devices may include, but are not limited to, a monitor or display, a speaker, etc. Input and / or output devices can be coupled to the data processing system directly or through an intermediate I / O controller.

[0133] In embodiments, the input and output devices may be implemented as a combined input / output device (in... Figure 12 (Dashed lines are used to illustrate input device 312 and output device 314). An example of such a combined device is a touch-sensitive display, sometimes also called a "touchscreen display" or simply a "touchscreen". In such embodiments, input to the device can be provided by the movement of a physical object (such as, for example, a stylus or a user's finger) on or near the touchscreen display.

[0134] Network adapter 316 can also be coupled to the data processing system to enable it to couple to other systems, computer systems, remote network devices, and / or remote storage devices via an intermediate private or public network. The network adapter may include a data receiver for receiving data transmitted to the data processing system 300 from the systems, devices, and / or networks, and a data transmitter for transmitting data from the data processing system 300 to the systems, devices, and / or networks. Modems, cable modems, and Ethernet cards are examples of different types of network adapters that can be used with the data processing system 300.

[0135] like Figure 12As illustrated, memory element 304 can store application program 318. In various embodiments, application program 318 may be stored in local memory 308, one or more mass storage devices 310, or separately from local memory and mass storage devices. It should be understood that data processing system 300 may further execute an operating system that facilitates the execution of application program 318. Figure 12 (Not shown in the image). The application program 318, implemented in the form of executable program code, can be executed by the data processing system 300 (e.g., by the processor 302). In response to executing the application program, the data processing system 300 can be configured to perform one or more operational or method steps described herein.

[0136] Various embodiments of the present invention can be implemented as a program product for use with a computer system, wherein the program(s) of the program product define the functionality of the embodiments (including the methods described herein). In one embodiment, the program(s) may be contained on a variety of non-transitory computer-readable storage media, wherein, as used herein, the expression “non-transitory computer-readable storage media” includes all computer-readable media, with the sole exception of transient propagation signals. In another embodiment, the program(s) may be contained on a variety of transient computer-readable storage media. Illustrative computer-readable storage media include, but are not limited to: (i) non-writable storage media on which information is permanently stored (e.g., read-only memory devices within a computer, such as CD-ROM discs readable by a CD-ROM drive, ROM chips, or any type of solid-state non-volatile semiconductor memory); and (ii) writable storage media on which changeable information is stored (e.g., flash memory, floppy disks within a floppy disk drive or hard disk drive, or any type of solid-state random access semiconductor memory). The computer program may run on the processor 302 described herein.

[0137] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that, when used in this specification, the terms “comprising” and / or “including” specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0138] All the means or steps plus functional elements in the following claims are intended to include any structure, material, action, and equivalent for performing a function in combination with other claimed elements as specifically claimed. Descriptions of embodiments of the invention have been shown for illustrative purposes, but are not intended to be exhaustive or limited to the embodiments in the disclosed forms. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. Embodiments have been chosen and described in order to best explain the principles of the invention and some practical applications, and to enable others skilled in the art to understand the invention with respect to various embodiments with various modifications suitable for particular intended uses.

Claims

1. A system (1, 51) for controlling a plurality of lighting devices (11-15) to present light effects accompanying audiovisual content (81), the audiovisual content (81) including an audio portion (83) and a video portion (83), the audio portion (83) including a plurality of audio channels, the system (1, 51) comprising: At least one input interface (3, 53); At least one transmitter (4, 54); and At least one processor (5, 55) is configured as follows: -The audiovisual content (81) is obtained via the at least one input interface (3, 53). - Determine a first characteristic of a first audio channel (86) among the plurality of audio channels, or of an audio object contained in the audio portion (83), the first characteristic indicating the location of the audio source. - Based on the first characteristic, the first audio channel (86) or the audio object is associated with the first lighting device (12) among the plurality of lighting devices (11-15), wherein the association is based on the location of the audio source relative to the position of the first lighting device (12). - Determine the second characteristic of the second audio channel (87) among the plurality of audio channels. Based on the second characteristic, the second audio channel (87) is associated with the first lighting device (12) and the second lighting device (13) among the plurality of lighting devices (11-15), while the first audio channel (86) is not associated with the second lighting device (13). - Determine whether the second audio content in the second audio channel (87) meets one or more predetermined criteria. -Based on the audiovisual content (81), the video portion (83) at least determines the chroma. - A first lighting effect is determined based on the determined chromaticity, wherein if one or more predetermined criteria are not met, the light intensity of the first lighting effect is based on the first audio content in the first audio channel (86) or the audio object, and if one or more predetermined criteria are met, the light intensity of the first lighting effect is based on the second audio content in the second audio channel (87), and - The first lighting device (12) is controlled via the at least one transmitter (4, 54) to present the first light effect.

2. The system (1, 51) according to claim 1, wherein the at least one processor (5, 55) is configured to further determine the light intensity of the first light effect based on the first audio content in the first audio channel (86) or the audio object if the one or more predetermined criteria are met.

3. The system (1, 51) according to claim 1 or 2, wherein the at least one processor (5, 55) is configured to further determine the light intensity of the first light effect based on the video portion (83) of the audiovisual content (81).

4. The system (1, 51) according to claim 1 or 2, wherein the second feature does not indicate the location of the audio source.

5. The system (1, 51) according to claim 4, wherein the second characteristic indicates whether the second audio channel (87) is a low-frequency effects channel.

6. The system (1, 51) according to claim 1 or 2, wherein the first characteristic is determined by the audio object and the second characteristic indicates the desired speaker position for the second audio channel.

7. The system (1, 51) according to claim 1 or 2, wherein the at least one processor (5, 55) is configured to determine whether the second audio content in the second audio channel (87) meets the one or more predetermined criteria by determining whether the audio intensity of the second audio content exceeds a threshold.

8. The system (1, 51) according to claim 1 or 2, wherein the at least one processor (5, 55) is configured to: -The spatial region in the current frame of the video portion (83) is selected based on whether one or more predetermined criteria are met, and - The chromaticity is determined only from the selected spatial region.

9. The system (1, 51) according to claim 1 or 2, wherein the first characteristic is determined by the first audio channel (86), and the at least one processor (5, 55) is configured to: - Determine whether the audio intensity of the first audio content exceeds a threshold. -The spatial region (95-97) in the current frame of the video portion (83) is selected based on whether the audio intensity of the first audio content exceeds the threshold, and - The chromaticity is determined at least from the selected spatial region (95-97).

10. The system (1, 51) according to claim 1 or 2, wherein the at least one processor (5, 55) is configured to determine one or more speaker signals for a loudspeaker based on the audio portion (83) of the audiovisual content (81).

11. The system (1, 51) of claim 10, wherein the at least one processor (5, 55) is configured to determine the light intensity of the first light effect based on the one or more speaker signals.

12. The system (1, 51) according to claim 1 or 2, wherein the at least one processor (5, 55) is configured to further determine the light intensity of the first light effect based on information about available speakers and / or information about the three-dimensional audio virtualization used.

13. A method for controlling multiple lighting devices to present light effects accompanying audiovisual content, the audiovisual content including an audio portion and a video portion, the audio portion including multiple audio channels, the method comprising: - Obtain the audiovisual content described in (101); - Determine (103) a first characteristic of a first audio channel among the plurality of audio channels or of an audio object contained in the audio portion, the first characteristic indicating the location of the audio source; - Based on the first characteristic, associate the first audio channel or the audio object with the first lighting device among the plurality of lighting devices (105), wherein the association is based on the location of the audio source relative to the location of the first lighting device; - Determine the second characteristic of the second audio channel among the plurality of audio channels (107); - Based on the second characteristic, the second audio channel is associated with the first lighting device and the second lighting device among the plurality of lighting devices (109), and the first audio channel is not associated with the second lighting device; - Determine whether the second audio content in the second audio channel (111) meets one or more predetermined criteria; - Based on the video portion of the audiovisual content, at least (113) chromaticity is determined; - Based on the determined chromaticity, determine (115) a first light effect, wherein if one or more predetermined criteria are not met, the light intensity of the first light effect is based on the first audio content in the first audio channel, and if one or more predetermined criteria are met, the light intensity of the first light effect is based on the second audio content in the second audio channel; and - Control (117) the first lighting device to present the first light effect.

14. A computer program product for a computing device, the computer program product comprising computer program code that executes the method of claim 13 when the computer program product is run on a processing unit of the computing device.

Images