Spatial audio rendering adapted to signal levels and loudspeaker playback limit thresholds

Abstract

Rendering the audio signal may involve a mapping of each audio signal to a loudspeaker signal that is calculated as a function of an expected perceived spatial position of the audio signal, a physical position associated with the loudspeaker, and a time-varying and frequency-varying representation of a maximum playback limit of a loudspeaker signal level relative to each loudspeaker. Each mapping may be calculated to substantially achieve an expected perceived spatial position of the associated audio signal when the loudspeaker signal is played back. A representation of a loudspeaker signal level relative to a maximum playback limit may be calculated for each audio signal. As the loudspeaker signal level increases above a threshold relative to the maximum playback limit, a mapping of the audio signal to a particular loudspeaker signal may be reduced while a mapping to one or more other loudspeakers for which the maximum playback limit is less than the threshold may be increased.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 392,794 filed on July 27, 2022, U.S. Provisional Patent Application No. 63 / 413,923 filed on October 6, 2022, and U.S. Provisional Patent Application No. 63 / 505,652 filed on June 1, 2023, all of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates to systems and methods for rendering audio for playback by a set of speakers. Background Art

[0004] Audio devices, including but not limited to smart audio devices, have been widely deployed and are becoming a common feature of many homes. Although existing systems and methods for controlling audio devices provide benefits, improved systems and methods would still be desirable.

[0005] Symbols and terminology

[0006] Throughout this disclosure, including in the claims, "speaker" and "loudspeaker" are used synonymously to refer to any sound-emitting transducer (or set of transducers) driven by a single speaker feed.A typical set of headphones includes two speakers.

[0007] Throughout this disclosure, including in the claims, the expression "performing an operation on" a signal or data (e.g., filtering, scaling, transforming, or applying a gain to the signal or data) is used in a broad sense to mean performing the operation directly on the signal or data or on a processed version of the signal or data (e.g., a version of the signal that has undergone preliminary filtering or preprocessing before the operation is performed on it).

[0008] Throughout this disclosure, including in the claims, the expression "system" is used in a broad sense to refer to a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, where the subsystem generates M inputs and the other XM inputs are received from external sources) may also be referred to as a decoder system.

[0009] Throughout this disclosure, including in the claims, the term "processor" is used in a broad sense to refer to a system or device that is programmable or otherwise configurable (e.g., with software or firmware) to perform operations on data (e.g., audio or video or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors that are programmed and / or otherwise configured to perform pipeline processing of audio or other sound data, programmable general purpose processors or computers, and programmable microprocessor chips or chipsets.

[0010] Throughout this disclosure, including in the claims, the terms "couple" or "coupled" are used to mean either a direct or indirect connection. Thus, if a first device couples to a second device, that connection may be through a direct connection or through an indirect connection via other devices and connections.

[0011] The expression "intelligent audio device" is used herein to represent an intelligent device, which is a single-purpose audio device or a virtual assistant (e.g., a connected virtual assistant). A single-purpose audio device is a device (e.g., TV or mobile phone) that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker), and the device is largely or primarily designed to achieve a single purpose. Although TV can usually play (and is considered to be able to play) audio from program materials, in most cases, modern TVs run some kind of operating system, and applications (including applications for watching TV) run locally on the operating system. Similarly, the audio input and output in a mobile phone can do a lot of things, but these are all served by applications running on the phone. In this sense, a single-purpose audio device with (multiple) speakers and (multiple) microphones is usually configured to run local applications and / or services to directly use (multiple) speakers and (multiple) microphones. Some single-purpose audio devices can be configured to be combined to achieve playing audio on a zone or user-configured area.

[0012] A virtual assistant (e.g., a connected virtual assistant) is a device (e.g., a smart speaker or voice assistant integrated device) that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker), and that can provide the ability to use multiple devices (other than the virtual assistant) for applications that are cloud-enabled in some sense or that are not implemented in or on the virtual assistant itself. Virtual assistants can sometimes work together, e.g., in discrete and conditionally defined ways. For example, two or more virtual assistants can work together in the sense that one of them (e.g., the virtual assistant that is most confident that it has heard the wake word) responds to the word. Connected devices can form a constellation that can be managed by a master application, which can be (or implements) a virtual assistant.

[0013] In this document, "wake-up word" is used in a broad sense to refer to any sound (e.g., a word spoken by a human or other sound) that the smart audio device is configured to wake up in response to detecting ("hearing") the sound (using at least one microphone included in or coupled to the smart audio device, or at least one other microphone). In this case, "waking up" means that the device enters a state of waiting (i.e., listening) for a voice command. In some instances, the so-called "wake-up word" in this document can include more than one word, for example, a phrase.

[0014] In this document, the expression "wake-up word detector" refers to a device configured to continuously search for alignment between real-time sound (e.g., speech) features and a trained model (or refers to software including instructions for configuring the device to continuously search for alignment between real-time sound features and a trained model). Typically, a wake-up word event is triggered whenever the wake-up word detector determines that the probability of detecting a wake-up word exceeds a predefined threshold. For example, the threshold may be a predetermined threshold that is adjusted to give a reasonable compromise between a false acceptance rate and a false rejection rate. After a wake-up word event, the device may enter a state (which may be referred to as an "awake" state or an "attention" state) in which the device listens for commands and passes received commands to a larger, more computationally intensive recognizer. Summary of the invention

[0015] At least some aspects of the present disclosure may be implemented via methods such as audio processing methods. In some instances, the methods may be implemented at least in part by control systems such as those disclosed herein. Some methods may involve receiving audio data by a control system and via an interface system. The audio data may include one or more audio signals and associated spatial data. The spatial data may indicate an expected perceived spatial position corresponding to the audio signal. In some examples, the expected perceived spatial position may correspond to a channel of a channel-based audio format, may correspond to metadata, or may correspond to both a channel and metadata. Some methods may involve rendering the audio data by a control system to be reproduced via a set of two or more loudspeakers of an environment, thereby generating a loudspeaker signal. Some methods may involve providing a loudspeaker signal to at least two loudspeakers in a set of loudspeakers of an environment via an interface system.

[0016] According to some examples, rendering each of the one or more audio signals included in the audio data may involve mapping each audio signal to a loudspeaker signal. In some instances, the mapping may be a time-varying and frequency-varying mapping. In some examples, the mapping of each audio signal may be calculated based on the expected perceived spatial position of the audio signal, the physical position associated with the loudspeaker, and the time-varying and frequency-varying representation of the loudspeaker signal level relative to the maximum playback limit of each loudspeaker. According to some examples, each mapping may be calculated to approximately achieve the expected perceived spatial position of the associated audio signal when the loudspeaker signal is played back on two or more corresponding loudspeakers located at the associated loudspeaker position. In some examples, the representation of the loudspeaker signal level relative to the maximum playback limit of each audio signal may be calculated based on one or more of the audio signals and their perceived spatial positions. According to some examples, as the representation of the loudspeaker signal level relative to the maximum playback limit increases above a threshold, the mapping of the audio signal to a specific loudspeaker signal may be reduced, while, for the representation of the signal level relative to the maximum playback limit of one or more other loudspeakers being less than a threshold, the mapping to the one or more other loudspeakers may be increased.

[0017] In some examples, the mapping may be calculated over the entire audible frequency range (eg, the audible frequency range of a human). However, in some examples, the mapping may be calculated over a subset of the audible frequency range.

[0018] According to some examples, mapping can involve minimizing a cost function including a first term that models how closely a desired perceptual spatial position can be realized as a mapping function of an audio signal to a loudspeaker signal, and a second term that assigns a cost to activating each of the loudspeakers. In some such examples, the cost of activating each loudspeaker can be based at least in part on a function of a representation of a loudspeaker signal level relative to a maximum playback limit.

[0019] In some examples, the representation of the loudspeaker signal level relative to the maximum playback limit may correspond to one or more of a digital signal level, a limiter gain, or an acoustic signal level. According to some examples, the representation of the loudspeaker signal level relative to the maximum playback limit may be calculated as the difference between the level estimate of each audio signal and the playback limit threshold of each loudspeaker. In some examples, the level estimate of each audio signal may be based at least in part on a zone-based rendering of all audio signals. According to some examples, the level estimate of each audio signal may be based at least in part on a previously calculated loudspeaker signal. In some examples, the level estimate of each audio signal may further depend on the participation of each loudspeaker in multiple spatial zones. Some methods may involve smoothing the level estimate of each audio signal in time, in frequency, or in both time and frequency.

[0020] According to some examples, the mapping of audio signals to loudspeaker signals may be determined by querying a data structure indexed by the expected perceptual spatial position and level estimate for each audio signal. In some examples, the mapping of audio signals to loudspeaker signals may be determined by interpolating from a precomputed set of speaker mappings. In some such examples, the set may be indexed by the expected perceptual spatial position and level estimate for each audio signal. In some examples, the set may be indexed by the expected level estimate for each audio signal.

[0021] In some examples, the level estimate of each audio signal may be expressed as a broadband gain multiplied by a spectral shape. According to some examples, the spectral shape may be selected from a plurality of spectral shapes. In some such examples, each spectral shape in the plurality of spectral shapes may correspond to a content type.

[0022] According to some examples, as the signal level relative to the representation of the maximum playback level increases above a threshold, mapping to one loudspeaker may be gradually reduced and mapping to another loudspeaker may be gradually increased.

[0023] In some examples, given available loudspeakers and associated loudspeaker positions, approximately achieving an intended perceptual spatial position of the associated audio signal may involve minimizing a difference between the perceptual spatial position and the intended perceptual spatial position. According to some examples, approximately achieving an intended perceptual spatial position of the associated audio signal may involve minimizing a cost function.

[0024] Some methods may involve controlling the extent of the mapping to one loudspeaker to be reduced and the extent of the mapping to another loudspeaker to be increased based on one or more of the audio format, codec, or metadata. Some methods may involve controlling the extent of the mapping to one loudspeaker to be reduced and the extent of the mapping to another loudspeaker to be increased based on an inflection point parameter.

[0025] Some or all of the operations, functions, and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include one or more memory devices as described herein, including but not limited to one or more random access memory (RAM) devices, read-only memory (ROM) devices, etc. Therefore, some innovative aspects of the subject matter described in this disclosure may be implemented in one or more non-transitory media having software stored thereon.

[0026] At least some aspects of the present disclosure may be implemented via an apparatus. For example, one or more devices may be capable of at least partially executing the methods disclosed herein. In some embodiments, the apparatus may include an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or a combination thereof.

[0027] The details of one or more embodiments of the subject matter described in this specification are set forth in the following drawings and description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Note that the relative sizes of the following drawings may not be drawn to scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a block diagram illustrating an example of components of an apparatus capable of implementing various aspects of the present disclosure.

[0029] Figure 2 Depicted is a plan view of a listening environment, which in this example is a living space.

[0030] Figure 3 is a block diagram illustrating an example of components of a system capable of implementing various aspects of the present disclosure.

[0031] Figure 4A , Figure 4B and Figure 4C Examples of playback limit thresholds and corresponding frequencies are shown.

[0032] Figure 5A and Figure 5B is a diagram showing an example of dynamic range compression data.

[0033] Figure 6 Examples of spatial zones of a listening environment are shown.

[0034] Figure 7 Shows Figure 6 Example of a loudspeaker within a spatial region of .

[0035] Figure 8 Shows coverage in Figure 7 Examples of spatial zones and nominal spatial positions on the loudspeakers.

[0036] Fig. 9 is a diagram indicating points of object to speaker mapping in an example embodiment.

[0037] Fig.10 is a graph indicating trilinear interpolation between points of an object-to-speaker mapping according to an example.

[0038] Fig.11 Examples of penalties for various inflection point parameters are shown.

[0039] Fig.12 is a flow chart outlining one example of a method that may be performed by an apparatus or system, such as those disclosed herein. DETAILED DESCRIPTION

[0040] The playback of spatial audio in consumer environments is often associated with a specified number of speakers placed in specified locations (such as those corresponding to Dolby 5.1 or 7.1 surround sound). In these cases, the content is authored specifically for the associated speakers and encoded as discrete channels, one for each speaker (e.g., Dolby Digital TM 、Dolby Digital Plus TM More recently, immersive, object-based spatial audio formats such as Dolby Atmos have been introduced. TM), which breaks this association between content and specific loudspeaker locations. Instead, content can be described as a collection of individual audio objects, each with potentially time-varying metadata describing the desired perceived position of the audio object in three-dimensional space and, in some examples, other properties of the audio object. At playback, the audio content is transformed into loudspeaker feeds by a renderer that adapts to the number and positions of loudspeakers in the playback system. However, many such renderers still restrict the positions of a set of loudspeakers to a set of specified layouts (e.g., using Dolby Atmos). TM Dolby 3.1.2, 5.1.2, 7.1.4, 9.1.6, etc.)

[0041] Beyond this limited rendering, methods have been developed that allow object-based audio to be flexibly rendered on a truly arbitrary number of loudspeakers placed in arbitrary locations. These methods typically require the renderer to know the number and physical location of the loudspeakers in the listening space. In order to make such systems practical for the average consumer, an automated method for locating the loudspeakers is desired. One such method relies on the use of multiple microphones that may be co-located with the loudspeakers. By playing an audio signal through a loudspeaker and recording it with a microphone, the distance between each loudspeaker and the microphone can be estimated. The positions of both the loudspeaker and the microphone can then be derived from these distances.

[0042] The introduction of object-based spatial audio in the consumer space coincided with the advent of so-called “smart speakers” such as the Amazon Echo TM The popularity of these devices can be attributed to their wireless connectivity and integrated voice interfaces (e.g., Amazon’s Alexa) TM ), but the sonic capabilities of these devices are often limited, especially when it comes to spatial audio. In most cases, these devices are limited to mono or stereo playback. However, combining the flexible rendering and automatic positioning technologies described above with multiple orchestrated smart speakers can produce systems with very sophisticated spatial playback capabilities that are still very simple for consumers to set up. Since wireless connectivity eliminates the need to run speaker wires, consumers can place as many speakers as they want wherever it is convenient, and the built-in microphones can be used to automatically position the speakers for the associated flexible renderer.

[0043] One method for rendering spatial audio on a set of loudspeakers is to map each component signal of a spatial mix on the set of loudspeakers based purely on assumed or measured loudspeaker locations and the expected perceived locations of the component signals. This method is described in U.S. Patents 9,712,939 and 11,172,318, which are hereby incorporated by reference. If there are playback capability variations between the set of loudspeakers, the perceived quality of the spatial rendering may be affected when using this method. Many smaller loudspeakers begin to distort and then reach their excursion limits as the playback level increases, especially for lower frequencies.

[0044] To reduce this distortion, each loudspeaker may implement dynamic processing that, in some examples, limits the playback level to below these limits in a manner that varies over frequency. When spatial audio rendered using the above method is then played on a set of loudspeakers, each loudspeaker applies its dynamic processing independently, making it possible for dramatically different relative modifications to the audio on different speaker feeds. For example, at high playback levels, less powerful loudspeakers will typically attenuate the audio more than more powerful loudspeakers. These processing variations across loudspeakers may dynamically shift the spatial balance of the mix in a perceptually distracting manner, and may also interfere with the overall relative balance of the mix. For example, if the front soundstage is primarily reproduced by the less powerful loudspeakers, the front soundstage may be attenuated overall relative to the rear soundstage.

[0045] The present assignee has developed methods to mitigate some of these problems by intelligently combining playback limit thresholds across loudspeakers and applying these playback limit thresholds in spatial zones throughout a spatial audio mixer before rendering the mix to the loudspeaker feeds. Some examples are disclosed herein. Zones can be selected to prevent perceptually distracting imaging from shifting from left to right while still allowing some independence of processing between portions of the audio mix. Some zone-based methods involve four zones: a front zone, a center zone, a surround zone, and a top zone.

[0046] These zone-based approaches do help stabilize the spatial imaging of the rendered audio. However, in some instances, such zone-based approaches may have the undesirable effect of limiting the overall playback level to the least powerful device in the set of loudspeakers.

[0047] The present disclosure provides improved rendering methods, including some improved zone-based methods, which better utilize the more powerful loudspeakers in the set of loudspeakers. An improved method for rendering spatial audio is disclosed, wherein the dynamic signal level of the spatial audio mix is ​​additionally considered when mapping each component signal of the mix to a loudspeaker feed signal. In some examples, when the level of the audio mix approaches a playback limit threshold of a particular loudspeaker, the mapping of the component to the loudspeaker is reduced in order to increase the mapping to other loudspeakers whose mix levels are further from their limit threshold. In this way, the overall level of the rendered audio may not be limited by the less powerful loudspeakers. However, when the audio signal level is below its limit threshold, the less powerful loudspeaker can still be used.

[0048] Constructing such a dynamic rendering system should be done carefully so as to prevent the introduction of additional perceptual artifacts in the process of attempting to maximize playback level. For example, consider a single component of a spatial audio mix whose intended perceptual position is "front left". If a loudspeaker is physically located near the intended perceptual position, then ideally a large proportion of the component signal energy should be mapped to that loudspeaker. However, if the signal level of the mix is ​​close to the playback limits of that loudspeaker, we would like to map a larger proportion of that component signal to other, more capable loudspeakers in order to reduce the activation of the dynamic processing of the first loudspeaker, thereby better maintaining the overall playback level of the mix. Since signal energy is dynamically transferred to these other loudspeakers, these other loudspeakers may be less suitable for achieving the intended perceptual position of the component signal because they are less physically close relative to the intended perceptual position, so the likelihood of perceiving this transfer as an unnecessary spatial shift of the component signal should be minimized.

[0049] To achieve this minimization, some disclosed methods employ several strategies simultaneously:

[0050] 1. First, in some examples, the mapping of each component signal to a loudspeaker signal takes into account the current signal level of the audio mix, and uses the loudspeakers that are considered available under the signal level conditions at that particular moment to make a "best effort" to achieve the desired perceptual position of the component audio signal. In some examples, making such a "best effort" may involve what will be described herein as approximately achieving the expected perceptual spatial position of the audio signal component. The expected perceptual spatial position may correspond to a channel of a channel-based audio format, to the position metadata of an audio object, or to both the channel and the position metadata. According to some examples, for example, given the available loudspeakers in a given audio environment, the capabilities of each loudspeaker, and the associated loudspeaker positions, approximately achieving the expected perceptual spatial position of the audio signal component may involve minimizing the difference between the perceptual spatial position and the expected perceptual spatial position. In some examples, approximately achieving the expected perceptual spatial position of the audio signal component may involve minimizing a cost function. In this way, this method optimizes the spatial mapping of each component signal with respect to the signal level conditions individually. This differs from simpler solutions, such as those that may render the loudspeaker signals using the above-described method of optimizing spatial imaging but ignoring signal level, and then redistribute the energy of the rendered signal among the loudspeakers based on a comparison of the rendered signal level with a limiting threshold for each loudspeaker.

[0051] 2. Second, in some examples, both the mapping of component signals to loudspeaker feeds and the characterization of signal levels relative to loudspeaker playback limitations on which the mapping depends can be calculated in a time-varying and frequency-varying manner. In this way, the transfer of energy of any component signal away from its spatially optimal loudspeaker occurs only in the frequency region and at the moments when the signal energy approaches the limiting thresholds of these optimal loudspeakers. This approach minimizes the amount of energy transferred and allows more of the energy of any component signal to remain in the loudspeaker that is optimal for its spatial reproduction. Therefore, the probability that the perceived position of the component signal remains in its desired spatial position remains high.

[0052] 3. Finally, a characterization of the signal level relative to the loudspeaker playback constraints on which the mapping of the component signal to the loudspeaker signal depends is calculated for each individual component signal based on one or more of the component signals of the mix and their expected perceptual positions. In this way, the transfer of signal energy between loudspeakers can be personalized not only for the expected perceptual position of each component signal (as outlined in the first strategy above), but also for an estimate of the overall signal level that is optimized in some way relative to the relationship of that component signal to the other components of the mix. For example, the overall signal level associated with any component signal can be calculated based on the spatial zone of the zone-based approach mentioned above. In this way, the transfer of component signals located in similar spatial zones will be similar, thereby stabilizing the perceived left / right balance of the dynamic rendering. In addition, component signals belonging to substantially different zones may be transferred in different ways if the overall levels associated with these zones are significantly different. For example, if the signal level of the surround zone is low, those audio signal components that are largely associated with the surround zone may be less transferred to its mapping. At the same time, the signal level of the front zone may be higher, and those components that are largely associated with the front zone may be more transferred to its mapping. Therefore, this strategy may also help minimize the amount of transferred energy across the entire spatial mix by applying the transfer only to component signals belonging to the spatial regions that require the transfer.

[0053] Figure 1 is a block diagram showing an example of components of an apparatus capable of implementing various aspects of the present disclosure. Figure 1 Sample, Figure 1 The types and quantities of the elements shown in are provided as examples only. Other embodiments may include more, fewer and / or different types and quantities of elements. According to some examples, the device 100 may be or may include an intelligent audio device configured to perform at least some of the methods disclosed herein. In other embodiments, the device 100 may be or may include another device configured to perform at least some of the methods disclosed herein, such as a laptop computer, a cellular phone, a tablet device, a smart home hub, etc. In some such embodiments, the device 100 may be or may include a server.

[0054] In this example, the device 100 includes an interface system 105 and a control system 110. In some embodiments, the interface system 105 can be configured to receive audio data. The audio data can include audio signals arranged to be reproduced by at least some speakers of the environment. The audio data can include one or more audio signals and associated spatial data. For example, the spatial data can include channel data and / or spatial metadata. The interface system 105 can be configured to provide rendered audio signals to at least some of a set of loudspeakers of the environment. In some embodiments, the interface system 105 can be configured to receive input from one or more microphones in the environment.

[0055] The interface system 105 may include one or more network interfaces and / or one or more external device interfaces (e.g., one or more universal serial bus (USB) interfaces). According to some embodiments, the interface system 105 may include one or more wireless interfaces. The interface system 105 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. In some examples, the interface system 105 may include a control system 110 and a memory system (e.g., Figure 1 However, in some examples, control system 110 may include a memory system.

[0056] For example, the control system 110 may include a general purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.

[0057] In some embodiments, the control system 110 may reside in more than one device. For example, a portion of the control system 110 may reside in a device within one of the environments depicted herein, and another portion of the control system 110 may reside in a device such as a server, a mobile device (e.g., a smart phone or a tablet computer) outside the environment. In other examples, a portion of the control system 110 may reside in a device within one of the environments depicted herein, and another portion of the control system 110 may reside in one or more other devices of the environment. For example, the functionality of the control system may be distributed across multiple smart audio devices of the environment, or may be shared by an orchestration device (such as a device that may be referred to as a smart home hub herein) and one or more other devices of the environment. In some such examples, the interface system 105 may also reside in more than one device.

[0058] In some implementations, the control system 110 can be configured to at least partially perform the methods disclosed herein.According to some examples, the control system 110 can be configured to implement a method of managing playback of multiple audio streams on multiple loudspeakers.

[0059] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices as described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. One or more non-transitory media may reside, for example, on a computer. Figure 1 1 and / or the control system 110. Thus, various innovative aspects of the subject matter described in the present disclosure may be implemented in one or more non-transitory media having software stored thereon. For example, the software may include instructions for controlling at least one device to process audio data. For example, the software may be provided by Figure 1 The control system 110 and other control system components are executed.

[0060] In some examples, apparatus 100 may include Figure 1 Optional microphone system 120 shown in FIG. Optional microphone system 120 may include one or more microphones. In some embodiments, one or more microphones may be part of or associated with another device (such as a speaker of a speaker system, a smart audio device, etc.).

[0061] According to some embodiments, the apparatus 100 may include Figure 1 Optional loudspeaker system 125 shown in . Optional loudspeaker system 125 may include one or more loudspeakers. Loudspeakers may sometimes be referred to as "speakers" in this article. In some examples, at least some of the loudspeakers of optional loudspeaker system 125 can be positioned arbitrarily. For example, at least some of the loudspeakers of optional loudspeaker system 125 can be placed in a position that does not correspond to a loudspeaker layout specified by any standard, such as Dolby 5.1, Dolby 5.1.2, Dolby 7.1, Dolby 7.1.4, Dolby 9.1, Hamasaki 22.2, etc. In some such examples, at least some of the loudspeakers of optional loudspeaker system 125 can be placed in a position that is convenient for the space (e.g., in a position where there is room to accommodate the loudspeakers), but not in a loudspeaker layout specified by any standard.

[0062] In some embodiments, the apparatus 100 may include Figure 1Optional sensor system 130 shown in . Optional sensor system 130 may include one or more cameras, touch sensors, gesture sensors, motion detectors, etc. According to some embodiments, optional sensor system 130 may include one or more cameras. In some embodiments, the camera may be a stand-alone camera. In some examples, one or more cameras of optional sensor system 130 may reside in a smart audio device, which may be a single-purpose audio device or a virtual assistant. In some such examples, one or more cameras of optional sensor system 130 may reside in a TV, a mobile phone, or a smart speaker.

[0063] In some embodiments, the apparatus 100 may include Figure 1 Optional display system 135 is shown in . Optional display system 135 may include one or more displays, such as one or more light emitting diode (LED) displays. In some instances, optional display system 135 may include one or more organic light emitting diode (OLED) displays. In some examples where device 100 includes display system 135, sensor system 130 may include a touch sensor system and / or a gesture sensor system proximate to one or more displays of display system 135. According to some such embodiments, control system 110 may be configured to control display system 135 to present a graphical user interface (GUI), such as one of the GUIs disclosed herein.

[0064] According to some examples, the apparatus 100 may be or may include a smart audio device. In some such implementations, the apparatus 100 may be or may include a wake-up word detector. For example, the apparatus 100 may be or may include a virtual assistant.

[0065] Figure 2 Depicts a plan view of a listening environment, which in this example is a living space. Figure 1 Sample, Figure 2 The types and quantities of elements shown in are provided as examples only. Other embodiments may include more, fewer and / or different types and quantities of elements. According to this example, environment 200 includes a living room 210 located at the upper left, a kitchen 215 located at the lower center, and a bedroom 222 located at the lower right. The boxes and circles distributed across the living space represent a group of loudspeakers 205a-205h, at least some of which may be smart speakers in some embodiments, placed in a location convenient for the space, but not adhering to any standard prescribed layout (arbitrary placement). In some examples, loudspeakers 205a-205h may be coordinated to implement one or more disclosed embodiments.

[0066] According to some examples, environment 200 may include a smart home hub for implementing at least some of the disclosed methods. According to some such implementations, the smart home hub may include at least a portion of the above-described control system 110. In some examples, smart devices (e.g., smart speakers, mobile phones, smart TVs, devices for implementing virtual assistants, etc.) may implement the smart home hub.

[0067] In this example, environment 200 includes cameras 211a-211e distributed throughout the environment. In some embodiments, one or more smart audio devices in environment 200 may also include one or more cameras. The one or more smart audio devices may be single-purpose audio devices or virtual assistants. In some such examples, one or more cameras of optional sensor system 130 may reside in or on television 230, in a mobile phone, or in a smart speaker (such as one or more of loudspeakers 205b, 205d, 205e, or 205h). Although cameras 211a-211e are not shown in each depiction of environment 200 presented in the present disclosure, in some embodiments, each environment 200 may still include one or more cameras.

[0068] In flexible rendering, spatial audio can be rendered on any number of arbitrarily placed speakers. With the widespread deployment of smart audio devices (e.g., smart speakers) in homes, there is a need to implement flexible rendering techniques that allow consumers to use smart audio devices to perform flexible rendering of audio and playback of the audio so rendered.

[0069] Several techniques have been developed to implement flexible rendering, including: Center of Mass Amplitude Translation (CMAP) and Flexible Virtualization (FV).

[0070] In the context of performing rendering (or rendering and playback) of spatial audio mixing (e.g., rendering an audio stream or multiple audio streams) for playback by a smart audio device (or another set of speakers) in a group of smart audio devices, the types of speakers (e.g., in or coupled to the smart audio device) may vary, and thus the corresponding acoustic capabilities of the speakers may vary quite significantly. Figure 2In the example shown in , loudspeakers 205d, 205f, and 205h are smart speakers with a single 0.6-inch speaker. In this example, loudspeakers 205b, 205c, 205e, and 205f are smart speakers with 2.5-inch woofers and 0.8-inch tweeters. According to this example, loudspeaker 205g is a smart speaker with a 5.25-inch woofer, three 2-inch midrange speakers, and a 1.0-inch tweeter. Here, loudspeaker 205a is a sound bar with sixteen 1.1-inch beam drivers and two 4-inch woofers. Therefore, the low-frequency capabilities of smart speakers 205d and 205f are significantly less than the low-frequency capabilities of other loudspeakers in environment 200, especially those with 4-inch or 5.25-inch woofers.

[0071] Examples of dynamic processing, including some related zone-based methods

[0072] Figure 3 is a block diagram showing an example of components of a system capable of implementing various aspects of the present disclosure. Figure 1 Sample, Figure 3 The types and quantities of the elements shown in the drawings are provided as examples only. Other implementations may include more, fewer and / or different types and quantities of elements.

[0073] According to this example, the system 300 includes a smart home hub 305 and speakers 205a to 205m. In this example, the smart home hub 305 includes Figure 1 3 and described above. According to this embodiment, the control system 110 includes a listening environment dynamics processing configuration data module 310, a listening environment dynamics processing module 315, and a rendering module 320. Some examples of the listening environment dynamics processing configuration data module 310, the listening environment dynamics processing module 315, and the rendering module 320 are described below. In some examples, the rendering module 320' can be configured for rendering and listening environment dynamics processing.

[0074] As indicated by the arrows between the smart home hub 305 and the loudspeakers 205a to 205m, the smart home hub 305 also includes Figure 1 According to some examples, the smart home hub 305 may be Figure 2. In some instances, the smart home hub 305 can be implemented by a smart speaker, a smart TV, a cellular phone, a laptop computer, etc. In some embodiments, the smart home hub 305 can be implemented by software, for example, via a downloadable software application or "app" software. In some instances, the smart home hub 305 can be implemented in each of the loudspeakers 205a-m, all of which operate in parallel to generate the same processed audio signal from the module 320. According to some such examples, in each loudspeaker, the rendering module 320 can then generate one or more speaker feeds associated with each loudspeaker or a group of loudspeakers, and these speaker feeds can be provided to each speaker dynamics processing module.

[0075] In some examples, the loudspeakers 205a to 205m may include Figure 2 In other examples, the loudspeakers 205a to 205h may be or may include other loudspeakers. Therefore, in this example, the system 300 includes M loudspeakers, where M is an integer greater than 2.

[0076] Smart speakers, as well as many other powered speakers, typically employ some type of internal dynamics processing to prevent speaker distortion. Often associated with such dynamics processing is a signal limiting threshold (e.g., a limiting threshold that is variable in frequency), below which signal levels are dynamically maintained. For example, the Dolby Audio Conditioner, one of several algorithms in the Dolby Audio Processing (DAP) audio post-processing suite, provides such processing. In some instances—but typically not via the dynamics processing module of a smart speaker—dynamics processing may also involve the application of one or more compressors, gates, expanders, duckers, and the like.

[0077] Thus, in this example, each of the loudspeakers 205a to 205m includes a corresponding loudspeaker dynamic processing (DP) module A to M. The loudspeaker dynamic processing modules are configured to apply individual loudspeaker dynamic processing configuration data to each individual loudspeaker of the listening environment. For example, loudspeaker DP module A is configured to apply individual loudspeaker dynamic processing configuration data suitable for loudspeaker 205a. In some examples, the individual loudspeaker dynamic processing configuration data may correspond to one or more capabilities of the individual loudspeaker, such as the ability of the loudspeaker to reproduce audio data within a specific frequency range and at a specific level without perceptible distortion.

[0078] When rendering spatial audio on a set of heterogeneous speakers (e.g., speakers of a smart audio device or speakers coupled to a smart audio device) that each have potentially different playback limitations, care should be taken when performing dynamic processing on the overall audio mix. A simple solution is to render the spatial mix as a speaker feed for each participating speaker, and then allow the dynamic processing module associated with each speaker to operate independently on its corresponding speaker feed according to the limitations of that speaker.

[0079] While this approach will prevent distortion in each speaker, it may dynamically shift the spatial balance of the mix in a perceptually distracting way. Figure 2 , assume that a television program is being played on television 230, and that the corresponding audio is being reproduced by the loudspeakers of environment 200. Assume that during the television program, audio associated with a stationary object, such as a heavy machinery unit in a factory, is intended to be rendered to location 244. Assume further that the dynamic processing module associated with loudspeaker 205d reduces the level of audio in the bass range much more than the dynamic processing module associated with loudspeaker 205b, because loudspeaker 205b has a substantially greater ability to reproduce sound in the bass range. If the volume of the signal associated with the stationary object fluctuates, then when the volume is high, the dynamic processing module associated with loudspeaker 205d will cause the level of the audio in the bass range to be reduced much more than the level of the same audio will be reduced by the dynamic processing module associated with loudspeaker 205b. This difference in level will cause the apparent location of the stationary object to change. Therefore, an improved solution is needed.

[0080] Some embodiments of the present disclosure are systems and methods for rendering (or rendering and playback) a spatial audio mix (e.g., rendering an audio stream or multiple audio streams) for playback by at least one (e.g., all or some) of the smart audio devices in a group of smart audio devices (e.g., a coordinated group of smart audio devices) and / or for playback by at least one (e.g., all or some) of the speakers in another group of speakers. Some embodiments are methods (or systems) for such rendering (e.g., including generation of speaker feeds) and playback of the rendered audio (e.g., playback of the generated speaker feeds). Examples of such embodiments include the following:

[0081] Systems and methods for audio processing may include rendering audio (e.g., such as rendering a spatial audio mix by rendering an audio stream or multiple audio streams) for playback by at least two speakers (e.g., all or some of a set of speakers), including rendering by:

[0082] (a) combining individual loudspeaker dynamics processing configuration data into limit thresholds (playback limit thresholds) for the individual loudspeakers, thereby determining listening environment dynamics processing configuration data (such as combined thresholds) for the plurality of loudspeakers;

[0083] (b) performing dynamic processing on audio (e.g., an audio stream indicative of a spatial audio mix) using listening environment dynamic processing configuration data (e.g., combining thresholds) for the plurality of loudspeakers to generate processed audio; and

[0084] (c) Render the processed audio as a speaker feed.

[0085] According to some embodiments, process (a) can be performed as follows Figure 3 . The smart home hub 305 can be configured to obtain the individual loudspeaker dynamic processing configuration data of each of the M loudspeakers via the interface system. In this embodiment, the individual loudspeaker dynamic processing configuration data includes an individual loudspeaker dynamic processing configuration data set for each of the multiple loudspeakers. According to some examples, the individual loudspeaker dynamic processing configuration data of one or more loudspeakers may correspond to one or more capabilities of one or more loudspeakers. In this example, each individual loudspeaker dynamic processing configuration data set includes at least one type of dynamic processing configuration data. In some examples, the smart home hub 305 can be configured to obtain an individual loudspeaker dynamic processing configuration data set by querying each of the loudspeakers 205a-205m. In other embodiments, the smart home hub 305 can be configured to obtain an individual loudspeaker dynamic processing configuration data set by querying the data structure of the previously obtained individual loudspeaker dynamic processing configuration data set stored in the memory.

[0086] In some examples, process (b) can be performed as follows Figure 3 The following describes some detailed examples of processes (a) and (b).

[0087] In some examples, the rendering of process (c) can be performed by Figure 3 In some embodiments, the audio processing may involve:

[0088] (d) performing dynamic processing on the rendered audio signal in accordance with individual loudspeaker dynamic processing configuration data for each loudspeaker (e.g., limiting the loudspeaker feeds in accordance with playback limiting thresholds associated with the corresponding loudspeakers, thereby generating limited loudspeaker feeds). For example, process (d) may be performed by Figure 3 It is performed by the dynamic processing modules A to M shown in .

[0089] The speaker may include at least one (e.g., all or some) of (or coupled to) a smart audio device in a group of smart audio devices. In some embodiments, to generate the limited speaker feed in step (d), the speaker feed generated in step (c) may be processed by a second stage of dynamic processing (e.g., by an associated dynamic processing system of each speaker), for example, to generate the speaker feed before ultimately playing back through the speaker. For example, the speaker feed (or a subset or portion thereof) may be provided to a dynamic processing system of each different speaker (e.g., a dynamic processing subsystem of a smart audio device, wherein the smart audio device includes or is coupled to a relevant one of the speakers), and the processed audio output from each of the dynamic processing systems may be used to generate a speaker feed for a relevant one of the speakers. After speaker-specific dynamic processing (in other words, dynamic processing performed independently for each speaker), the processed (e.g., dynamically limited) speaker feed may be used to drive the speaker to cause playback of the sound.

[0090] The first stage of dynamic processing (in step (b)) can be designed to reduce perceptually distracting shifts in spatial balance that would result if steps (a) and (b) were omitted, and the dynamically processed (e.g., limited) speaker feeds resulting from step (d) can be generated in response to the original audio (rather than in response to the processed audio generated in step (b)). This can prevent undesirable shifts in the spatial balance of the mix. The second stage of dynamic processing operating on the rendered speaker feeds from step (c) can be designed to ensure that there is no speaker distortion, as the dynamic processing of step (b) may not necessarily guarantee that the signal level has been reduced to below the threshold for all speakers. In some examples, the combination of individual loudspeaker dynamic processing configuration data (e.g., the combination of thresholds in the first stage (step (a))) can involve a step of averaging individual loudspeaker dynamic processing configuration data (e.g., limiting thresholds) across speakers (e.g., across smart audio devices) or taking the minimum value of individual loudspeaker dynamic processing configuration data (e.g., limiting thresholds) across speakers (e.g., across smart audio devices).

[0091] In some embodiments, when the first stage of dynamics processing (in step (b)) operates on audio indicative of a spatial mix (e.g., audio of an object-based audio program, including at least one object channel and optionally also at least one loudspeaker channel), the first stage may be implemented in accordance with a technique for performing audio object processing using spatial zones. In this case, the combined individual loudspeaker dynamics processing configuration data (e.g., combined limit thresholds) associated with each zone may be derived by (or as) a weighted average of the individual loudspeaker dynamics processing configuration data (e.g., individual loudspeaker limit thresholds), and the weighting may be given or determined at least in part by each loudspeaker's spatial proximity to the zone and / or position within the zone.

[0092] In the example embodiment, a plurality of M loudspeakers (M ≥ 2) is assumed, where each loudspeaker is indexed by a variable i. Associated with each loudspeaker i is a set of frequency-varying playback limit thresholds T i [f], where the variable f represents an index into a finite set of frequencies that specifies a threshold. (Note that if the size of the frequency set is one, then the corresponding single threshold can be considered broadband, applying to the entire frequency range.) According to this example, these thresholds are used by each speaker in its own independent dynamics processing function to limit the audio signal to frequencies below the threshold T i [f] for specific purposes, such as preventing a speaker from distorting or preventing a speaker from playing above a level that would be considered objectionable in its vicinity.

[0093] Figure 4A , Figure 4B and Figure 4C The example of playback limit threshold and corresponding frequency is shown. For example, the frequency range shown can span the frequency range (for example, 20Hz to 20kHz) that ordinary human can hear. In these examples, the playback limit threshold is indicated by the vertical axis of Figure 400a, 400b and 400c, and the vertical axis is marked as "level threshold" in these examples. The playback limit threshold / level threshold increases in the direction of the arrow on the vertical axis. For example, the playback limit threshold / level threshold can be expressed in decibels. In these examples, the horizontal axis of Figure 400a, 400b and 400c indicates frequency, and frequency increases along the direction of the arrow on the horizontal axis. For example, the playback limit threshold indicated by curves 400a, 400b and 400c can be implemented by the dynamic processing module of a separate loudspeaker.

[0094] Figure 4A Graph 400a of FIG. 400 shows a first example of a playback limit threshold as a function of frequency. Curve 405a indicates the playback limit threshold for each corresponding frequency value. In this example, at the bass frequency f b At the input level T iThe received input audio will be processed by the dynamics module at an output level T o Output. For example, the bass frequency f b can be in the range of 60 Hz to 250 Hz. However, in this example, at the treble frequency f t At the input level T i The received input audio will be processed by the dynamics processing module at the same level (input level T i ) output. For example, the treble frequency f t may be in the range above 1280 Hz. Thus, in this example, curve 405a corresponds to a dynamics processing module that applies a significantly lower threshold to bass frequencies than to treble frequencies. Such a dynamics processing module may be suitable for a loudspeaker without a woofer (e.g., Figure 2 loudspeaker 205d).

[0095] Figure 4B Graph 400b of FIG. 400b shows a second example of a playback limit threshold as a function of frequency. Curve 405b indicates that Figure 4A The same bass frequency f shown in b At the input level T i The input audio received at the lower level will be processed by the dynamics module at a higher output level T o Thus, in this example, curve 405b corresponds to a dynamic processing module that does not apply a threshold for bass frequencies below curve 405a. Such a dynamic processing module may be suitable for a loudspeaker having at least a small woofer (e.g., Figure 2 loudspeaker 205b).

[0096] Figure 4C Graph 400c of FIG. 400c shows a third example of a playback limit threshold as a function of frequency. Curve 405c (a straight line in this example) indicates that Figure 4A The same bass frequency f shown in b At the input level T i Input audio received at 405C will be output by the dynamics processing module at the same level. Thus, in this example, curve 405C corresponds to a loudspeaker that may be suitable for being able to reproduce a wide range of frequencies, including bass frequencies. It will be observed that, for simplicity, the dynamics processing module may approximate curve 405C by implementing curve 405D, which applies the same threshold to all indicated frequencies.

[0097] A spatial audio mix for multiple speakers may be rendered using a known rendering system such as Center of Mass Amplitude Panning (CMAP) or Flexible Virtualization (FV). The rendering system generates a speaker feed for each of the multiple speakers from the constituent components of the spatial audio mix. In some previous examples, the threshold T is then used by the associated dynamics processing function of each speaker. i [f] to process the speaker feeds independently. Without the benefit of the present disclosure, this described rendering scenario may result in a distracting shift in the perceived spatial balance of the rendered spatial audio mix. For example, one of the M speakers (e.g., on the right side of the listening area) may be much less capable than the other speakers (e.g., rendering audio in the bass range), and thus the threshold T for that speaker may be at least slightly lower in a certain frequency range. i [f] may be significantly lower than the thresholds of the other speakers. During playback, the dynamics processing module of that speaker will reduce the level of the right component of the spatial mix significantly more than the left component. Listeners are very sensitive to this dynamic shift between the left / right balance of the spatial mix and may find the result very distracting.

[0098] To address this issue, in some examples, individual loudspeaker dynamics processing configuration data (e.g., playback limiting thresholds) for individual speakers of a listening environment are combined to create listening environment dynamics processing configuration data for all loudspeakers of the listening environment. The listening environment dynamics processing configuration data can then be used to perform dynamic processing first in the context of the entire spatial audio mix before rendering it as a speaker feed. Because this first stage of dynamics processing has access to the entire spatial mix and not just an individual speaker feed, processing can be performed in a manner that does not cause a distracting shift in the perceived spatial balance of the mix. The individual loudspeaker dynamics processing configuration data (e.g., playback limiting thresholds) can be combined in a manner that eliminates or reduces the amount of dynamic processing performed by any individual dynamics processing function for an individual speaker.

[0099] In one example of determining listening environment dynamics processing configuration data, individual loudspeaker dynamics processing configuration data for individual loudspeakers (e.g., playback limiting thresholds) may be combined into a single set of listening environment dynamics processing configuration data (e.g., frequency-dependent playback limiting thresholds) that is applied to all components of the spatial mix in a first stage of dynamics processing. According to some such examples, because the limits for all components are the same, the spatial balance of the mix can be maintained. One way to dynamically process configuration data (e.g., playback limit thresholds) for the combined individual loudspeakers is to take the minimum value across all loudspeakers i:

[0100]

[0101] This combination essentially eliminates the need for individual dynamics processing for each speaker, because the spatial mix is ​​first constrained to be below the threshold of the speaker with the least capability at each frequency. However, this strategy can be too aggressive. Many speakers may be playing back at levels below their capabilities, and the combined playback level of all speakers may be objectionably low. For example, if Figure 4A The thresholds in the bass range shown in Figure 4C If the loudspeaker corresponding to the threshold of the latter loudspeaker is not necessarily low in the bass range. An alternative combination for determining the listening environment dynamic processing configuration data is to average (mean) the individual loudspeaker dynamic processing configuration data across all loudspeakers of the listening environment. For example, in the context of a playback limit threshold, the mean may be determined as follows:

[0102]

[0103] For this combination, the overall playback level may increase compared to taking the minimum because the first stage of dynamics processing is limiting to a higher level, thereby allowing the more capable speakers to play back louder. For speakers whose individual limiting thresholds fall below the mean, their independent dynamics processing functions may still limit their associated speaker feeds if necessary. However, the first stage of dynamics processing may reduce the need for this limiting because some initial limiting has already been performed on the spatial mix.

[0104] According to some examples of determining the listening environment dynamics processing configuration data, an adjustable combination can be created by interpolating between the minimum and mean values ​​of the individual loudspeaker dynamics processing configuration data by adjusting the parameter α. For example, in the context of a playback limit threshold, the interpolation can be determined as follows:

[0105]

[0106] Other combinations of individual loudspeaker dynamics processing configuration data are possible, and the present disclosure is intended to cover all such combinations.

[0107] Figure 5A and Figure 5B500a and 500b, the input signal level in decibels is shown on the horizontal axis, and the output signal level in decibels is shown on the vertical axis. As with other disclosed examples, specific thresholds, ratios, and other values ​​are shown by way of example only and are not intended to be limiting.

[0108] exist Figure 5A In the example shown in , the output signal level is equal to the input signal level below the threshold, in this example the threshold is -10 dB. Other examples may involve different thresholds, for example, -20 dB, -18 dB, -16 dB, -14 dB, -12 dB, -8 dB, -6 dB, -4 dB, -2 dB, 0 dB, 2 dB, 4 dB, 6 dB, etc. In the case of being above the threshold, various examples of compression ratios are shown. A ratio of N:1 means that for every N dB increase in the input signal above the threshold, the output signal level will increase by 1 dB. For example, a compression ratio of 10:1 (line 505e) means that for every 10 dB increase in the input signal above the threshold, the output signal level will only increase by 1 dB. A compression ratio of 1:1 (line 505a) means that even above the threshold, the output signal level is still equal to the input signal level. Lines 505b, 505c and 505d correspond to compression ratios of 3:2, 2:1 and 5:1. Other embodiments may provide different compression ratios, such as 2.5:1, 3:1, 3.5:1, 4:3, 4:1, etc.

[0109] Figure 5B An example of a "knee point" is shown, which controls how the compression ratio changes at or near a threshold, in this example 0 dB. According to this example, a compression curve with a "hard" knee point consists of two straight line segments, segment 510a reaching the threshold, and segment 510b being above the threshold. A hard knee point can be simpler to implement, but may result in artifacts.

[0110] exist Figure 5B , an example of a "soft" knee is also shown. In this example, the soft knee spans 10 dB. According to this embodiment, above and below the 10 dB span, the compression ratio of the compression curve with the soft knee is the same as the compression ratio of the compression curve with the hard knee. Other embodiments may provide various other shapes of "soft" knees, which may span more or fewer decibels, may indicate different compression ratios over the span, etc.

[0111] Other types of dynamic range compression data may include "attack" data and "release" data. The attack is the time period over which the compressor reduces gain (e.g., in response to an increasing level at the input) to reach a gain determined by the compression ratio. The attack time of a compressor is typically between 25 milliseconds and 500 milliseconds, but other attack times are possible. The release is the time period over which the compressor increases gain (e.g., in response to a decreasing level at the input) to reach an output gain determined by the compression ratio (or to the input level if the input level has fallen below a threshold). For example, the release time may be in the range of 25 milliseconds to 2 seconds.

[0112] Therefore, in some examples, for each loudspeaker in a plurality of loudspeakers, the individual loudspeaker dynamic processing configuration data may include a dynamic range compression data set. The dynamic range compression data set may include threshold data, input / output ratio data, attack data, release data and / or inflection point data. One or more of these types of individual loudspeaker dynamic processing configuration data may be combined to determine the listening environment dynamic processing configuration data. As described above, with reference to the combination performed on the playback limit threshold, in some examples, the dynamic range compression data may be averaged to determine the listening environment dynamic processing configuration data. In some instances, the minimum or maximum value of the dynamic range compression data may be used to determine the listening environment dynamic processing configuration data (e.g., the maximum compression ratio). In other embodiments, an adjustable combination interpolated between the minimum and mean values ​​of the dynamic range compression data for the individual loudspeaker dynamic processing may be created (e.g., via the adjustment parameters described above with reference to equation (C)).

[0113] In some of the examples described above, in a first stage of dynamic processing, a single listening environment dynamic processing configuration data set (eg, a combined threshold set) is processed. ) is applied to all components of the spatial mix. Such an implementation may maintain the spatial balance of the mix, but may produce other unwanted artifacts. For example, "spatial ducking" may occur when a very loud portion of the spatial mix in an isolated spatial region causes the entire mix to be turned down. Other softer components of the mix that are spatially distant from the loud component may be perceived as becoming unnaturally soft. For example, soft background music may be heard in the surround field of the spatial mix at frequencies below the combining threshold. The first stage of dynamics processing will then reduce the level of the entire mix to the threshold. Because the music is spatially separated from the gunfire, this may be perceived as an unnatural ducking in the continuous stream of music.

[0114] Some examples of zone-based approaches

[0115] To address these issues, some embodiments allow for independent or partially independent dynamic processing of different "spatial zones" of a spatial mix. A spatial zone can be thought of as a subset of the spatial region over which the entire spatial mix is ​​rendered. Although much of the following discussion provides examples of dynamic processing based on playback limiting thresholds, these concepts are equally applicable to other types of individual loudspeaker dynamic processing configuration data and listening environment dynamic processing configuration data.

[0116] Figure 6 Examples of spatial zones of a listening environment are shown. Figure 6 An example of a spatial mixing area (represented by the entire square) is depicted, subdivided into three spatial zones: front zone, center zone, and surround zone. Other examples may include more spatial zones, fewer spatial zones, different spatial zones, or a combination thereof. For example, some examples may include one or more top zones.

[0117] Although Figure 6 The spatial zones in the square are depicted as having hard boundaries, but in reality, it is helpful to treat the transition from one spatial zone to another as continuous. For example, a spatial mix component located in the middle of the left edge of the square may have half of its level assigned to the front zone and half to the surround zone. The signal level of each component from the spatial mix can be allocated and accumulated into each spatial zone in this continuous manner. The dynamic processing function can then operate independently for each spatial zone on the overall signal level assigned to it from the mix. For each component of the spatial mix, the results of the dynamic processing from each spatial zone (e.g., the time-varying gain per frequency) can then be combined and applied to the component. In some examples, this combination of spatial zone results is different for each component and is a function of the allocation to each zone for that particular component. The end result is that components of the spatial mix with similar spatial zone allocations receive similar dynamic processing, but independence between spatial zones is allowed. Spatial zones can be advantageously selected to prevent objectionable spatial shifts, such as left / right imbalance, while still allowing some spatially independent processing (e.g., to reduce other artifacts such as spatial ducking as described).

[0118] The technique of processing the spatial mix by spatial zones may be advantageously employed in one or more of the first stages of dynamics processing mentioned above (e.g., stage (a), stage (b), or both). For example, different combinations of individual loudspeaker dynamics processing configuration data (e.g., playback limit thresholds) across loudspeakers i may be calculated for each spatial zone. The set of combined zone thresholds may be given by , where index j refers to one of the plurality of spatial regions. The dynamic processing module may be configured to have an associated threshold The method operates independently on each spatial region of the spatial mix, and according to the techniques described above, the results can be applied back to the constituent components of the spatial mix.

[0119] Consider that the rendered spatial signal consists of a total of K individual component signals x k [t], each component signal has an associated expected spatial position (possibly time-varying). One particular method for implementing zone processing involves computing a representation describing each audio signal x based on its expected spatial position relative to the position of zone j. k [t] The time-varying translation gain α of the contribution to the area kj [t]. These translation gains can be advantageously designed to follow the power-preserving translation law that requires the sum of the squares of the gains to be equal to unity. Based on these translation gains, the region signal s ( [t] can be calculated as the sum of the component signals weighted by the panning gain of the zone:

[0120]

[0121] Then, each zone signal s j [t] can be determined by the zone threshold The parameterized dynamic processing functions DP are processed independently to generate the frequency-varying and time-varying zone correction gains G j :

[0122]

[0123] The zone correction gain can then be calculated by adding it to each individual component signal x k The zone translation gains of [t] are proportionally combined to calculate the frequency-varying and time-varying correction gains for the individual component signals:

[0124]

[0125] These signals correct the gain G k A filter bank may then be applied to each component signal to produce a dynamically processed component signal which can then be rendered as a speaker signal.

[0126] The combination of individual loudspeaker dynamics processing configuration data (such as speaker playback limit thresholds) for each spatial zone can be performed in various ways. As an example, the spatial zone and speaker dependency weights w can be used. ij [f] Set the spatial zone playback limit threshold Calculated as the speaker playback limit threshold T i The weighted sum of [f]:

[0127]

[0128] Similar weighting functions may be applied to other types of individual loudspeaker dynamics processing configuration data. Advantageously, the combined individual loudspeaker dynamics processing configuration data (e.g., playback limiting thresholds) for a spatial zone may be biased towards the individual loudspeaker dynamics processing configuration data (e.g., playback limiting thresholds) of the loudspeakers that are most responsible for playing back the components of the spatial mix associated with that spatial zone. This may be accomplished by assigning a weight w to the frequency f. ij [f] is set to be implemented as a function of each speaker's responsibility for rendering the component of the spatial mix associated with that zone.

[0129] Figure 7 Shows Figure 6 Example of a loudspeaker within a spatial region of . Figure 7 Depicted with Figure 6 The same zones, but with the positions of five example loudspeakers (speakers 1, 2, 3, 4, and 5) responsible for rendering the superimposed spatial mix. In this example, loudspeakers 1, 2, 3, 4, and 5 are represented by diamond shapes. In this particular example, loudspeaker 1 is primarily responsible for rendering the center zone, loudspeakers 2 and 5 are responsible for the front zone, and loudspeakers 3 and 4 are responsible for the surround zone. The weights w can be created based on this conceptual one-to-one mapping of loudspeakers to spatial zones. ij [f], but as with the spatial zone-based processing of the spatial mix, a more continuous mapping may be preferred. For example, speaker 4 is very close to the front zone, and components of the audio mix located between speakers 4 and 5 (although in the conceptual front zone) will likely be played back primarily by a combination of speakers 4 and 5. As such, it makes sense that the individual loudspeaker dynamics processing configuration data (e.g., playback limit thresholds) for speaker 4 contribute to the combined individual loudspeaker dynamics processing configuration data (e.g., playback limit thresholds) for the front and surround zones.

[0130] One way to achieve this continuous mapping is to set the weights w ij [f] is set equal to a speaker participation value that describes the relative contribution of each speaker i in rendering the component associated with the spatial zone,. Such values ​​may be derived directly from a rendering system responsible for rendering to the speakers (e.g., from step (c) described above) and a set of one or more nominal spatial positions associated with each spatial zone. The set of nominal spatial positions may include a set of positions within each spatial zone.

[0131] Figure 8 Shows coverage in Figure 7 Examples of spatial zones and nominal spatial positions on the loudspeakers. The nominal positions are indicated by numbered circles: associated with the front zone are two positions at the top corners of the square, associated with the central zone is a single position at the top middle of the square, and associated with the surround zone are two positions at the bottom corners of the square.

[0132] To compute speaker participation values ​​for a spatial zone, each nominal position associated with the zone may be rendered by a renderer to generate speaker activations associated with that position. For example, in the case of CMAP, these activations may be the gain of each speaker, or in the case of FV, a complex value for each speaker at a given frequency. Next, for each speaker and zone, these activations may be accumulated across each nominal position associated with the spatial zone to produce a value g ij [f]. This value represents the total activation of speaker i for rendering the entire set of nominal positions associated with spatial zone j. Finally, the speaker participation value in a spatial zone can be calculated as the cumulative activation g normalized by the sum of all these accumulated cross-speaker activations ij [f]. The weight can then be set to that speaker participation value:

[0133]

[0134] The described normalization ensures that w across all speakers i ij The sum of [f] equals one, which is a desirable property of the weights in equation (G).

[0135] According to some embodiments, the process described above for calculating speaker participation values ​​and combining thresholds based on these values ​​can be performed as a static process, where the resulting combined threshold is calculated once during a setup procedure that determines the layout and functionality of the speakers in the environment. In such a system, it can be assumed that once set up, both the dynamic processing configuration data for the individual loudspeakers and the manner in which the rendering algorithm activates the loudspeakers based on the desired audio signal positions remain static. However, in some systems, both of these aspects may change over time (e.g., in response to changing conditions in the playback environment), and therefore, it may be desirable to update the combined threshold in accordance with the process described above in a continuous or event-triggered manner to account for such changes.

[0136] Example of mapping audio signal components related to workload signal

[0137] The time-varying and frequency-varying mapping of the component signals of a spatial audio mix (also referred to herein as an audio object) can generally be represented by the following equations:

[0138]

[0139] Here, variables t and f represent the audio object signal O i , loudspeaker signal S j and the mapping H from object i to loudspeaker signal, ij The number of audio objects is determined by N. oGiven, where N o ≥2, and the number of loudspeaker signals is determined by N s Given, where N s ≥ 2. Mapping H ij It can generally be considered as a time-varying filter, and its form and application can take many forms, such as real or complex time-varying gains applied to various frequency bands such as the following filter groups: time-varying finite impulse response (FIR) filter or time-varying infinite impulse response (IIR) filter, orthogonal mirror filter (QMF) or short-time Fourier transform (STFT) applied to audio objects in the time domain.

[0140] According to this example, with each audio object signal O i Associated is the expected perceived spatial location which can change over time As an example, such a position may correspond to a Dolby Atmos TM The time-varying 3D (x, y, z) metadata of an audio object that is part of a spatial audio mix. In another example, the audio object signal may correspond to a channel in a multi-channel signal (such as a Dolby 5.1 signal), and the expected position may be fixed. In either case, the expected perceived spatial position may have been selected by the audio content creator. In this example, with each loudspeaker signal S j Associated with this is the assumed physical location of the loudspeaker And the collection Represents loudspeaker k = 1...N s all loudspeaker locations.

[0141] For a given audio object signal i and loudspeaker j, the signal E ij (f, t) represents a time-varying and frequency-varying representation of the loudspeaker signal level associated with object i relative to the maximum playback limit of loudspeaker j. For simplicity, this will therefore be referred to as the workload signal. As the level of the workload signal increases, this indicates that rendering object i on loudspeaker j will cause loudspeaker j to approach or exceed its playback limit threshold. The set {E ik (f, t)} represents all loudspeakers k=1...N s The workload signal associated with object i.

[0142] Then, we can Megaphone Location Set and workload signal set {E ik (f, t)} to calculate the mapping H ij The calculation, for example, is as follows:

[0143]

[0144] In this example, the mapping function M j Compute the mapping H across all loudspeaker signals ij , the purpose of which is to make all loudspeaker signals at a given position When playing back the audio object O on the loudspeakers at the same time i The perceived spatial position of In other words, each mapping is calculated so that the expected perceptual spatial position of the associated audio signal is approximately achieved when the loudspeaker signal is played back on two or more corresponding loudspeakers located at the associated loudspeaker positions. In some examples, "approximately achieving" the expected perceptual spatial position of the associated audio signal can involve minimizing the difference between the perceptual spatial position and the expected perceptual spatial position given the available loudspeakers and the associated loudspeaker positions. Again, in this example, achieving this approximation is subject to the behavioral constraints imposed by the workload signal. For a fixed object position, as the workload signal E ij (f, t) increases to a certain threshold μ j Above, mapping H ij should be reduced, while at the same time for the workload signal to be less than the threshold μ k One or more other loudspeakers k, mapping H ik should increase. This behavior can be described more precisely by considering the quantity in question at two instants t1 and t2:

[0145] For one or more other loudspeakers k≠j, where E ik (f, t1) and E ik (f, t2)<μ k ,

[0146] If E ij (f,t2)>E ij (f,t1)>μ j ,in

[0147] but

[0148] |H ij (f, t2)|<|H ij (f, t1)|

[0149] and

[0150] |H ik(f, t2)|>|H ik (f, t1)|.

[0151] The mathematically stated behavior above summarizes the high-level idea that, according to some examples, when the level of the spatial mix approaches the playback limiting threshold of a particular loudspeaker, the mapping of components to that loudspeaker is reduced in order to increase the mapping to other loudspeakers where the spatial mix level is further from the limiting threshold of the other loudspeakers.

[0152] Typically, the frequency-variant mapping described above can be implemented over the entire audible frequency range using a frequency resolution comparable to that of human perception. For example, in one embodiment, the mapping can be calculated for 20 discrete frequency bands with a resolution of approximately 2ERB (Equivalent Rectangular Bandwidth). Utilizing this spacing helps maintain the perceptual transparency of the system.

[0153] However, in other embodiments, it may be advantageous to calculate the dynamics mapping only over a subset of the available frequency range of the loudspeaker. For example, the dynamics mapping may be calculated only over frequencies less than a threshold frequency (e.g., 500 Hz), where there may be differences in the capabilities of the loudspeakers. In some examples, above 500 Hz—or above another threshold—all loudspeakers have the same capabilities, in which case the mapping may be independent of the signal level.

[0154] According to some examples, "approximately achieving" the desired perceived spatial location of the associated audio signal can involve minimizing a cost function. In some such examples, the mapping described by Equation 2a can be advantageously achieved by minimizing the following objective as one of the cost functions: When playing back simultaneously on the loudspeakers at i The perceived spatial position of According to some such examples, the cost function can be expressed as follows:

[0155]

[0156] In Equation 2b, C(g) represents the cost C as a function of g, which represents the N number of speaker activations. s dimensional vector. In equation 2b, the set Represents a set of N s the location of the loudspeakers, and represents the desired perceived spatial position of the audio signal. In this example, the first term C in Equation 2b spatial The second term C models how close the desired spatial impression is to being realized as a mapping function of audio objects to loudspeaker signals, andproximity Assign a cost to each loudspeaker in the activated loudspeakers. Term C proximity One purpose of is to create a sparse solution in which only loudspeakers close to the expected spatial location of the object signal are activated. According to this example, the cost function includes one or more additional dynamically configurable terms for the activation penalty, thereby allowing the spatial rendering to be modified in response to various other controls. In equation 2b, the term These additional cost items are represented by represents a set of one or more properties of an audio signal being rendered (e.g., of an object-based audio program), a collection of one or more properties representing the speakers through which the audio is being rendered, and Represents one or more additional external inputs. Each item Returns the cost as a function of the activation g associated with a combination of one or more properties of the audio signal, the loudspeaker, and / or the external input, typically a set of In some examples, the set At least from or Only one element of any one of .

[0157] Examples include, but are not limited to:

[0158] The desired perceived spatial location of the audio signal;

[0159] the level of the audio signal (which may vary over time); and / or

[0160] The frequency spectrum of the audio signal (possibly varying over time).

[0161] Examples include, but are not limited to:

[0162] The location of the loudspeakers in the listening space;

[0163] The frequency response of the loudspeaker;

[0164] ·Playback level limitation of loudspeakers;

[0165] Parameters of the dynamics processing algorithms within the loudspeaker, such as limiter gain;

[0166] The measurement or estimation of the acoustic transmission from each loudspeaker to the other loudspeakers;

[0167] · Measurement of the performance of echo cancellers on loudspeakers; and / or

[0168] • The relative synchronization of the loudspeakers with respect to each other.

[0169] Examples include, but are not limited to:

[0170] The location of one or more listeners or speakers in the playback space;

[0171] The measurement or estimate of the acoustic transmission from each loudspeaker to the listening position;

[0172] The measurement or estimation of the acoustic transmission from a speaker to a set of loudspeakers;

[0173] The location of some other landmarks in the playback space; and / or

[0174] A measurement or estimate of the acoustic transmission from each loudspeaker to some other landmark in the playback space.

[0175] By converting the workload signal E ij (f, t) is mapped to the loudspeaker-by-loudspeaker activation penalty of Eq. 2b, the mapping function M j The stated goal of can be achieved with a process having the following property: for each individual audio object i all loudspeakers j=1...N are optimized simultaneously s Mapping H on ij More specifically, for any particular frequency f, time t, and object signal i,

[0176]

[0177] in, is the vector that minimizes the cost function in Equation 2b The jth element of :

[0178]

[0179] Workload signal E ij The exact manner in which (f, t) may be used to represent the loudspeaker signal level with respect to the maximum playback limit varies, and the inventors contemplate many possible options. In some embodiments, the workload signal may be or may correspond to a digital level that is an input or output of a limiter. In other embodiments, the workload signal may be the actual gain applied by the limiter, i.e., a signal indicating that the loudspeaker level has exceeded the playback threshold. In some other embodiments, the workload signal may be an acoustic signal (e.g., measured in decibels of sound pressure level, i.e., dBSPL, at a particular distance). For example, the acoustic signal may be derived from the digital level using the loudspeaker sensitivity and a known analog amplifier gain.

[0180] Regardless of the specific form of representation, the workload signal E ijThe construction of (f, t) is an important component to the functionality of many examples of the present disclosure because in those examples, the workload signal E ij (f, t) determines where and when the signal energy transfer occurs between the loudspeakers. As mentioned above, for each audio object signal, the workload signal can be calculated as a function of one or more audio object signals in the audio object signal and their perceived spatial positions, for example, as shown below:

[0181]

[0182] By considering the entire spatial mix in its construction, signal energy from any part of the spatial mix can be combined into a workload signal for each object for various purposes. One possible purpose is to combine object signals in a way that represents how the audio object signals will likely accumulate into a loudspeaker signal. Another possible purpose is to use specific spatial relationships to bundle the energy redistribution behavior of the present invention between objects. Some preferred constructions of the workload signal can achieve both of these purposes.

[0183] In considering the preferred construction of the workload signal, it may be convenient to revisit its more detailed definition: a time-varying and frequency-varying representation of the loudspeaker signal level associated with object i relative to the maximum playback limit of loudspeaker i. This general representation concept can be literally translated into the following specific construction of the workload signal:

[0184] E ij (f, t) = 10log 10 (L i (f, t))-τ j (f) (4)

[0185] In Equation 4, the term L i (f, t) represents the time-varying and frequency-varying signal level associated with object i, and τ j (f) represents the frequency-dependent playback limit of loudspeaker L. In some examples, similar to the loudspeaker position, it is assumed that the playback limit of the loudspeaker (expressed in decibels in this example) has been characterized and provided to the control system. In this way, the calculation of the workload signal of the audio object across all loudspeakers is simplified to a single level signal L i Calculation of (f, t).

[0186] As mentioned above, the mapping H ij These audio object level signals L i (f, t) should represent how the audio object signal will likely be accumulated into the loudspeaker signal. This is a recurring relationship because the mapping H ijis used to explicitly generate the loudspeaker signals, as shown in Equation 2a. One solution to this cyclic problem is to produce a simple region-based rendering of the audio object, where no information about the loudspeakers is required. Instead, the signal energy of a finite set of spatial regions can be generated by a simple panning rule. In general, the control system can adopt N z In one useful embodiment, the rendering process may involve four zones, which may be a front zone, a center zone, a surround zone, and a top zone. Other examples may involve more or fewer zones. In some such examples, the rendering of audio objects into zones may be governed by a simple broadband panning rule that is a function of the expected spatial position of the audio objects:

[0187]

[0188] In Equation 5, g il (t) represents the translation gain of audio object i to spatial region l. z It is beneficial to maintain the power of the panning gain in a collection of zones, for example, as follows:

[0189]

[0190] According to these zone translation gains, the time-varying and frequency-varying zone power spectrum Z l (f, t) can be calculated by accumulating the power spectra of the audio objects weighted by their corresponding panning gains, for example, as follows:

[0191]

[0192] These zone power spectra correspond to the energy distribution of the overall spatial mix within the designed spatial zones. Assuming that the loudspeakers are distributed across the zones - e.g., it is expected that each zone includes at least one loudspeaker - these power spectra represent a reasonable approximation of the signal levels that will appear in the loudspeaker signals within the zones.

[0193] The present disclosure is unique in that—to the best of the inventors' knowledge—in some examples the zone power spectrum can then be mapped back to the object level signal L by applying a zone panning gain per object i (f, t), for example, as follows:

[0194]

[0195] In the case where an audio object i is panned across multiple zones, according to L in Equation 8 i The construction of (f, t) combines the region power spectra in a manner proportional to this translation. i(f, t) is thus used as an approximation of the level of the loudspeaker signal to which object i may be rendered. i (f, t) will be similar, which means that the behavior of the dynamic energy redistribution of the present invention will be similar for these objects. In some advantageous embodiments, the zones can be designed to group objects across the left / right axis, thereby limiting the behavior of energy redistribution to be similar for the left and right sides of the spatial mix. Such zone design helps reduce perceptually distracting left / right image shifts.

[0196] Finally, in some examples, the object level signal L i (f, t) can be compared with the speaker limit threshold τ j (f) Combining to calculate workload signal E ij (f, t), for example, as shown in Equation 4.

[0197] In the case where the spatial audio mix consists of channel-based audio (i.e., where each audio object signal may correspond to a channel in a multi-channel signal, such as a Dolby 5.1 signal or a Dolby 5.1.4 signal, and the desired position of each object is a fixed spatial position), the zone-based construction of the level signal just described may be simplified along multiple dimensions. Because the positions of the object / channel signals are fixed, the mapping of channels to zones may not require a dynamic panning function as shown in Equation 5. Furthermore, this mapping may be simplified to a set of one-to-one mappings from channels to zones and back. For example, for a 5.1.4 signal, the left and right front channels may be mapped to the front zone, the center channel may be mapped to the center zone, the left and right surround channels may be mapped to the surround zone, and all four top channels may be mapped to the top zone. With this mapping, Equation 7 for calculating the zone power spectrum simplifies to summing over the power spectra of each channel belonging to the zone, and Equation 8 simplifies to making the level signal for each channel equal to the zone power spectrum corresponding to the single zone to which the channel is mapped.

[0198] The above-described method of calculating the object level signal relies on an approximation of the way in which objects are accumulated into the loudspeaker signal. To eliminate this approximation, an alternative construction of the object level signal may be employed which employs direct feedback of the loudspeaker signal from the previous time interval, but at the expense of introducing a delay in the activation of the energy redistribution of the present invention. Thus, this second category of methods may be referred to herein as feedback methods.

[0199] One such alternative embodiment is similar to

[0200] The method represented by Equation 8, except that the zone power spectrum is replaced by the power spectrum of the loudspeaker signal from the previous time interval, and the object-to-zone panning gain is replaced by a normalized version of the object-to-loudspeaker mapping from the previous time interval, for example, as shown below:

[0201]

[0202] The normalization of the object-to-speaker mapping shown in equation 9b ensures that the weighted combination of the speaker signals in equation 9a is performed in a power-preserving manner. If the object-to-speaker mapping is inherently calculated in a power-preserving manner, as may be the case in some advantageous embodiments, this normalization step may be unnecessary.

[0203] The construction of the object level signal in equation 9a combines all the speaker signals from the previous time interval in proportion to the mapping of the audio object to each speaker signal. As such, it is a direct representation of the speaker signal level to which the audio object is mapped. However, one problem with this construction is that it lacks any concept of associating audio objects in similar spatial regions. The absence of this feature can introduce instabilities in the imaging of the resulting rendered audio.

[0204] As a third example of constructing an audio object level signal, the zone-based processing of the first example may be combined with a more accurate representation of the loudspeaker signal level provided by the feedback method of the second example. In some such alternative examples, the audio object level signal may be constructed as a weighted sum of the zone signals, as in the first example, but with the zone power spectrum Z l (f, t) can be represented by the loudspeaker power spectrum V l (f, t) instead, for example, as follows:

[0205]

[0206] While the zone power spectrum is computed directly from the object signal, the loudspeaker zone power spectrum is computed as a weighted sum of the loudspeaker signal power spectra from previous time intervals:

[0207]

[0208] Weighted P jl This may be referred to herein as a "speaker zone participation value", which means a measure of the portion of the energy of a zone signal, , that will be rendered to speaker j. jl can be obtained by participating in the values ​​across a set of original speaker zones Normalize to get:

[0209]

[0210] This can be done by using the mapping function M j To simulate each area, a set of N l Spatial location , thereby calculating the original speaker zone participation value The N generated by this simulated rendering l The power spectra of the maps can be summed together to compute the original zone speaker participation, for example, as follows:

[0211]

[0212] In some examples, the zone power spectrum Z defined in Equation 7 may be l (f, t) to calculate the workload signal associated with the simulated rendering

[0213]

[0214] As a summary, the above describes the method for constructing the object level signal L i Three different approaches to (f, t): 1) a region-based approach; 2) a feedback-based approach; and 3) a hybrid approach that combines elements of the region-based approach and the feedback-based approach.

[0215] In all the methods described above for constructing the object level signal, further processing may be applied to reduce perceptual artifacts. For example, the object level signal L may be processed in time, frequency, or both. i (f, t) is smoothed in order to regularize the variation in the resulting energy spread across these dimensions.

[0216] Define the workload signal E as shown in Equation 4 ij (f, t) allows the invention to be implemented very efficiently using a single data structure, such as a matrix consisting of the expected audio object positions and the audio object level signal L i The lookup table may be constructed by mapping audio objects to speakers indexed by (f, t). The lookup table may be constructed, for example, by mapping all possible expected object positions (or at least a reasonable number of expected object positions) to the same audio object. Sampling is performed and L is passed through the object level signal value across a meaningful range i (f, t) is constructed by sampling. As mentioned before, the expected object position May correspond to time-varying 3D (x, y, z) metadata of an audio object.

[0217] Fig. 9is a graph of points indicating the mapping of audio objects to speakers as a function of the x, y, and z coordinates of the audio objects in an example embodiment. In this example, 15 points are sampled for the x and y dimensions, and 5 points are sampled for the z dimension. Other implementations may include more samples or fewer samples. According to this example, each point represents a set of j=1...N for audio object i s Mapping of loudspeakers H ij , where the (x,y,z) position corresponds to the point.

[0218] At runtime, to determine the actual mapping for each speaker, in some examples trilinear interpolation between the speaker mappings of the nearest 8 points may be used. Fig.10 is a diagram of trilinear interpolation between points indicating speaker mapping according to an example. In this example, the process of continuous linear interpolation includes interpolating each pair of points in the top plane to determine a first interpolation point 1005a and a second interpolation point 1005b, interpolating each pair of points in the bottom plane to determine a third interpolation point 1010a and a fourth interpolation point 1010b, interpolating the first interpolation point 1005a and the second interpolation point 1005b to determine a fifth interpolation point 1015 in the top plane, interpolating the third interpolation point 1010a and the fourth interpolation point 1010b to determine a sixth interpolation point 1020 in the bottom plane, and interpolating the fifth interpolation point 1015 and the sixth interpolation point 1020 to determine a seventh interpolation point 1025 between the top plane and the bottom plane. Although trilinear interpolation is an effective interpolation method, those skilled in the art will appreciate that trilinear interpolation is only one possible interpolation method that can be used to implement various aspects of the present disclosure, and other examples may include other interpolation methods.

[0219] The above approach can be further extended by adding a fourth dimension to the lookup table to cover varying audio object signal levels. In some examples, this lookup along the fourth dimension of signal level can be performed independently across a set of frequency bands, thereby explicitly capturing the L i The exact properties of (f, t) vary in frequency.

[0220] Some alternative channel-based examples involve interpolating levels, such as linearly, without trilinear interpolation of positions (or in some examples, without any interpolation of positions).

[0221] In another alternative embodiment, the object level signal may be approximated as a broadband gain multiplied by a prototype spectral shape, for example, as follows:

[0222]

[0223] In Equation 15, the prototype spectral shape L p(f) is chosen to represent the average spectral shape associated with the content expected to be processed by the system. i (t) Calculate to minimize the estimated object level signal L calculated by one of the above methods i (f, t) and its approximate value A lookup table can be constructed assuming that the target signal level spectrum follows the form of Equation 15, thereby allowing a single broadband gain g for all frequency bands to be i (t) to index the lookup of the signal level. Therefore, this approximation can reduce the computational complexity by reducing the number of operations required to interpolate between points in the table.

[0224] Now returning to the equation 2b as the workload signal E ij The mapping H of the function of (f, t) ij During the optimization of the mapping for each object i, it may be necessary to map the workload signal to a per-speaker penalty P j (f). Considering the construction of the workload signal in Equation 4, we note that when the audio object level signal is less than the speaker playback limit threshold, the workload signal is less than zero, and when the audio object signal is greater than the threshold, the workload signal is greater than zero. When mapping these workload signals to speaker penalties, it may be useful to apply a transformation so that the penalty increases smoothly and monotonically from a zero value as the workload signal increases above a certain specified transition point. Designating this transition point as corresponding to a workload value less than zero means that the penalty will begin to activate before the signal level has reached the playback threshold of the loudspeaker. Such examples have the advantageous effect of gradually diverting energy away from the loudspeaker before the signal level reaches the playback limit. According to some such examples, an inflection point parameter K can be used to control the transition point and the rate at which the penalty increases as the workload signal increases.

[0225] Fig.11 Examples of penalties for various inflection point parameters are shown. Fig.11 Depicted is how the loudspeaker penalty increases for knee values ​​ranging from -24dB to -6dB. It can be observed that below the knee value, the penalty is zero (this value has no effect on the computational map). According to these examples, the penalty increases monotonically above the knee value, meaning that as the workload signal increases, energy will be gradually transferred from the associated loudspeaker. Additionally, we see that larger knee values ​​correspond to a more gradual activation of the loudspeaker penalty. In practice, the inventors have determined that knee values ​​in the range of -18dB to -12dB work well.

[0226] The knee value K is an example of a parameter that can be used to control the degree of energy transfer between loudspeakers as a function of signal level. In some embodiments, the degree of such energy transfer can be fixed for all content processed by the system. In other embodiments, the degree of such transfer can be additionally controlled based on the input audio format, codec, or metadata. For example, some codecs can incorporate metadata that indicates that certain components of the mix are more eligible for more energy transfer than other components. This metadata or the basis for the metadata can be controlled, for example, by the content creator, which in Dolby Atmos TM Different inflection point values ​​are specified for different audio objects in the mix. Additionally, in some examples, the degree of energy transfer can depend at least in part on the content format. For example, the degree of energy transfer for the center channel of a Dolby 5.1 mix can be set to be less than the degree of energy transfer for other channels in order to keep the perceived dialog location in the center.

[0227] Fig.12 1200 is a flowchart outlining one example of a method that may be performed by an apparatus or system, such as an apparatus or system disclosed herein. As with other methods described herein, the blocks of method 1200 do not have to be performed in the order indicated. In some embodiments, one or more blocks of method 1200 may be performed simultaneously. In addition, some embodiments of method 1200 may include more or fewer blocks than shown and / or described. The blocks of method 1200 may be performed by one or more devices, which may be (or may include) a control system, such as in Figure 1 The control system 110 shown in and described above, or one of the other disclosed control system examples.

[0228] According to this example, block 1205 involves receiving audio data by the control system and via the interface system. In this example, the audio data includes one or more audio signals and associated spatial data. Here, the spatial data indicates an expected perceived spatial location corresponding to the audio signal. In some examples, the spatial data may be or may include an object-based audio format such as Dolby Atmos. TM ). In some examples, the expected perceived spatial location can be expressed as As disclosed herein. In some examples, the spatial data may be or may correspond to channels of a channel-based audio format, such as Dolby 5.1, Dolby 5.1.2, Dolby 7.1, Dolby 7.1.4, or Dolby 9.1 formats. Thus, the expected perceived spatial position may correspond to a channel of a channel-based audio format, may correspond to metadata, or may correspond to both a channel and metadata.

[0229] In this example, block 1210 involves rendering, by a control system, audio data for reproduction via a set of two or more loudspeakers of an environment, thereby generating loudspeaker signals. According to this example, rendering each of the one or more audio signals included in the audio data involves a time-varying and frequency-varying mapping of each audio signal to a loudspeaker signal. In this example, the mapping of each audio signal is calculated as a function of an expected perceived spatial position of the audio signal, a physical position associated with the loudspeaker, and a time-varying and frequency-varying representation of the loudspeaker signal level relative to a maximum playback limit of each loudspeaker. According to this example, each mapping is calculated to approximately achieve the expected perceived spatial position of the associated audio signal when the loudspeaker signal is played back on two or more corresponding loudspeakers located at the associated loudspeaker positions.

[0230] According to some examples, "approximately achieving" an expected perceptual spatial position of an associated audio signal, given available loudspeakers and associated loudspeaker positions, can involve minimizing a difference between the perceptual spatial position and the expected perceptual spatial position. In some examples, approximately achieving the expected perceptual spatial position of an associated audio signal can involve minimizing a cost function, such as one of the cost functions disclosed herein. Equation 2b of the present disclosure encompasses many different possibilities, depending on the choice of cost term. For example, the cost term can be calculated based on the additional cost term and to implement various implementations.

[0231] In this example, block 1210 involves, for each audio signal, calculating a representation of a loudspeaker signal level relative to a maximum playback limit as a function of one or more audio signals in the audio signal and their perceived spatial locations. According to this example, block 1210 involves decreasing the mapping of the audio signal to a particular loudspeaker signal as the representation of the loudspeaker signal level relative to the maximum playback limit increases above a threshold. Additionally, in this example, block 1210 involves increasing the mapping to one or more other loudspeakers for a signal level relative to the representation of the maximum playback limit of one or more other loudspeakers being less than a threshold.

[0232] According to some examples, the mapping may be calculated over the entire audible frequency range for a normal person. However, in some examples, the mapping may be calculated over a subset of the audible frequency range. According to some examples, the mapping may involve minimizing a cost function including a first term and a second term, the first term modeling the closeness of implementing the expected perceptual spatial position as a mapping function of the audio signal to the loudspeaker signal, and the second term assigning a cost to each loudspeaker in the activated loudspeakers. Equation 2b provides an example of both the first term and the second term. In some examples, the cost of activating each loudspeaker may be based at least in part on a representation of the loudspeaker signal level relative to a function of a maximum playback limit.

[0233] In some examples, the representation of the loudspeaker signal level relative to the maximum playback limit may correspond to one or more of a digital signal level, a limiter gain, or an acoustic signal level. According to some examples, the representation of the loudspeaker signal level relative to the maximum playback limit may be calculated as the difference between the level estimate of each audio signal and the playback limit threshold of each loudspeaker. In some such examples, the level estimate of each audio signal may be based at least in part on the zone-based rendering of all audio signals. In some examples, the level estimate of each audio signal may be based at least in part on a previously calculated loudspeaker signal. In some such examples, the level estimate of each audio signal may further depend on the participation of each loudspeaker in a plurality of spatial zones. According to some examples, block 1210 - or another block of method 1200 - may involve smoothing the level estimate of each audio signal in time, in frequency, or in both time and frequency.

[0234] According to some examples, a mapping of audio signals to loudspeaker signals may be determined by querying a data structure indexed by an expected perceptual spatial position and a level estimate for each audio signal. In some examples, the mapping of audio signals to loudspeaker signals may involve determining loudspeaker activations. In some such examples, loudspeaker activations may be determined by interpolating from a precomputed set of loudspeaker activations. According to some such examples, the set may be indexed by an expected perceptual spatial position and a level estimate for each audio signal.

[0235] In some examples, the level estimate of each audio signal can be expressed as a broadband gain multiplied by a spectral shape. In some such examples, the spectral shape can be selected from a plurality of spectral shapes. Each spectral shape in the plurality of spectral shapes can, for example, correspond to a content type. The content type can, for example, include movie content, television program content, podcast content, music performance content, game content, etc.

[0236] According to some examples, as the signal level relative to the representation of the maximum playback level increases above a threshold, mapping to one loudspeaker may be gradually reduced and mapping to another loudspeaker may be gradually increased. Fig.11 and corresponding descriptions provide some examples. In some examples, method 1200 may involve controlling a reduction in mapping to one loudspeaker and an increase in mapping to another loudspeaker based on one or more of an audio format, a codec, or metadata. According to some examples, method 1200 may involve controlling a reduction in mapping to one loudspeaker and an increase in mapping to another loudspeaker based on an inflection point parameter.

[0237] In this embodiment, block 1215 involves providing a microphone signal to at least two microphones of the set of microphones of the environment via the interface system.

[0238] Some disclosed embodiments include a system or device configured (e.g., programmed) to perform any embodiment of the disclosed method, and a tangible computer-readable medium (e.g., disk) storing code for implementing any embodiment of the disclosed method or its steps. For example, the disclosed system may be or include a programmable general-purpose processor, digital signal processor, or microprocessor that is programmed and / or otherwise configured with software or firmware to perform any of a variety of operations on data, including embodiments of the disclosed method or its steps. Such a general-purpose processor may be or include a computer system that includes an input device, a memory, and a processing subsystem that is programmed (and / or otherwise configured) to perform embodiments of the disclosed method (or its steps) in response to data asserted thereto.

[0239] Some embodiments of the disclosed system are implemented as a configurable (e.g., programmable) digital signal processor (DSP) that is configured (e.g., programmed and otherwise configured) to perform required processing on (multiple) audio signals, including the execution of embodiments of the disclosed method. Alternatively, embodiments of the disclosed system (or its elements) are implemented as a general-purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor, which may include an input device and a memory), which is programmed and / or otherwise configured with software or firmware to perform any of a variety of operations, including embodiments of the disclosed method. Alternatively, elements of some embodiments of the disclosed system are implemented as a general-purpose processor or DSP that is configured (e.g., programmed) to perform embodiments of the disclosed method, and the system also includes other elements (e.g., one or more loudspeakers and / or one or more microphones). A general-purpose processor configured to perform embodiments of the disclosed method will typically be coupled to an input device (e.g., a mouse and / or keyboard), a memory, and a display device.

[0240] Another aspect of the present disclosure is a computer-readable medium (eg, a disk or other tangible storage medium) storing code for performing any disclosed method or steps thereof (eg, an encoder executable to perform any disclosed method or steps thereof).

[0241] Although specific embodiments and applications have been described herein, it will be apparent to those of ordinary skill in the art that many changes may be made to the embodiments and applications described herein without departing from the scope described and claimed herein. It should be understood that although certain forms have been shown and described, the scope of the present disclosure is not limited to the specific embodiments described and shown or the specific methods described.

Claims

1. An audio processing method, comprising: receiving, by the control system and via the interface system, audio data comprising one or more audio signals and associated spatial data indicating intended perceived spatial locations corresponding to the audio signals; The audio data is rendered by the control system to produce a loudspeaker signal, the rendering being for reproduction via a set of two or more loudspeakers of an environment, wherein: Rendering each of the one or more audio signals included in the audio data involves a mapping of each audio signal to the loudspeaker signal, the mapping being a time-variant and frequency-variant mapping; The mapping of each audio signal is calculated as a function of an intended perceived spatial location of the audio signal, a physical location associated with the loudspeakers, and a representation of the loudspeaker signal level relative to a maximum playback limit of each loudspeaker, wherein the representation is time-varying and frequency-varying; Each mapping is calculated such that when the loudspeaker signal is played back on the two or more corresponding loudspeakers located at the associated loudspeaker positions, the intended perceived spatial position of the associated audio signal is approximately achieved; for each audio signal, a representation of the loudspeaker signal level relative to a maximum playback limit is calculated as a function of one or more of the audio signals and their perceived spatial locations; and decreasing said mapping of the audio signal to a particular loudspeaker signal as said representation of a loudspeaker signal level relative to a maximum playback limit increases above a threshold, while increasing said mapping to one or more other loudspeakers for a signal level relative to said representation of a maximum playback limit for said one or more other loudspeakers being less than a threshold; as well as The loudspeaker signal is provided to at least two loudspeakers of the set of loudspeakers of the environment via the interface system.

2. The audio processing method according to claim 1, wherein: The mapping is calculated over the entire audible frequency range.

3. The audio processing method according to claim 1, wherein: The mapping is computed over a subset of the audible frequency range.

4. The method according to any one of claims 1 to 3, wherein: The mapping involves minimizing a cost function comprising a first term that models how closely the desired perceptual spatial position is realized as a mapping function of audio signals to loudspeaker signals, and a second term that assigns a cost to activating each of the loudspeakers.

5. The method of claim 4, wherein: The cost of activating each loudspeaker is based at least in part on a function of loudspeaker signal level relative to the representation of the maximum playback limit.

6. The method according to any one of claims 1 to 5, wherein: The representation of loudspeaker signal level relative to the maximum playback limit corresponds to one or more of a digital signal level, a limiter gain, or an acoustic signal level.

7. The method according to any one of claims 1 to 5, wherein: The representation of loudspeaker signal levels relative to the maximum playback limit is calculated as the difference between the level estimate of each audio signal and a playback limit threshold for each loudspeaker.

8. The method of claim 7, wherein: The level estimate of each audio signal is based at least in part on a region-based rendering of all of the audio signals.

9. The method of claim 7 or claim 8, wherein: The level estimate for each audio signal is at least partly based on a previously calculated loudspeaker signal.

10. The method of claim 9, wherein: The level estimate of each audio signal further depends on the participation of each loudspeaker in the plurality of spatial zones.

11. A method as claimed in any one of claims 7 to 10, further comprising smoothing the level estimate of each audio signal across time, across frequency or across both time and frequency.

12. The method according to any one of claims 7 to 11, wherein: The mapping from audio signals to loudspeaker signals is determined by querying a data structure indexed by the expected perceptual spatial position and the level estimate for each audio signal.

13. The method according to any one of claims 7 to 11, wherein: The mapping from audio signals to loudspeaker signals is determined by interpolation from a set of pre-computed loudspeaker mappings, the set being indexed by the expected perceptual spatial position and the level estimate for each audio signal.

14. The method according to any one of claims 7 to 11, wherein: The mapping of audio signals to loudspeakers is determined by interpolation from a set of pre-computed loudspeaker mappings, the set being indexed by the expected level estimate for each audio signal.

15. The method of claim 12 or claim 13, wherein: The level estimate for each audio signal is expressed as a broadband gain multiplied by a spectral shape.

16. The method of claim 15, wherein: The spectral shape is selected from a plurality of spectral shapes, each spectral shape of the plurality of spectral shapes corresponding to a content type.

17. The method according to any one of claims 1 to 16, wherein: As the signal level relative to the representation of the maximum playback level increases above a threshold, mapping to one loudspeaker is gradually reduced and mapping to the other loudspeaker is gradually increased.

18. The method of any one of claims 1 to 17, further comprising controlling the extent of the reduction of mapping to one loudspeaker and the increase of mapping to another loudspeaker based on one or more of an audio format, a codec, or metadata.

19. The method of any one of claims 1 to 18, further comprising controlling the extent of the reduction in mapping to one loudspeaker and the extent of the increase in mapping to another loudspeaker in accordance with the inflection point parameter.

20. The method of any one of claims 1 to 19, wherein: The expected perceptual spatial location corresponds to a channel of a channel-based audio format, to metadata, or to both the channel and the metadata.

21. The method according to any one of claims 1 to 20, wherein: Given available loudspeakers and associated loudspeaker positions, substantially achieving the intended perceptual spatial position of the associated audio signal involves minimizing a difference between the perceptual spatial position and the intended perceptual spatial position.

22. The method of any one of claims 1 to 21, wherein: Approximately achieving the desired perceptual spatial position of the associated audio signal involves minimizing a cost function.

23. An apparatus configured to perform the method according to any one of claims 1 to 22.

24. A system configured to perform the method of any one of claims 1 to 22.

25. One or more non-transitory media having instructions stored thereon, the instructions being used to control one or more devices to perform the method of any one of claims 1 to 22.

Images