Rendering based on loudspeaker orientation

JP2024542069APending Publication Date: 2024-11-13DOLBY LABORATORIES LICENSING CORP
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2024526478
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2022-05-10
Filing Date
2022-11-07
Publication Date
2024-11-13

AI Technical Summary

Benefits of technology

【0023】 本明細書に記載された主題の一つまたは複数の実装の詳細が、添付の図面および以下の説明に記載される。他の特徴、側面、および利点は、本稿、図面、および特許請求の範囲から明白になるであろう。以下の図の相対的な寸法は、一定の縮尺で描かれていない場合があることに留意されたい。

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024542069000001_ABST
    Figure 2024542069000001_ABST
Patent Text Reader

Abstract

An audio processing method involves receiving an audio signal and associated spatial data, listener position data, loudspeaker position data, and loudspeaker orientation data, and rendering the audio data for playback based at least in part on the spatial data, the listener position data, the loudspeaker position data, and the loudspeaker orientation data to generate a rendered audio signal. The rendering involves applying a loudspeaker orientation factor that tends to reduce relative activation of the loudspeakers based at least in part on the increased loudspeaker orientation angle. In some examples, the rendering may involve modifying an effect of the loudspeaker orientation factor based at least in part on a loudspeaker importance metric. The loudspeaker importance metric may correspond to the importance of the loudspeakers for rendering the audio signal at the intended perceived spatial location of the audio signal.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 277,225, filed November 9, 2021, U.S. Provisional Application No. 63 / 364,322, filed May 6, 2022, and European Patent Application No. 22172447.9, filed May 10, 2022, each of which is incorporated by reference in its entirety herein. Technical Field The present disclosure relates to devices, systems, and methods for rendering audio data for playback on an audio device. [Background technology]

[0002] Audio devices, including but not limited to smart audio devices, are being widely deployed and are becoming a common feature of many homes. While existing systems and methods for controlling audio devices provide benefits, improved systems and methods are desirable.

[0003] Notation and Nomenclature Throughout this disclosure, including the claims, the terms "speaker," "loudspeaker," and "audio reproduction transducer" are used interchangeably to refer to any sound emitting transducer (or set of transducers). A typical headphone set includes two speakers. A speaker may be implemented to include multiple transducers (e.g., woofers and tweeters) that may be driven by a single common speaker feed or multiple speaker feeds. In some examples, the speaker feeds may receive different processing in different circuit branches coupled to different transducers.

[0004] Throughout this disclosure, including the claims, the phrase performing an operation "on" a signal or data (e.g., filtering, scaling, transforming, or applying a gain to a signal or data) is used broadly to refer to performing an operation on the signal or data directly, or on a processed version of the signal or data (e.g., on a version of the signal that has been subjected to pre-filtering or pre-processing before performing the operation).

[0005] Throughout this disclosure, including the claims, the term "system" is used broadly to denote a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, where the subsystem generates M of the inputs and the other XM inputs are received from external sources) may also be referred to as a decoder system.

[0006] Throughout this disclosure, including the claims, the term "processor" is used broadly to denote a system or device that is programmable or configurable (e.g., with software or firmware) to perform operations on data (e.g., audio or video or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors programmed and / or otherwise configured to perform pipeline processing on audio or other sound data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets.

[0007] Throughout this disclosure, including the claims, the terms "couple" or "coupled" are used to mean either a direct connection or an indirect connection. Thus, if a first device couples to a second device, the connection may be through a direct connection or through an indirect connection via other devices and connections.

[0008] As used herein, a "smart device" is generally an electronic device that is configured to communicate with one or more other devices (or networks) via various wireless protocols, such as Bluetooth, Zigbee, near field communications, Wi-Fi, light fidelity (Li-Fi), 3G, 4G, 5G, etc., and can operate interactively and / or autonomously to some degree. Some notable types of smart devices are smartphones, smart cars, smart thermostats, smart doorbells, smart locks, smart refrigerators, phablets and tablets, smart watches, smart bands, smart keychains, and smart audio devices. The term "smart device" may also refer to devices that exhibit some characteristics of ubiquitous computing, such as artificial intelligence.

[0009] The term "smart audio device" is used herein to denote a smart device that is either a single-purpose audio device or a multi-purpose audio device (e.g., an audio device that implements at least some aspects of a virtual assistant function). A single-purpose audio device is a device (e.g., a television (TV)) that includes or is coupled to at least one microphone (and, optionally, includes or is coupled to at least one speaker and / or at least one camera) and is designed primarily or primarily to achieve a single purpose. For example, a TV can typically play (and is considered capable of playing) audio from program material, but in most cases, modern TVs run some operating system on which applications run locally, including applications for watching television. In this sense, a single-purpose audio device having a speaker and microphone is often configured to run local applications and / or services to directly use the speaker and microphone. Several single-purpose audio devices may be configured to be grouped together to achieve playback of audio over a zone or user-configured area.

[0010] One common type of multipurpose audio device is an audio device that implements at least some aspects of the virtual assistant functionality, although other aspects of the virtual assistant functionality may be implemented by one or more other devices, such as one or more servers with which the multipurpose audio device is configured to communicate. Such multipurpose audio devices may be referred to herein as "virtual assistants." A virtual assistant is a device (e.g., a smart speaker or a voice assistant integrated device) that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker and / or at least one camera). In some examples, a virtual assistant may provide the ability to utilize multiple devices (different from the virtual assistant) for applications that are in some sense cloud-enabled or otherwise not fully implemented in or on the virtual assistant itself. In other words, at least some aspects of the virtual assistant functionality, e.g., speech recognition functionality, may be implemented (at least in part) by one or more servers or other devices with which the virtual assistant can communicate over a network such as the Internet. Virtual assistants may sometimes operate together, e.g., in a discrete, conditionally defined manner. For example, two or more virtual assistants can cooperate in the sense that one of them, e.g., the one that is most confident that it heard the wake word, will respond to the wake word. Connected virtual assistants can, in some implementations, form a kind of constellation, which may be managed by one main application that may be (or may implement) a virtual assistant.

[0011] Here, "wake word" is used broadly to mean any sound (e.g., a word spoken by a human being, or any other sound) that the smart audio device is configured to wake up in response to detecting ("listening to") that sound (using at least one microphone included in or coupled to the smart audio device, or at least one other microphone). In this context, "awakening" refers to the device entering a state in which it waits for (i.e., listens for) a voice command. In some instances, what may be referred to herein as a "wake word" may include multiple words, e.g., a phrase.

[0012] Here, the term "wake word detector" refers to a device (or software including instructions for configuring a device) configured to continuously look for alignment between real-time audio (e.g., speech) features and a trained model. Typically, a wake word event is triggered whenever the wake word detector determines that the probability that the wake word has been detected exceeds a predetermined threshold. For example, the threshold may be a predetermined threshold tuned to provide a reasonable compromise between false accept and false reject rates. Following the wake word event, the device may enter a state (which may be referred to as an "awake" or "attentive" state) in which it listens for commands and passes received commands to a larger, more computationally intensive recognizer.

[0013] As used herein, the terms "program stream" and "content stream" refer to a collection of one or more audio signals, and possibly video signals, that are intended to be listened to at least in part together. Examples include selections of music, movie soundtracks, movies, television programs, audio portions of television programs, podcasts, live voice calls, synthesized voice responses from a smart assistant, etc. In some instances, a content stream may contain multiple versions of at least a portion of an audio signal, e.g., the same dialogue in multiple languages. In such instances, only one version of the audio data or portion thereof is intended to be played at a time (e.g., the version corresponding to a single language). Summary of the Invention [Means for solving the problem]

[0014] At least some aspects of the present disclosure may be implemented via one or more audio processing methods. In some cases, the method(s) may be implemented, at least in part, by a control system and / or via instructions (e.g., software) stored on one or more non-transitory media. Some such methods may involve receiving audio data by the control system and via an interface system, the audio data including one or more audio signals and associated spatial data. The spatial data may indicate an intended perceived spatial location corresponding to one audio signal of the one or more audio signals. The intended perceived spatial location may correspond, for example, to a channel of a channel-based audio format. Alternatively or additionally, the intended perceived spatial location corresponds to position metadata, for example, position metadata of an object-based audio format.

[0015] In some examples, the method may involve receiving, by the control system, via the interface system, listener position data indicative of a listener position corresponding to a person in the audio environment. According to some examples, the method may involve receiving, by the control system, via the interface system, loudspeaker position data indicative of a position of each loudspeaker of a plurality of loudspeakers in the audio environment. In some examples, the method may involve receiving, by the control system and via the interface system, loudspeaker orientation data indicative of a position of each loudspeaker of a plurality of loudspeakers in the audio environment. In some examples, the method may involve receiving, by the control system and via the interface system, loudspeaker orientation data. In some such examples, the loudspeaker orientation data may indicate a loudspeaker orientation angle between (a) a direction of maximum sound radiation for each loudspeaker of a plurality of loudspeakers in the audio environment and (b) a listener position. In some such examples, the listener position may be relative to a position of the corresponding loudspeaker. According to some examples, the loudspeaker orientation angle for a particular loudspeaker may be an angle between (a) a direction of maximum sound radiation for the particular loudspeaker and (b) a line between the position of the particular loudspeaker and the listener position.

[0016] According to some examples, the method may involve rendering, by a control system, the audio data for playback through at least a subset of a plurality of loudspeakers in an audio environment to generate a rendered audio signal. In some examples, the rendering may be based at least in part on spatial data, listener position data, loudspeaker position data, and loudspeaker orientation data. In some examples, the rendering may involve applying a loudspeaker orientation factor that tends to reduce relative activation of the loudspeakers based at least in part on an increased loudspeaker orientation angle.

[0017] In some examples, the method may involve providing the rendered audio signal to at least the subset of loudspeakers of a plurality of loudspeakers in the audio environment via an interface system.

[0018] According to some examples, the method may involve estimating a loudspeaker importance metric for at least the subset of loudspeakers. For example, the method may include estimating a loudspeaker importance metric for each loudspeaker of the subset of loudspeakers. In some examples, the loudspeaker importance metric may correspond to an importance of the loudspeaker for rendering the audio signal at an intended perceived spatial location of the audio signal. According to some examples, the rendering for each loudspeaker may be based at least in part on the loudspeaker importance metric. In some examples, the rendering for each loudspeaker may involve modifying an effect of a loudspeaker orientation factor based at least in part on the loudspeaker importance metric. According to some examples, the rendering for each loudspeaker may involve reducing an effect of a loudspeaker orientation factor based at least in part on the increased loudspeaker importance metric.

[0019] In some examples, the method may involve determining whether the loudspeaker orientation angle is equal to or exceeds a threshold loudspeaker orientation angle. According to some examples, the audio processing method may involve applying the loudspeaker orientation factor only if the loudspeaker orientation angle is equal to or exceeds the threshold loudspeaker orientation angle. In some examples, the loudspeaker importance metric may be based at least in part on a distance between the qualified loudspeaker and a line between (a) a first loudspeaker having the shortest clockwise angular distance from the qualified loudspeaker and (b) a second loudspeaker having the shortest counterclockwise angular distance from the qualified loudspeaker. In some such examples, the qualified loudspeaker may be a loudspeaker having a loudspeaker orientation angle equal to or exceeds the threshold loudspeaker orientation angle. In some instances, the first loudspeaker and the second loudspeaker may be disqualified loudspeakers having loudspeaker orientation angles less than the threshold loudspeaker orientation angle.

[0020] According to some examples, rendering may involve determining a relative activation for at least the subset of loudspeakers by optimizing a cost that is a function of: a model of a perceived spatial location of an audio signal of the one or more audio signals when played on the subset of loudspeakers in an audio environment; a measure of proximity of an intended perceived spatial location of the audio signal to a location of each loudspeaker of the subset of loudspeakers; and one or more additional dynamically configurable functions. In some such examples, at least one of the one or more additional dynamically configurable functions may be based at least in part on a loudspeaker orientation factor. According to some such examples, at least one of the one or more additional dynamically configurable functions may be based at least in part on a loudspeaker importance metric. In some such examples, at least one of the one or more additional dynamically configurable functions may be based at least in part on measurements or estimates of acoustic transmission from each loudspeaker in the audio environment to other loudspeakers in the audio environment.

[0021] Aspects of some disclosed implementations include a control system configured (e.g., programmed) to perform one or more of the disclosed methods or steps thereof, as well as non-transitory storage of data (e.g., disk or other tangible storage medium) storing code (e.g., executable code to perform) for performing one or more of the disclosed methods or steps thereof. For example, some disclosed embodiments may be or include a programmable general-purpose processor, digital signal processor, or microprocessor programmed with software or firmware and / or otherwise configured to perform any of a variety of operations on data, including one or more of the disclosed methods or steps thereof. Such a general-purpose processor may be or include a computer system including an input device, a memory, and a processing subsystem programmed (and / or otherwise configured) to perform one or more of the disclosed methods (or steps thereof) in response to asserted data.

[0022] Some or all of the operations, functions, and / or methods described herein may be executed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Thus, some innovative aspects of the subject matter described in this disclosure may be implemented in non-transitory media that store software.

[0023] Details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. It should be noted that the relative dimensions of the following figures may not be drawn to scale. [Brief description of the drawings]

[0024] [Figure 1] FIG. 1 is a block diagram illustrating example components of an apparatus capable of implementing various aspects of the disclosure. [Diagram 2] 1 shows an example of an audio environment. [Diagram 3] 3 shows another example of an audio environment. [Figure 4] 1 shows an example of loudspeakers positioned on the circumference of a unit circle. [Diagram 5] FIG. 5 shows a loudspeaker arrangement with strings connecting the loudspeaker locations. [Figure 6] 6 shows the loudspeaker arrangement of FIG. 5 with one string omitted. [Figure 7] 4 shows alternative examples of loudspeakers positioned on the circumference of a unit circle. [Figure 8] 8 and 9 show alternative examples of loudspeakers positioned on the circumference of a unit circle. [Figure 9] 8 and 9 show alternative examples of loudspeakers positioned on the circumference of a unit circle. [Figure 10] Formula 6 of the present disclosure is shown, with the elements of the formula identified. [Figure 11] Formula 7 of the present disclosure is shown, with the elements of the formula identified. [Figure 12] 6A and 6B are graphs corresponding to equation 6 of the present disclosure. [Figure 13A] 7 is a graph corresponding to Equation 7 of the present disclosure. [Figure 13B] 7 is a graph corresponding to Equation 7 of the present disclosure. [Figure 13C] 13 is a graph showing an example of a penalty function based on loudspeaker orientation and importance metric. [Figure 14] FIG. 1 is a flow diagram outlining an example of the disclosed method. [Figure 15] FIG. 1 illustrates an exemplary set of speaker activations. [Figure 16]FIG. 1 illustrates an exemplary set of object rendering positions. [Figure 17] 2 is a flow diagram outlining an example of a method that may be performed by an apparatus or system such as that shown in FIG. 1. [Figure 18] 1 is a graph of speaker activation in an exemplary embodiment. [Figure 19] 4 is a graph of object rendering positions in an exemplary embodiment. [Figure 20] 1 is a graph of speaker activation in an exemplary embodiment. [Figure 21] 4 is a graph of object rendering positions in an exemplary embodiment. [Figure 22] 1 is a graph of speaker activation in an exemplary embodiment. [Diagram 23] 4 is a graph of object rendering positions in an exemplary embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] Spatial audio playback in consumer environments is typically tied to a given number of loudspeakers located in given locations. Some examples include Dolby 5.1 and Dolby 7.1 surround sound. More recently, immersive object-based spatial audio formats such as Dolby Atmos™ have been introduced, which break this association between audio content and specific loudspeaker locations. Instead, content may be described as a collection of individual audio objects, each of which may have associated time-varying metadata, such as positional metadata to describe the desired perceived location of said audio object in three-dimensional space. At playback, content is converted to loudspeaker feeds by a renderer that matches the number and locations of the loudspeakers in the playback system. However, many such renderers still constrain the position of a set of loudspeakers to be one of a set of prescribed layouts (e.g., Dolby 3.1.2, Dolby 5.1.2, Dolby 7.1.4, Dolby 9.1.6, etc. with Dolby Atmos).

[0026] "Flexible rendering" methods have been recently developed that allow object-based audio and legacy channel-based audio to be flexibly rendered across any number of loudspeakers located in any position. These methods generally require the renderer to have knowledge of the number and physical locations of the loudspeakers in the listening space. For such systems to be practical for the average consumer, an automated method for localizing the loudspeakers is desirable. Thus, methods have been recently developed for automatically localizing the positions of loudspeakers within a listening space, which may also be referred to herein as the "audio environment." Detailed examples of flexible rendering and automatic audio device localization are provided herein.

[0027] Concurrent with the introduction of object-based spatial audio in consumer spaces has been the rapid adoption of so-called “smart speakers” such as the Amazon Echo™ line. While the tremendous popularity of these devices can be attributed to the simplicity and convenience offered by their wireless connectivity and integrated voice interface (e.g., Amazon’s Alexa™), the acoustic capabilities of these devices are generally limited, especially when it comes to spatial audio. In most cases, these devices are constrained to mono or stereo playback. However, combining the aforementioned flexible rendering and automatic localization techniques with multiple coordinated smart speakers can result in a system with very sophisticated spatial playback capabilities, yet remains extremely simple for consumers to set up. Consumers can place as many or as few speakers as desired wherever convenient, without the need to run speaker wires because of the wireless connectivity, and can use the built-in microphones to automatically localize the speakers for their associated flexible renderers.

[0028] The flexible rendering methods described above take into account the location of the loudspeakers relative to the listening position or area, but not the orientation of the loudspeakers relative to the listening position or area. In general, these methods model the speakers as radiating directly towards the listening position, which in reality may not be the case. The further the loudspeaker orientation points away from the intended listening position, the more some acoustic properties may change, two being the most prominent. First, the overall equalization heard at the listening position may change, usually with a drop in high frequencies due to most loudspeakers exhibiting a higher degree of directionality at higher frequencies. Second, the ratio of direct to reflected sound at the listening position may decrease, as more acoustic energy is directed away from the listening position and interacts with the room before it is finally heard.

[0029] In light of the potential effects of loudspeaker orientation, some disclosed implementations may involve one or more of the following: For any given position of the loudspeaker, activation of the loudspeaker may be reduced as the orientation of the loudspeaker increases away from the listening position; The degree of said reduction may be reduced as a function of a measure of the importance of that loudspeaker for rendering a given audio signal at its desired perceived spatial location.

[0030] A detailed example is given below.

[0031] FIG. 1 is a block diagram illustrating examples of components of a device capable of implementing various aspects of the disclosure. As with other figures provided herein, the types and numbers of elements shown in FIG. 1 are provided merely as examples. Other implementations may include more, fewer, and / or different types and numbers of elements. According to some examples, device 150 may be configured to perform at least some of the methods disclosed herein. In some implementations, device 150 may be or include one or more components of an audio system. For example, device 150 may be an audio device, such as a smart audio device, in some implementations. In other examples, device 150 may be a mobile device (such as a mobile phone), a laptop computer, a tablet device, a television, a vehicle or component thereof, or another type of device.

[0032] According to some alternative implementations, apparatus 150 may be or include a server. In some such examples, apparatus 150 may be or include an encoder. Thus, in some cases, apparatus 150 may be a device configured for use in an audio environment, and in other cases, apparatus 150 may be a device configured for use in the "cloud," e.g., a server.

[0033] In this example, device 150 includes an interface system 155 and a control system 160. Interface system 155, in some implementations, may be configured to communicate with one or more other devices of an audio environment. The audio environment, in some examples, may be a home audio environment. In other examples, the audio environment may be another type of environment, such as an office environment, an automobile environment, a train environment, a street or sidewalk environment, a park environment, etc. Interface system 155, in some implementations, may be configured to exchange control information and associated data with audio devices of the audio environment. The control information and associated data, in some examples, may relate to one or more software applications that device 150 is executing.

[0034] The interface system 155 may be configured to receive, provide, or both receive and provide a content stream in some implementations. The content stream may include audio data. The audio data may include, but is not limited to, an audio signal. In some instances, the audio data may include spatial data, such as channel data and / or spatial metadata. The metadata may be provided, for example, by what may be referred to herein as an “encoder.” In some instances, the content stream may include video data and audio data corresponding to the video data.

[0035] The interface system 155 may include one or more network interfaces and / or one or more external device interfaces (such as one or more universal serial bus (USB) interfaces). According to some implementations, the interface system 155 may include one or more wireless interfaces. The interface system 155 may include one or more devices for implementing a user interface, such as one or more microphones, one or more loudspeakers, a display system, a touch sensor system, and / or a gesture sensor system. In some examples, the interface system 155 may include one or more interfaces between the control system 160 and a memory system, such as the optional memory system 165 shown in FIG. 1. However, the control system 160 may include a memory system in some cases. The interface system 155 may be configured to receive input from one or more microphones in the environment in some implementations.

[0036] The control system 160 may include, for example, a general-purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.

[0037] In some implementations, the control system 160 may reside in more than one device. For example, in some implementations, a portion of the control system 160 may reside in a device in one of the environments illustrated herein, and another portion of the control system 160 may reside in a device that is external to the environment, such as a server, a mobile device (e.g., a smartphone or tablet computer), etc. In other examples, a portion of the control system 160 may reside in a device in one of the environments illustrated herein, and another portion of the control system 160 may reside in one or more other devices of the environment. For example, the control system functionality may be distributed across multiple smart audio devices of the environment, or may be shared by a leader device (such as what may be referred to herein as a smart home hub) and one or more other devices of the environment. In other examples, a portion of the control system 160 may reside in a device implementing a cloud-based service, such as a server, and another portion of the control system 160 may reside in another device implementing said cloud-based service, such as another server, a memory device, etc. The interface system 155 may also reside in more than one device, in some examples.

[0038] In some implementations, the control system 160 may be configured to at least partially perform the methods disclosed herein. According to some examples, the control system 160 may be configured to receive audio data, listener position data, loudspeaker position data, and loudspeaker orientation data via the interface system 155. The audio data may include one or more audio signals and associated spatial data indicative of an intended perceived spatial location corresponding to the audio signals. The listener position data may indicate a listener position corresponding to a person in the audio environment. The loudspeaker position data may indicate a location of each loudspeaker of a plurality of loudspeakers in the audio environment. The loudspeaker orientation data may indicate a loudspeaker orientation angle between (a) a direction of maximum acoustic radiation for each loudspeaker of a plurality of loudspeakers in the audio environment and (b) a listener position relative to the corresponding loudspeaker.

[0039] In some such examples, the control system 160 may be configured to render the audio data for playback through at least a subset of a plurality of loudspeakers in the audio environment to generate a rendered audio signal. According to some such examples, the rendering may be based at least in part on the spatial data, the listener position data, the loudspeaker position data, and the loudspeaker orientation data. In some such examples, the rendering may involve applying a loudspeaker orientation factor that tends to reduce relative activation of the loudspeakers based at least in part on the increased loudspeaker orientation angle.

[0040] In some examples, control system 160 may be configured to estimate a loudspeaker importance metric for at least the subset of loudspeakers. The loudspeaker importance metric may correspond to the importance of the loudspeaker for rendering the audio signal at the intended perceived spatial location of the audio signal. In some such examples, the rendering for each loudspeaker may be based at least in part on the loudspeaker importance metric.

[0041] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored in one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, and the like. The one or more non-transitory media may reside, for example, in optional memory system 165 and / or in control system 160 shown in FIG. 1. Thus, various inventive aspects of the subject matter described in this disclosure may be implemented in one or more non-transitory media having software stored thereon. The software may include, for example, instructions for controlling at least one device to perform some or all of the methods disclosed herein. The software may be executable by one or more components of a control system, such as, for example, control system 160 of FIG. 1.

[0042] In some examples, the device 150 may include an optional microphone system 170 as shown in FIG. 1. The optional microphone system 170 may include one or more microphones. According to some examples, the optional microphone system 170 may include an array of microphones. In some examples, the control system 160 may be configured to determine direction of arrival (DOA) and / or time of arrival (TOA) information, for example, according to signals from the array of microphones. The array of microphones may be configured for receiver beamforming in some cases, for example, according to instructions from the control system 160. In some implementations, one or more of the microphones may be part of or associated with another device, such as a speaker of a speaker system, a smart audio device, etc. In some examples, the device 150 may not include the microphone system 170. However, in some such implementations, the device 150 may still be configured to receive microphone data for one or more microphones in the audio environment via the interface system 160. In some such implementations, a cloud-based implementation of device 150 may be configured to receive microphone data, or data corresponding to microphone data, from one or more microphones in the audio environment via interface system 160.

[0043] According to some implementations, device 150 may include an optional loudspeaker system 175 shown in FIG. 1. Optional loudspeaker system 175 may include one or more loudspeakers. A loudspeaker may also be referred to herein as a "speaker" or, more generally, an "audio reproduction transducer." In some examples (e.g., cloud-based implementations), device 150 may not include loudspeaker system 175.

[0044] In some implementations, the device 150 may include the optional sensor system 180 shown in FIG. 1. The optional sensor system 180 may include one or more touch sensors, gesture sensors, motion detectors, and the like. According to some implementations, the optional sensor system 180 may include one or more cameras. In some implementations, the camera may be a freestanding camera. In some examples, the one or more cameras of the optional sensor system 180 may be present in a smart audio device, which in some examples may be configured to at least partially implement a virtual assistant. In some such examples, the one or more cameras of the optional sensor system 180 may be present in a television, a mobile phone, or a smart speaker. In some examples, the device 150 may not include the sensor system 180. However, in some such implementations, the device 150 may still be configured to receive sensor data for one or more sensors in the audio environment via the interface system 160.

[0045] In some implementations, the device 150 may include an optional display system 185 as shown in FIG. 1. The optional display system 185 may include one or more displays, such as one or more light emitting diode (LED) displays. In some cases, the optional display system 185 may include one or more organic light emitting diode (OLED) displays. In some examples, the optional display system 185 may include one or more displays of a smart audio device. In other examples, the optional display system 185 may include a television display, a laptop display, a mobile device display, or another type of display. In some examples in which the device 150 includes a display system 185, the sensor system 180 may include a touch sensor system and / or a gesture sensor system in proximity to one or more displays of the display system 185. According to some such implementations, the control system 160 may be configured to control the display system 185 to present one or more graphical user interfaces (GUIs).

[0046] According to some such examples, device 150 may be or include a smart audio device. In some such implementations, device 150 may be or include a wake word detector. For example, device 150 may be or include a virtual assistant.

[0047] The previously implemented flexible rendering methods mentioned above take into account the position of the loudspeaker relative to the listening position or area, but not the orientation of the loudspeaker relative to the listening position or area. In general, these methods model the speaker as radiating directly towards the listening position, which in reality may not be the case. Most loudspeakers have an associated direction in which acoustic energy is maximally radiated, and ideally this direction is pointed towards the listening position or area. For a simple device with a single loudspeaker driver mounted in an enclosure, the side of the enclosure in which the loudspeaker is mounted is considered the "front" of the device, and ideally the device is oriented so that this front faces the listening position or area. More complex devices may include multiple individually addressable loudspeakers that point in different directions relative to the device. In such cases, the orientation of each individual loudspeaker relative to the listening position or area may be taken into account when the overall orientation of the device relative to the listening position or area is set. Additionally, a device may include speakers with non-zero elevation angles (e.g., pointed upward from the device), and the orientation of these speakers relative to the listening position may be considered in three dimensions rather than simply two dimensions.

[0048] Figure 2 illustrates an example audio environment. Figure 2 illustrates an example loudspeaker orientation relative to a listening position or area. Figure 2 represents an overhead view of the audio environment, with the listening position represented by the head of a listener 205. As with other figures provided herein, the types, numbers, and arrangements of elements shown in Figure 2 are provided merely as examples. Other implementations may include more, fewer, and / or different types and numbers of elements, different arrangements of elements, etc.

[0049] According to this example, audio environment 200 includes audio devices 210A, 210B, and 210C. Audio devices 210A-210C may, in some examples, be instances of apparatus 150 of FIG. 1. In this example, audio device 210A includes a single loudspeaker L1, audio device 210B includes a single loudspeaker L2, and audio device 210C includes three individual loudspeakers L3, L4, and L5. Arrows emanating from each speaker represent the direction of maximum acoustic radiation associated with each. For audio devices 210A and 210B, each of which includes a single loudspeaker, these arrows may be considered the "front" of the device. For audio device 210C, loudspeakers L3, L4, and L5 may be considered to be the front, left, and right speakers, respectively. Thus, the arrow associated with L3 may be seen as the front of audio device 210C.

[0050] The orientation of each loudspeaker may be represented in various ways, depending on the specific implementation. In this example, the orientation of each loudspeaker is represented by the angle between the loudspeaker's maximum radiation direction and a line connecting its associated device to the listening position. This orientation angle may vary between -180 degrees and 180 degrees, with 0 degrees indicating that the loudspeaker is pointed directly at the listening position and -180 degrees or 180 degrees indicating that the loudspeaker is pointed completely away from the listening position. The orientation angle of L1, represented by the value q1 in the figure, is close to zero, indicating that loudspeaker L1 is pointed almost directly at the listening position. Meanwhile, q2 is close to 180 degrees, meaning that speaker L2 is pointed almost directly away from the listening position. In audio device 210C, q3 and q4 have relatively small values ​​and absolute values ​​are less than 90 degrees, indicating that L3 and L4 are pointed substantially toward the listening position. However, q5 has a relatively large value and an absolute value greater than 90 degrees, indicating that L5 is oriented substantially away from the listening position. The positions and orientations of the set of loudspeakers may be determined, or at least estimated, according to various techniques, including but not limited to those disclosed herein.

[0051] As mentioned above, the further a loudspeaker's orientation points away from the intended listening position, the more significant some acoustic properties may change. Two acoustic properties are most prominent. First, the overall equalization heard at the listening position may change. Since most loudspeakers have a higher degree of directivity at higher frequencies, the higher frequencies are usually reduced. Second, the ratio of direct to reflected sound at the listening position may be reduced, since relatively more sound energy is directed away from the listening position and interacts with walls, floors, objects, etc. in the audio environment before finally being heard. While the first problem can often be mitigated to some extent by equalization, the second problem cannot.

[0052] This second problem can be particularly problematic when loudspeakers that point away from the intended listening position are combined with others for spatial reproduction purposes. Localization of the elements of a spatial mix at their desired location is generally best achieved when the loudspeakers that contribute to this localization all have a relatively high direct-to-reflected sound ratio at the listening position. If this is not the case for a particular loudspeaker because it is oriented away from the listening position, the localization may be inaccurate or "smudged." In some instances, it may be beneficial to exclude this loudspeaker from the rendering process to improve localization. However, in some cases, excluding such a loudspeaker from the rendering process may cause even greater disturbance to the overall spatial rendering than including that loudspeaker in the rendering process. For example, if a loudspeaker points away from the listening position, but is the only loudspeaker to the left of the listening position, it may be better to keep this loudspeaker as part of the rendering rather than causing the entire spatial mix to collapse to the right of the listening position due to its exclusion.

[0053] Some disclosed examples involve navigating such selections for rendering systems in which both the position and orientation of the loudspeakers are specified relative to a listening position. For example, some disclosed examples involve rendering a set of one or more audio signals across a set of two or more loudspeakers, each audio signal having an associated desired perceived spatial location. In some such examples, the position and orientation (e.g., relative to a desired listening position or area) of each loudspeaker of the set of loudspeakers is provided to a renderer. According to some such examples, the relative activation of each loudspeaker may be calculated as a function of the desired perceived spatial location of the one or more audio signals and the position and orientation of the loudspeaker. In some such examples, for any given position of a loudspeaker, the activation of the loudspeaker may be reduced as the orientation of the loudspeaker increases away from the listening position. According to some such examples, the extent of this reduction itself may be reduced as a function of a measure of the importance of the loudspeaker for rendering any audio signal at its desired perceived spatial location.

[0054] 3 illustrates another example of an audio environment. According to this example, audio environment 200 includes audio devices 210A, 210B, and 210C of FIG. 2, as well as an additional audio device 210D. Audio device 210D may, in some examples, be an instance of device 150 of FIG. 1. In this example, audio device 210D includes a single loudspeaker L6. An arrow emanating from loudspeaker L6 represents the direction of maximum acoustic radiation associated with loudspeaker L6, indicating that q6 is close to 180 degrees. This means that loudspeaker L6 is pointed almost diametrically away from the listening position corresponding to listener 205.

[0055] 3 also illustrates an example of applying certain aspects of the present disclosure to audio devices 210A-210D. A summary of the behavior resulting from applying this aspect of the present disclosure to each loudspeaker is as follows.

[0056] L1: The orientation angle q1 is small (less than 30 degrees in this example), therefore this loudspeaker is fully utilized (ON).

[0057] L2: The orientation angle q2 is large (close to 180 degrees in this example), and therefore some aspects of the present disclosure indicate that this loudspeaker should be fully or substantially disabled (turned OFF). However, in this example, since L2 is the only loudspeaker behind the listener, the indication of the importance of this loudspeaker for spatial rendering is high. As a result, in this example, loudspeaker L2 is not penalized and is left fully enabled (ON).

[0058] L3: The orientation angle q3 is relatively small (less than 60 degrees in this example), therefore this loudspeaker is fully utilized (ON).

[0059] L4: The orientation angle q4 is relatively small (less than 60 degrees in this example), therefore this loudspeaker is fully utilized (ON).

[0060] L5: The orientation angle q5 is relatively large (in this example, between 130 and 150 degrees), and therefore some aspects of the present disclosure indicate that this loudspeaker should be completely (or at least partially) disabled. Furthermore, in this example, the presence in close proximity of other loudspeakers L3 and L4 in the same enclosure, which are substantially aimed at the listening position, provides a low indication of the importance of this loudspeaker for spatial rendering. As a result, in this example, loudspeaker L5 is left completely disabled (off).

[0061] L6: The orientation angle q6 is relatively large (close to 180 degrees in this example), and therefore some aspects of the present disclosure indicate that this loudspeaker should be fully or at least partially disabled. According to this example, the indication of the importance of this loudspeaker for spatial rendering is relatively low, since there are other loudspeakers L3 and L4 in different enclosures and relatively close together, substantially aimed at the listening position. As a result, in this example, speaker L6 is completely disabled (off).

[0062] The following paragraphs disclose implementations that may achieve the results described with reference to FIG. 3. A flexible rendering system is described in detail below that casts the rendering problem as one of cost function minimization, where the cost function includes two terms. The first term models how close the desired spatial impression is achieved as a function of speaker activation, and the second term assigns a cost to activating a speaker. In some examples, one purpose of this second term is to create a sparse solution in which only speakers that are very close to the desired spatial location of the audio being rendered are activated. According to some examples, the cost function includes one or more additional dynamically configurable terms to this activation penalty, allowing the spatial rendering to be modified in response to various possible controls.

[0063] In some aspects, this cost function may be expressed by the following equation:

number

[0064] The derivation of Equation 1 is explained in detail below. In this example, the set

number

number

number

[0065] In some examples, one or more aspects of the present disclosure provide a method for characterizing loudspeakers in an audio environment using {^s i}, one or more additional cost terms C j According to some such examples, the cost may be calculated as a function of both the position and orientation of each speaker relative to the listening position.

[0066] In some such examples, the general cost function of Equation 1 may be expressed as a quadratic form of a matrix, as follows:

number

[0067] The derivation of Equation 2 is described in detail below. In some examples, each additional cost term may be parameterized by a diagonal matrix of speaker penalty terms, for example, as follows:

number

[0068] Some aspects of this disclosure incorporate these speaker penalty terms w ij may be implemented by computing a set of penalty terms x, y, y, y, y, y, y, y, y, as a function of both the position and orientation of each speaker i. According to some examples, the penalty terms may be computed across different subsets of loudspeakers across frequency depending on the capabilities of each loudspeaker (e.g., according to each loudspeaker's ability to accurately reproduce low frequencies).

[0069] The following discussion assumes that the position and orientation of each loudspeaker i is known, in this example relative to the listening position. Some detailed examples of determining, or at least estimating, the position and orientation of each loudspeaker i are given below. Some flexible rendering methods previously disclosed already take into account the position of each loudspeaker relative to the listening position. Some flexible rendering methods of the present disclosure further incorporate the orientation of the loudspeakers relative to the listening position, as well as the positions of the loudspeakers relative to each other. Loudspeaker orientation has already been defined in this disclosure as an orientation angle θ i The positions of the loudspeakers relative to each other, which may reflect possible disturbances to the spatial rendering introduced by speaker penalty, are parameterized as α iα is sometimes referred to herein as the “loudspeaker importance metric.”

[0070] According to some disclosed examples, loudspeakers may be nominally divided into two categories, "qualified" and "ineligible," meaning that they are either eligible or ineligible for penalty according to the loudspeaker orientation. In some such examples, the determination of whether a loudspeaker is eligible or ineligible depends on the loudspeaker's orientation angle θ i In some such examples, the determination of whether a loudspeaker is qualified or disqualified may be based at least in part on the orientation angle θ of the loudspeaker. i is the orientation angle threshold T θ In some such examples, the loudspeaker may be at least partially based on whether the condition |θ i |>T θ If it satisfies, then the loudspeaker is eligible to be penalized according to the loudspeaker orientation; otherwise, the loudspeaker is ineligible. In one example, the orientation angle threshold T θ = (11 / 18)π radians (110 degrees). However, in other examples, the orientation angle threshold T θ may be greater or less than 110 degrees, e.g., 100 degrees, 105 degrees, 115 degrees, 120 degrees, etc. According to some examples, the position of each qualified speaker may be considered with respect to the positions of unqualified or well-oriented loudspeakers. In some such examples, for a qualified loudspeaker i, loudspeakers i1 and i2 with the shortest clockwise and counterclockwise angular distances φ1 and φ2 from i may be identified in a set of unqualified loudspeakers. The angular distances between the speakers may be determined, in some such examples, by casting the loudspeaker positions onto a unit circle. The listening position is at the center of the unit circle.

[0071] To encapsulate potential impairments, in some examples, a loudspeaker importance metric α may be devised as a function of φ and φ. In some examples, the loudspeaker importance metric α for loudspeaker i may be i corresponds to the unit perpendicular distance from loudspeaker i to the line connecting loudspeakers i1 and i2, the two loudspeakers adjacent to loudspeaker i. Below is one such example where the loudspeaker importance metric α is expressed as a function of φ1 and φ2:

[0072] 4 shows an example of loudspeakers positioned on the circumference of a unit circle. In this example, loudspeakers i, i1, and i2 are located on the circumference of a circle 400, and loudspeaker i is located between loudspeaker i1 and loudspeaker i2. According to this example, the center 405 of the circle 400 corresponds to the listener position. In this example, the angular distance between loudspeaker i and loudspeaker i1 is φ1, the angular distance between loudspeaker i and loudspeaker i2 is φ2, and the angular distance between loudspeaker i1 and loudspeaker i2 is φ3. Since the circle contains 2π radians, φ1+φ2+φ3=2π.

[0073] Figure 5 shows the loudspeaker arrangement of Figure 4 with strings connecting the loudspeaker positions. In this example, string C1 connects loudspeaker i to loudspeaker i1, string C2 connects loudspeaker i to loudspeaker i2, and string C3 connects loudspeaker i1 to loudspeaker i2. By definition, the angle φ N The chord length C on the unit circle spanning N is C N =sin(φ N / 2).

[0074] The inner triangles 505a, 505b, and 505c are isosceles triangles with central angles φ1, φ2, and φ3, respectively. Any inner triangle is also isosceles and has central angles φ nThe sum of the interior angles of a triangle is π radians. Therefore, each of the remaining congruent angles of any interior triangle is (π-φ n ) radians. One such angle, ξ n =(π-φ n ) / 2 is shown in FIG.

[0075] Figure 6 shows the loudspeaker arrangement of Figure 5 with one string omitted. In this example, string C2 of Figure 5 has been omitted to better show triangle 605. Triangle 605 includes a side α that is perpendicular to string C3 and extends from string C3 to loudspeaker i. According to this example, the interior angle a of triangle 605 can be expressed as a=ξ1+ξ2.

[0076] The sine law defines the relationship between the interior angles a, b, and c of a triangle and the lengths of the sides opposite each angle, α, β, and γ, as (sina) / α = (sinb) / β = (sinc) / γ.

[0077] In the example of triangle 605, the sine theorem states that:

number

number

[0078] In some implementations, φ1 or φ2 may be greater than π radians. In such cases, if α were calculated according to Equation 4, it would fall outside the circle. In some such instances, Equation 4 may be modified as follows:

number

[0079] This is a better representation of the energy error introduced by penalizing the corresponding loudspeaker.

[0080] In some examples, if φ1=φ2, then α may be calculated as α=sin(φ1 / 2), since this function fits Equation 4 continuously when φ1 and φ2 are similar.

[0081] In the loudspeaker layouts shown in Figures 4, 5, and 6, according to some implementations, loudspeaker i is not turned off (and in some examples, the relative activation of loudspeaker i is not reduced) regardless of the loudspeaker orientation angle of loudspeaker i, because the distance between loudspeaker i and the line connecting loudspeakers i1 and i2, and therefore the corresponding loudspeaker importance metric of loudspeaker i, is too large.

[0082] FIG. 7 shows an alternative example of loudspeakers located on the circumference of a unit circle. In this example, loudspeakers i, i1, and i2 are located at different positions on the circumference of circle 400 compared to the positions shown in FIGS. 4, 5, and 6. Here, loudspeakers i, i1, and i2 are all located on the same half of circle 400. However, loudspeaker i is still located between loudspeaker i1 and loudspeaker i2, the angular distance between loudspeaker i and loudspeaker i1 is still φ1, the angular distance between loudspeaker i and loudspeaker i2 is still φ2, and the angular distance between loudspeaker i1 and loudspeaker i2 is still φ3. Also, the relationship α i 6, the distance between loudspeaker i and the line 705 connecting loudspeakers i1 and i2, and therefore the corresponding loudspeaker importance metric α i Therefore, according to some implementations, the loudspeaker orientation angle θ i is the orientation angle threshold T θIf the threshold voltage V i is equal to or exceeds V t i , then loudspeaker i may be turned off, or the relative activation of loudspeaker i may at least be reduced.

[0083] 8 and 9 show alternative examples of loudspeakers positioned on the circumference of a unit circle. In this example, loudspeakers L1, L2, and L3 are all located on the same half of circle 400. However, loudspeaker L4 is located on the other half of circle 400. Arrows emanating from each of loudspeakers L1-L4 indicate the direction of maximum sound radiation for each loudspeaker, and therefore the loudspeaker orientation angle θ for each loudspeaker. FIGS. 8 and 9 also show the convex hull of loudspeaker 805 formed by loudspeakers L1-L4.

[0084] As before, the loudspeaker being evaluated is referred to as loudspeaker i, and the loudspeakers adjacent to the loudspeaker being evaluated are referred to as loudspeakers i1 and i2. Thus, in Figure 8, loudspeaker L3 is denoted as speaker i, speaker L1 is denoted as speaker i1, and speaker L2 is denoted as speaker i2. In Figure 8, the loudspeaker importance metric α i indicates the relative importance of loudspeaker L3 for rendering the audio signal at its intended perceived spatial location. In this example, the loudspeaker importance metric α i is much smaller than the loudspeaker importance metric α corresponding to loudspeaker i in FIG. 6, for example. The relatively small loudspeaker importance metric α corresponding to loudspeaker L3 i Therefore, loudspeaker L3 (for example, the orientation angle threshold T θ The spatial impairment introduced by penalizing (by having a loudspeaker orientation angle θ greater than or equal to 100 Hz) may be acceptable.

[0085] 9, loudspeaker L2 is denoted as loudspeaker i, loudspeaker L3 is denoted as loudspeaker i1, and loudspeaker L4 is denoted as speaker i2, where the loudspeaker importance metric α i indicates the relative importance of loudspeaker L2 for rendering the audio signal at its intended perceived spatial location. In this example, the loudspeaker importance metric α i is the importance metric α corresponding to loudspeaker L3 in Figure 8. i is greater than the loudspeaker importance metric α corresponding to loudspeaker L2. i 6. However, in some implementations, the loudspeaker importance metric α corresponding to loudspeaker i in FIG. 6 may be smaller than the loudspeaker importance metric α corresponding to loudspeaker L2 (e.g., the orientation angle threshold T θ The spatial impairment introduced by penalizing (by having a loudspeaker orientation angle θ greater than or equal to 100 Hz) may not be acceptable.

[0086] In some examples, the loudspeaker importance metric α imay correspond to a specific behavior of the spatial cost system described above. When the target audio object position is outside the convex hull of the loudspeakers 805, according to some examples, the solution with the smallest possible error places the audio object on the convex hull of the speakers. In some such examples, if loudspeaker i is penalized to the extent that it is deactivated, the line connecting loudspeakers i1 and i2 becomes part of the convex hull of the loudspeakers 805. This line therefore becomes part of the smallest error solution. For example, with reference to FIG. 8, if loudspeaker L3 is deactivated, the convex hull of the loudspeaker 805 will include line 810 instead of the chord between loudspeakers L1, L3, and L2. With reference to FIG. 9, if loudspeaker L2 is deactivated, the convex hull of the loudspeaker 805 will include line 815 instead of the chord between loudspeakers L3, L2, and L4. The loudspeaker importance metric α i It can be easily seen that α directly correlates with the reduction in the size of the convex hull of a loudspeaker 805 caused by deactivating the corresponding loudspeaker. That is, the perpendicular distance from the speaker in question to the lines connecting the neighboring loudspeakers is the point of maximum divergence between solutions with and without a deactivation penalty for that loudspeaker. For at least these reasons, the loudspeaker importance metric α i is an appropriate metric to represent the potential spatial impairment introduced when penalizing a speaker.

[0087] According to some examples, for each loudspeaker that is eligible for penalty based on the orientation angle of that loudspeaker, a loudspeaker importance metric α i can be calculated. i The larger the value of α, the greater the probability of error. This is demonstrated in Figs. 8 and 9. i is the α in Figure 9. i9. Therefore, the spatial impairment introduced by penalizing i in FIG. 8 may be acceptable. On the other hand, the spatial impairment introduced by penalizing i in FIG. 9 may not be acceptable. To this effect, the importance metric threshold T α is α i In some such examples, the α i <T α and |θ i |>T θ If both are true, the penalty w ij may be calculated (e.g., according to Equation 3) and applied to the loudspeakers as a function of the loudspeaker orientation angle. According to some examples, the importance metric threshold T α can be in the range of 0.1 to 0.35, for example, 0.1, 0.15, 0.2, 0.25, 0.30, or 0.35. In other examples, the importance metric threshold T α may be set to a higher or lower value.

[0088] Depending on the relative magnitude of the penalty in the cost function optimization, any particular penalty can be designed to elicit absolute or incremental behavior. In the case of the renderer cost function, a sufficiently large penalty may exclude or disable the loudspeaker entirely, while a smaller penalty may quiet it down without muting it. The arctangent function, tan -1 x is an advantageous functional form for the penalty because it can be manipulated to reflect this behavior. -1 (x→±∞) is effectively a step function or switch. On the other hand, tan -1(x→0) is effectively a linear ramp; intermediate ranges yield intermediate behavior. Therefore, choosing the arctangent range around x=0 as the functional form of the penalty allows a significant level of control over the system behavior.

[0089] For example, the penalty w in Eq. ij is generally scaled by α with a scaling factor η for precise penalty behavior. i and θ i Equation 5 provides one such example.

number

[0090] In some examples, x and y ∈ [0,1]. The specific scaling factors and their respective arctangent functions are given by θ i and α i may be constructed to ensure precise and gradual deactivation of loudspeaker i from the use of both x and y as a function of η. In some examples, the arctangent functions x and y in Equation 5 may be constructed as follows, with a scale factor η=5.0 in these examples:

number

[0091] In Equations 6 and 7, "r" represents an arctangent function tuning factor that corresponds to half the range of the sampled arctangent function. For r=1, the full output space of the sampled arctangent function has length 2. FIGS. 10 and 11 show Equations 6 and 7, respectively, of the present disclosure, with the elements of each equation identified. According to these examples, elements 1010a and 1010b respectively represent the threshold T Θ and T α, where x is the input variable that is scaled according to x. According to these examples, elements 1015a and 1015b allow the input variables to be expanded across the desired arctangent range. According to these examples, elements 1020a and 1020b shift the input variables so that the centers are aligned as desired with the arctangent function, e.g., x is centered at 0. In these examples, elements 1025a, 1025b, and 1025c scale the output of Equation 6 and Equation 7 to be in the range [0,1]. Element 1025d normalizes the function output by the maximum numerator input.

[0092] Figures 12A and 12B are graphs corresponding to equation 6 of the present disclosure. Figures 13A and 13B are graphs corresponding to equation 7 of the present disclosure. Figures 12A and 13A are sections of an arctangent with a domain of length 2r. Figures 12B and 13B correspond to the same arctangent curve segment as Figures 12A and 13A, respectively, in the range [0,1] over the domain of the input variable to which the penalty is applied, transformed according to equations 6 and 7, respectively.

[0093] 12A-13B show features that make the arctangent function an advantageous functional form for the penalty. In the examples of FIG. 12A and FIG. 12B, r=1, so the full output space of the sampled arctangent function has a length of 2. In the intermediate portions of these curves (e.g., from −0.5 to 0.5), the function approximates a linear ramp. In the examples of FIG. 13A and FIG. 13B, r=2, so the full output space of the sampled arctangent function has a length of 4. In these examples, a relatively small portion of the displayed arctangent function approximates a linear ramp. For values ​​in the range of 1.5 to 3, there is much less change in the function than for values ​​close to zero. Thus, using the arctangent as the functional form for the penalty, along with selecting a desired value of r, allows a significant level of control over system behavior.

[0094] 13C is a graph showing an example of a penalty function based on loudspeaker orientation and importance metric. In this example, graph 1300 shows the penalty function w ij (θ i ,α i ) is shown below. According to this example, the penalty function w ij (θ i ,α i ) is T θ <|θ i |≦π and 0≦α i <T α The former condition requires that the loudspeaker be directed far enough away from the listening position, while the latter condition requires that the speaker be close enough to other speakers so that the spatial image is not compromised by its deactivation or reduced activation. If these conditions are met, then the penalty w ij The application of α to speaker i results in improved localization of audio objects via flexible rendering. i For any particular value of , the penalty w ij The value of |θ i | is T θ increases as |θ increases to π. Thus, the activation of speaker i is reduced as its orientation increases away from the listening position. Furthermore, i For any fixed value of |, the penalty w ij is α i from 0 to T α This means that the amount by which the activation of speaker i is reduced is proportional to the importance metric α i This means that as increases, it becomes smaller.

[0095] FIG. 14 is a flow diagram outlining examples of the disclosed methods. In some examples, method 1400 may be performed by an apparatus such as that shown in FIG. 1. In some examples, method 1400 may be performed by a control system of a coordinating device, which may be an audio device in some examples. The blocks of method 1400, as well as other methods described herein, are not necessarily performed in the order shown. Additionally, such methods may include more or fewer blocks than illustrated and / or described.

[0096] In this example, block 1405 involves receiving audio data by the control system via the interface system. According to this example, the audio data includes one or more audio signals and associated spatial data. In this example, the spatial data indicates an intended perceived spatial location corresponding to one audio signal of the one or more audio signals. In some such examples, the spatial data may be or include metadata. According to some examples, the metadata may correspond to an audio object. In some such examples, the audio signal may correspond to an audio object. In some instances, the audio data may be part of a content stream of audio signals and possibly video signals, at least some of which are intended to be heard together. Examples include selections of music, movie soundtracks, movies, television programs, audio portions of television programs, podcasts, live voice calls, synthesized voice responses from a smart assistant, etc. In some instances, the audio data may be received from another device, e.g., via wireless communication. In other instances, the audio data may be received or retrieved from a memory of the same device that includes the control system.

[0097] According to this example, block 1410 involves receiving listener position data by the control system via the interface system. In this example, the listener position data indicates listener positions corresponding to people in the audio environment. In some instances, the listener position data may indicate a listener's head position. In some implementations, block 1410, or another block of method 1400, may involve receiving listener orientation data. Various methods of estimating listener positions and orientations are disclosed herein.

[0098] In this example, block 1415 involves receiving, by the control system via the interface system, loudspeaker position data indicating a position of each loudspeaker of a plurality of loudspeakers in the audio environment, in some examples, the plurality may include all loudspeakers in the audio environment, while in other examples, the plurality may include only a subset of the total number of loudspeakers in the audio environment.

[0099] According to this example, block 1420 involves receiving loudspeaker orientation data by the control system via the interface system. The loudspeaker orientation data may vary according to a specific implementation. In this example, the loudspeaker orientation data indicates a loudspeaker orientation angle between (a) a direction of maximum sound radiation for each loudspeaker of a plurality of loudspeakers in an audio environment and (b) a listener position relative to the corresponding loudspeaker. According to some such examples, the loudspeaker orientation angle for a particular loudspeaker may be an angle between (a) a direction of maximum sound radiation for that particular loudspeaker and (b) a line between the position of that particular loudspeaker and the listener position. In other examples, the loudspeaker orientation data may indicate loudspeaker orientation angles according to another reference frame, such as an audio environment coordinate system, an audio device reference frame, etc. Alternatively or additionally, in some examples, the loudspeaker orientation angle may not be defined according to a direction of maximum sound radiation for each loudspeaker, but may instead be defined in another manner, for example, by an orientation of a device including the loudspeakers.

[0100] In this example, block 1425 involves rendering, by a control system, the audio data for playback through at least a subset of a plurality of loudspeakers in the audio environment to generate a rendered audio signal. According to this example, the rendering is based at least in part on spatial data, listener position data, loudspeaker position data, and loudspeaker orientation data. In this example, the rendering involves applying a loudspeaker orientation factor that tends to reduce relative activation of the loudspeakers based at least in part on the increased loudspeaker orientation angle. In this example, block 1430 involves providing, via an interface system, the rendered audio signal to at least said subset of loudspeakers of the plurality of loudspeakers in the audio environment.

[0101] In some examples, method 1400 may involve estimating a loudspeaker importance metric for at least the subset of loudspeakers. According to some examples, the loudspeaker importance metric may correspond to the importance of the loudspeaker for rendering the audio signal at an intended perceived spatial location of the audio signal. In some examples, the rendering for each loudspeaker may be based at least in part on the loudspeaker importance metric.

[0102] According to some implementations, the rendering for each loudspeaker may involve modifying an effect of a loudspeaker orientation factor based at least in part on the loudspeaker importance metric. In some such examples, the rendering for each loudspeaker may involve reducing an effect of a loudspeaker orientation factor based at least in part on the increased loudspeaker importance metric.

[0103] According to some examples, the method 1400 may involve determining whether a loudspeaker orientation angle is equal to or exceeds a threshold loudspeaker orientation angle. In some such examples, the method 1400 may involve applying the loudspeaker orientation factor only if the loudspeaker orientation angle is equal to or exceeds the threshold loudspeaker orientation angle. In some examples, a "qualified loudspeaker" may be a loudspeaker that has a loudspeaker orientation angle that is equal to or exceeds the threshold loudspeaker orientation angle. In this context, a "qualified loudspeaker" is a loudspeaker that is eligible to be penalized, e.g., to be attenuated (reducing relative speaker activation) or turned off.

[0104] In some examples, the loudspeaker importance metric for a particular loudspeaker may be based at least in part on the location of that particular loudspeaker relative to the locations of one or more other loudspeakers. For example, if a loudspeaker is relatively close to other loudspeakers, the perceptual change caused by penalizing any of these closely spaced loudspeakers may be less than the perceptual change caused by penalizing another loudspeaker that is not close to other loudspeakers in the audio environment.

[0105] According to some examples, the loudspeaker importance metric may be based at least in part on a distance between a qualified loudspeaker and a line between (a) a first loudspeaker with the shortest clockwise angular distance from the qualified loudspeaker and (b) a second loudspeaker with the shortest counterclockwise angular distance from the qualified loudspeaker. This distance may, in some examples, correspond to the loudspeaker importance metric α disclosed herein. As mentioned above, in some examples, a "qualified" loudspeaker is a loudspeaker with a loudspeaker orientation angle that is equal to or exceeds a threshold loudspeaker orientation angle. In some examples, the first loudspeaker and the second loudspeaker may be ineligible loudspeakers with loudspeaker orientation angles less than the threshold loudspeaker orientation angle. These ineligible loudspeakers may be ineligible for penalization, for example, for being attenuated (reducing relative speaker activation) or turned off.

[0106] In some examples, the rendering of block 1425 may involve determining a relative activation for at least the subset of loudspeakers by optimizing a cost function. In some such examples, block 1425 may involve determining a relative activation for at least the subset of loudspeakers by optimizing a cost that is a function of: a model of a perceived spatial location of an audio signal of the one or more audio signals when played on the subset of loudspeakers in an audio environment; a measure of the proximity of the intended perceived spatial location of the audio signal to a location of each loudspeaker of the subset of loudspeakers; and one or more additional dynamically configurable functions.

[0107] According to some examples, at least one of the one or more additional dynamically configurable functions may be based at least in part on a loudspeaker orientation factor. In some examples, at least one of the one or more additional dynamically configurable functions may be based at least in part on a loudspeaker importance metric. According to some examples, at least one of the one or more additional dynamically configurable functions may be based at least in part on measurements or estimates of acoustic transmission from each loudspeaker in the audio environment to one or more other loudspeakers in the audio environment.

[0108] Example of how to estimate the position and orientation of an audio device As described in the description of FIG. 14 and elsewhere herein, in some examples, audio processing modifications (such as those corresponding to loudspeaker orientations, loudspeaker importance metrics, or both) may be based at least in part on information of audio device locations and audio device orientations. The locations and orientations of audio devices in an audio environment may be determined or estimated by a variety of methods, including but not limited to those described in the following paragraphs. While this discussion refers to the locations and orientations of audio devices, one skilled in the art will recognize that, given information about how one or more loudspeakers are positioned in a corresponding audio device, the locations and orientations of the loudspeakers may be determined according to the locations and orientations of the audio devices.

[0109] Some such methods may involve receiving direct instructions by a user, for example, using a smartphone or tablet device to mark or indicate the approximate location of the audio device on a floor plan or similar pictorial representation of the environment. Such digital interfaces are already common in managing the configuration, grouping, names, purposes and identities of smart home devices. For example, such direct instructions may be provided via an Amazon Alexa smartphone application, a Sonos S2 controller application, or a similar application.

[0110] Some examples may involve solving a basic triangulation problem using measured signal strengths (sometimes referred to as Received Signal Strength Indication (RSSI)) of common wireless communication technologies such as Bluetooth, Wi-Fi, ZigBee, etc., to generate an estimate of the physical distance between audio devices, e.g., as disclosed in non-patent document 1, and / or as disclosed in non-patent document 2, both of which are incorporated herein by reference. [Non-Patent Document 1] J. Yang and Y. Chen, "Indoor Localization Using Improved RSS-Based Lateration Methods", GLOBECOM 2009 - 2009 IEEE Global Telecommunications Conference, Honolulu, HI, 2009, pp. 1-6, doi: 10.1109 / GLOCOM.2009.5425237 [Non-Patent Document 2] Mardeni, R. & Othman, Shaiful & Nizam, “Node Positioning in ZigBee Network Using Trilateration Method Based on the Received Signal Strength Indicator (RSSI)”, 2010, 46.

[0111] US Patent No. 5,399,633, entitled "Automatic Discovery and Localization of Speaker Locations in Surround Sound Systems," which is incorporated herein by reference, describes a system that can automatically locate loudspeakers and microphones in a listening environment by acoustically measuring the time of arrival (TOA) between each speaker and the microphone. [Patent Document 1] U.S. Patent No. 10,779,084

[0112] International Applications PCT / US21 / 61506 and PCT / US21 / 61533, entitled "AUTOMATIC LOCALIZATION OF AUDIO DEVICES" ("Automatic Localization Application"), which are incorporated herein by reference, disclose methods, devices, and systems for automatically determining the location and orientation of audio devices. Figs. 4-9B and the corresponding description on pages 17-47 are specifically incorporated herein by reference. Some disclosed examples of the automatic localization application involve receiving direction of arrival (DOA) data corresponding to sounds emitted by at least a first smart audio device of an audio environment. In some implementations, the first smart audio device may include a first audio transmitter and a first audio receiver. In some examples, the DOA data may correspond to sounds received by at least a second smart audio device of the audio environment. In some instances, the second smart audio device may include a second audio transmitter and a second audio receiver. In some examples, the DOA data may also correspond to sounds emitted by at least a second smart audio device and received by at least a first smart audio device.

[0113] Some such methods may involve receiving, by a control system, configuration parameters. In some examples, the configuration parameters may correspond to an audio environment and / or to one or more audio devices of the audio environment. Some such methods may involve minimizing, by the control system, a cost function based at least in part on the DOA data and the configuration parameters to estimate a position and / or orientation of at least the first smart audio device and the second smart audio device.

[0114] According to some examples, the DOA data may also correspond to sounds received by one or more passive audio receivers of the audio environment. In some examples, each of the one or more passive audio receivers may include a microphone array, but in some cases may lack an audio emitter. In some such examples, minimizing the cost function may also provide an estimated position and orientation of each of the one or more passive audio receivers.

[0115] In some examples, the DOA data may also correspond to sounds emitted by one or more audio emitters of the audio environment. In some cases, each of the one or more audio emitters may include at least one sound emitting transducer, but in some cases may lack a microphone array. In some such examples, minimizing the cost function may also provide an estimated position of each of the one or more audio emitters.

[0116] In some implementations, the DOA data may also correspond to sounds emitted by a third through Nth smart audio device of the audio environment, where N corresponds to a total number of smart audio devices in the audio environment. In some examples, the DOA data may also correspond to sounds received by each of the first through Nth smart audio devices from every other smart audio device in the audio environment. In some such examples, minimizing the cost function may involve estimating positions and / or orientations of the third through Nth smart audio devices.

[0117] According to some examples, the configuration parameters may include the number of audio devices in the audio environment, one or more dimensions of the audio environment, and / or one or more constraints on the positions and / or orientations of the audio devices. In some instances, the configuration parameters may include disambiguation data for rotation, translation, and / or scaling.

[0118] Some methods can include receiving, by a control system, a seed layout of the cost function, which in some examples can specify a correct number of audio transmitters and receivers in the audio environment and arbitrary positions and orientations for each of the audio transmitters and receivers in the audio environment.

[0119] Some methods may involve receiving, by the control system, a weighting factor associated with one or more elements of the DOA data, which may indicate, for example, availability and / or reliability of the one or more elements of the DOA data.

[0120] Some methods may involve the control system obtaining one or more elements of DOA data using beamforming methods, steered power response methods, time difference of arrival methods, structured signal methods, or combinations thereof.

[0121] Some methods may involve receiving, by the control system, time of arrival (TOA) data corresponding to sounds emitted by at least one audio device of the audio environment and received by at least one other audio device of the audio environment. In some such examples, a cost function may be based at least in part on the TOA data. Some such methods may involve estimating at least one playback latency and / or estimating at least one recording latency. In some examples, the cost function may operate using rescaled positions, rescaled latencies, and / or rescaled arrival times.

[0122] According to some examples, the cost function may include a first term that depends only on the DOA data. In some such examples, the cost function may include a second term that depends only on the TOA data. In some such examples, the first term may include a first weighting factor and the second term may include a second weighting factor. In some instances, one or more TOA elements in the second term may have a TOA element weighting factor indicative of the availability and / or reliability of each of the one or more TOA elements.

[0123] In some examples, the configuration parameters may include playback latency data, recording latency data, data for disambiguating latency symmetry, rotation disambiguation data, translation disambiguation data, scaling disambiguation data, and / or one or more combinations thereof.

[0124] Some other aspects of the disclosure may be implemented via methods. Some such methods may involve device localization. For example, some methods may include locating a device in an audio environment. Some such methods may involve obtaining, by a control system, direction of arrival (DOA) data corresponding to a transmission of at least a first transceiver of a first device in the environment. The first transceiver, in some examples, may include a first transmitter and a first receiver. In some cases, the DOA data may correspond to a transmission received by at least a second transceiver of a second device in the environment. In some examples, the second transceiver may include a second transmitter and a second receiver. In some cases, the DOA data may correspond to a transmission from at least the second transceiver received by the at least the first transceiver.

[0125] In some examples, the first device and the second device may be audio devices and the environment may be an audio environment. According to some such examples, the first transmitter and the second transmitter may be audio transmitters. In some such examples, the first receiver and the second receiver may be audio receivers. In some implementations, the first transceiver and the second transceiver may be configured to transmit and receive electromagnetic waves.

[0126] Some such methods may involve receiving, by a control system, configuration parameters. In some cases, the configuration parameters may correspond to an environment and / or to one or more devices in the environment. Some such methods may involve minimizing, by the control system, a cost function based at least in part on the DOA data and the configuration parameters to estimate positions and / or orientations of at least the first device and the second device.

[0127] In some examples, the DOA data may also correspond to transmissions received by one or more passive receivers in the environment. Each of the one or more passive receivers may, for example, include a receiver array but may lack a transmitter. In some such examples, minimizing the cost function may also provide an estimated position and / or orientation of each of the one or more passive receivers.

[0128] According to some examples, the DOA data may also correspond to transmissions from one or more transmitters in the environment. In some cases, each of the one or more transmitters may lack a receiver array. In some such examples, minimizing the cost function may also provide an estimated location of each of the one or more transmitters.

[0129] In some examples, the DOA data may also correspond to transmissions emitted by third through Nth transceivers of third through Nth devices in the environment, where N corresponds to a total number of transceivers in the environment. In some such examples, the DOA data may also correspond to transmissions received by each of the first through Nth transceivers from every other transceiver in the environment. In some such examples, minimizing the cost function may involve estimating positions and / or orientations of the third through Nth transceivers.

[0130] WO 2021 / 127286 A1, entitled "Audio Device Auto-Location," which is incorporated herein by reference, discloses a method for estimating a location of an audio device in an audio environment, a location of a listener, and an orientation of the listener. Some disclosed methods involve estimating an audio device location in an environment via direction of arrival (DOA) data and by determining interior angles for each of a plurality of triangles based on the DOA data. In some examples, each triangle has a vertex that corresponds to a location of an audio device. Some disclosed methods involve determining a side length for each side of each of the triangles and performing a forward alignment process that aligns each of the plurality of triangles to generate a forward alignment matrix. Some disclosed methods involve determining to perform a backward alignment process that aligns each of the plurality of triangles in a reverse sequence to generate a backward alignment matrix. A final estimate of each audio device location may be based, at least in part, on values ​​of the forward alignment matrix and values ​​of the backward alignment matrix.

[0131] Other disclosed methods of WO 2021 / 127286 A1 involve estimating the listener position, and in some cases, the listener position. Some such methods involve prompting the listener (e.g., via audio prompts from one or more loudspeakers in the environment) to make one or more utterances, and estimating the listener position according to DOA data. The DOA data may correspond to microphone data acquired by multiple microphones in the environment. The microphone data may correspond to detection of one or more utterances by the microphones. At least some of the microphones may be co-located with the loudspeakers. According to some examples, estimating the listener position may involve a triangulation process. Some such examples involve triangulating the user's voice by finding an intersection between DOA vectors passing through an audio device. Some disclosed methods of determining the listener's orientation involve prompting the user to identify one or more loudspeaker positions. Some such examples involve prompting the user to identify one or more loudspeaker positions by moving next to the loudspeaker positions and making an utterance. Another example involves prompting a user to identify one or more loudspeaker locations by pointing to each of the one or more loudspeaker locations with a handheld device, such as a mobile phone, that includes an inertial sensor system and a wireless interface configured to communicate with a control system (such as a control system of a master device) that controls audio devices in the audio environment. Determining the listener orientation involves having the loudspeakers render audio objects such that they appear to rotate around the listener, and prompting the listener to speak (e.g., "Stop!") when the listener perceives the audio object to be at a loudspeaker location, a television location, or the like.Some disclosed methods involve determining the position and / or orientation of a listener via camera data, for example, by determining a relative position of the listener and one or more audio devices of the audio environment in accordance with the camera data, by determining an orientation of the listener relative to one or more audio devices of the audio environment in accordance with the camera data (e.g., in accordance with the direction the listener is facing), etc.

[0132] In Non-Patent Document 3, which is incorporated herein by reference, a system is described in which a single linear microphone array associated with a component of a playback system whose position is predictable, such as a soundbar, a front center speaker, measures the time difference of arrival (TDOA) for both the satellite loudspeakers and the listener to localize the positions of both the loudspeakers and the listener. In this case, the listening orientation is inherently defined as a line connecting the detected listening position and a component of the playback system that includes a linear microphone array, such as a soundbar that is co-located with the television (placed directly above or below the television). Since the position of the soundbar is predictably placed directly above or below the video screen, the geometry of the measured distances and angles of incidence can be converted to absolute positions relative to any point in front of that reference soundbar position using simple trigonometric principles. The distance between the loudspeaker and the microphones of the linear microphone array can be estimated by playing a test signal and measuring the time of flight (TOF) between the radiating loudspeaker and the receiving microphone. For this purpose, the time delay of the DC component of the measured impulse response can be used. The impulse response between the loudspeaker and the microphone array elements can be obtained by playing a test signal through the loudspeaker to be analyzed. For example, either a maximum length sequence (MLS) or a chirp signal (also known as a logarithmic sine sweep) can be used as the test signal. The room impulse response can be obtained by calculating the circular cross-correlation between the captured signal and the MLS input. Figure 2 of this paper shows the echo impulse response obtained with the MLS input. This impulse response is said to be similar to measurements made in a typical office or living room. The delay of the direct component is used to estimate the distance between the loudspeaker and the microphone array elements.For loudspeaker distance estimation, any loopback latency of the audio device used to play the test signal should be calculated and removed from the measured TOF estimate. [Non-Patent Document 3] Shi, Guangi et al, Spatial Calibration of Surround Sound Systems including Listener Position Estimation, AES 137th Convention, October 2014

[0133] Example of estimating a person's position and orientation in an audio environment The position and orientation of a person in an audio environment may be determined or estimated by a variety of methods, including but not limited to those described in the following paragraphs.

[0134] In Non-Patent Document 4, which is incorporated herein by reference, a number of commercially available techniques are presented for tracking both the position and orientation of listeners' heads in the context of spatial audio playback systems. One particular example described is the Microsoft Kinect. With its depth sensing and standard camera, and publicly available software (Windows Software Development Kit (SDK)), the head positions and orientations of several listeners in space can be tracked simultaneously, using a combination of skeletal tracking and face recognition. Kinect for Windows has been discontinued, but the Azure Kinect Development Kit (DK) is now available, which implements the next generation of Microsoft's depth sensor. [Non-Patent Document 4] Hess, Wolfgang, Head-Tracking Techniques for Virtual Acoustic Applications, AES 133rd Convention, October 2012

[0135] U.S. Patent No. 10,779,084, entitled "Automatic Discovery and Localization of Speaker Locations in Surround Sound Systems," which is incorporated herein by reference, describes a system that can automatically locate loudspeakers and microphones in a listening environment by acoustically measuring the time of arrival (TOA) between each speaker and the microphone. A listening location may be detected by placing and locating a microphone (e.g., a microphone in a mobile phone held by the listener) at a desired listening position, and an associated listening orientation may be defined by placing another microphone at a point in the listener's viewing direction, e.g., on a TV. Alternatively, a listening orientation may be defined by locating a loudspeaker in the viewing direction, e.g., a loudspeaker on a TV.

[0136] WO 2021 / 127286 A1, entitled "Audio Device Auto-Location," which is incorporated herein by reference, discloses a method for estimating a position of an audio device, a position of a listener, and a location of a listener in an audio environment. Some disclosed methods involve estimating an audio device position in an environment via direction of arrival (DOA) data and by determining interior angles for each of a plurality of triangles based on the DOA data. In some examples, each triangle has a vertex that corresponds to a position of an audio device. Some disclosed methods involve determining a side length of each side of each of the triangles and performing a forward alignment process that aligns each of the plurality of triangles to generate a forward alignment matrix. Some disclosed methods involve determining to perform a backward alignment process that aligns each of the plurality of triangles in a backward sequence to generate a backward alignment matrix. A final estimate of the position of each audio device may be based, at least in part, on values ​​of the forward alignment matrix and values ​​of the backward alignment matrix.

[0137] Other disclosed methods of WO 2021 / 127286 A1 involve estimating the listener position, and in some cases, the listener position. Some such methods involve prompting the listener (e.g., via audio prompts from one or more loudspeakers in the environment) to make one or more utterances, and estimating the listener position according to DOA data. The DOA data may correspond to microphone data acquired by multiple microphones in the environment. The microphone data may correspond to detection of one or more utterances by the microphones. At least some of the microphones may be co-located with the loudspeakers. According to some examples, estimating the listener position may involve a triangulation process. Some such examples involve triangulating the user's voice by finding an intersection between DOA vectors passing through an audio device. Some disclosed methods of determining the listener's orientation involve prompting the user to identify one or more loudspeaker positions. Some such examples involve prompting the user to identify one or more loudspeaker positions by moving next to the loudspeaker positions and making an utterance. Another example involves prompting a user to identify one or more loudspeaker locations by pointing to each of the one or more loudspeaker locations with a handheld device, such as a mobile phone, that includes an inertial sensor system and a wireless interface configured to communicate with a control system (such as a control system of a master device) that controls audio devices in the audio environment. Determining the listener orientation involves having the loudspeakers render audio objects such that they appear to rotate around the listener, and prompting the listener to speak (e.g., "Stop!") when the listener perceives the audio object to be at a loudspeaker location, a television location, or the like.Some disclosed methods involve determining the position and / or orientation of a listener via camera data, for example, by determining a relative position of the listener and one or more audio devices of the audio environment in accordance with the camera data, by determining an orientation of the listener relative to one or more audio devices of the audio environment in accordance with the camera data (e.g., in accordance with the direction the listener is facing), etc.

[0138] In Non-Patent Document 3, which is incorporated herein by reference, a system is described in which a single linear microphone array associated with a component of a playback system whose position is predictable, such as a soundbar, a front center speaker, measures the time difference of arrival (TDOA) for both the satellite loudspeakers and the listener to localize the positions of both the loudspeakers and the listener. In this case, the listening orientation is inherently defined as a line connecting the detected listening position and a component of the playback system that includes a linear microphone array, such as a soundbar that is co-located with the television (placed directly above or below the television). Since the position of the soundbar is predictably placed directly above or below the video screen, the geometry of the measured distances and angles of incidence can be converted to absolute positions relative to any point in front of that reference soundbar position using simple trigonometric principles. The distance between the loudspeaker and the microphones of the linear microphone array can be estimated by playing a test signal and measuring the time of flight (TOF) between the radiating loudspeaker and the receiving microphone. For this purpose, the time delay of the DC component of the measured impulse response can be used. The impulse response between the loudspeaker and the microphone array elements can be obtained by playing a test signal through the loudspeaker to be analyzed. For example, either a maximum length sequence (MLS) or a chirp signal (also known as a logarithmic sine sweep) can be used as the test signal. The room impulse response can be obtained by calculating the circular cross-correlation between the captured signal and the MLS input. Figure 2 of this paper shows the echo impulse response obtained with the MLS input. This impulse response is said to be similar to measurements made in a typical office or living room. The delay of the direct component is used to estimate the distance between the loudspeaker and the microphone array elements.For loudspeaker distance estimation, any loopback latency of the audio device used to play the test signal should be calculated and removed from the measured TOF estimate.

[0139] Further examples of audio processing modifications involving cost function optimization As noted elsewhere herein, in various disclosed examples, one or more types of audio processing modifications may be based on optimizing a cost function. Some such examples involve elastic rendering.

[0140] Flexible rendering allows spatial audio to be rendered across any number of arbitrarily positioned speakers. In light of the widespread deployment of audio devices, including but not limited to smart audio devices (e.g., smart speakers), in the home, there is a need to implement flexible rendering techniques that allow consumer products to perform flexible rendering of audio and playback of the audio so rendered.

[0141] Several techniques have been developed to achieve flexible rendering. They cast the rendering problem as one of cost function minimization, where the cost function consists of two terms: a first term that models the desired spatial impression the renderer is trying to achieve, and a second term that assigns a cost to activating speakers. To date, this second term has focused on creating a sparse solution, where only speakers close to the desired spatial location of the audio being rendered are activated.

[0142] Spatial audio playback in consumer environments is typically tied to a given number of loudspeakers located in given locations, e.g., 5.1 and 7.1 surround sound. In these cases, content is specifically authored for the relevant loudspeakers and encoded as one discrete channel for each loudspeaker (e.g., Dolby Digital, or Dolby Digital Plus, etc.). More recently, immersive object-based spatial audio formats have been introduced that break this association between content and specific loudspeaker locations (Dolby Atmos). Instead, content may be described as a collection of individual audio objects, each with possibly time-varying metadata that describes the desired perceived location of said audio object in three-dimensional space. At playback, the content is transformed into loudspeaker feeds by a renderer that matches the number and location of the loudspeakers in the playback system. However, many such renderers still constrain the position of a set of loudspeakers to be one of a set of prescribed layouts (e.g., for Dolby Atmos: 3.1.2, 5.1.2, 7.1.4, 9.1.6, etc.).

[0143] Going beyond such constrained rendering, methods have been developed that allow object-based audio to be flexibly rendered across a truly arbitrary number of loudspeakers placed in arbitrary positions. These methods require the renderer to have knowledge of the number and physical locations of the loudspeakers in the listening space. For such systems to be practical for the average consumer, an automated method for localizing the loudspeakers is desirable. One such method relies on the use of multiple microphones, possibly co-located with the loudspeakers. By playing audio signals through the loudspeakers and recording with the microphones, the distance between each loudspeaker and microphone is estimated. From these distances, the locations of both the loudspeakers and the microphones are then estimated.

[0144] Concurrent with the introduction of object-based spatial audio in consumer spaces has been the rapid adoption of so-called “smart speakers” such as the Amazon Echo product line. While the tremendous popularity of these devices can be attributed to their simplicity and convenience afforded by wireless connectivity and integrated voice interfaces (e.g., Amazon’s Alexa), the acoustic capabilities of these devices are generally limited, especially with regard to spatial audio. In most cases, these devices are constrained to mono or stereo playback. However, combining the aforementioned flexible rendering and automatic localization techniques with multiple coordinated smart speakers can result in a system with very sophisticated spatial playback capabilities, yet remains extremely simple for consumers to set up. Consumers can place as many or as few speakers as desired, wherever convenient, without the need to run speaker wires for wireless connectivity, and the built-in microphones can be used to automatically localize the speakers for their associated flexible renderers.

[0145] Conventional flexible rendering algorithms are designed to achieve as closely as possible a particular desired perceived spatial impression. In a coordinated smart speaker system, sometimes maintaining this spatial impression may not be the most important or desired objective. For example, if someone is attempting to simultaneously speak to an integrated voice assistant, it may be desirable to momentarily modify the spatial rendering to reduce the relative playback level at speakers near certain microphones in order to increase the signal-to-noise ratio and / or signal-to-echo ratio (SER) of microphone signals containing detected speech. Some embodiments described herein may be implemented as modifications to existing flexible rendering methods to allow such dynamic modifications to the spatial rendering, e.g., for the purpose of achieving one or more additional objectives.

[0146] Existing flexible rendering techniques include Center of Mass Amplitude Panning (CMAP) and Flexible Virtualization (FV). From a high level, both of these techniques render a set of one or more audio signals, each with an associated desired perceived spatial location, for playback on a set of two or more speakers, where the relative activation of the set of speakers is a function of a model of the perceived spatial location of the audio signals played on those speakers and the proximity of the desired perceived spatial location of the audio signals to the positions of those speakers. The model ensures that the audio signals are heard by the listener near the intended spatial location, and the proximity term controls which speakers are used to achieve this spatial impression. In particular, the proximity term prioritizes activation of speakers that are close to the desired perceived spatial location of the audio signals. For both CMAP and FV, this functional relationship is conveniently derived from a cost function written as a sum of two terms, a term for spatial aspects and a term for proximity.

number

[0147] Here, the set { → s i} denotes the positions of the set of M loudspeakers, → Let o denote the desired perceived spatial location of the audio signal, and g denote an M-dimensional vector of speaker activations. For CMAP, each activation in the vector represents a gain per speaker, whereas for FV, each activation represents a filter (in this second case, g may be equivalently viewed as a vector of complex values ​​at specific frequencies, with different g's being calculated across multiple frequencies to form filters). The optimal vector of activations is found by minimizing a cost function over the activations.

number

[0148] In some definitions of the cost function, g opt Even if the relative levels between the components of are appropriate, it is difficult to control the absolute level of optimal activation obtained from the above minimization. To address this issue, we set g so that the absolute level of activation is controlled. opt Subsequent normalization of the vectors may be performed. For example, it may be desirable to normalize the vectors to have unit length, which is consistent with the commonly used constant power panning convention.

number

[0149] The exact behavior of the flexible rendering algorithm is determined by the two terms of the cost function C spatial and C proximity For CMAP, C spatial represents the perceived spatial location of audio signals reproduced from a set of loudspeakers in terms of their associated activation gains gi The loudspeaker positions are derived from a model that places them at the center of gravity of the loudspeaker positions weighted by (elements of vector g).

number

[0150] Equation 10 is then manipulated into a spatial cost that represents the squared error between the desired audio position and the position produced by the activated loudspeakers.

number

[0151] In FV, the spatial terms of the cost function are defined differently. Here, the goal is to find the audio object position at the left and right ears of the listener. → The goal is to generate a binaural response b corresponding to o. Conceptually, b is a 2×1 vector of filters (one filter for each ear), but it is more convenient to treat it as a 2×1 vector of complex values ​​at specific frequencies. Proceeding with this representation at specific frequencies, the desired binaural response can be retrieved from a set of HRTFs indexed by object position.

number

[0152] Together, the 2×1 binaural response e produced at the listener's ears by the loudspeakers is modeled as a 2×M acoustic transfer matrix H multiplied by an M×1 vector g of complex speaker activation values:

number

[0153] The acoustic transfer matrix H is a set of loudspeaker positions relative to the listener positions { → s iFinally, the spatial component of the cost function is defined as the squared error between the desired binaural response (Equation 12) and the binaural response produced by the loudspeakers (Equation 13):

number

[0154] Conveniently, the spatial terms of the CMAP and FV cost functions defined in Equation 11 and Equation 14 can both be rearranged into quadratic forms of matrices as functions of the speaker activations g.

number

[0155] To this end, the second term of the cost function, C proximity can be defined as the distance-weighted sum of the squared absolute values ​​of the speaker activations, which can be compactly expressed in matrix form as

number

number

[0156] The distance penalty function can take many forms, but the following is a useful parameterization:

number

[0157] The overall cost function is obtained by combining the two terms of the cost function defined in Equation 15 and Equation 16a.

number

[0158] By setting the derivative of this cost function with respect to g to zero and solving for g, we obtain the optimal speaker activation solution.

number

[0159] In general, the optimal solution in Equation 18 may give speaker activations that are negative in value. For the CMAP construction of a flexible renderer, such negative activations may be undesirable, and thus Equation 18 may be minimized, subject to all activations remaining positive.

[0160] 15 and 16 are diagrams illustrating an example set of speaker activation and object rendering positions. In these examples, the speaker activation and object rendering positions correspond to speaker positions of 4, 64, 165, -87, and -4 degrees. FIG. 15 shows speaker activations 1505a, 1510a, 1515a, 1520a, and 1525a that contain the optimal solution to Equation 11 for these particular speaker positions. FIG. 16 plots the individual speaker positions as points 1605, 1610, 1615, 1620, and 1625, which correspond to speaker activations 1505a, 1510a, 1515a, 1520a, and 1525a, respectively. FIG. 16 also shows ideal object positions (in other words, the positions where the audio objects should be rendered) for a number of possible object angles as dots 1630a, and the corresponding actual rendering positions for those objects as dots 1635a connected to the ideal object positions by dotted lines 1640a.

[0161] One class of embodiments involves a method of rendering audio for playback by at least one (e.g., all or some) of multiple coordinated (coordinated) smart audio devices. For example, a set of smart audio devices present in a user's home (a system in a user's home) may be coordinated to handle a variety of simultaneous use cases, including flexible rendering (in some embodiments) of audio for playback by all or some of the smart audio devices (i.e., by all or some speakers). Many interactions with the system are possible that require dynamic modifications to the rendering. Such modifications may, but do not necessarily, focus on spatial fidelity.

[0162] Some embodiments are methods for rendering audio for playback by at least one (e.g., in whole or in part) of a set of smart audio devices (or for playback by at least one (e.g., in whole or in part) of a speaker of another set of speakers). The rendering may include minimization of a cost function, the cost function including at least one dynamic speaker activation term. Examples of such dynamic speaker activation terms include, but are not limited to, the following: · The proximity of the speaker to the listener or listeners; ·The speaker's proximity to attractive or repulsive forces; · Audibility of the loudspeaker relative to some position (e.g. the listener's position, or the baby's room); · Speaker capabilities (e.g. frequency response and distortion); · Synchronization of speakers relative to other speakers; · Wake word performance; -Echo canceller performance.

[0163] The dynamic speaker activation term may enable at least one of a variety of behaviors, including distorting the spatial presentation of audio away from a particular smart audio device so that its microphone can better hear the speaker or so that a secondary audio stream can be better heard from the smart audio device's speaker.

[0164] Some embodiments implement rendering for playback through speakers of multiple coordinated smart audio devices, while other embodiments implement rendering for playback through speakers of a separate set of speakers.

[0165] Pairing the flexible rendering method (implemented according to some embodiments) with a set of wireless smart speakers (or other smart audio devices) can provide an extremely capable and easy-to-use spatial audio rendering system. Considering interactions with such a system, it becomes apparent that dynamic modifications to the spatial rendering may be desirable to optimize other objectives that may arise during use of the system. To achieve this goal, one class of embodiments augments existing flexible rendering algorithms (wherein speaker activation is a function of spatial and proximity terms as previously disclosed) with one or more additional dynamically configurable functions that depend on one or more characteristics of the audio signal being rendered, the set of speakers, and / or other external inputs. According to some embodiments, the existing cost function of flexible rendering given in Equation 1 is augmented with these one or more additional dependencies according to the following equation:

number

[0166] Equation 19 corresponds to the above equation 1. Thus, the above discussion explains the derivation of equation 1 as well as the derivation of equation 19.

[0167] In Equation 19, the term C j (g,{{^o},{^s i},{^e}} j ) represents an additional cost term, {^o} represents a set of one or more characteristics of the audio signal being rendered (e.g., for an object-based audio program), and {^s i} represents a set of one or more characteristics of the speakers to which the audio is being rendered, and {^e} represents one or more additional external inputs. j (g,{{^o},{^s i},{^e}} j ) is the set {{^o},{^s i Returns a cost as a function of activations g associated with a combination of one or more characteristics of the audio signal, the speakers, and / or the external input, generally denoted by},{^e}}. i},{^e}} is at least {^o}, {^s i It should be understood that the expression may contain only one element from either {^e}, {^n}, or {^n}.

[0168] Examples of {^o} include, but are not limited to, the following: The desired perceived spatial location of the audio signal; the level of the audio signal (possibly time-varying); and / or The spectrum of the audio signal (possibly time-varying).

[0169] {^s i Examples of} include, but are not limited to: · The location of the loudspeakers in the listening space; ·Loudspeaker frequency response; ·Loudspeaker playback level limits; Parameters of dynamics processing algorithms within the speaker, such as limiter gain; ·Measurements or estimates of the acoustic transmission from each loudspeaker to the other loudspeakers; a measure of echo canceller performance for said loudspeaker; and / or -Synchronization of speakers relative to each other.

[0170] Examples of {^e} include, but are not limited to, the following: · the position of one or more listeners or speakers within the reproduction space; ·Measurements or estimates of the acoustic transmission from each loudspeaker to the listening position; ·Measurements or estimates of acoustic transmission from a talker to a set of loudspeakers; the location of any other landmarks within the playback space; and / or A measurement or estimate of the acoustic transmission from each loudspeaker to some other landmark in the playback space.

[0171] Using the new cost function defined in Equation 28, an optimal set of activations can be found through minimization with respect to g and possible a posteriori regularization as specified previously in Equations 28a and 28b.

[0172] Figure 17 is a flow diagram outlining one example of a method that may be performed by an apparatus or system such as that shown in Figure 1. The blocks of method 1700, as well as other methods described herein, are not necessarily performed in the order shown. Moreover, such methods may include more or fewer blocks than shown and / or described. The blocks of method 1700 may be performed by one or more devices that may be (or may include) a control system, such as control system 160 shown in Figure 1.

[0173] In this implementation, block 1705 involves receiving audio data by a control system via an interface system. In this example, the audio data includes one or more audio signals and associated spatial data. According to this implementation, the spatial data indicates an intended perceived spatial location corresponding to the audio signal. In some cases, the intended perceived spatial location may be explicit, e.g., as indicated by position metadata, such as Dolby Atmos position metadata. In other cases, the intended perceived spatial location may be implicit, e.g., the intended perceived spatial location may be an assumed location associated with a channel according to Dolby 5.1, Dolby 7.1, or another channel-based audio format. In some examples, block 1705 involves a rendering module of the control system receiving the audio data via an interface system.

[0174] According to this example, block 1710 involves rendering, by the control system, the audio data for playback through a set of loudspeakers in the environment to generate rendered audio signals. In this example, rendering each of the one or more audio signals included in the audio data involves determining a relative activation of the set of loudspeakers in the environment by optimizing a cost function. According to this example, the cost is a function of a model of the perceived spatial location of the audio signal when played on the set of loudspeakers in the environment. In this example, the cost is also a function of a measure of the proximity of the intended perceived spatial location of the audio signal to the location of each loudspeaker in the set of loudspeakers. In this implementation, the cost is also a function of one or more additional dynamically configurable functions. In this example, the dynamically configurable function is based on one or more of the following: the proximity of the loudspeaker to one or more listeners; the proximity of the loudspeaker to an attraction location (attraction is a factor that prioritizes relatively higher loudspeaker activation in proximity closer to an attraction location); the proximity of the loudspeaker to a repulsion location (repulsion is a factor that prioritizes relatively lower loudspeaker activation in proximity closer to a repulsion location); the capabilities of each loudspeaker relative to other loudspeakers in the environment; the synchronization of the loudspeaker to other loudspeakers; wake word performance; or echo cancellation performance.

[0175] In this example, block 1715 involves providing the rendered audio signals via an interface system to at least some loudspeakers of a set of loudspeakers in the environment.

[0176] According to some examples, the model of perceived spatial location may generate a binaural response corresponding to an audio object location at the left and right ear of a listener. Alternatively or additionally, the model of perceived spatial location may place the perceived spatial location of an audio signal reproduced from a set of loudspeakers at the centroid of the positions of the loudspeakers of said set weighted by the associated activation gains of the loudspeakers.

[0177] In some examples, the one or more additional dynamically configurable functions may be based at least in part on a level of the one or more audio signals, hi some instances, the one or more additional dynamically configurable functions may be based at least in part on a spectrum of the one or more audio signals.

[0178] Some examples of method 1700 involve receiving loudspeaker layout information. In some examples, the one or more additional dynamically configurable functions may be based at least in part on respective positions of loudspeakers within the environment.

[0179] Some examples of method 1700 involve receiving loudspeaker specification information. In some examples, the one or more additional dynamically configurable functions may be based at least in part on the capabilities of each loudspeaker, which may include one or more of frequency response, playback level limits, or parameters of one or more loudspeaker dynamics processing algorithms.

[0180] According to some examples, the one or more additional dynamically configurable functions may be based at least in part on measurements or estimates of acoustic transmission from each loudspeaker to the other loudspeakers. Alternatively or additionally, the one or more additional dynamically configurable functions may be based at least in part on listener or speaker positions of one or more people in the environment. Alternatively or additionally, the one or more additional dynamically configurable functions may be based at least in part on measurements or estimates of acoustic transmission from each loudspeaker to the listener or speaker positions. The estimates of acoustic transmission may be based at least in part on walls, furniture or other objects that may be between each loudspeaker and the listener or speaker positions, for example.

[0181] Alternatively or additionally, the one or more additional dynamically configurable functions may be based at least in part on object positions of one or more non-loudspeaker objects or landmarks in the environment, hi some such implementations, the one or more additional dynamically configurable functions may be based at least in part on measurements or estimates of acoustic transmission from each loudspeaker to the object or landmark positions.

[0182] By using one or more well-defined additional cost terms to implement flexible rendering, a large number of new and useful behaviors can be achieved. All the example behaviors listed below are cast in terms of penalizing certain loudspeakers under certain conditions that are considered undesirable. The end result is that these loudspeakers are activated less in the spatial rendering of the set of audio signals. In many of these cases, one might consider simply attenuating the undesirable loudspeakers, independent of any modifications to the spatial rendering, but such a strategy may significantly degrade the overall balance of the audio content. For example, certain components of the mix may become completely inaudible. On the other hand, in the disclosed embodiments, integrating these penalties into the core optimization of the rendering allows the rendering to adapt to perform the best possible spatial rendering with the speakers with fewer remaining penalties. This is a much more elegant, adaptable and effective solution.

[0183] Example use cases include, but are not limited to: ●Provides a more balanced spatial presentation around the listening area. o It has been found that spatial audio is best presented across loudspeakers that are approximately the same distance from the intended listening area. A cost may be constructed such that loudspeakers that are significantly closer or farther away than the average distance of the loudspeakers to the listening area are penalized, thus reducing their activation.

[0184] ●Move the audio away from or towards the listener or speaker. o When a user of the system is attempting to speak to a smart voice assistant of or associated with the system, it may be beneficial to generate costs that penalize loudspeakers that are closer to the speaker, in this way allowing these loudspeakers to be activated less and their associated microphones to hear the speaker better. o To provide a more intimate experience for a single listener that minimizes playback levels for others in the listening space, speakers far from the listener's position may be heavily penalized, so that only the speakers closest to the listener are activated most significantly.

[0185] ● Move the audio away from or towards a landmark, zone, or area. Certain locations in the vicinity of the listening space may be considered sensitive, such as a baby's room, a baby's bed, offices, reading areas, study areas, etc. Alternatively, for the same (or similar) cases as above, the system of speakers may generate a measurement of the acoustic transmission from each speaker to the baby's room, especially if one of the speakers (with an attached or associated microphone) is in the baby's room itself. In this case, rather than using the physical proximity of the speakers to the baby's room, a cost may be constructed that penalizes the use of speakers with high measured acoustic transmission into that room. And / or

[0186] ●Optimal use of speaker capabilities. ○ The capabilities of different loudspeakers can vary significantly. For example, one popular smart speaker contains only a single 1.6 inch full-range driver with limited low frequency capabilities, while another smart speaker contains a much more capable 3 inch woofer. These capabilities are generally reflected in the frequency response of the speaker, and thus the set of responses associated with the speaker may be utilized in cost terms. At certain frequencies, speakers with less capabilities compared to other speakers, as measured by their frequency response, may be penalized and therefore activated to a lesser extent. In some implementations, such frequency response values ​​may be stored in the smart loudspeaker and then reported to a computational unit responsible for optimizing the flexible rendering.

[0187] Many speakers include two or more drivers, each responsible for reproducing a different frequency range. For example, one popular smart speaker is a two-way design that includes a woofer for low frequencies and a tweeter for high frequencies. Typically, such speakers include crossover circuitry to split the full-range playback audio signal into appropriate frequency ranges and send them to the respective drivers. Alternatively, such speakers may provide the flexible renderer with playback access to each individual driver, as well as information about the capabilities of each individual driver, such as frequency response. By applying cost terms such as those described immediately above, in some examples, the flexible renderer may automatically construct a crossover between the two drivers based on their relative capabilities at different frequencies.

[0188] o The above-mentioned exemplary uses of frequency response focus on the inherent capabilities of the speaker, but may not accurately reflect the capabilities of the speaker when placed in a listening environment. In some cases, the measured frequency response of the speaker at the intended listening position may be available through some calibration procedure. Such measurements may be used instead of pre-calculated responses to better optimize the use of the speaker. For example, a speaker may be inherently very capable at certain frequencies, but because of its placement (e.g., behind a wall or furniture) may produce a very limited response at the intended listening position. A measurement capturing this response and feeding it into the appropriate cost terms may prevent significant activation of such speakers.

[0189] o Frequency response is only one aspect of a loudspeaker's reproduction capabilities. Many small loudspeakers begin to distort as the reproduction level increases, especially for low frequencies, and then reach their excursion limits. To reduce such distortion, many loudspeakers implement dynamics processing that constrains the reproduction level below some limiting thresholds that may be variable over frequency. If a speaker is close to or at these thresholds, but other speakers participating in the flexible rendering are not, it makes sense to reduce the signal level at the limiting speaker and divert this energy to other, less loaded speakers. Such behavior may be achieved automatically according to some embodiments by appropriately configuring the relevant cost terms. Such cost terms may include one or more of the following:

[0190] ■ Monitor the global playback volume in relation to a loudspeaker's limit threshold. For example, loudspeakers whose volume levels are closer to their limit threshold may be penalized more heavily. ■ Monitoring dynamic signal levels, possibly varying over frequency, in relation to a speaker limit threshold, also possibly varying over frequency. For example, loudspeakers whose monitored signal levels are closer to their limit threshold may be penalized more heavily. ■ Directly monitoring loudspeaker dynamics processing parameters, such as limiting gain. In some such examples, loudspeakers whose parameters indicate stronger limiting may be penalized more heavily. ■ The actual instantaneous voltage, current, and power delivered by the amplifier to the loudspeaker are monitored to determine whether the loudspeaker is operating in a linear range. For example, loudspeakers operating with less linearity may be penalized more.

[0191] o Smart speakers with integrated microphones and interactive voice assistants typically employ some type of echo cancellation to reduce the level of the audio signal played from the speaker that is picked up by the recording microphone. The greater this reduction, the more likely the speaker is to hear and understand the speaker in the space. If the echo canceller residual is consistently high, this may be an indication that the speaker is being driven into a non-linear region where the echo path becomes difficult to predict. In such cases, it may make sense to divert signal energy away from that speaker, and thus a cost term that takes echo canceller performance into account may be beneficial. Such a cost term may assign a higher cost to speakers whose associated echo cancellers are performing poorly.

[0192] o To achieve predictable localization when rendering spatial audio across multiple loudspeakers, it is generally required that the playback across a set of loudspeakers be reasonably synchronized over time. While this is a given for wired loudspeakers, with a large number of wireless loudspeakers, synchronization can be difficult and the end result variable. In such cases, each loudspeaker may be able to report its relative degree of synchronization with the target, and this degree may then enter into the synchronization cost term. In some such instances, loudspeakers with a lower degree of synchronization may be penalized more heavily and thus excluded from rendering. Furthermore, tight synchronization may not be required for certain types of audio signals, for example components of an audio mix that are intended to be diffuse or non-directional. In some implementations, components may be tagged as such using metadata, and the synchronization cost term may be modified so that the penalty is reduced.

[0193] Further examples of embodiments are now described. Similar to the proximity costs defined in Equations 25a and 25b, the new cost function term C j (g,{{^o},{^s i},{^e}} j ) as a weighted sum of the squared absolute values ​​of the speaker activations, e.g.

number

number

[0194] Equation 20b corresponds to equation 3 above.

[0195] Combining Equations 20a and 20b with a matrix-quadratic version of the CMAP and FV cost functions given in Equation 15 gives a potentially useful implementation of the general extended cost function (for some embodiments) given in Equation 19.

number

[0196] Equation 21 corresponds to the above equation 2. Thus, the above discussion explains the derivation of equation 2 as well as the derivation of equation 21.

[0197] With this definition of the new cost function term, the overall cost function remains in matrix quadratic form, and the optimal set of activations g opt can be found through differentiation of Equation 21 to give:

number

[0198] Weight term w ij , respectively, with respect to a given consecutive penalty value p ij =p ij ({{^o},{^s i},{^e}} j ) as a function of the penalty value. In one exemplary embodiment, this penalty value is the distance from the (rendered) object to the loudspeaker being considered. In another exemplary embodiment, this penalty value represents the inability of a given loudspeaker to reproduce some frequencies. Based on this penalty value, the weighting term w ij can be parameterized as follows:

number

[0199] Here, α j denotes a prefactor (taking into account the global strength of the weight terms), and τ j represents the penalty threshold (around or above which the weight term becomes significant), and f j (x) represents a monotonically increasing function. For example, f j (x)=x βj Then the weight terms have the form:

number

[0200] If all loudspeakers are penalized, it is often convenient to subtract a minimum penalty from all weight terms in post-processing to ensure that at least one speaker is not penalized.

number

[0201] As mentioned above, there are many possible use cases that can be realized using the new cost function terms described herein (and similar new cost function terms used according to other embodiments). More specific details are now provided using three examples: moving audio towards a listener or speaker, moving audio away from a listener or speaker, and moving audio away from a landmark.

[0202] In a first example, what is referred to herein as a "gravitational force" is used to attract audio towards a location, which in some examples may be a listener or speaker location, a landmark location, a furniture location, etc. This location may be referred to herein as an "attractive location" or an "attractor location." According to this example, a weight w ij is the fixed attractor position of the i-th speaker → l j A continuous penalty value p given by the distance from ij and a threshold τ given by the maximum of these distances across all speakers. j This takes the form of Equation 17:

number

[0203] To illustrate the use case of "pulling" audio towards a listener or speaker, we specifically j = 20, β j = 3, → l j Set α to the vector corresponding to the 180 degree listener / speaker position (bottom center of the plot). j , β j , and → l j These values ​​of are just examples. In some implementations, α j may range from 1 to 100, and β jmay range from 1 to 25. FIG. 18 is a graph of speaker activations in an example embodiment. In this example, FIG. 18 shows speaker activations 1505b, 1510b, 1515b, 1520b, and 1525b, which are applied to the cost function for the same speaker positions from FIG. 15 and FIG. 16. ij FIG. 19 is a graph of object rendering positions in an example embodiment. In this example, FIG. 19 shows the corresponding ideal object positions 1630b for a number of possible object angles and the corresponding actual rendering positions 1635b for those objects connected to the ideal object positions 1630b by dotted lines 1640b. Fixed positions of the actual rendering positions 1635b → l j The diagonal orientation towards indicates the influence of attractor weighting on the optimal solution to the cost function.

[0204] In the second and third examples, a "repulsive force" is used to "push" audio away from a location. The location may be a person's location (e.g., a listener's location, a speaker's location, etc.) or another location, e.g., a landmark location, a furniture location, etc. In some examples, the repulsive force may be used to push audio away from an area or zone of a listening environment, such as an office area, a reading area, a bed or sleeping area (e.g., a baby's bed or sleeping area). According to some such examples, a particular location may be used to represent a zone or area. For example, a location representing a baby's bed may be an estimated location of the baby's head, an estimated sound source location corresponding to the baby, etc. Said locations may be referred to herein as "repulsive force locations" or "repulsive locations." As used herein, a "repulsive force" is a factor that favors relatively lower loudspeaker activation in closer proximity to the repulsive force location. According to this example, a fixed repulsive location, similar to the attractive force in Equations 26a and 26b, → l j Regarding p ijand τ j Define.

number

[0205] To illustrate the use case of pushing audio away from a listener or speaker, one example specifically j = 5, β j = 2, → l j may be set to the vector corresponding to a 180 degree listener / speaker position (bottom center of the plot). j , β j , and → l j These values ​​of are merely examples. As noted above, in some examples, j may range from 1 to 100, and β j may range from 1 to 25. FIG. 20 is a graph of speaker activations in an example embodiment. In this example, FIG. 20 shows speaker activations 1505c, 1510c, 1515c, 1520c, and 1525c, which are applied to the cost function for the same speaker positions as in the previous figure. ij FIG. 21 is a graph of object rendering positions in an example embodiment. In this example, FIG. 21 shows ideal object positions 1630c for a number of possible object angles and the corresponding actual rendering positions 1635c for those objects connected to the ideal object positions 1630c by dotted lines 1640c. Fixed positions of actual rendering positions 1635c → l j The diagonal orientation away from indicates the influence of repulsive weighting on the optimal solution to the cost function.

[0206] A third example use case is to "push" audio away from an acoustically sensitive landmark, such as a door to a sleeping baby's room. Similar to the last example, → l jto the vector corresponding to the 180 degree door position (bottom center of the plot). To achieve a stronger repulsion force and tilt the sound field completely forward of the primary listening space, α j = 20, β j =5. FIG. 22 is a graph of speaker activations in an example embodiment. Again in this example, FIG. 22 shows speaker activations 1505d, 1510d, 1515d, 1520d, and 1525d, which include optimal solutions for the same set of speaker positions with stronger repulsion added. FIG. 23 is a graph of object rendering positions in an example embodiment. Again in this example, FIG. 23 shows ideal object positions 1630d for a number of possible object angles and corresponding actual rendering positions 1635d for those objects connected to the ideal object positions 1630d by dotted lines 1640d. The diagonal orientation of the actual rendering positions 1635d shows the effect of stronger repulsion weighting on the optimal solution to the cost function.

[0207] Aspects of some disclosed implementations include a system or device configured (e.g., programmed) to perform one or more of the disclosed methods, and a tangible computer-readable medium (e.g., disk) storing code for implementing one or more of the disclosed methods, or steps thereof. For example, a system may be or include a programmable general-purpose processor, digital signal processor, or microprocessor programmed with software or firmware and / or otherwise configured to perform any of a variety of operations on data, including one or more of the disclosed methods, or steps thereof. Such a general-purpose processor may be or include a computer system that includes an input device, a memory, and a processing subsystem programmed (and / or otherwise configured) to perform one or more of the disclosed methods (or steps thereof) in response to asserted data.

[0208] Some disclosed embodiments are implemented as a configurable (e.g., programmable) digital signal processor (DSP) configured (e.g., programmed and otherwise configured) to perform necessary processing on the audio signal(s), including performing one or more of the disclosed methods. Alternatively, some embodiments (or elements thereof) may be implemented as a general-purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor, which may include an input device and memory) that is programmed with software or firmware and / or otherwise configured to perform any of a variety of operations, including one or more of the disclosed methods or steps thereof. Alternatively, elements of some disclosed embodiments are implemented as a general-purpose processor or DSP configured (e.g., programmed) to perform one or more of the disclosed methods or steps thereof, the system also including other elements (e.g., one or more loudspeakers and / or one or more microphones). A general-purpose processor configured to perform one or more of the disclosed methods or steps thereof is typically coupled to an input device (e.g., a mouse and / or a keyboard), a memory, and a display device.

[0209] Another aspect of some disclosed implementations is a computer-readable medium (e.g., a disk or other tangible storage medium) storing code (e.g., executable code to perform) for performing any embodiment of one or more of the disclosed methods or steps thereof.

[0210] While specific embodiments and applications are described herein, it will be apparent to those skilled in the art that many variations to the embodiments and applications described herein are possible without departing from the scope of the material described and claimed herein. Although certain embodiments have been shown and described, it will be understood that the disclosure is not limited to the specific embodiments described and shown, or to the specific methods described.

Claims

1. 13. A method of audio processing comprising: receiving, by a control system and via an interface system, audio data including one or more audio signals and associated spatial data, the spatial data indicating an intended perceived spatial location corresponding to an audio signal of the one or more audio signals; receiving, by the control system via the interface system, listener position data indicative of listener positions corresponding to persons in an audio environment; receiving, by the control system via the interface system, loudspeaker position data indicative of a position of each loudspeaker of a plurality of loudspeakers in the audio environment; receiving, by the control system via the interface system, loudspeaker orientation data indicating (a) a direction of maximum sound radiation for each loudspeaker of the plurality of loudspeakers in the audio environment and (b) a loudspeaker orientation angle between the listener position relative to a position of a corresponding loudspeaker; rendering, by the control system, the audio data for playback through at least a subset of the plurality of loudspeakers in the audio environment to generate a rendered audio signal, the rendering being based at least in part on the spatial data, the listener position data, the loudspeaker position data, and the loudspeaker orientation data, the rendering involving applying a loudspeaker orientation factor that tends to reduce relative activation of loudspeakers based at least in part on increased loudspeaker orientation angles; and providing, via the interface system, the rendered audio signals to at least the subset of loudspeakers of the plurality of loudspeakers in the audio environment. Audio processing methods.

2. The audio processing method of claim 1 , further comprising estimating loudspeaker importance metrics for at least the subset of the loudspeakers.

3. 3. The audio processing method of claim 2, wherein the loudspeaker importance metric corresponds to a loudspeaker's importance for rendering an audio signal at an intended perceived spatial location of the audio signal.

4. The audio processing method of claim 2 , wherein the rendering for each loudspeaker is based at least in part on the loudspeaker importance metric.

5. The audio processing method of claim 2 , wherein the rendering for each loudspeaker involves modifying an effect of the loudspeaker orientation factor based at least in part on the loudspeaker importance metric.

6. 3. The audio processing method of claim 2, wherein the rendering for each loudspeaker involves reducing an influence of the loudspeaker orientation factor based at least in part on an increased loudspeaker importance metric.

7. 2. The audio processing method of claim 1, wherein the loudspeaker orientation angle for a particular loudspeaker is the angle between (a) a direction of maximum sound radiation for the particular loudspeaker and (b) a line between a position of the particular loudspeaker and the listener position.

8. 2. The audio processing method of claim 1, further comprising the step of determining whether a loudspeaker orientation angle is greater than or equal to a threshold loudspeaker orientation angle, the audio processing method involving applying the loudspeaker orientation factor only if the loudspeaker orientation angle is greater than or equal to the threshold loudspeaker orientation angle.

9. 9. The audio processing method of claim 8, wherein the loudspeaker importance metric is based at least in part on a distance between a qualified loudspeaker and a line between (a) a first loudspeaker that is closest in clockwise angular distance from the qualified loudspeaker and (b) a second loudspeaker that is closest in counterclockwise angular distance from the qualified loudspeaker, a qualified loudspeaker being a loudspeaker that has a loudspeaker orientation angle that is greater than or equal to the threshold loudspeaker orientation angle.

10. 10. The audio processing method of claim 9, wherein the first loudspeaker and the second loudspeaker are disqualified loudspeakers having loudspeaker orientation angles less than the threshold loudspeaker orientation angle.

11. The rendering may include: a model of the perceived spatial location of audio signals of the one or more audio signals when played through the subset of loudspeakers in the audio environment; a measure of the proximity of an intended perceived spatial location of the audio signal to a location of each loudspeaker of the subset of loudspeakers; and One or more additional dynamically configurable functions determining a relative activation for at least said subset of said loudspeakers by optimizing a cost that is a function of at least one of the one or more additional dynamically configurable functions is based at least in part on the loudspeaker orientation factor; 2. The audio processing method of claim 1.

12. 12. The audio processing method of claim 11, wherein at least one of the one or more additional dynamically configurable functions is based at least in part on the loudspeaker importance metric.

13. 12. The audio processing method of claim 11, wherein at least one of the one or more additional dynamically configurable functions is based at least in part on measurements or estimates of acoustic transmission from each loudspeaker in the audio environment to other loudspeakers in the audio environment.

14. 2. The audio processing method of claim 1, wherein the intended perceived spatial location corresponds to channel or location metadata of a channel-based audio format.

15. Apparatus arranged to carry out an audio processing method according to any one of claims 1 to 14.

16. A system arranged to carry out an audio processing method according to any one of claims 1 to 14.

17. One or more non-transitory media having software stored thereon, the software including instructions for controlling one or more devices to perform the audio processing method of any one of claims 1 to 14.

Citation Information

Patent Citations

  • Adaptive panner of audio objects

    EP3223542A2

  • Sound field controller and sound system

    JP2005057545A

  • An audio processor and a method considering acoustic obstacles and providing loudspeaker signals

    WO2020030304A1

  • Compensating for binaural loudspeaker directivity

    WO2020081103A1

  • Rendering audio over multiple speakers with multiple activation criteria

    WO2021021682A1