Spatial audio frequency domain multiplexing for optimal listening positions of multiple listeners
Patent Information
- Application Number
- CN202180088317.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-08-24
- Filing Date
- 2021-12-02
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2041-12-02
Smart Images

Figure CN116830603B_ABST
Abstract
Description
[0001] Cross-references to related applications
[0002] This application claims priority to U.S. Provisional Application No. 63 / 120,963, filed December 3, 2020, and U.S. Provisional Application No. 63 / 260,529, filed August 24, 2021, each of which is incorporated herein by reference in its entirety. Technical Field
[0003] This disclosure relates to systems and methods for rendering audio for playback by some or all of a set of speakers (e.g., each active speaker). Background Technology
[0004] Audio devices are widely deployed in many homes, vehicles, and other environments. While existing systems and methods for controlling audio devices offer benefits, improvements in systems and methods will continue to be desired.
[0005] Symbols and terms
[0006] Throughout this disclosure, including in the claims, the terms "speaker," "loudspeaker," and "audio reproduction transducer" are used synonymously to refer to any sound-emitting transducer (or a group of transducers). A typical set of headphones includes two speakers. A speaker can be implemented to include multiple transducers (e.g., a woofer and a tweeter), which can be driven by a single common speaker feed or multiple speaker feeds. In some examples, the speaker feeds can undergo different processing in different circuit branches coupled to different transducers.
[0007] Throughout this disclosure, including in the claims, the expression “to” perform an operation on a signal or data (e.g., to filter, scale, transform, or apply gain to a signal or data) is used broadly to indicate performing an operation directly on a signal or data or on a processed version of a signal or data (e.g., a version of a signal that has undergone preliminary filtering or preprocessing before the operation is performed on it).
[0008] Throughout this disclosure, including in the claims, the term "system" is used broadly to refer to a device, system, or subsystem. For example, a subsystem implementing a decoder may be referred to as a decoder system, and a system including such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, wherein the subsystem generates M inputs, and the other XM inputs are received from an external source) may also be referred to as a decoder system.
[0009] Throughout this disclosure, including in the claims, the term "processor" is used broadly to refer to a system or device that is programmable or otherwise configurable (e.g., with software or firmware) for performing operations on data (e.g., audio or video or other image data). Examples of processors include field-programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors programmed and / or otherwise configured to perform pipelined processing of audio or other sound data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets.
[0010] As used herein, a “smart device” is an electronic device that can operate interactively and / or autonomously to some extent, and is typically configured to communicate with one or more other devices (or networks) via various wireless protocols such as Bluetooth, Zigbee, Near Field Communication, Wi-Fi, Li-Fi, 3G, 4G, 5G, etc. Several notable types of smart devices include smartphones, smart cars, smart thermostats, smart doorbells, smart locks, smart refrigerators, phablets and tablets, smartwatches, smart bracelets, smart keychains, and smart audio devices. The term “smart device” can also refer to devices that exhibit properties of ubiquitous computing, such as artificial intelligence.
[0011] In this document, the term "smart audio device" is used to refer to a smart device, which is either a single-purpose audio device or a multi-purpose audio device (e.g., a smart speaker or other audio device that implements at least some aspects of a virtual assistant function). A single-purpose audio device is a device that includes or is coupled to at least one microphone (and optionally includes or is coupled to at least one speaker and / or at least one camera) and is largely or primarily designed to perform a single purpose (e.g., a television (TV)). For example, although a TV can generally play (and is considered capable of playing) audio from program material, in most instances, modern TVs run some kind of operating system on which applications (including TV-watching applications) run natively. In this sense, a single-purpose audio device with a speaker and a microphone is typically configured to run local applications and / or services to directly use the speaker and microphone. Some single-purpose audio devices may be configured to be combined to enable audio playback in a specific zone or user-configured zone.
[0012] A common type of multipurpose audio device is an audio device that implements at least some aspects of virtual assistant functionality (e.g., a smart speaker), although other aspects of the virtual assistant functionality may be implemented by one or more other devices, such as one or more servers, and the multipurpose audio device is configured to communicate with said one or more servers. Such a multipurpose audio device may be referred to herein as a "virtual assistant." A virtual assistant is a device that includes or is coupled to at least one microphone (and optionally includes or is coupled to at least one speaker and / or at least one camera) (e.g., a smart speaker or a voice assistant integrated device). In some examples, a virtual assistant may provide the ability to use multiple devices (different from the virtual assistant) for applications that are, in a sense, cloud-enabled or otherwise not fully implemented in or on top of the virtual assistant itself. In other words, at least some aspects of the virtual assistant functionality (e.g., speech recognition functionality) may (at least partially) be implemented by one or more servers or other devices, and the virtual assistant may communicate with said one or more servers or other devices via a network (such as the Internet). Virtual assistants may sometimes work together, for example, in a discrete and conditionally defined manner. For example, two or more virtual assistants may work together in the sense that one of them (e.g., the virtual assistant most certain that it has heard the wake word) responds to the wake word. In some implementations, the connected virtual assistants can form a constellation, which can be managed by a main application, which may be (or implement) the virtual assistants.
[0013] As used herein, the terms "program stream" and "content stream" refer to a collection of one or more audio signals, and in some instances, a collection of video signals, at least a portion of which is intended to be heard together. Examples include music anthologies, movie soundtracks, movies, television programs, audio portions of television programs, podcasts, live voice calls, synthesized voice responses from intelligent assistants, and so on. In some instances, a content stream may include multiple versions of at least a portion of an audio signal, such as the same dialogue in more than one language. In such instances, only one version of the audio data or a portion thereof is intended to be reproduced at a time (e.g., a version corresponding to a single language). Summary of the Invention
[0014] At least some aspects of this disclosure can be implemented via methods. Some such methods may involve audio data processing. For example, some methods may involve receiving audio data by a control system configured to implement multiple renderers. Some such methods may involve receiving listening configuration data for multiple listening configurations by the control system. Each of the multiple listening configurations may correspond to a listening position and listening orientation in an audio environment. Some such methods may involve rendering the audio data by each of the multiple renderers according to the listening configuration data to obtain a renderer-specific amplifier feed signal set for the corresponding listening configuration. Each renderer may be configured to render the audio data for a different listening configuration.
[0015] Some such methods may involve the control system decomposing each renderer-specific loudspeaker feed signal set into renderer-specific frequency band sets for each renderer. Some such methods may involve the control system combining the renderer-specific frequency band sets of each renderer to produce a loudspeaker feed signal output set. Some such methods may involve the control system outputting the loudspeaker feed signal output set to multiple loudspeakers.
[0016] In some examples, decomposing each renderer-specific loudspeaker feed signal set into each renderer-specific frequency band set may involve: analyzing the renderer-specific loudspeaker feed signal set by an analysis filter bank associated with each renderer to generate a global frequency band set; and selecting a frequency band subset of the global frequency band set to generate the renderer-specific frequency band set. The frequency band subset of the global frequency band set can be selected such that when the renderer-specific frequency band sets of all renderers in the plurality of renderers are combined, each frequency band in the global frequency band set is represented only once in the loudspeaker feed signal output set.
[0017] Combining the renderer-specific set of frequency bands can involve synthesizing the amplifier feed signal output set in the time domain using a synthesis filter bank. In some examples, the analysis filter bank may be a short-time discrete Fourier transform (STDFT) filter bank, a hybrid composite quadrature mirror (HCQMF) filter bank, or a quadrature mirror (QMF) filter bank.
[0018] In some examples, each set in a renderer-specific frequency band set can be uniquely associated with one of the plurality of renderers and uniquely associated with one of the plurality of listening configurations. In some implementations, each listening configuration can correspond to a person's listening position and listening orientation. In some such examples, the listening position can correspond to the person's head position, and the listening orientation can correspond to the person's head orientation.
[0019] According to some examples, the audio data may be or may include spatial channel-based audio data and / or spatial object-based audio data. In some instances, the audio data may have one of the following formats: stereo, Dolby 3.1.2, Dolby 5.1, Dolby 5.1.2, Dolby 7.1, Dolby 7.1.2, Dolby 7.1.4, Dolby 9.1, Dolby 9.1.6, or Dolby Atmos audio formats. In some instances, the rendering may involve performing double-balanced amplitude shift in the time domain or performing crosstalk cancellation in the frequency domain.
[0020] Some methods may involve: receiving audio data by a control system; and receiving listening configuration data for multiple listening configurations by the control system. For example, each listening configuration may correspond to a listening location and listening orientation. Some such methods may involve analyzing the audio data using an analysis filter bank implemented via the control system to generate a global frequency band set corresponding to the audio data. Some such methods may involve the control system selecting a subset of the global frequency band set for each of a plurality of renderers implemented by the control system to generate a renderer-specific frequency band set for each renderer.
[0021] Some such methods may involve each of the plurality of renderers rendering a renderer-specific set of frequency bands according to the listening configuration data to obtain a renderer-specific set of loudspeaker feed signals for a corresponding listening configuration. In some such examples, each renderer may be configured to render frequency bands in the renderer-specific set of frequency bands for different listening configurations. Some such methods may involve the control system combining the renderer-specific set of loudspeaker feed signals from each of the plurality of renderers to produce a set of loudspeaker feed signal outputs. Some such methods may involve the control system outputting the set of loudspeaker feed signal outputs to multiple loudspeakers in an audio environment.
[0022] Some such methods may involve transforming the loudspeaker feed signal output set from the frequency domain to the time domain using a synthesis filter bank. In some such examples, the analysis filter bank may be a short-time discrete Fourier transform (STDFT) filter bank, a hybrid composite quadrature mirror (HCQMF) filter bank, or a quadrature mirror (QMF) filter bank.
[0023] In some examples, each renderer-specific amplifier feed signal set may be uniquely associated with one of the plurality of renderers. In some examples, each renderer-specific amplifier feed signal set may be uniquely associated with one of the plurality of listening configurations. According to some examples, the listening configuration may be or may include the listening position and / or listening orientation of a person in the audio environment. In some instances, the listening position may correspond to the person's head position. In some examples, the listening orientation may correspond to the person's head orientation.
[0024] In some embodiments, the listening position and the listening orientation may be relative to an audio environment coordinate system. In some embodiments, the listening position and the listening orientation may be relative to a coordinate system corresponding to a person within the audio environment (e.g., corresponding to the position and orientation of the person's head). In some instances, the listening position may be relative to the position of one or more loudspeakers within the audio environment.
[0025] According to some implementations, the listening configuration data may correspond to sensor data obtained from one or more sensors in the audio environment. In some examples, the sensors may be or may include cameras, motion sensors, and / or microphones.
[0026] According to some examples, the audio data may be or may include spatial channel-based audio data and / or spatial object-based audio data. In some instances, the audio data may have one of the following formats: stereo, Dolby 3.1.2, Dolby 5.1, Dolby 5.1.2, Dolby 7.1, Dolby 7.1.2, Dolby 7.1.4, Dolby 9.1, Dolby 9.1.6, or Dolby Atmos audio formats. In some examples, combining the amplifier feed signal set may involve multiplexing each of the renderer-specific amplifier feed signal sets.
[0027] In some instances, the rendering may involve performing a double-balanced amplitude shift in the time domain or crosstalk cancellation in the frequency domain.
[0028] In some examples, the rendering may involve generating multiple data structures. For example, each data structure may include a set of renderer-specific speaker activations for a corresponding listening configuration and corresponding to each of a plurality of points in two-dimensional or three-dimensional space. According to some such examples, the combination may involve combining the multiple data structures into a single data structure.
[0029] Some implementations may involve a method for rendering audio data in a vehicle. Some such methods may involve: receiving audio data by a control system; and receiving sensor signals indicating the presence of multiple people in the vehicle by the control system. Some such methods may involve the control system estimating multiple listening configurations relative to multiple loudspeakers in the vehicle, at least in part based on the sensor signals. For example, each listening configuration may correspond to the listening position and listening orientation of one of the multiple people.
[0030] Some such methods may involve the control system rendering received audio data for each of the plurality of listening configurations to generate a set of loudspeaker feed signals. Some such methods may involve the control system providing the set of loudspeaker feed signals to the plurality of loudspeakers.
[0031] In some examples, the rendering of the audio data may be performed by multiple renderers. In some instances, each of the multiple renderers may be configured to render the audio data for different listening configurations to obtain a renderer-specific loudspeaker feed signal set. In some such examples, the method may involve the control system decomposing each renderer-specific loudspeaker feed signal set into a renderer-specific frequency band set for each renderer. Some such methods may involve the control system combining the renderer-specific frequency band sets of each renderer to produce a loudspeaker feed signal output set. Some such methods may involve the control system outputting the loudspeaker feed signal output set.
[0032] In some examples, decomposing the renderer-specific loudspeaker feed signal set into the renderer-specific frequency band set may involve analyzing the renderer-specific loudspeaker feed signal set by an analysis filter bank associated with each renderer to produce a global frequency band set. Some such methods may involve selecting a subset of the global frequency band set to produce the renderer-specific frequency band set. In some examples, a subset of the global frequency band set may be selected such that when the renderer-specific frequency bands of each of the plurality of renderers are combined, each frequency band in the global frequency band set is represented only once in the loudspeaker feed signal output set.
[0033] According to some examples, combining the multiple renderer-specific frequency bands may involve synthesizing the amplifier feed signal output set in the time domain by a synthesis filter bank. In some examples, the analysis filter bank may be a short-time discrete Fourier transform (STDFT) filter bank, a hybrid composite quadrature mirror (HCQMF) filter bank, or a quadrature mirror (QMF) filter bank.
[0034] In some examples, each set in the renderer-specific frequency band set may be uniquely associated with one of the plurality of renderers. In some examples, each set in the renderer-specific frequency band set may be uniquely associated with one of the plurality of listening configurations. According to some examples, the rendering may involve performing double-balanced amplitude shift in the time domain or crosstalk cancellation in the frequency domain. In some implementations, combining renderer-specific frequency band sets may involve multiplexing the renderer-specific frequency band sets.
[0035] According to some implementations, the rendering of the audio data can be performed by multiple renderers. In some such examples, each renderer can be configured to render the audio data for different listening configurations among the multiple listening configurations. According to some such examples, the method may involve analyzing the received audio by an analysis filter bank implemented by the control system to generate a global frequency band set of the received audio data. Some such methods may involve the control system selecting a subset of the global frequency band set for each of the multiple renderers to generate a renderer-specific frequency band set for each renderer. Some such methods may involve each of the multiple renderers rendering the renderer-specific frequency band set to obtain a loudspeaker feed signal set for a corresponding listening configuration. Some such methods may involve combining the loudspeaker feed signal sets from each renderer to generate a loudspeaker feed signal output set. Some such methods may involve outputting the loudspeaker feed signal output set.
[0036] According to some examples, combining the loudspeaker feed signal set may involve synthesizing the loudspeaker feed signal output set in the time domain using a synthesis filter bank. In some examples, the synthesis filter bank may be a short-time discrete Fourier transform (STDFT), hybrid composite quadrature mirror (HCQMF), or quadrature mirror (QMF) filter bank.
[0037] In some instances, each renderer-specific frequency band set can be uniquely associated with a single renderer. In some examples, each renderer-specific frequency band set can be uniquely associated with a listening configuration. According to some examples, listening position can correspond to head position. In some examples, listening orientation can correspond to head orientation.
[0038] According to some examples, the audio data may be or may include spatial channel-based audio data and / or spatial object-based audio data. In some instances, the audio data may have one of the following formats: stereo, Dolby 3.1.2, Dolby 5.1, Dolby 5.1.2, Dolby 7.1, Dolby 7.1.2, Dolby 7.1.4, Dolby 9.1, Dolby 9.1.6, or Dolby Atmos audio formats. In some instances, the rendering may involve performing double-balanced amplitude shift in the time domain or performing crosstalk cancellation in the frequency domain. In some examples, combining the amplifier feed signal sets from each renderer may involve multiplexing the amplifier feed signal sets from each renderer.
[0039] According to some implementations, the sensor signals may include signals from one or more seat sensors. For example, the seat sensors may include one or more cameras, one or more seat belt sensors, one or more headrest sensors, one or more seat back sensors, one or more seat bottom sensors, and / or one or more elbow rest sensors.
[0040] Some methods may also involve selecting a rendering mode among multiple rendering modes. In some examples, each of the multiple rendering modes may be based on a corresponding listening configuration among multiple listening configurations.
[0041] In some examples, at least one listening configuration may be associated with a person's identity. In some such examples, at least one such listening configuration may be stored in the vehicle's memory.
[0042] According to some examples, the rendering may involve generating a set of coefficients corresponding to the listening configuration for each renderer. In some such examples, these coefficients may be used for the rendering. In some examples, the coefficients may be panner coefficients.
[0043] Some or all of the operations, functions, and / or methods described herein can be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Therefore, some innovative aspects of the subject matter described in this disclosure can be implemented in non-transitory media on which software is stored.
[0044] At least some aspects of this disclosure can be implemented via apparatus. For example, one or more devices may be able to perform at least partially the methods disclosed herein. In some embodiments, the apparatus may include an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or combinations thereof. In some examples, the apparatus may be one of the audio devices referenced above. However, in some embodiments, the apparatus may be another type of device, such as a mobile device, laptop computer, server, vehicle, etc. For example, a vehicle control system may be configured to perform at least some of the disclosed methods. An audio device control system may be configured to perform at least some of the disclosed methods.
[0045] Details of one or more embodiments of the subject matter described in this specification are set forth in the following figures and description. Other features, aspects, and advantages will become apparent from the description, figures, and claims. Note that the relative dimensions in the following figures may not be drawn to scale. Attached Figure Description
[0046] Figure 1 This is a block diagram illustrating examples of components of an apparatus capable of implementing various aspects of the present disclosure.
[0047] Figure 2A A floor plan of the listening environment is depicted, which in this example is a living space.
[0048] Figure 2B It shows Figure 2A Examples of audio environments at different times.
[0049] Figure 2C Another example of an audio environment is shown.
[0050] Figure 3 An example box of one of the disclosed implementations is shown.
[0051] Figure 4 An example box of another disclosed implementation is shown.
[0052] Figure 5 It outlines what can be achieved by, for example Figures 1 to 4 A flowchart illustrating an example of a method performed by an apparatus or system, such as the one shown.
[0053] Figure 6A An example box of another disclosed implementation is shown.
[0054] Figure 6BThis is a graph indicating the points where the speaker is activated in the example embodiment.
[0055] Figure 6C It is a graph of trilinear interpolation between points indicating speaker activation, based on an example.
[0056] Figure 7 This is a flowchart outlining another example of a method that can be performed by an apparatus or system such as those disclosed herein.
[0057] Figure 8 An example of the interior of a vehicle according to one embodiment is shown.
[0058] Figure 9 An example box of another disclosed implementation is shown.
[0059] Figure 10 This is a flowchart outlining an example of a method that can be performed by an apparatus or system such as those disclosed herein.
[0060] Figure 11 An example of the geometric relationship between four audio devices in an environment is shown.
[0061] Figure 12 It shows the location Figure 11 An audio transmitter within an audio environment.
[0062] Figure 13 It shows the location Figure 11 An audio receiver within an audio environment.
[0063] Figure 14 It outlines what can be made by, for example Figure 1 The flowchart shows an example of a method performed by the control system of the device.
[0064] Figure 15 This is a flowchart outlining an example of a method for automatically estimating device location and orientation based on DOA data.
[0065] Figure 16 This is a flowchart outlining an example of a method for automatically estimating device location and orientation based on DOA and TOA data.
[0066] Figure 17 This is a flowchart outlining another example of a method for automatically estimating device location and orientation based on DOA and TOA data.
[0067] Figure 18A An example of an audio environment is shown.
[0068] Figure 18BAdditional examples of determining listener angle orientation data are shown.
[0069] Figure 18C Additional examples of determining listener angle orientation data are shown.
[0070] Figure 18D It is shown according to the reference Figure 18C The described method is an example of determining the appropriate rotation of the audio device coordinates.
[0071] Figure 19 An example of the geometric relationship between three audio devices in an environment is shown.
[0072] Figure 20 It shows Figure 19 Another example of the geometric relationship between three audio devices in the environment shown in the image.
[0073] Figure 21A It shows Figure 19 and Figure 20 The two triangles depicted in the image do not contain any corresponding audio devices or other environmental features.
[0074] Figure 21B An example of estimating the interior angles of a triangle formed by three audio devices is shown.
[0075] Figure 22 It outlines what can be made by, for example Figure 1 The flowchart shows an example of a method performed by the apparatus of the device shown.
[0076] Figure 23 An example is shown where each audio device in the environment is the vertex of multiple triangles.
[0077] Figure 24 An example of a portion of the forward alignment process is provided.
[0078] Figure 25 Examples of several estimates of the audio device position that have occurred during the forward alignment process are shown.
[0079] Figure 26 An example of a part of the reverse alignment process is provided.
[0080] Figure 27 An example of multiple audio device position estimations that have occurred during the reverse alignment process is shown.
[0081] Figure 28 A comparison between the estimated audio device location and the actual audio device location is shown.
[0082] Figure 29 It outlines what can be made by, for example Figure 1The flowchart shows another example of a method performed by the apparatus of the device shown in the diagram.
[0083] Figure 30 This is a flowchart outlining another example of a positioning method.
[0084] Figure 31 This is a flowchart outlining another example of a positioning method. Detailed Implementation
[0085] Figure 1 This is a block diagram illustrating examples of components of an apparatus capable of implementing various aspects of the present disclosure. According to some examples, apparatus 100 may be, or may include, a smart audio device configured to perform at least some of the methods disclosed herein. In other embodiments, apparatus 100 may be, or may include, another device configured to perform at least some of the methods disclosed herein, such as a laptop computer, cellular phone, tablet device, smart home hub, etc. In some such embodiments, apparatus 100 may be, or may include, a server. In some embodiments, apparatus 100 may be configured to implement a device that may be referred to herein as an "orchestration device" or "audio session manager."
[0086] In this example, device 100 includes an interface system 105 and a control system 110. In some embodiments, the interface system 105 may be configured to communicate with one or more devices executing or configured to execute a software application. Such a software application may be referred to herein as an "application" or simply an "app". In some embodiments, the interface system 105 may be configured to exchange control information and associated data related to the application. In some embodiments, the interface system 105 may be configured to communicate with one or more other devices in an audio environment. In some examples, the audio environment may be a home audio environment. In other examples, the audio environment may be another type of environment, such as an office environment, a vehicle environment, a park, or other outdoor environment. In some embodiments, the interface system 105 may be configured to exchange control information and associated data with audio devices in the audio environment. In some examples, the control information and associated data may relate to one or more applications with which device 100 is configured to communicate.
[0087] In some embodiments, the interface system 105 may be configured to receive an audio program stream. The audio program stream may include audio signals arranged to be reproduced by at least some speakers in the environment. The audio program stream may include spatial data such as channel data and / or spatial metadata. In some embodiments, the interface system 105 may be configured to receive input from one or more microphones in the environment.
[0088] Interface system 105 may include one or more network interfaces and / or one or more external device interfaces (such as one or more Universal Serial Bus (USB) interfaces). According to some embodiments, interface system 105 may include one or more wireless interfaces. Interface system 105 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. In some examples, interface system 105 may include a control system 110 and a memory system (such as...) Figure 1 The system 110 may include one or more interfaces between the optional memory system 115 shown in the diagram. However, in some instances, the control system 110 may include a memory system.
[0089] For example, the control system 110 may include a general-purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic and / or discrete hardware components.
[0090] In some implementations, the control system 110 may reside in a single device. For example, a portion of the control system 110 may reside in a device within one of the environments depicted herein, and another portion of the control system 110 may reside in a device outside the environment, such as a server, mobile device (e.g., a smartphone or tablet). In other examples, a portion of the control system 110 may reside in a device within one of the environments depicted herein, and another portion of the control system 110 may reside in one or more other devices within the environment. For example, control system functionality may be distributed across multiple smart audio devices in the environment, or may be shared by an orchestration device (such as a device that may be referred to herein as a smart home hub) and one or more other devices in the environment. In some such examples, the interface system 105 may also reside in more than one device.
[0091] In some embodiments, the control system 110 may be configured to at least partially perform the methods disclosed herein. Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as the memory devices described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. For example, one or more non-transitory media may reside on... Figure 1The optional memory system 115 and / or control system 110 shown are included. Therefore, various innovative aspects of the subject matter described in this disclosure can be implemented in one or more non-transitory media on which software is stored. For example, the software may include instructions for controlling at least one device to process audio data. For example, the software may be operable by a control system (such as...) Figure 1 The control system 110 is executed by one or more components.
[0092] In some examples, device 100 may include Figure 1 The optional microphone system 120 is shown. The optional microphone system 120 may include one or more microphones. In some embodiments, the one or more microphones may be part of or associated with another device, such as a speaker in a speaker system, a smart audio device, etc. In some examples, the device 100 may not include the microphone system 120. However, in some such embodiments, the device 100 may still be configured to receive microphone data from one or more microphones in an audio environment via the interface system 110.
[0093] According to some embodiments, device 100 may include Figure 1 The optional loudspeaker system 125 is shown in the diagram. The optional loudspeaker system 125 may include one or more loudspeakers, which may also be referred to herein as "loudspeakers". In some examples, at least some of the loudspeakers of the optional loudspeaker system 125 may be arbitrarily positioned. For example, at least some of the loudspeakers of the optional loudspeaker system 125 may be placed in locations that do not correspond to any standard-defined loudspeaker layout (such as Dolby 5.1, Dolby 5.1.2, Dolby 7.1, Dolby 7.1.4, Dolby 9.1, Hamasaki 22.2, etc.). In some such examples, at least some of the loudspeakers of the optional loudspeaker system 125 may be placed in locations that are convenient for space (e.g., where there is space to accommodate the loudspeakers), but not in any standard-defined loudspeaker layout. In some examples, the device 100 may not include the loudspeaker system 125.
[0094] In some embodiments, device 100 may include Figure 1The optional sensor system 129 is shown. The optional sensor system 129 may include one or more cameras, touch sensors, gesture sensors, motion detectors, etc. According to some embodiments, the optional sensor system 129 may include one or more cameras. In some embodiments, the camera may be a stand-alone camera. In some examples, one or more cameras of the optional sensor system 129 may reside in a smart audio device, which may be a single-purpose audio device or a virtual assistant. In some such examples, one or more cameras of the optional sensor system 129 may reside in a TV, mobile phone, or smart speaker. In some examples, the device 100 may not include the sensor system 129. However, in some such embodiments, the device 100 may still be configured to receive sensor data from one or more sensors in the audio environment via the interface system 110.
[0095] In some embodiments, device 100 may include Figure 1 The optional display system 135 is shown. The optional display system 135 may include one or more displays, such as one or more light-emitting diode (LED) displays. In some instances, the optional display system 135 may include one or more organic light-emitting diode (OLED) displays. In some examples where the device 100 includes the display system 135, the sensor system 129 may include a touch sensor system and / or gesture sensor system for proximity to one or more displays of the display system 135. According to some such embodiments, the control system 110 may be configured to control the display system 135 to present one or more graphical user interfaces (GUIs).
[0096] According to some such examples, device 100 may be or may include a smart audio device. In some such embodiments, device 100 may be or may include a wake word detector. For example, device 100 may be or may include a virtual assistant.
[0097] The term "sweet spot," used by audiophiles and recording engineers, refers to the location from which an individual can hear the playback audio in the way the mixing engineer expects to hear it. In a stereo context, and assuming the left and right speakers have equal playback levels, the sweet spot can be considered the location of the vertices of an equilateral triangle, where the left and right speakers are the other vertices. In the case of surround sound, the sweet spot can be considered the focal point of sound propagating from four or more speakers, for example, the location where the wavefronts from all the speakers arrive simultaneously. In some publications, the sweet spot is referred to as the "reference listening point."
[0098] Therefore, in some examples, the optimal listening position can be defined based on exemplary amplifier layouts such as a left / right speaker stereo layout or a left / right / center / left surround / right surround Dolby 5.1 amplifier layout. However, in many audio environments, including but not limited to home audio environments, the amplifier does not necessarily need to be positioned in a location corresponding to the amplifier in the exemplary amplifier layout.
[0099] Figure 2A A floor plan of the listening environment is depicted, which in this example is a living space. (Compared to other listening environments provided herein...) Figure 1 Sample, Figure 2A The types, quantities, and arrangements of the components shown are provided by way of example only. Other implementations may include more, fewer, and / or different types, quantities, and / or arrangements of components. In other examples, the audio environment may be another type of environment, such as an office environment, a vehicle environment, a park, or other outdoor environment. Some detailed examples involving vehicle environments are described below.
[0100] According to this example, the audio environment 200 includes a living room 210 in the upper left, a kitchen 215 in the lower center, and a bedroom 222 in the lower right. Figure 2A In the example, the boxes and circles distributed throughout the living space represent a group of loudspeakers 205a, 205b, 205c, 205d, 205e, 205f, 205g, and 205h, and in some embodiments, at least some of these loudspeakers may be smart speakers. In this example, loudspeakers 205a to 205h are placed in locations convenient for the living space, but they do not correspond to any standard “exemplary” loudspeaker layout such as Dolby 5.1, Dolby 7.1, etc. In some examples, loudspeakers 205a to 205h may be coordinated to implement one or more of the disclosed embodiments.
[0101] Flexible rendering is used for any number of loudspeakers placed arbitrarily (such as... Figure 2A This refers to the technique of rendering spatial audio on a loudspeaker (as shown in the image). With the widespread deployment of smart audio devices (e.g., smart speakers) and other audio devices not positioned according to any standard "paradigm" loudspeaker layout in the home, it may be advantageous to implement flexible rendering of audio data and playback of such rendered audio data.
[0102] Several techniques have been developed to implement flexible rendering, including centroid amplitude translation (CMAP) and flexible virtualization (FV). Both techniques treat the rendering problem as one of the minimization of a cost function, which includes at least a first term modeling the desired spatial impression the renderer is trying to achieve and a second term allocating costs for activating speakers. Detailed examples of CMAP, FV, and combinations thereof are described in International Publication No. WO 2021 / 021707 A1, published on February 4, 2021, entitled “MANAGING PLAYBACK OF MULTIPLE STREAMS OF AUDIOOVER MULTIPLE SPEAKERS” (see page 25, line 8 to page 31, line 27), which is hereby incorporated by reference.
[0103] However, the methods disclosed herein concerning flexible rendering are not limited to CMAP- and / or FV-based flexible rendering. Such methods can be implemented by any suitable type of flexible rendering, such as Vector Base Amplitude Panning (VBAP). The relevant VBAP method is disclosed by Pulkki and Ville in “Virtual Sound Source Positioning Using Vector Base Amplitude Panning” (see Journal of the Audio Engineering Society, Vol. 45, No. 6 (June 1997)), which is hereby incorporated by reference. Other suitable types of flexible rendering include, but are not limited to, methods based on double-balanced panning and ambisonics, such as the method described by D. Arteaga in “An Ambisonics Decoder for Irregular3-DLoudspeaker Arrays” (see page 8918, (May 2013)), which is hereby incorporated by reference.
[0104] In some instances, it can be relative to, for example Figure 2A The audio environment coordinate system 217 shown performs flexible rendering. According to this example, the audio environment coordinate system 217 is a two-dimensional Cartesian coordinate system. In this example, the origin of the audio environment coordinate system 217 is within the loudspeaker 205a, and the x-axis corresponds to the major axis of the loudspeaker 205a. In other embodiments, the audio environment coordinate system 217 may be a three-dimensional coordinate system, which may or may not be a Cartesian coordinate system.
[0105] Furthermore, the origin of the coordinate system need not be associated with the loudspeaker or loudspeaker system. In some implementations, the origin of the coordinate system can be at another location within the audio environment 200. An example of such an example is provided by the location of an alternative audio environment coordinate system 217'. In this example, the origin of the alternative audio environment coordinate system 217' has been chosen such that the values of x and y are positive for all locations within the audio environment 200. In some instances, the origin and orientation of the coordinate system can be chosen to correspond to the position and orientation of a person's head within the audio environment 200. In some such implementations, the person's viewing direction can be along an axis of the coordinate system (e.g., along the positive y-axis).
[0106] In some implementations, the control system may control the flexible rendering process at least in part based on the location (and in some examples, orientation) of each participating loudspeaker (e.g., each active loudspeaker and / or each loudspeaker for which audio data will be rendered) in the audio environment. According to some such implementations, the control system may have previously determined the location (and in some examples, orientation) of each participating loudspeaker according to a coordinate system such as audio environment coordinate system 217, and may have stored the corresponding loudspeaker location data in a data structure. Some methods for determining the location of audio devices are described below.
[0107] According to some such implementations, a control system for an orchestration device (in some instances, which may be one of loudspeakers 205a to 205h) can render audio data such that specific elements or areas of the audio environment 200 (such as television 230) represent the front and center of the audio environment. Such implementations may be advantageous for some use cases such as playing back audio of movies, television programs, or other content being displayed on television 230.
[0108] However, for other use cases (such as playing back music that is not associated with the content being displayed on the television 230), such a rendering method may not be optimal. In such alternative use cases, it might be desirable to render audio data for playback such that the front and center of the rendered sound field correspond to the position and orientation of a person within the audio environment 200.
[0109] For example, referring to person 220a, it might be desirable to render audio data for playback such that the front and center of the rendered sound field correspond to the viewing direction of person 220a, indicated by the direction of arrow 223a from person 220a's position. In this example, person 220a's position is indicated by point 221a at the center of person 220a's head. In some examples, the "optimal listening position" of the audio data rendered for playback by person 220a may correspond to point 221a. Several methods for determining a person's position and orientation in an audio environment are described below. In some such examples, a person's position and orientation can be determined based on the position and orientation of a piece of furniture (such as the position and orientation of chair 225).
[0110] According to this example, the positions of people 220b and 220c are represented by points 221b and 221c, respectively. Here, the front portions of people 220b and 220c are represented by arrows 223b and 223c, respectively. The positions of points 221a, 221b, and 221c, and the orientations of arrows 223a, 223b, and 223c, can be determined relative to a coordinate system such as the audio environment coordinate system 217. As mentioned above, in some examples, the origin and orientation of the coordinate system can be chosen to correspond to the position and orientation of the person's head within the audio environment 200.
[0111] In some examples, the "optimal listening position" of the audio data rendered for playback of person 220b can correspond to point 221b. Similarly, the "optimal listening position" of the audio data rendered for playback of person 220c can correspond to point 221c. It can be observed that if the "optimal listening position" of the audio data rendered for playback of person 220a corresponds to point 221a, then this optimal listening position will not correspond to either point 221b or point 221c.
[0112] Furthermore, the front and central regions of the sound field rendered for person 220b should ideally correspond to the direction of arrow 223b. Similarly, the front and central regions of the sound field rendered for person 220c should ideally correspond to the direction of arrow 223c. It can be observed that the front and central regions are different for persons 220a, 220b, and 220c. Therefore, the audio data rendered via the previously disclosed method and based on the position and orientation of any one of these persons will not be optimal for the position and orientation of the other two persons.
[0113] However, various disclosed implementations can satisfactorily render audio data for multiple optimal listening locations and, in some instances, for multiple orientations. Some such methods involve creating two or more distinct spatial renders of the same audio content for different listening configurations on a set of public loudspeakers and combining the renders by multiplexing the distinct spatial renders across frequencies. In some such examples, the spectrum corresponding to the human hearing range (e.g., 20 Hz to 20,000 Hz) can be divided into multiple frequency bands. According to some such examples, each of the distinct spatial renders will be played back via a different set of frequency bands. In some such examples, the rendered audio data corresponding to each set of frequency bands can be combined into a single loudspeaker feed signal output set. The result is that spatial audio can be provided for each of multiple locations and, in some instances, for each of multiple orientations.
[0114] Some such implementations may involve rendering spatial audio separately for two or more individuals in the vehicle (e.g., both the driver and the front passenger). According to some examples, the number of listeners and their locations (and in some instances, their orientation) can be determined based on sensor data. In the context of the vehicle, the number of listeners and their locations (and in some instances, their orientation) can be determined based on seat sensor data.
[0115] In some implementations, it can be based on the audio environment (such as...) Figure 2A Data from one or more cameras in the audio environment 200 is used to determine the number of listeners and their locations (and, in some instances, their orientation). In this example, the audio environment 200 includes cameras 211a to 211e distributed throughout the environment. In some embodiments, one or more smart audio devices in the audio environment 200 may also include one or more cameras. The one or more smart audio devices may be single-purpose audio devices or virtual assistants. In some such examples, one or more cameras of the optional sensor system 130 may reside in or on a television 230, a mobile phone, or a smart speaker (such as one or more of loudspeakers 205b, 205d, 205e, or 205h). Although cameras 211a to 211e are not shown in every depiction of the audio environment presented in this disclosure, in some embodiments, each audio environment may still include one or more cameras.
[0116] Figure 2B It shows Figure 2A An example of the audio environment at different times. In this example, people 220a and 220b have changed their positions and orientations. Figure 2BAt the time depicted, person 220a has moved to chair 225, and person 220b is standing between sofa 240 and table 233. In some embodiments, the new positions and orientations of persons 220a and 220b can be determined, and an audio signal can be rendered for each of the new positions and orientations. In some examples, as disclosed herein, the rendered audio signals can be processed and combined.
[0117] Figure 2C Another example of an audio environment is shown. In this example, audio environment 200 includes loudspeakers 205i, 205j, and 205k. According to this example, a single listening position (corresponding to point 221d) and two listening directions (corresponding to arrows 223d and 223e) are shown. In this example, the two listening directions are orthogonal to each other. In some implementations, two sets of rendered audio signals can be generated corresponding to each of the two orientations and to the single position. In some examples, as disclosed herein (e.g., by cross-frequency multiplexing), the rendered audio signals can be processed and combined. Such implementations can provide listeners with a more uniform sense of space regardless of their orientation within audio environment 200. Therefore, such implementations may be desirable for parties or other social gatherings.
[0118] Figure 3 An example box of one of the disclosed implementations is shown. (Compared to other implementations provided herein...) Figure 1 Sample, Figure 3 The types, quantities, and arrangements of elements shown are provided by way of example only. Other embodiments may include more, fewer, and / or different types, quantities, and / or arrangements of elements. According to some embodiments, Figure 3 At least some of the frames can be accessed via Figure 1 The device 100 is used to implement this. In this example, elements 310a to 310n, 315a to 315n, and 320 are implemented via an instance of the control system 110 of the device 100. In some such examples, elements 310a to 310n, 315a to 315n, and 320 may be implemented by the control system 110 according to instructions stored on one or more non-transitory computer-readable media, in some instances, which may correspond to one or more memory devices of the memory system 115.
[0119] In this example, spatial audio stream 305 is received and rendered by a set of N spatial audio renderers 310a to 310n. In some examples, spatial audio stream 305 may include an audio signal and associated spatial data. The spatial data may indicate the expected perceived spatial location corresponding to the audio signal. According to some examples (e.g., for systems such as Dolby Atmos...), TMIn the implementation of an audio object, spatial data may be or may include location metadata. However, in some instances, it is anticipated that the perceived spatial location may correspond to the channels of a channel-based audio format (e.g., to the left channel, right channel, center channel, etc.). Therefore, examples of spatial audio streams 305 that may be received by spatial audio renderers 310a to 310n include stereo, Dolby 5.1, Dolby 7.1, and object-based audio content (such as Dolby Atmos).
[0120] In this example, N is at least three, meaning there are at least three spatial audio renderers. However, in some alternative examples, N can be two or more. In some examples, one or more of the spatial audio renderers 310a to 310n can operate in the time domain. In some instances, one or more of the spatial audio renderers 310a to 310n can operate in the frequency domain.
[0121] According to this example, each of the spatial audio renderers 310a to 310n is configured to render audio data for a single listening configuration. For example, the listening configuration may be defined according to a coordinate system. The listening configuration may correspond to a person's listening position (or listening area) in an audio environment. In some examples, the listening configuration may correspond to a person's listening orientation in the audio environment. In some examples, the listening configuration may be determined relative to the position (and in some instances, orientation) of each of a group of two or more loudspeakers numbered. In some instances, the listening position (or listening area) may correspond to the position and orientation of a piece of furniture in the audio environment. For example, refer to... Figure 2A The listening position may correspond to the position and orientation of chair 225. In some examples, the listening area may correspond to the position and orientation of at least a portion of sofa 240 (e.g., with section 205a or section 205b).
[0122] In this example, each of the spatial audio renderers 310a to 310n is configured to generate a loudspeaker feed signal, which is provided to a corresponding decomposition module in the decomposition modules 315a to 315n. In this embodiment, each of the decomposition modules 315a to 315n is configured to decompose the loudspeaker feed signal into a selected set of frequency bands. For embodiments where one or more of the decomposition modules 315a to 315n are receiving the loudspeaker feed signal in the time domain, the decomposition module receiving such a loudspeaker feed signal can be configured to transform the loudspeaker feed signal to the frequency domain. In this context, the “frequency band” generated by the decomposition modules 315a to 315n is a frequency domain representation of the loudspeaker feed signal within each of a set of frequency ranges. However, as described below, in some examples, some or all of the spatial audio renderers 310a to 310n and the corresponding decomposition modules 315a to 315n can operate in the time domain. In some such examples, the “frequency band” can be a loudspeaker feed signal that has been filtered in the time domain to have a desired energy distribution in a selected frequency band.
[0123] According to this example, the combining module 320 is configured to combine the renderer-specific loudspeaker feed signal sets 317a to 317n output by each of the decomposition modules 315a to 315n to produce a loudspeaker feed signal output set 325. According to some examples, the combining module 320 may be configured to combine the renderer-specific loudspeaker feed signals 317a to 317n (e.g., add them together). The operation of the combining module 320 can be viewed as a multiplexing process. Alternatively, the combined operation of the decomposition modules 315a to 315n and the combining module 320 can be viewed as a multiplexing process. In some examples, the combining module 320 may be configured to transform the combined renderer-specific loudspeaker feed signal sets 317a to 317n from the frequency domain to the time domain, such that the loudspeaker feed signal output set 325 is in the time domain. However, in some implementations, some or all of the spatial audio renderers 310a to 310n and the corresponding decomposition modules 315a to 315n may operate in the time domain. In some such examples, some or all of the decomposition modules 315a to 315n may be implemented as comb filters in the time domain. In some examples, some or all of the decomposition modules 315a to 315n may be implemented as finite impulse response (FIR) or infinite impulse response (IIR) filters in the time domain. In some examples, the loudspeaker feed signal output set 325 may be provided to a group of loudspeakers in an audio environment. According to some embodiments, the loudspeaker feed signal output set 325 may be played back by the group of loudspeakers.
[0124] In some instances, each frequency band set generated by each of the decomposition modules 315a to 315n can be a renderer-specific frequency band set: for example, a different renderer-specific frequency band set can be specifically selected for each of the spatial audio renderers 310a to 310n. According to some implementations, these renderer-specific frequency band sets can be advantageously selected such that the loudspeaker feed signal output set 325 includes all frequencies within the audible range, or all frequencies within the frequency range included in the spatial audio stream 305.
[0125] In one such example, spatial audio stream 305 may include (and / or loudspeaker feed signal output set 325 may represent) in a range from F min To F max The audio data in the frequency range. In this example, the combined frequency band set of the renderer-specific loudspeaker feed signals 317a to 317n (in other words, the frequency band of the loudspeaker feed signal output set 325) may include a range from F min To F max (including F) min and F max The adjacent frequency bands B1 to B) X Where X is an integer corresponding to the total number of frequency bands. In some such examples, the decomposition module 315a can generate frequency band sets B1, B... 1+N B 1+2N In some such examples, the decomposition module 315b can generate frequency band sets B2, B... 2+N B 2+2N In some such examples, the decomposition module 315c can generate frequency band sets B3, B... 3+N B 3+2N wait.
[0126] For example, in an implementation with four spatial audio renderers and 64 frequency bands, the decomposition module 315a can generate frequency band sets B1, B5, B9, B... 13 B 17 B 21 B 25 B 29 B 33 B 37 B 41 B 45 B 49 B 53 B 57 and B 61 In one such example, the decomposition module 315b can generate frequency band sets B2, B6, and B... 10 B 14 B 18 B 22 B26 B 30 B 34 B 38 B 42 B 46 B 50 B 54 B 58 and B 62 In one such example, the decomposition module 315c can generate the frequency band sets B3, B7, and B... 11 B 15 B 19 B 23 B 27 B 31 B 35 B 39 B 43 B 47 B 51 B 55 B 59 and B 63 In one such example, the decomposition module 315d can generate the frequency band sets B4, B8, and B... 12 B 16 B 20 B 24 B 28 B 32 B 36 B 40 B 44 B 48 B 52 B 56 B 60 and B 64 In some such examples, the loudspeaker feed signal output set 325 includes all 64 frequency bands B1 to B2. 64 The foregoing is an example of what may be called a "non-overlapping" implementation, in which each set in the renderer-specific frequency band set includes different frequency bands.
[0127] However, in some alternative examples, one or more overlapping or non-unique frequency bands may exist, generated by decomposition modules 315a to 315n. In some such examples, one or more of the lowest frequency bands may be generated by two or more decomposition modules 315a to 315n. For example, in an example similar to the foregoing example, decomposition module 315d may generate frequency band sets B1, B4, B8, B... 12 B 16 B 20 B 24 B 28 B 32 B 36B 40 B 44 B 48 B 52 B 56 B 60 and B 64 Decomposition modules 315a to 315c can generate the frequency band set indicated in the preceding paragraphs. It can be observed that, in such an example, the loudspeaker feed signal output set 325 includes two contributions from frequency band B1. Some such implementations may involve matching the playback amplitude of the overlapping frequency bands to the playback amplitude of the non-overlapping example described above, wherein only the frequency band set generated by decomposition modules 315a to 315d includes frequency band B1. For example, some such implementations may involve selecting the levels of two instances of frequency band B1 such that the total sound pressure level in frequency band B1 is the same in the overlapping case as in the non-overlapping case.
[0128] Figure 4 An example box of another disclosed implementation is shown. (Compared to other implementations provided herein...) Figure 1 Sample, Figure 4 The types, quantities, and arrangements of elements shown are provided by way of example only. Other embodiments may include more, fewer, and / or different types, quantities, and / or arrangements of elements. According to some embodiments, Figure 4 At least some of the frames can be accessed via Figure 1 The device 100 is used to implement this. In this example, elements 310a to 310n, 315a to 315n, and 320 are implemented via an instance of the control system 110 of the device 100. In some such examples, elements 310a to 310n, 315a to 315n, and 320 may be implemented by the control system 110 according to instructions stored on one or more non-transitory computer-readable media, in some instances, which may correspond to one or more memory devices of the memory system 115.
[0129] In this example, spatial audio stream 305 is received and rendered by a set of N spatial audio renderers 310a to 310n. According to this example, spatial audio stream 305 and spatial audio renderers 310a to 310n are as described above. Figure 3 As described, these descriptions will not be repeated here.
[0130] According to this embodiment, each of the spatial audio renderers 310a to 310n is configured to generate a loudspeaker feed signal, which is provided to a corresponding decomposition module in the decomposition modules 315a to 315n. In this embodiment, each of the decomposition modules 315a to 315n includes a corresponding filter bank analysis block in filter bank analysis blocks 405a to 405n, each of which is configured to decompose the loudspeaker feed signal 403a to 403n from the corresponding spatial audio renderer in the spatial audio renderers 310a to 310n into one of the global frequency band sets 407a to 407n. The filter bank analysis blocks 405a to 405n can be configured to implement a short-time discrete Fourier transform (STDFT) filter bank, a hybrid composite quadrature mirror (HCQMF) filter bank, a quadrature mirror (QMF) filter bank, or another type of filter bank. According to some examples, the global frequency band set can be the same as the one referenced above. Figure 3 The described adjacent frequency bands B1 to B X Correspondingly.
[0131] According to this example, each of the decomposition modules 315a to 315n includes a corresponding frequency band selection block among the frequency band selection blocks 410a to 410n, each of which is configured to select a renderer-specific set of frequency bands from a global frequency band set generated by the corresponding filter bank analysis blocks among the filter bank analysis blocks 405a to 405n. For example, the renderer-specific set of frequency bands may be as referenced above. Figure 3 As described. However, other implementations may provide different renderer-specific frequency band sets. For one or more of the decomposition modules 315a to 315n that are receiving a speaker feed signal in the time domain, the decomposition module receiving such a speaker feed signal may be configured to transform the speaker feed signal to the frequency domain.
[0132] According to this example, the combination module 320 includes a combination block 415 configured to combine renderer-specific loudspeaker feed signals 317a to 317n output from each of the decomposition modules 315a to 315n to produce a loudspeaker feed signal output set 417 in the frequency domain. In some examples, the combination block 415 may be configured to combine the renderer-specific loudspeaker feed signals 317a to 317n via a multiplexing process. In this example, the combination module 320 also includes a filter combination block 420 configured to transform the loudspeaker feed signal output set 417 from the frequency domain to the time domain, such that the loudspeaker feed signal output set 325 is in the time domain. In some examples, the loudspeaker feed signal output set 325 may be provided to a group of loudspeakers in an audio environment. According to some embodiments, the loudspeaker feed signal output set 325 may be played back by the group of loudspeakers.
[0133] Figure 5 It outlines what can be made by, for example Figures 1 to 4 The flowchart illustrates an example of a method performed by the apparatus or system shown herein. As with other methods described herein, the blocks of method 500 need not be performed in the indicated order. Furthermore, such a method may include more or fewer blocks than those shown and / or described. The blocks of method 500 may be performed by one or more devices, which may be (or may include) a control system, such as... Figure 1 , Figure 3 and Figure 4 The control system 110 shown and described above, or one of other disclosed control system examples.
[0134] In this implementation, block 505 relates to receiving audio data by a control system configured to implement multiple renderers. In some examples, the audio data may include an audio signal and associated spatial data, for example, as referenced above. Figure 3 or Figure 4 The spatial audio stream 305 describes this. Therefore, in some examples, the audio data may include spatial channel-based audio data and / or spatial object-based audio data. In some instances, the audio data may have one of the following audio formats: stereo, Dolby 3.1.2, Dolby 5.1, Dolby 5.1.2, Dolby 7.1, Dolby 7.1.2, Dolby 7.1.4, Dolby 9.1, Dolby 9.1.6, or Dolby Atmos. TM .
[0135] According to this example, box 510 relates to a control system receiving listening configuration data for multiple listening configurations. In this example, each listening configuration corresponds to a listening position and listening orientation in an audio environment. For example, each listening configuration may correspond to a person's listening position and listening orientation in an audio environment. For example, the listening position may correspond to a person's head position. For example, the listening orientation may correspond to a person's head orientation. For example, the listening position and orientation may correspond to... Figure 2A and Figure 2B The positions and orientations of people 220a to 220c shown correspond to those in the diagram. Figure 2C In the example shown, box 510 may involve receiving listening configuration data for two listening configurations corresponding to the same listening location and two different listening directions.
[0136] According to this embodiment, block 515 relates to each of the plurality of renderers rendering the audio data according to the listening configuration data to obtain a renderer-specific amplifier feed signal set for a corresponding listening configuration. In this example, each renderer is configured to render the audio data for a different listening configuration. In some embodiments, one or more renderers may operate in the time domain, for example, to perform double-balanced amplitude translation in the time domain. According to some embodiments, one or more renderers may operate in the frequency domain, for example, to perform crosstalk cancellation in the frequency domain. In some examples, block 515 may be... Figure 3 or Figure 4 The spatial audio renderer 310a to 310n is executed.
[0137] In this example, block 520 relates to the control system decomposing each renderer-specific set of loudspeaker feed signals into renderer-specific sets of frequency bands for each renderer. In some examples, the “frequency band” generated in block 520 may be a frequency domain representation of the renderer-specific loudspeaker feed signal in each frequency range of a set of frequency ranges. However, as described elsewhere herein, in some examples, the “frequency band” may be a loudspeaker feed signal in the time domain that has been filtered in block 520 to have a desired energy distribution in the selected frequency band. In some examples, block 520 may be... Figure 3 or Figure 4 The decomposition modules 315a to 315n are executed. In some "non-overlapping" implementations, each set in the renderer-specific band set may include different bands. However, in some "overlapping" implementations, one or more bands may be included in two or more sets within the renderer-specific band set.
[0138] According to this embodiment, block 525 relates to combining renderer-specific frequency bands of each renderer by a control system to generate a set of loudspeaker feed signal outputs. In some examples, block 525 may relate to adding the renderer-specific frequency band sets. The combination process of block 525 can be viewed as a process of multiplexing the renderer-specific frequency band sets. However, some may consider the operation of block 520 and the combination process of block 525 together as a process of multiplexing the renderer-specific frequency band sets. According to some embodiments, block 525 may relate to transforming the loudspeaker feed signal output set from the frequency domain to the time domain by a synthesis filter bank. In this example, block 530 relates to outputting the loudspeaker feed signal output set to multiple loudspeakers by the control system.
[0139] In some examples, decomposing a renderer-specific loudspeaker feed signal set into a renderer-specific frequency band set may involve analyzing the renderer-specific loudspeaker feed signal set by an analysis filter bank associated with each renderer to produce a global frequency band set. For example, the analysis filter bank may be a short-time discrete Fourier transform (STDFT) filter bank, a hybrid composite quadrature mirror (HCQMF) filter bank, or a quadrature mirror (QMF) filter bank. For example, the global frequency band set may include adjacent frequency bands B1 to B2. X (including B1 and B) X ), where X is an integer corresponding to the total number of frequency bands.
[0140] In some examples, decomposing each renderer-specific loudspeaker feed signal set into each renderer-specific frequency band set may involve selecting a frequency band subset of the global frequency band set to generate the renderer-specific frequency band set. According to some implementations, each set in the renderer-specific frequency band set may be uniquely associated with one of the plurality of renderers and uniquely associated with one of the plurality of listening configurations. In some examples, a subset of the global frequency band set may be selected such that when combining the renderer-specific frequency bands of all renderers in the plurality of renderers, each frequency band in the global frequency band set is represented only once in the loudspeaker feed signal output set.
[0141] In some implementations... Figure 3 and Figure 4 Some or all of the renderers described can utilize different strategies to perform their rendering, and in some instances can operate in different signal domains. For example, one renderer might perform double-balanced amplitude translation in the time domain, while another renderer might employ crosstalk cancellation implemented in the frequency domain. However, the loudspeaker feeds from each renderer must ultimately be in a common domain (e.g., the time or frequency domain) and then combined with the outputs of one or more other renderers.
[0142] Further efficiency can be achieved when all renderers operate on the output from the same filter bank. An example of this will now be described.
[0143] Figure 6A An example box of another disclosed implementation is shown. (Compared to other implementations provided herein...) Figure 1 Sample, Figure 6AThe types, quantities, and arrangements of the elements shown are provided by way of example only. Other implementations may include more, fewer, and / or different types, quantities, and / or arrangements of elements. In this example, elements 310a to 310n, 405, 410a to 410n, 415, and 420 are implemented via an instance of the control system 110 of device 100. In some such examples, elements 310a to 310n, 405, 410a to 410n, 415, and 420 may be implemented by control system 110 according to instructions stored on one or more non-transitory computer-readable media, in some instances corresponding to one or more memory devices of memory system 115.
[0144] In this example, spatial audio stream 305 is received by filter bank analysis block 405. Here, only a single instance of filter bank analysis block 405 is implemented, and filter bank analysis is applied to the input spatial audio stream 305, rather than to other streams such as... Figure 3 and Figure 4 The illustration shows multiple instances of speaker feeds for each of the spatial audio renderers 310a to 310n. According to this example, filter bank analysis block 405 generates a global set of frequency bands 607 corresponding to the audio data of the spatial audio stream 305. In this context, the "frequency band" generated by filter bank analysis block 405 is a frequency domain representation of the audio data of the spatial audio stream 305 within each frequency range of a set of frequency ranges. In this example, filter bank analysis block 405 is the one referenced above. Figure 4 The description of a single instance of filter bank analysis blocks 405a to 405n is so specific that the description will not be repeated here.
[0145] In this implementation, each of the frequency band selection blocks 410a to 410n is configured to select a corresponding set from the global frequency band set 607 within the set of renderer-specific frequency bands 617a to 617n and provide one of the renderer-specific frequency band sets 617a to 617n to the corresponding spatial audio renderer among the spatial audio renderers 310a to 310n. Therefore, for each of the spatial audio renderers 310a to 310n, only the renderer-specific frequency bands of the spatial audio stream belonging to its selected frequency band subset are processed to generate speaker feeds for those frequency bands, thereby potentially reducing the complexity of the operations performed by each of the spatial audio renderers.
[0146] If the input spatial audio stream 305 includes spatial metadata, in some implementations that spatial metadata is also provided to spatial audio renderers 310a to 310n. In some such examples, the spatial metadata may accompany the global band set 607 as well as each set in the renderer-specific band sets 617a to 617n.
[0147] According to this example, control system 110 is configured to implement the above reference. Figure 4 The described combination block 415 is configured to combine renderer-specific loudspeaker feed signals 317a to 317n output by spatial audio renderers 310a to 310n to produce a loudspeaker feed signal output set 417 in the frequency domain. In some examples, combination block 415 may be configured to combine renderer-specific loudspeaker feed signals 317a to 317n via a summation process. In this example, control system 110 is configured to implement filter combination block 420, which is configured to transform the combined renderer-specific loudspeaker feed signal sets 317a to 317n from the frequency domain to the time domain, such that the loudspeaker feed signal output set 325 is in the time domain. In some examples, the loudspeaker feed signal output set 325 may be provided to a group of loudspeakers in an audio environment. According to some embodiments, the loudspeaker feed signal output set 325 may be played back by the group of loudspeakers.
[0148] In one example, each of the spatial audio renderers 310a to 310n can be configured to implement centroid amplitude shift (CMAP) and flexible virtualization (FV), or a combination thereof. In other examples, each of the spatial audio renderers 310a to 310n can be configured to implement vector basis amplitude shift (VBAP), double-balanced shift, or another type of flexible rendering. According to some such implementations, each of the spatial audio renderers 310a to 310n can be implemented to operate in the frequency domain using an HCQMF filter bank. Such a flexible renderer is inherently adaptable to different listening positions relative to a set of common loudspeakers, and therefore, each of the N renderers can be implemented as an instantiation of a different configuration of the same core renderer operating in the HCQMF domain. The same HCQMF filter bank is also suitable for multiplexing renderers across frequencies, and therefore, Figure 6A The efficient implementation shown is applicable. In some such examples, the HCQMF filter bank may contain 77 frequency bands. However, alternative implementations may involve different types of filter banks, some of which may have more or fewer frequency bands.
[0149] (According to some embodiments) One practical consideration for implementing flexible rendering is complexity. In some cases, given the processing power of a particular device, performing accurate rendering for each frequency band of each audio object in real time may be impractical. One challenge is that the audio object locations of at least some of the audio objects to be rendered (which, in some instances, may be indicated by metadata) can change multiple times per second. Since rendering can be performed for each of multiple listening configurations, the complexity can be compounded for some of the disclosed implementations.
[0150] An alternative approach to reducing complexity at the expense of memory is to use one or more lookup tables (or other such data structures) that include samples of all possible object locations in three-dimensional space (e.g., speaker-activated locations). Depending on the specific implementation, the samples may be the same or different across all dimensions. In some such examples, one such data structure may be created for each of multiple listening configurations. Alternatively or additionally, a single data structure can be created by summing multiple data structures, each of which may correspond to different listening configurations among the multiple listening configurations.
[0151] Figure 6B This is a graph indicating the points of speaker activation in an example embodiment. In this example, 15 points are sampled along the x and y dimensions, and 5 points are sampled along the z dimension. According to this example, each point represents M speaker activations, and one speaker activation is used for each of the M speakers in the audio environment. Speaker activation can be... Figure 6A The filter bank analyzes the gain or composite value of each of the N frequency bands associated with 405. A single data structure can be created by multiplexing data structures associated with multiple listening configurations across these frequency bands. In other words, for each frequency band of the data structure, activations from each of the multiple listening configurations can be selected. Once this single multiplexed data structure is created, it can be used to implement [the desired configuration]. Figure 6A The functionality is associated with a single instance of the renderer that is equivalent to the function of the function. Based on this example, Figure 6B The points shown can correspond to the speaker activation values of a single data structure created by multiplexing multiple data structures, each of which corresponds to a different listening configuration.
[0152] Other implementations may include more or fewer samples. For example, in some implementations, the spatial sampling used for speaker activation may be non-uniform. Some implementations may involve more than... Figure 6B The example shows more or fewer speaker activation samples in the x,y plane. Some implementations can determine speaker activation samples in only one x,y plane. According to this example, each point represents M speaker activations for CMAP, FV, VBAP, or other flexible rendering methods. In some implementations, such as Figure 6B The speaker activations shown in the diagram, and a set of speaker activations, can be stored in a data structure, which may be referred to herein as a "table" (or "Cartesian table," such as...). Figure 6B (As indicated in the document).
[0153] The desired rendering location does not necessarily correspond to the calculated location of speaker activation. At runtime, some form of interpolation can be implemented to determine the actual activation of each speaker. In some such examples, trilinear interpolation between the speaker activations of the eight points closest to the desired rendering location can be used.
[0154] Figure 6C It is a graph of trilinear interpolation between points indicating speaker activation, based on an example. Figure 6C The solid circle 603 at or near the vertex of the rectangular prism shown corresponds to the positions of the eight points closest to the calculated desired rendering position of the speaker activation. In this example, the desired rendering position is... Figure 6C Points within a rectangular prism are presented in the image. In this example, the continuous linear interpolation process includes interpolating each pair of points in the top plane to determine a first interpolation point 605a and a second interpolation point 605b; interpolating each pair of points in the bottom plane to determine a third interpolation point 610a and a fourth interpolation point 610b; interpolating the first interpolation point 605a and the second interpolation point 605b to determine a fifth interpolation point 615 in the top plane; interpolating the third interpolation point 610a and the fourth interpolation point 610b to determine a sixth interpolation point 620 in the bottom plane; and interpolating the fifth interpolation point 615 and the sixth interpolation point 620 to determine a seventh interpolation point 625 between the top and bottom planes.
[0155] While trilinear interpolation is an efficient interpolation method, those skilled in the art will understand that it is merely one possible interpolation method that can be used to implement various aspects of this disclosure, and other examples may include other interpolation methods. For instance, some implementations may involve methods that are more efficient than... Figure 6B The x,y plane shown is interpolated in more or less of the x,y plane. Some such implementations may involve interpolation in only one x,y plane. In some implementations, the speaker activation for the desired rendering location will be simply set to the speaker activation of the location closest to the desired rendering location where speaker activation has been calculated.
[0156] Figure 7 This is a flowchart outlining another example of a method that can be performed by a device or system such as the apparatus or system disclosed herein. As with other methods described herein, the blocks of method 700 need not be performed in the indicated order. Furthermore, such a method may include more or fewer blocks than those shown and / or described. The blocks of method 700 may be performed by one or more devices, which may be (or may include) a control system, such as... Figure 6A The control system 110 shown and described above, or one of other disclosed control system examples.
[0157] In this embodiment, block 705 relates to receiving audio data by a control system. In some examples, the audio data may include an audio signal and associated spatial data, for example, as referenced above. Figure 3 , Figure 4 This is as described in Figure 6, specifically the spatial audio stream 305. Therefore, in some examples, the audio data may include spatial channel-based audio data and / or spatial object-based audio data. In some instances, the audio data may have one of the following audio formats: stereo, Dolby 3.1.2, Dolby 5.1, Dolby 5.1.2, Dolby 7.1, Dolby 7.1.2, Dolby 7.1.4, Dolby 9.1, Dolby 9.1.6, or Dolby Atmos. TM .
[0158] According to this example, box 710 relates to a control system receiving listening configuration data for multiple listening configurations. In this example, each listening configuration corresponds to a listening position and a listening orientation. For example, each listening configuration may correspond to a person's listening position and listening orientation in an audio environment. For example, a listening position may correspond to a person's head position. For example, a listening orientation may correspond to a person's head orientation.
[0159] According to some examples, the listening configuration data may correspond to sensor data obtained from one or more sensors in the audio environment. For example, the sensors may include one or more cameras, one or more motion sensors, and / or one or more microphones. In some instances, the listening position and the listening direction may be relative to the coordinate system of the audio environment. According to some examples, the listening position may be relative to the location of one or more loudspeakers in the audio environment.
[0160] According to this embodiment, block 715 relates to analyzing received audio data by an analysis filter bank implemented via a control system to generate a global frequency band set corresponding to the audio data. In some instances, the analysis filter bank may be a short-time discrete Fourier transform (STDFT) filter bank, a hybrid composite quadrature mirror (HCQMF) filter bank, or a quadrature mirror (QMF) filter bank. In some examples, block 715 may be performed by a filter bank analysis block 405 that generates a global frequency band set 607 corresponding to the audio data of the spatial audio stream 305.
[0161] In this example, block 720 relates to the selection of a subset of a global frequency band set by a control system and for each of a plurality of renderers implemented by the control system to generate a renderer-specific frequency band set for each renderer. In some such implementations, each renderer-specific loudspeaker feed signal set may be uniquely associated with one of the plurality of renderers and uniquely associated with one of the plurality of listening configurations. In some examples, block 720 may be executed by each of frequency band selection blocks 410a to 410n, which are configured to select a corresponding set from the global frequency band set 607 of renderer-specific frequency bands 617a to 617n and provide one of the renderer-specific frequency band sets 617a to 617n to the corresponding spatial audio renderer among the spatial audio renderers 310a to 310n.
[0162] According to this embodiment, block 725 relates to each of the plurality of renderers rendering a renderer-specific set of frequency bands according to the listening configuration data to obtain a renderer-specific set of loudspeaker feed signals for a corresponding listening configuration. In this example, each renderer is configured to render frequency bands in the renderer-specific set of frequency bands for different listening configurations. In some examples, block 725 may be... Figure 6A The spatial audio renderers 310a to 310n are executed. According to some examples, the rendering of box 725 may involve crosstalk cancellation in the frequency domain.
[0163] In this example, block 730 relates to the control system combining renderer-specific loudspeaker feed signal sets from each of a plurality of renderers to produce a set of loudspeaker feed signal outputs. In some examples, combining the set of loudspeaker feed signals may involve multiplexing each of the renderer-specific loudspeaker feed signal sets. In some examples, block 730 may be at least partially derived from the above reference. Figure 6A The described combination block 415 is executed, and is configured to combine renderer-specific loudspeaker feed signals 317a to 317n output by spatial audio renderers 310a to 310n to produce a loudspeaker feed signal output set 417 in the frequency domain. According to some examples, block 730 (or another block of method 700) may involve (e.g., via a synthesis filter bank) transforming the loudspeaker feed signal output set in the frequency domain into a loudspeaker feed signal output set in the time domain.
[0164] In some alternative examples, box 725 may involve generating multiple data structures. Each data structure may include a set of renderer-specific speaker activations for a corresponding listening configuration and corresponding to each of a plurality of points in two-dimensional or three-dimensional space. In some such examples, one such data structure may be created for each of the multiple listening configurations, for example, as referenced above. Figure 6B and Figure 6C As described. In some such examples, box 730 may involve creating a single data structure (e.g., a single lookup table) by summing multiple data structures, each of which corresponds to a different listening configuration in multiple listening configurations.
[0165] In this embodiment, block 735 relates to the control system outputting a set of loudspeaker feed signals to multiple loudspeakers. In some examples, method 700 may involve causing multiple loudspeakers to reproduce the set of loudspeaker feed signals.
[0166] In some implementations, the audio environment may be or may include the vehicle environment. Figure 8 An example of a vehicle interior according to one embodiment is shown. In this example, vehicle 800 includes seats 805a, 805b, 805c, and 805d, each of which includes a seat back 807, a seat bottom 809, and one of headrests 810a, 810b, 810c, and 810d. In this embodiment, each seat has one or more associated elbow rests 811 and seat belts 813.
[0167] In this example, vehicle 800 includes multiple loudspeakers, although the loudspeakers are... Figure 8 The speakers are not visible in the vehicle. A potential advantage of the vehicle audio environment is that the position and orientation of the speakers within the vehicle are usually fixed. Therefore, the position and orientation of the speakers within the vehicle are typically known and do not need to be determined, for example, based on an automatic audio equipment positioning process.
[0168] Based on some examples, a vehicle control system (which may be...) Figure 1 An example of the control system 110 can be configured to determine the listening location and direction of one or more persons in the vehicle 800. In some such examples, it can be configured to determine the listening location and direction of one or more persons in the vehicle 800 based on sensor data obtained from one or more sensors of the vehicle 800. The one or more sensors can be... Figure 1 An example of sensor system 129. In Figure 8 In the example shown, the vehicle control system has determined the positions of listener 1, who is sitting in the driver's seat, and listener 2, who is sitting in the front passenger seat, based on sensor data obtained from one or more sensors of the vehicle 800.
[0169] In some examples, one or more sensors may be seat sensors, such as one or more cameras, one or more seatbelt sensors, one or more headrest sensors, one or more seat back sensors, one or more seat bottom sensors, and / or one or more elbow rest sensors. Depending on the specific implementation, if one or more seat sensors include one or more cameras, the cameras may or may not be attached to the seat. For example, each of the one or more cameras may be attached to a portion of the vehicle interior near the seat, such as the dashboard, windshield, rearview mirror, steering wheel, etc., and may be positioned to obtain an image of a person in any of the seats 805a to 804d.
[0170] According to some such implementations, if sensor data indicates that a person is sitting in a seat, the listening position can be assumed to correspond to the seat position (and / or headrest position), and the assumed listening orientation of the person can be correlated with the orientation of the seat. In some implementations, the vehicle control system can determine the person's listening position based on the position of their head. In some examples, the position of the person's head can be determined based on the headrest position. According to some examples, the vehicle control system can determine the person's listening orientation based on the orientation of the seat the person is sitting in. In some implementations (such as...) Figure 8 In the example shown, all seats 805a to 804d face forward. Therefore, the vehicle control system can determine that the orientation of the person in any of the seats 805a to 804d is facing forward.
[0171] However, in some implementations, the vehicle control system may determine the position and / or orientation of a person (e.g., the person's head) at least in part based on the position of the seat back. For example, the vehicle control system may determine (e.g., based on seat sensor data or from seat mechanisms used for positioning the seat (including, but not limited to, seat mechanisms used for adjusting the seat back angle)) whether the person's seat back is in an upright position, a reclined position, etc., and may thus determine the person's position and / or orientation.
[0172] Furthermore, in some alternative implementations, one or more seats in the vehicle may be configured to rotate so that one or more seats in the vehicle face the side of the vehicle, the rear of the vehicle, etc. In some such implementations, the vehicle control system may determine the position and / or orientation of a person (e.g., the person's head) based at least in part on a determined seat rotation angle (e.g., based on seat sensor data). As autonomous vehicles become more popular and accepted by consumers, in some instances, even a person sitting in what is typically the driver's seat of a vehicle may not always be facing forward.
[0173] Figure 9 An example box of another disclosed implementation is shown. (Compared to other implementations provided herein...) Figure 1 Sample, Figure 9 The types, quantities, and arrangements of the elements shown are provided by way of example only. Other implementations may include more, fewer, and / or different types, quantities, and / or arrangements of elements. In this example, elements 310a, 310b, 410, 415, 420, 905, 915a, and 915b are implemented via an instance of the control system 110 of device 100 (which in this instance is a vehicle control system). In some such examples, elements 310a, 310b, 410, 415, 420, 905, 915a, and 915b may be implemented by control system 110 according to instructions stored on one or more non-transitory computer-readable media, which in some instances may correspond to one or more memory devices of memory system 115.
[0174] In this example, the encoded spatial audio stream 305 is received and decoded by the decoder 905, and the decoded spatial audio stream 907 is provided to the filter bank analysis block 405. Here, as in... Figure 6A In this example, only a single instance of filter bank analysis block 405 is implemented, and filter bank analysis is applied to the input decoded spatial audio stream 907, rather than to the speaker feed of the spatial audio renderer. According to this example, filter bank analysis block 405 generates a global set of frequency bands 607 corresponding to the audio data of the decoded spatial audio stream 907. In this context, the "frequency band" generated by filter bank analysis block 405 is a frequency domain representation of the audio data of the decoded spatial audio stream 907 within each of a set of frequency ranges. In this example, filter bank analysis block 405 is the one referenced above. Figure 4 The description of a single instance of filter bank analysis blocks 405a to 405n is so specific that the description will not be repeated here.
[0175] In this embodiment, the frequency band selection block 410 has the same characteristics as described above. Figure 4 The described frequency band selection blocks 410a to 410n function similarly. However, in this embodiment, frequency band selection block 410 is configured to select two renderer-specific frequency band sets 617a and 617b from the global frequency band set 607. In this example, frequency band selection block 410 is configured to provide renderer-specific frequency band set 617a to spatial audio renderer 310a and renderer-specific frequency band set 617b to spatial audio renderer 310b. Therefore, for each of spatial audio renderers 310a and 310b, only the renderer-specific frequency bands of the spatial audio stream belonging to its selected frequency band subset are processed to generate speaker feeds for these frequency bands, thus similar to the above reference. Figure 3 and Figure 4The described operations potentially reduce the complexity of the operations performed by each of the spatial audio renderers 310a and 310b.
[0176] In this example, it will be with Figure 8 Listener position data 910a corresponding to listener 1 is provided to a translation coefficient generation block 915a, which is configured to generate translation coefficients corresponding to the listener position data 910a and provide the translation coefficients to a spatial audio renderer 310a. In some embodiments, listener position data 910a may include both listener position and listener orientation data. In some such examples, depending on the capabilities of seat 805a, the listener orientation data may indicate that listener 1 is facing forward.
[0177] Based on this example, it will be with Figure 8 Listener position data 910b corresponding to listener 2 is provided to a translation coefficient generation block 915b, which is configured to generate translation coefficients corresponding to the listener position data 910b and provide the translation coefficients to a spatial audio renderer 310b. In some embodiments, listener position data 910b may include both listener position and listener orientation data. In some such examples, depending on the capabilities of seat 805b, the listener orientation data may indicate that listener 2 is facing forward. According to some examples, listener position data 910a and listener position data 910b may be or may be based on vehicle sensor data, such as seat sensor data.
[0178] Some alternative implementations may omit the translation coefficient generation blocks 915a and 915b, which are separate from the spatial audio renderers 310a and 310b. In some such implementations, listener position data 910a may be provided to the spatial audio renderer 310a, and listener position data 910a may be provided to the spatial audio renderer 310b. According to some such examples, spatial audio renderer 310a may be configured to generate translation coefficients corresponding to listener position data 910a, and spatial audio renderer 310b may be configured to generate translation coefficients corresponding to listener position data 910b.
[0179] If the input spatial audio stream 305 includes spatial metadata, in some implementations that spatial metadata is also provided to the spatial audio renderers 310a and 310b. In some such examples, the spatial metadata may accompany each set in the global band set 607 and the renderer-specific band sets 617a and 617b.
[0180] According to this example, control system 110 is configured to implement the above reference. Figure 4The described combination block 415 is configured to combine renderer-specific loudspeaker feed signals 317a and 317b output by spatial audio renderers 310a and 310b to generate a loudspeaker feed signal output set 417 in the frequency domain. In some examples, combination block 415 may be configured to combine renderer-specific loudspeaker feed signals 317a and 31b via a multiplexing process. In this example, control system 110 is configured to implement filter combination block 420, which is configured to transform the loudspeaker feed signal output set 417 from the frequency domain to the time domain, such that loudspeaker feed signal output set 325 is in the time domain. In some examples, loudspeaker feed signal output set 325 may be provided to a group of loudspeakers in vehicle 800. According to some embodiments, loudspeaker feed signal output set 325 may be played back by this group of loudspeakers in vehicle 800.
[0181] Figure 10 This is a flowchart outlining an example of a method that can be performed by a device or system such as the apparatus or system disclosed herein. As with other methods described herein, the blocks of method 1000 need not be performed in the indicated order. Furthermore, such a method may include more or fewer blocks than those shown and / or described. The blocks of method 1000 may be performed by one or more devices, which may be (or may include) a control system, such as... Figure 8 The control system 110 shown and described above, or one of other disclosed control system examples.
[0182] In this embodiment, block 1005 relates to receiving audio data by a control system. In some embodiments, the control system may be or may include a vehicle control system. In some examples, the audio data may include an audio signal and associated spatial data, for example, as referenced above. Figure 3 , Figure 4 Figure 6 and Figure 9 The spatial audio stream 305 describes this. Therefore, in some examples, the audio data may include spatial channel-based audio data and / or spatial object-based audio data. In some instances, the audio data may have one of the following audio formats: stereo, Dolby 3.1.2, Dolby 5.1, Dolby 5.1.2, Dolby 7.1, Dolby 7.1.2, Dolby 7.1.4, Dolby 9.1, Dolby 9.1.6, or Dolby Atmos. TM .
[0183] According to this example, box 1010 relates to the control system receiving sensor signals indicating the presence of multiple people in the vehicle. In some instances, the sensor signals may be or include signals from one or more seat sensors. In some such examples, the seat sensors may include one or more cameras, one or more seatbelt sensors, one or more headrest sensors, one or more seat back sensors, one or more seat bottom sensors, and / or one or more elbow rest sensors. However, in some alternative examples, the sensor signals may be or may include signals from one or more doors of the vehicle, signals from one or more non-seat surfaces of the vehicle (e.g., one or more dashboard surfaces, interior panel surfaces, floor surfaces, ceiling surfaces, steering wheel surfaces), etc. For example, the sensors may include one or more cameras, one or more pressure sensors, one or more touch sensors, one or more motion sensors, and / or one or more microphones.
[0184] According to this embodiment, block 1015 relates to estimating multiple listening configurations by a control system and at least in part based on sensor signals. In this example, each listening configuration corresponds to the listening position and listening orientation of a person among a plurality of people. In some instances, the listening position and listening orientation may be relative to a vehicle coordinate system. According to some examples, the listening configuration may be relative to the position of one or more loudspeakers in the vehicle. In some such examples, the listening position may correspond to a head position. According to some such examples, the listening orientation may correspond to a head orientation. In some embodiments, at least one listening configuration may be associated with a person's identity and stored in memory. For example, in some such embodiments, the head position and / or orientation may correspond to a saved preset seat position and / or orientation for a particular individual. The memory may be vehicle memory accessible by the control system or remote memory, such as the memory of a server for implementing cloud-based services.
[0185] In this example, block 1020 relates to the control system rendering received audio data for each of a plurality of listening configurations to generate a loudspeaker feed signal output set. In this embodiment, block 1025 relates to the control system providing the loudspeaker feed signal output set to a plurality of loudspeakers in a vehicle. In some examples, method 1000 may involve causing the plurality of loudspeakers to reproduce the loudspeaker feed signal output set.
[0186] In some examples, the rendering of the audio data may be performed by multiple renderers. In some such examples, each of the multiple renderers may be configured to render the audio data for different listening configurations to obtain a renderer-specific loudspeaker feed signal set. In some such examples, method 1000 may involve the control system decomposing each renderer-specific loudspeaker feed signal set into a renderer-specific frequency band set for each renderer. In some instances, each set in the renderer-specific frequency band set may be uniquely associated with one of the multiple renderers. In some instances, each set in the renderer-specific frequency band set may be uniquely associated with one of the multiple listening configurations. In some instances, rendering may involve rendering in the time domain (e.g., performing double-balanced amplitude shift in the time domain) or rendering in the frequency domain (e.g., crosstalk cancellation in the frequency domain).
[0187] In some such examples, method 1000 may involve the control system combining the renderer-specific frequency band sets of each renderer to produce a loudspeaker feed signal output set. In some instances, combining the multiple sets of renderer-specific frequency bands may involve multiplexing the renderer-specific frequency band sets. In some such examples, method 1000 may involve the control system outputting the loudspeaker feed signal output set.
[0188] In some examples, decomposing the renderer-specific loudspeaker feed signal set into the renderer-specific frequency band set may involve: analyzing the renderer-specific loudspeaker feed signal set by an analysis filter bank associated with each renderer to generate a global frequency band set; and selecting a subset of the global frequency band set to generate the renderer-specific frequency band set. In some such examples, a subset of the global frequency band set may be selected such that when the renderer-specific frequency bands of each of the plurality of renderers are combined, each frequency band in the global frequency band set is represented only once in the loudspeaker feed signal output set.
[0189] According to some examples, combining the multiple renderer-specific frequency bands may involve synthesizing the loudspeaker feed signal output set in the time domain by a synthesis filter bank. In some examples, the analysis filter bank and / or the synthesis filter bank may be a short-time discrete Fourier transform (STDFT) filter bank, a hybrid composite quadrature mirror (HCQMF) filter bank, or a quadrature mirror (QMF) filter bank.
[0190] In some alternative examples, the rendering of audio data may also be performed by multiple renderers. In some such examples, each of the multiple renderers may be configured to render the audio data for different listening configurations to obtain a renderer-specific loudspeaker feed signal set. In some such examples, method 1000 may involve analyzing the received audio by an analysis filter bank implemented by the control system to generate a global frequency band set of the received audio data. In some such examples, method 1000 may involve the control system selecting a subset of the global frequency band set for each of the multiple renderers to generate a renderer-specific frequency band set for each renderer. In some such examples, method 1000 may involve each of the multiple renderers rendering the renderer-specific frequency band set to obtain a loudspeaker feed signal set for a corresponding listening configuration. According to some implementations, each renderer-specific frequency band set may be uniquely associated with one renderer. In some implementations, each renderer-specific frequency band set may be uniquely associated with one listening configuration.
[0191] In some implementations, the rendering may involve generating, or generating for each renderer, a set of coefficients corresponding to the listening configuration. These coefficients can be used in the rendering. In some instances, the coefficients may be translation coefficients.
[0192] Some examples may involve selecting a rendering mode from multiple rendering modes. In some such examples, each rendering mode may be based on a corresponding listening configuration among multiple listening configurations. In some examples, at least one listening configuration may be associated with a person's identity and stored in memory. According to some such examples, the memory may be vehicle memory. In other examples, the memory may be remote memory accessible by a control system, such as the memory of a server for implementing cloud-based services.
[0193] In some examples, method 1000 may involve combining a set of loudspeaker feed signals from each renderer to produce a set of loudspeaker feed signal outputs. According to some examples, combining the set of loudspeaker feed signals from each renderer may involve multiplexing the set of loudspeaker feed signals from each renderer. In some examples, method 1000 may involve outputting the set of loudspeaker feed signal outputs.
[0194] According to some such examples, combining the loudspeaker feed signal set may involve synthesizing the loudspeaker feed signal output set in the time domain by a synthesis filter bank. In some examples, the synthesis filter bank or the analysis filter bank may be a short-time discrete Fourier transform (STDFT) filter bank, a hybrid composite quadrature mirror (HCQMF) filter bank, or a quadrature mirror (QMF) filter bank.
[0195] The listening position and orientation associated with each of the multiple listening configurations can be obtained through many mechanisms known in the art. In some applications, such as a car cab, these positions and orientations are fixed and can be physically measured, for example, using a measuring tape or according to a CAD design. Figures 2A to 2B Other applications in the home environment illustrated may require more adaptive methods that can detect these positions and orientations dynamically and automatically through a single setup procedure or even over time. In Hess, Wolfgang, Head-Tracking Techniques for Virtual Acoustic Applications (AES 133rd Convention, October 2012), cited herein, numerous commercially available techniques are presented for tracking both the position and orientation of a listener's head in the context of a spatial audio reproduction system. A specific example discussed is the Microsoft Kinect. Utilizing its depth sensing and standard camera, along with publicly available software (Windows Software Development Kit (SDK)), a combination of skeletal tracking and facial recognition can be used to simultaneously track the position and orientation of the heads of multiple listeners in space. Although the Windows Kinect is discontinued, the Azure Kinect Development Kit (DK), which implements Microsoft's next-generation depth sensor, is currently available.
[0196] U.S. Patent No. 10,779,084, entitled "Automatic Discovery and Localization of Speaker Locations in Surround Sound Systems," which is hereby incorporated by reference, describes a system for automatically locating loudspeakers and microphones in a listening environment by acoustically measuring the time of arrival (TOA) between each speaker and microphone. The listening location can be detected by placing and positioning a microphone (e.g., a microphone in a mobile phone held by the listener) at the desired listening location, and the associated listening direction can be defined by placing another microphone at a point in the listener's viewing direction (e.g., at a TV). Alternatively, the listening direction can be defined by positioning a loudspeaker (e.g., a loudspeaker on a TV) in the viewing direction.
[0197] This is cited in connection with Shi, Guangi et al., Spatial Calibration of SurroundSound Systems including Listener Position Estimation, (AES 137). th A system described in the Convention (October 2014) involves a single linear microphone array associated with components of a predictably positioned reproduction system (such as a soundbar or front center speaker) measuring the time difference of arrival (TDOA) between a satellite loudspeaker and a listener to pinpoint their locations. In this case, the listening direction is inherently defined as a line connecting the detected listening location to components of the reproduction system, including the linear microphone array (such as a soundbar positioned co-located with the television (placed directly above or below the television)). Since the soundbar is predictably positioned directly above or below the video screen, the geometry of the measured distance and angle of incidence can be converted to an absolute position relative to any point in front of this reference soundbar location using simple trigonometry. The distance between the loudspeaker and microphones of the linear microphone array can be estimated by playing a test signal and measuring the time of flight (TOF) between the transmitting loudspeaker and the receiving microphone. The time delay of the direct component of the measured impulse response can be used for this purpose. The impulse response between the loudspeaker and the microphone array elements can be obtained by playing the test signal via the loudspeaker being analyzed. For example, a maximum length sequence (MLS) or chirped signal (also known as a logarithmic sinusoidal sweep) can be used as the test signal. The room impulse response can be obtained by calculating the cyclic cross-correlation between the captured signal and the MLS input. Figure 2 of this reference shows the echo impulse response obtained using the MLS input. This impulse response is described as analogous to measurements taken in a typical office or living room. The delay of the direct component is used to estimate the distance between the loudspeaker and the microphone array elements. For loudspeaker distance estimation, any loop-back delay of the audio equipment used to play back the test signal should be calculated and removed from the measured TOF estimate.
[0198] As can be seen, there are many mechanisms that can obtain multiple listening configurations, listening positions, and orientations, and all such methods (as well as related future methods that can be developed) are intended to be applicable to embodiments of this disclosure. Therefore, the specific details disclosed herein should be considered merely as examples.
[0199] Figure 11An example of the geometric relationship between four audio devices in an environment is shown. In this example, audio environment 1100 is a room that includes a television 1101 and audio devices 1105a, 1105b, 1105c, and 1105d. According to this example, audio devices 1105a to 1105d are located at positions 1 to 4 of audio environment 1100, respectively. Similar to other examples disclosed herein, Figure 11 The types, quantities, positions, and orientations of the elements shown are merely examples. Other implementations may have elements of different types, quantities, and arrangements, such as more or fewer audio devices, audio devices in different locations, audio devices with different capabilities, etc.
[0200] In this embodiment, each of the audio devices 1105a to 1105d is a smart speaker comprising a microphone system and a speaker system, the speaker system including at least one speaker. In some embodiments, each microphone system includes an array of at least three microphones. According to some embodiments, the television 1101 may include a speaker system and / or a microphone system. In some such embodiments, an automatic positioning method may be used to automatically position the television 1101 or a portion thereof (e.g., a television amplifier, a television transceiver, etc.), for example, as described below with reference to audio devices 1105a to 1105d.
[0201] Some embodiments described in this disclosure allow a group of audio devices (such as...) to be configured based on the direction of arrival (DOA) between each pair of audio devices, the time of arrival (TOA) of the audio signal between each pair of devices, or both the DOA and TOA of the audio signal between each pair of devices. Figure 11 The automatic positioning of audio devices 1105a to 1105d shown in the figure. In some instances, such as in Figure 11 In the example shown, each of the audio devices is enabled with at least one driver unit and a microphone array, the microphone array being able to provide the direction of arrival of incoming sound. According to this example, double-headed arrow 1110ab represents sound transmitted by audio device 1105a and received by audio device 1105b, and sound transmitted by audio device 1105b and received by audio device 1105a. Similarly, double-headed arrows 1110ac, 1110ad, 1110bc, 1110bd, and 1110cd represent sound transmitted and received by audio devices 1105a and 1105c, sound transmitted and received by audio devices 1105a and 1105d, sound transmitted and received by audio devices 1105b and 1105c, sound transmitted and received by audio devices 1105b and 1105d, and sound transmitted and received by audio devices 1105c and 1105d, respectively.
[0202] In this example, each of the audio devices 1105a to 1105d has an orientation, represented by arrows 1115a to 1115d, which can be defined in various ways. For example, the orientation of an audio device with a single loudspeaker can correspond to the direction the single loudspeaker faces. In some examples, the orientation of an audio device with multiple loudspeakers facing different directions can be indicated by the direction one of the loudspeakers faces. In other examples, the orientation of an audio device with multiple loudspeakers facing different directions can be indicated by the direction of a vector corresponding to the sum of the different directions in which the audio output is faced by each of the multiple loudspeakers. Figure 11 In the example shown, the orientation of arrows 1115a to 1115d is defined with reference to a Cartesian coordinate system. In other examples, the orientation of arrows 1115a to 1115d may be defined with reference to another type of coordinate system (such as a spherical or cylindrical coordinate system).
[0203] In this example, television 1101 includes an electromagnetic interface 1103 configured to receive electromagnetic waves. In some examples, electromagnetic interface 1103 may be configured to transmit and receive electromagnetic waves. According to some embodiments, at least two of audio devices 1105a to 1105d may include an antenna system configured as a transceiver. The antenna system may be configured to transmit and receive electromagnetic waves. In some examples, the antenna system includes an antenna array having at least three antennas. Some embodiments described in this disclosure allow a group of devices (such as...) to be based at least in part on the DOA of electromagnetic waves transmitted between devices. Figure 11 The automatic positioning of audio devices 1105a to 1105d and / or television 1101 shown in the figure. Therefore, the double-headed arrows 1110ab, 1110ac, 1110ad, 1110bc, 1110bd and 1110cd can also represent electromagnetic waves transmitted between audio devices 1105a to 1105d.
[0204] According to some examples, the antenna system of a device (such as an audio device) may be co-located with the device's loudspeaker, for example, adjacent to the loudspeaker. In some such examples, the antenna system orientation may correspond to the loudspeaker orientation. Alternatively or additionally, the device's antenna system may have a known or predetermined orientation relative to one or more loudspeakers of the device.
[0205] In this example, audio devices 1105a to 1105d are configured to communicate wirelessly with each other and with other devices. In some examples, audio devices 1105a to 1105d may include a network interface configured to communicate with other devices via the Internet. In some embodiments, the automatic location process disclosed herein may be performed by a control system of one of the audio devices 1105a to 1105d. In other examples, the automatic location process may be performed by another device of the audio environment 1100 (such as a device sometimes referred to as a smart home hub), which is configured to communicate wirelessly with the audio devices 1105a to 1105d. In other examples, the automatic location process may be performed at least in part by a device outside the audio environment 1100 (such as a server), for example, based on information received from one or more of the audio devices 1105a to 1105d and / or the smart home hub.
[0206] Figure 12 It shows the location Figure 11 An audio transmitter within an audio environment. Some implementations provide one or more audio transmitters (such as...) Figure 12 Automatic positioning of person 1205. In this example, person 1205 is at location 5. Here, the sound emitted by person 1205 and received by audio device 1105a is represented by single-headed arrow 1210a. Similarly, the sound emitted by person 1205 and received by audio devices 1105b, 1105c, and 1105d is represented by single-headed arrows 1210b, 1210c, and 1210d. The audio transmitter can be positioned based on the DOA of the audio transmitter sound captured by audio devices 1105a to 1105d and / or television 1101, the TOA difference of the audio transmitter sound measured by audio devices 1105a to 1105d and / or television 1101, or both the DOA and the TOA difference.
[0207] Alternatively or additionally, some implementations may provide automatic positioning of one or more electromagnetic wave transmitters. Some embodiments described in this disclosure allow automatic positioning of one or more electromagnetic wave transmitters, at least in part, based on the DOA of electromagnetic waves emitted by the one or more electromagnetic wave transmitters. If the electromagnetic wave transmitter is at position 5, the electromagnetic waves emitted by the electromagnetic wave transmitter and received by audio devices 1105a, 1105b, 1105c, and 1105d may also be represented by single-headed arrows 1210a, 1210b, 1210c, and 1210c.
[0208] Figure 13 It shows the location Figure 11An audio receiver within an audio environment. In this example, the microphone of smartphone 1305 is enabled, but the speaker of smartphone 1305 is not currently emitting sound. Some embodiments provide for one or more passive audio receivers (such as...) when smartphone 1305 is not emitting sound. Figure 13 Automatic positioning of the smartphone 1305. Here, the sound emitted by the audio device 1105a and received by the smartphone 1305 is represented by a single-headed arrow 1310a. Similarly, the sounds emitted by the audio devices 1105b, 1105c, and 1105d and received by the smartphone 1305 are represented by single-headed arrows 1310b, 1310c, and 1310d.
[0209] If the audio receiver is equipped with a microphone array and configured to determine the DOA of the received sound, the audio receiver can be located at least in part based on the DOA of the sound emitted by the audio devices 1105a to 1105d and captured by the audio receiver. In some examples, the audio receiver can be located at least in part based on the TOA difference of the smart audio devices, such as that captured by the audio receiver, regardless of whether the audio receiver is equipped with a microphone array. Other embodiments may allow automatic localization of a group of smart audio devices, one or more audio transmitters, and one or more receivers by combining the methods described above based solely on DOA or based on both DOA and TOA.
[0210] Arrival Direction Positioning
[0211] Figure 14 It outlines what can be made by, for example Figure 1 The flowchart illustrates an example of a method performed by the control system of the apparatus shown. As with other methods described herein, the blocks of method 1400 need not be performed in the indicated order. Furthermore, such a method may include more or fewer blocks than those shown and / or described.
[0212] Method 1400 is an example of an audio device localization process. In this example, method 1400 involves determining the location and orientation of two or more smart audio devices, each of which includes a loudspeaker system and a microphone array. According to this example, method 1400 involves determining the location and orientation of the smart audio devices based at least in part on audio emitted by each smart audio device and captured by each other smart audio device, according to a DOA estimation. In this example, the initial framework of method 1400 relies on the ability of the control system of each smart audio device to extract the DOA from the input audio obtained by the microphone array of that smart audio device, for example, by using the time difference of arrival between the individual microphone diaphragms of the microphone array.
[0213] In this example, box 1405 relates to obtaining audio emitted by each smart audio device in the audio environment and captured by each other smart audio device in the audio environment. In some such examples, box 1405 may relate to causing each smart audio device to emit sound, which in some instances may be a sound having a predetermined duration, frequency content, etc. This predetermined type of sound may be referred to herein as a structured source signal. In some implementations, the smart audio devices may be or may include Figure 11 Audio devices 1105a to 1105d.
[0214] In some such examples, box 1405 may relate to a sequential process in which a single smart audio device emits sound and other smart audio devices “listen” to the sound. For example, see reference Figure 11 Box 1405 may involve: (a) causing audio device 1105a to emit sound and receiving microphone data corresponding to the emitted sound from the microphone arrays of audio devices 1105b to 1105d; then (b) causing audio device 1105b to emit sound and receiving microphone data corresponding to the emitted sound from the microphone arrays of audio devices 1105a, 1105c, and 1105d; then (c) causing audio device 1105c to emit sound and receiving microphone data corresponding to the emitted sound from the microphone arrays of audio devices 1105a, 1105b, and 1105d; then (d) causing audio device 1105d to emit sound and receiving microphone data corresponding to the emitted sound from the microphone arrays of audio devices 1105a, 1105b, and 1105c. Depending on the specific implementation, the emitted sounds may be the same or different.
[0215] In other examples, box 1405 may involve a synchronization process that causes all smart audio devices to emit sound while other smart audio devices “listen” to the sound. For example, block 1405 may involve simultaneously performing the following steps: (1) causing audio device 1105a to emit a first sound and receiving microphone data corresponding to the emitted first sound from the microphone arrays of audio devices 1105b to 1105d; (2) causing audio device 1105b to emit a second sound different from the first sound and receiving microphone data corresponding to the emitted second sound from the microphone arrays of audio devices 1105a, 1105c and 1105d; (3) causing audio device 1105c to emit a third sound different from the first and second sounds and receiving microphone data corresponding to the emitted third sound from the microphone arrays of audio devices 1105a, 1105b and 1105d; (4) causing audio device 1105d to emit a fourth sound different from the first, second and third sounds and receiving microphone data corresponding to the emitted fourth sound from the microphone arrays of audio devices 1105a, 1105b and 1105c.
[0216] In some examples, box 1405 can be used to determine the interaudibility of audio devices in an audio environment. Some detailed examples are disclosed in this document.
[0217] In this example, box 1410 relates to a process of preprocessing an audio signal obtained via a microphone. For example, box 1410 may involve applying one or more filters, noise or echo suppression processes, etc. Some additional preprocessing examples are described below.
[0218] According to this example, box 1415 relates to determining DOA candidates from the preprocessed audio signal generated by box 1410. For example, if box 1405 relates to transmitting and receiving structured source signals, then box 1415 may relate to one or more deconvolution methods for generating impulse responses and / or “pseudo-ranges,” according to which the arrival time difference of major peaks, along with the known microphone array geometry of the smart audio device, can be used to estimate DOA candidates.
[0219] However, not all implementations of method 1400 involve obtaining a microphone signal based on the emission of a predetermined sound. Therefore, some examples of block 1415 include “blind” methods applied to arbitrary audio signals, such as controlled response power, receiver-side beamforming, or other similar methods, according to which one or more DOAs can be extracted by peak picking. Some examples are described below. It will be understood that while DOA data can be determined via blind methods or using structured source signals, in most instances, only structured source signals can be used to determine DOA data. Furthermore, structured source signals are often used to obtain more accurate DOA information.
[0220] According to this example, box 1420 relates to selecting a DOA corresponding to the sound emitted by each of the other smart audio devices. In many instances, the microphone array can detect both directly arriving and reflected sounds emitted by the same audio device. Box 1420 may relate to selecting the audio signal most likely to correspond to the directly emitted sound. Some additional examples of determining DOA candidates and selecting a DOA from two or more candidate DOAs are described below.
[0221] In this example, block 1425 relates to receiving DOA information generated by the implementation of each smart audio device in block 1420 (in other words, receiving a set of DOAs corresponding to the sounds transmitted from each smart audio device to each other smart audio device in the audio environment) and performing a localization method based on the DOA information (e.g., implementing a localization algorithm via a control system). In some disclosed implementations, block 1425 relates to minimizing a cost function that may be subject to some constraints and / or weights, for example, as referenced below. Figure 15 As described. In some such examples, the cost function takes the DOA value from each smart audio device to each other smart device as input data and returns the estimated location and estimated orientation of each of the smart audio devices as output. Figure 14 In the example shown, box 1430 represents the estimated smart audio device location and estimated smart audio device orientation generated in box 1425.
[0222] Figure 15 This is a flowchart outlining another example of a method for automatically estimating device location and orientation based on DOA data. For example, method 1500 can be performed via, for instance, through methods such as Figure 1 The control system of the device shown implements a positioning algorithm to perform this action. As with other methods described herein, the blocks of method 1500 need not be performed in the indicated order. Furthermore, such a method may include more or fewer blocks than those shown and / or described.
[0223] According to this example, DOA data is obtained in box 1505. According to some implementations, box 1505 may involve obtaining acoustic DOA data, for example, as referenced above. Figure 14 The blocks 1405 to 1420 are described. Alternatively or additionally, block 1505 may relate to obtaining DOA data corresponding to electromagnetic waves transmitted and received by each of a plurality of devices in the environment.
[0224] In this example, the localization algorithm receives DOA data from each smart device to each other smart device in the audio environment within box 1505, along with any configuration parameters 1510 specified for the audio environment, as input. In some examples, optional constraints 1525 may be applied to the DOA data. For example, configuration parameters 1510, minimum weights 1515, optional constraints 1525, and seed layout 1530 may be obtained from memory by a control system, which is execution software for implementing the cost function 1520 and the nonlinear search algorithm 1535. For example, configuration parameters 1510 may include data corresponding to external inputs such as maximum room size, loudspeaker layout constraints, setting global translation (e.g., two parameters), global rotation (one parameter), and global scaling (one parameter).
[0225] According to this example, configuration parameter 1510 is provided to cost function 1520 and nonlinear search algorithm 1535. In some examples, configuration parameter 1510 is provided to optional constraint 1525. In this example, cost function 1520 considers the difference between the measured DOA and the DOA estimated by the positioning solution through the optimizer.
[0226] In some embodiments, optional constraint 1525 imposes restrictions on possible audio device locations and / or orientations, such as imposing a condition that minimizes the distance between audio devices. Alternatively or additionally, optional constraint 1525 may impose restrictions on dummy minimization variables that are conveniently introduced, for example, as described below.
[0227] In this example, the weight 1515 is also minimized and fed into the nonlinear search algorithm 1535. Several examples are described below.
[0228] According to some implementations, the nonlinear search algorithm 1535 is an algorithm that can find local solutions to continuous optimization problems of the following form:
[0229] min C(x)
[0230] xeC n
[0231] Make g L ≤g(x)≤g U
[0232] And x L ≤x≤x U
[0233] In the aforementioned expression, C(x): R n ->R represents the cost function 1520, and g(x):R n ->R m This represents the constraint function corresponding to optional constraint 1525. In these examples, vector g L and g U This represents the lower and upper bounds of the constraint, and the vector x L and x U It represents the boundary of variable x.
[0234] The nonlinear search algorithm 1535 can vary depending on the specific implementation. Examples of the nonlinear search algorithm 1535 include gradient descent, the Broyden-Fletcher-Gold-Sano (BFGS) method, interior-point optimization (IPOPT) methods, etc. While some nonlinear search algorithms only require the values of the cost function and constraints, some other methods may also require the first derivatives (gradient, Jacobian) of the cost function and constraints, and some other methods may also require the second derivatives (Hessians) of the same function. If derivatives are required, they can be explicitly provided, or they can be automatically calculated using automatic or numerical differentiation techniques.
[0235] Some nonlinear search algorithms require seed point information to begin minimization, such as Figure 15The seed layout 1530 provided to the nonlinear search algorithm 1535 is as suggested. In some examples, seed point information can be provided as a layout consisting of the same number of smart audio devices with corresponding positions and orientations (in other words, the same number as the actual number of smart audio devices that obtained the DOA data). The positions and orientations can be arbitrary and do not need to be the actual or approximate positions and orientations of the smart audio devices. In some examples, the seed point information can indicate the positions of smart audio devices along an axis or another arbitrary line of the audio environment, the positions of smart audio devices along circles, rectangles, or other geometric shapes within the audio environment, etc. In some examples, the seed point information can indicate any smart audio device orientation, which can be a predetermined smart audio device orientation or a random smart audio device orientation.
[0236] In some embodiments, the cost function 1520 can be formulated using complex plane variables as follows:
[0237]
[0238] Wherein, the asterisk indicates the complex conjugate, the bar indicates the absolute value, and where:
[0239] ·Z nm =exp(iDOA nm ) represents the complex plane value of the direction of arrival of intelligent device m as measured from device n, where i represents the imaginary unit;
[0240] ·x n =x nx +ix ny This represents the complex plane value used to encode the x and y positions of the smart device n;
[0241] ·z n =exp(iα) n α represents the orientation angle of the smart device n. n The composite value to be encoded;
[0242] · Indicates giving DOA nm The weight of the measurement;
[0243] • N represents the number of smart audio devices that have obtained DOA data; and
[0244] ·x=(x1,...,x N ) and z = (z1, ..., z N ) represent the vectors of the composite position and composite orientation of all N smart audio devices, respectively.
[0245] Based on this example, the result of minimization is that the device location data 1540 indicates the 2D location x of the smart device. k(This represents two real unknowns per device) and the device orientation data 1545 indicates the orientation vector z of the smart device. k (This represents two additional real-valued variables per device). From the perspective of the orientation vector, only the smart device α... k The orientation angle is related to the problem (one real unknown per device). Therefore, in this example, there are three relevant unknowns per smart device.
[0246] In some examples, the outcome evaluation box 1550 involves calculating the residuals of the cost function at the outcome location and orientation. A relatively lower residual indicates a relatively more accurate device positioning value. According to some implementations, the outcome evaluation box 1550 may involve a feedback process. For example, some such examples may implement a feedback process involving comparing the residuals of a given DOA candidate combination with the residuals of another DOA candidate combination, as illustrated in the discussion of DOA robustness measures below.
[0247] As described above, in some embodiments, block 1505 may relate to obtaining acoustic DOA data, as referenced above to determine DOA candidates and select DOA candidates. Figure 14 The boxes 1405 to 1420 are described. Therefore, Figure 15 The dashed lines from result evaluation boxes 1550 to 1505 represent a flow of optional feedback procedures. Furthermore, Figure 14 The dashed line from box 1430 (which in some examples may involve outcome evaluation) to DOA candidate selection box 1420 indicates the flow of another optional feedback process.
[0248] In some embodiments, the nonlinear search algorithm 1535 may not accept complex-valued variables. In such cases, each complex-valued variable can be replaced by a pair of real-valued variables.
[0249] In some implementations, additional prior information regarding the availability or reliability of each DOA measurement may exist. In some such examples, a loudspeaker may be located using only a subset of all possible DOA elements. For example, a missing DOA element can be masked with its corresponding zero weight in the cost function. In some such examples, the weight w nm It can be zero or one; for example, zero for measurements that are lost or considered insufficiently reliable, and one for reliable measurements. In some other embodiments, the weight w nm It can have continuous values from zero to one, serving as a function of the reliability of the DOA measurement. In these embodiments where no prior information is available, the weight w nm It can be simply set to one.
[0250] In some implementations, the condition |z can be added. kThe constraint |x| = 1 (one condition per smart audio device) serves as a constraint to ensure the normalization of the vector indicating the orientation of the smart audio device. In other examples, these additional constraints may not be necessary, and the vector indicating the orientation of the smart audio device may not be normalized. Other implementations may add constraints on the proximity of the smart audio device, for example, indicating |x|. n -x m |≥D, where D is the minimum distance between smart audio devices.
[0251] Minimizing the cost function described above does not completely determine the absolute position and orientation of the smart audio devices. According to this example, the cost function remains invariant under global rotation (1 independent parameter), global translation (2 independent parameters), and global rescaling (1 independent parameter), thus affecting the position and orientation of all smart devices simultaneously. This global rotation, translation, and rescaling cannot be determined from minimizing the cost function. Different layouts associated with symmetry transformations are completely indistinguishable within this framework and are referred to as belonging to the same equivalence class. Therefore, the configuration parameters should provide a criterion that allows for a unique definition of the smart audio device layout representing the entire equivalence class. In some embodiments, it may be advantageous to choose a criterion such that the smart audio device layout definition is close to a reference frame of reference for listeners near the reference listening position. An example of such a criterion is provided below. In some other examples, the criterion may be purely mathematical and disconnected from real-world reference frames.
[0252] Symmetry disambiguation criteria may include: a reference position that fixes global translational symmetry (e.g., smart audio device 1 should be at the origin); and a reference orientation that fixes two-dimensional rotational symmetry (e.g., smart device 1 should be oriented to the region of the audio environment in front, such as television 1101). Figures 11 to 13 The parameters are: the position of device 2 in the minimization problem; and a reference distance with fixed global scaling symmetry (e.g., smart device 2 should be at a unit distance from smart device 1). In total, there are four parameters in this example that cannot be determined from the minimization problem and should be provided as external inputs. Therefore, in this example, there are 3N-4 unknowns that can be determined from the minimization problem.
[0253] As described above, in some examples, in addition to the group of smart audio devices, there may be one or more passive audio receivers equipped with microphone arrays, and / or one or more audio transmitters. In such cases, the localization process can use techniques based on DOA estimation to determine the location and orientation of the smart audio devices, the transmitter location, and the passive receiver location and orientation from the audio emitted by each smart audio device and each transmitter and captured by each other smart audio device and each passive receiver.
[0254] In some such examples, the localization process can continue in a manner similar to that described above. In some instances, the localization process can be based on the same cost function described above, which is shown below for the reader's convenience:
[0255]
[0256] However, if the localization process involves passive audio receivers and / or audio transmitters that are not audio receivers, the variables in the aforementioned equations need to be interpreted in a slightly different way. Now, N represents the total number of devices, including N... smart One smart audio device, N rec One passive audio receiver and N emit There are n transmitters such that N = N smart +N rec +N emit In some examples, weights It can have a sparse structure to mask data loss due to passive receiver or transmitter devices (or other audio sources without receivers, such as humans), such that if device n is an audio transmitter without a receiver, then for all m, And if device m is an audio receiver, then for all n, For both smart audio devices and passive receivers, both position and angle can be determined; however, for audio transmitters only, position can be obtained. The total number of unknowns is 3N. smart +3N rec +2N emit -4.
[0257] Combined arrival time and arrival direction positioning
[0258] In the following discussion, the differences between the DOA-based localization process described above and the combined DOA and TOA localization discussed in this section will be emphasized. It can be assumed that these details not explicitly given are the same as those in the DOA-based localization process described above.
[0259] Figure 16 This is a flowchart outlining an example of a method for automatically estimating device location and orientation based on DOA and TOA data. For example, method 1600 can be performed via, for example, Figure 1 The control system of the device shown implements a positioning algorithm to perform this action. As with other methods described herein, the blocks of method 1600 need not be performed in the indicated order. Furthermore, such a method may include more or fewer blocks than those shown and / or described.
[0260] According to this example, DOA data is obtained in blocks 1605 to 1620. According to some implementations, blocks 1605 to 1620 may involve obtaining acoustic DOA data from multiple smart audio devices, for example, as referenced above. Figure 14 The blocks 1405 to 1420 are described. In some alternative embodiments, blocks 1605 to 1620 may relate to obtaining DOA data corresponding to electromagnetic waves transmitted and received by each of a plurality of devices in the environment.
[0261] In this example, however, box 1605 also relates to obtaining TOA data. According to this example, the TOA data includes the measured TOA of the audio transmitted and received by each smart audio device in the audio environment (e.g., each pair of smart audio devices in the audio environment). In some embodiments involving the transmission of structured source signals, the audio used to extract the TOA data may be the same as the audio used to extract the DOA data. In other embodiments, the audio used to extract the TOA data may be different from the audio used to extract the DOA data.
[0262] According to this example, box 1616 relates to detecting TOA candidates in the audio data, and box 1618 relates to selecting a single TOA for each pair of smart audio devices from the TOA candidates. Some examples are described below.
[0263] Various techniques can be used to obtain TOA data. One approach is to use room-calibrated audio sequences, such as sweep frequencies (e.g., logarithmic sine tones) or maximum-length sequences (MLS). Alternatively, the aforementioned sequences can be used with the frequency band limited to a near-ultrasonic audio frequency range (e.g., 18 kHz to 24 kHz). Within this audio frequency range, the most standard audio equipment can emit and record sound, but such signals are imperceptible to humans because they are beyond normal human hearing. Some alternative implementations may involve recovering TOA elements from hidden signals within the primary audio signal, such as a direct-sequence spread spectrum signal.
[0264] Given a DOA dataset from each smart audio device to each other smart audio device and a TOA dataset from each pair of smart audio devices. Figure 16 The positioning method 1625 can be based on minimizing a specific cost function that may be subject to some constraints. In this example, Figure 16 The localization method 1625 receives the DOA and TOA values described above as input data and outputs estimated location data and orientation data 630 corresponding to the smart audio device. In some examples, the localization method 1625 may also output the playback and recording latency of the smart audio device, for example, until some global symmetries cannot be determined from the minimization problem. Some examples are described below.
[0265] Figure 17 This is a flowchart outlining another example of a method for automatically estimating device location and orientation based on DOA and TOA data. For example, method 1700 can be performed via, for example, Figure 1 The control system of the device shown implements a positioning algorithm to perform this operation. As with other methods described herein, the blocks of method 1700 need not be performed in the indicated order. Furthermore, such a method may include more or fewer blocks than those shown and / or described.
[0266] In addition to those described below, in some examples, boxes 1705, 1710, 1715, 1720, 1725, 1730, 1735, 1740, 1745, and 1750 can be referenced as above. Figure 15 The boxes 1505, 1510, 1515, 1520, 1525, 1530, 1535, 1540, 1545, and 1550 are described. However, in this example, relative to... Figure 15 The cost function 1520 and nonlinear optimization method 1535 are modified to operate on both DOA and TOA data. In some examples, TOA data for box 1708 can be obtained, as referenced above. Figure 16 As described. With Figure 15 Another difference in the process is that, in this example, the nonlinear optimization method 1735 also outputs recording and playback latency data 1747 corresponding to the smart audio device, for example, as described below. Therefore, in some implementations, the result evaluation block 1750 may involve evaluating both DOA data and / or TOA data. In some such examples, the operation of block 1750 may include a feedback process involving DOA data and / or TOA data. For example, some such examples may implement a feedback process involving comparing the residuals of a given TOA / DOA candidate combination with the residuals of another TOA / DOA candidate combination, for example, as illustrated in the discussion of TOA / DOA robustness measures below.
[0267] In some examples, the outcome evaluation box 1750 involves calculating the residuals of the cost function at the outcome location and orientation. Relatively lower residuals generally indicate relatively more accurate device positioning values. According to some implementations, the outcome evaluation box 1750 may involve a feedback process. For example, some such examples may implement a feedback process involving comparing the residuals of a given TOA / DOA candidate combination with the residuals of another TOA / DOA candidate combination, for example, as illustrated in the discussion of TOA and DOA robustness measures below.
[0268] therefore, Figure 16The dashed lines from box 630 (which in some examples may involve result evaluation) to DOA candidate selection box 1620 and TOA candidate selection box 1618 represent the flow of the optional feedback process. In some implementations, box 1705 may involve obtaining acoustic DOA data, as referenced above regarding determining and selecting DOA candidates. Figure 16 Boxes 1605 to 1620 are described. In some examples, box 1708 may relate to obtaining acoustic TOA data, as described above in the references concerning determining and selecting TOA candidates. Figure 16 The boxes described in 1605 to 1618. Although Figure 17 Not shown, but some optional feedback processes may involve reverting from result evaluation box 1750 to box 1705 and / or box 1708.
[0269] According to this example, the localization algorithm continues by minimizing a cost function that may be subject to some constraints, and can be described as follows. In this example, the localization algorithm receives DOA data 1705 and TOA data 1708, as well as configuration parameters 1710 specified for the listening environment and possibly some optional constraints 1725 as input. In this example, the cost function takes into account the difference between the measured DOA and the estimated DOA, and the difference between the measured TOA and the estimated TOA. In some embodiments, constraints 1725 impose restrictions on possible device locations, orientations, and / or delays, such as imposing a condition that the distance between audio devices is minimized and / or imposing a condition that some device delays should be zero.
[0270] In some implementations, the cost function can be formulated as follows:
[0271]
[0272] In the aforementioned equation, and k = (k1, ..., k N ) represent the vectors of the playback and recording devices for each device, respectively, and where W DOA and W TOA These represent the global weights (also known as pre-factors) of the minimized parts of DOA and TOA, respectively, reflecting the relative importance of each of the two terms. In some such examples, the TOA cost function can be formulated as:
[0273]
[0274] in,
[0275] TOA nm This represents the measured arrival time of a signal traveling from smart device m to smart device n;
[0276] · Indicates granting TOA nm The weight of the measurement; and
[0277] •c represents the speed of sound.
[0278] Each smart audio device has a maximum of 5 real-valued unknowns: device location x n (Each device has 2 real unknowns), device orientation α n (One real unknown per device) and recording and playback delays and k n (Two additional unknowns per device). From this, it appears that only device location and latency relate to the TOA portion of the cost function. In some implementations, the number of effective unknowns can be reduced if there are prior known constraints or relationships between latency.
[0279] In some examples, there may be additional prior information, such as regarding the availability or reliability of each TOA measurement. In some of these examples, weights... The weights can be zero or one; for example, zero for measurements that are unavailable (or considered insufficiently reliable) and one for reliable measurements. This allows device location to be estimated using only a subset of all possible DOA and / or TOA elements. In some other implementations, the weights can have continuous values from zero to one, for example, as a function of the reliability of the TOA measurement. In some examples where no prior reliability information is available, the weights can simply be set to one.
[0280] According to some implementations, one or more additional constraints may be imposed on the possible values of delay and / or the relationship between different delays themselves.
[0281] In some examples, the location of the audio device can be measured in standard units of length (such as meters), and delay and arrival time can be indicated in standard units of time (such as seconds). However, it is generally true that nonlinear optimization methods perform better when minimizing the variations of the different variables used in the process at the same level. Therefore, some implementations may involve rescaling the location measurement so that the variation of the smart device's location ranges between -1 and 1, and rescaling the delay and arrival time so that these values also range between -1 and 1.
[0282] Minimizing the cost function described above does not fully determine the absolute position, orientation, or latency of the smart audio device. TOA information provides an absolute distance scale, meaning the cost function is no longer invariant under scale transformations, but remains invariant under global rotation and global translation. Furthermore, latency is subject to additional global symmetry: if the same global quantity is added simultaneously to all playback and recording latency, the cost function remains unchanged. These global transformations cannot be determined from minimizing the cost function. Similarly, configuration parameters should provide a standard that allows for a unique definition of device layout representing the entire equivalence class.
[0283] In some examples, the symmetry disambiguation criteria may include the following: a reference position for fixing the global translational symmetry (e.g., smart device 1 should be at the origin); a reference orientation for fixing the two-dimensional rotational symmetry (e.g., smart device 1 should be oriented in front of the orientation); and a reference delay (e.g., the recording delay of device 1 should be zero). In total, in this example, there are four parameters that cannot be determined from the minimization problem and should be provided as external inputs. Therefore, there are 5N-4 unknowns that can be determined from the minimization problem.
[0284] In some implementations, in addition to the set of smart audio devices, there may be one or more passive audio receivers that may not be equipped with a properly functioning microphone array and / or one or more audio transmitters. Including latency as a minimized variable allows some disclosed methods to locate receivers and transmitters whose transmission and reception times are not precisely known. In some such implementations, the TOA cost function described above can be implemented. For the reader's convenience, this cost function is shown again below:
[0285]
[0286] As described above with reference to the DOA cost function, if the cost function is used for location estimation involving passive receivers and / or transmitters, the cost function variables need to be described in a slightly different way. Now, N represents the total number of devices, including N... smart One smart audio device, N rec One passive audio receiver and N emit There are n transmitters such that N = N smart +N rec +N emit Weight It can have a sparse structure to mask data loss due to passive receivers or transmitters alone, for example, such that if device n is an audio transmitter, then for all m, And if device m is an audio receiver, then for all n, According to some implementations, for smart audio devices, location, orientation, and recording and playback delays must be determined; for passive receivers, location, orientation, and recording delays must be determined; and for audio transmitters, location and playback delays must be determined. According to some such examples, the total number of unknowns is therefore 5N. smart +4N rec +3N emit -4.
[0287] Disambiguation of global translation and rotation
[0288] Solutions to the DOA-only problem, as well as the combined TOA and DOA problems, are subject to global translation and rotation ambiguities. In some examples, translation ambiguity can be resolved by treating only the transmitter source as the listener and translating all devices so that the listener is at the origin.
[0289] Rotational ambiguity can be addressed by imposing constraints on the solution. For example, some multi-loudspeaker environments may include a television (TV) loudspeaker and a sofa positioned for TV viewing. After positioning the loudspeaker in the environment, some methods may involve finding a vector combining the listener's direction of view with the TV viewing direction. Some such methods may then involve causing the TV to emit sound from its loudspeaker and / or prompting the user to move toward the TV and positioning the user's speech. Some implementations may involve rendering an audio object that has been translated around the environment. The user can provide user input (e.g., saying "stop") indicating when the audio object is in one or more predetermined locations within the environment (such as in front of the environment, at the TV position in the environment, etc.). Some implementations involve a mobile application equipped with an inertial measurement unit that prompts the user to point the phone in two defined directions: a first direction toward a specific device (e.g., a device with an emitting LED), and a second direction toward the user's desired viewing direction, such as in front of the environment, at the TV position in the environment, etc. Reference will now be made to... Figures 18A to 18D Describe some detailed examples of disambiguation.
[0290] Figure 18A An example of an audio environment is shown. According to some examples, the audio device location data output by one of the disclosed positioning methods may include an estimate of the audio device location of each of audio devices 1 to 5 using a reference audio device coordinate system 1807. In this embodiment, the audio device coordinate system 1807 is a Cartesian coordinate system with the location of the microphone of audio device 2 as its origin. Here, the x-axis of the audio device coordinate system 1807 corresponds to line 1803 between the microphone locations of audio device 2 and audio device 1.
[0291] In this example, the listener's location is determined by prompting a listener 1805 seated on sofa 1103 (e.g., via an audio prompt from one or more loudspeakers in environment 1800a) to speak one or more utterances 1827 and by estimating the listener's location based on time-of-arrival (TOA) data. The TOA data corresponds to microphone data obtained from multiple microphones in the environment. In this example, the microphone data corresponds to the detection of one or more utterances 1827 by microphones from at least some (e.g., three, four, or all five) of the audio devices 1 to 5.
[0292] Alternatively or additionally, the listener's location can be estimated based on DOA data provided by microphones from at least some (e.g., two, three, four, or all five) of the audio devices 1 to 5. According to some such examples, the listener's location can be determined based on the intersection of lines 1809a, 1809b, etc., corresponding to the DOA data.
[0293] According to this example, the listener's position corresponds to the origin of the listener coordinate system 1820. In this example, the listener's angular orientation data is indicated by the y' axis of the listener coordinate system 1820, which corresponds to line 1813a between the listener's head 1810 (and / or the listener's nose 1825) and the soundbar 1830 of the television 1101. Figure 18A In the example shown, line 1813a is parallel to the y' axis. Therefore, angle This represents the angle between the y-axis and the y'-axis. Therefore, although the origin of the audio device coordinate system 1807 is shown as... Figure 18A Corresponding to audio device 2 in the context, but some implementations involve rotating the audio device coordinates around the origin of the listener coordinate system 1820 by an angle. Previously, the origin of the audio device coordinate system 1807 and the origin of the listener coordinate system 1820 were jointly located. This joint location can be performed through a coordinate transformation from the audio device coordinate system 1807 to the listener coordinate system 1820.
[0294] In some examples, the location of the soundbar 1830 and / or the television 1101 can be determined by causing the soundbar to emit sound and estimating its location based on DOA and / or TOA data, which can correspond to the detection of sound by the microphones of at least some (e.g., 3, 4, or all 5) of the audio devices 1 to 5. Alternatively or additionally, the location of the soundbar 1830 and / or the television 1101 can be determined by prompting the user to walk towards the TV and locating the user's speech using DOA and / or TOA data, which can correspond to the detection of sound by the microphones of at least some (e.g., 3, 4, or all 5) of the audio devices 1 to 5. Some such methods may involve applying a cost function, such as that described above. Some such methods may involve triangulation. Such examples can be advantageous in cases where the soundbar 1830 and / or the television 1101 do not have associated microphones.
[0295] In some other examples where the soundbar 1830 and / or television 1101 do indeed have associated microphones, the positions of the soundbar 1830 and / or television 1101 can be determined according to TOA and / or DOA methods (such as those disclosed herein). According to some of these methods, the microphone can be co-located with the soundbar 1830.
[0296] According to some embodiments, the soundbar 1830 and / or television 1101 may have an associated camera 1811. The control system may be configured to capture an image of the listener's head 1810 (and / or the listener's nose 1825). In some such examples, the control system may be configured to determine a line 1813a between the listener's head 1810 (and / or the listener's nose 1825) and the camera 1811. Listener angular orientation data may correspond to line 1813a. Alternatively or additionally, the control system may be configured to determine the angle between line 1813a and the y-axis of the audio device coordinate system.
[0297] Figure 18B Additional examples of determining listener angular orientation data are shown. According to this example, the listener position has been determined. Here, the control system controls the loudspeaker of environment 1800b to render audio object 1835 to various positions within environment 1800b. In some such examples, the control system may cause the loudspeaker to render audio object 1835 such that audio object 1835 appears to rotate around listener 1805, for example, by rendering audio object 1835 such that audio object 1835 appears to rotate around the origin of listener coordinate system 1820. In this example, curved arrow 1840 shows a portion of the trajectory of audio object 1835 as it rotates around listener 1805.
[0298] According to some such examples, the listener 1805 may provide user input indicating when the audio object 1835 is in the direction the listener 1805 is facing (e.g., saying "stop"). In some such examples, the control system may be configured to determine a line 1813b between the listener's position and the position of the audio object 1835. In this example, line 1813b corresponds to the y' axis of the listener's coordinate system, which indicates the direction the listener 1805 is facing. In alternative embodiments, the listener 1805 may provide user input indicating when the audio object 1835 is in front of the environment, at a TV position in the environment, at a position of the audio device, etc.
[0299] Figure 18C An additional example of determining listener angular orientation data is shown. According to this example, the listener's position has been determined. Here, listener 1805 is using a handheld device 1845 to provide input regarding the listener's viewing direction by pointing the handheld device 1845 at the television 1101 or the soundbar 1830. In this example, the dashed outline of the handheld device 1845 and the listener's arm indicate the time prior to when listener 1805 points the handheld device 1845 at the television 1101 or the soundbar 1830, at which point listener 1805 points the handheld device 1845 at audio device 2. In other examples, listener 1805 may have already pointed the handheld device 1845 at another audio device, such as audio device 1. According to this example, the handheld device 1845 is configured to determine an angle α between audio device 2 and the television 1101 or the soundbar 1830, said angle approximating the angle between audio device 2 and the listener 1805's viewing direction.
[0300] In some examples, the handheld device 1845 may be a cellular phone including an inertial sensor system and a wireless interface configured to communicate with the control system of the audio equipment of the control environment 1800c. In some examples, the handheld device 1845 may run an application or "app" configured to perform necessary functions, such as by providing user prompts (e.g., via a graphical user interface), by receiving input instructing the handheld device 1845 to point in a desired direction, by saving corresponding inertial sensor data and / or transmitting corresponding inertial sensor data to the control system of the audio equipment of the control environment 1800c.
[0301] According to this example, the control system (which may be the control system of handheld device 1845, the control system of intelligent audio device of environment 1800c, or the control system of audio device controlling environment 1800c) is configured to determine the orientation of lines 1813c and 1850 based on inertial sensor data (e.g., based on gyroscope data). In this example, line 1813c is parallel to axis y' and can be used to determine the listener angular orientation. According to some examples, the control system can determine the appropriate rotation of the audio device coordinates about the origin of listener coordinate system 1820 based on the angle α between audio device 2 and the viewing direction of listener 1805.
[0302] Figure 18D It is shown according to the reference Figure 18C The described method provides an example of determining the appropriate rotation of the audio device coordinates. In this example, the origin of the audio device coordinate system 1807 is co-located with the origin of the listener coordinate system 1820. It is possible to co-locate the origin of the audio device coordinate system 1807 with the origin of the listener coordinate system 1820 after determining the listener's position. Co-locating the origin of the audio device coordinate system 1807 with the origin of the listener coordinate system 1820 may involve transforming the audio device position from the audio device coordinate system 1807 to the listener coordinate system 1820. Angle α has been determined, as referenced above. Figure 18C As described. Therefore, angle α corresponds to the desired orientation of audio device 2 in listener coordinate system 1820. In this example, angle β corresponds to the orientation of audio device 2 in audio device coordinate system 1807. In this example, it is angle β-α. Indicates the rotation necessary to align the y-axis of the audio device coordinate system 1807 with the y'-axis of the listener coordinate system 1820.
[0303] DOA robustness measures
[0304] As referenced above Figure 14 In some examples of using "blind" methods applied to arbitrary signals (including controlled response power, beamforming, or other similar methods), robustness measures can be added to improve accuracy and stability. Some implementations include time integration of the controlled response of the beamformer to filter out transients and detect only persistent peaks, while averaging out random errors and fluctuations in these persistent DOAs. Other examples may use only a limited frequency band as input, which can be adjusted according to room or signal type for better performance.
[0305] For examples using "supervised" methods (involving the use of structured source signals and deconvolution methods to generate impulse responses), preprocessing measures can be implemented to enhance the accuracy and prominence of the DOA peak. In some examples, such preprocessing may include truncating an amplitude window of a certain time width starting at the beginning of the impulse response on each microphone channel. Such examples can incorporate an impulse response start detector, allowing the start of each channel to be found independently.
[0306] In some examples, further processing can be added to improve DOA accuracy based on the “blind” or “supervised” methods described above. It is important to note that peak detection-based DOA selection (e.g., during controlled response power (SRP) or impulse response analysis) is sensitive to ambient acoustics, which can lead to the capture of non-master path signals due to reflections and device obstruction that suppress received and transmitted energy. These occurrences can reduce the accuracy of the device’s DOA and introduce errors into the optimizer’s localization solution. Therefore, a prudent approach is to consider all peaks within a predetermined threshold as candidates for ground-based DOAs. An example of a predetermined threshold is requiring peaks to be greater than the average controlled response power (SRP). For all detected peaks, salience thresholding and removal of candidates below the average signal level have proven to be simple yet effective initial filtering techniques. As used herein, “salience” is a measure of how much a local peak is compared to its neighboring local minima, which differs from power-based thresholding alone. An example of a salience threshold is requiring the power difference between a peak and its neighboring local minima to be at or above a threshold. Retaining feasible candidates improves the chances that a device pair will include a usable DOA in its group (within an acceptable error tolerance to ground reality), although there is a chance that the device pair will not include a usable DOA if the signal is destroyed by strong reflections / blockage. In some examples, a selection algorithm can be implemented to do one of the following: 1) select the best usable DOA candidate for each device pair; 2) determine that no candidate is available and therefore make the optimization contribution of the pair zero using a cost function weighting matrix; or 3) select the best inferred candidate, but apply a non-binary weighting to the DOA contribution if it is difficult to disambiguate the amount of error carried by the best candidate.
[0307] After initial optimization using the best-inferred candidates, in some examples, the positioning solution can be used to calculate the residual cost contribution for each DOA. Outlier analysis of the residual costs can provide evidence of the DOA pairs that have the greatest impact on the positioning solution, where extreme outliers mark these DOAs as potentially incorrect or suboptimal. A recursive run of optimizing the peripheral DOA pairs based on the residual cost contributions of the remaining candidates, weighted and applied to the contributions of that device pair, can then be used for candidate processing, according to one of the three options mentioned above. This is as referenced above. Figures 14 to 17 An example of the described feedback process. According to some implementations, iterative optimization and decision processing can be performed until all detected candidates are evaluated and the residual cost contributions of the selected DOA are balanced.
[0308] The drawback of candidate selection based on optimizer evaluation is that it is computationally intensive and sensitive to the order of candidate traversal. Alternative techniques with fewer computational weights involve determining all permutations of candidates in the set and running a triangle alignment method for device localization on these candidates. A relevant triangle alignment method is disclosed in U.S. Provisional Patent Application No. 62 / 992,068, filed March 19, 2020, entitled “Audio Device Auto-Location,” which is hereby incorporated by reference for all purposes. The localization result can then be evaluated by the total cost and residual cost generated relative to the DOA candidate calculations used in triangulation. Decision logic used to analyze these metrics can be used to determine the optimal candidates and their corresponding weights to be provided to the nonlinear optimization problem. In cases where the candidate list is large and thus produces a high permutation count, filtering and intelligent traversal of the permutation list can be applied.
[0309] TOA robust measures
[0310] As referenced above Figure 16 As described, using multiple candidate TOA solutions adds robustness compared to systems utilizing a single or minimum TOA value, and ensures that errors have minimal impact on finding the optimal speaker layout. After obtaining the system's impulse response, in some examples, each TOA matrix element can be recovered by searching for the peak corresponding to the direct sound. Under ideal conditions (e.g., no noise, no obstructions in the direct path between the source and the receiver and speaker pointing directly at the microphone), this peak can be easily identified as the maximum peak in the impulse response. However, in the presence of noise, obstructions, or misalignment between the speaker and microphone, the peak corresponding to the direct sound may not correspond to the maximum value. Furthermore, under such conditions, the peak corresponding to the direct sound may be difficult to isolate from other reflections and / or noise. In some instances, direct sound identification can be a challenging process. Incorrect identification of the direct sound will degrade the automatic localization process (and in some instances may completely disrupt it). Therefore, considering multiple candidates for the direct sound can be effective when errors may exist in the direct sound identification process. In some such instances, the peak selection process may include two parts: (1) a direct sound search algorithm for finding suitable peak candidates, and (2) a peak candidate evaluation process to increase the probability of picking the correct TOA matrix element.
[0311] In some implementations, the process of searching for direct sound candidate peaks may include methods for identifying relevant candidates for direct sound. Some such methods may be based on the following steps: (1) identifying a first reference peak (e.g., the maximum absolute value of the impulse response (IR), i.e., the “first peak”); (2) assessing the noise level around (before and after) the first peak; (3) searching for alternative peaks above the noise level before (and in some cases after) the first peak; (4) ranking the peaks according to the probability of the corresponding correct TOA of the found peaks; and optionally (5) grouping adjacent peaks (to reduce the number of candidates).
[0312] Once direct sound candidate peaks are identified, some implementations may involve a multi-peak evaluation step. As a result of the direct sound candidate peak search, in some examples, for each TOA matrix element, there will be one or more candidate values sorted according to their estimated probabilities. Multiple TOA matrices can be formed by selecting from different candidate values. To evaluate the likelihood of a given TOA matrix, a minimization process (as described above) can be implemented. This process can generate minimized residuals that are a good estimate of the internal consistency of the TOA and DOA matrices. A perfectly noise-free TOA matrix will result in zero residuals, while a TOA matrix with incorrect matrix elements will result in large residuals. In some implementations, the method will search for a set of candidate TOA matrix elements that create a TOA matrix with minimum residuals. This is referenced above. Figure 16 and Figure 17 An example of the described evaluation process may involve the result evaluation box 1750. In one example, the evaluation process may involve performing the following steps: (1) selecting an initial TOA matrix; (2) evaluating the initial matrix with the residuals of the minimization process; (3) changing one element of the TOA matrix from the TOA candidate list; (4) re-evaluating the matrix with the residuals of the minimization process; (5) accepting the change if the residual is small, otherwise not accepting the change; and (6) iterating through steps 3 to 5. In some examples, the evaluation process may stop when all TOA candidates have been evaluated or when a predefined maximum number of iterations has been reached.
[0313] Some disclosed alternative implementations also involve acoustically locating loudspeakers and / or listeners using direction of arrival (DOA) data. In some examples, DOA data can be obtained via a microphone array that is co-located with some or all of the loudspeakers.
[0314] Figure 19An example of the geometric relationship between three audio devices in an environment is shown. In this example, environment 1900 is a room including a television 1901, a sofa 1903, and five audio devices 1905. According to this example, the audio devices 1905 are located at positions 1 to 5 in environment 1900. In this embodiment, each audio device 1905 includes a microphone system 1920 having at least three microphones and a speaker system 1925 including at least one speaker. In some embodiments, each microphone system 1920 includes a microphone array. According to some embodiments, each audio device 1905 may include an antenna system containing at least three antennas.
[0315] As with other examples disclosed in this article, Figure 19 The types, quantities, and arrangements of the components shown are merely examples. Other embodiments may have components of different types, quantities, and arrangements, such as more or fewer audio devices 1905, audio devices 1905 in different locations, etc.
[0316] In this example, the vertices of triangle 1910a are at positions 1, 2, and 3. Here, triangle 1910a has sides 12, 23a, and 13a. According to this example, the angle between sides 12 and 23 is θ2, the angle between sides 12 and 13a is θ1, and the angle between sides 23a and 13a is θ3. These angles can be determined from DOA data, as described in more detail below.
[0317] In some implementations, only the relative lengths of the triangle sides can be determined. In alternative implementations, the actual lengths of the triangle sides can be estimated. According to some such implementations, the actual lengths of the triangle sides can be estimated based on TOA data, for example, based on the arrival times of sounds generated by an audio device located at one vertex of the triangle and detected by an audio device located at another vertex of the triangle. Alternatively or additionally, the length of the triangle sides can be estimated based on electromagnetic waves generated by an audio device located at one vertex of the triangle and detected by an audio device located at another vertex of the triangle. For example, the length of the triangle sides can be estimated based on the signal strength of electromagnetic waves generated by an audio device located at one vertex of the triangle and detected by an audio device located at another vertex of the triangle. In some implementations, the length of the triangle sides can be estimated based on the phase shift of the detected electromagnetic waves.
[0318] Figure 20 It shows Figure 19Another example of the geometric relationship between three audio devices in the environment is shown. In this example, the vertices of triangle 1910b are at positions 1, 3, and 4. Here, triangle 1910b has sides 13b, 14, and 34a. According to this example, the angle between sides 13b and 14 is θ4, the angle between sides 13b and 34a is θ5, and the angle between sides 34a and 14 is θ6.
[0319] By comparison Figure 11 and Figure 12 It can be observed that the length of side 13a of triangle 1910a should be equal to the length of side 13b of triangle 1910b. In some implementations, the side length of a triangle (e.g., triangle 1910a) can be assumed to be correct, and the length of the side shared by adjacent triangles will be constrained to that length.
[0320] Figure 21A It shows Figure 19 and Figure 20 The two triangles depicted in the image do not contain any corresponding audio devices or other environmental features. Figure 21A Estimates of the side lengths and angular orientations of triangles 1910a and 1910b are shown. Figure 21A In the example shown, the length of side 13b of triangle 1910b is constrained to be the same as the length of side 13a of triangle 1910a. The lengths of the other sides of triangle 1910b are scaled proportionally to the change in the length of side 13b. The resulting triangle 1910b' is... Figure 21A It is shown as adjacent to triangle 1910a.
[0321] According to some implementations, the side lengths of other triangles adjacent to triangles 1910a and 1910b can be determined in a similar manner until the locations of all audio devices in environment 1900 have been determined.
[0322] Some examples of audio device location can be presented as follows. Each audio device can report the DOA of each other audio device in the environment (e.g., a room) based on the sound produced by each other audio device in the environment. The Cartesian coordinates of the i-th audio device can be represented as x. i =[x i y i ] T The superscript T indicates the vector transpose. Given M audio devices in an environment, i = {1...M}.
[0323] Figure 21B An example of estimating the interior angles of a triangle formed by three audio devices is shown. In this example, the audio devices are i, j, and k. The DOA of the sound source emanating from device j, as observed from device i, can be represented as θ.ji The DOA of the sound source emitted from device k, as observed from device i, can be represented as θ. ki .exist Figure 21B In the example shown, θ ji and θ ki It is measured from axis 2105a, the orientation of which is arbitrary, and which may, for example, correspond to the orientation of audio device i. The interior angle α of triangle 2110 can be expressed as α = θ. ki -θ ji It can be observed that the calculation of interior angle α does not depend on the orientation of axis 2105α.
[0324] exist Figure 21B In the example shown, θ ij and θ kj It is measured from axis 2105b, the orientation of which is arbitrary and can correspond to the orientation of audio device j. The interior angle b of triangle 2110 can be expressed as b = θ. ij -θ kj Similarly, in this example, θ jk and θ ik It is measured from axis 2105c. The interior angle c of triangle 2110 can be expressed as c = θ. jk -θ ik .
[0325] In the presence of measurement error, a + b + c ≠ 180°. Robustness can be improved by predicting each angle from the other two angles and averaging the results, for example, as shown below:
[0326]
[0327] In some implementations, edge lengths (A, B, C) can be calculated by applying a sine rule (up to scaling errors). In some examples, an arbitrary value, such as 1, can be assigned to an edge length. For example, by setting A = 1 and setting the vertex... If placed at the origin, the positions of the remaining two vertices can be calculated as follows:
[0328]
[0329] However, arbitrary rotations are acceptable.
[0330] According to some implementations, the process of triangle parameterization can be repeated for all possible subsets of three audio devices in the environment, within a size of... Enumerate in the superset v. In some examples, T lThis can represent the l-th triangle. Triangles may not be enumerated in any particular order, depending on the implementation. Due to possible errors in DOA and / or side length estimation, triangles may overlap and may not be perfectly aligned.
[0331] Figure 22 It outlines what can be made by, for example Figure 1 The flowchart illustrates an example of a method performed by the apparatus shown herein. As with other methods described herein, the blocks of method 2200 need not be performed in the indicated order. Furthermore, such a method may include more or fewer blocks than those shown and / or described. In this embodiment, method 2200 relates to estimating the position of a speaker in the environment. The blocks of method 2200 may be performed by one or more devices, which may be (or may include) Figure 1 The device 100 shown in the figure.
[0332] In this example, box 2205 involves obtaining the direction of arrival (DOA) data for each of a plurality of audio devices. In some examples, the plurality of audio devices may include all audio devices in the environment, such as Figure 19 All audio devices shown in the figure are from 1905.
[0333] However, in some instances, multiple audio devices may only include a subset of all audio devices in the environment. For example, multiple audio devices may include all smart speakers in the environment, but exclude one or more other audio devices in the environment.
[0334] Depending on the specific implementation, DOA data can be obtained in various ways. In some instances, determining DOA data may involve determining the DOA data of at least one of a plurality of audio devices. For example, determining DOA data may involve receiving microphone data from each of a plurality of audio device microphones corresponding to a single audio device among the plurality of audio devices, and determining the DOA data of the single audio device at least in part based on the microphone data. Alternatively or additionally, determining DOA data may involve receiving antenna data from one or more antennas corresponding to a single audio device among the plurality of audio devices, and determining the DOA data of the single audio device at least in part based on the antenna data.
[0335] In some such examples, a single audio device can determine its own DOA data. According to some implementations, each of a plurality of audio devices can determine its own DOA data. However, in other implementations, another device, which may be local or remote, can determine the DOA data of one or more audio devices in the environment. According to some implementations, a server can determine the DOA data of one or more audio devices in the environment.
[0336] According to this example, box 2210 involves determining the interior angles of each of a plurality of triangles based on DOA data. In this example, each of the plurality of triangles has vertices corresponding to the audio device positions of the three audio devices. Some such examples have been described above.
[0337] Figure 23 An example is shown where each audio device in the environment is represented by the vertices of multiple triangles. The sides of each triangle correspond to the distance between two audio devices 1905.
[0338] In this implementation, box 2215 relates to determining the side length of each side of each triangle. (The sides of a triangle may also be referred to herein as “edges.”) According to this example, the side length is based at least in part on the interior angles. In some instances, the side lengths can be calculated by determining a first length of the first side of the triangle and determining the lengths of the second and third sides of the triangle based on the interior angles of the triangle. Some such examples have been described above.
[0339] According to some such implementations, determining the first length may involve setting the first length to a predetermined value. However, in some examples, determining the first length may be based on time-of-arrival data and / or received signal strength data. In some implementations, the time-of-arrival data and / or received signal strength data may correspond to sound waves detected by a second audio device in the environment from a first audio device in the environment. Alternatively or additionally, the time-of-arrival data and / or received signal strength data may correspond to electromagnetic waves (e.g., radio waves, infrared waves, etc.) detected by a second audio device in the environment from a first audio device in the environment.
[0340] According to this example, box 2220 involves performing a forward alignment process that aligns each of the plurality of triangles in a first order. According to this example, the forward alignment process produces a forward alignment matrix.
[0341] Based on some such examples, it is expected that the triangle will have an edge (x) i x j Alignment is done in a way that matches adjacent edges, for example, as shown below. Figure 21A As shown in the diagram and described above. Let the magnitude of ε be... The set of all edges. In some such implementations, box 2220 may involve traversing ε and aligning the common edges of the triangle in a forward order by forcing the edges to align with the edges of previously aligned edges.
[0342] Figure 24 An example of a portion of the forward alignment process is provided. Figure 24 The numbers 1 to 5 shown in bold are... Figure 1 Figure 2 and Figure 5 The locations of the audio devices shown correspond to those in the diagram. Figure 24 The order of the forward alignment process shown and described herein is merely an example.
[0343] In this example, as in Figure 21A In this case, the length of side 13b of triangle 1910b is forced to be the same as the length of side 13a of triangle 1910a. Figure 24 The resulting triangle 1910b' is shown, where the same interior angles are maintained. According to this example, the length of side 13c of triangle 1910c is also forced to be the same as the length of side 13a of triangle 1910a. Figure 24 The resulting triangle 1910c' is shown in the figure, where the same interior angles are maintained.
[0344] Next, in this example, the length of side 34b of triangle 1910d is forced to be the same as the length of side 34a of triangle 1910b'. Furthermore, in this example, the length of side 23b of triangle 1910d is forced to be the same as the length of side 23a of triangle 1910a. Figure 24 The resulting triangle 1910d' is shown, where the same interior angles are maintained. Based on some such examples, Figure 5 The remaining triangles shown can be processed in the same way as triangles 1910b, 1910c, and 1910d.
[0345] The result of the forward alignment process can be stored in a data structure. According to some examples, the result of the forward alignment process can be stored in a forward alignment matrix. For instance, the result of the forward alignment process can be stored in a matrix. In the diagram, N indicates the total number of triangles.
[0346] Multiple audio device position estimations will occur when the DOA data and / or initial side length determination contain errors. These errors typically increase during the forward alignment process.
[0347] Figure 25 Examples of multiple estimates of audio device positions that have occurred during the forward alignment process are shown. In this example, the forward alignment process is based on a triangle with the seven audio device positions as vertices. Here, the triangle is not perfectly aligned due to additional errors in the DOA estimation. Figure 25 The positions of the numbers 1 through 7 shown correspond to the estimated audio device positions generated by the forward alignment process. In this example, the audio device position estimate marked "1" is consistent, but the audio device position estimates for audio devices 6 and 7 show significant differences, as indicated by the relatively large areas where the numbers 6 and 7 are located.
[0348] return Figure 22 In this example, box 2225 relates to a reverse alignment process that aligns each of a plurality of triangles in a second order, which is the reverse of the first order. According to some implementations, the reverse alignment process may involve traversing ε in the same but reverse order as before. In alternative examples, the reverse alignment process may not be exactly the reverse of the order of operations of the forward alignment process. According to this example, the reverse alignment process produces a reverse alignment matrix, which can be represented herein as...
[0349] Figure 26 An example of a part of the reverse alignment process is provided. Figure 26 The numbers 1 to 5 shown in bold are... Figure 19 Figure 21 and Figure 23 The locations of the audio devices shown correspond to those in the diagram. Figure 26 The order of the reverse alignment process shown and described herein is merely an example.
[0350] exist Figure 26 In the example shown, triangle 1910e is based on audio device positions 3, 4, and 5. In this implementation, it is assumed that the side length (or "edge") of triangle 1910e is correct, and the side lengths of adjacent triangles are forced to be consistent with it. According to this example, the length of side 45b of triangle 1910f is forced to be consistent with the length of side 45a of triangle 1910e. Figure 26 The resulting triangle 1910f' is shown, where the interior angles remain the same. In this example, the length of side 35b of triangle 1910c is forced to be the same as the length of side 35a of triangle 1910e. Figure 26 The resulting triangle 1910c is shown, where the interior angles remain the same. Based on some such examples, Figure 23 The remaining triangles shown can be processed in the same way as triangles 1910c and 1910f until the reverse alignment process has included all the remaining triangles.
[0351] Figure 27 An example of multiple audio device position estimates that have occurred during the reverse alignment process is shown. In this example, the reverse alignment process is based on [the method described above]. Figure 25 The triangle describes the locations of the seven audio device positions with the same vertices. Figure 27 The positions of numbers 1 through 7 shown correspond to the estimated audio device positions generated by the reverse alignment process. Again, the triangles are not perfectly aligned due to additional errors in the DOA estimation. In this example, the audio device position estimates labeled 6 and 7 are consistent, but the audio device position estimates for audio devices 1 and 2 show greater differences.
[0352] return Figure 22Box 2230 relates to generating a final estimate of the location of each audio device based at least in part on the values of the forward alignment matrix and the reverse alignment matrix. In some examples, generating the final estimate of the location of each audio device may involve translating and scaling the forward alignment matrix to produce a translated and scaled forward alignment matrix, and translating and scaling the reverse alignment matrix to produce a translated and scaled reverse alignment matrix.
[0353] For example, by moving the centroid to the origin and forcing the unit Frobenius norm (e.g., and Use this to fix translation and scaling.
[0354] According to some such examples, generating a final estimate of the position of each audio device may also involve generating a rotation matrix based on a translated and scaled forward alignment matrix and a translated and scaled reverse alignment matrix. The rotation matrix may include multiple estimated audio device positions for each audio device. For example, the optimal rotation between forward and reverse alignment can be found through singular value decomposition. In some such examples, generating the rotation matrix may involve performing singular value decomposition on the translated and scaled forward alignment matrix and the translated and scaled reverse alignment matrix, for example, as follows:
[0355]
[0356] In the aforementioned equations, U represents the matrix respectively. The left singular vector of the matrix is denoted by V, and the right singular vector of the matrix is denoted by ∑. ∑ represents the matrix of singular values. The aforementioned equation produces the rotation matrix R = VU. T Matrix product VU T Generate a rotation matrix such that Optimally rotated to be with Alignment.
[0357] Based on some examples, when determining the rotation matrix R = VU T Then, the alignment can be averaged, for example, as follows:
[0358]
[0359] In some implementations, generating a final estimate of the location of each audio device may also involve averaging the estimated audio device locations for each audio device to produce a final estimate of the location of each audio device. Various disclosed implementations have proven robust, even when the DOA data and / or other calculations include significant errors. For example, due to overlapping vertices from multiple triangles, Containing the same nodes A total of several estimates are calculated. The final estimate is obtained by averaging across common nodes.
[0360] Figure 28 A comparison between the estimated audio device location and the actual audio device location is shown. Figure 28 In the example shown, the audio device location is the same as in the reference above. Figure 17 and Figure 19 The audio device positions estimated during the described forward alignment and reverse alignment processes correspond to each other. In these examples, the DOA estimation error has a standard deviation of 15 degrees. Nevertheless, the final estimate of each audio device position (each of the final estimates in...) Figure 28 (represented by "x") and the actual audio device location (each of the actual audio device locations is in...) Figure 28 The circle in the middle corresponds well.
[0361] Much of the foregoing discussion pertains to the automatic positioning of audio devices. The following discussion expands upon some of the methods briefly described above for determining listener position and listener angular orientation. In the foregoing description, the term "rotation" was used in essentially the same way as the term "orientation" used in the following description. For example, the "rotation" referenced above could refer to a global rotation of the final speaker geometry, rather than a single triangle as described above. Figure 14 The rotation during the process described above. This global rotation or orientation can be analyzed by referring to the listener's angular orientation, such as the direction the listener is looking, the direction the listener's nose is pointing, etc.
[0362] Various satisfactory methods for estimating listener position are described below. However, estimating listener angular orientation can be challenging. Some relevant methods are described in detail below.
[0363] Determining the listener's position and angular orientation allows for the realization of desired features, such as audio equipment directional positioning relative to the listener. Knowing the listener's position and angular orientation allows for determining, for example, which speakers in the environment are in front of the listener, which are behind, and which are closer to the center (if any).
[0364] After establishing a correlation between the audio device location and the listener's location and orientation, some implementations may involve providing audio device location data, audio device angular orientation data, listener location data, and listener angular orientation data to an audio rendering system. Alternatively or additionally, some implementations may involve an audio data rendering process that is at least partially based on the audio device location data, audio device angular orientation data, listener location data, and listener angular orientation data.
[0365] Figure 29 It outlines what can be made by, for example Figure 1A flowchart of another example of a method performed by the apparatus shown herein. As with other methods described herein, the blocks of method 2900 need not be performed in the indicated order. Furthermore, such a method may include more or fewer blocks than those shown and / or described. In this example, the blocks of method 2900 are performed by a control system, which may be (or may include) Figure 1 The control system 110 is shown in the figure. As described above, in some embodiments, the control system 110 may reside in a single device, while in other embodiments, the control system 110 may reside in two or more devices.
[0366] In this example, box 2905 relates to obtaining the direction of arrival (DOA) data for each of a plurality of audio devices in the environment. In some examples, the plurality of audio devices may include all audio devices in the environment, such as Figure 27 All audio devices shown in the figure are from 1905.
[0367] However, in some instances, multiple audio devices may only include a subset of all audio devices in the environment. For example, multiple audio devices may include all smart speakers in the environment, but exclude one or more other audio devices in the environment.
[0368] Depending on the specific implementation, DOA data can be obtained in various ways. In some instances, determining DOA data may involve determining the DOA data of at least one of a plurality of audio devices. In some examples, DOA data can be obtained by controlling each of a plurality of loudspeakers in an environment to reproduce a test signal. For example, determining DOA data may involve receiving microphone data from each of a plurality of audio device microphones corresponding to a single audio device among a plurality of audio devices, and determining the DOA data of the single audio device at least in part based on the microphone data. Alternatively or additionally, determining DOA data may involve receiving antenna data from one or more antennas corresponding to a single audio device among a plurality of audio devices, and determining the DOA data of the single audio device at least in part based on the antenna data.
[0369] In some such examples, a single audio device can determine its own DOA data. According to some implementations, each of a plurality of audio devices can determine its own DOA data. However, in other implementations, another device, which may be local or remote, can determine the DOA data of one or more audio devices in the environment. According to some implementations, a server can determine the DOA data of one or more audio devices in the environment.
[0370] according to Figure 29The example shown, box 2910, relates to generating audio device location data via a control system, at least in part, based on DOA data. In this example, the audio device location data includes an estimate of the audio device location for each audio device referenced in box 2905.
[0371] For example, audio device location data can be (or include) coordinates in a coordinate system (such as a Cartesian, spherical, or cylindrical coordinate system). This coordinate system may be referred to herein as the audio device coordinate system. In some such examples, the audio device coordinate system may be oriented with reference to one of the audio devices in the environment. In other examples, the audio device coordinate system may be oriented with reference to an axis defined by a line between two audio devices in the environment. However, in still other examples, the audio device coordinate system may be oriented with reference to another part of the environment (such as a television, a wall in a room, etc.).
[0372] In some examples, box 2910 may refer to the above reference. Figure 22 The process described. According to some such examples, box 2910 may involve determining the interior angles of each of a plurality of triangles based on DOA data. In some instances, each of the plurality of triangles may have vertices corresponding to the audio device positions of the three audio devices. Some such methods may involve determining the side length of each side of each triangle based at least in part on the interior angles.
[0373] Some such methods may involve performing a forward alignment process that aligns each of a plurality of triangles in a first order to produce a forward alignment matrix. Some such methods may involve performing a reverse alignment process that aligns each of the plurality of triangles in a second order that is the reverse of the first order to produce a reverse alignment matrix. Some such methods may involve generating a final estimate of the position of each audio device based at least in part on the values of the forward alignment matrix and the reverse alignment matrix. However, in some embodiments of method 2900, block 2910 may involve applying additional methods beyond those referenced above. Figure 22 Methods other than those described.
[0374] In this example, box 2915 relates to listener location data, determined via a control system, indicating the listener's location within the environment. For example, the listener location data may be referenced to an audio device coordinate system. However, in other examples, the coordinate system may be oriented with reference to the listener or a portion of the environment (such as a television, a room wall, etc.).
[0375] In some examples, box 2915 may involve prompting a listener (e.g., via audio cues from one or more loudspeakers in the environment) to speak one or more utterances and estimating the listener's location based on DOA data. The DOA data may correspond to microphone data obtained from multiple microphones in the environment. The microphone data may correspond to the detection of one or more utterances by the microphones. At least some microphones may be co-localized with the loudspeakers. According to some examples, box 2915 may involve a triangulation process. For example, box 2915 may involve triangulating the user's speech by finding the intersections between the DOA vectors passing through the audio device, for example, as referenced above. Figure 18A As described. According to some implementations, block 2915 (or another operation of method 2900) may involve locating the origin of the audio device coordinate system and the origin of the listener coordinate system together after determining the listener's location. Locating the origin of the audio device coordinate system and the origin of the listener coordinate system together may involve transforming the audio device location from the audio device coordinate system to the listener coordinate system.
[0376] According to this embodiment, block 2920 relates to determining listener angular orientation data indicating listener angular orientation via a control system. For example, the listener angular orientation data may be obtained with reference to a coordinate system (such as an audio device coordinate system) used to represent listener position data. In some such examples, the listener angular orientation data may be obtained with reference to the origin and / or axes of the audio device coordinate system.
[0377] However, in some implementations, listener angular orientation data can be obtained with reference to an axis defined by the listener's position and another point in the environment (such as a television, audio device, wall, etc.). In some such implementations, the listener's position can be used to define the origin of the listener's coordinate system. In some such examples, listener angular orientation data can be obtained with reference to the axis of the listener's coordinate system.
[0378] This document discloses various methods for implementing block 2920. According to some examples, the listener's angular orientation can correspond to the listener's viewing direction. In some such examples, the listener's viewing direction can be inferred, for example, by assuming the listener is viewing a specific object (such as a television), referring to listener position data. In some such implementations, the listener's viewing direction can be determined based on the listener's position and the television's position. Alternatively or additionally, the listener's viewing direction can be determined based on the listener's position and the television soundbar's position.
[0379] However, in some examples, the listener's viewing direction can be determined based on listener input. According to some such examples, listener input may include inertial sensor data received from a device held by the listener. The listener can use said device to point to a location in the environment, for example, a location corresponding to the direction the listener is facing. For example, the listener can use said device to point to a loudspeaker (a loudspeaker that reproduces sound). Therefore, in such examples, the inertial sensor data may include inertial sensor data corresponding to the loudspeaker that is emitting the sound.
[0380] In some such instances, listener input may include an indication of the audio device selected by the listener. In some examples, the audio device indication may include inertial sensor data corresponding to the selected audio device.
[0381] However, in other examples, instructions for audio devices can be given based on one or more of the listener's utterances (e.g., "The TV is now in front of me," "Speaker 2 is now in front of me," etc.). Other examples of determining listener orientation data based on one or more of the listener's utterances are described below.
[0382] according to Figure 29 In the example shown, block 2925 relates to determining audio device angular orientation data via a control system, the audio device angular orientation data indicating the audio device angular orientation of each audio device relative to the listener position and the listener angular orientation. According to some such examples, block 2925 may relate to rotating the audio device coordinates about a point defined by the listener position. In some embodiments, block 2925 may relate to transforming audio device position data from an audio device coordinate system to a listener coordinate system.
[0383] Figure 30 This is a flowchart outlining another example of a localization method. As with other methods described herein, the boxes of method 3000 need not be performed in the indicated order. Furthermore, such a method may include more or fewer boxes than those shown and / or described. In this embodiment, method 3000 involves estimating the location and orientation of an audio device in the environment. The boxes of method 3000 may be performed by one or more devices, which may be (or may include) Figure 1 The device 100 shown in the figure.
[0384] In this example, block 3005 obtains direction of arrival (DOA) data corresponding to the sound emitted by at least a first intelligent audio device in the audio environment from the control system. For example, the control system may be as described above. Figure 1The described control system 110. According to this example, a first smart audio device includes a first audio transmitter and a first audio receiver, and DOA data corresponds to sound received by at least a second smart audio device in the audio environment. Here, the second smart audio device includes a second audio transmitter and a second audio receiver. In this example, the DOA data also corresponds to sound transmitted by at least the second smart audio device and received by at least the first smart audio device. In some examples, the first smart audio device and the second smart audio device may be... Figure 11 Two of the audio devices 1105a to 1105d shown in the figure.
[0385] Depending on the specific implementation method, DOA data can be obtained in various ways. In some instances, determining DOA data may involve the references above. Figure 14 And / or one or more of the DOA-related methods described in the "DOA Robustness Measures" section. Some implementations may involve obtaining DOA data by a control system using one or more elements of beamforming methods, controllable power response methods, time difference of arrival methods, and / or structured signaling methods.
[0386] According to this example, block 3010 relates to receiving configuration parameters by the control system. In this embodiment, the configuration parameters correspond to the audio environment itself, one or more audio devices of the audio environment, or both the audio environment and one or more audio devices of the audio environment. According to some examples, the configuration parameters may indicate the number of audio devices in the audio environment, one or more dimensions of the audio environment, one or more constraints on the position or orientation of the audio devices, and / or disambiguation data for at least one of rotation, translation, or scaling. In some examples, the configuration parameters may include playback delay data, recording delay data, and / or data used to disambiguate delay symmetry.
[0387] In this example, box 3015 involves minimizing a cost function by the control system, at least in part, based on DOA data and configuration parameters, to estimate the location and orientation of at least the first and second smart audio devices.
[0388] According to some examples, DOA data can also correspond to sounds emitted by the third to Nth smart audio devices in the audio environment, where N corresponds to the total number of smart audio devices in the audio environment. In such examples, DOA data can also correspond to sounds received by each of the first to Nth smart audio devices from all other smart audio devices in the audio environment. In such instances, minimizing the cost function can involve estimating the location and orientation of the third to Nth smart audio devices.
[0389] In some examples, DOA data can also correspond to sound received by one or more passive audio receivers in an audio environment. Each of the one or more passive audio receivers may include a microphone array but may lack an audio transmitter. Minimizing the cost function can also provide an estimated location and orientation for each of the one or more passive audio receivers. According to some examples, DOA data can also correspond to sound emitted by one or more audio transmitters in an audio environment. Each of the one or more audio transmitters may include at least one acoustic transducer but may lack a microphone array. Minimizing the cost function can also provide an estimated location for each of the one or more audio transmitters.
[0390] In some examples, method 3000 may involve receiving a seed layout for a cost function by a control system. For example, the seed layout may specify the correct number of audio transmitters and receivers in the audio environment, as well as the arbitrary position and orientation of each of the audio transmitters and receivers in the audio environment.
[0391] According to some examples, method 3000 may involve receiving a weighting factor associated with one or more elements of the DOA data by a control system. For example, the weighting factor may indicate the availability and / or reliability of one or more elements of the DOA data.
[0392] In some examples, method 3000 may involve receiving, by a control system, time-of-arrival (TOA) data corresponding to sound emitted by at least one audio device in the audio environment and received by at least one other audio device in the audio environment. In some such examples, a cost function may be based at least in part on the TOA data. Some such implementations may involve estimating at least one playback delay and / or at least one recording delay. According to some such examples, the cost function may operate using a rescaled location, a rescaled delay, and / or a rescaled time of arrival.
[0393] In some examples, the cost function may include a first term that depends only on the DOA data and a second term that depends only on the TOA data. In some such examples, the first term may include a first weighting factor, and the second term may include a second weighting factor. According to some such examples, one or more TOA elements of the second term may have TOA element weighting factors that indicate the availability or reliability of each of the one or more TOA elements.
[0394] Figure 31This is a flowchart outlining another example of a positioning method. As with other methods described herein, the boxes of method 3100 need not be performed in the indicated order. Furthermore, such a method may include more or fewer boxes than those shown and / or described. In this embodiment, method 3100 involves estimating the location and orientation of a device in its environment. The boxes of method 3100 may be performed by one or more devices, which may be (or may include) Figure 1 The device 100 shown in the figure.
[0395] In this example, block 3105 obtains direction of arrival (DOA) data corresponding to the transmission of at least a first transceiver of a first device in the environment from the control system. For example, the control system may be as described above. Figure 1 The described control system 110. According to this example, the first transceiver includes a first transmitter and a first receiver, and DOA data corresponds to transmissions received by at least a second transceiver from a second device in the environment. The second transceiver also includes a second transmitter and a second receiver. In this example, the DOA data also corresponds to transmissions received by at least the first transceiver from at least the second transceiver. According to some examples, the first and second transceivers can be configured to transmit and receive electromagnetic waves. In some examples, the first and second smart audio devices can be... Figure 11 Two of the audio devices 1105a to 1105d shown in the figure.
[0396] Depending on the specific implementation method, DOA data can be obtained in various ways. In some instances, determining DOA data may involve the references above. Figure 14 And / or one or more of the DOA-related methods described in the "DOA Robustness Measures" section. Some implementations may involve obtaining DOA data by a control system using one or more elements of beamforming methods, controllable power response methods, time difference of arrival methods, and / or structured signaling methods.
[0397] According to this example, block 3110 relates to receiving configuration parameters by the control system. In this embodiment, the configuration parameters correspond to the environment itself, one or more devices in the audio environment, or both the environment and one or more devices in the audio environment. According to some examples, the configuration parameters may indicate the number of audio devices in the environment, one or more dimensions of the environment, one or more constraints on the position or orientation of the devices, and / or disambiguation data for at least one of rotation, translation, or scaling. In some examples, the configuration parameters may include playback delay data, recording delay data, and / or data used to disambiguate delay symmetry.
[0398] In this example, box 3115 involves minimizing a cost function by the control system, at least in part, based on DOA data and configuration parameters, to estimate the position and orientation of at least the first and second devices.
[0399] According to some implementations, DOA data may also correspond to transmissions made by the third to Nth transceivers of the third to Nth devices in the environment, where N corresponds to the total number of transceivers in the environment, and where the DOA data corresponds to transmissions received by each of the first to Nth transceivers from all other transceivers in the environment. In some such implementations, minimizing the cost function may also involve estimating the location and orientation of the third to Nth transceivers.
[0400] In some examples, the first and second devices can be smart audio devices, and the environment can be an audio environment. In some such examples, the first and second transmitters can be audio transmitters. In some such examples, the first and second receivers can be audio receivers. According to some such examples, the DOA data can also correspond to the sound emitted by the third to Nth smart audio devices in the audio environment, where N corresponds to the total number of smart audio devices in the audio environment. In such examples, the DOA data can also correspond to the sound received by each of the first to Nth smart audio devices from all other smart audio devices in the audio environment. In such instances, minimizing the cost function can involve estimating the location and orientation of the third to Nth smart audio devices. Alternatively or additionally, in some examples, the DOA data can correspond to electromagnetic waves emitted and received by devices in the environment.
[0401] In some examples, DOA data can also correspond to sound received by one or more passive receivers in the environment. Each of the one or more passive receivers may include a receiver array but may lack a transmitter. Minimizing the cost function can also provide an estimated location and orientation for each of the one or more passive receivers. According to some examples, DOA data can also correspond to transmissions from one or more transmitters in the environment. In some such examples, each of the one or more transmitters may lack a receiver array. Minimizing the cost function can also provide an estimated location for each of the one or more transmitters.
[0402] In some examples, method 3100 may involve receiving a seed layout for a cost function by a control system. For example, the seed layout may specify the correct number of transmitters and receivers in the audio environment, as well as the arbitrary position and orientation of each of the transmitters and receivers in the audio environment.
[0403] According to some examples, method 3100 may involve receiving a weighting factor associated with one or more elements of the DOA data by a control system. For example, the weighting factor may indicate the availability and / or reliability of one or more elements of the DOA data.
[0404] In some examples, method 3100 may involve receiving, by a control system, time-of-arrival (TOA) data corresponding to sound emitted by at least one audio device in the audio environment and received by at least one other audio device in the audio environment. In some such examples, a cost function may be based at least in part on the TOA data. Some such implementations may involve estimating at least one playback delay and / or at least one recording delay. According to some such examples, the cost function may operate using a rescaled location, a rescaled delay, and / or a rescaled time of arrival.
[0405] In some examples, the cost function may include a first term that depends only on the DOA data and a second term that depends only on the TOA data. In some such examples, the first term may include a first weighting factor, and the second term may include a second weighting factor. According to some such examples, one or more TOA elements of the second term may have TOA element weighting factors that indicate the availability or reliability of each of the one or more TOA elements.
[0406] The various features and aspects will be understood from the following enumerated example embodiments (“EEE”):
[0407] EEE1. An audio processing method, comprising:
[0408] - Audio data is received by a control system configured to implement multiple renderers;
[0409] - The control system receives listening configuration data for multiple listening configurations, each of which corresponds to a listening position and listening orientation in the audio environment;
[0410] - The audio data is rendered by each of the plurality of renderers and according to the listening configuration data to obtain a renderer-specific loudspeaker feed signal set for a corresponding listening configuration, wherein each renderer is configured to render the audio data for a different listening configuration;
[0411] -The control system decomposes each renderer-specific loudspeaker feed signal set into renderer-specific frequency band sets for each renderer;
[0412] - The control system combines the renderer-specific frequency band set of each renderer to generate a loudspeaker feed signal output set; and
[0413] The control system outputs the amplifier feed signal output set to multiple amplifiers.
[0414] EEE2. The method as described in EEE1, wherein decomposing each renderer-specific loudspeaker feed signal set into each renderer-specific frequency band set includes:
[0415] - The renderer-specific loudspeaker feed signal set is analyzed by an analysis filter bank associated with each renderer to generate a global frequency band set; and
[0416] - Select a subset of the global frequency band set to generate the renderer-specific frequency band set.
[0417] EEE3. The method as described in EEE2, wherein a subset of the global frequency band set is selected such that when combining renderer-specific frequency band sets of all renderers among the plurality of renderers, each frequency band in the global frequency band set is represented only once in the loudspeaker feed signal output set.
[0418] EEE4. The method as described in EEE2 or EEE3, wherein combining the renderer-specific set of frequency bands includes synthesizing the amplifier feed signal output set in the time domain by a synthesis filter bank.
[0419] EEE5. The method of any one of EEE2 to EEE4, wherein the analysis filter bank is selected from the group consisting of filter banks of the following: short-time discrete Fourier transform (STDFT) filter bank, hybrid composite quadrature mirror (HCQMF) filter bank, and quadrature mirror (QMF) filter bank.
[0420] EEE6. The method of any one of EEE1 to EEE5, wherein each set in the renderer-specific frequency band set is uniquely associated with one of the plurality of renderers and uniquely associated with one of the plurality of listening configurations.
[0421] EEE7. The method as described in any one of EEE1 to EEE6, wherein each listening configuration corresponds to the person's listening position and listening orientation.
[0422] EEE8. The method as described in EEE7, wherein the listening position corresponds to the head position of the person, and wherein the listening direction corresponds to the head orientation of the person.
[0423] EEE9. The method of any one of EEE1 to EEE8, wherein the audio data includes at least one of spatial channel-based audio data or spatial object-based audio data.
[0424] EEE10. The method of any one of EEE1 to EEE9, wherein the audio data has a format selected from the group consisting of audio formats such as stereo, 3.1.2, 5.1, 5.1.2, 7.1, 7.1.2, 7.1.4, 9.1, 9.1.6 and Dolby Atmos audio formats.
[0425] EEE11. The method of any one of EEE1 to EEE10, wherein rendering by a renderer among the plurality of renderers includes performing a double-balanced amplitude shift in the time domain or performing crosstalk cancellation in the frequency domain.
[0426] EEE12. An apparatus configured to perform a method as described in any one of EEE1 to EEE11.
[0427] EEE13. A system configured to perform the method as described in any one of EEE1 to EEE11.
[0428] EEE14. One or more non-transitory media having instructions stored thereon that, when executed by a device or system, cause the device or system to perform the method as described in any one of EEE1 to EEE11.
[0429] While specific embodiments and applications have been described herein, it will be apparent to those skilled in the art that many changes can be made to the embodiments and applications described herein without departing from the scope of the description and claims herein. It should be understood that although certain forms have been shown and described, the scope of this disclosure is not limited to the specific embodiments described and shown or the specific methods described.
Claims
1. An audio processing method, comprising: - Audio data is received by the control system; - The control system receives listening configuration data for multiple listening configurations, each of which corresponds to a listening location and listening direction; - The audio data is analyzed by an analysis filter bank implemented via the control system to generate a global frequency band set corresponding to the audio data; - The control system selects a subset of the global frequency band set for each of the plurality of renderers implemented by the control system to generate a renderer-specific frequency band set for each renderer; - Each of the plurality of renderers renders a renderer-specific set of frequency bands according to the listening configuration data to obtain a renderer-specific set of loudspeaker feed signals for a corresponding listening configuration, wherein each renderer is configured to render frequency bands in the renderer-specific set of frequency bands for different listening configurations; - The control system combines the renderer-specific loudspeaker feed signal sets of each of the plurality of renderers to generate a loudspeaker feed signal output set; and - The control system outputs the amplifier feed signal set to multiple amplifiers in the audio environment. Each frequency band subset of the global frequency band set is selected such that when all renderers of the plurality of renderers combine the renderer-specific frequency band set, each frequency band of the global frequency band set is represented only once in the loudspeaker feed signal output set.
2. The method of claim 1, further comprising transforming the loudspeaker feed signal output set from the frequency domain to the time domain by a synthesis filter bank.
3. The method of claim 1 or claim 2, wherein, The analysis filter bank is selected from the following filter banks: Short-Time Discrete Fourier Transform (STDFT) filter bank, Hybrid Composite Quadrature Mirror (HCQMF) filter bank, and Quadrature Mirror (QMF) filter bank.
4. The method of claim 1 or claim 2, wherein, Each renderer-specific set of loudspeaker feed signals is uniquely associated with one of the plurality of renderers and uniquely associated with one of the plurality of listening configurations.
5. The method of claim 1 or claim 2, wherein, The listening configuration includes at least one of the person's listening position and listening direction in the audio environment.
6. The method of claim 5, wherein, The listening position corresponds to the position of the person's head, and the listening direction corresponds to the orientation of the person's head.
7. The method of claim 1 or claim 2, wherein, The audio data includes at least one of spatial channel-based audio data and / or spatial object-based audio data.
8. The method of claim 1 or claim 2, wherein, The audio data has an audio format selected from the group consisting of the following: stereo, 3.1.2, 5.1, 5.1.2, 7.1, 7.1.2, 7.1.4, 9.1, 9.1.6 and Dolby Atmos audio formats.
9. The method of claim 1 or claim 2, wherein, The rendering process includes performing crosstalk cancellation in the frequency domain.
10. The method of claim 1 or claim 2, wherein, Combining the loudspeaker feed signal set involves multiplexing each of the renderer-specific loudspeaker feed signal sets.
11. The method as claimed in claim 1 or claim 2, wherein, The listening configuration data corresponds to sensor data obtained from one or more sensors in the audio environment.
12. The method of claim 11, wherein, The sensor includes at least one of a camera, a motion sensor, or a microphone.
13. The method as claimed in claim 1 or claim 2, wherein, The listening position and the listening direction are relative to the audio environment coordinate system.
14. The method as claimed in claim 1 or claim 2, wherein, The listening position is relative to the location of one or more loudspeakers in the audio environment.
15. The method of claim 1 or claim 2, wherein: The rendering involves generating multiple data structures, each including a set of renderer-specific speaker activations for a corresponding listening configuration and corresponding to each of a plurality of points in two-dimensional or three-dimensional space; and The combination involves combining the multiple data structures into a single data structure.
16. An apparatus for audio processing, the apparatus being configured to perform the method as claimed in any one of claims 1 to 15.
17. A system for audio processing, the system being configured to perform the method as claimed in any one of claims 1 to 15.
18. One or more non-transitory media storing instructions that, when executed by a device or system, cause the device or system to perform the method as claimed in any one of claims 1 to 15.
Citation Information
Patent Citations
Automatic discovery and localization of speaker locations in surround sound systems
US10779084B2
Managing playback of multiple streams of audio over multiple speakers
WO2021021707A1
An apparatus for determining a spatial output multi-channel audio signal
CN102165797A
Apparatus and method for driving an array of loudspeakers with drive signals
US20180098175A1
