Audio device automatic location selection

Abstract

A method for estimating an audio device location within an environment may include obtaining direction of arrival (DOA) data for each audio device among a plurality of audio devices within the environment and determining an interior angle for each triangle among a plurality of triangles based on the DOA data. Each triangle may have a vertex corresponding to an audio device location. The method may include determining a side length for each side of each triangle, performing a forward alignment process that aligns each triangle among the plurality of triangles to generate a forward alignment matrix, and performing a backward alignment process that aligns each triangle among the plurality of triangles in a reverse sequence to generate a backward alignment matrix. A final estimate of each audio device location may be based at least in part on values ​​of the forward alignment matrix and the backward alignment matrix.

Description

[Technical field]

[0001] (Reference to Related Application) This application claims priority to U.S. Provisional Patent Application No. 62 / 949,998, filed December 18, 2019, European Patent Application No. 19217580.0, filed December 18, 2019, and U.S. Provisional Patent Application No. 62 / 992,068, filed March 19, 2020, which are incorporated herein by reference.

[0002] This disclosure relates to systems and methods for automatically locating audio devices. [Background technology]

[0003] Audio devices, including but not limited to smart audio devices, are widely deployed and have become a common feature of many homes. While existing systems and methods for locating audio devices provide benefits, improved systems and methods are desirable. Summary of the Invention

[0004] In this specification, the expression "smart audio device" is used to indicate a smart device that is either a single-purpose audio device or a virtual assistant (e.g., a connected virtual assistant). A single-purpose audio device is a device (e.g., a smart speaker, a television (TV) or a mobile phone) that includes or is coupled to at least one microphone (in some examples, it may also include or be coupled to at least one microphone) that is designed to achieve a largely or primarily single purpose. Although a TV can typically play (or is thought to be able to play) audio from program material, in most cases, modern TVs run some operating system on which applications run locally, including applications for watching television. Similarly, audio inputs and outputs on a mobile phone may do many things, but they are serviced by applications running on the phone. In this sense, a single-purpose audio device with a speaker and a microphone is often configured to run local applications and / or services for directly using the speaker and microphone. Several single-purpose audio devices may be configured to group together to achieve audio playback in a zone or user-defined area.

[0005] In this specification, a "virtual assistant" (e.g., a connected virtual assistant) is a device (e.g., a smart speaker, a smart display, or a voice assistant integrated device) that includes or is coupled to at least one microphone (optionally includes or is coupled to at least one speaker) and that may be cloud-enabled in some sense or provide the ability to utilize multiple devices (separate from the virtual assistant) for applications not implemented in or on the virtual assistant itself. Virtual assistants may sometimes cooperate, for example, in a very discrete and conditionally defined manner. For example, two or more virtual assistants may cooperate in the sense that one of them, i.e., the virtual assistant that is most confident that it has heard a wake word, responds to that word. The connected devices may form a kind of constellation, which may be managed by one main application that may be (or include or implement) a virtual assistant.

[0006] As used herein, "wake word" is used broadly to denote any sound (e.g., a word spoken by a human being, or any other sound) that the smart audio device is configured to wake up in response to detecting (using at least one microphone included in or coupled to the smart audio device, or at least one other microphone) that sound. In this context, "wake up" means that the device enters a state in which it waits for (i.e., listens for) a sound command.

[0007] As used herein, the term "wake word detector" refers to a device (or software including instructions for configuring the device) configured to continuously search for alignment between real-time sound (e.g., speech) mechanisms and a trained model. Typically, a wake word event is triggered whenever the wake word detector determines that the probability that a wake word has been detected exceeds a predefined threshold. For example, the threshold may be a predefined threshold that is adjusted to provide a good compromise between false acceptance and false rejection. Following a wake word event, the device may enter a state (sometimes referred to as a state of "awakened" or "attentiveness") in which the device listens for commands and passes received commands to a larger, more computationally intensive recognizer.

[0008] Throughout this disclosure, including the claims, "speaker" and "loudspeaker" are used interchangeably to refer to any sound emitting transducer (or set of transducers) driven by a single speaker feed. A typical set of headphones includes two speakers. A speaker may be implemented to include multiple transducers (e.g., a woofer and a tweeter) all driven by a single common speaker feed. The speaker feeds may in some instances undergo different processing in different circuitry branches coupled to different transducers.

[0009] Throughout this disclosure, including the claims, expressions of performing an operation to turn a signal or data "on" (e.g., filtering, scaling, transforming, or applying a gain to the signal or data) are used broadly to indicate performing the operation directly on the signal or data or on a processed version of the signal or data (e.g., on a version of the signal that has undergone preliminary filtering or preprocessing before the operation is performed).

[0010] Throughout this disclosure, including the claims, the term "system" is used broadly to denote a device, a system, or a subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, where the subsystem generates M inputs and the other XM inputs are received from external sources) may be referred to as a decoder system.

[0011] Throughout this disclosure, including the claims, the term "processor" is used broadly to denote a system or device that is programmable or otherwise configurable (e.g., with software or firmware) to perform operations on data (e.g., audio, video or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors programmed and / or otherwise configured to perform pipeline processing on audio or other sound data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets.

[0012] At least some aspects of the present disclosure may be implemented via methods. Some such methods may include methods for determining audio device locations, i.e., locations of a plurality of (e.g., at least four or more) audio devices in an environment. For example, some methods may include obtaining direction of arrival (DOA) data for each audio device of the plurality of audio devices and determining an interior angle for each triangle of the plurality of triangles based on the DOA data. In some examples, each triangle of the plurality of triangles may have vertices that correspond with locations of three audio devices of the audio devices. Some such methods may include determining a side length for each side of each triangle of the triangle based at least in part on the interior angle.

[0013] Some such methods may include performing a forward alignment process to align each triangle of the plurality of triangles in a first sequence to generate a forward alignment matrix. Some such methods may include performing a reverse alignment process to align each triangle of the plurality of triangles in a second sequence that is the reverse of the first sequence to generate a reverse alignment matrix. Some such methods may include generating a final estimate of each audio device location based at least in part on values ​​of the forward alignment matrix and values ​​of the reverse alignment matrix.

[0014] According to some examples, generating a final estimate of each audio device location may include translating and scaling a forward alignment matrix to generate a translated and scaled forward alignment matrix, and translating and scaling a reverse alignment matrix to generate a translated and scaled reverse alignment matrix. Some such methods may include generating a rotation matrix based on the translated and scaled forward alignment matrix and the translated and scaled reverse alignment matrix. The rotation matrix may include multiple estimated audio device locations for each audio device. In some implementations, generating the rotation matrix may include performing singular value decomposition on the translated and scaled forward alignment matrix and the translated and scaled reverse alignment matrix. According to some examples, creating a final estimate of each audio device location may include averaging the estimated audio device locations for each audio device to create a final estimate of each audio device location.

[0015] In some implementations, determining the side lengths may include determining a first length of a first side of the triangle and determining a second side length and a third side length of the triangle based on an interior angle of the triangle. Determining the first length may, in some examples, include setting the first length to a predetermined value. Determining the first length may, in some examples, be based on time of arrival data and / or received signal strength data.

[0016] According to some examples, obtaining the DOA data may include determining DOA data for at least one audio device of the plurality of audio devices. In some examples, determining the DOA data may include receiving microphone data from each microphone of the plurality of audio device microphones corresponding to a single audio device of the plurality of audio devices and determining the DOA data for the single audio device based at least in part on the microphone data. According to some examples, determining the DOA data may include receiving antenna data from one or more antennas corresponding to the single audio device of the plurality of audio devices and determining the DOA data for the single audio device based at least in part on the antenna data.

[0017] In some implementations, the method may include controlling at least one of the audio devices based at least in part on the final estimate of the at least one audio device location. In some such examples, controlling the at least one audio device of the audio devices may include controlling a loudspeaker of the at least one audio device of the audio devices.

[0018] Some or all of the operations, functions and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including, but not limited to, random access memory (RAM) devices, read-only memory (ROM) devices, etc. Thus, some innovative aspects of the subject matter described in this disclosure may be implemented in a non-transitory medium having software stored thereon.

[0019] For example, the software may include instructions for controlling one or more devices to perform a method including audio device locations. Some methods may include obtaining DOA data for each audio device of the plurality of audio devices and determining an interior angle for each triangle of the plurality of triangles based on the DOA data. In some examples, each triangle of the plurality of triangles may have vertices corresponding to audio device locations of three audio devices of the audio devices. Some such methods may include determining a side length for each side of each triangle of the triangles based at least in part on the interior angle.

[0020] Some such methods may include performing a forward alignment process to align each triangle of the plurality of triangles in a first sequence to generate a forward alignment matrix. Some such methods may include performing a reverse alignment process to align each triangle of the plurality of triangles in a second sequence that is the reverse of the first sequence to generate a reverse alignment matrix. Some such methods may include generating a final estimate of each audio device location based at least in part on values ​​of the forward alignment matrix and values ​​of the reverse alignment matrix.

[0021] According to some examples, generating a final estimate of each audio device location may include translating and scaling a forward alignment matrix to generate a translated and scaled forward alignment matrix, and translating and scaling a reverse alignment matrix to generate a translated and scaled reverse alignment matrix. Some such methods may include generating a rotation matrix based on the translated and scaled forward alignment matrix and the translated and scaled reverse alignment matrix. The rotation matrix may include multiple estimated audio device locations for each audio device. In some implementations, generating the rotation matrix may include performing singular value decomposition on the translated and scaled forward alignment matrix and the translated and scaled reverse alignment matrix. According to some examples, creating a final estimate of each audio device location may include averaging the estimated audio device locations for each audio device to create a final estimate of each audio device location.

[0022] In some implementations, determining the side lengths may include determining a first length of a first side of the triangle and determining a second side length and a third side length of the triangle based on an interior angle of the triangle. Determining the first length may, in some examples, include setting the first length to a predetermined value. Determining the first length may, in some examples, be based on time of arrival data and / or received signal strength data.

[0023] According to some examples, obtaining the DOA data may include determining DOA data for at least one audio device of the plurality of audio devices. In some examples, determining the DOA data may include receiving microphone data from each microphone of the plurality of audio device microphones corresponding to a single audio device of the plurality of audio devices and determining the DOA data for the single audio device based at least in part on the microphone data. According to some examples, determining the DOA data may include receiving antenna data from one or more antennas corresponding to the single audio device of the plurality of audio devices and determining the DOA data for the single audio device based at least in part on the antenna data.

[0024] In some implementations, the method may include controlling at least one of the audio devices based at least in part on the final estimate of the at least one audio device location. In some such examples, controlling the at least one audio device of the audio devices may include controlling a loudspeaker of the at least one audio device of the audio devices.

[0025] At least some aspects of the present disclosure may be implemented via an apparatus. For example, one or more devices may be capable of performing, at least in part, the methods disclosed herein. In some implementations, the apparatus may include an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or combinations thereof. In some examples, the apparatus may be one of the audio devices described above. However, in some implementations, the apparatus may be another type of device, such as a mobile device, a laptop, a server, or the like.

[0026] In some aspects of the present disclosure, any of the methods described may be implemented in a computer program (computer program product) including instructions that, when executed by a computer, cause the computer to perform any of the methods or steps of the methods described in the present disclosure.

[0027] In some aspects of the present disclosure, a computer-readable medium that includes a computer program product is described.

[0028] Details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. It should be noted that the relative dimensions of the following figures may not be drawn to scale. [Brief description of the drawings]

[0029] [Figure 1] 1 shows an example of the geometric relationships between three audio devices in an environment. [Diagram 2]2 illustrates another example of the geometric relationships between three audio devices in the environment shown in FIG. 1. [Figure 3A] FIG. 3 shows both triangles shown in FIGS. 1 and 2 without the corresponding audio devices and other configurations of the environment. [Figure 3B] This shows an example of estimating the interior angles of a triangle formed by three audio devices. [Figure 4] 12 is a flow diagram outlining an example of a method that may be performed by an apparatus such as that shown in FIG. [Diagram 5] An example is shown in which each audio device in the environment is a vertex of multiple triangles. [Figure 6] 1 provides an example of a portion of the forward alignment process. [Figure 7] 13 shows examples of multiple estimates of audio device locations generated during the forward alignment process. [Figure 8] 1 provides an example of a portion of the reverse alignment process. [Figure 9] 13 shows examples of multiple estimates of audio device locations generated during the backward alignment process. [Figure 10] 1 shows a comparison of estimated and actual audio device locations. [Figure 11] 1 is a block diagram illustrating example components of an apparatus capable of implementing various aspects of the disclosure. [Figure 12] 12 is a flow diagram outlining an example of a method that may be performed by an apparatus such as that shown in FIG. [Figure 13A] 13 shows examples of some of the blocks in FIG. [Figure 13B] 13 illustrates an additional example of determining angular viewer orientation data. [Figure 13C] 13 illustrates an additional example of determining angular viewer orientation data. [Figure 13D]13D illustrates an example of determining an appropriate rotation for audio device coordinates according to the method described with reference to FIG. 13C. [Figure 14] 11 shows the speaker activation including the optimal solution of Equation 11 for these particular speaker positions. [Figure 15] Speaker activation is plotted for each individual speaker position shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] Like reference numbers and designations in the various drawings indicate like elements.

[0031] In addition to existing audio devices including televisions and sound bars, and new microphone- and loudspeaker-enabled connected devices such as light bulbs and microwaves, the emergence of smart speakers incorporating multiple driving units and microphone arrays has created the problem of needing to position dozens of microphones and loudspeakers relative to each other to achieve orchestration. Audio devices cannot be considered to be in a standard layout (such as a discrete Dolby 5.1 loudspeaker layout). In some cases, audio devices in an environment may be randomly positioned, or at least distributed in an irregular and / or asymmetric manner in the environment.

[0032] Furthermore, audio devices cannot be assumed to be heterogeneous or synchronous. As used herein, audio devices may be referred to as "synchronous" or "synchronized" if sounds are detected or emitted by the audio devices according to the same or synchronized sample clocks. For example, a first synchronized microphone of a first audio device in an environment may digitally sample audio data according to a first sample clock, and a second microphone of a second synchronized audio device in the environment may digitally sample audio data according to the first sample clock. Alternatively or additionally, a first synchronized speaker of a first audio device in an environment may emit sounds according to a speaker setup clock, and a second synchronized speaker of a second audio device in the environment may emit sounds according to its speaker setup clock.

[0033] Some fully disclosed methods for automatic speaker location require synchronized microphones and / or speakers. For example, some previous tools for device localization rely on sample synchrony between all microphones in the system, require known test stimuli, and pass full bandwidth audio data between sensors.

[0034] The assignee has produced several loudspeaker location techniques for cinema and home that are excellent solutions in the use cases for which they were designed. Some such methods are based on time-of-flight derived from the impulse response between the sound source and a microphone that is approximately co-located with each loudspeaker. System latencies in the record and playback chains may also be estimated, but sample synchrony between clocks is required, along with the need for known test stimuli to estimate the impulse response.

[0035] Recent examples of source localization in this context relax the constraint by requiring intra-device microphone synchrony but not inter-device synchrony. In addition, some such methods forgo the need to pass sound between sensors by low-bandwidth message passing, such as through detection of the time of arrival (TOA) of direct (unreflected) sound or through detection of the dominant direction of arrival (DOA) of direct sound. Each approach has some potential advantages and potential disadvantages. For example, TOA methods can determine device geometry up to unknown translation, rotation, and reflection about one of three axes. If there is only one microphone per device, the rotation of each individual device is also unknown. DOA methods can determine device geometry up to unknown translation, rotation, and scale. Some such methods produce satisfactory results under ideal conditions, but the robustness of such methods to measurement errors has not been demonstrated.

[0036] Some implementations of the present disclosure automatically locate the positions of multiple audio devices within an environment (e.g., within a room) by applying a geometry-based optimization that uses asynchronous DOA estimates from uncontrolled sound sources observed by microphone arrays within each device. The various disclosed audio device location approaches have been proven to be robust to large DOA estimation errors.

[0037] Some such implementations include iteratively aligning triangles derived from the set of DOA data. In some such examples, each audio device may include a microphone array that estimates the DOA from uncontrolled sources. In some implementations, the microphone array may be co-located with at least one loudspeaker. However, at least some of the disclosed methods generalize to cases where not all microphone arrays are co-located with the loudspeakers.

[0038] According to some disclosed methods, DOA data from every audio device to every other audio device in the environment may be aggregated. Audio device locations may be estimated by iteratively aligning a triangle parameterized by pairs of DOAs. Some such methods may yield correct results up to unknown scales and rotations. In many applications, absolute scale is not necessary and rotations can be resolved by adding additional constraints to the solution. For example, some multi-speaker environments may include television (TV) speakers and a couch positioned for TV viewing. After locating the speakers in the environment, some methods may include finding a vector pointing to the TV and locating the speech of a user sitting on the couch by triangulation. Some such methods may then include having the TV radiate sound from its speakers and / or prompting the user to walk to the TV and locating the user's speech by triangulation. Some implementations may include rendering an audio object that pans around the environment. A user may provide user input (e.g., saying "stop") indicating when an audio object is at one or more predetermined locations in the environment, such as at the front of the environment, at a TV location in the environment, etc. According to some such examples, after locating speakers in the environment and determining their orientations, a user may be located by finding the intersection of directions of arrival of sounds emitted by multiple speakers. Some implementations include determining an estimated distance between at least two audio devices and scaling distances between other audio devices in the environment according to the estimated distance.

[0039] FIG. 1 illustrates an example of a geometric relationship between three audio devices in an environment. In this example, the environment 100 is a room including a television 101, a sofa 105, and five audio devices 105. According to this example, the audio devices 105 are located at locations 1-5 in the environment 100. In this implementation, each of the audio devices 105 includes a microphone system 120 having at least three microphones and a speaker system 125 including at least one speaker. In some implementations, each microphone system 120 includes an array of microphones. According to some implementations, each of the audio devices 105 may include an antenna system including at least three antennas.

[0040] 1 are made by way of example only. Other implementations may have different types, numbers and arrangements of elements, e.g., more or fewer audio devices 105, audio devices 105 in different locations, etc.

[0041] In this example, triangle 110a has its vertices at locations 1, 2, and 3. Here, triangle 110a has sides 12, 23a, and 13a. According to this example, the angle between sides 12 and 23 is θ2, the angle between sides 12 and 13a is θ1, and the angle between sides 23a and 13a is θ3. These angles may be determined according to the DOA data, as described in more detail below.

[0042] In some implementations, only the relative lengths of the triangle sides may be determined. In alternative implementations, the actual lengths of the triangle sides may be estimated. According to some such implementations, the actual lengths of the triangle sides may be estimated according to TOA data, for example, according to the arrival time of a sound generated by an audio device located at one triangle vertex and detected by an audio device located at another triangle vertex. Alternatively or additionally, the lengths of the triangle sides may be estimated according to an electromagnetic wave generated by an audio device located at one triangle vertex and detected by an audio device located at another triangle vertex. For example, the lengths of the triangle sides may be estimated according to a signal strength of an electromagnetic wave generated by an audio device located at one triangle vertex and detected by an audio device located at another triangle vertex. In some implementations, the lengths of the triangle sides may be estimated according to a detected phase shift of the electromagnetic wave.

[0043] Figure 2 shows another example of the geometric relationships between three audio devices in the environment shown in Figure 1. In this example, triangle 110b has its vertices at locations 1, 3, and 4. Here, triangle 110b has sides 13b, 14, and 34a. According to this example, the angle between sides 13b and 14 is θ4, the angle between sides 13b and 34a is θ5, and the angle between sides 34a and 14 is θ6.

[0044] 1 and 2, it is observed that the length of side 13a of triangle 110a should be equal to the length of side 13b of triangle 110b. In some implementations, the length of the side of one triangle (e.g., triangle 110a) may be assumed to be correct, and the length of the side shared by adjacent triangles is constrained to this length.

[0045] FIG. 3A shows both triangles shown in FIG. 1 and FIG. 2 without the corresponding audio devices and other configurations of the environment. FIG. 3 shows estimates of the side lengths and angular orientations of triangles 110a and 110b. In the example shown in FIG. 3A, the length of side 13b of triangle 110b is constrained to be the same length as side 13a of triangle 110a. The lengths of the other sides of triangle 110b are scaled in proportion to the resulting change in the length of side 13b. The resulting triangle 110b' is shown in FIG. 3A adjacent to triangle 110a.

[0046] According to some implementations, the side lengths of all other triangles adjacent to triangles 110a and 110b may be determined in a similar manner until all audio device locations within environment 100 have been determined.

[0047] Some examples of audio device locations follow: Each audio device may report the DOA of every other audio device in an environment (e.g., a room) based on the sounds produced by every other audio device in the environment. The Cartesian coordinates of the ith audio device are x i =[x i ,y i ] T where the superscript T denotes a vector transposition. If there are M audio devices in the environment, then i={1...M}.

[0048] FIG. 3B shows an example of estimating the interior angles of a triangle formed by three audio devices. In this example, audio devices are i, j, and k. The DOA of a sound source emanating from device j as observed from device i is expressed as θ ji The DOA of a source emanating from device k as observed by device i is sometimes expressed as θ ki In the example shown in FIG. ji and θ kiare measured from axis 305a, and their orientation may be arbitrary, for example, they may correspond to the orientation of audio device i. The interior angle a of triangle 310 is a=θ ki -θ ji It may be observed that the calculation of the internal angle a is independent of the orientation of the axis 305a.

[0049] In the example shown in FIG. ij and θ kj are measured from axis 305b, and their orientation may be arbitrary and may correspond to the orientation of audio device j. The interior angle b of triangle 310 is given by b=θ ij -θ kj Similarly, θ jk and θ ik In this example, the angle c of the triangle 310 is measured from the axis 305c. jk -θ ik It may be expressed as.

[0050] In the presence of measurement errors, a+b+c≠180°. Robustness can be improved by predicting each angle from the other two angles and averaging, for example:

number

[0051] In some implementations, the edge lengths (A,B,C) may be calculated by applying the sine law (up to scaling error). In some instances, an edge length may be assigned an arbitrary value, such as 1. For example, make A=1 and vertex (outside 1) By placing TIFF0007665630000002.tif7170 at the origin, the locations of the remaining two vertices may be calculated as follows:

number

[0052] However, any rotation may be acceptable.

[0053] In some implementations, the process of triangle parameterization is (outside 2) The process may be repeated for all possible subsets of three audio devices in the environment, as listed in the superset ζ of TIFF0007665630000004.tif12170. l may represent the l-th triangle. Depending on the implementation, the triangles may not be listed in any particular order. The triangles may overlap and may not be perfectly aligned due to possible errors in the DOA and / or side length estimates.

[0054] 4 is a flow diagram outlining an example of a method that may be performed by an apparatus such as that shown in FIG. 11. The blocks of method 400, as well as other methods described herein, are not necessarily performed in the order shown. Moreover, such methods may include more or fewer blocks than shown and / or described. In this implementation, method 400 includes estimating a location of a speaker within an environment. The blocks of method 400 may be performed by one or more devices, which may be (or may include) apparatus 1100 shown in FIG. 11.

[0055] In this example, block 405 includes obtaining direction of arrival (DOA) data for each audio device of the plurality of audio devices. In some examples, the plurality of audio devices may include all audio devices in an environment, such as all audio devices 105 shown in FIG.

[0056] However, in some cases, the plurality of audio devices may include only a subset of all audio devices in the environment, for example, the plurality of audio devices may include all smart speakers in the environment, but not one or more of the other audio devices in the environment.

[0057] The DOA data may be obtained in various ways, depending on the particular implementation. In some examples, determining the DOA data may include determining DOA data for at least one audio device of the multiple audio devices. For example, determining the DOA data may include receiving microphone data from each microphone of a plurality of audio device microphones corresponding to a single audio device of the multiple audio devices, and determining the DOA data for the single audio device based at least in part on the microphone data. Alternatively or additionally, determining the DOA data may include receiving antenna data from one or more antennas corresponding to the single audio device of the multiple audio devices, and determining the DOA data for the single audio device based at least in part on the antenna data.

[0058] In some such instances, a single audio device itself may determine the DOA data. According to some such implementations, each audio device of the multiple audio devices may determine its own DOA data. However, in other implementations, another device, which may be a local or remote device, may determine the DOA data for one or more audio devices in the environment. According to some implementations, a server may determine the DOA data for one or more audio devices in the environment.

[0059] According to this example, block 410 includes determining interior angles of each triangle of the plurality of triangles based on the DOA data. In this example, each triangle of the plurality of triangles has vertices corresponding to audio device locations of three audio devices. Some such examples are described above.

[0060] 5 shows an example in which each audio device in the environment is a vertex of multiple triangles, with each triangle side corresponding to the distance between two audio devices 105.

[0061] In this implementation, block 415 includes determining a side length for each side of each triangle. (A side of a triangle may also be referred to herein as an "edge.") According to this example, the side lengths are based at least in part on the interior angles. In some cases, the side lengths may be calculated by determining a first length of a first side of the triangle and determining lengths of a second and third side of the triangle based on the interior angles of the triangle. Some such examples are described above.

[0062] According to some such implementations, determining the first length may include setting the first length to a predetermined value. The lengths of the second and third sides may then be determined based on the interior angles of the triangle. All sides of the triangle may be determined based on a predetermined value, e.g., a reference value. To obtain the actual distances (lengths) between the audio devices in the environment, a standardized scaling may be applied to the geometric shapes resulting from the alignment process described below with reference to blocks 420 and 425 of FIG. 4. This standardized scaling may include scaling the aligned triangles such that they fit into a bounding shape of a size corresponding to the environment, e.g., a circle, a polygon, etc. The size of the shape may be the size of a typical home environment or any size suitable for a particular implementation. However, scaling the aligned triangles is not limited to fitting the geometric shapes into a particular bounding shape, and any other scaling criteria suitable for a particular implementation may be used.

[0063] In some examples, determining the first length may be based on time of arrival data and / or received signal strength data. The time of arrival data and / or received signal strength data may, in some implementations, correspond to sound waves from a first audio device in the environment detected by a second audio device in the environment. Alternatively or additionally, the time of arrival data and / or received signal strength data may correspond to electromagnetic waves (e.g., radio waves, infrared waves, etc.) from a first audio device in the environment detected by a second audio device in the environment. When time of arrival data and / or received signal strength data is not available, the first strength may be set to a predetermined value as described above.

[0064] According to this example, block 420 includes performing a forward alignment process to align each triangle of the plurality of triangles in the first sequence. According to this example, the forward alignment process generates a forward alignment matrix.

[0065] According to some such examples, a triangle may be defined as having edges (x i ,x j ) is equal to the adjacent edges. Let ε be the set of all edges of size (Outside 3) TIFF0007665630000005.tif12170. In some such implementations, block 420 may include aligning common edges of triangles in the forward direction by traversing through ε and matching the edges with previously aligned edges.

[0066] Figure 6 provides an example of a portion of the forward alignment process. The numbers 1-5 shown in bold in Figure 6 correspond to the audio device locations shown in Figures 1, 2 and 5. The sequence of the forward alignment process shown in Figure 6 and described herein is merely an example.

[0067] In this example, as in Figure 3A, the length of side 13b of triangle 110b is made to match the length of side 13a of triangle 110a. The resulting triangle 110b' is shown in Figure 6, with the same interior angles maintained. According to this example, the length of side 13c of triangle 110c is also made to match the length of side 13a of triangle 110a. The resulting triangle 110c' is shown in Figure 6, with the same interior angles maintained.

[0068] Next, in this example, the length of side 34b of triangle 110d is made to match the length of side 34a of triangle 110b'. Further, in this example, the length of side 23b of triangle 110d is made to match the length of side 23a of triangle 110a. The resulting triangle 110d', with the same interior angles maintained, is shown in Figure 6. According to some such examples, the remaining triangles shown in Figure 5 may be treated in the same manner as triangles 110b, 110c, and 110d.

[0069] The results of the forward alignment process may be stored in a data structure. According to some such examples, the results of the forward alignment process may be stored in a forward alignment matrix. For example, the results of the forward alignment process may be stored in a matrix (outside 4) The resulting image may be stored in file TIFF0007665630000006.tif8170, where N indicates the total number of triangles.

[0070] Multiple estimates of audio device location are generated when the DOA data and / or initial edge length determinations contain errors, and the errors typically increase during the forward alignment process.

[0071] FIG. 7 shows an example of multiple estimates of audio device locations generated during the forward alignment process. In this example, the forward alignment process is based on a triangle with seven audio device locations as vertices, where the triangle does not align perfectly due to additive errors in the DOA estimates. Locations numbered 1-7 shown in FIG. 7 correspond to estimated audio device locations generated by the forward alignment process. In this example, the audio device location estimates marked "1" match, but the audio device location estimates for audio devices "6" and "7" show larger differences, as indicated by the relatively larger regions in which the numbers 6 and 7 are located.

[0072] Returning to FIG. 4, in this example, block 425 includes a backward alignment process that aligns each triangle of the plurality of triangles in a second sequence that is the reverse of the first sequence. According to some implementations, the backward alignment process may include traversing through ε as before, but in reverse order. In alternative examples, the backward alignment process may not be exactly the reverse of the sequence of operations of the forward alignment process. According to this example, the backward alignment process is referred to herein as (outside 5) Generates a reverse alignment matrix, sometimes represented as TIFF0007665630000007.tif8170.

[0073] Figure 8 provides an example of a portion of the backward alignment process. The numbers 1-5 shown in bold in Figure 8 correspond to the audio device locations shown in Figures 1, 2 and 5. The sequence of the backward alignment process shown in Figure 8 and described herein is merely an example.

[0074] In the example shown in FIG. 8, triangle 110e is based on audio device locations 3, 4, and 5. In this implementation, the side lengths (or "edges") of triangle 110e are assumed to be correct, and the side lengths of adjacent triangles are matched with them. According to this example, the length of side 45b of triangle 110f is matched with the length of side 45a of triangle 110e. The resulting triangle 110f', in which the interior angles remain the same, is shown in FIG. 8. In this example, the length of side 35b of triangle 110c is matched with the length of side 35a of triangle 110e. The resulting triangle 110c" in which the interior angles remain the same is shown in FIG. 8. According to some such examples, the remaining triangles shown in FIG. 5 may be processed similarly to triangles 110c and 110f until the backward alignment process includes all remaining triangles.

[0075] FIG. 9 shows an example of multiple estimates of audio device locations generated during the backward alignment process. In this example, the backward alignment process is based on a triangle with the seven audio device locations described above with reference to FIG. 7 as vertices. Locations numbered 1-7 shown in FIG. 9 correspond to the estimated audio device locations generated by the backward alignment process. Again, the triangle does not align perfectly due to additive errors in the DOA estimates. In this example, the audio device location estimates marked 6 and 7 match, while the audio device location estimates for audio devices 1 and 2 show a larger difference.

[0076] Returning to Figure 4, block 430 includes generating a final estimate of each audio device location based at least in part on the forward alignment matrix values ​​and the reverse alignment matrix values. In some examples, generating a final estimate of each audio device location may include translating and scaling the forward alignment matrix to generate a translated and scaled forward alignment matrix, and translating and scaling the reverse alignment matrix to generate a translated and scaled reverse alignment matrix.

[0077] For example, translation and scaling involve moving the center of gravity to the origin and making the units the Frobenius norm, e.g. (outside 6) TIFF0007665630000008.tif10170 and (outer 7) Fixed by TIFF0007665630000009.tif10170.

[0078] According to some such examples, generating a final estimate of each audio device location may include generating a rotation matrix based on a translated and scaled forward alignment matrix and a translated and scaled reverse alignment matrix. The rotation matrix may include a plurality of estimated audio device locations for each audio device. An optimal rotation between the forward alignment and the reverse alignment may be found, for example, by singular value decomposition. In some such examples, generating the rotation matrix may include, for example, as follows:

number

[0079] In the above formula, U and V are respectively the matrices (outside 8) represents the left and right singular vectors of TIFF0007665630000011.tif8170. Σ represents the matrix of singular values. The above formula is the rotation matrix R=VU T The matrix multiplication VU T teeth, (outer 9) TIFF0007665630000012.tif8170 is optimally rotated (Outside 10) This results in a rotation matrix that aligns the image with TIFF0007665630000013.tif8170.

[0080] According to some examples, the rotation matrix R = VU T After determining, the alignments may be averaged, for example, as follows:

number

[0081] In some implementations, generating a final estimate of each audio device location may include averaging the estimated audio device locations of each audio device to generate a final estimate of each audio device location. Various disclosed implementations have proven robust even when the DOA data and / or other calculations contain significant errors. For example, (Outside 11) TIFF0007665630000015.tif8170 is (Outside 12) TIFF0007665630000016.tif11170, i.e., multiple estimates of the same node due to overlapping vertices of multiple triangles. Averaging over the common nodes results in a final estimate (Outside 13) results in TIFF0007665630000017.tif9170.

[0082] FIG. 10 shows a comparison of the estimated audio device locations and the actual audio device locations. In the examples shown in FIG. 10, the audio device locations correspond to the locations estimated during the forward and reverse alignment processes described above with reference to FIG. 7 and FIG. 9. In these examples, the errors in the DOA estimates had a standard deviation of 15 degrees. Nevertheless, the final estimates of each audio device location (each of them is represented by an "x" in FIG. 10) match well with the actual audio device locations (each of them is represented by a circle in FIG. 10). By performing a forward alignment process in a first sequence and a reverse alignment process in a second sequence opposite to the first sequence, the errors / inaccuracies in the direction of arrival estimates (data) are averaged out, thereby reducing the overall error of the estimates of the audio device locations in the environment. As shown in FIG. 7 (where a higher number of vertices indicates a larger spread of the alignment) and FIG. 9 (where a lower number of vertices indicates a larger spread), the errors tend to accumulate in the alignment sequences. The process of traversing the sequences in reverse order also reverses the alignment errors, thereby averaging out the overall errors in the final location estimate.

[0083] 11 is a block diagram illustrating examples of components of an apparatus capable of implementing various aspects of the disclosure. According to some examples, the apparatus 1100 may be or include a smart audio device (such as a smart speaker) configured to perform at least a portion of the methods disclosed herein. In other implementations, the apparatus 1100 may be or include another device configured to perform at least a portion of the methods disclosed herein. In some such implementations, the apparatus 1100 may be or include a server.

[0084] In this example, the device 1100 includes an interface system 1105 and a control system 1110. The interface system 1105 may be configured to receive input from each of a plurality of microphones in an environment in some implementations. The interface system 1105 may include one or more network interfaces and / or one or more external device interfaces (such as one or more Universal Serial Bus (USB) interfaces). According to some implementations, the interface system 1105 may include one or more wireless interfaces. The interface system 1105 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. In some examples, the interface system 1105 may include one or more interfaces between the control system 1110 and a memory system, such as the optional memory system 1115 shown in FIG. 11. However, the control system 1110 may include a memory system.

[0085] The control system 1110 may include, for example, a general purpose single or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components. In some implementations, the control system 1110 may reside in more than one device. For example, a portion of the control system 1110 may reside in a device within the environment 100 shown in FIG. 1, and another portion of the control system 1110 may reside in a device outside the environment 100, such as a server, a mobile device (e.g., a smartphone or tablet computer), etc. The interface system 1105 may reside in more than one device in some such examples.

[0086] In some implementations, the control system 1110 may be configured to perform, at least in part, the methods disclosed herein. According to some examples, the control system 1110 may be configured to implement, for example, the methods described above with reference to Figure 4 and / or the methods described below with reference to Figure 12 et seq. In some such examples, the control system 1110 may be configured to determine an estimate of each location of a plurality of audio device locations in the environment based at least in part on output from a classifier.

[0087] In some examples, the device 1100 may include an optional microphone system 1120 shown in FIG. 11. The microphone system 1120 may include one or more microphones. In some examples, the microphone system 1120 may include an array of microphones. In some examples, the device 1100 may include an optional speaker system 1125 shown in FIG. 11. The speaker system 1125 may include one or more loudspeakers. In some examples, the microphone system 1120 may include an array of loudspeakers. In some such examples, the device 1100 may be or include an audio device. For example, the device 1100 may be or include one of the audio devices 105 shown in FIG. 1.

[0088] In some examples, the apparatus 1100 may include an optional antenna system 1130 shown in FIG. 11. According to some examples, the antenna system 1130 may include an array of antennas. In some examples, the antenna system 1130 may be configured to transmit and / or receive electromagnetic waves. According to some implementations, the control system 1110 may be configured to estimate a distance between two audio devices in the environment based on antenna data from the antenna system 1130. For example, the control system 1110 may be configured to estimate a distance between two audio devices in the environment according to a time of arrival of the antenna data and / or a signal strength of the received antenna data.

[0089] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored in one or more non-transitory media. Such non-transitory media may include memory devices, such as those described herein, including, but not limited to, random access memory (RAM) devices, read-only memory (ROM) devices, and the like. The one or more non-transitory media may be present, for example, in optional memory system 1115 shown in FIG. 11 and / or in control system 1110. Thus, various innovative aspects of the subject matter described in this disclosure may be implemented in one or more non-transitory media, the one or more non-transitory media having software stored thereon. The software may include, for example, instructions for controlling at least one device to process audio data. The software may be executable by one or more components of a control system, such as, for example, control system 1110 of FIG. 11.

[0090] Much of the above discussion relates to audio device auto-location. The following discussion is based on several methods of determining listener location and listener angular orientation briefly described above. In the above description, the term "rotation" is used in essentially the same way as the term "orientation" is used in the following description. For example, the above "rotation" may refer to the overall rotation of the final speaker geometry, rather than the rotation of individual triangles during the process described above with reference to FIG. 4 et seq. This overall rotation or orientation may be determined by, for example, the direction the listener is looking, the direction the listener's nose is pointing, etc., with reference to the listener angular orientation.

[0091] Various satisfactory methods of estimating viewer location are known in the art, some of which are described below. However, estimating viewer angular orientation can be difficult. Some relevant methods are detailed below.

[0092] Determining the audience location and audience angular orientation can enable a number of desired configurations, such as orienting positioned audio devices relative to the audience. Knowing the audience location and angular orientation can enable, for example, determining which speakers in the environment are in front of the audience, which are in the back, which (if any) are near the center, etc.

[0093] After performing a correlation between the audio device location and the viewer location and orientation, some implementations may include providing the audio device location data, the audio device angular orientation data, the viewer location data, and the viewer angular orientation data to an audio rendering system. Alternatively or additionally, some implementations may include an audio data rendering process based, at least in part, on the audio device location data, the audio device angular orientation data, the viewer location data, and the viewer angular orientation data.

[0094] FIG. 12 is a flow diagram outlining an example of a method that may be performed by an apparatus such as that shown in FIG. 11. The blocks of method 1200, as well as other methods described herein, are not necessarily performed in the order shown. Furthermore, such methods may include more or fewer blocks than shown and / or described. In this example, the blocks of method 1200 are performed by a control system that may be (or may include) control system 1110 shown in FIG. 11. As mentioned above, in some implementations, control system 1110 may reside in a single device, while in other embodiments, control system 1110 may reside in two or more devices.

[0095] In this example, block 1205 includes obtaining direction of arrival (DOA) data for each audio device of a plurality of audio devices in the environment. In some examples, the plurality of audio devices may include all audio devices in the environment, such as all audio devices 105 shown in FIG.

[0096] However, in some cases, the plurality of audio devices may include only a subset of all audio devices in the environment, for example, the plurality of audio devices may include all smart speakers in the environment, but may not include one or more of the other audio devices in the environment.

[0097] The DOA data may be obtained in various ways, depending on the particular implementation. In some examples, determining the DOA data may include determining DOA data of at least one audio device of the multiple audio devices. In some examples, the DOA data may be obtained by controlling each loudspeaker of the multiple loudspeakers in the environment to reproduce a test signal. For example, determining the DOA data may include receiving microphone data from each microphone of the multiple audio device microphones corresponding to a single audio device of the multiple audio devices, and determining the DOA data for the single audio device based at least in part on the microphone data. Alternatively or additionally, determining the DOA data may include receiving antenna data from one or more antennas corresponding to the single audio device of the multiple audio devices, and determining the DOA data for the single audio device based at least in part on the antenna data.

[0098] In some such examples, a single audio device may determine the DOA data itself. According to some such implementations, each audio device of the multiple audio devices may determine its own DOA data. However, in other implementations, other devices, which may be local or remote devices, may determine the DOA data for one or more audio devices in the environment. According to some implementations, a server may determine the DOA data for one or more audio devices in the environment.

[0099] 12, block 1210 includes generating, via the control system, audio device location data based at least in part on the DOA data. In this example, the audio device location data includes an estimate of an audio device location for each audio device referenced in block 1205.

[0100] The audio device location data may be (or may include) coordinates in a coordinate system, such as, for example, a Cartesian coordinate system, a spherical coordinate system, or a cylindrical coordinate system. The coordinate system may be referred to herein as an audio device coordinate system. In some such examples, the audio device coordinate system may be oriented with reference to one of the audio devices in the environment. In other examples, the audio device coordinate system may be oriented with reference to an axis defined by a line between two audio devices in the environment. However, in other examples, the audio device coordinate system may be oriented with reference to another part of the environment, such as a television, a wall of a room, etc.

[0101] In some examples, block 1210 may include the processes described above with reference to FIG. 4. According to some such examples, block 1210 may include determining an interior angle for each triangle of the plurality of triangles based on the DOA data. In some cases, each triangle of the plurality of triangles may have vertices that correspond with audio device locations of three audio devices. Some such methods may include determining a length for each side of each triangle of the triangles based at least in part on the interior angle.

[0102] Some such methods may include performing a forward alignment process to align each triangle of the plurality of triangles in a first sequence to generate a forward alignment matrix. Some such methods may include performing a reverse alignment process to align each triangle of the plurality of triangles in a second sequence that is a reverse of the first sequence to generate a reverse alignment matrix. Some such methods may include generating a final estimate of each audio device location based at least in part on values ​​of the forward alignment matrix and values ​​of the reverse alignment matrix. However, in some implementations of method 1200, block 1210 may include applying a method other than those described above with reference to FIG. 4.

[0103] In this example, block 1215 includes determining, via the control system, viewer location data indicative of a viewer location within the environment. The viewer location data may, for example, reference an audio device coordinate system. However, in other examples, the coordinate system may be oriented with reference to the viewer or with reference to a portion of the environment, such as a television, a wall of a room, etc.

[0104] In some examples, block 1215 may include prompting the viewer to make one or more utterances (e.g., via audio prompts from one or more loudspeakers in the environment) and estimating the viewer location according to the DOA data. The DOA data may correspond to microphone data acquired by a plurality of microphones in the environment. The microphone data may correspond to detection of one or more utterances by the microphones. At least some of the microphones may be co-located with the loudspeakers. According to some examples, block 1215 may include a triangulation process. For example, block 1215 may include triangulating the user's voice by finding an intersection between DOA vectors that pass through the audio device, e.g., as described below with reference to FIG. 13A. According to some implementations, block 1215 (or another operation of method 1200) may include co-locating the origin of the audio device coordinate system and the viewer coordinate system, after the viewer location is determined. Co-locating the origins of the audio device coordinate system and the viewer coordinate system may include transforming the position of the audio device from the audio device coordinate system to the viewer coordinate system.

[0105] According to this embodiment, block 1220 includes determining, via the control system, viewer angular orientation data indicative of a viewer angular orientation. The viewer angular orientation data may be generated with reference to a coordinate system used to represent the viewer location data, such as, for example, an audio device coordinate system. In some such examples, the viewer angular orientation data may be generated with reference to an origin and / or axes of the audio device coordinate system.

[0106] However, in some implementations, the viewer angular orientation data may be generated with reference to axes defined by the viewer location and another point in the environment, such as a television, an audio device, a wall, etc. In some such implementations, the viewer location may be used to define the origin of a viewer coordinate system. The viewer angular orientation data may be generated with reference to axes of the viewer coordinate system in some such instances.

[0107] Various methods for performing block 1220 are disclosed herein. According to some examples, the viewer angle orientation may correspond to a viewer viewing direction. In some such examples, the viewer's viewing direction may be inferred with reference to viewer location data, for example, by assuming that the viewer is looking at a particular object such as a television. In some such implementations, the viewer viewing direction may be determined according to the viewer location and the television location. Alternatively or additionally, the viewer viewing direction may be determined according to the viewer location and the television soundbar location.

[0108] However, in some examples, the viewer viewing direction may be determined according to viewer input. According to some such examples, the viewer input may include inertial sensor data received from a device held by the viewer. The viewer may use the device to point to a location in the environment, e.g., a location corresponding to a direction in which the viewer is facing. For example, the viewer may use the device to point to a sounding loudspeaker. Thus, in such examples, the inertial sensor data may include inertial sensor data corresponding to the sounding loudspeaker.

[0109] In some such cases, the viewer input may include an indication of an audio device selected by the viewer, which in some examples may include inertial sensor data corresponding to the selected audio device.

[0110] However, in other examples, the indication of the audio devices may be made according to one or more utterances of the viewer (e.g., "The TV is now in front of me," "Speaker 2 is now in front of me," etc.) Other examples of determining viewer angle orientation data according to one or more utterances of the viewer are described below.

[0111] According to the example shown in Figure 12, block 1225 includes determining, via the control system, audio device angular orientation data indicative of audio device angular orientation data for each audio device relative to the viewer location and the viewer angular orientation. According to some such examples, block 1225 may include rotating audio device coordinates about a point defined by the viewer location. In some implementations, block 1225 may include transforming the audio device location data from the audio device coordinate system to the viewer coordinate system. Some examples are described below.

[0112] Figure 13A shows examples of some of the blocks of Figure 12. According to some such examples, the audio device location data includes an estimate of the audio device location for each of audio devices 1-5, with reference to audio device coordinate system 1307. In this implementation, audio device coordinate system 1307 is a Cartesian coordinate system with the location of the microphone of audio device 2 as its origin, where the x-axis of audio device coordinate system 1307 corresponds to line 1303 between the location of the microphone of audio device 2 and the location of the microphone of audio device 1.

[0113] In this example, the audience location is determined by prompting an audience member 1305, shown seated on a couch 103 (e.g., via audio prompts from one or more loudspeakers in the environment 1300a), to make one or more utterances 1327, and estimating the audience location according to time of arrival (TOA) data. The TOA data corresponds to microphone data obtained from multiple microphones in the environment. In this example, the microphone data corresponds to detection of the one or more utterances 1327 by microphones of at least some (e.g., three, four, or all five) of audio devices 1-5.

[0114] Alternatively or additionally, the audience location according to the DOA data is provided by at least some (e.g., two, three, four or all five) microphones of audio devices 1-5. According to some such examples, the audience location may be determined according to the intersection of lines 1309a, 1309b, etc., that correspond to the DOA data.

[0115] According to this example, the viewer location corresponds to the origin of the viewer coordinate system 1320. In this example, the viewer angular orientation data is represented by the y' axis of the viewer coordinate system 1320, which corresponds to the line 1313a between the viewer's head 1310 (and / or the viewer's nose 1325) and the soundbar 1330 of the television 101. In the example shown in FIG. 13A, the line 1313a is parallel to the y' axis. Thus, the angle θ represents the angle between the y and y' axes. In this example, block 1225 of FIG. 12 may include a rotation of the audio device coordinates by the angle θ about the origin of the viewer coordinate system 1320. Thus, although the origin of audio device coordinate system 1307 is shown to correspond to audio device 2 in Figure 13A, some implementations include co-locating the origin of audio device coordinate system 1307 with the origin of viewer coordinate system 1320 prior to rotating the audio device coordinates by angle θ about the origin of viewer coordinate system 1320. This co-location may be performed by a coordinate transformation from audio device coordinate system 1307 to viewer coordinate system 1320.

[0116] In some examples, the location of the soundbar 1330 and / or the television 101 may be determined by having the soundbar emit a sound and estimating the location of the soundbar according to DOA and / or TOA data, which may correspond to detection of the sound by microphones of at least some (e.g., three, four, or all five) of the audio devices 1-5. Alternatively or additionally, the location of the soundbar 1330 and / or the television 101 may be determined by prompting a user to walk to the television and locating the user's speech by DOA and / or TOA data, which may correspond to detection of the sound by microphones of at least some (e.g., three, four, or all five) of the audio devices 1-5. Such methods may include triangulation. Such examples may be useful in situations where the soundbar 1330 and / or the television 101 do not have associated microphones.

[0117] In some other examples where the soundbar 1330 and / or television 101 have associated microphones, the location of the soundbar 1330 and / or television 101 may be determined according to a TOA method or a DOA method, such as the DOA methods disclosed herein. According to some such methods, the microphones may be co-located with the soundbar 1330.

[0118] According to some implementations, the soundbar 1330 and / or the television 101 may have an associated camera 1311. A control system may be configured to capture an image of the viewer's head 1310 (and / or the viewer's nose 1325). In some such examples, the control system may be configured to determine a line 1313a between the viewer's head 1310 (and / or the viewer's nose 1325) and the camera 1311. The viewer angular orientation data may correspond to the line 1313a. Alternatively or additionally, the control system may be configured to determine an angle θ between the line 1313a and the y-axis of the audio device coordinate system.

[0119] 13B shows an additional example of determining viewer angular orientation data. According to this example, the viewer location has already been determined in block 1215 of FIG. 12. Here, a control system controls loudspeakers of environment 1300b to render audio object 1335 at various locations within environment 1300b. In some such examples, the control system may cause the loudspeakers to render audio object 1335 such that it appears to rotate around viewer 1305, for example, by rendering audio object 1335 such that it rotates around the origin of viewer coordinate system 1320. In this example, curved arrow 1340 indicates a portion of the trajectory of audio object 1335 as it rotates around viewer 1305.

[0120] According to some such examples, the viewer 1305 may provide user input (e.g., saying "stop") indicating when the audio object 1335 is in the direction the viewer 1305 is facing. In some such examples, the control system may be configured to determine a line 1313b between the viewer location and the location of the audio object 1335. In this example, the line 1313b corresponds to the y' axis of a viewer coordinate system indicating the direction the viewer 1305 is facing. In alternative implementations, the viewer 1305 may provide user input indicating when the audio object 1335 is in front of the environment, at a TV location of the environment, at an audio device location, etc.

[0121] 13C illustrates an additional example of determining viewer angular orientation data. According to this example, the viewer location has already been determined in block 1215 of FIG. 12. Here, the viewer 1305 uses the handheld device 1345 to provide input regarding the viewer's 1305 viewing direction by pointing the handheld device 1345 toward the television 101 or soundbar 1330. The dashed outline of the handheld device 1345 and the viewer's arm indicates that the viewer 1305 was pointing the handheld device 1345 toward audio device 2, in this example, at a time prior to when the viewer 1305 was pointing the handheld device 1345 toward the television 101 or soundbar 1330. In other examples, the viewer 1305 may point the handheld device 1345 toward another audio device, such as audio device 1. According to this example, the handheld device 1345 is configured to determine an angle α between audio device 2 and the television 101 or soundbar 1330, which approximates the angle between audio device 2 and the viewing direction of the viewer 1305.

[0122] The handheld device 1345 may, in some examples, be a mobile phone including an inertial sensor system and a wireless interface configured to communicate with a control system controlling the audio devices of the environment 1300c. In some examples, the handheld device 1345 may execute an application or “app” configured to control the handheld device 1345 to perform a required functionality, such as by providing user prompts (e.g., via a graphical user interface), by receiving input indicating that the handheld device 1345 is pointing in a desired direction, by storing corresponding inertial sensor data and / or by transmitting corresponding inertial sensor data to a control system controlling the audio devices of the environment 1300c.

[0123] According to this example, a control system (which may be a control system of handheld device 1345 or a control system controlling an audio device of environment 1300c) is configured to determine the orientation of lines 1313c and 1350 according to inertial sensor data, e.g., gyroscope data. In this example, line 1313c is parallel to axis y' and may be used to determine a viewer angular orientation. According to some examples, the control system may determine an appropriate rotation for audio device coordinates around the origin of viewer coordinate system 1320 according to angle α between audio device 2 and the viewing direction of viewer 1305.

[0124] FIG. 13D illustrates an example of determining an appropriate rotation for audio device coordinates according to the method described with reference to FIG. 13C. In this example, the origin of audio device coordinate system 1307 is co-located with the origin of viewer coordinate system 1320. Co-locating the origins of audio device coordinate system 1307 and viewer coordinate system 1320 is enabled after the process of viewer location is determined 1215. Co-locating the origins of audio device coordinate system 1307 and viewer coordinate system 1320 may include transforming the audio device location from audio device coordinate system 1307 to viewer coordinate system 1320. Angle α was determined as described above with reference to FIG. 13C. Thus, angle α corresponds to a desired orientation of audio device 2 in viewer coordinate system 1320. In this example, angle β corresponds to an orientation of audio device 2 in audio device coordinate system 1307. The angle θ, which in this example is β-α, indicates the rotation required to align the y-axis of audio device coordinate system 1307 with the y′-axis of viewer coordinate system 1320 .

[0125] In some implementations, the method of FIG. 12 may include controlling at least one of the audio devices in the environment based at least in part on the corresponding audio device location, the corresponding audio device angular orientation, the viewer location data, and the viewer angular orientation data.

[0126] For example, some implementations may include providing the audio device location data, the audio device angular orientation data, the viewer location data, and the viewer angular orientation data to an audio rendering system. In some examples, the audio rendering system may be implemented by a control system, such as the control system 1110 of FIG. 11. Some implementations may include controlling an audio data rendering process based at least in part on the audio device location data, the audio device angular orientation data, the viewer location data, and the viewer angular orientation data. Some such implementations may include providing loudspeaker acoustic capability data to the rendering system. The loudspeaker acoustic capability data may correspond to one or more loudspeakers of the environment. The loudspeaker acoustic capability data may indicate orientations of one or more drivers, multiple drivers, or driver frequency responses of one or more drivers. In some examples, the loudspeaker acoustic capability data is retrieved from memory and then provided to the rendering system.

[0127] Existing flexible rendering techniques include Center of Mass Amplitude Panning (CMAP) and Flexible Virtualization (FV). From a high level, both of these techniques render a set of one or more audio signals, each having an associated desired perceived spatial location, for playback on a set of two or more loudspeakers, where the relative activation of the set of loudspeakers is a function of a model of the perceived spatial location of the audio signal played on the loudspeakers and the proximity of the desired perceived spatial location of the audio signal to the location of the loudspeakers. The model ensures that the audio signal is heard by the viewer near its intended spatial location, and the proximity term controls which loudspeakers are used to achieve this spatial impression. In particular, the proximity term favors activation of loudspeakers close to the desired perceived spatial location of the audio signal. For both CMAP and FV, this functional relationship is conveniently derived from a cost function written as a sum of two terms, one for the spatial aspect and one for the proximity, as follows:

number

[0128] Here, set (Outside 14) TIFF0007665630000019.tif7170 shows the location of a set of M loudspeakers, (Outside 15) TIFF0007665630000020.tif7170 denotes the desired perceived spatial location of an audio signal, and g denotes an M-dimensional vector of speaker activations. For CMAP, each activation in the vector represents a gain per speaker, while for FV, each activation represents a filter (in this second case, g can be equivalently thought of as a vector of complex values ​​at specific frequencies, and different g's are calculated across multiple frequencies to form filters). The optimal vector of activations is found by minimizing a cost function over the activations, as follows:

number

[0129] With a particular definition of the cost function, it is difficult to control the absolute level of optimal activation resulting from the above minimization, but g opt The relative levels between the components of are relevant. To address this issue, we use g opt Subsequent normalization of may be performed. For example, normalization of vectors with unit length is desirable, which is consistent with commonly used constant power panning rules such as:

number

[0130] The exact behavior of the flexible rendering algorithm depends on two terms in the cost function: C spatial and C proximity It is determined by the specific configuration of CMAP. spatial defines the perceived spatial location of an audio signal played from a set of loudspeakers in terms of the associated activation gains g i From the model, which places the loudspeaker positions in the center of mass weighted by (elements of vector g), it is derived as follows:

number

[0131] Equation 3 is then manipulated into a spatial cost that represents the square of the error between the desired audio position and the position produced by the activated loudspeakers, as follows:

number

[0132] In FV, the spatial term of the cost function is defined differently. Here, the goal is to find the audio object location at the listener's left and right ears. (Outside 16) The goal of the study is to generate a binaural response b corresponding to TIFF0007665630000025.tif7170. Conceptually, b is a 2×1 vector of filters (one filter for each ear), but is more conveniently treated as a 2×1 vector of complex values ​​at specific frequencies. Continuing this representation at specific frequencies, the desired binaural response may be extracted from a set of HRTF indices by object position, as follows:

number

[0133] At the same time, the 2×1 binaural response e produced at the listener's ears by the loudspeakers is modeled as a 2×M acoustic transmission matrix H multiplied by an M×1 vector of complex speaker activation values ​​g, as follows:

number

[0134] The acoustic transfer matrix H is the relationship between the loudspeaker positions and the listener position. (Outside 17) The model is based on the set of TIFF0007665630000028.tif7170. Finally, the spatial component of the cost function is defined as the squared error between the desired binaural response (Equation 5) and the response produced by the loudspeakers (Equation 6) as follows:

number

[0135] Conveniently, the spatial terms of the CMAP and FV cost functions defined in Equation 4 and Equation 7 can both be reconstructed to matrix quadratic order as a function of the speaker activation g as follows:

number

[0136] For this purpose, the second term of the cost function is, namely, C proximity is defined as the distance-weighted sum of the absolute squared values ​​of the audience activations, which can be compactly expressed in matrix form as

number

number

[0137] The distance penalty function can take many forms, but the following is a useful parameterization:

number

[0138] Combining the two terms of the cost function defined in Equation 8 and Equation 9a results in an overall cost function as follows:

number

number

[0139] In general, an optimal solution to Equation 11 may result in speaker activations that are negative in value. For the CMAP structure of the flexible renderer, such negative activations may be undesirable, and thus Equation 11 may be minimized, provided that all activations remain positive.

[0140] 14 and 15 are diagrams illustrating an example set of speaker activations and object rendering positions, given speaker positions of 4, 64, 165, -87, and -4 degrees. FIG. 14 shows speaker activations including the optimal solution of Equation 11 for these particular speaker positions. FIG. 15 plots the individual speaker positions as orange, purple, green, gold, and blue dots, respectively. FIG. 15 shows ideal object positions (i.e., positions where audio objects should be rendered) for a number of possible object angles as green dots, and the corresponding actual rendering positions for those objects as red dots, connected to the ideal object positions by black dotted lines.

[0141] While specific embodiments and applications of the present disclosure have been described herein, it will be apparent to those skilled in the art that many variations of the embodiments and applications described herein are possible without departing from the scope of this disclosure.

[0142] Various aspects of the present disclosure may be understood from the following exemplary embodiments (EEE).

[0143] 1. A method for selecting an audio device location, comprising: obtaining direction of arrival (DOA) data for each of a plurality of audio devices; determining an interior angle of each triangle of the plurality of triangles based on the DOA data, each triangle of the plurality of triangles having vertices corresponding to audio device locations of three of the audio devices; determining a side length for each side of each of the triangles based at least in part on the interior angles; performing a forward alignment process to align each triangle of the plurality of triangles in the first sequence to generate a forward alignment matrix; performing a reverse alignment process to align each triangle of the plurality of triangles in a second sequence that is a reverse of the first sequence to generate a reverse alignment matrix; generating a final estimate of each audio device location based at least in part on the forward alignment matrix values ​​and the reverse alignment matrix values; method.

[0144] 2. Producing a final estimate of each audio device location is translating and scaling the forward alignment matrix to generate a translated and scaled forward alignment matrix; translating and scaling the reverse alignment matrix to generate a translated and scaled reverse alignment matrix. EEE1 method.

[0145] 3. Generating a final estimate of each audio device location further includes generating a rotation matrix based on the translated and scaled forward alignment matrix and the translated and scaled reverse alignment matrix, the rotation matrix including a plurality of estimated audio device locations for each audio device; EEE2 method.

[0146] 4. Generating the rotation matrix includes performing a singular value decomposition on the translated and scaled forward alignment matrix and the translated and scaled reverse alignment matrix; EEE3 method.

[0147] 5. Producing a final estimate of each audio device location further includes averaging the audio device locations estimated for each audio device to generate a final estimate of each audio device location. EEE3 or EEE4 method.

[0148] 6. Determining the length of a side is Determining a first length of a first side of the triangle; determining a second side length and a third side length of the triangle based on the interior angles of the triangle. One of the methods EEE1 to 5.

[0149] 7. The method of EEE6, wherein determining the first length includes setting the first length to a predetermined value.

[0150] 8. The method of EEE6, wherein determining the first length is based on at least one of time of arrival data or received signal strength data.

[0151] 9. The method of any one of EEE1-8, wherein obtaining the DOA data includes determining DOA data for at least one audio device of the plurality of audio devices.

[0152] 10. The method of EEE9, wherein the step of determining the DOA data includes receiving microphone data from each microphone of a plurality of audio device microphones corresponding to a single audio device among the plurality of audio devices, and determining the DOA data for the single audio device based at least in part on the microphone data.

[0153] 11. The method of EEE9, wherein determining the DOA data includes receiving antenna data from one or more antennas corresponding to a single audio device of the multiple audio devices, and determining the DOA data for the single audio device based at least in part on the antenna data.

[0154] 12. The method of any one of EEE1-11, further comprising controlling at least one of the audio devices based at least in part on the final estimate of the at least one audio device location.

[0155] 13. The method of EEE12, wherein controlling at least one of the audio devices includes controlling a loudspeaker of the at least one of the audio devices.

[0156] 14. An apparatus configured to perform any one of methods EEE1-13.

[0157] One or more non-transitory media containing software recorded thereon, the software including instructions for controlling one or more devices to perform any of the methods of EEE1-13.

[0158] 1. A method for configuring an audio device, comprising: obtaining, via a control system, audio device direction of arrival (DOA) data for each audio device of a plurality of audio devices in the environment; generating, via the control system, audio device location data based at least in part on the DOA data, the audio device location data including an estimate of an audio device location for each audio device; determining, via the control system, viewer location data indicative of viewer locations within the environment; determining, via the control system, viewer angular orientation data indicative of a viewer angular orientation; and determining, via the control system, audio device angular orientation data indicative of an audio device angular orientation for each audio device relative to a viewer location and a viewer angular orientation. method.

[0159] 17. The method of EEE16, further comprising controlling at least one of the audio devices based at least in part on the corresponding audio device location, the corresponding audio device angular orientation, the viewer location data, and the viewer angular orientation data.

[0160] 18. The method of EEE16, further comprising providing the audio device location data, the audio device angular orientation data, the viewer location data, and the viewer angular orientation data to an audio rendering system.

[0161] 19. The method of EEE16, further comprising controlling an audio data rendering process based at least in part on the audio device location data, the audio device angular orientation data, the viewer location data, and the viewer angular orientation data.

[0162] 20. The method of any one of EEE16-19, wherein obtaining the DOA data includes controlling each loudspeaker of a plurality of loudspeakers in the environment to play the test signal.

[0163] 21. The method of any one of EEE16-20, wherein at least one of the viewer location data or the viewer angular orientation data is based on DOA data corresponding to one or more utterances of the viewer.

[0164] 22. The viewer angle orientation corresponds to the viewer viewing direction, and is one of the methods EEE16 to 21.

[0165] 23. The method of EEE22, in which the viewer viewing direction is determined according to the viewer location and the television location.

[0166] 24. The method of EEE22, wherein the viewer viewing direction is determined according to the viewer location and the TV soundbar location.

[0167] 25. The method of EEE22, in which viewer viewing direction is determined according to viewer input.

[0168] 26. The method of EEE25, wherein the viewer input includes inertial sensor data received from a device held by the viewer.

[0169] 27. The method of EEE25, wherein the inertial sensor data includes inertial sensor data corresponding to a loudspeaker emitting the sound.

[0170] 28. The method of EEE25, wherein the viewer input includes an indication of an audio device selected by the viewer.

[0171] 29. The method of any one of EEE16-28, further comprising providing loudspeaker acoustic capability data to the rendering system, the loudspeaker acoustic capability data indicating at least one of an orientation of one or more drivers, a plurality of drivers, or a driver frequency response of the one or more drivers.

[0172] 30. Generating audio device location data determining an interior angle for each triangle of the plurality of triangles based on the audio device DOA data, each triangle of the plurality of triangles having vertices corresponding to audio device locations of three of the audio devices; determining a side length for each side of each of the triangles based at least in part on the interior angles; performing a forward alignment process to align each triangle of the plurality of triangles in the first sequence to generate a forward alignment matrix; performing a reverse alignment process to align each triangle of the plurality of triangles in a second sequence that is a reverse of the first sequence to generate a reverse alignment matrix; generating a final estimate of each audio device location based at least in part on the forward alignment matrix values ​​and the reverse alignment matrix values; One of the methods EEE16~29.

[0173] 20. An apparatus configured to perform any one of the methods of EEE16 to 30.

[0174] One or more non-transitory media having software recorded thereon, the software including instructions for controlling one or more devices to perform any one of the methods set forth in EEE16-30.

Claims

1. 1. A method for determining a location of a plurality of at least four audio devices in an environment, each audio device configured to detect a signal generated by a different audio device of the plurality of audio devices, the method comprising: obtaining direction of arrival data based on a detected direction of the signal generated by another one of the plurality of audio devices in the environment; determining an interior angle of each triangle of a plurality of triangles based on the direction of arrival data, each triangle of the plurality of triangles having vertices corresponding to locations of three audio devices of the plurality of audio devices; determining a side length for each side of each triangle of the triangles based on the interior angles and based on the signals generated by the audio device separated by the side lengths to be determined; or determining side lengths based on the interior angles, wherein a side length of one of the triangles is set to a predetermined value; performing a forward alignment process to align each triangle of the plurality of triangles in a first sequence to generate a forward alignment matrix, the forward alignment process being performed by matching a side length of each triangle with a side length of an adjacent triangle and using the interior angles determined for the adjacent triangles; performing a reverse alignment process to align each triangle of the plurality of triangles to generate a reverse alignment matrix, the reverse alignment process being performed as the forward alignment process but in a second sequence that is the reverse of the first sequence; generating a final estimate of each audio device location based at least in part on the forward alignment matrix values ​​and the reverse alignment matrix values. method.

2. generating the final estimate of each audio device location, translating and scaling the forward alignment matrix to generate a translated and scaled forward alignment matrix; translating and scaling the reverse alignment matrix to generate a translated and scaled reverse alignment matrix; Translating and scaling the forward alignment matrix and the reverse alignment matrix includes moving the center of mass of each matrix to the origin and forcing the Frobenius norm of each matrix to zero. The method of claim 1.

3. 3. The method of claim 2, wherein generating the final estimate of each audio device location further comprises generating a further matrix based on the translated and scaled forward alignment matrix and the translated and scaled reverse alignment matrix, the further matrix comprising a plurality of estimated audio device locations for each audio device.

4. The method of claim 3 , wherein generating the further matrices comprises performing a singular value decomposition on the translated and scaled forward alignment matrix and the translated and scaled reverse alignment matrix.

5. 5. The method of claim 1, wherein generating the final estimate of each audio device location further comprises averaging multiple estimates of the location of the audio device obtained from overlapping vertices of multiple triangles.

6. Determining the length of the side comprises: Determining a first length of a first side of the triangle; determining a second side length and a third side length of the triangle based on the interior angles of the triangle; determining the first length includes setting the first length to a predetermined value, or determining the first length is based on at least one of time of arrival data or received signal strength data. The method according to any one of claims 1 to 5.

7. each audio device includes a plurality of audio device microphones, and determining the direction of arrival data includes receiving microphone data from each microphone of a plurality of audio device microphones corresponding to a single audio device of the plurality of audio devices, and determining the direction of arrival data for the single audio device based at least in part on the microphone data. The method according to any one of claims 1 to 6.

8. 7. The method of claim 1, wherein each audio device includes one or more antennas, and determining the direction of arrival data includes receiving antenna data from one or more antennas corresponding to a single audio device of the plurality of audio devices, and determining the direction of arrival data for the single audio device based at least in part on the antenna data.

9. 9. The method of claim 1, further comprising controlling at least one of the audio devices based at least in part on the final estimate of at least one audio device location.

10. 10. The method of claim 9, wherein each audio device of the plurality of audio devices includes a loudspeaker, and controlling at least one of the audio devices includes controlling a loudspeaker based on the at least one audio device of the audio devices.

11. Apparatus arranged to carry out the method according to any one of claims 1 to 10.

12. A computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out a method according to any one of claims 1 to 10.

13. A computer readable medium comprising a computer program according to claim 12.

Images