Automatic positioning of audio devices

Abstract

A method for estimating a location of an audio device in an environment may involve obtaining direction of arrival (DOA) data for each of a plurality of audio devices in the environment and determining an interior angle of each of a plurality of triangles based on the DOA data. Each triangle may have a vertex corresponding to an audio device location. The method may involve determining a length of each side of each of the triangles, performing a forward alignment process to align each of the plurality of triangles to produce a forward alignment matrix, and performing a reverse alignment process to align each of the plurality of triangles in a reverse order to produce a reverse alignment matrix. A final estimate of each audio device location may be based at least in part on a value of the forward alignment matrix and a value of the reverse alignment matrix.

Description

Background Art

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 949,998 filed on December 18, 2019, European Patent Application No. 19217580.0 filed on December 18, 2019, and U.S. Provisional Patent Application No. 62 / 992,068 filed on March 19, 2020, all of which are incorporated herein by reference. Technical Field

[0004] The present disclosure relates to systems and methods for automatically locating audio devices. Background Art

[0005] Audio devices, including but not limited to smart audio devices, have been widely deployed and are becoming a common feature of many homes. Although existing systems and methods for locating audio devices provide benefits, improved systems and methods would still be desirable.

[0006] Symbols and terminology

[0007] The expression "smart audio device" is used herein to represent a smart device, which is a single-purpose audio device or a virtual assistant (e.g., a connected virtual assistant). A single-purpose audio device is a device (e.g., a smart speaker, a television (TV) or a mobile phone) that includes or is coupled to at least one microphone (and in some examples may also include or be coupled to at least one speaker) and is largely or primarily designed to implement a single purpose. Although TVs can usually play (and are considered to be able to play) audio from program materials, in most cases, modern TVs run some kind of operating system, and applications (including applications for watching TV) run locally on the operating system. Similarly, the audio input and output in a mobile phone can do a lot of things, but these are all served by applications running on the phone. In this sense, a single-purpose audio device with (multiple) speakers and (multiple) microphones is typically configured to run local applications and / or services to directly use (multiple) speakers and (multiple) microphones. Some single-purpose audio devices can be configured to be combined together to achieve audio playback on a region or user-configured region.

[0008] In this document, a "virtual assistant" (e.g., a connected virtual assistant) is a device (e.g., a smart speaker, smart display, or voice assistant integrated device) that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker), and that can provide the ability to use multiple devices (other than the virtual assistant) for applications that are cloud-enabled in some sense or that are not implemented in or on the virtual assistant itself. Virtual assistants can sometimes work together, for example, in very discrete and conditionally defined ways. For example, two or more virtual assistants can work together in the sense that one of them (i.e., the virtual assistant that is most confident that it has heard the wake word) responds to the word. The connected devices can form a series that can be managed by a master application that can be (or include or implement) a virtual assistant.

[0009] In this document, "wake-up word" is used in a broad sense to refer to any sound (e.g., a word spoken by a human or some other sound) that the smart audio device is configured to wake up in response to detecting ("hearing") the sound (using at least one microphone included in or coupled to the smart audio device, or at least one other microphone). In this case, "waking up" means that the device enters a state of waiting (i.e., listening) for a voice command.

[0010] In this document, the expression "wake-up word detector" refers to a device (or software including instructions for configuring the device to continuously search for alignment between real-time sound (e.g., speech) features and a training model) that is configured to continuously search for alignment between real-time sound features and a training model. Typically, a wake-up word event is triggered whenever the wake-up word detector determines that the probability of detecting a wake-up word exceeds a predefined threshold. For example, the threshold may be a predetermined threshold that is adjusted to give a reasonable compromise between a false acceptance rate and a false rejection rate. After a wake-up word event, the device may enter a state (which may be referred to as an "awake" state or an "attention" state) in which the device listens for commands and passes received commands to a larger, more computationally intensive recognizer.

[0011] Throughout this disclosure, including in the claims, "speaker" and "loudspeaker" are used synonymously to refer to any sound-emitting transducer (or set of transducers) driven by a single speaker feed. A typical set of headphones includes two speakers. A speaker may be implemented to include multiple transducers (e.g., a woofer and a tweeter) that are all driven by a single common speaker feed. In some cases, the speaker feed may undergo different processing in different circuit branches coupled to different transducers.

[0012] Throughout this disclosure, including in the claims, expressions referring to “performing an operation on” a signal or data (e.g., filtering, scaling, transforming, or applying a gain to the signal or data) are used in a broad sense to mean performing the operation directly on the signal or data or on a processed version of the signal or data (e.g., a version of the signal that has undergone preliminary filtering or preprocessing before the operation is performed on it).

[0013] Throughout this disclosure, including in the claims, the expression "system" is used in a broad sense to refer to a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to a plurality of inputs, where the subsystem generates M of the inputs and the other XM inputs are received from external sources) may also be referred to as a decoder system.

[0014] Throughout this disclosure, including in the claims, the term "processor" is used in a broad sense to refer to a system or device that is programmable or otherwise configurable (e.g., with software or firmware) to perform operations on data (e.g., audio, video or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors that are programmed and / or otherwise configured to perform pipeline processing of audio or other sound data, programmable general purpose processors or computers, and programmable microprocessor chips or chipsets. Summary of the invention

[0015] At least some aspects of the present disclosure may be implemented via methods. Some such methods may involve audio device positioning, i.e., methods for determining the positions of multiple (e.g., at least four or more) audio devices in an environment. For example, some methods may involve obtaining direction of arrival (DOA) data for each of a plurality of audio devices and determining an interior angle of each of a plurality of triangles based on the DOA data. In some instances, each of the plurality of triangles may have vertices corresponding to audio device positions of three audio devices. Some such methods may involve determining the length of each side of each triangle based at least in part on the interior angle.

[0016] Some such methods may involve performing a forward alignment process that aligns each of the plurality of triangles in a first order to produce a forward alignment matrix. Some such methods may involve performing a reverse alignment process that aligns each of the plurality of triangles in a second order that is reverse of the first order to produce a reverse alignment matrix. Some such methods may involve generating a final estimate of the position of each audio device based at least in part on values ​​of the forward alignment matrix and values ​​of the reverse alignment matrix.

[0017] According to some examples, generating a final estimate of each audio device position may involve translating and scaling a forward alignment matrix to generate a translated and scaled forward alignment matrix, and translating and scaling a reverse alignment matrix to generate a translated and scaled reverse alignment matrix. Some such methods may involve generating a rotation matrix based on the translated and scaled forward alignment matrix and the translated and scaled reverse alignment matrix. The rotation matrix may include multiple estimated audio device positions for each audio device. In some embodiments, generating a rotation matrix may involve performing singular value decomposition on the translated and scaled forward alignment matrix and the translated and scaled reverse alignment matrix. According to some examples, generating a final estimate of each audio device position may also involve averaging the estimated audio device positions of each audio device to generate a final estimate of each audio device position.

[0018] In some embodiments, determining the side length may involve determining a first length of a first side of the triangle and determining the lengths of a second side and a third side of the triangle based on an interior angle of the triangle. In some examples, determining the first length may involve setting the first length to a predetermined value. In some examples, determining the first length may be based on arrival time data and / or received signal strength data.

[0019] According to some examples, obtaining DOA data may involve determining DOA data for at least one of the plurality of audio devices. In some instances, determining DOA data may involve receiving microphone data from each of a plurality of audio device microphones corresponding to a single audio device in the plurality of audio devices and determining DOA data for the single audio device based at least in part on the microphone data. According to some examples, determining DOA data may involve receiving antenna data from one or more antennas corresponding to the single audio device in the plurality of audio devices and determining DOA data for the single audio device based at least in part on the antenna data.

[0020] In some implementations, the method may also involve controlling at least one of the audio devices based at least in part on the final estimate of the position of the at least one audio device. In some such examples, controlling at least one of the audio devices may involve controlling a loudspeaker of the at least one of the audio devices.

[0021] Some or all of the operations, functions, and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Therefore, some innovative aspects of the subject matter described in this disclosure may be implemented in non-transitory media having software stored thereon.

[0022] For example, the software may include instructions for controlling one or more devices to perform methods involving audio device positioning. Some methods may involve obtaining DOA data for each of a plurality of audio devices and determining an interior angle of each of a plurality of triangles based on the DOA data. In some instances, each of the plurality of triangles may have vertices corresponding to audio device locations of three audio devices. Some such methods may involve determining the length of each side of each triangle based at least in part on the interior angles.

[0023] Some such methods may involve performing a forward alignment process that aligns each of the plurality of triangles in a first order to produce a forward alignment matrix. Some such methods may involve performing a reverse alignment process that aligns each of the plurality of triangles in a second order that is reverse of the first order to produce a reverse alignment matrix. Some such methods may involve generating a final estimate of the position of each audio device based at least in part on values ​​of the forward alignment matrix and values ​​of the reverse alignment matrix.

[0024] According to some examples, generating a final estimate of each audio device position may involve translating and scaling a forward alignment matrix to generate a translated and scaled forward alignment matrix, and translating and scaling a reverse alignment matrix to generate a translated and scaled reverse alignment matrix. Some such methods may involve generating a rotation matrix based on the translated and scaled forward alignment matrix and the translated and scaled reverse alignment matrix. The rotation matrix may include multiple estimated audio device positions for each audio device. In some embodiments, generating a rotation matrix may involve performing singular value decomposition on the translated and scaled forward alignment matrix and the translated and scaled reverse alignment matrix. According to some examples, generating a final estimate of each audio device position may also involve averaging the estimated audio device positions of each audio device to generate a final estimate of each audio device position.

[0025] In some embodiments, determining the side length may involve determining a first length of a first side of the triangle and determining the lengths of a second side and a third side of the triangle based on an interior angle of the triangle. In some examples, determining the first length may involve setting the first length to a predetermined value. In some examples, determining the first length may be based on arrival time data and / or received signal strength data.

[0026] According to some examples, obtaining DOA data may involve determining DOA data for at least one of the plurality of audio devices. In some instances, determining DOA data may involve receiving microphone data from each of a plurality of audio device microphones corresponding to a single audio device in the plurality of audio devices and determining DOA data for the single audio device based at least in part on the microphone data. According to some examples, determining DOA data may involve receiving antenna data from one or more antennas corresponding to the single audio device in the plurality of audio devices and determining DOA data for the single audio device based at least in part on the antenna data.

[0027] In some implementations, the method may also involve controlling at least one of the audio devices based at least in part on the final estimate of the position of the at least one audio device. In some such examples, controlling at least one of the audio devices may involve controlling a loudspeaker of the at least one of the audio devices.

[0028] At least some aspects of the present disclosure may be implemented via an apparatus. For example, one or more devices may be capable of at least partially performing the methods disclosed herein. In some embodiments, the apparatus may include an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or a combination thereof. In some examples, the apparatus may be one of the audio devices cited above. However, in some embodiments, the apparatus may be another type of device, such as a mobile device, a laptop computer, a server, etc.

[0029] In some aspects of the present disclosure, any of the described methods may be implemented in a computer program product comprising instructions, which, when executed by a computer, cause the computer to perform any of the methods or steps of a method described in the present disclosure.

[0030] In some aspects of the present disclosure, a computer-readable medium including a computer program product is described.

[0031] The details of one or more embodiments of the subject matter described in this specification are set forth in the following drawings and description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Note that the relative sizes of the following drawings may not be drawn to scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 An example of the geometric relationship between three audio devices in an environment is shown.

[0033] Figure 2 Shows Figure 1 Another example of the geometric relationship between three audio devices in the environment shown in .

[0034] Figure 3A Shows Figure 1 and Figure 2 The two triangles depicted in the figure have no corresponding audio equipment and other features of the environment.

[0035] Figure 3B An example of estimating the interior angle of a triangle formed by three audio devices is shown.

[0036] Figure 4 is an overview of what can be done by Fig.11 A flowchart of an example of a method performed by an apparatus such as an apparatus is shown in FIG.

[0037] Figure 5 An example is shown where each audio device in the environment is a vertex of multiple triangles.

[0038] Figure 6 An example of a portion of the forward alignment process is provided.

[0039] Figure 7 An example of multiple audio device position estimates that have occurred during the forward alignment process is shown.

[0040] Figure 8 An example of a portion of the reverse alignment process is provided.

[0041] Fig. 9 An example of multiple audio device position estimates that have occurred during the reverse alignment process is shown.

[0042] Fig.10 A comparison of the estimated audio device position and the actual audio device position is shown.

[0043] Fig.11 is a block diagram illustrating an example of components of an apparatus capable of implementing various aspects of the present disclosure.

[0044] Fig.12 is an overview of what can be done by Fig.11 A flowchart of an example of a method performed by an apparatus such as an apparatus is shown in FIG.

[0045] Fig.13A Shows Fig.12 Examples of some boxes.

[0046] Fig. 13B Additional examples of determining listener angular orientation data are shown.

[0047] Fig. 13C Additional examples of determining listener angular orientation data are shown.

[0048] Fig.13D According to the reference Fig. 13C An example of a method for determining the appropriate rotation of audio device coordinates is described.

[0049] Fig.14 Speaker activations are shown that include the optimal solution to Eq. 11 for these specific speaker locations.

[0050] Fig.15 The positions of the various speakers are depicted, Fig.14 Speaker activation is shown for such a position.

[0051] Like reference numbers and designations throughout the various drawings indicate like elements. DETAILED DESCRIPTION

[0052] In addition to existing audio devices including televisions and sound bars, and new microphone- and speaker-enabled connected devices such as light bulbs and microwave ovens, the advent of smart speakers incorporating multiple drivers and microphone arrays creates the problem that dozens of microphones and speakers need to be positioned relative to each other to achieve orchestration. A typical layout of audio devices (such as a discrete Dolby 5.1 speaker layout) cannot be assumed. In some cases, audio devices in an environment may be randomly positioned, or at least distributed within the environment in an irregular and / or asymmetrical manner.

[0053] Furthermore, it cannot be assumed that audio devices are non-uniform or synchronized. As used herein, an audio device may be referred to as "synchronized" or "synchronized" if the audio devices detect or emit sound according to the same sampling clock or synchronized sampling clock. For example, a first synchronized microphone of a first audio device within an environment may digitally sample audio data according to a first sampling clock, and a second microphone of a second synchronized audio device within the environment may digitally sample audio data according to the first sampling clock. Alternatively or in addition, a first synchronized speaker of a first audio device within an environment may emit sound according to a speaker setup clock, and a second synchronized speaker of a second audio device within the environment may emit sound according to a speaker setup clock.

[0054] Some previously disclosed methods for automatic speaker localization require synchronized microphones and / or speakers. For example, some previously existing tools for device localization rely on sampling synchronization between all microphones in the system, which requires known test stimuli and the transfer of full bandwidth audio data between sensors.

[0055] The present assignee has presented several speaker positioning techniques for cinemas and homes that are excellent solutions for the use cases for which they are designed. Some such methods are based on the time of flight derived from the impulse response between the sound source and the microphone(s) approximately co-located with each loudspeaker. Although system delays in the recording and playback chain can be estimated, sampling synchronization between clocks is required and known test stimuli are required from which to estimate the impulse responses.

[0056] Recent examples of source localization in this context have relaxed the constraints by requiring intra-device microphone synchronization but not inter-device synchronization. Additionally, some such methods do not require audio to be transmitted between sensors via low-bandwidth messaging (such as via detecting the time of arrival (TOA) of direct (non-reflected) sound or via detecting the main direction of arrival (DOA) of direct sound. Each method has some potential advantages and potential disadvantages. For example, the TOA method can determine the device geometry based on unknown translation, rotation, and reflection about one of the three axes. If each device has only one microphone, the rotation of a single device is also unknown. The DOA method can determine the device geometry based on unknown translation, rotation, and scaling. Although some such methods can give satisfactory results under ideal conditions, the robustness of these methods to measurement errors has not yet been demonstrated.

[0057] Some embodiments of the present disclosure automatically locate the positions of multiple audio devices in an environment (e.g., a room) by applying geometry-based optimization using asynchronous DOA estimates from uncontrolled sound sources observed by a microphone array in each device. Various disclosed audio device localization methods have been shown to be robust to large DOA estimation errors.

[0058] Some such embodiments involve iteratively aligning triangles obtained from a DOA dataset. In some such examples, each audio device may include a microphone array that estimates the DOA from an uncontrolled source. In some embodiments, the microphone array may be placed in parallel with at least one loudspeaker. However, at least some disclosed methods generalize to situations where not all microphones are placed in parallel with the loudspeaker.

[0059] According to some disclosed methods, DOA data from each audio device to each other audio device in the environment can be aggregated. The audio device position can be estimated by iteratively aligning triangles parameterized by DOA pairs. Some such methods can produce results corrected for unknown scaling and rotation. In many applications, absolute scaling is unnecessary, and the rotation can be solved by applying additional constraints to the solution. For example, some multi-speaker environments may include television (TV) speakers and sofas positioned for TV viewing. After positioning the speakers in the environment, some methods may involve finding a vector pointing to the TV and locating the speech of a user sitting on the sofa by triangulation. Some such methods may then involve making the TV emit a sound from its speakers and / or prompting the user to walk towards the TV and locating the user's speech by triangulation. Some embodiments may involve rendering an audio object that pans around the environment. The user may provide a user input (e.g., saying "stop") indicating when the audio object is located at one or more predetermined positions in the environment (such as in front of the environment, at the TV position of the environment, etc.). According to some such examples, after positioning the speakers in the environment and determining their orientation, the user can be positioned by finding the intersection of the directions of arrival of the sounds emitted by multiple speakers. Some implementations involve determining an estimated distance between at least two audio devices and scaling distances between other audio devices in an environment based on the estimated distance.

[0060] Figure 1 An example of a geometric relationship between three audio devices in an environment is shown. In this example, the environment 100 is a room including a TV 101, a sofa 105, and five audio devices 105. According to this example, the audio devices 105 are in positions 1 to 5 of the environment 100. In this embodiment, each audio device 105 includes a microphone system 120 having at least three microphones and a speaker system 125 including at least one speaker. In some embodiments, each microphone system 120 includes a microphone array. According to some embodiments, each audio device 105 may include an antenna system including at least three antennas.

[0061] As with other examples disclosed herein, Figure 1 The types, quantities, and arrangements of the elements shown in the drawings are for example only. Other implementations may have different types, quantities, and arrangements of elements, for example, more or fewer audio devices 105, audio devices 105 in different locations, etc.

[0062] In this example, the vertices of triangle 110a are at positions 1, 2, and 3. Here, triangle 110a has sides 12, 23a, and 13a. According to this example, the angle between side 12 and side 23 is θ2, the angle between side 12 and side 13a is θ1, and the angle between side 23a and side 13a is θ3. These angles can be determined from DOA data, as described in more detail below.

[0063] In some embodiments, only the relative length of the triangle sides can be determined. In alternative embodiments, the actual length of the triangle sides can be estimated. According to some such embodiments, the actual length of the triangle sides can be estimated based on TOA data, for example, based on the arrival time of the sound generated by an audio device located at one triangle vertex and detected by an audio device located at another triangle vertex. Alternatively or in addition, the length of the triangle sides can be estimated based on the electromagnetic waves generated by an audio device located at one triangle vertex and detected by an audio device located at another triangle vertex. For example, the length of the triangle sides can be estimated based on the signal strength of the electromagnetic waves generated by an audio device located at one triangle vertex and detected by an audio device located at another triangle vertex. In some embodiments, the length of the triangle sides can be estimated based on the phase shift of the detected electromagnetic waves.

[0064] Figure 2 Shows Figure 1 Another example of the geometric relationship between three audio devices in the environment shown in . In this example, the vertices of triangle 110b are at positions 1, 3, and 4. Here, triangle 110b has sides 13b, 14, and 34a. According to this example, the angle between side 13b and side 14 is θ4, the angle between side 13b and side 34a is θ5, and the angle between side 34a and side 14 is θ6.

[0065] By comparison Figure 1 and Figure 2 , it can be observed that the length of side 13a of triangle 110a should be equal to the length of side 13b of triangle 110b. In some embodiments, the side length of a triangle (e.g., triangle 110a) can be assumed to be correct, and the length of the side shared by adjacent triangles will be constrained to be that length.

[0066] Figure 3A Shows Figure 1 and Figure 2 The two triangles depicted in FIG. 1 have no corresponding audio devices and other features of the environment. FIG. 3 shows estimates of the side lengths and angular orientations of triangles 110a and 110b. Figure 3AIn the example shown in FIG. 1 , the length of side 13b of triangle 110b is constrained to be the same as the length of side 13a of triangle 110a. The lengths of the other sides of triangle 110b are scaled in proportion to the resulting change in the length of side 13b. The resulting triangle 110b′ is Figure 3A Shown adjacent to triangle 110a.

[0067] According to some implementations, the side lengths of other triangles adjacent to triangles 110a and 110b may be determined in a similar manner until all audio device locations in environment 100 have been determined.

[0068] Some examples of audio device locations may be performed as follows. Each audio device may report the DOA of each other audio device in the environment (e.g., room) based on the sound produced by each other audio device in the environment. The Cartesian coordinates of the ith audio device may be represented as x i =[x i ,y i ] T , where the superscript T indicates the vector transpose. Given M audio devices in an environment, i = {1...M}.

[0069] Figure 3B An example of estimating the interior angle of a triangle formed by three audio devices is shown. In this example, the audio devices are i, j, and k. The DOA of a sound source emanating from device j as observed by device i can be expressed as θ ji The DOA of the sound source emitted from device k observed by device i can be expressed as θ ki .exist Figure 3B In the example shown in ji and θ ki is measured from axis 305a, the orientation of which is arbitrary and which may, for example, correspond to the orientation of audio device i. The interior angle a of triangle 310 may be expressed as a=θ ki -θ ji It can be observed that the calculation of the interior angle a is independent of the orientation of the axis 305a.

[0070] exist Figure 3B In the example shown in ij and θ kj is measured from axis 305b, the orientation of which is arbitrary and which may correspond to the orientation of audio device j. The interior angle b of triangle 310 may be expressed as b=θ ij -θ kj Similarly, in this example, θ jk and θ ik is measured from axis 305c. The interior angle c of triangle 310 can be expressed as c=θjk -θ ik .

[0071] In the presence of measurement errors, a+b+c≠180°. This can be improved by predicting each angle from the other two angles and averaging them, for example, as follows:

[0072]

[0073] In some embodiments, the edge lengths (A, B, C) can be calculated (based on the scaling error) by applying the sine rule. In some examples, an edge length can be assigned an arbitrary value, such as 1. For example, by setting A=1 and placing the vertex Placed at the origin, the positions of the remaining two vertices can be calculated as follows:

[0074]

[0075] However, any rotation may be acceptable.

[0076] According to some embodiments, the process of triangle parameterization can be repeated for all possible subsets of three audio devices in the environment, In some examples, T l The lth triangle may be represented. The triangles may not be enumerated in any particular order, depending on the implementation. Due to possible errors in DOA and / or edge length estimation, triangles may overlap and may not be perfectly aligned.

[0077] Figure 4 is an overview of what can be done by Fig.11 400. As with other methods described herein, the blocks of method 400 need not be performed in the order indicated. In addition, such a method may include more or fewer blocks than those shown and / or described. In this embodiment, method 400 involves estimating the position of a speaker in an environment. The blocks of method 400 may be performed by one or more devices, which may be (or may include) Fig.11 The device 1100 is shown in FIG.

[0078] In this example, block 405 involves obtaining direction of arrival (DOA) data for each of a plurality of audio devices. In some examples, the plurality of audio devices may include all audio devices in an environment, such as Figure 1 All audio devices 105 are shown.

[0079] However, in some instances, the plurality of audio devices may include only a subset of all audio devices in the environment. For example, the plurality of audio devices may include all smart speakers in the environment, but not one or more of the other audio devices in the environment.

[0080] The DOA data may be obtained in a variety of ways, depending on the particular implementation. In some instances, determining the DOA data may involve determining the DOA data for at least one of the plurality of audio devices. For example, determining the DOA data may involve receiving microphone data from each of a plurality of audio device microphones corresponding to a single audio device in the plurality of audio devices and determining the DOA data for the single audio device based at least in part on the microphone data. Alternatively or additionally, determining the DOA data may involve receiving antenna data from one or more antennas corresponding to a single audio device in the plurality of audio devices and determining the DOA data for the single audio device based at least in part on the antenna data.

[0081] In some such examples, a single audio device itself may determine the DOA data. According to some such embodiments, each of the plurality of audio devices may determine its own DOA data. However, in other embodiments, another device (which may be a local or remote device) may determine the DOA data for one or more audio devices in the environment. According to some embodiments, a server may determine the DOA data for one or more audio devices in the environment.

[0082] According to this example, block 410 involves determining an interior angle of each of a plurality of triangles based on the DOA data. In this example, each of the plurality of triangles has vertices corresponding to audio device locations of three audio devices. Some such examples are described above.

[0083] Figure 5 An example is shown in which each audio device in the environment is a vertex of multiple triangles. The sides of each triangle correspond to the distance between two audio devices 105.

[0084] In this embodiment, block 415 involves determining the length of each side of each triangle. (The sides of a triangle may also be referred to herein as "edges.") According to this example, the side lengths are based at least in part on the interior angles. In some instances, the side lengths may be calculated by determining a first length of a first side of the triangle and determining the lengths of a second side and a third side of the triangle based on the interior angles of the triangle. Some such examples are described above.

[0085] According to some such embodiments, determining the first length may involve setting the first length to a predetermined value. The lengths of the second side and the third side may then be determined based on the interior angle of the triangle. All sides of the triangle may be determined based on predetermined values ​​(e.g., reference values). To obtain the actual distance (length) between audio devices in the environment, a normalized scaling may be applied to the reference value given by: Figure 4 The geometric shapes produced by the alignment process described in blocks 420 and 425 of FIG. 4 can be standardized scaling. The standardized scaling can include scaling the aligned triangles so that the triangles fit within a bounding shape having a size corresponding to the environment, such as a circle, a polygon, etc. The size of the shape can be the size of a typical home environment or any size suitable for a particular implementation. However, scaling the aligned triangles is not limited to fitting the geometric shapes to a particular bounding shape, and any other scaling criteria suitable for a particular implementation can be used.

[0086] In some examples, determining the first length can be based on arrival time data and / or received signal strength data. In some embodiments, the arrival time data and / or received signal strength data can correspond to sound waves from a first audio device in the environment detected by a second audio device in the environment. Alternatively or additionally, the arrival time data and / or received signal strength data can correspond to electromagnetic waves (e.g., radio waves, infrared waves, etc.) from a first audio device in the environment detected by a second audio device in the environment. When the arrival time data and / or received signal strength data are not available, the first length can be set to a predetermined value as described above.

[0087] According to this example, block 420 involves performing a forward alignment process that aligns each of the plurality of triangles in a first order. According to this example, the forward alignment process generates a forward alignment matrix.

[0088] According to some of these examples, we expect the triangle to have edges (x i , x j ) are aligned in a way that is equal to adjacent edges, for example, Figure 3A As shown in and described above. Let ε be In some such implementations, block 420 may involve traversing ε and aligning common edges of triangles in forward order by forcing edges to be consistent with edges of previously aligned edges.

[0089] Figure 6 An example of a portion of the forward alignment process is provided. Figure 6 The numbers 1 to 5 in bold are Figure 1 , Figure 2 and Figure 5 corresponds to the audio device position shown in . Figure 6The order of the forward alignment process shown in and described herein is only an example.

[0090] In this example, as in Figure 3A , the length of side 13b of triangle 110b is forced to be consistent with the length of side 13a of triangle 110a. Figure 6 The resulting triangle 110b' is shown in , where the same internal angle is maintained. According to this example, the length of the side 13c of the triangle 110c is also forced to coincide with the length of the side 13a of the triangle 110a. Figure 6 The resulting triangle 110c' is shown in , where the same interior angles are maintained.

[0091] Next, in this example, the length of side 34b of triangle 110d is forced to coincide with the length of side 34a of triangle 110b'. Also, in this example, the length of side 23b of triangle 110d is forced to coincide with the length of side 23a of triangle 110a. Figure 6 The resulting triangle 110d' is shown in FIG. 1 , where the same interior angle is maintained. According to some such examples, Figure 5 The remaining triangles shown in FIG. 1 may be processed in the same manner as triangles 110 b , 110 c , and 110 d .

[0092] The results of the forward alignment process can be stored in a data structure. According to some such examples, the results of the forward alignment process can be stored in a forward alignment matrix. For example, the results of the forward alignment process can be stored in a matrix , where N indicates the total number of triangles.

[0093] Multiple audio device position estimates will occur when the DOA data and / or initial side length determination contain errors. The errors typically increase during the forward alignment process.

[0094] Figure 7 An example of multiple audio device position estimates that have occurred during the forward alignment process is shown. In this example, the forward alignment process is based on a triangle with the seven audio device positions as its vertices. Here, due to the additional errors in the DOA estimates, the triangles are not perfectly aligned. Figure 7 The positions of numbers 1 through 7 shown in correspond to the estimated audio device positions produced by the forward alignment process. In this example, the audio device position estimates labeled "1" agree, but the audio device position estimates for audio devices 6 and 7 show large differences, as shown by the relatively large areas where numbers 6 and 7 are located.

[0095] return Figure 4In this example, block 425 involves a reverse alignment process of aligning each of the plurality of triangles in a second order that is reversed from the first order. According to some implementations, the reverse alignment process may involve traversing ε as before but in a reverse order. In an alternative example, the reverse alignment process may not be exactly the reverse order of operations of the forward alignment process. According to this example, the reverse alignment process produces a reverse alignment matrix, which may be represented herein as

[0096] Figure 8 An example of a portion of the reverse alignment process is provided. Figure 8 The numbers 1 to 5 in bold are Figure 1 , Figure 2 and Figure 5 corresponds to the audio device position shown in . Figure 8 The order of the reverse alignment process shown in and described herein is only an example.

[0097] exist Figure 8 In the example shown in , triangle 110e is based on audio device locations 3, 4, and 5. In this implementation, the side lengths (or "edges") of triangle 110e are assumed to be correct, and the side lengths of adjacent triangles are forced to be consistent with them. According to this example, the length of side 45b of triangle 110f is forced to be consistent with the length of side 45a of triangle 110e. Figure 8 The resulting triangle 110f' is shown in , where the interior angles remain the same. In this example, the length of side 35b of triangle 110c is forced to coincide with the length of side 35a of triangle 110e. Figure 8 The resulting triangle 110c" is shown in FIG. 1, where the interior angles remain the same. According to some such examples, Figure 5 The remaining triangles shown in can be processed in the same manner as triangles 110c and 110f until the reverse alignment process has included all remaining triangles.

[0098] Fig. 9 An example of multiple audio device position estimates that have occurred during the reverse alignment process is shown. In this example, the reverse alignment process is based on having the same Figure 7 A triangle with seven identical vertices describing the location of the audio device. Fig. 9 The positions of numbers 1 to 7 shown in correspond to the estimated audio device positions resulting from the inverse alignment process. Here again, the triangles are not perfectly aligned due to the additional errors in the DOA estimates. In this example, the audio device position estimates labeled 6 and 7 agree, but the audio device position estimates for audio devices 1 and 2 show greater differences.

[0099] return Figure 4, block 430 involves generating a final estimate of the position of each audio device based at least in part on the value of the forward alignment matrix and the value of the reverse alignment matrix. In some examples, generating the final estimate of the position of each audio device may involve translating and scaling the forward alignment matrix to generate a translated and scaled forward alignment matrix, and translating and scaling the reverse alignment matrix to generate a translated and scaled reverse alignment matrix.

[0100] For example, by moving the center of mass to the origin and enforcing a unit Frobenius norm (e.g., and ) to fix pan and zoom.

[0101] According to some such examples, generating a final estimate of each audio device position may also involve generating a rotation matrix based on the translated and scaled forward alignment matrix and the translated and scaled reverse alignment matrix. The rotation matrix may include multiple estimated audio device positions for each audio device. For example, the best rotation between the forward alignment and the reverse alignment may be found by singular value decomposition. In some such examples, generating the rotation matrix may involve performing singular value decomposition on the translated and scaled forward alignment matrix and the translated and scaled reverse alignment matrix, for example, as follows:

[0102]

[0103] In the above equations, U represents the matrix The left singular vector of and V represents the matrix The above equation produces the rotation matrix R = VU T . Matrix product VU T Generate a rotation matrix such that is optimally rotated to Alignment.

[0104] According to some examples, when determining the rotation matrix R=VU T Afterwards, the alignments can be averaged, for example, as follows:

[0105]

[0106] In some embodiments, generating a final estimate of each audio device position may also involve averaging the estimated audio device positions of each audio device to generate a final estimate of each audio device position. Various disclosed embodiments have been shown to be robust even when the DOA data and / or other calculations include significant errors. For example, due to overlapping vertices from multiple triangles, Contains the same nodes The final estimate is produced by averaging across common nodes.

[0107] Fig.10 A comparison of the estimated audio device position and the actual audio device position is shown. Fig.10 In the example shown in FIG. 1 , the audio device position is the same as in the example shown in FIG. 1 . Figure 7 and Fig. 9 The forward alignment process and the backward alignment process described above correspond to the estimated audio device positions during the forward alignment process. In these examples, the error in the DOA estimates has a standard deviation of 15 degrees. Nevertheless, the final estimate of each audio device position (each of the final estimates in Fig.10 ) and the actual audio device position (each of which is represented by an "x" in Fig.10 By performing the forward alignment process in a first order and the backward alignment process in a second order opposite to the first order, the errors / inaccuracies of the arrival direction estimates (data) are averaged out, thereby producing an overall error in the estimate of the position of the audio device in the environment. Errors tend to accumulate in the alignment order, such as Figure 7 (where a larger number of vertices shows a larger alignment spread) and Fig. 9 (where a smaller number of vertices shows a larger spread). The process of traversing the sequence in reverse order also reverses the alignment error, thereby averaging the overall error in the final position estimate.

[0108] Fig.11 1 is a block diagram showing an example of components of an apparatus capable of implementing various aspects of the present disclosure. According to some examples, the apparatus 1100 may be or may include an intelligent audio device (such as a smart speaker) configured to perform at least some of the methods disclosed herein. In other embodiments, the apparatus 1100 may be or may include other devices configured to perform at least some of the methods disclosed herein. In some such embodiments, the apparatus 1100 may be or may include a server.

[0109] In this example, the apparatus 1100 includes an interface system 1105 and a control system 1110. In some embodiments, the interface system 1105 can be configured to receive input from each of a plurality of microphones in the environment. The interface system 1105 can include one or more network interfaces and / or one or more external device interfaces (such as one or more universal serial bus (USB) interfaces). According to some embodiments, the interface system 1105 can include one or more wireless interfaces. The interface system 1105 can include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. In some examples, the interface system 1105 can include a control system 1110 and a memory system (such as Fig.11 However, the control system 1110 may include a memory system.

[0110] For example, the control system 1110 may include a general purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components. In some embodiments, the control system 1110 may reside in more than one device. For example, a portion of the control system 1110 may reside in Figure 1 100, and another portion of the control system 1110 may reside in a device outside of the environment 100, such as a server, a mobile device (e.g., a smart phone or tablet computer), etc. In some such examples, the interface system 1105 may also reside in more than one device.

[0111] In some embodiments, the control system 1110 can be configured to at least partially perform the methods disclosed herein. According to some examples, the control system 1110 can be configured to implement the above-mentioned Figure 4 The methods described and / or referenced below Fig.12 and the methods described below.In some such examples, the control system 1110 can be configured to determine an estimate for each of a plurality of audio device locations in the environment based at least in part on output from the classifier.

[0112] In some examples, apparatus 1100 may include Fig.11 Optional microphone system 1120 is depicted in FIG. Microphone system 1120 may include one or more microphones. In some examples, microphone system 1120 may include a microphone array. In some examples, apparatus 1100 may include Fig.11Optional speaker system 1125 is depicted in FIG. Speaker system 1125 may include one or more loudspeakers. In some examples, microphone system 1120 may include a loudspeaker array. In some such examples, apparatus 1100 may be or may include an audio device. For example, apparatus 1100 may be or may include Figure 1 One of the audio devices 105 shown in .

[0113] In some examples, apparatus 1100 may include Fig.11 Optional antenna system 1130 shown in FIG. According to some examples, antenna system 1130 may include an antenna array. In some examples, antenna system 1130 may be configured to transmit and / or receive electromagnetic waves. According to some embodiments, control system 1110 may be configured to estimate the distance between two audio devices in an environment based on antenna data from antenna system 1130. For example, control system 1110 may be configured to estimate the distance between two audio devices in an environment based on the arrival time of antenna data and / or the received signal strength of antenna data.

[0114] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. The one or more non-transitory media may be located, for example, in a Fig.11 110 and / or the control system 1110. Thus, various innovative aspects of the subject matter described in the present disclosure may be implemented in one or more non-transitory media having software stored thereon. For example, the software may include instructions for controlling at least one device to process audio data. For example, the software may be provided by Fig.11 The control system 1110 and other control system components are executed.

[0115] Much of the foregoing discussion has dealt with automatic positioning of audio devices. The following discussion expands upon some of the methods for determining listener position and listener angular orientation briefly described above. In the foregoing description, the term "rotation" is used in substantially the same manner as the term "orientation" is used in the following description. For example, the "rotation" mentioned above may refer to a global rotation of the final loudspeaker geometry, rather than to a rotation of the final loudspeaker geometry as in the above reference to the rotation of the final loudspeaker geometry. Figure 4 and the rotation of the individual triangles during the process described below.This global rotation or orientation may be resolved with reference to the listener angular orientation, for example by the direction the listener is looking, the direction the listener's nose is pointing, etc.

[0116] Various satisfactory methods for estimating the listener position are known in the art, some of which are described below. However, estimating the listener angular orientation can be challenging. Some relevant methods are described in detail below.

[0117] Determining the listener position and the listener angular orientation can enable some desirable features, such as directionally positioned audio devices relative to the listener. Knowing the listener position and angular orientation allows determining, for example, which speakers in the environment are in front, which are in the back, which are near the center (if any), etc. relative to the listener.

[0118] After establishing an association between the audio device position and the listener's position and orientation, some embodiments may involve providing the audio device position data, the audio device angular orientation data, the listener position data, and the listener angular orientation data to an audio rendering system. Alternatively or additionally, some embodiments may involve an audio data rendering process based at least in part on the audio device position data, the audio device angular orientation data, the listener position data, and the listener angular orientation data.

[0119] Fig.12 is an overview of what can be done by Fig.11 1200. As with other methods described herein, the blocks of method 1200 need not be performed in the order indicated. In addition, such a method may include more or fewer blocks than those shown and / or described. In this example, the blocks of method 1200 are performed by a control system, which may be (or may include) Fig.11 As described above, in some embodiments, the control system 1110 may be located in a single device, while in other embodiments, the control system 1110 may be located in two or more devices.

[0120] In this example, block 1205 involves obtaining direction of arrival (DOA) data for each of a plurality of audio devices in the environment. In some examples, the plurality of audio devices may include all audio devices in the environment, such as Figure 1 All audio devices 105 are shown.

[0121] However, in some instances, the plurality of audio devices may include only a subset of all audio devices in the environment. For example, the plurality of audio devices may include all smart speakers in the environment, but not one or more of the other audio devices in the environment.

[0122] DOA data can be obtained in various ways, depending on the specific implementation. In some instances, determining DOA data can involve determining DOA data of at least one audio device in a plurality of audio devices. In some examples, DOA data can be obtained by controlling each of a plurality of loudspeakers in an environment to reproduce a test signal. For example, determining DOA data can involve receiving microphone data from each of a plurality of audio device microphones corresponding to a single audio device in a plurality of audio devices and determining DOA data of a single audio device based at least in part on the microphone data. Alternatively or additionally, determining DOA data can involve receiving antenna data from one or more antennas corresponding to a single audio device in a plurality of audio devices and determining DOA data of a single audio device based at least in part on the antenna data.

[0123] In some such examples, a single audio device itself may determine the DOA data. According to some such embodiments, each of the plurality of audio devices may determine its own DOA data. However, in other embodiments, another device (which may be a local or remote device) may determine the DOA data for one or more audio devices in the environment. According to some embodiments, a server may determine the DOA data for one or more audio devices in the environment.

[0124] according to Fig.12 In the example shown in FIG. 12 , block 1210 involves generating audio device position data via the control system based at least in part on the DOA data. In this example, the audio device position data includes an estimate of the audio device position of each audio device referenced in block 1205 .

[0125] The audio device position data may be, for example, (or include) coordinates of a coordinate system (such as a Cartesian coordinate system, a spherical coordinate system, or a cylindrical coordinate system). The coordinate system may be referred to herein as an audio device coordinate system. In some such examples, the audio device coordinate system may be oriented with reference to one of the audio devices in the environment. In other examples, the audio device coordinate system may be oriented with reference to an axis defined by a line between two audio devices in the audio devices in the environment. However, in other examples, the audio device coordinate system may be oriented with reference to other parts of the environment (such as a television, a wall of a room, etc.).

[0126] In some examples, block 1210 may refer to the above reference Figure 4According to some such examples, block 1210 may involve determining an interior angle of each of a plurality of triangles based on the DOA data. In some instances, each of the plurality of triangles may have vertices corresponding to audio device locations of three audio devices. Some such methods may involve determining a length of each side of each triangle based at least in part on the interior angles.

[0127] Some such methods may involve performing a forward alignment process that aligns each of the plurality of triangles in a first order to produce a forward alignment matrix. Some such methods may involve performing a reverse alignment process that aligns each of the plurality of triangles in a second order that is reverse of the first order to produce a reverse alignment matrix. Some such methods may involve generating a final estimate of the position of each audio device based at least in part on the values ​​of the forward alignment matrix and the values ​​of the reverse alignment matrix. However, in some embodiments of method 1200, block 1210 may involve applying a method other than the one described above with reference to FIG. Figure 4 Methods other than those described.

[0128] In this example, block 1215 involves determining, via the control system, listener position data indicating a listener position within the environment. For example, the listener position data may be referenced to an audio device coordinate system. However, in other examples, the coordinate system may be oriented with reference to the listener or to a portion of the environment (e.g., a television, a wall of a room, etc.).

[0129] In some examples, block 1215 may involve prompting a listener (e.g., via an audio prompt from one or more loudspeakers in the environment) to speak one or more utterances and estimating the listener position based on the DOA data. The DOA data may correspond to microphone data obtained by multiple microphones in the environment. The microphone data may correspond to detection of one or more utterances by the microphones. At least some of the microphones may be co-located with the loudspeakers. According to some examples, block 1215 may involve a triangulation process. For example, block 1215 may involve triangulating the user's voice by finding intersections between DOA vectors passing through the audio device, for example, as described below with reference to Fig.13A According to some embodiments, block 1215 (or another operation of method 1200) may involve, after determining the listener position, co-locating the origin of the audio device coordinate system and the origin of the listener coordinate system. Co-locating the origin of the audio device coordinate system and the origin of the listener coordinate system may involve transforming the audio device position from the audio device coordinate system to the listener coordinate system.

[0130] According to the embodiment, block 1220 involves determining, via the control system, listener angular orientation data indicating the listener angular orientation. For example, the listener angular orientation data may be obtained with reference to a coordinate system (such as an audio device coordinate system) used to represent the listener position data. In some such examples, the listener angular orientation data may be obtained with reference to an origin and / or axis of the audio device coordinate system.

[0131] However, in some embodiments, the listener angular orientation data may be derived with reference to an axis defined by the listener position and other points in the environment (e.g., a television, an audio device, a wall, etc.). In some such embodiments, the listener position may be used to define the origin of a listener coordinate system. In some such examples, the listener angular orientation data may be derived with reference to the axis of the listener coordinate system.

[0132] Various methods for performing block 1220 are disclosed herein. According to some examples, the listener angular orientation may correspond to the listener viewing direction. In some such examples, the listener viewing direction may be inferred from the listener position data, for example, by assuming that the listener is viewing a particular object, such as a television. In some such implementations, the listener viewing direction may be determined based on the listener position and the television position. Alternatively or additionally, the listener viewing direction may be determined based on the listener position and the television soundbar position.

[0133] However, in some examples, the listener viewing direction can be determined based on listener input. According to some such examples, the listener input can include inertial sensor data received from a device held by the listener. The listener can use the device to point to a location in the environment (e.g., a location corresponding to the direction the listener is facing). For example, the listener can use the device to point to a loudspeaker that is emitting sound (a loudspeaker that is reproducing sound). Therefore, in such examples, the inertial sensor data can include inertial sensor data corresponding to the loudspeaker that is emitting sound.

[0134] In some such instances, the listener input can include an indication of an audio device selected by the listener. In some examples, the indication of the audio device can include inertial sensor data corresponding to the selected audio device.

[0135] However, in other examples, the indication of the audio device may be made based on one or more utterances of the listener (e.g., "The TV is now in front of me," "Speaker 2 is now in front of me," etc. Other examples of determining listener angular orientation data based on one or more utterances of the listener are described below.

[0136] according to Fig.12In the example shown in , block 1225 involves determining, via the control system, audio device angular orientation data indicating the audio device angular orientation of each audio device relative to the listener position and the listener angular orientation. According to some such examples, block 1225 may involve rotating the audio device coordinates around a point defined by the listener position. In some implementations, block 1225 may involve transforming the audio device position data from an audio device coordinate system to a listener coordinate system. Some examples are described below.

[0137] Fig.13A Shows Fig.12 According to some such examples, the audio device position data includes an audio device position estimate for each of the audio devices 1 to 5 with reference to an audio device coordinate system 1307. In this embodiment, the audio device coordinate system 1307 is a Cartesian coordinate system with the position of the microphone of the audio device 2 as the origin. Here, the x-axis of the audio device coordinate system 1307 corresponds to the line 1303 between the microphone position of the audio device 2 and the microphone position of the audio device 1.

[0138] In this example, the listener position is determined by prompting a listener 1305, shown sitting on a couch 103 (e.g., via an audio prompt from one or more loudspeakers in the environment 1300a), to speak one or more utterances 1327 and estimating the listener position based on time of arrival (TOA) data. The TOA data corresponds to microphone data obtained by multiple microphones in the environment. In this example, the microphone data corresponds to detection of one or more utterances 1327 by microphones of at least some (e.g., 3, 4, or all 5) of the audio devices 1 to 5.

[0139] Alternatively or additionally, the listener position is based on DOA data provided by microphones of at least some (e.g., 2, 3, 4, or all 5) of the audio devices 1 to 5. According to some such examples, the listener position can be determined based on the intersection of lines 1309a, 1309b, etc. corresponding to the DOA data.

[0140] According to this example, the listener position corresponds to the origin of the listener coordinate system 1320. In this example, the listener angular orientation data is indicated by the y' axis of the listener coordinate system 1320, which corresponds to the line 1313a between the listener's head 1310 (and / or the listener's nose 1325) and the sound bar 1330 of the television 101. Fig.13A In the example shown in FIG. 1 , line 1313a is parallel to the y′ axis. Therefore, the angle represents the angle between the y-axis and the y'-axis. In this example, Fig.12Block 1225 may involve rotating the audio device coordinates around the origin of the listener coordinate system 1320 by an angle Thus, although the origin of the audio device coordinate system 1307 is shown as Fig.13A , but some embodiments involve rotating the audio device coordinates around the origin of the listener coordinate system 1320 by an angle Previously, the origin of the audio device coordinate system 1307 was co-located with the origin of the listener coordinate system 1320. This co-location may be performed by a coordinate transformation from the audio device coordinate system 1307 to the listener coordinate system 1320.

[0141] In some examples, the location of the soundbar 1330 and / or the TV 101 can be determined by causing the soundbar to emit a sound and estimating the location of the soundbar based on the DOA and / or TOA data, which can correspond to the detection of the sound by the microphones of at least some (e.g., 3, 4, or all 5) of the audio devices 1 to 5. Alternatively or in addition, the location of the soundbar 1330 and / or the TV 101 can be determined by prompting the user to approach the TV and locating the user's speech by the DOA and / or TOA data, which can correspond to the detection of the sound by the microphones of at least some (e.g., 3, 4, or all 5) of the audio devices 1 to 5. This method can involve triangulation. Such an example can be beneficial in the case where the soundbar 1330 and / or the TV 101 do not have an associated microphone.

[0142] In some other examples, where the soundbar 1330 and / or the television 101 do have an associated microphone, the location of the soundbar 1330 and / or the television 101 can be determined according to a TOA or DOA method (such as the DOA method disclosed herein). According to some such methods, the microphone can be co-located with the soundbar 1330.

[0143] According to some embodiments, the soundbar 1330 and / or the television 101 may have an associated camera 1311. The control system may be configured to capture an image of the listener's head 1310 (and / or the listener's nose 1325). In some such examples, the control system may be configured to determine a line 1313a between the listener's head 1310 (and / or the listener's nose 1325) and the camera 1311. The listener angular orientation data may correspond to the line 1313a. Alternatively or additionally, the control system may be configured to determine an angle between the line 1313a and the y-axis of the audio device coordinate system.

[0144] Fig. 13B An additional example of determining listener angular orientation data is shown. According to this example, the listener position is already in Fig.12 1305. Here, the control system controls the loudspeakers of the environment 1300b to render the audio objects 1335 to various locations within the environment 1300b. In some such examples, the control system may cause the loudspeakers to render the audio objects 1335 so that the audio objects 1335 appear to rotate around the listener 1305, for example, by rendering the audio objects 1335 so that the audio objects 1335 appear to rotate around the origin of the listener coordinate system 1320. In this example, the curved arrow 1340 shows a portion of the trajectory of the audio object 1335 as it rotates around the listener 1305.

[0145] According to some such examples, the listener 1305 may provide user input (e.g., saying "stop") indicating when the audio object 1335 is in the direction that the listener 1305 is facing. In some such examples, the control system may be configured to determine a line 1313b between the listener position and the position of the audio object 1335. In this example, the line 1313b corresponds to the y' axis of the listener coordinate system indicating the direction that the listener 1305 is facing. In alternative embodiments, the listener 1305 may provide user input indicating when the audio object 1335 is in front of the environment, at the TV position of the environment, at the audio device position, etc.

[0146] Fig. 13C An additional example of determining listener angular orientation data is shown. According to this example, the listener position is already in Fig.12 Here, the listener 1305 is using a handheld device 1345 to provide input regarding the viewing direction of the listener 1305 by pointing the handheld device 1345 toward the television 101 or the sound bar 1330. In this example, the dashed outlines of the handheld device 1345 and the listener's arm indicate that before the listener 1305 pointed the handheld device 1345 toward the television 101 or the sound bar 1330, the listener 1305 pointed the handheld device 1345 toward the audio device 2. In other examples, the listener 1305 may have pointed the handheld device 1345 toward another audio device, such as the audio device 1. According to this example, the handheld device 1345 is configured to determine an angle α between the audio device 2 and the television 101 or the sound bar 1330, which approximates the angle between the audio device 2 and the viewing direction of the listener 1305.

[0147] In some examples, handheld device 1345 may be a cellular telephone that includes an inertial sensor system and a wireless interface configured to communicate with a control system for controlling audio devices of environment 1300c. In some examples, handheld device 1345 may run an application or "app" that is configured to control handheld device 1345 to perform necessary functions, such as by providing user prompts (e.g., via a graphical user interface), by receiving input indicating that handheld device 1345 is pointed in a desired direction, by saving corresponding inertial sensor data and / or transmitting corresponding inertial sensor data to the control system for controlling audio devices of environment 1300c, etc.

[0148] According to this example, the control system (which can be a control system of the handheld device 1345 or a control system of the audio device of the control environment 1300c) is configured to determine the orientation of the lines 1313c and 1350 based on the inertial sensor data (e.g., based on the gyroscope data). In this example, the line 1313c is parallel to the axis y' and can be used to determine the listener angular orientation. According to some examples, the control system can determine the appropriate rotation of the audio device coordinates around the origin of the listener coordinate system 1320 based on the angle α between the audio device 2 and the viewing direction of the listener 1305.

[0149] Fig.13D According to the reference Fig. 13C An example of a method for determining an appropriate rotation of audio device coordinates is described. In this example, the origin of the audio device coordinate system 1307 is co-located with the origin of the listener coordinate system 1320. After the process of 1215 in which the listener position is determined, it is possible to co-locate the origin of the audio device coordinate system 1307 and the origin of the listener coordinate system 1320. Co-locating the origin of the audio device coordinate system 1307 and the origin of the listener coordinate system 1320 may involve transforming the audio device position from the audio device coordinate system 1307 to the listener coordinate system 1320. As described above with reference to Fig. 13C The angle α is determined as described. Thus, the angle α corresponds to the desired orientation of the audio device 2 in the listener coordinate system 1320. In this example, the angle β corresponds to the orientation of the audio device 2 in the audio device coordinate system 1307. (β-α in this example) indicates the rotation necessary to align the y-axis of the audio device coordinate system 1307 with the y′ axis of the listener coordinate system 1320 .

[0150] In some embodiments, Fig.12 The method may involve controlling at least one of the audio devices in the environment based at least in part on the corresponding audio device position, the corresponding audio device angular orientation, the listener position data, and the listener angular orientation data.

[0151] For example, some embodiments may involve providing audio device position data, audio device angular orientation data, listener position data, and listener angular orientation data to an audio rendering system. In some examples, the audio rendering system may be controlled by a control system (e.g., Fig.11 Some embodiments may involve controlling the audio data rendering process based at least in part on audio device position data, audio device angular orientation data, listener position data, and listener angular orientation data. Some such embodiments may involve providing loudspeaker acoustic capability data to the rendering system. The loudspeaker acoustic capability data may correspond to one or more loudspeakers of the environment. The loudspeaker acoustic capability data may indicate the orientation of one or more drivers, the number of drivers, or the driver frequency response of one or more drivers. In some examples, the loudspeaker acoustic capability data may be retrieved from a memory and then provided to the rendering system.

[0152] Existing flexible rendering techniques include Centroid Amplitude Translation (CMAP) and Flexible Virtualization (FV). At a high level, these two techniques render a set of one or more audio signals, each with an associated desired perceptual spatial position, for playback on a set of two or more speakers, where the relative activation of the set of speakers is a function of a model of the perceptual spatial position of the audio signal played back by the speakers and the proximity of the desired perceptual spatial position of the audio signal to the speaker positions. The model ensures that the listener hears the audio signal near its expected spatial position, and the proximity term controls which speakers are used to achieve this spatial impression. Specifically, the proximity term favors activating speakers that are close to the desired perceptual spatial position of the audio signal. For both CMAP and FV, this functional relationship can be conveniently derived from a cost function written as the sum of two terms, one for the spatial aspect and one for proximity:

[0153]

[0154] Here, collection represents the positions of a set of M loudspeakers, denotes the desired perceptual spatial location of the audio signal, and g denotes the M-dimensional vector of loudspeaker activations. For CMAP, each activation in the vector represents a gain for each loudspeaker, while for FV, each activation represents a filter (in the second case, g can equivalently be viewed as a vector of complex values ​​at a particular frequency, and different g are computed across multiple frequencies to form the filter). The optimal vector of activations (g opt ) is found by minimizing a cost function across activations:

[0155]

[0156] Under certain definitions of the cost function, it is difficult to control the absolute level of the optimal activation resulting from the above minimization, although g opt The relative levels between the components are appropriate. To solve this problem, you can perform g opt Subsequent normalization of in order to control the absolute level of activation. For example, it may be desirable to normalize the vector to have unit length, which complies with the commonly used constant power shift rule:

[0157]

[0158] The exact behavior of the flexible rendering algorithm is determined by the two terms in the cost function C 空间 and C 接近度 For CMAP, C 空间 is obtained from a model that places the perceived spatial position of an audio signal played from a set of loudspeakers at the locations indicated by the associated activation gains g of the loudspeakers. i The centroids of the positions of these loudspeakers weighted by (elements of vector g):

[0159]

[0160] Equation 3 is then manipulated to represent the spatial cost of the squared error between the desired audio position and the audio position produced by the activated loudspeakers:

[0161]

[0162] For FV, the spatial term of the cost function is defined differently. The goal is to produce the same spatial frequencies at the left and right ears as the audio object positions. The corresponding binaural response b. Conceptually, b is a 2×1 vector of filters (one filter for each ear), but it is more convenient to think of it as a 2×1 vector of complex values ​​at a specific frequency. Continuing with this representation at a specific frequency, the desired binaural response can be obtained from a set of HRTF indices at the object location:

[0163]

[0164] Meanwhile, the 2x1 binaural response e produced by the loudspeaker at the listener's ears is modeled as a 2 x M acoustic transfer matrix H multiplied by an M x 1 vector g of complex loudspeaker activation values:

[0165] e=Hg (6)

[0166] The acoustic transmission matrix H is based on the set of loudspeaker positions relative to the listener position: Finally, the spatial component of the cost function is defined as the squared error between the desired binaural response (Equation 5) and the binaural response produced by the loudspeakers (Equation 6):

[0167]

[0168] Conveniently, the spatial terms of the cost functions for CMAP and FV defined in both Equations 4 and 7 can be rearranged as matrix quadratic functions as a function of the speaker activation g:

[0169]

[0170] where A is an M x M square matrix, B is a 1 x M vector, and C is a scalar. The rank of the matrix A is 2, and therefore when M > 2, there are infinitely many speaker activations g for which the spatial error term is equal to zero. The second term C introduced in the cost function 接近度 This uncertainty is removed and a specific solution is produced that has perceptually beneficial properties compared to other possible solutions. For both CMAP and FV, C 接近度 is constructed so that the position Far from the expected audio signal location The activation of speakers located close to the desired position is penalized more than the activation of speakers located close to the desired position. This construction produces a sparse optimal set of speaker activations, where only speakers close to the position of the desired audio signal are significantly activated, and in fact leads to a spatial reproduction of the audio signal that is perceptually more robust to listener movement around the set of speakers.

[0171] To this end, the second term C of the cost function 接近度 can be defined as the distance-weighted sum of the squared absolute values ​​of the speaker activations. This is concisely expressed in matrix form as:

[0172]

[0173] Where D is the diagonal matrix of distance penalties between the desired audio position and each speaker:

[0174]

[0175] The distance penalty function can take many forms, but the following is a useful parameterization

[0176]

[0177] in, is the Euclidean distance between the desired audio position and the speaker position, and α and β are adjustable parameters. The parameter α indicates the global strength of the penalty; d0 corresponds to the spatial extent of the distance penalty (loudspeakers at approximately d0 distance or further will be penalized), and β explains the abruptness of the onset of the penalty at distance d0.

[0178] Combining the two terms of the cost functions defined in Equations 8 and 9a yields the overall cost function

[0179] C(g)=g * Ag+Bg+C+g * Dg=g * (A+D)g+Bg+C (10)

[0180] Setting the derivative of this cost function with respect to g to zero and solving for g yields the optimal speaker activation solution:

[0181]

[0182] Typically, the optimal solution in Equation 11 may produce speaker activations with negative values. For CMAP construction of a flexible renderer, such negative activations may be undesirable, and thus Equation (11) may be minimized while keeping all activations positive.

[0183] Fig.14 and Fig.15 is a diagram illustrating a set of example speaker activation and object rendering positions given speaker positions of 4 degrees, 64 degrees, 165 degrees, -87 degrees, and -4 degrees. Fig.14 Speaker activations are shown that include the optimal solution to Eq. 11 for these specific speaker locations. Fig.15 The various speaker locations are depicted as orange, purple, green, gold, and blue dots. Fig.15 The ideal object positions (ie, the positions where the audio objects are to be rendered) for a number of possible object angles are also shown as green dots, and the corresponding actual rendered positions of these objects are shown as red dots connected to the ideal object positions by black dashed lines.

[0184] Although specific embodiments and applications of the present disclosure have been described herein, it will be apparent to those of ordinary skill in the art that many changes may be made to the embodiments and applications described herein without departing from the scope of the present disclosure.

[0185] Various aspects of the present disclosure may be understood from the following enumerated example embodiments (EEE):

[0186] 1. A method for locating an audio device, comprising:

[0187] obtaining direction of arrival (DOA) data for each of a plurality of audio devices;

[0188] determining an interior angle of each of a plurality of triangles based on the DOA data, each triangle of the plurality of triangles having vertices corresponding to audio device positions of three of the audio devices;

[0189] determining a length of each side of each of the triangles based at least in part on the interior angles;

[0190] performing a forward alignment process of aligning each of the plurality of triangles in a first order to generate a forward alignment matrix;

[0191] performing a reverse alignment process of aligning each of the plurality of triangles in a second order reverse to the first order to generate a reverse alignment matrix; and

[0192] A final estimate of the position of each audio device is generated based at least in part on the values ​​of the forward alignment matrix and the values ​​of the reverse alignment matrix.

[0193] 2. The method of claim 1, wherein generating the final estimate of the location of each audio device comprises:

[0194] translating and scaling the forward alignment matrix to produce a translated and scaled forward alignment matrix; and

[0195] The inverse alignment matrix is ​​translated and scaled to produce a translated and scaled inverse alignment matrix.

[0196] 3. A method as described in EEE 2, wherein generating the final estimate of each audio device position further includes generating a rotation matrix based on the translated and scaled forward alignment matrix and the translated and scaled reverse alignment matrix, the rotation matrix including multiple estimated audio device positions for each audio device.

[0197] 4. The method of EEE 3, wherein generating the rotation matrix comprises performing singular value decomposition on the translated and scaled forward alignment matrix and the translated and scaled reverse alignment matrix.

[0198] 5. The method of EEE 3 or EEE 4, wherein generating the final estimate of each audio device position further comprises averaging the estimated audio device position of each audio device to generate the final estimate of each audio device position.

[0199] 6. The method of any one of EEE 1 to 5, wherein determining the side length involves:

[0200] determining a first length of a first side of the triangle; and

[0201] The lengths of the second side and the third side of the triangle are determined based on the interior angle of the triangle.

[0202] 7. The method of EEE 6, wherein determining the first length involves setting the first length to a predetermined value.

[0203] 8. The method of EEE 6, wherein determining the first length is based on at least one of arrival time data or received signal strength data.

[0204] 9. The method of any one of EEEs 1 to 8, wherein obtaining the DOA data involves determining the DOA data for at least one audio device of the plurality of audio devices.

[0205] 10. The method of EEE 9, wherein determining the DOA data involves receiving microphone data from each of a plurality of audio device microphones corresponding to a single audio device of the plurality of audio devices and determining the DOA data for the single audio device based at least in part on the microphone data.

[0206] 11. The method of EEE 9, wherein determining the DOA data involves receiving antenna data from one or more antennas corresponding to a single audio device among the plurality of audio devices and determining the DOA data for the single audio device based at least in part on the antenna data.

[0207] 12. The method of any one of EEEs 1 to 11, further comprising controlling at least one of the audio devices based at least in part on the final estimate of the position of at least one audio device.

[0208] 13. The method of EEE 12, wherein controlling at least one of the audio devices involves controlling a loudspeaker of at least one of the audio devices.

[0209] 14. An apparatus configured to perform the method as described in any one of EEE 1 to 13.

[0210] 15. One or more non-transitory media having software recorded thereon, the software comprising instructions for controlling one or more devices to perform the method as described in any one of EEE 1 to 13.

[0211] 16. A method for configuring an audio device, comprising:

[0212] obtaining, via the control system, audio device direction of arrival (DOA) data for each audio device of a plurality of audio devices in the environment;

[0213] generating, via the control system, audio device position data based at least in part on the DOA data, the audio device position data comprising an estimate of an audio device position of each audio device;

[0214] determining, via the control system, listener position data indicative of a listener position within the environment;

[0215] determining, via the control system, listener angular orientation data indicative of a listener angular orientation; and

[0216] Audio device angular orientation data is determined via the control system, the audio device angular orientation data indicating an audio device angular orientation of each audio device relative to the listener position and the listener angular orientation.

[0217] 17. The method of EEE 16, further comprising controlling at least one of the audio devices based at least in part on a corresponding audio device position, a corresponding audio device angular orientation, the listener position data, and the listener angular orientation data.

[0218] 18. The method of EEE 16, further comprising providing the audio device position data, the audio device angular orientation data, the listener position data, and the listener angular orientation data to an audio rendering system.

[0219] 19. The method of EEE 16, further comprising controlling an audio data rendering process based at least in part on the audio device position data, the audio device angular orientation data, the listener position data, and the listener angular orientation data.

[0220] 20. The method of any one of EEEs 16 to 19, wherein obtaining the DOA data involves controlling each of a plurality of loudspeakers in the environment to reproduce a test signal.

[0221] 21. The method of any one of EEEs 16 to 20, wherein at least one of the listener position data or the listener angular orientation data is based on DOA data corresponding to one or more utterances of the listener.

[0222] 22. The method of any one of EEE 16 to 21, wherein the listener angular orientation corresponds to a listener viewing direction.

[0223] 23. The method of EEE 22, wherein the listener viewing direction is determined based on the listener position and the television position.

[0224] 24. The method of EEE 22, wherein the listener viewing direction is determined based on the listener position and a television sound bar position.

[0225] 25. The method of EEE 22, wherein the listener viewing direction is determined based on listener input.

[0226] 26. The method of EEE 25, wherein the listener input comprises inertial sensor data received from a device held by the listener.

[0227] 27. The method of EEE 25, wherein the inertial sensor data comprises inertial sensor data corresponding to a microphone emitting sound.

[0228] 28. The method of EEE 25, wherein the listener input comprises an indication of an audio device selected by the listener.

[0229] 29. The method of any one of EEEs 16 to 28, further comprising providing loudspeaker acoustic capability data to a rendering system, the loudspeaker acoustic capability data indicating at least one of an orientation of one or more drivers, a number of drivers, or a driver frequency response of one or more drivers.

[0230] 30. The method of any one of EEEs 16 to 29, wherein generating the audio device location data comprises:

[0231] determining an interior angle of each of a plurality of triangles based on the audio device DOA data, each triangle of the plurality of triangles having vertices corresponding to audio device positions of three of the audio devices;

[0232] determining a length of each side of each of the triangles based at least in part on the interior angles;

[0233] performing a forward alignment process of aligning each of the plurality of triangles in a first order to generate a forward alignment matrix;

[0234] performing a reverse alignment process of aligning each of the plurality of triangles in a second order reverse to the first order to generate a reverse alignment matrix; and

[0235] A final estimate of the position of each audio device is generated based at least in part on the values ​​of the forward alignment matrix and the values ​​of the reverse alignment matrix.

[0236] 31. An apparatus configured to perform the method as described in any one of EEE 16 to 30.

[0237] 32. One or more non-transitory media having software recorded thereon, the software comprising instructions for controlling one or more devices to perform the method as described in any one of EEE 16 to 30.

Claims

1. A method of determining the location of a plurality of audio devices in an environment, each audio device being configured to detect a signal generated by a different audio device of the plurality of audio devices, the method comprising: obtaining direction of arrival (DOA) data based on a detected direction of the signal generated by another audio device of the plurality of audio devices in the environment; determining an interior angle of each of a plurality of triangles based on the direction of arrival data, each triangle of the plurality of triangles having vertices corresponding to locations of three audio devices of the plurality of audio devices; determining the length of each side of each of the triangles based on the interior angles and the signals generated by the audio devices separated by the side lengths to be determined; or determining the side length based on the interior angle, wherein a side length of one of the triangles is set to a predetermined value; performing a forward alignment process to align each of the plurality of triangles in a first order to generate a forward alignment matrix, wherein the forward alignment process is performed by forcing a side length of each triangle to be consistent with a side length of an adjacent triangle and using the interior angles determined for the adjacent triangles; performing a reverse alignment process of aligning each of the plurality of triangles to generate a reverse alignment matrix, wherein the reverse alignment process is performed in a manner similar to the forward alignment process but in a second order that is opposite to the first order; and A final estimate of the position of each audio device is generated based at least in part on the values ​​of the forward alignment matrix and the values ​​of the reverse alignment matrix.

2. The method of claim 1, wherein: Generating the final estimate of the position of each audio device comprises: translating and scaling the forward alignment matrix to produce a translated and scaled forward alignment matrix; and The reverse alignment matrix is ​​translated and scaled to produce a translated and scaled reverse alignment matrix, wherein translating and scaling the forward alignment matrix and the reverse alignment matrix includes moving the centroid of the respective matrices to an origin and forcing the Frobenius norm of each matrix to be one.

3. The method of claim 2, wherein: Generating the final estimate of each audio device position further comprises generating an additional matrix based on the translated and scaled forward alignment matrix and the translated and scaled back alignment matrix, the additional matrix comprising a plurality of estimated audio device positions for each audio device.

4. The method of claim 3, wherein: Generating the further matrix includes performing a singular value decomposition on the translated and scaled forward alignment matrix and the translated and scaled reverse alignment matrix.

5. The method of claim 1, wherein: Generating the final estimate of the position of each audio device further comprises averaging multiple estimates of the position of the audio device obtained from overlapping vertices of multiple triangles.

6. The method according to any one of claims 1 to 5, wherein: Determining the side length involves: determining a first length of a first side of the triangle; and Determining lengths of a second side and a third side of the triangle based on the interior angle of the triangle, wherein determining the first length involves setting the first length to a predetermined value, or wherein determining the first length is based on at least one of arrival time data or received signal strength data.

7. The method according to any one of claims 1 to 5, wherein: Each audio device includes multiple audio device microphones, and wherein determining the arrival direction data involves receiving microphone data from each of the multiple audio device microphones corresponding to a single audio device among the multiple audio devices, and determining the arrival direction data for the single audio device based at least in part on the microphone data.

8. The method according to any one of claims 1 to 5, wherein: Each audio device comprises one or more antennas, and wherein determining the direction of arrival data involves receiving antenna data from one or more antennas corresponding to a single audio device of the plurality of audio devices, and determining the direction of arrival data for the single audio device based at least in part on the antenna data.

9. The method of any one of claims 1 to 5, further comprising controlling at least one of the audio devices based at least in part on the final estimate of at least one audio device position.

10. The method of claim 9, wherein: Each audio device of the plurality of audio devices comprises a loudspeaker, and wherein controlling at least one of the audio devices involves controlling the loudspeaker of the at least one of the audio devices.

11. A device for automatically positioning an audio device, the device comprising: Interface system; and A control system configured to perform the method according to any one of claims 1 to 10.

12. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 10.

13. A computer-readable medium comprising the computer program product of claim 12.

Images