Techniques for estimating room boundaries and layouts using microphone pairs

Abstract

In various embodiments, a computer-implemented method includes: emitting a reference audio signal using one or more speakers of an audio processing device located in an acoustic environment; receiving, by each microphone of a microphone pair of the audio processing device, a response of the acoustic environment to the reference audio signal; determining an impulse response of the acoustic environment based on the reference audio signal and the received response of the acoustic environment; determining a plurality of candidate wall distance estimates and a plurality of confidence scores based on the impulse response; and determining a set of wall distance estimates based on a plurality of candidate wall distance estimates and a plurality of confidence scores, including candidate wall distance estimates and confidence scores from one or more other audio processing devices, each wall distance estimate of the set of wall distance estimates is associated with a different wall of the plurality of walls.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application Serial No. 63 / 582,464, filed on September 13, 2023, entitled “BOUNDARY ESTIMATION USING MICROPHONE PAIRS” and U.S. Provisional Patent Application Serial No. 63 / 582,467, filed on September 13, 2023, entitled “ROOM LAYOUT ESTIMATION USING MICROPHONE PAIRS”. The subject matter of these related applications is hereby incorporated by reference herein. Technical Field

[0003] Various embodiments relate generally to audio systems and, more particularly, to techniques for estimating room boundaries and layout using microphone pairs. Background Art

[0004] The layout of a room, such as the size and shape of the room, can significantly affect the acoustics of the room. Knowing the size and shape of the room can help the audio system provide better sound quality to the listener. The first step for an audio system to determine the layout of a room is to estimate the wall boundaries of the room. However, conventional audio systems cannot accurately determine the wall boundaries of a room without using special measurement hardware, and therefore cannot accurately determine the layout of a room.

[0005] One method for determining the wall boundaries of a room is to perform a sine sweep test. A sine sweep is an audio signal that gradually changes frequency over time and can be used to identify the resonant frequencies of an object or structure. A device with a speaker and a microphone can broadcast a sine sweep in the room and capture impulse responses from the walls of the room. These impulse responses can be measured to estimate the distance from the device to the wall that generated the reflection captured in the response. However, this technique relies on the relative position of the device in the room. For example, if the device is placed too close to one or more walls, the accuracy of the estimate will be affected. In addition, because the technique uses a single device with a speaker and a microphone, the impulse response from one wall may interfere with another wall. Conventional audio systems need to compensate for this interference to determine the boundaries of other walls, which is difficult and inaccurate. Therefore, an audio system with a single device using a speaker and a microphone cannot accurately estimate the wall boundaries.

[0006] Another method for determining the boundaries of a room is to require that the audio system be equipped with one or more cameras. The audio system can use the camera to capture a picture or video of each wall relative to the location of the camera. The audio system can then estimate the wall boundaries based on the placement of the camera. However, not all audio systems are equipped with one or more cameras or the ability to estimate wall boundaries based on pictures or videos.

[0007] One drawback of both of these approaches is that even if the audio system successfully estimates the wall boundaries of the room, the audio system does not know how to use the wall boundaries to estimate the layout of the room. Without a room layout estimate, the audio system cannot determine the acoustics of the room and therefore cannot adapt to produce sound quality that will match the acoustics.

[0008] As indicated above, there is a need in the art for more efficient techniques for audio systems to estimate wall boundaries and room layouts. Summary of the invention

[0009] In various embodiments, a computer-implemented method includes: receiving, by each microphone of a microphone pair of an audio processing device, a response of an acoustic environment to a reference audio signal; determining an impulse response of the acoustic environment based on the reference audio signal and the received response of the acoustic environment; determining a plurality of candidate wall distance estimates and a plurality of confidence scores based on the impulse response, each confidence score in the plurality of confidence scores being associated with a different candidate wall distance estimate in the plurality of candidate wall distance estimates; and determining a set of wall distance estimates based on the plurality of candidate wall distance estimates, the plurality of confidence scores, and the candidate wall distance estimates and confidence scores received from one or more other audio processing devices, each wall distance estimate in the set of wall distance estimates being associated with a different wall in the plurality of walls in the acoustic environment.

[0010] At least one technical advantage of the disclosed technology over the prior art is that, using the disclosed technology, an audio system can accurately estimate the wall boundaries of a room with minimal setup. The disclosed technology is not affected by the relative positions of the devices of the audio system in the room, thereby allowing more flexible placement of the devices to estimate the boundaries and layout of the room. In addition, the audio system does not need to be equipped with a camera or image processing software. In addition, the estimated room layout produces a better room acoustic model and correspondingly improves the sound quality output by the audio system. These technical advantages provide one or more technical improvements over prior art methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order that the manner in which the above-mentioned features of various embodiments can be understood in detail, the inventive concept briefly summarized above may be described in more detail by reference to various embodiments, some of which are shown in the accompanying drawings. It should be noted, however, that the drawings illustrate only typical embodiments of the inventive concept and are therefore not to be considered as limiting its scope in any way, and that other equally effective embodiments exist.

[0012] Figure 1 is a conceptual illustration of an apparatus configured to implement one or more aspects of various embodiments;

[0013] Figure 2 is a conceptual illustration of an example acoustic environment according to various embodiments;

[0014] Figure 3 is a conceptual illustration of an example table detailing wall directions based on device orientation according to various embodiments;

[0015] Figure 4 is a conceptual illustration of an example real room layout matrix according to various embodiments;

[0016] Figure 5 is a conceptual illustration of an example ordered room layout matrix according to various embodiments;

[0017] Figure 6 is a conceptual illustration of an example final room layout matrix according to various embodiments;

[0018] Figure 7 is a conceptual illustration of an example table describing the distance of each wall from an example ordinal device and a graph mapping each wall surrounding the device in an example acoustic environment, according to various embodiments;

[0019] Figure 8 is a flow chart of method steps for determining a wall distance estimate for an acoustic environment according to various embodiments; and

[0020] Fig. 9 is a flow chart of method steps for determining a room layout estimate of an acoustic environment according to various embodiments. DETAILED DESCRIPTION

[0021] In the following description, numerous specific details are set forth to provide a more thorough understanding of various embodiments. However, it will be apparent to one skilled in the art that these inventive concepts may be practiced without one or more of these specific details.

[0022] System Overview

[0023] Figure 11 is a schematic diagram illustrating an audio processing device 100 configured to implement one or more aspects of the present disclosure according to various embodiments. In various embodiments, the audio processing device 100 is a central unit in a home theater system, a sound bar, a vehicle system, etc. In some embodiments, the audio processing device 100 is included in a network of multiple audio processing devices 100 that communicate with each other to perform the techniques disclosed herein. In some embodiments, the audio processing device 100 is included in one or more devices, such as consumer products ( For example , portable speakers, games, etc.), vehicles ( For example , head unit of a car, truck, van, etc.), smart home devices ( For example , smart lighting systems, security systems, digital assistants, etc.), communication systems ( For example , telephone conference system, video conference system, speaker amplification system, etc.) etc. In various embodiments, the audio processing device 100 is located in various acoustic environments, including but not limited to indoor environments ( For example , living room, meeting room, conference room, home office, etc.) and / or outdoor environment ( For example , terrace, roof, garden, etc.). As shown, the audio processing device 100 includes, but is not limited to, one or more speakers 160, a microphone pair 150, and a computing device 110. In addition, the computing device 110 includes, but is not limited to, a processing unit 112 and a memory 114. The memory 114 stores various data, including, but not limited to, an audio processing application 120, device location data 122, confidence score data 124, wall distance estimates (WDE) 126, WDE confidence pairs 128, and a room layout matrix (RLM) 130.

[0024] The audio processing device 100 includes one or more speakers 160, which can broadcast or otherwise transmit one or more reference audio signals, such as sine sweeps, in an acoustic environment. In some embodiments, one or more speakers 160 may include, but are not limited to, a sound bar with multiple speaker units, one or more loudspeakers, or any other speaker system that reproduces one or more reference audio signals by generating sound waves. In some embodiments, one or more speakers 160 can play audio signals to test the acoustic effects of an acoustic environment. In some embodiments, one or more reference audio signals may be sine sweeps. Sine sweeps are reference tones used to check the adverse effects of frequency response or acoustic effects in an acoustic environment. In some embodiments, the frequency of one or more reference audio signals may be between 20 Hz and 20 kHz. However, any frequency capable of generating sine sweeps or similar audio signals is within the scope of the disclosed embodiments. One or more speakers 160 may include multiple element instances, even if not shown, still within the scope of the disclosed embodiments.

[0025] Microphone pair 150( For example , microphone 150(1) and microphone 150(2), in Figure 1 The microphone pair 150 may receive or otherwise acquire sound data associated with audio in the acoustic environment of the audio processing device 100 (not shown). The received audio may include reflected audio signals associated with sound emitted by one or more speakers 160. The microphone pair 150 may include a plurality of types of sound ( For example , subsonic pulses, ultrasonic waves, voice commands, etc.) to determine various types of sensors about the acoustic environment. In some embodiments, the audio processing device 100 includes other types of sensors (not shown) in addition to the microphone to obtain information about the acoustic environment. Other types of sensors include optical sensors (such as RGB cameras, time-of-flight cameras, infrared cameras, depth cameras, quick response (QR) code tracking systems), motion sensors (such as accelerometers or inertial measurement units (IMUs) ( For example , three-axis accelerometer, gyroscope sensor and / or magnetometer)), pressure sensor, etc. In addition, in some embodiments, the sensor 150 may include a wireless sensor (including a radio frequency (RF) sensor ( example like , sonar and radar)) and / or wireless communication protocols including Bluetooth, Bluetooth Low Energy (BLE), cellular protocols and / or Near Field Communication (NFC).

[0026] Processing unit 112 may be any suitable processor, such as a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), and / or any other type of processing unit or combination of different processing units, such as a CPU configured to operate in conjunction with a GPU. In general, processing unit 112 may be any technically feasible hardware unit capable of processing data and / or executing software applications.

[0027] The memory 114 may be one or more of readily available memories, such as random access memory (RAM) modules, read-only memory, floppy disks, hard disks, or any other form of digital storage (local or remote) or a combination thereof. The processing unit 112 is configured to read data from and write data to the memory 114. In various embodiments, the memory 114 includes non-volatile memory, such as an optical drive, a magnetic drive, a flash drive, or other storage device. In some embodiments, a separate data repository (such as an external data repository) included in a network ("cloud storage device") may supplement the memory 114. The audio processing application 120 within the memory 114 may be executed by the processing unit 112 to implement the overall functionality of the computing device 110, and thus coordinate the operation of the audio processing device 100 as a whole. In various embodiments, an interconnect bus (not shown) connects the processing unit 112, the memory 114, the speaker 160, the sensor 150, and any other components of the computing device 110. The memory 114 stores, but is not limited to, an audio processing application 120 , position data 122 , impulse response data 124 , and wall distance estimates 126 .

[0028] The computing device 110 is configured to execute one or more software applications. In some alternative embodiments, each of any number of instances of the computing device 110 may include any number of instances of the processor 112 and any number of instances of the memory 114 in any combination. Specifically, any number of instances of the computing device 110 (including one instance) may provide a multi-processing environment in any technically feasible manner. For explanation purposes only, each software application is described as residing in the memory 114 of the computing device 110 and executing on the processor 112 of the computing device 110. However, in some embodiments, the functions of any number of software applications may be distributed among instances of the memory 114 of any number of instances of the computing device 110 and executed on any number of other software applications in any combination on instances of the processor 112 of any number of instances of the computing device 110. In addition, the functions of any number of software applications may be merged into a single software application (such as the audio processing application 120).

[0029] As shown, the computing device 110 is configured to execute an audio processing application 120 that implements one or more optimization techniques to accurately determine a wall distance estimate (WDE) for each wall in the acoustic environment and one or more fusion algorithms to accurately determine a room layout (WDE) of the acoustic environment. For example , determine the length, width, and height of the acoustic environment). As previously described herein, the room layout of an acoustic environment, such as the size and shape of the acoustic environment, can significantly affect the acoustic effect. Knowing the size and shape of the acoustic environment can help the audio system provide better sound quality. However, conventional methods for determining the room layout of an acoustic environment are not practical or accurate.

[0030] The audio processing application 120 resides in the memory 114 of the computing device 110 and executes on the processor 112 of the computing device 110. More generally, the functionality of the audio processing application 120 may be distributed across any number of software applications. Each of the software applications may reside in any number of instances of the memory 114 of any number of instances of the computing device 110 and execute in any combination on any number of instances of the processor 112 of any number of instances of the computing device 110. For purposes of explanation, multiple instances of the same object are represented, where necessary, by a reference number identifying the object and an alphanumeric character in parentheses identifying the instance.

[0031] Generate wall distance estimates

[0032] In operation, the audio processing application 120 performs various techniques to determine the exact WDE of each wall in the acoustic environment. In various embodiments, the audio processing application instructs the speaker 160 to broadcast or otherwise transmit a reference audio signal within the acoustic environment. In some embodiments, the reference audio signal is a sine sweep, but any reference audio signal that is reflected from the walls of the acoustic environment and can then be received by the microphone pair 150 can be broadcast or otherwise transmitted. Each microphone in the microphone pair 150 can acquire or otherwise receive a reflected audio signal from an obstacle in the acoustic environment. The audio processing application 120 combines the reflected audio signal with the reference audio signal to generate an impulse response. The impulse response is the reaction of the acoustic environment to the reference audio signal, expressed as a function of time.

[0033] The audio processing application 120 generates a WDE, such as the wall distance estimate 126, using the impulse response to generate an estimated distance from a wall of the acoustic environment to a microphone of the microphone pair 150, which is a function of the speed of sound and the time associated with the peak amplitude of the impulse response ( For example, due to the round trip time for the reference audio signal to travel from the speaker 160 to the wall, reflect from the wall, and then return to the microphone). Each WDE is associated with a microphone used to calculate the distance and is associated with the audio processing device 100 that includes the microphone. Each WDE can be expressed as a unit of measurement, such as meters, feet, or any other measurement metric. The WDEs 126 can be stored in the memory 114 of the audio processing device 100. In some embodiments, the WDEs 126 may include other WDEs derived from other audio processing devices 100 in the acoustic environment that are sent to the audio processing device 100.

[0034] The audio processing application 120 uses the amplitude of the peak from the impulse response data to calculate a confidence score to rank the corresponding WDE 126. The confidence score is a measure of the likelihood that the corresponding WDE accurately represents the actual distance from the audio processing device to the wall of the acoustic environment. The confidence score can be calculated based on the peak amplitude of the impulse response peak. For example, a confidence score of 1.0 represents the highest confidence, while a confidence score of 0.0 represents the lowest confidence. The confidence score can be stored in the memory 114 of the audio processing device 100 as confidence score data 124. The confidence score data 124 may include confidence scores of WDEs in other audio processing devices 100 in the acoustic environment. The audio processing application 120 can generate WDE confidence pairs 128 by associating each WDE in the WDE 126 with a corresponding confidence score in the confidence score data 124.

[0035] The audio processing application 120 executes one or more rule-based optimization algorithms that combine multiple WDEs in the WDEs 126 of multiple microphone pairs 150 from multiple audio processing devices 100 to derive a valid set of WDEs representing each wall (including ceiling and floor) in the acoustic environment. The audio processing application 120 removes global outliers from the WDEs 126 based on various adjustable thresholds and confidence score data 124. For example, if a given WDE 126 is less than a minimum wall distance threshold or greater than a maximum wall distance threshold, the given WDE is removed. In another example, if a given WDE 126 is associated with a confidence score less than a minimum confidence threshold, the given WDE 126 is removed from the WDEs 126.

[0036] The one or more rule-based optimization algorithms may include a clustering algorithm. One example of clustering is clustering based on the associated microphone pairs WDEs 126 in the microphone pairs 150 of each audio processing device 100. For example, if the distance difference between two or more WDEs associated with the same microphone in the microphone pair 150 is less than a minimum distance threshold, each WDE except the WDE associated with the highest confidence score is removed from the WDEs 126. In some embodiments, the confidence scores associated with the remaining WDEs 126 may be averaged with the other confidence scores of the removed WDEs. The minimum distance threshold may be a predefined threshold. Another example of clustering is clustering based on the associated microphone pairs WDEs 126 in the microphone pairs 150 of the same audio processing device 100. For example, if the distance difference between two or more WDEs 126 associated with different microphones of the microphone pair 150 in the same audio processing device 100 is less than a minimum distance threshold, the WDE is determined to be a "valid estimate".

[0037] The one or more rule-based optimization algorithms may include identifying and extracting a ceiling estimate and / or a floor estimate of the acoustic environment from the WDE 126. One example of identifying and extracting a ceiling estimate and / or a floor estimate is calculating the mode of the WDE 126. For example, the audio processing application 120 calculates the mode of the WDE 126. If the associated confidence score of each WDE equal to the mode is greater than a minimum confidence threshold, then one of the WDEs of the mode may be identified as a "ceiling estimate" or a "floor estimate" and the remaining WDEs may be removed. The minimum confidence threshold may be a predefined threshold.

[0038] After the above steps, the audio processing application 120 combines the remaining WDEs associated with the two microphones in the microphone pair 150 whose separation distance is less than the minimum distance threshold in a specific order other than the confidence score. These combined or unified WDEs accurately represent each wall of the acoustic environment from the perspective of the associated audio processing device 100. The above steps may be implemented in each audio processing device 100 in the acoustic environment to generate a unified WDE that accurately represents each wall of the acoustic environment from the perspective of each audio processing device 100 in the acoustic environment.

[0039] Generate room layout estimates

[0040] The audio processing application 120 performs various techniques to determine the room layout of the acoustic environment. The audio processing application 120 determines an ordinal device in the audio processing device 100 in the acoustic environment. An ordinal device is defined as a reference device, where the center of the device is considered the origin of the room layout of the acoustic environment.

[0041] The audio processing application 120 uses a device layout estimation algorithm to determine the device distance estimate. For example, two audio processing devices 100 (each device includes at least one speaker 160) in an acoustic environment may each transmit an audio signal including different characteristic tones into the acoustic environment. The audio processing application 120 of one of the audio processing devices may instruct the speaker 160 to use one or more microphones to receive the two audio signals. The audio processing application 120 determines the estimated distance between the audio processing devices based on the different characteristic tones and time-frequency masking operations. The device distance estimate may be stored in the memory 114 of the audio processing device 100 as device location data 122. The device location data 122 includes location information of one or more locations within the listening environment, such as the location of each device. In some embodiments, the device location data 122 includes specific coordinates relative to a reference point. For example, the device location data 122 may store the current position and / or orientation of each corresponding audio processing device 100 in the acoustic environment as a distance and angle from a specific reference point. In some embodiments, the device location data 122 includes additional orientation information, such as a set of angles relative to a reference orientation ( For example , In such an example, the position and orientation of a given audio processing device 100 and / or microphone in a microphone pair 150 is stored as a set of distances and angles relative to a reference point in the device position data 122. The device layout estimation algorithm is described in more detail in U.S. patent application Ser. No. 18 / 860,604, filed May 31, 2024, which is incorporated herein by reference.

[0042] Reference Figure 2 The acoustic environment in 200 together with Figures 3 to 7 The examples of room layout matrices, tables, and graphs in further describe the audio processing application 120. However, it should be understood that the described techniques are also applicable to other acoustic environments having different layouts, sizes, and / or different numbers of audio processing devices.

[0043] Device setup in an example acoustic environment

[0044] Figure 2 2 is a conceptual illustration of an example acoustic environment 200 according to various embodiments. As shown, the acoustic environment 200 has four walls (not including the ceiling and floor) represented as walls 202a to 202d. Audio processing devices 204a to 204d (which may be Figure 1 The example of the audio processing device 100 in FIG. 1 is arranged in the acoustic environment 200 to determine the wall distance estimation values ​​of the walls 202a to 202d and the room layout estimation value of the acoustic environment 200. In order to accurately determine the wall distance estimation values ​​and the room layout estimation value, the audio processing devices 204a to 204d should face at least two different walls. For example, Figure 2 As shown, audio processing devices 204a and 204b face wall 202a, while audio processing devices 204c and 204d face wall 202c. In addition, in order to accurately determine the wall distance estimate and the room layout estimate, audio processing devices 204a to 204d should be at the same height in the acoustic environment, such as 1 meter above the floor. Other arrangements are also possible, and Figure 2 The arrangements shown in are not meant to be limiting.

[0045] Example of determining room layout estimates

[0046] Return to reference Figure 1 , now use Figures 2 to 7 The audio processing application 120 is described by way of example in FIG. The audio processing application 120 determines the orientation of each wall within the acoustic environment relative to each audio processing device 100 in the acoustic environment. For example, Figure 3 Table 300 shows that based on Figure 2 The column for wall 202a shows the orientation of wall 202a relative to audio processing devices 204a, 204b, 204c, and 204d. The other columns of table 300 show Figure 2 The other walls in the use case acoustic environment 200 of the audio processing device are relative to the orientation of the audio processing devices 204a, 204b, 204c, and 204d. The term "front" means that the device faces the relevant wall. The term "rear" means that the device faces away from the relevant wall. The term "left" means that the wall is located in the left direction of the device. The term "right" means that the wall is located in the right direction of the device.

[0047] Figure 3 is a conceptual illustration of an example table detailing wall directions based on device orientation according to various embodiments. As shown, table 300 shows Figure 2 The direction of each wall in the acoustic environment 200 relative to each device in the acoustic environment 200. The audio processing application 120 can use the device position data 122 to determine the orientation of each wall relative to each device.

[0048] As shown, audio processing device 204a is labeled as an ordinal device in acoustic environment 200. Wall 202a is a front wall relative to audio processing devices 204a and 204b, and a rear wall relative to audio processing devices 204c and 204d. The audio processing application 120 of audio processing device 204a determines that wall 202b is a left wall relative to audio processing devices 204a and 204b, and a right wall relative to audio processing devices 204c and 204d. The audio processing application 120 of audio processing device 204a determines that wall 202c is a rear wall relative to audio processing devices 204a and 204b, and a front wall relative to audio processing devices 204c and 204d. The audio processing application 120 of audio processing device 204a determines that wall 202d is a right wall relative to audio processing devices 204a and 204b, and a left wall relative to audio processing devices 204c and 204d.

[0049] Figure 4 4 is a conceptual illustration of an example real room layout matrix according to various embodiments. As shown, matrices 400a and 400b are the true values ​​of the distance and confidence score of each wall relative to each audio processing device 204a to 204d in the acoustic environment 200, respectively. The values ​​in matrix 400a are the true distances (in meters (m)) from each corresponding wall to each corresponding device. The values ​​in matrix 400b are all equal to 1.0 because the confidence score of the true distance is always 1.0.

[0050] Return to reference Figure 1 , the audio processing application 120 generates a room layout matrix (RLM) 130 and arranges the WDE 126 of each audio processing device 100 in the acoustic environment in descending order of confidence scores in the RLM 130. In various embodiments, only the first n confidence scores of each audio processing device 100 are used in the RLM 130, where n is equal to the number of non-ceiling, non-floor walls in the acoustic environment. For example, in an acoustic environment with four non-ceiling, non-floor walls, the first four confidence scores are used to populate the RLM 130. In some embodiments, if the audio processing application 120 determines that all rows or all columns of the RLM 130 are zero, the audio processing application 120 issues an error flag. Figure 5 RLMs 500a and 500b in FIG. 130 show examples of ordered RLMs for four devices in an acoustic environment with four non-ceiling, non-floor walls. The labels for the wall columns in RLM 130 begin with the placeholder labels walls W, X, Y, and Z. The following techniques help disambiguate which walls of the acoustic environment should be associated with which WDE.

[0051] Figure 5is a conceptual illustration of an example ordered room layout matrix according to various embodiments. RLM500a represents the calculated WDE 126 for each audio processing device 204a to 204d in the acoustic environment 200, sorted in descending order based on the associated confidence score 124. The columns of RLM500a include placeholder labels Wall W, X, Y, and Z until the audio processing application 120 determines the wall associated with each WDE. RLM500b represents the calculated confidence scores associated with the WDE 126 in RLM500a, which are also sorted in descending order for each device. For exemplary purposes only, the values ​​in RLM500a are expressed in meters and are not meant to be limiting in any way. Any unit of measurement is acceptable. The values ​​in RLM500b are expressed as a range between 0.0 and 1.0, where 0.0 is the lowest possible confidence score and 1.0 is the highest possible confidence score. The audio processing device 204 a uses the values ​​in the RLMs 500 a and 500 b to determine an estimate of the room layout of the acoustic environment 200 .

[0052] Return to reference Figure 1 , the audio processing application 120 determines that one or more audio processing devices in the RLM 130 have a different orientation than the ordinal device. If the audio processing application 120 determines that a device has a different orientation than the ordinal device, the audio processing application 120 cyclically shifts the values ​​in the row for that device one position to the right for every 90 degrees that the orientation differs from the ordinal device. For example, in RLM 600a, the audio processing application 120 determines based on the device position data 122 that the audio processing device 204c is rotated 180 degrees relative to the audio processing device 204a (the ordinal device). In this example, the audio processing device shifts the values ​​in the row associated with the device 204c two positions to the right ( For example , and shift one position to the right for every 90 degree rotation difference).

[0053] Once each row correctly represents the correct wall relative to the orientation of each device, the audio processing application 120 reorders the values ​​in each row of the RLM based on the associated confidence score. For example, with respect to the orientation in table 300, the WDE associated with the highest confidence score is placed in the front wall column of each device. The WDE associated with the lowest confidence score is placed in the rear wall column of each device. The WDE associated with the second highest confidence score is placed in the left wall column of devices 204a and 204c and in the right wall column of devices 204b and 204d. The WDE associated with the third highest confidence score is placed in the right wall column of devices 204a and 204c and in the left wall column of devices 204b and 204d.

[0054] In addition, the audio processing application 120 removes certain outliers from the matrix 600a based on the outlier determination techniques disclosed herein. For example, as shown in the acoustic environment 200, the audio processing device 204c is located between the audio processing device 204d and the wall 202d. If the distance between the audio processing devices 204d and 204c is 2m, then the distance between the wall 202d and the audio processing device 204a is physically impossible to be less than 2m. Therefore, the audio processing application 120 can use the device location data 122 to determine that the WDE with a value of 2m associated with the audio processing device 204d in the RLM500a (which is associated with a confidence of 0.72 in the RLM500d) cannot be associated with the wall 202d. The audio processing application 120 can use the device location data 122 relative to the audio processing device 204a to calculate a similar analysis for the wall 202a. Because the WDE with a value of 2m is associated with a lower confidence score than the other WDEs associated with audio processing device 204d, and cannot be associated with wall 202a or wall 202d, the audio processing application identifies the WDE as an outlier and removes it from the RLM.

[0055] Once the outliers are removed from the RLM 130, the audio processing application determines the final WDE for each wall for the orientation of the ordinal device. For example, from the perspective of the ordinal audio processing device 204a, wall 202a is the front wall, wall 202b is the left wall, wall 202c is the back wall, and wall 202d is the right wall. To determine the final WDE for each wall, the audio processing application 120 determines the WDE with the highest confidence score in each wall column. For example, for the column in RLM 600a representing wall 202a, the audio processing application 120 determines that the WDE associated with the highest confidence score is associated with the audio processing device 204a. Because the audio processing device 204a is an ordinal device, the WDE of 3.9m is the final distance and no additional calculations need to be performed. In another example, for the column in RLM 600a representing wall 202d, the audio processing application 120 determines that a WDE of 1.0m is associated with the highest confidence score of 0.68, but is associated with a non-ordinal device. Because the WDE of 1.0 m and the confidence score of 0.68 are associated with audio processing device 204 b and not with the ordinal device (audio processing device 204 a), the audio processing application 120 uses the device position data 122 to determine the final WDE. For example, if audio processing device 204 b is 1.0 meter away from the ordinal device in the direction toward wall 202 d, and audio processing device 204 b is closer to wall 202 d than the ordinal device, as shown in acoustic environment 200, the audio processing application 120 calculates the final distance value of wall 202 d by adding the WDE of 1.0 m to the distance between audio processing device 204 b and the ordinal device in the direction of wall 202 d. In some embodiments, the distance between the two devices in the direction of the wall is zero. For example, for the column in RLM 600 a representing wall 202 b, the audio processing application 120 determines that a WDE of 1.0 m is associated with the highest confidence score of 0.94, but is associated with a non-ordinal device. Because the WDE of 1.0 m and the confidence score of 0.94 are associated with the audio processing device 204 d and not the ordinal device (audio processing device 204 a), the audio processing application 120 uses the device position data 122 to determine the final WDE. For example, as shown in the acoustic environment 200, the audio processing device 204 d is equidistant from the ordinal device with respect to the wall 202 b. Therefore, the audio processing application 120 calculates the final WDE for the wall 202 b by subtracting or adding 0.0 m from the WDE of 1.0 m. This process may be repeated for each wall in the acoustic environment.

[0056] Figure 6is a conceptual illustration of an example final room layout matrix according to various embodiments. RLM 600a shows the orientation of each wall relative to each device, and the WDE 126 associated with each wall relative to each device. The values ​​in each RLM 600a and 600b are based on the orientation of each wall by the audio processing application 120 of the ordinal device. Figure 5 For example, the audio processing application 120 moves the WDE 126 in the RLM 500a based on the orientation of each wall relative to each device, such as Figure 3 300. In addition, the audio processing application 120 then reorders the WDEs in each row based on the associated confidence scores in RLM500b. In addition, the audio processing application 120 removes certain outliers from the matrix 600a based on the outlier determination techniques disclosed herein. For example, as shown in the acoustic environment 200, the audio processing device 204c is located between the audio processing device 204d and the wall 202d. If the distance between the audio processing devices 204d and 204c is 2 meters, then the distance between the wall 202d and the audio processing device 204a is physically impossible to be less than 2 meters. Therefore, the audio processing application 120 can use the device location data 122 to determine that the WDE associated with the audio processing device 204d and the wall 202d in RLM500a with a value of 2 meters (which is associated with a confidence score of 0.72 in RLM500d) cannot be used to estimate the distance to the wall 202d. The audio processing application 120 can calculate a similar analysis for wall 202a using the device location data 122 relative to the audio processing device 204a. Because the WDE with a value of 2 meters is associated with a lower confidence score than the other WDEs associated with the audio processing device 204d, and cannot be associated with either wall 202a or wall 202d, the audio processing application identifies the WDE as an outlier and removes it from the RLMs 600a and 600b.

[0057] Return to reference Figure 1Once the final WDE relative to the ordinal device is calculated for each wall, the audio processing application 120 determines room layout estimates of the acoustic environment, such as the length, width, height, and volume of the acoustic environment. The audio processing application 120 determines the length of the acoustic environment by adding the distance from the ordinal device to the front wall and the distance from the ordinal device to the rear wall. The audio processing application 120 determines the width of the acoustic environment by adding the distance from the ordinal device to the left wall and the distance from the ordinal device to the right wall. The audio processing application 120 determines the height of the acoustic environment by adding the WDE associated with the ceiling and the distance of the ordinal device from the ground, which is determined based on the device layout data 122. The audio processing application 120 determines the volume of the acoustic environment by multiplying the length of the acoustic environment by the width of the acoustic environment and the height of the acoustic environment.

[0058] Figure 7 700a shows the final WDE of each wall relative to the ordinal device based on the WDE associated with the highest confidence score in RLM600a and 600b across all devices. For example, for the column in RLM600a representing wall 202a, the audio processing application 120 determines that the WDE associated with the highest confidence score is associated with audio processing device 204a. Because audio processing device 204a is an ordinal device, the WDE of 3.9m is the final distance and no other calculations are required. In another example, for the column in RLM600a representing wall 202d, the audio processing application 120 determines that a WDE of 1.0m is associated with the highest confidence score of 0.68, but is associated with a non-ordinal device. Because the WDE of 1.0 m and the confidence score of 0.68 are associated with audio processing device 204 b and not with the ordinal device (audio processing device 204 a), the audio processing application 120 uses the device location data 122 to determine the final WDE. For example, if audio processing device 204 b is 1.0 meter away from the ordinal device in the direction toward wall 202 d, and audio processing device 204 b is closer to wall 202 d than the ordinal device, as shown in acoustic environment 200, the audio processing application 120 calculates a final distance value for wall 202 d by adding the WDE of 1.0 m to the distance between audio processing device 204 b and the ordinal device in the direction of wall 202 d. This process is repeated for each wall in the acoustic environment. Graph 700 b shows an example of a final WDE estimate relative to an ordinal device.

[0059] Once the final WDE is calculated, the audio processing application 120 determines the room layout estimates based on the final wall estimates in table 700a. For example, the audio processing application 120 determines the room length estimate as 5.9 meters based on adding the front wall estimate of 3.9 meters to the rear wall estimate of 2.0 meters (4.0 meters minus the distance between the audio processing devices 204d and 204a in the direction of the rear wall). The audio processing application 120 determines the room width estimate as 4.1 meters based on adding the left wall estimate of 1.0 meters (1.0 meters minus the distance between the audio processing devices 204d and 204a in the direction of the left wall) to the right wall estimate of 3.1 meters (1.0 meters plus the distance between the audio processing devices 204b and 204a in the direction of the right wall). The audio processing application 120 determines the room height estimate as 2.8 meters based on adding the ceiling distance estimate of 1.8 meters to the floor distance estimate of 1 meter. The values ​​of the ceiling estimate and the height are based on extracting the ceiling estimate and the floor estimate when determining the valid set of WDEs 126. The audio processing application 120 determines the estimated volume of the room to be 67.73 m based on multiplying the estimated length of the room 5.9 m by the estimated width of the room 4.1 m by the estimated height of the room 2.8 m.

[0060] Method for determining wall distance estimate

[0061] Figure 8 A flow chart illustrating the steps of a method for determining an estimate of the distance to a wall of an acoustic environment. Figures 1 to 7 The method steps are described with reference to a system, but those skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the present invention.

[0062] As shown, method 800 begins at step 802, where the audio processing application 120 of each audio processing device 100 in the acoustic environment emits a reference audio signal using one or more speakers 160. In some embodiments, the reference audio signal can be a sine sweep, or any other reference audio signal that is reflected from the walls of the acoustic environment and can then be received by the microphone pair 150 as an impulse response.

[0063] At step 804, each microphone in the microphone pair 150 of each audio processing device 100 receives a reflected audio signal from the acoustic environment. The audio processing application 120 determines an impulse response based on the reflected audio signal. The impulse response represents the reaction of the acoustic environment to the reference audio signal, expressed as a time function. In some embodiments, in response to emitting the reference audio signal, the microphones in the microphone pair 150 may be instructed to listen to the reflected audio signal through the audio processing application 120 of the audio processing device 100.

[0064] At step 806, the audio processing application 120 of each audio processing device 100 calculates a WDE based on each impulse response received by the microphone of the microphone pair 150 of the audio processing device 100. The WDE is stored in each audio processing device 100 as a WDE 126. The audio processing application 120 generates the WDE 126 by generating an estimated peak distance from the wall of the acoustic environment to the microphone of the microphone pair 150 based on the speed of sound and the peak amplitude of the impulse response. Each WDE is associated with the microphone used to calculate the peak distance and is associated with the audio processing device 100 that includes the microphone. Each WDE may be expressed as a unit of measurement, such as meters, feet, or any other measurement. The WDE 126 may be stored in the memory 114 of the audio processing device 100. In some embodiments, the WDE 126 may include other WDEs derived from other audio processing devices 100 in the acoustic environment that are sent to the audio processing device 100.

[0065] At step 808, the audio processing application 120 of each audio processing device 100 calculates a confidence score for each WDE. The confidence score is a measure of the likelihood that the corresponding WDE accurately represents the actual distance from the audio processing device to the wall of the acoustic environment. The confidence score may be calculated based on the peak amplitude of the impulse response peak. For example, a confidence score of 1.0 represents the highest confidence, while a confidence score of 0.0 represents the lowest confidence. The confidence scores may be stored in the memory 114 of the audio processing device 100 as confidence score data 124. The confidence score data 124 may include confidence scores of WDEs in other audio processing devices 100 in the acoustic environment. The audio processing application 120 may generate WDE confidence pairs 128 by associating each WDE in the WDE 126 with a corresponding confidence score in the confidence score data 124.

[0066] At step 810, the audio processing application 120 removes global outliers from the WDEs 126 based on various adjustable thresholds and the confidence score data 124. For example, if a given WDE 126 is less than a minimum wall distance threshold or greater than a maximum wall distance threshold, the given WDE is removed. In another example, if a given WDE 126 is associated with a confidence score less than a minimum confidence threshold, the given WDE 126 is removed from the WDEs 126. The threshold used may be a predetermined threshold.

[0067] At step 812, the audio processing application 120 clusters the WDEs for each microphone in the microphone pair 150. One example of clustering is clustering based on the associated microphone pair WDEs 126 in the microphone pair 150 of each audio processing device 100. For example, if the distance difference between two or more WDEs associated with the same microphone in the microphone pair 150 is less than a minimum distance threshold, each WDE except the WDE associated with the highest confidence score is removed from the WDEs 126. In some embodiments, the confidence scores associated with the remaining WDEs 126 may be averaged with the other confidence scores of the removed WDEs. The minimum distance threshold may be a predefined threshold.

[0068] At step 814, the audio processing application 120 clusters the WDEs of the two microphones in the microphone pair 150 of each audio processing device 100. Another example of clustering is clustering based on associated microphone pair WDEs 126 in the microphone pair 150 of the same audio processing device 100. For example, if the distance difference between two or more WDEs 126 associated with different microphones of the microphone pair 150 in the same audio processing device 100 is less than a minimum distance threshold, then the WDE is determined to be a “valid estimate”.

[0069] At step 816, the audio processing application 120 may identify and extract WDEs representing the ceiling and / or floor of the acoustic environment. One example of identifying and extracting ceiling estimates and / or floor estimates is to calculate the mode of the WDEs 126 identified by each of the audio processing devices 100 in the acoustic environment. For example, the audio processing application 120 calculates the mode of the WDEs 126. If the associated confidence score of each WDE equal to the mode is greater than a minimum confidence threshold, then one of the WDEs of the mode may be identified as a "ceiling estimate" or "floor estimate" and the remaining WDEs may be removed. The minimum confidence threshold may be a predefined threshold.

[0070] At step 818, the audio processing application 120 combines the remaining WDEs 126 of the two microphones in the microphone pair 150 whose separation distance is less than the minimum distance threshold in a specific order other than the confidence score. The combined or unified WDEs 126 accurately represent each wall of the acoustic environment. These combined or unified WDEs 126 accurately represent each wall of the acoustic environment from the perspective of the associated audio processing device 100.

[0071] At step 820, the audio processing application 120 determines an acoustic model of the acoustic environment based on the combined or unified WDE 126. The audio processing application 120 processes one or more audio signals using the acoustic model. The audio processing application 120 emits the one or more processed audio signals based on the combined or unified WDE 126 using one or more speakers 160.

[0072] Methods for determining room layout estimates

[0073] Fig. 9 A flow chart illustrating method steps for determining a room layout estimate for an acoustic environment. Figures 1 to 7 The method steps are described with reference to a system, but those skilled in the art will understand that any system configured to perform the method steps in any order falls within the scope of the present invention.

[0074] As shown, method 900 begins at step 902, where the audio processing application 120 of each audio processing device 100 processes one or more audio signals based on the WDE 126 to determine an initial WDE 126 that accurately represents each wall of the acoustic environment from the perspective of each associated audio processing device 100 in the acoustic environment. For example, Figure 8 The steps of may be used to determine the initial WDE 126 used in method 900, but this is not meant to be limiting.

[0075] At step 904, the audio processing application 120 of one of the audio processing devices 100 in the acoustic environment selects one of the audio processing devices 100 as an ordinal device. Any of the audio processing devices 100 in the acoustic environment can be an ordinal device. An ordinal device is defined as a reference device, where the center of the device is considered to be the origin of the room layout of the acoustic environment.

[0076] At step 906, the audio processing application 120 determines the orientation of each wall in the acoustic environment relative to each audio processing device 100 in the acoustic environment. The orientation of each wall can be the front, left, back, or right. The term "front" means that the device is facing the relevant wall. The term "back" means that the device is facing away from the relevant wall. The term "left" means that the wall is located in the left direction of the device. The term "right" means that the wall is located in the right direction of the device. For example, Figure 3 Table 300 shows that based on Figure 2 The column for wall 202a shows the orientation of wall 202a relative to audio processing devices 204a, 204b, 204c, and 204d. The other columns of table 300 show Figure 2The orientation of other walls in the use case acoustic environment 200 relative to the audio processing devices 204a, 204b, 204c, and 204d.

[0077] At step 908, the audio processing application 120 generates a room layout matrix, such as an RLM 130, based on each audio processing device 100 in the acoustic environment and each wall in the acoustic environment. In various embodiments, only the first n confidence scores of each audio processing device 100 are used in the RLM 130, where n is equal to the number of non-ceiling, non-floor walls in the acoustic environment. For example, in an acoustic environment with four non-ceiling, non-floor walls, the first four confidence scores are used to populate the RLM 130. In some embodiments, if the audio processing application 120 determines that all rows or all columns of the RLM 130 are zero, the audio processing application 120 issues an error flag. Figure 5 RLMs 500a and 500b in FIG. 130 show examples of ordered RLMs for four devices in an acoustic environment with four non-ceiling, non-floor walls. The labels for the wall columns in RLM 130 begin with the placeholder labels walls W, X, Y, and Z. The following techniques help disambiguate which walls of the acoustic environment should be associated with which WDE.

[0078] At step 910, the audio processing application 120 sorts the unified WDEs (such as WDE 126) by descending confidence scores based on the confidence score data 124 and places the sorted WDEs in a room layout matrix. In various embodiments, only the first n confidence scores of each audio processing device 100 are used in the RLM 130, where n is equal to the number of non-ceiling, non-floor walls in the acoustic environment. For example, in an acoustic environment with four non-ceiling, non-floor walls, the first four confidence scores are used to populate the RLM 130. In some embodiments, if the audio processing application 120 determines that all rows or all columns of the RLM 130 are zero, the audio processing application 120 issues an error flag.

[0079] At step 912, the audio processing application 120 shifts the columns of the room layout matrix based on the difference in rotation of each device relative to the ordinal device. The rotational difference may be determined based on the device position data 122. For example, if the audio processing application 120 determines that the non-ordinal device has a different orientation than the ordinal device, the audio processing application 120 cyclically shifts the value in the row of the device of the RLM 130 one position to the right for each 90 degree difference in orientation compared to the ordinal device.

[0080] At step 914, the audio processing application 120 reorders each WDE in each row of the room layout matrix based on the determined orientation of the device corresponding to the row so that the WDE with the highest confidence score is placed in the column associated with the front wall relative to the device. For example, with respect to the orientation in table 300, the WDE associated with the highest confidence score is placed in the front wall column of each device. The WDE associated with the lowest confidence score is placed in the rear wall column of each device. Because some devices face opposite directions, the walls associated with the second highest confidence score and the third highest confidence score may change. For example, if Figure 6 As shown in RLM 600a of FIG. 5 , the WDE associated with the second highest confidence score is placed in the left wall column of devices 204a and 204c and in the right wall column of devices 204b and 204d. The WDE associated with the third highest confidence score is placed in the right wall column of devices 204a and 204c and in the left wall column of devices 204b and 204d.

[0081] At step 916, the audio processing application 120 removes the outlier WDE from the RLM 130. The removal of the outlier may be based on a multidimensional scaling method. The removal of the outlier may be based on the device location data 122. For example, as shown in the acoustic environment 200, the audio processing device 204c is located between the audio processing device 204d and the wall 202d. If the distance between the audio processing devices 204d and 204c is 2m, then the distance between the wall 202d and the audio processing device 204a is physically impossible to be less than 2m. Therefore, the audio processing application 120 may use the device location data 122 to determine that the WDE with a value of 2m associated with the audio processing device 204d in the RLM 500a (which is associated with a confidence level of 0.72 in the RLM 500d) cannot be associated with the wall 202d. The audio processing application 120 may use the device location data 122 relative to the audio processing device 204a to calculate a similar analysis for the wall 202a. Because the WDE with a value of 2m is associated with a lower confidence score than the other WDEs associated with audio processing device 204d, and cannot be associated with wall 202a or wall 202d, the audio processing application identifies the WDE as an outlier and removes the WDE and associated confidence score from the RLM, as shown in FIG. Figure 6 The lack of WDE and confidence score in RLM600a and 600b, respectively, is shown in FIG.

[0082] At step 918, the audio processing application 120 determines a final WDE for each wall based on the WDE with the highest confidence score among all audio processing devices 100 in the acoustic environment. For example, based on the acoustic environment 200, from the perspective of the ordinal audio processing device 204a, wall 202a is the front wall, wall 202b is the left wall, wall 202c is the back wall, and wall 202d is the right wall. To determine the final WDE for each wall, the audio processing application 120 determines the WDE with the highest confidence score in each wall column. For example, for the column representing wall 202a in RLM 600a, the audio processing application 120 determines that the WDE associated with the highest confidence score is associated with audio processing device 204a. Because audio processing device 204a is an ordinal device, the WDE of 3.9m is the final distance, and no other calculations need to be performed. In another example, for the column in RLM600a representing wall 202d, the audio processing application 120 determines that a WDE of 1.0 m is associated with the highest confidence score of 0.68, but is associated with a non-ordinal device. Because the WDE of 1.0 m and the confidence score of 0.68 are associated with audio processing device 204b and not with the ordinal device (audio processing device 204a), the audio processing application 120 uses the device position data 122 to determine the final WDE. For example, if audio processing device 204b is 1.0 meter away from the ordinal device in a direction toward wall 202d, and audio processing device 204b is closer to wall 202d than the ordinal device, as shown in acoustic environment 200, the audio processing application 120 calculates the final distance value of wall 202d by adding the WDE of 1.0 m to the distance between audio processing device 204b and the ordinal device in the direction of wall 202d. In some embodiments, the distance between the two devices in the direction of the wall is zero. For example, for the column in RLM 600a representing wall 202b, the audio processing application 120 determines that a WDE of 1.0m is associated with the highest confidence score of 0.94, but is associated with a non-ordinal device. Because the WDE of 1.0m and the confidence score of 0.94 are associated with audio processing device 204d and not with an ordinal device (audio processing device 204a), the audio processing application 120 uses the device position data 122 to determine the final WDE. For example, as shown in the acoustic environment 200, the audio processing device 204d is equidistant from the ordinal device with respect to wall 202b. Therefore, the audio processing application 120 calculates the final WDE for wall 202b by subtracting or adding 0.0m from the WDE of 1.0m. This process may be repeated for each wall in the acoustic environment.

[0083] At step 920, the audio processing application 120 calculates room layout estimates for the acoustic environment, such as the length, width, height, and volume of the acoustic environment. The audio processing application 120 determines the length of the acoustic environment by adding the distance from the ordinal device to the front wall and the distance from the ordinal device to the rear wall. The audio processing application 120 determines the width of the acoustic environment by adding the distance from the ordinal device to the left wall and the distance from the ordinal device to the right wall. The audio processing application 120 determines the height of the acoustic environment by adding the WDE associated with the ceiling and the WDE associated with the floor. The audio processing application 120 determines the volume of the acoustic environment by multiplying the length of the acoustic environment by the width of the acoustic environment and the height of the acoustic environment.

[0084] At step 922, the audio processing application 120 determines an acoustic model of the acoustic environment based on the room layout estimate. The audio processing application 120 processes one or more audio signals using the acoustic model. The audio processing application 120 uses one or more speakers 160 to emit one or more processed audio signals based on the room layout estimate.

[0085] In summary, the disclosed technology allows an audio processing system to accurately estimate the distance to the walls of an acoustic environment in which a device is located. A speaker in each device transmits a reference audio signal, such as a sine sweep, and in response, a microphone pair in each device receives an impulse response from each wall of the acoustic environment. The audio processing system calculates a wall distance estimate (WDE) for each wall (including ceiling and floor) based on the peak distance of the impulse response from the wall to each microphone in the microphone pair, resulting in six WDEs for each microphone of each device. The audio processing system calculates a confidence score for each wall (including ceiling and floor). The audio processing system removes outliers, clusters the estimates, removes duplicate estimates, and extracts ceiling estimates and floor estimates. The audio processing system combines the remaining WDEs for each microphone of each device with the confidence scores to determine an accurate, unified WDE for each device.

[0086] In some embodiments, the audio processing system combines the unified WDE for each device with the device distance estimate to determine the room layout of the acoustic environment. The audio processing system selects the device as an ordinal device. The audio processing system determines the orientation of each wall within the acoustic environment relative to each device in the acoustic environment. The audio processing system creates a room layout matrix in which rows are associated with devices in the acoustic environment and columns are associated with walls of the acoustic environment. The audio processing system sorts the unified WDEs starting from the ordinal device in descending order of confidence scores and places the WDEs in the room layout matrix. If one or more of the devices has a different orientation relative to the acoustic environment compared to the ordinal device, the audio processing system shifts the value in the row of the device one position to the right for every 90 degrees of orientation difference compared to the ordinal device. The audio processing system reorders the WDEs in each row based on the determined orientation of each wall relative to each device, so that the WDE with the highest confidence score for each device is associated with the front wall of the device. The audio processing system removes outlier WDEs. The audio processing system calculates a final WDE for each wall based on the WDE of that wall with the highest confidence score among all devices. The audio processing system calculates a room layout estimate based on the final WDE distance relative to the ordinal device.

[0087] At least one technical advantage of the disclosed technology over the prior art is that, using the disclosed technology, an audio system can accurately estimate the wall boundaries of a room with minimal setup. The disclosed technology is not affected by the relative positions of the devices of the audio system in the room, thereby allowing more flexible placement of the devices to estimate the boundaries and layout of the room. In addition, the audio system does not need to be equipped with a camera or image processing software. In addition, the estimated room layout produces a better room acoustic model and correspondingly improves the sound quality output by the audio system. These technical advantages provide one or more technical improvements over prior art methods.

[0088] 1. In some embodiments, a computer-implemented method includes: emitting a reference audio signal using one or more speakers of an audio processing device located in an acoustic environment; receiving, by each microphone of a microphone pair of the audio processing device, a response of the acoustic environment to the reference audio signal; determining an impulse response of the acoustic environment based on the reference audio signal and the received response of the acoustic environment; determining a plurality of candidate wall distance estimates and a plurality of confidence scores based on the impulse response, each confidence score in the plurality of confidence scores being associated with a different candidate wall distance estimate in the plurality of candidate wall distance estimates; and determining a set of wall distance estimates based on the plurality of candidate wall distance estimates, the plurality of confidence scores, and candidate wall distance estimates and confidence scores received from one or more other audio processing devices, each wall distance estimate in the set of wall distance estimates being associated with a different wall in the plurality of walls in the acoustic environment.

[0089] 2. The computer-implemented method of clause 1, wherein determining a set of wall distance elements comprises clustering the plurality of candidate wall distance estimates associated with a first microphone in the pair of microphones.

[0090] 3. A computer-implemented method as described in clause 2, wherein clustering the multiple candidate wall distance estimates associated with the first microphone includes: determining that a first candidate wall distance estimate among the multiple candidate wall distance estimates associated with the first microphone is within a threshold distance from a second candidate wall distance estimate among the multiple candidate wall distance estimates associated with the first microphone; and removing the first candidate wall distance estimate from the multiple candidate wall distance estimates.

[0091] 4. A computer-implemented method as described in clause 3, wherein removing the first candidate wall distance estimate from the multiple candidate wall distance estimates includes: determining that the confidence score associated with the second candidate wall distance estimate is higher than the confidence score associated with the first candidate wall distance estimate; and determining a new confidence score associated with the second candidate wall distance estimate based on averaging the confidence score associated with the first candidate wall distance estimate and the confidence score associated with the second candidate wall distance estimate.

[0092] 5. The computer-implemented method of clauses 1 to 4, wherein determining the set of wall distance estimates comprises clustering together the plurality of candidate wall distance estimates associated with different microphones in the microphone pair.

[0093] 6. A computer-implemented method as described in clause 5, wherein clustering the multiple candidate wall distance estimates associated with the different microphones includes: determining that a first candidate wall distance estimate among the multiple candidate wall distance estimates associated with a first microphone in the microphone pair is within a threshold distance from a second candidate wall distance estimate among the multiple candidate wall distance estimates associated with a second microphone in the microphone pair; and determining that the set of wall distance estimates includes at least one of the first candidate wall distance estimate among the multiple candidate wall distance estimates associated with the first microphone or the second candidate wall distance estimate among the multiple candidate wall distance estimates associated with the second microphone.

[0094] 7. A computer-implemented method as described in clauses 1 to 6, further comprising: determining a most common wall distance estimate among the multiple candidate wall distance estimates and the candidate wall distance estimates received from the one or more other audio processing devices; and identifying the most common wall distance estimate as a ceiling distance estimate.

[0095] 8. The computer-implemented method of clauses 1 to 7, wherein the reference audio signal comprises a sine sweep.

[0096] 9. The computer-implemented method of clauses 1 to 8, wherein the audio processing device and the other audio processing devices are located at the same height from a floor of the acoustic environment.

[0097] 10. A computer-implemented method as described in clauses 1 to 9, wherein the candidate wall distance estimate and confidence score received from a second audio processing device among the one or more other audio processing devices is based on one or more other impulse responses determined by the second audio processing device.

[0098] 11. A computer-implemented method as described in clauses 1 to 10, which also includes: determining an acoustic model of the acoustic environment based on the set of wall distance estimation values; processing one or more audio signals using the acoustic model; and emitting one or more processed audio signals using the one or more speakers.

[0099] 12. In some embodiments, one or more non-transitory computer-readable media include instructions that, when executed by one or more processors, cause the one or more processors to perform the following steps: emitting a reference audio signal using one or more speakers of an audio processing device located in an acoustic environment; receiving a response of the acoustic environment to the reference audio signal by each microphone in a pair of microphones of the audio processing device; determining an impulse response of the acoustic environment based on the reference audio signal and the received response of the acoustic environment; determining a plurality of candidate wall distance estimates and a plurality of confidence scores based on the impulse response, each confidence score in the plurality of confidence scores being associated with a different candidate wall distance estimate in the plurality of candidate wall distance estimates; and determining a set of wall distance estimates based on the plurality of candidate wall distance estimates, the plurality of confidence scores, and candidate wall distance estimates and confidence scores received from one or more other audio processing devices, each wall distance estimate in the set of wall distance estimates being associated with a different wall in the plurality of walls in the acoustic environment.

[0100] 13. The one or more non-transitory computer-readable media of clause 12, wherein determining a set of wall distance elements comprises clustering the plurality of candidate wall distance estimates associated with the first microphone of the pair of microphones.

[0101] 14. One or more non-transitory computer-readable media as described in claim 13, wherein the step of clustering the multiple candidate wall distance estimates associated with the first microphone includes: determining that a first candidate wall distance estimate among the multiple candidate wall distance estimates associated with the first microphone is within a threshold distance from a second candidate wall distance estimate among the multiple candidate wall distance estimates associated with the first microphone; and removing the first candidate wall distance estimate from the multiple candidate wall distance estimates.

[0102] 15. One or more non-transitory computer-readable media as described in claim 14, wherein the step of removing the first candidate wall distance estimate from the multiple candidate wall distance estimates includes: determining that the confidence score associated with the second candidate wall distance estimate is higher than the confidence score associated with the first candidate wall distance estimate; and determining a new confidence score associated with the second candidate wall distance estimate based on averaging the confidence score associated with the first candidate wall distance estimate and the confidence score associated with the second candidate wall distance estimate.

[0103] 16. The one or more non-transitory computer-readable media of clauses 12 to 15, wherein determining the set of wall distance estimates comprises clustering together the plurality of candidate wall distance estimates associated with different microphones in the microphone pair.

[0104] 17. One or more non-transitory computer-readable media as described in claim 16, wherein the step of clustering the multiple candidate wall distance estimates associated with the different microphones includes: determining that a first candidate wall distance estimate among the multiple candidate wall distance estimates associated with a first microphone in the microphone pair is within a threshold distance from a second candidate wall distance estimate among the multiple candidate wall distance estimates associated with a second microphone in the microphone pair; and determining that the set of wall distance estimates includes at least one of the first candidate wall distance estimate among the multiple candidate wall distance estimates associated with the first microphone or the second candidate wall distance estimate among the multiple candidate wall distance estimates associated with the second microphone.

[0105] 18. One or more non-transitory computer-readable media as described in clauses 12 to 17, wherein the steps further include: determining a most common wall distance estimate among the multiple candidate wall distance estimates and the candidate wall distance estimates received from the one or more other audio processing devices; and identifying the most common wall distance estimate as a ceiling distance estimate.

[0106] 20. In some embodiments, a system for estimating room layout includes: a plurality of audio processing devices in an acoustic environment, wherein a first audio processing device in the plurality of audio processing devices includes: one or more speakers; a microphone pair; one or more memories, wherein the one or more memories store an audio processing application; and one or more processors, wherein the one or more processors are coupled to the one or more memories, and wherein the one or more processors perform the following steps when executing the audio processing application: using the one or more speakers to emit a reference audio signal; receiving, by each microphone in the microphone pair, a response of the acoustic environment to the reference audio signal; and based on the reference audio signal, The method comprises determining an impulse response of the acoustic environment based on an audio signal and a received response of the acoustic environment; determining a plurality of candidate wall distance estimates and a plurality of confidence scores based on the impulse response, each of the plurality of confidence scores being associated with a different candidate wall distance estimate among the plurality of candidate wall distance estimates; and determining a set of wall distance estimates based on the plurality of candidate wall distance estimates, the plurality of confidence scores, and candidate wall distance estimates and confidence scores received from one or more other audio processing devices, each wall distance estimate in the set of wall distance estimates being associated with a different wall among a plurality of walls in the acoustic environment.

[0107] Any and all combinations of any of the claim elements as described in any of the claims and / or any of the elements described in this application, in any form, are within the intended scope of the invention and protection.

[0108] Descriptions of various embodiments have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

[0109] Aspects of the present embodiment may be embodied as a system, method or computer program product. Therefore, aspects of the present disclosure may take the following forms: a complete hardware implementation, a complete software implementation (including firmware, resident software, microcode, etc.) or an implementation combining software aspects and hardware aspects, which are generally all referred to as "modules", "systems" or "computers" in this article. In addition, any hardware and / or software technology, process, function, component, engine, module or system described in the present disclosure may be implemented as a circuit or a circuit set. In addition, aspects of the present disclosure may take the form of a computer program product, which is embodied in one or more computer-readable media on which a computer-readable program code is embodied.

[0110] Any combination of one or more computer-readable media may be utilized. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer-readable storage media will include the following media: an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing media. In the context of this document, a computer-readable storage medium may be any tangible medium that may contain or store a program for use by an instruction execution system, device, or apparatus, or that may be connected to an instruction execution system, device, or apparatus.

[0111] Aspects of the present disclosure are described above with reference to the flowchart illustration and / or block diagram of the method, device (system) and computer program product according to the embodiments of the present disclosure.It should be understood that each frame in the flowchart illustration and / or block diagram and the frame combination in the flowchart illustration and / or block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine.These instructions enable the function / action specified in one or more frames of the flowchart and / or block diagram to be realized when executed via the processor of a computer or other programmable data processing device.Such processors can be, but are not limited to, general-purpose processors, special-purpose processors, special-purpose processors or field programmable gate arrays.

[0112] The flow charts and block diagrams in the accompanying drawings illustrate the architecture, functions and operations of possible implementations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this respect, each box in the flow chart or block diagram may represent a module, section or part of a code, and the code includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions proposed in the box may not occur in the order proposed in the accompanying drawings. For example, in fact, depending on the functions involved, the two boxes shown in succession can be executed substantially simultaneously, or sometimes these boxes can be executed in reverse order. It should also be noted that each box illustrated in the block diagram and / or flow chart and the combination of boxes in the block diagram and / or flow chart can be implemented by a system based on special-purpose hardware that performs a specified function or action or by a combination of special-purpose hardware and computer instructions.

[0113] While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the present disclosure may be devised without departing from the basic scope thereof, the scope of which is determined by the following claims.

Claims

1. A computer-implemented method comprising: emitting a reference audio signal using one or more speakers of an audio processing device located in an acoustic environment; receiving, by each microphone of a pair of microphones of the audio processing device, a response of the acoustic environment to the reference audio signal; determining an impulse response of the acoustic environment based on the reference audio signal and a received response of the acoustic environment; determining a plurality of candidate wall distance estimates and a plurality of confidence scores based on the impulse response, each confidence score in the plurality of confidence scores being associated with a different candidate wall distance estimate in the plurality of candidate wall distance estimates; as well as Based on the multiple candidate wall distance estimates, the multiple confidence scores, and candidate wall distance estimates and confidence scores received from one or more other audio processing devices, a set of wall distance estimates is determined, each wall distance estimate in the set of wall distance estimates being associated with a different wall of the multiple walls in the acoustic environment. 2 . The method of claim 1 , wherein determining a set of wall distance elements comprises clustering the plurality of candidate wall distance estimates associated with the first microphone of the pair of microphones.

3. The method of claim 2, wherein clustering the plurality of candidate wall distance estimates associated with the first microphone comprises: determining that a first candidate wall distance estimate of the plurality of candidate wall distance estimates associated with the first microphone is within a threshold distance of a second candidate wall distance estimate of the plurality of candidate wall distance estimates associated with the first microphone; as well as The first candidate wall distance estimate is removed from the plurality of candidate wall distance estimates.

4. The method of claim 3, wherein removing the first candidate wall distance estimate from the plurality of candidate wall distance estimates comprises: determining that a confidence score associated with the second candidate wall distance estimate is higher than a confidence score associated with the first candidate wall distance estimate; as well as A new confidence score associated with the second candidate wall distance estimate is determined based on averaging the confidence score associated with the first candidate wall distance estimate and the confidence score associated with the second candidate wall distance estimate.

5. The method of claim 1, wherein determining the set of wall distance estimates comprises clustering together the plurality of candidate wall distance estimates associated with different microphones in the microphone pair.

6. The method of claim 5, wherein clustering the plurality of candidate wall distance estimates associated with the different microphones comprises: determining that a first candidate wall distance estimate of the plurality of candidate wall distance estimates associated with a first microphone of the pair of microphones is within a threshold distance of a second candidate wall distance estimate of the plurality of candidate wall distance estimates associated with a second microphone of the pair of microphones; as well as Determining the set of wall distance estimates includes at least one of the first candidate wall distance estimate of the plurality of candidate wall distance estimates associated with the first microphone or the second candidate wall distance estimate of the plurality of candidate wall distance estimates associated with the second microphone.

7. The method of claim 1, further comprising: determining a most common wall distance estimate among the plurality of candidate wall distance estimates and the candidate wall distance estimates received from the one or more other audio processing devices; as well as The most common wall distance estimate is identified as the ceiling distance estimate.

8. The method of claim 1, wherein the reference audio signal comprises a swept sine.

9. The method of claim 1, wherein the audio processing device and the other audio processing devices are located at the same height from a floor of the acoustic environment.

10. The method of claim 1, wherein the candidate wall distance estimate and confidence score received from a second audio processing device of the one or more other audio processing devices is based on one or more other impulse responses determined by the second audio processing device.

11. The method of claim 1, further comprising: determining an acoustic model of the acoustic environment based on the set of wall distance estimates; processing one or more audio signals using the acoustic model; as well as One or more processed audio signals are emitted using the one or more speakers.

12. One or more non-transitory computer-readable media comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the following steps: emitting a reference audio signal using one or more speakers of an audio processing device located in an acoustic environment; receiving, by each microphone of a pair of microphones of the audio processing device, a response of the acoustic environment to the reference audio signal; determining an impulse response of the acoustic environment based on the reference audio signal and a received response of the acoustic environment; determining a plurality of candidate wall distance estimates and a plurality of confidence scores based on the impulse response, each confidence score in the plurality of confidence scores being associated with a different candidate wall distance estimate in the plurality of candidate wall distance estimates; as well as Based on the multiple candidate wall distance estimates, the multiple confidence scores, and candidate wall distance estimates and confidence scores received from one or more other audio processing devices, a set of wall distance estimates is determined, each wall distance estimate in the set of wall distance estimates being associated with a different wall of the multiple walls in the acoustic environment.

13. The one or more non-transitory computer-readable media of claim 12, wherein determining a set of wall distance elements comprises clustering the plurality of candidate wall distance estimates associated with the first microphone of the pair of microphones.

14. The one or more non-transitory computer-readable media of claim 13, wherein clustering the plurality of candidate wall distance estimates associated with the first microphone comprises: determining that a first candidate wall distance estimate of the plurality of candidate wall distance estimates associated with the first microphone is within a threshold distance of a second candidate wall distance estimate of the plurality of candidate wall distance estimates associated with the first microphone; as well as The first candidate wall distance estimate is removed from the plurality of candidate wall distance estimates.

15. The one or more non-transitory computer-readable media of claim 14, wherein removing the first candidate wall distance estimate from the plurality of candidate wall distance estimates comprises: determining that a confidence score associated with the second candidate wall distance estimate is higher than a confidence score associated with the first candidate wall distance estimate; as well as A new confidence score associated with the second candidate wall distance estimate is determined based on averaging the confidence score associated with the first candidate wall distance estimate and the confidence score associated with the second candidate wall distance estimate.

16. The one or more non-transitory computer-readable media of claim 12, wherein determining the set of wall distance estimates comprises clustering together the plurality of candidate wall distance estimates associated with different microphones in the microphone pair.

17. The one or more non-transitory computer-readable media of claim 16, wherein clustering the plurality of candidate wall distance estimates associated with the different microphones comprises: determining that a first candidate wall distance estimate of the plurality of candidate wall distance estimates associated with a first microphone of the pair of microphones is within a threshold distance of a second candidate wall distance estimate of the plurality of candidate wall distance estimates associated with a second microphone of the pair of microphones; as well as Determining the set of wall distance estimates includes at least one of the first candidate wall distance estimate of the plurality of candidate wall distance estimates associated with the first microphone or the second candidate wall distance estimate of the plurality of candidate wall distance estimates associated with the second microphone.

18. The one or more non-transitory computer readable media of claim 12, wherein the steps further comprise: determining a most common wall distance estimate among the plurality of candidate wall distance estimates and the candidate wall distance estimates received from the one or more other audio processing devices; as well as The most common wall distance estimate is identified as the ceiling distance estimate.

19. A system for estimating a room layout, comprising: A plurality of audio processing devices in an acoustic environment, wherein a first audio processing device of the plurality of audio processing devices comprises: one or more speakers; Microphone pair; one or more memories storing an audio processing application; and One or more processors, the one or more processors being coupled to the one or more memories, the one or more processors performing the following steps when executing the audio processing application: emitting a reference audio signal using the one or more speakers; receiving, by each microphone of the pair of microphones, a response of the acoustic environment to the reference audio signal; determining an impulse response of the acoustic environment based on the reference audio signal and a received response of the acoustic environment; determining a plurality of candidate wall distance estimates and a plurality of confidence scores based on the impulse response, each confidence score in the plurality of confidence scores being associated with a different candidate wall distance estimate in the plurality of candidate wall distance estimates; and Based on the multiple candidate wall distance estimates, the multiple confidence scores, and candidate wall distance estimates and confidence scores received from one or more other audio processing devices, a set of wall distance estimates is determined, each wall distance estimate in the set of wall distance estimates being associated with a different wall of the multiple walls in the acoustic environment.

Images