Method and system for multi-temporal resolution audio processing

Abstract

The invention relates to a multi-temporal resolution audio processing method and system. Aspects of the present disclosure provide a method for voice control, including transforming a multi-channel audio stream into a complex valued frequency domain representation using a short-time Fourier transform (STFT) applied to data in each window aligned across each input channel of the multi-channel audio stream. For the current window, the method further comprises the following steps: updating a first complex value covariance matrix corresponding to the slow adaptive beam former, and forming single-channel denoising estimation for each frequency band in the STFT; calculating a voice activity detection (VAD) estimate for each frequency band in the STFT by comparing the amplitude of the single-channel de-noising estimate with the amplitude of each input channel of the multi-channel audio stream; and selectively updating or not updating a second complex valued covariance matrix corresponding to the fast adaptive beamformer in response to the VAD estimates indicating the presence or absence of speech, respectively.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. non-provisional application serial No. 18 / 450,784, filed on August 16, 2023, entitled “Methods and Systems for Multi-channel Time-Resolution Audio Processing,” the entire contents of which are expressly incorporated herein by reference. Technical Field

[0003] Aspects of the present disclosure generally relate to systems and methods for multiple time-resolution audio processing. Background Art

[0004] People are increasingly interacting with Voice User Interface (VUI) platforms. VUI platforms can be given commands, asked questions, and even in some cases react to gestures to issue commands and eye contact to get their attention. Behind all of this technology is audio and video processing, which combines many other sources of information to reason about how to best respond to the user's input.

[0005] In the multi-channel audio processing part of VUI, there are multiple technologies that work together. One example is acoustic echo cancellation, which helps the device listen only to the user's voice and not its own. Another is noise suppression.

[0006] However, further improvements in the multi-channel audio processing part are needed. Summary of the invention

[0007] A simplified summary of one or more aspects is presented below to provide a basic understanding of these aspects. This summary is not an extensive overview of all contemplated aspects, and is neither intended to identify key or important elements of all aspects, nor to describe the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to a more detailed description that is presented later.

[0008] According to aspects of the present disclosure, a computer-implemented method for voice control is provided. The method includes transforming a multi-channel audio stream into a complex-valued frequency domain representation using a short-time Fourier transform (STFT) applied to data in each of a plurality of windows aligned across each input channel of the multi-channel audio stream. For a current one of the plurality of windows, the method further includes updating a first complex-valued covariance matrix corresponding to a slow adaptive beamformer and forming a single-channel denoised estimate for each frequency band in the STFT. For a current one of the plurality of windows, the method further includes calculating a voice activity detection (VAD) estimate for each frequency band in the STFT by comparing the amplitude of the single-channel denoised estimate with the amplitude of each input channel of the multi-channel audio stream. For a current one of the plurality of windows, the method further includes selectively updating or not updating a second complex-valued covariance matrix corresponding to a fast adaptive beamformer in response to a VAD estimate indicating the presence or absence of speech, respectively. The method includes controlling a device based on a voice user interface by a hardware processor to perform a user-perceivable action in response to at least the output of a fast adaptive beamformer.

[0009] According to other aspects of the present disclosure, a computer program product configured to enable voice control is provided. The computer program product includes one or more non-transitory computer-readable media having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform a method alone or in combination. The method includes transforming a multi-channel audio stream into a complex-valued frequency domain representation using a short-time Fourier transform (STFT) applied to data in each of a plurality of windows aligned across each input channel of the multi-channel audio stream. For a current one of the plurality of windows, the method also includes updating a first complex-valued covariance matrix corresponding to a slow adaptive beamformer and forming a single-channel denoised estimate for each frequency band in the STFT. For a current one of the plurality of windows, the method also includes calculating a voice activity detection (VAD) estimate for each frequency band in the STFT by comparing the amplitude of the single-channel denoised estimate formed by the slow adaptive beamformer with the amplitude of each input channel of the multi-channel audio stream. For a current one of the plurality of windows, the method also includes selectively updating or not updating a second complex-valued covariance matrix corresponding to a fast adaptive beamformer in response to a VAD estimate indicating the presence or absence of speech, respectively. The method includes controlling, by a hardware processor, a voice user interface-based device to perform a user-perceivable action in response to at least an output of a fast adaptive beamformer.

[0010] According to another aspect of the present disclosure, a voice control system is provided. The system includes one or more memories, alone or in combination, with instructions. The system also includes one or more processors, each coupled to at least one of the one or more memories, and configurable to execute the instructions to: transform the multi-channel audio stream into a complex-valued frequency domain representation using a short-time Fourier transform (STFT) applied to the data in each of the multiple windows aligned across each input channel of the multi-channel audio stream. The one or more processors also execute the instructions to update the first complex-valued covariance matrix corresponding to the slow adaptive beamformer for the current one of the multiple windows by the slow adaptive beamformer, and form a single-channel denoised estimate for each frequency band in the STFT by the slow adaptive beamformer. The one or more processors also execute the instructions to calculate a voice activity detection (VAD) estimate for each frequency band in the STFT for the current one of the multiple windows by comparing the amplitude of the single-channel denoised estimate formed by the slow adaptive beamformer with the amplitude of each input channel of the multi-channel audio stream. The one or more processors further execute instructions to selectively update or not update, for a current one of the plurality of windows, a second complex-valued covariance matrix corresponding to the fast adaptive beamformer in response to a VAD estimate indicating the presence or absence of speech, respectively. The one or more processors further execute instructions to control the device based on the voice user interface to perform a user-perceivable action in response to at least an output of the fast adaptive beamformer.

[0011] To accomplish the above and related ends, one or more aspects include the features hereinafter fully described and particularly pointed out in the claims. The following description and the accompanying drawings set forth in detail certain illustrative features of one or more aspects. However, these features are indicative of only a few of the various ways in which the principles of the various aspects may be employed, and this description is intended to include all such aspects and their equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The disclosed aspects will hereinafter be described in conjunction with the accompanying drawings, which are provided for illustration and not limitation of the disclosed aspects, wherein like reference numerals denote like elements and wherein:

[0013] Figure 1 A block diagram illustrating an example of a computer environment according to example aspects of the present disclosure is shown.

[0014] Figure 2 The present disclosure shows an example aspect of Figure 1 A block diagram of another client computer is shown in relation to a client computer.

[0015] Figure 3 The present disclosure shows an example aspect Figure 2 A block diagram of one or more memory devices.

[0016] Figure 4 Further illustrating exemplary aspects of the present disclosure Figure 2 A block diagram of one or more memory devices.

[0017] Figure 5 A block diagram of an example user environment is shown according to example aspects of the present disclosure.

[0018] Figure 6-10 A flow chart illustrating an example of a method for multi-time resolution audio processing and voice control according to example aspects of the present disclosure is shown.

[0019] Fig.11 A block diagram illustrating an example of dual beamformer processing according to example aspects of the present disclosure is shown. DETAILED DESCRIPTION

[0020] Aspects of the present disclosure relate to a method and system for multi-time resolution audio processing.

[0021] In one aspect, two beamformers, a slow adaptive beamformer and a fast adaptive beamformer, adapt at different speeds to improve the performance of, for example, a voice user interface (VUI).

[0022] In one aspect, the input to the system is a multi-channel audio stream from a microphone array and a desired source direction of arrival (DOA). The multi-channel audio stream is converted to a complex-valued frequency domain representation by Fourier transforming aligned overlapping windows (also called frames) on all input channels. This is often referred to as the Short Time Fourier Transform (STFT). That is, the SHTF is a transform that converts a function into a form that describes the frequencies present in the original function. The output of the transform is a complex-valued function of frequency.

[0023] In one aspect, frequency domain processing is applied to the complex-valued frequency domain representation of the data and the processed audio is transformed from the frequency domain back to the time domain using a standard inverse STFT and overlap-add process. The resulting time domain signal is the output of the system.

[0024] In one aspect, the algorithm works by tracking two complex-valued covariance matrices for each band in the Fourier transform, one for the slow adaptive beamformer and one for the fast adaptive beamformer. In one aspect, the two beamformers use the same equations for the update rule but different settings for the parameters that set the adaptation speed. In another aspect, the beamformers use different equations for the update rule and different settings for the parameters that set the adaptation speed.

[0025] In one aspect, after the covariance matrix is ​​updated, given the frequency domain representation of the current window, a minimum variance distortion-free response (MVDR) beamformer vector and / or a minimum power distortion-free response (MPDR) is calculated at each frequency and multiplied by the corresponding beamformer weight vector of the input STFT to produce a single-channel denoising result. The single-channel denoising result is determined by subtracting the noise signal from a single input channel of the input multi-channel audio stream. The beamformer weight vector can be an M×N vector, where M is an integer representing the number of sensor (microphone) elements and N is an integer representing the direction of arrival (DOA). In a given window, the process of updating the covariance and forming a single-channel denoised estimate is first applied in the slow beamformer. Then, in one aspect, a voice activity detection (VAD) estimate is calculated for each frequency bin (band) by comparing the amplitude of the denoised estimate output by the slow beamformer with the amplitude in any channel of the original signal to see if they are different within a threshold amount. The threshold amount can be selected empirically.

[0026] In one aspect, the VAD estimate is used to determine whether the data in the current frame should be used to update the covariance matrix of the fast beamformer. If the VAD indicates that speech is present, the fast beamformer updates the fast beamforming covariance matrix. Otherwise, the fast beamformer does update the fast beamforming covariance matrix.

[0027] In one aspect, the output of the fast beamformer is used as the output of the full voice user interface system. In fact, the slow beamformer is slow to adapt to the acoustic background and is not affected by speech as much as the fast beamformer because speech is relatively transient in time. When speech is present, the VAD output by the slow beamformer informs the fast beamformer so that the slow beamformer can adapt to fluctuations in the target speech signal not only quickly but also more accurately than if a single beamformer is used for both noise and signal. This improves the performance of the entire denoising system compared to the output of a single beamformer. This all assumes that the target DOA is known (given or estimated in some way). By specifically restricting the fast adaptive beamformer to only process speech, the performance of the fast adaptive beamformer will be improved over a single beamformer that processes both noise and speech. This is because the background changes much more slowly than the speaker; and the fast adaptive beamformer has a higher noise floor than the slow adaptive beamformer. Thus, adapting the fast adaptive beamformer to speech only helps adapt the slow beamformer to noise, and the fast beamformer can also adapt to rapidly changing speech signals. In addition, various aspects of the present disclosure enable the noise to be averaged slowly over a long period of time by the slow adaptive beamformer in order to capture the stationarity of the noise by using a slow adaptation rate compared to the fast adaptive beamformer.

[0028] Methods according to various aspects may be performed online or offline. In one aspect, an online implementation may use a first buffer for collecting input samples and a second buffer for outputting processed audio. The covariance update and beamformer vector calculation steps may be rewritten in a mathematically equivalent form to avoid computing matrix inversions. After this, all operations only include matrix-vector products, vector-vector products, scalar divisions, and other such simple algebraic operations. This makes the implementation more hardware-friendly and significantly speeds up the runtime, allowing multiple such beamformers to be run simultaneously on an embedded device compared to the case where the covariance update and beamformer vector calculation steps are not mathematically equivalent.

[0029] Reference Figure 1 , illustrates an example computing environment 100 according to one aspect of the present disclosure.

[0030] The computing environment 100 includes an example of an environment for executing at least some of the computer code 177 involved in performing a method (e.g., multi-time resolution audio processing). In addition to the computer code 177, the computing environment 100 also includes, for example, a client computer 101, a wide area network (WAN) 102, a remote server 104, a public cloud 105, and a private cloud 106. In this embodiment, the computer 101 includes a processor group 110 (including a processing circuit 120 and a cache 121), a communication structure 111, a volatile memory 112, a persistent memory 113 (including an operating system 122 and a block 177, as described above), a peripheral device group 114 (including a user interface (UI), a device group 123, a memory 124, and an Internet of Things (IoT) sensor group 125), and a network module 115. The remote server 104 includes a remote database 130. The public cloud 105 includes a gateway 140, a cloud orchestration module 141, a host physical machine group 142, a virtual machine group 143, and a container group 144.

[0031] Computer 101 may take the form of a desktop computer, laptop computer, tablet computer, smartphone, smart watch or other wearable computer, mainframe computer, quantum computer, or any other form of computer or mobile device now known or developed in the future that is capable of running programs, accessing a network, or querying a database, such as remote database 130. As is well known in the art of computer technology, the performance of computer-implemented methods may be distributed among multiple computers and / or among multiple locations, depending on the technology. On the other hand, in this demonstration of computing environment 100, the detailed discussion focuses on a single computer, particularly computer 101, to keep the demonstration as simple as possible. Even though computer 101 is Figure 1Although not shown in the cloud, the computer 101 may also be located in the cloud. On the other hand, the computer 101 does not need to be in the cloud unless it can be positively indicated. The computer 101 may be interchangeably referred to as a "VUI-based device" 101 herein.

[0032] Processor group 110 includes one or more computer processors of any type now known or developed in the future. Processing circuit 120 can be distributed over multiple packages, for example, multiple coordinated integrated circuit chips. Processing circuit 120 can implement multiple processor threads and / or multiple processor cores. Cache 121 is a memory located in the processor chip package, typically used for data or code that should be quickly accessible to threads or cores running on processor group 110. Cache memory is typically organized into multiple levels based on relative proximity to the processing circuit. Alternatively, some or all of the cache of the processor group can be located "off-chip". In some computing environments, processor group 110 can be designed to work with quantum bits and perform quantum computing.

[0033] Computer-readable program instructions are typically loaded onto the computer 101 to cause the processor group 110 of the computer 101 to perform a series of operating steps to implement a computer-implemented method, such that the instructions executed thereby will instantiate the method specified in the flowchart and / or narrative description of the computer-implemented method included in this document (collectively referred to as the "method of the present invention"). These computer-readable program instructions are stored in various types of computer-readable storage media, such as cache 121 and other storage media discussed below. The program instructions and related data are accessed by the processor group 110 to control and direct the execution of the method of the present invention. In the computing environment 100, at least some of the instructions for performing the method of the present invention may be stored in a block 177 in the persistent memory 113.

[0034] Communications fabric 111 is the signaling pathways that allow the various components of computer 101 to communicate with each other. Typically, such fabrics are comprised of switches and conductive pathways, such as those that make up a bus, a bridge, physical input / output ports, etc. Other types of signal communication pathways may be used, such as fiber optic communication pathways and / or wireless communication pathways.

[0035] The volatile memory 112 is any type of volatile memory now known or developed in the future. Examples include dynamic random access memory (RAM) or static RAM. Typically, volatile memory is characterized by random access, but this is not required unless explicitly stated. In the computer 101, the volatile memory 112 is located in a single package and is internal to the computer 101, but, alternatively or additionally, the volatile memory may be distributed across multiple packages and / or located externally relative to the computer 101. Regarding the online aspect, the volatile memory 112 may include a first buffer 112A for collecting input samples and a second buffer 112B for outputting processed audio.

[0036] Persistent memory 113 is any form of non-volatile memory for computers now known or developed in the future. The non-volatility of this storage means that the stored data will be maintained regardless of whether power is supplied to computer 101 and / or directly to persistent memory 113. Persistent memory 113 can be a read-only memory (ROM), but typically at least a portion of the persistent memory allows data to be written, deleted, and rewritten. Some common forms of persistent storage include disks and solid-state storage devices. Operating system 122 can take a variety of forms, such as various known proprietary operating systems or operating systems of the type of open source portable operating system interface using a kernel. The code included in block 177 typically includes at least some computer code involved in executing the method of the present invention.

[0037] The peripheral device group 114 includes a peripheral device group of the computer 101. The data communication connection between the peripheral device and other components of the computer 101 can be implemented in various ways, such as a Bluetooth connection, a near field communication (NFC) connection, a connection through a cable (such as a universal serial bus (USB) type cable), a plug-in type connection (such as a secure digital (SD) card), a connection through a local area network, or even a connection through a wide area network such as the Internet. In various embodiments, the UI device group 123 may include one or more components such as a display screen, a speaker, a microphone array, a wearable device (such as goggles and smart watches), a keyboard, a mouse, a printer, a touchpad, a virtual reality goggles, an augmented reality goggles, a mixed reality goggles, a game controller, a voice user interface (VUI), an automatic speech recognition system (ASR), a text to speech (TTS) system, and a tactile device. The memory 124 is an external memory, such as an external hard drive, or a pluggable memory, such as an SD card. The memory 124 can be persistent and / or volatile. In some embodiments, the memory 124 can be in the form of a quantum computing storage device for storing data in the form of quantum bits. In embodiments where computer 101 is required to have a large amount of storage (e.g., computer 101 locally stores and manages a large database), the storage may be provided by a peripheral storage device designed to store large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. IoT sensor group 125 is composed of one or more sensors that can be used in IoT applications. For example, one sensor may be a thermometer and another sensor may be a motion detector.

[0038] The network module 115 is a collection of computer software, hardware, and firmware that allows the computer 101 to communicate with other computers via the WAN 102. The network module 115 may include hardware, such as a modem or a Wi-Fi signal transceiver, software for packetizing and / or unpacking data transmitted over a communication network, and / or web browser software for transmitting data over the Internet. In some embodiments, the network control function and the network forwarding function of the network module 115 are executed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing software defined networks (SDN)), the control function and the forwarding function of the network module 115 are executed on physically separate devices, so that the control function manages several different network hardware devices. Computer-readable program instructions for executing the method of the present invention can typically be downloaded to the computer 101 from an external computer or an external storage device via a network adapter card or a network interface included in the network module 115.

[0039] WAN 102 is any wide area network (e.g., the Internet) capable of communicating computer data over non-local distances by any technology for communicating computer data now known or developed in the future. In some embodiments, a WAN may be replaced and / or supplemented by a local area network (LAN), which is designed to transmit data between devices located in a local area network (e.g., a Wi-Fi network). A WAN and / or LAN typically includes computer hardware such as copper transmission cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and edge servers.

[0040] Remote server 104 is any computer system that provides at least some data and / or functionality to computer 101. Remote server 104 may be controlled and used by the same entity that operates computer 101. Remote server 104 represents a machine that collects and stores useful data for use by other computers (such as computer 101).

[0041] The public cloud 105 is any computer system available to multiple entities that provides on-demand availability of computer system resources and / or other computer capabilities, particularly data storage (cloud storage) and computing power, without direct active management by users. Cloud computing typically utilizes resource sharing to achieve consistency and economies of scale. The direct and active management of the computing resources of the public cloud 105 is performed by the computer hardware and / or software of the cloud orchestration module 141. The computing resources provided by the public cloud 105 are typically implemented by virtual computing environments running on various computers that constitute the host physical machine group 142, which is a universe of physical computers in and / or available to the public cloud 105. Virtual computing environments (VCEs) are typically in the form of virtual machines from the virtual machine group 143 and / or containers from the container group 144. It should be understood that these VCEs can be stored as images and can be transferred between various physical machine hosts as images or after the VCE is instantiated. The cloud orchestration module 141 manages the transmission and storage of images, deploys new instances of VCEs, and manages active instances of VCE deployments. Gateway 140 is a collection of computer software, hardware, and firmware that allows public cloud 105 to communicate over WAN 102 .

[0042] Public cloud 105 can provide a subscription service for human interaction to multiple users (e.g., users of computer 101). The service can be used for a variety of purposes of human interaction. Such purposes of human interaction can include dating, friendship, and business.

[0043] In one aspect, public cloud 105 works in conjunction with remote server 104 to retrieve and provide profile information of a user to a user, such as a user using computer 101 and / or another user operating a device similar to computer 101 .

[0044] Private cloud 106 is similar to public cloud 105, except that the computing resources are only available to a single enterprise. Although private cloud 106 is depicted as communicating with WAN 102, in other embodiments, the private cloud may be completely disconnected from the Internet and only accessible via a local / private network. A hybrid cloud is a combination of multiple clouds of different types (e.g., private, community, or public), typically implemented separately by different vendors. Each of the multiple clouds remains a separate and discrete entity, but the larger hybrid cloud architecture is bound together by standardized or proprietary technologies that enable orchestration, management, and / or data / application portability between the multiple constituent clouds. In this embodiment, both public cloud 105 and private cloud 106 are part of a larger hybrid cloud.

[0045] Reference Figure 2 , shows another client computer 200 according to various aspects of the present disclosure.

[0046] The client computer 200 is Figure 1 A variation of the client computer 101 and may be used with Figure 1 The same device interface as shown. The client device 200 may be interchangeably referred to as a “Voice User Interface (VUI)-based device” 200 .

[0047] The VUI-based device 200 may include a voice user interface 290 that includes or interfaces with any of the following elements: one or more processors 201; one or more memory devices 202 operably coupled to one or more processors 201; one or more microphone arrays 203; one or more speakers 204; one or more displays 205; one or more input devices 206; one or more automatic speech recognition (ASR) systems 207; and one or more text-to-speech (TTS) systems 208.

[0048] The one or more processors 201 and the one or more memory devices 202 are configured to execute program code to perform a method such as method 600 .

[0049] The one or more microphone arrays 203 are configured to receive speech and background noise, which are used in the manner described herein to improve audio processing of the VUI-based device 200. In particular, the methods described herein use the slow adaptive beamformer to determine when to have the fast adaptive beamformer selectively update the covariance matrix corresponding to the fast adaptive beamformer.

[0050] The one or more speakers 204 are configured to reproduce sounds, such as user prompts, responses to user commands and / or user inquiries, and the like.

[0051] One or more displays 205 may display the status of the device (eg, an LED to show power) as well as responses to user commands and / or user queries and other information relevant to the user's use of the device.

[0052] One or more input devices 206 may be used to power the device on or off, adjust volume, enter network information including network name and password, and the like.

[0053] One or more ASR systems 207 are configured to decode pronunciations of one or more languages. In other aspects, a single ASR can be used with different language models for different languages ​​and / or different domains (work, home, etc.).

[0054] The one or more TTS systems 208 are configured to convert responses to user commands and / or user queries into speech that is reproduced by the one or more speakers 204 .

[0055] Reference Figure 3 According to various aspects of the present disclosure, it is further shown that Figure 2 One or more memory devices 202.

[0056] The one or more memory devices 202 include a slow adaptive beamformer 202A and a fast adaptive beamformer 202B. The one or more processors 201 execute code corresponding to the slow adaptive beamformer 202A and the fast adaptive beamformer 202B to perform the multi-time resolution processing described herein. Beamforming is a signal processing technique used for sensor arrays (such as microphones) for directional signal transmission or reception.

[0057] The slowly adaptive beamformer 202A is configured to slowly adapt to the background including the background noise to take into account the stationarity of the noise.

[0058] The fast adaptive beamformer 202B is configured to quickly adapt to speech to take into account the temporal nature of speech.

[0059] In one aspect, the relative speeds of "slow" and "fast" are achieved by using the same covariance matrix update rule equation, but with different adaptation speeds, where the slow adaptive beamformer 202A is slower than the fast adaptive beamformer 202B.

[0060] In one aspect, for each of a plurality of windows aligned across each input channel of a multi-channel audio stream, an STFT is applied to the data in the window (frame).

[0061] For the current window, a voice activity detection (VAD) estimate is generated after updating the covariance matrix corresponding to the slow adaptive beamformer 202B, which is used along with the direction of arrival to selectively update or avoid updating the covariance matrix corresponding to the fast adaptive beamformer. In this way, the fast adaptive beamformer can adapt to fluctuations in the target speech signal more quickly and accurately, which enhances the performance of the overall denoising system compared to the output of a single beamformer, which must consider both speech and noise simultaneously and then be general to both, rather than specific to either one.

[0062] Reference Figure 4 According to various aspects of the present disclosure, it is further shown that Figure 2 One or more memory devices 202. Figure 5 , illustrating a user environment 500 according to various aspects of the present disclosure.

[0063] The one or more memory devices 202 include a slow adaptive beamformer 202A and a fast adaptive beamformer 202B. The one or more processors 201 execute code corresponding to the slow adaptive beamformer 202A and the fast adaptive beamformer 202B.

[0064] Covariance matrix 502A1 corresponds to the slow adaptive beamformer 202A, and therefore corresponds to noise. Covariance matrix 502B1 corresponds to the fast adaptive beamformer 202B, and therefore corresponds to speech. The covariance matrix is ​​a square matrix that gives the covariance between each pair of elements of a given random vector. Here, vector is the beamforming vector described herein.

[0065] A short-time Fourier transform is applied to each of a plurality of windows aligned on each input channel of the multi-channel audio stream.

[0066] For each window, the code / algorithm slowly updates the covariance matrix 502A1 corresponding to the slow adaptive beamformer 202A, while selectively updating the covariance matrix 502B1 corresponding to the fast adaptive beamformer 202B, to better adapt to speech. The statistics of speech change faster than the statistics of noise. In other words, speech is "less stable" than noise, so the statistics need to be adjusted more quickly. This is what the "fast adaptive" beamformer is able to achieve by forgetting the past faster (i.e., having a shorter effective memory window).

[0067] Therefore, for each current window being processed associated with the frequency band of the STFT, the covariance matrix of the slow adaptive beamformer 202A is updated and a VAD is formed. Based on the VAD indicating the absence or presence of speech, the covariance matrix of the fast adaptive beamformer 202B is updated or not, respectively.

[0068] Reference Figure 5 , illustrating a user environment 500 according to various aspects of the present disclosure.

[0069] The user environment 500 corresponds to the interior of a motor vehicle and includes a first microphone 501, a second microphone 502, and a third microphone 503 for receiving speech (e.g., user commands and / or user inquiries). In another aspect, the first microphone 501, the second microphone 502, and the third microphone 503 can each be a microphone array. These and other variations are included in aspects of the present disclosure. The microphones are arranged to have a known or easily determined direction of arrival (DOA). The DOA is used to calculate a minimum variance distortionless response (MVDR) and / or a minimum power distortionless response (MPDR).

[0070] The microphones or microphone arrays 501-503 can be arranged to be directional and optimized to capture speech from various seats while minimizing the capture of background noise. Thus, in one embodiment, they can be directed away from the motor and / or other noise sources and directed toward the mouth of the user (driver and / or passenger).

[0071] In one aspect, speech captured in the direction of the driver is weighted more heavily than speech captured in the direction of the passenger. In one aspect, the front seat passenger weighs more than the rear seat passenger.

[0072] Reference now Figure 6-10 , a method 600 for multi-time resolution audio processing and voice control is shown and described according to various exemplary aspects. Blocks shown with dashed lines are optional functions.

[0073] Method 600 may be performed by one or more processors (e.g., Figure 1 The processor group 110 and Figure 2201) which are operably coupled to one or more memories (e.g. Figure 1 The persistent memory 113 and Figure 2 202 in the memory device 202). The computer code (e.g., Figure 1 The computer code 177 and the slowly adaptive beamformer 502A and the fast adaptive beamformer 502B) implement the slowly adaptive beamformer 502A and the fast adaptive beamformer 502B.

[0074] At block 601, method 600 includes performing method 600 offline. This involves updating the coefficients of the beamformer offline.

[0075] At block 602, the method 600 includes performing the method 600 online. This involves updating the coefficients of the beamformer online.

[0076] In one aspect, the offline and online methods can be combined, for example, the offline method can be used to perform initialization and then the online method can be used when a new frequency snapshot is obtained.

[0077] At block 603 , the method 600 includes applying the method 600 in a beamforming strategy configured to use a correlation matrix of any one of the multi-channel audio streams and the noise.

[0078] At block 604 , the method 600 includes executing the method 600 by the automatic speech recognition system 207 operably coupled to the voice user interface 290 of the voice user interface based device 200 .

[0079] At block 605, the method 600 includes configuring the slow adaptive beamformer 202A and the fast adaptive beamformer 202B, both using the same equation for the update rule but with different adaptation speeds, or both having different update rule equations and different adaptation speeds.

[0080] At block 606, the method 600 includes configuring the slow adaptive beamformer 202A to slowly adapt to the acoustic background including background noise in the multi-channel audio stream, and configuring the fast adaptive beamformer 202B to quickly adapt to speech in the multi-channel audio stream.

[0081] At block 607 , the method 600 includes receiving a multi-channel audio stream and a desired source direction of arrival (DOA) from the microphone array 203 .

[0082] At block 610, method 600 includes transforming the multi-channel audio stream into a complex-valued frequency domain representation using a short-time Fourier transform (STFT) and an inverse STFT, and transforming back into a time domain representation. The STFT is applied to data in each of a plurality of windows aligned on each input channel of the multi-channel audio stream. In one aspect, the windows can be exponential windows. In one aspect, the windows can be rectangular windows of varying lengths. In one aspect, a slow adaptive beamformer can capture the stationarity of noise using a time window that is longer than a fast adaptive beamformer.

[0083] In one aspect, block 610 may include one or more of blocks 610A-610B.

[0084] At block 610A, the method 600 includes applying frequency domain processing to the complex-valued frequency domain representation to obtain processed audio, and transforming the processed video from the frequency domain to the time domain using an inverse STFT and overlap-add process and providing a time domain representation.

[0085] At block 610B, method 600 includes performing method 600 online using a first buffer configured to collect input samples and a second buffer configured to feed out processed audio.

[0086] At block 610C, method 600 includes obtaining a time domain representation using an inverse of the STFT with an overlap-add process.

[0087] At block 620 , for a current one of the plurality of windows, the method 600 includes updating, by the slowly adaptive beamformer 202A, a first complex-valued covariance matrix 202A1 corresponding to the slowly adaptive beamformer 202A and forming, by the slowly adaptive beamformer 202A, a single channel denoised estimate for each frequency band in the STFT.

[0088] In one aspect, block 620 may include one or more of blocks 620A-620F.

[0089] At block 620A, method 600 includes computing a beamformer vector at each frequency band in the STFT.

[0090] At block 620B, method 600 includes calculating a beamformer vector at each frequency band in the STFT in response to a frequency domain representation of a current window in the plurality of windows. The beamformer vector is calculated to maximize (e.g., amplify) signals in the DOA and minimize (e.g., attenuate) signals outside the DOA. To this end, various types of beamforming vectors may be used, some of which will be described below with respect to blocks 620D and 620E.

[0091] At block 620C, method 600 includes determining a denoised estimate by multiplying the beamformer vector by a corresponding beamformer weight vector of the STFT. The beamformer weight vector may be an MxN vector, where M represents the number of sensor (microphone) elements and N represents the direction of arrival (DOA).

[0092] At block 620D, the method 600 includes computing a minimum variance distortionless response (MVDR) beamformer vector at each frequency band in the STFT.

[0093] At block 620E, method 600 includes calculating a minimum power distortion-free response (MPDR) beamformer vector at each frequency band in the STFT.

[0094] At block 620F, the method 600 includes writing the covariance update step and the beamformer vector calculation step in a mathematically equivalent form to avoid computing matrix inversions and to be limited to algebraic operations.

[0095] At block 630, for a current one of the plurality of windows, the method 600 includes computing a voice activity detection (VAD) estimate for each frequency band in the STFT by comparing the magnitude of the single channel denoised estimate formed by the slow adaptive beamformer 202A to the magnitude of each input channel of the multi-channel audio stream.

[0096] At block 640 , for a current one of the plurality of windows, the method 600 includes selectively updating or not updating, by the fast adaptive beamformer 202B, a second complex-valued covariance matrix corresponding to the fast adaptive beamformer 202B in response to the VAD estimate indicating the absence or presence of speech, respectively.

[0097] In one aspect, block 640 may include block 640A.

[0098] At block 640A, the method 600 includes, in response to the amplitude of the single-channel denoised estimate being lower than the amplitude of a given input channel of the multi-channel audio stream by a threshold amount, treating a particular frequency band in the STFT corresponding to the single-channel denoised estimate as noise, and selectively refraining from updating the second complex-valued covariance matrix 220B1 corresponding to the fast adaptive beamformer 202B.

[0099] At block 650 , method 600 includes performing an inverse STFT.

[0100] At block 660, the method 600 includes controlling the device 200 based on the voice user interface 290 to perform a user-perceivable action in response to at least the output of the fast adaptive beamformer 202B. In one aspect, the output of the slow adaptive beamformer 202A may also be used along with the output of the fast adaptive beamformer 202B.

[0101] In one aspect, block 660 may include one or more of blocks 660A-660C.

[0102] At block 660A, the method 600 includes executing the method 600 by the automatic speech recognition system 207 configured to convert speech into commands to control the voice user interface-based device 200 to perform user-perceivable actions.

[0103] At block 660B, the method 600 includes weighting speech captured from the driver position more heavily than speech captured from the passenger position.

[0104] At block 660C, method 600 includes weighting speech captured from the position of the front seat passenger more heavily than speech captured from the position of the rear seat passenger.

[0105] Reference Fig.11 , showing a dual beamformer process 1100 according to various aspects of the present disclosure. The dual beamformer process 1100 is Figure 6-10 An overview of method 600 is provided.

[0106] The dual beamformer process 1100 involves a short time Fourier transform (SHFT) 1102 for transforming a multi-channel audio stream (audio input 1101) into a complex-valued frequency domain representation.

[0107] The dual beamformer process 1100 involves updating 1103 the slowly adaptive beamformer covariance matrix 502A1, computing 1104 the slowly adaptive beamformer weight vector based on the DOA, applying 1105 the slowly adaptive beamformer weight vector to the input STFT, and computing 1106 a voice activity detection (VAD) estimate.

[0108] The dual beamformer processing 1100 also includes updating 1121 the fast adaptive covariance matrix 502B1 in response to the VAD estimate indicating the presence of speech, calculating 1122 a fast adaptive beamformer weight vector based on the DOA, and calculating 1122 the fast adaptive beamformer weight vector based on the DOA, applying 1123 the fast adaptive beamformer weight vector to the input STFT, performing 1124 an inverse short time Fourier transform, and outputting 1131 audio to the device, for example, to control the device to perform an action such as a user-perceivable action. Such operations may include queries, commands (e.g., play music, tell me the weather, etc.), etc.

[0109] In one aspect, applying (1105 and / or 1123) the beamformer weight vector includes applying the weights in the vector to the magnitude of each frequency band of the STFT.

[0110] Various aspects of the present disclosure may take the form of a combination of hardware aspects, software aspects, or software and hardware aspects in whole or in part. In addition, as described herein, various aspects of the present disclosure (e.g., systems and methods) may take the form of a computer program product, which includes a computer-readable non-transient storage medium having computer-accessible instructions (e.g., computer-readable and / or computer-executable instructions), such as computer software encoded or otherwise contained in such a storage medium. These instructions may be read or otherwise accessed and executed by one or more processors to perform or allow the operations described herein to be performed. Instructions may be provided in any suitable form, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, assembly code, a combination of the above codes, etc. Any suitable computer-readable non-transient storage medium may be used to form a computer program product. For example, a computer-readable medium may include any tangible non-transient medium for storing information in a form that can be read or otherwise accessed by one or more computers or processors functionally coupled thereto. Non-transitory storage media may include read-only memory (ROM); random access memory (RAM); magnetic disk storage media; optical storage media; flash memory, etc.

[0111] Aspects of the present disclosure are described herein with reference to block diagrams and flow charts of methods, systems, apparatuses, and computer program products. It is understood that each block of the block diagrams and flow charts, and the combination of blocks in the block diagrams and flow charts, respectively, can be implemented by computer-accessible instructions. In some implementations, the computer-accessible instructions can be loaded or otherwise incorporated into a general-purpose computer, a special-purpose computer, or another programmable information processing device to produce a specific machine so that the operations or functions specified in the flow chart blocks can be implemented in response to the execution of the computer or processing device.

[0112] Unless expressly stated otherwise, any protocol, procedure, process, or method described herein should not be construed as requiring that its acts or steps be performed in a particular order. Therefore, if a process or method claim does not actually recite the order in which its acts or steps are to be followed, or if the claims or description of the present disclosure do not otherwise specifically recite that the steps are to be limited to a particular order, no order is intended to be inferred in any respect. This applies to any possible non-express basis for interpretation, including: problems of logic related to arrangement of steps or operational flow; plain meaning derived from grammatical organization or punctuation; the number or type of aspects described in the specification or drawings; or the like.

[0113] As used in this disclosure, including the accompanying drawings, the terms "component", "module", "system", etc. are intended to refer to a computer-related entity or an entity related to a device having one or more specific functions. The entity can be hardware, a combination of hardware and software, software, or software in execution. One or more such entities are also referred to as "functional elements". For example, a component can be a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. For example, an application running on a server or a network controller and a server or network control program can be a component. One or more components can reside in a process and / or an execution thread, and a component can be localized on a computer and / or distributed between two or more computers. In addition, these components can be executed from various computer-readable media on which various data structures are stored. These components can communicate through local and / or remote processes, such as according to signals with one or more data packets (e.g., data from a component that interacts with another component in a local system, a distributed system, and / or interacts with other systems through a network such as the Internet through signals). As another example, a component can be a device having a specific function provided by mechanical parts operated by electrical or electronic circuits, which can be controlled or otherwise operated by program code executed by a processor. As another example, a component can be a device that provides a specific function through an electronic component without mechanical parts, and the electronic component can include a processor to execute a program code that at least partially provides the function of the electronic component. As another example, an interface can include an I / O component or an application programming interface (API) component. Although the above examples are directed to aspects of components, aspects or features of the examples are also applicable to systems, modules, etc.

[0114] In addition, the word "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless specified otherwise or clear from the context, "X employs A or B" is intended to mean any of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, then "X employs A or B" holds true in any of the above cases. In addition, "a" and "an" as used in this specification and the drawings should be construed as "one or more" unless specified otherwise or the context clearly dictates a singular form.

[0115] In addition, the terms "example" and "for example" are used herein to indicate use as an example or illustration. Any aspect or design described herein as an "example" or related to an "for example" clause is not necessarily to be construed as being superior to other aspects or designs described herein. On the contrary, the use of the word "example" or "for example" is intended to present the concept in a concrete way. Unless the context clearly indicates otherwise, the terms "first", "second", "third", etc. used in the claims and the specification are for clarity only and do not necessarily indicate or imply any temporal or spatial order.

[0116] The term "processor" as used in this disclosure may refer to any computational processing unit or device, including processing circuits that can operate on data and / or signaling. Computational processing units or devices may include, for example, single-core processors; single processors with software multi-threaded execution capabilities; multi-core processors; multi-core processors with software multi-threaded execution capabilities; multi-core processors using hardware multi-threading technology; parallel platforms; and parallel platforms with distributed shared memory. In addition, the processor may include an integrated circuit, an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic controller (PLC), a complex programmable logic device (CPLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof, which is designed to perform the functions described herein. In some cases, the processor may utilize nanoscale architectures, such as transistors, switches, and gates based on molecules and quantum dots, to optimize space usage or improve the performance of user devices. The processor may also be implemented as a combination of computational processing units.

[0117] In addition, terms such as "memory", "data storage", "database", and any other information storage components related to the operation and function of the component refer to the "memory component" or entities contained in the "memory" or components that constitute the memory. It should be understood that the memory components described herein can be volatile memory or non-volatile memory, or can include volatile and non-volatile storage. In addition, the memory component can be detachable or attached to a functional element (e.g., a device, a server).

[0118] As a simple example, non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmed ROM (EPROM), electrically erasable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) used as an external cache. By way of illustration and not limitation, RAM comes in many forms, such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and direct Rambus RAM (DRRAM). In addition, the memory components of the systems or methods disclosed herein are intended to include, but are not limited to, including these and any other suitable types of memory.

[0119] The various aspects described herein can be implemented as methods, apparatuses, or articles of manufacture using standard programming and / or engineering techniques. In addition, the various aspects disclosed herein can also be implemented by program modules or other types of computer program instructions stored in a memory device and executed by a processor, or other combinations of hardware and software or hardware and firmware. Such program modules or computer program instructions can be loaded onto a general-purpose computer, a special-purpose computer, or another type of programmable data processing device to produce a machine, so that the instructions executed on the computer or other programmable data processing device create a means for implementing the functions disclosed herein.

[0120] The term "article of manufacture" as used herein is intended to encompass a computer program accessible from any computer-readable device, carrier, or medium. For example, computer-readable media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., compact disks (CDs), digital versatile disks (DVDs), Blu-ray disks (BDs), etc.), smart cards, and flash memory devices (e.g., cards, sticks, key drives, etc.).

[0121] The detailed description set forth herein in conjunction with the accompanying drawings is intended as a description of various configurations or implementations and is not intended to represent the only configuration or implementation in which the concepts described herein may be practiced. The detailed description includes specific details for the purpose of providing a thorough understanding of the various concepts. However, it will be appreciated by those skilled in the art that these concepts may be practiced without these specific details or variations of these specific details. In some cases, well-known components are shown in block diagram form, and some blocks may represent one or more well-known components.

[0122] The previous description of the present disclosure is provided to enable those skilled in the art to make or use the present disclosure. Various modifications to the present disclosure will be apparent to those skilled in the art, and the common principles defined herein may be applied to other variations without departing from the scope of the present disclosure. In addition, although the elements of the described aspects may be described or claimed in the singular, the plural form may be considered unless a limitation to the singular is explicitly stated. In addition, unless otherwise stated, all or a portion of any aspect may be used together with all or a portion of any other aspect. Therefore, the present disclosure is not limited to the examples and designs described herein, but should conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for voice control, comprising: transforming the multi-channel audio stream into a complex-valued frequency domain representation using a short-time Fourier transform (STFT) applied to data in each of a plurality of windows aligned across each input channel of the multi-channel audio stream, Wherein, for a current one of the multiple windows, the method comprises: updating a first complex-valued covariance matrix corresponding to the slow adaptive beamformer and forming a single-channel denoised estimate for each frequency band in the STFT; computing a voice activity detection (VAD) estimate for each frequency band in the STFT by comparing the magnitude of the single channel denoised estimate to the magnitude of each input channel of the multi-channel audio stream; and selectively updating or not updating a second complex-valued covariance matrix corresponding to the fast adaptive beamformer in response to the VAD estimate indicating the presence or absence of speech, respectively; and The voice user interface based device is controlled by a hardware processor to perform a user-perceivable action in response to at least the output of the fast adaptive beamformer.

2. The method of claim 1, further comprising configuring the slow adaptive beamformer and the fast adaptive beamformer to use different equations for update rules and different adaptation speeds.

3. The method of claim 1, further comprising configuring the slow adaptive beamformer and the fast adaptive beamformer to both use the same equation as an update rule but have different adaptation speeds.

4. The method according to claim 3, further comprising: configuring the slowly adaptive beamformer to slowly adapt to an acoustic background including background noise in the multi-channel audio stream; and A fast adaptive beamformer is configured to quickly adapt to speech in the multi-channel audio stream.

5. The method according to claim 1, wherein the method further comprises: In response to the amplitude of the single-channel denoised estimate being lower than the amplitude of a given input channel of the multi-channel audio stream by a threshold amount, treating a particular frequency band in the STFT corresponding to the single-channel denoised estimate as noise and selectively refraining from updating a second complex-valued covariance matrix corresponding to the fast adaptive beamformer.

6. The method of claim 1, further comprising applying the method in a beamforming strategy configured to use a correlation matrix of any one of the multi-channel audio streams and noise.

7. The method of claim 1, wherein forming the single channel denoised estimate comprises computing a beamformer vector at each frequency band in the STFT.

8. The method of claim 7, wherein the beamformer vector is calculated at each frequency band in the STFT in response to a frequency domain representation of a current window of the plurality of windows.

9. The method of claim 7, wherein the denoised estimate is determined by multiplying the beamformer vector by a corresponding beamformer weight vector of the STFT.

10. The method of claim 1, wherein forming the single channel denoised estimate comprises computing a minimum variance distortionless response (MVDR) beamformer vector at each frequency band in the STFT.

11. The method of claim 1 , wherein forming the single channel denoised estimate comprises computing a minimum power distortionless response (MPDR) beamformer vector at each frequency band in the STFT.

12. The method of claim 1, further comprising receiving a multi-channel audio stream and a desired source direction of arrival (DOA) from a microphone array.

13. The method of claim 1, wherein the transform comprises applying frequency domain processing to the complex-valued frequency domain representation to obtain processed audio, and the method further comprises transforming the processed video from the frequency domain to the time domain using an inverse STFT and overlap-add process and providing a time domain representation.

14. The method of claim 13, further comprising performing the method online using a first buffer configured to collect input samples and a second buffer configured to feed out processed audio.

15. The method of claim 1 further comprises executing the method via an automatic speech recognition system, wherein the automatic speech recognition technology is operably coupled to a voice user interface of the voice user interface-based device and is configured to convert the speech into a command to control the audio user interface-based device to perform the user-perceivable action.

16. The method of claim 1, wherein the window is an exponential window.

17. The method of claim 1, wherein the windows are rectangular windows of different lengths, wherein the slow adaptive beamformer uses a longer time window than the fast adaptive beamformer to capture temporal stationarity.

18. The method of claim 1, wherein the method is performed off-line.

19. The method of claim 1, further comprising writing the covariance update step and the beamformer vector calculation step in a mathematically equivalent form to avoid computing matrix inversions and to be limited to algebraic operations.

20. The method of claim 1 further comprising weighting speech captured from a driver position more heavily than speech captured from a passenger position.

21. The method of claim 1 further comprising weighting speech captured from a front seat passenger's position more heavily than speech captured from a rear seat passenger's position.

22. A computer program product configured to implement voice control, the computer program product comprising one or more non-transitory computer-readable media having instructions stored thereon, the instructions, when executed by one or more processors, causing the one or more processors to perform a method, alone or in combination, comprising: transforming the multi-channel audio stream into a complex-valued frequency domain representation using a short-time Fourier transform (STFT) applied to data in each of a plurality of windows aligned across each input channel of the multi-channel audio stream, Wherein for a current one of the plurality of windows, the method comprises: updating a first complex-valued covariance matrix corresponding to the slow adaptive beamformer and forming a single-channel denoised estimate for each frequency band in the STFT; computing a voice activity detection (VAD) estimate for each frequency band in the STFT by comparing the magnitude of the single channel denoised estimate to the magnitude of each input channel of the multi-channel audio stream; and selectively updating or not updating a second complex-valued covariance matrix corresponding to the fast adaptive beamformer in response to the VAD estimate indicating the presence or absence of speech, respectively; and The voice user interface based device is controlled by a hardware processor to perform a user-perceivable action in response to at least the output of the fast adaptive beamformer.

23. A voice control system, comprising: one or more memories, alone or in combination, having instructions; one or more processors, each coupled to at least one of the one or more memories and configurable to execute the instructions to: transforming the multi-channel audio stream into a complex-valued frequency domain representation using a short-time Fourier transform (STFT) applied to data in each of a plurality of windows aligned across each input channel of the multi-channel audio stream, Wherein, for a current one of the plurality of windows, the one or more processors may be configured to execute the instructions to: updating a first complex-valued covariance matrix corresponding to the slow adaptive beamformer and forming a single-channel denoised estimate for each frequency band in the STFT; computing a voice activity detection (VAD) estimate for each frequency band in the STFT by comparing the magnitude of the single channel denoised estimate to the magnitude of each input channel of the multi-channel audio stream; and selectively updating or not updating a second complex-valued covariance matrix corresponding to the fast adaptive beamformer in response to the VAD estimate indicating the presence or absence of speech, respectively; and In response to at least the output of the fast adaptive beamformer, the voice user interface based device is controlled to perform a user-perceivable action.

Images