Speech enhancement using clustering of cue

Abstract

To provide a method and a system for enhancing speech in echo environment.SOLUTION: A method includes: a step for receiving or generating a sound sample 202 indicating a sound signal which an array 201 of a microphone receives during a prescribed time period; a step 203 for frequency-converting a sound sample for providing a sample whose frequency is converted; a step 204 for clustering the sample whose frequency is converted to a speaker based on a spatial cue related to the received sound signal and an acoustic cue related to the speaker for providing a speaker-related cluster; a step 205 for determining a relative transfer function for each speaker for providing a speaker-related relative transfer function; a step 206 for applying multiple input multiple output MIMO beam formation operation to the speaker-related relative transfer function for providing a beam-formed signal; and a step 207 for inverse-frequency-converting the beam-formed signal for providing a voice signal.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] (Background technology) The performance of voice enhancement modules is determined by their ability to filter out all interfering signals, leaving only the desired voice signal. Interfering signals can be, for example, other speakers, noise from air conditioning, music, motor noise (e.g. in a car or airplane), and loud crowd noise, also known as "cocktail party noise". The performance of voice enhancement modules is usually judged by their ability to improve the speech-to-noise-ratio (SNR) or speech-to-interference-ratio (SIR), which reflects the ratio (often in dB scale) of the power of the desired voice signal to the total power of noise and other interfering signals, respectively. [Prior art documents] [Non-patent literature]

[0002] [Non-Patent Document 1] "Lessons in Digital Estimation Theory" by Jerry M. Mendel [Non-Patent Document 2] “New Features for Emotional Speech Recognition” by Palo et. al. Summary of the Invention [Problem to be solved by the invention]

[0003] There is an increasing need to implement voice reinforcement in reverberant environments. [Means for solving the problem]

[0004] A method for sound enhancement may be provided, the method including the steps of receiving or generating sound samples representative of sound signals received by an array of microphones during a given time period; frequency converting the sound samples to provide frequency converted samples; clustering the frequency converted samples to speakers to provide speaker-related clusters, the clustering step may be based on (i) spatial cues associated with the received sound signals and (ii) acoustic cues associated with the speakers; determining relative transfer functions for each individual speaker of the speakers to provide speaker-related relative transfer functions; applying a multiple input multiple output (MIMO) beamforming operation to the speaker-related relative transfer functions to provide a beamformed signal; and inverse frequency converting the beamformed signal to provide a voice signal.

[0005] The method may include generating an audio cue associated with a speaker.

[0006] Generating the acoustic cues may include searching for keywords in the sound samples and extracting the acoustic cues from the keywords.

[0007] The method may include extracting spatial cues associated with the keywords.

[0008] The method may include using spatial queries associated with the keywords as clustering seeds.

[0009] The acoustic cues may include pitch frequency, pitch intensity, one or more harmonics of the pitch frequency, and intensity of one or more harmonics of the pitch frequency.

[0010] The method may include associating a reliability attribute with each pitch and determining that a speaker that may be associated with a pitch may be silenced when the reliability of the pitch falls below a predefined threshold.

[0011] The clustering step may include processing the frequency transformed samples to provide acoustic and spatial cues, using the acoustic cues to keep track of the state of the speakers, segmenting the spatial cues of individual frequency components of the frequency transformed signals into groups, and assigning to each group of frequency transformed signals the acoustic cue associated with the currently active speaker.

[0012] The assigning step may include a step of calculating, for each group of frequency transformed signals, a cross-correlation between elements of iso-frequency lines of the time-frequency map that belong to other lines of the time-frequency map and have elements that can be associated with the group of frequency transformed signals.

[0013] The tracking step may include applying an extended Kalman filter.

[0014] The tracking step may include applying multiple hypothesis tracking.

[0015] The tracking step may include applying a particle filter.

[0016] The segmenting step may include assigning a single frequency component associated with a single time frame to a single speaker.

[0017] The method may include monitoring at least one monitored acoustic feature from among speech rate, speech intensity and emotional voicing.

[0018] The method may include providing the at least one monitored acoustic feature to an extended Kalman filter.

[0019] The frequency transformed samples can be arranged in multiple vectors, one vector for each microphone of the microphone array, and the method can include calculating an intermediate vector by weighted averaging the multiple vectors, and searching for acoustic cue candidates by ignoring elements of the intermediate vector having values ​​that may be below a predefined threshold.

[0020] The method may include determining the predefined threshold to be three times the standard deviation of the noise.

[0021] A non-transitory computer-readable medium may be provided that stores instructions that, when executed by a computerized system, will cause the computerized system to receive or generate sound samples representing sound signals received by an array of microphones during a given period of time, frequency convert the sound samples to provide frequency converted samples, cluster the frequency converted samples into speakers to provide speaker-related clusters (the clustering may be based on (i) spatial cues associated with the received sound signals and (ii) acoustic cues associated with the speakers), determine relative transfer functions for each individual speaker of the speakers to provide speaker-related relative transfer functions, apply a multiple-input multiple-output (MIMO) beamforming operation to the speaker-related relative transfer functions to provide a beamformed signal, and inverse frequency convert the beamformed signal to provide an audio signal.

[0022] The non-transitory computer readable medium may store instructions for generating an audio cue associated with a speaker.

[0023] Generating the acoustic cues can include searching for keywords in sound samples and extracting the acoustic cues from the keywords.

[0024] Generating the acoustic cues can include searching for keywords in sound samples and extracting the acoustic cues from the keywords.

[0025] The non-transitory computer readable medium may store instructions for extracting spatial cues associated with keywords.

[0026] The non-transitory computer readable medium can store instructions for using spatial queries associated with keywords as clustering seeds.

[0027] The acoustic cues may include pitch frequency, pitch intensity, one or more harmonics of the pitch frequency, and intensity of one or more harmonics of the pitch frequency.

[0028] The non-transitory computer-readable medium can store instructions for associating reliability attributes with individual pitches and determining that a speaker that may be associated with a pitch may be silenced when the reliability of the pitch falls below a defined threshold.

[0029] Clustering may include processing the frequency-converted samples to provide acoustic and spatial cues, constantly tracking the state of the speakers using the acoustic cues, segmenting the spatial cues of individual frequency components of the frequency-converted signals into groups, and assigning to each group of frequency-converted signals the acoustic cue associated with the currently active speaker.

[0030] The assignment may include, for each group of frequency transformed signals, calculating a cross-correlation between elements of iso-frequency lines of the time-frequency map that belong to other lines of the time-frequency map and have elements that can be associated with the group of frequency transformed signals.

[0031] The tracking can include applying an extended Kalman filter.

[0032] The tracking may include applying multiple hypothesis tracking.

[0033] The tracking can include applying a particle filter.

[0034] Segmenting may include assigning a single frequency component associated with a single time frame to a single speaker.

[0035] The non-transitory computer readable medium can store instructions for monitoring at least one monitored acoustic feature from among voice rate, voice intensity, and emotional voicing.

[0036] The non-transitory computer-readable medium can store instructions for providing at least one monitored acoustic feature to an extended Kalman filter.

[0037] The frequency transformed samples can be arranged in multiple vectors, one vector for each microphone of the microphone array, and the non-transitory computer readable medium can store instructions for calculating an intermediate vector by weighted averaging the multiple vectors and searching for acoustic cue candidates by ignoring elements of the intermediate vector having values ​​that may be below a predefined threshold.

[0038] The non-transitory computer readable medium can store instructions for determining the pre-defined threshold to be three times the standard deviation of the noise.

[0039] A computerized system may be provided that may include an array of microphones, a storage device, and a processor. The processor may be configured to receive or generate sound samples representative of sound signals received by the array of microphones during a given time period, frequency convert the sound samples to provide frequency converted samples, cluster the frequency converted samples into speakers to provide speaker-related clusters (the clustering may be based on (i) spatial cues associated with the received sound signals and (ii) acoustic cues associated with the speakers), determine relative transfer functions for each individual speaker of the speakers to provide speaker-related relative transfer functions, apply a multiple-input multiple-output (MIMO) beamforming operation to the speaker-related relative transfer functions to provide beamformed signals, and inverse frequency convert the beamformed signals to provide audio signals, and the storage device may be configured to store at least one of the sound samples and the audio signals.

[0040] The computerized system may not include an array of microphones, but may be capable of receiving signals from the array of microphones that are representative of the sound signals received by the array of microphones during a given period of time.

[0041] The processor may be configured to generate an audio cue associated with the speaker.

[0042] Generating the acoustic cues can include searching for keywords in sound samples and extracting the acoustic cues from the keywords.

[0043] The processor may be configured to extract spatial cues associated with the keywords.

[0044] The processor may be configured to use spatial queries associated with the keywords as clustering seeds.

[0045] The acoustic cues may include pitch frequency, pitch intensity, one or more harmonics of the pitch frequency, and intensity of one or more harmonics of the pitch frequency.

[0046] The processor may be configured to associate a reliability attribute with each pitch and to determine that a speaker that may be associated with a pitch may be silenced when the reliability of the pitch falls below a predefined threshold.

[0047] The processor may be configured to process and cluster the frequency-converted samples to provide acoustic and spatial cues, keep track of the state of the speakers using the acoustic cues, segment the spatial cues of individual frequency components of the frequency-converted signals into groups, and assign to each group of frequency-converted signals an acoustic cue associated with the currently active speaker.

[0048] The processor may be configured to assign, for each group of frequency transformed signals, by calculating cross-correlation between elements of iso-frequency lines of the time-frequency map that belong to other lines of the time-frequency map and have elements that can be associated with the group of frequency transformed signals.

[0049] The processor may be configured to perform tracking by applying an extended Kalman filter.

[0050] The processor may be configured to track by applying multiple hypothesis tracking.

[0051] The processor may be configured to track by applying a particle filter.

[0052] The processor may be configured to segment by assigning a single frequency component associated with a single time frame to a single speaker.

[0053] The processor may be configured to monitor at least one of the monitored acoustic features: speech rate, speech intensity, and emotional voicing.

[0054] The processor may be configured to provide the at least one monitored acoustic feature to an extended Kalman filter.

[0055] The frequency transformed samples may be arranged in multiple vectors, one vector for each microphone in the microphone array, and the processor may be configured to calculate an intermediate vector by weighted averaging the multiple vectors and to search for candidate acoustic cues by ignoring elements of the intermediate vector having values ​​that may be below a predefined threshold.

[0056] The processor may be configured to determine the predefined threshold to be three times the standard deviation of the noise.

[0057] In order to understand the invention and to better see how it can be carried out in practice, preferred embodiments will now be described, by way of non-limiting example only, with reference to the accompanying drawings, in which: [Brief description of the drawings]

[0058] [Figure 1] FIG. [Diagram 2] FIG. 1 illustrates an example of a method. [Diagram 3] FIG. 3 shows an illustration of the clustering step of the method of FIG. 2. [Figure 4] FIG. 1 illustrates an example of pitch detection on a time-frequency map. [Diagram 5] FIG. 13 illustrates an example of a time-frequency-cue map. [Figure 6] FIG. 2 illustrates an example of a voice recognition chain for offline training. [Figure 7] FIG. 2 shows an example of a voice recognition chain in real-time training. [Figure 8]FIG. 13 illustrates an example of a training mechanism. [Figure 9] FIG. 1 illustrates an example of a method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0059] All references to a system shall apply mutatis mutandis to a method performed by the system and / or to a non-transitory computer readable medium storing instructions that, when executed by the system, cause the system to perform the method.

[0060] All references to a method shall apply mutatis mutandis to a system configured to perform the method and / or to a non-transitory computer readable medium storing instructions that, when executed by the system, cause the system to perform the method.

[0061] All references to a non-transitory computer readable medium shall apply, mutatis mutandis, to methods performed by a system and / or performed by a system configured to execute instructions stored on the non-transitory computer readable medium.

[0062] The term "and / or" means additionally or alternatively.

[0063] The term "system" means a computerized system.

[0064] Audio enhancement methods focus on extracting audio signals from desired sources (speakers) when the signals are disrupted by noise and other speakers. In anechoic environments, spatial filtering in the form of directional beamforming is effective. However, in reverberant environments, audio from individual sources is smeared across several directions and is not necessarily continuous, which does not take advantage of conventional beamformers. Addressing this problem using a beamformer based on transfer functions (TFs), or using relative transfer functions (RTFs) as the TFs themselves, is a promising direction. However, in multi-speaker environments, the ability to predict the RTFs for each speaker remains a challenge when audio signals are captured simultaneously. A solution is provided that involves tracking acoustic and spatial cues to cluster simultaneous speakers, thereby facilitating the prediction of the RTFs of speakers in reverberant environments.

[0065] A loudspeaker clustering algorithm is provided that assigns each frequency component to its original loudspeaker, especially in a multi-loudspeaker reverberant environment. The clustering algorithm provides the necessary conditions for the RTF estimator to work properly in a multi-loudspeaker reverberant environment. The prediction of the RTF matrix is ​​then used to calculate the weight vector of a transfer function based linear constrained minimum variance (TF-LCMV) beamformer (see equation (10) below), thus satisfying the necessary conditions for TF-LCMV to work. It is assumed that each human loudspeaker is endowed with a different pitch, and thus pitch is a bijective indicator for the loudspeakers. Multi-pitch detection is known to be a challenging task, especially in a noisy reverberant multi-loudspeaker environment. To address this challenge, the W-Disjoint Orthogonality (W-DO) assumption is adopted, and a set of spatial cues, such as signal strength, azimuth and elevation angles, are used as additional features. To overcome temporarily inactive speakers and pitch changes, an acoustic cue - the pitch value - is constantly tracked using an extended Kalman filter (EKF), and spatial cues are used to segment the last L frequency components and assign each individual frequency component to a different source. The results of the EKF and segmentation are combined by cross-correlation to facilitate the clustering of frequency components to a specific speaker with a specific pitch.

[0066] Figure 1 illustrates the paths that frequency components of an audio signal travel in a reverberant environment from a human speaker 11 to a microphone array 12. Walls 13 and other elements in the environment 14 reflect the impinging signal with attenuation and angles of reflection determined by the material and texture of the wall. Different frequency components of the human voice will follow different paths. These paths may be direct paths 15, which lie on the shortest path between the human speaker 11 and the microphone array 12, or indirect paths 16, 17. It should be noted that frequency components may travel along one or more paths.

[0067] FIG. 2 illustrates the algorithm. The signal is acquired by a microphone array 201 containing M≧2 microphones, M=7 microphones being one example. The microphones can be spread in a series of clusters, for example evenly spaced on a line, a circle or a sphere, or even non-uniformly spaced to form any shape. The signals from the individual microphones are sampled, digitized and stored in M ​​frames, each containing T consecutive samples 202. The size of the frame T can be chosen to be large enough so that the short-time Fourier Transform (STFT) is accurate, but short enough so that the signal is stationary along an equivalent time duration. A typical value for T is 4,096 samples for a sampling rate of 16 kHz, i.e. a frame is equivalent to ¼ second. Successive frames are often overlapped with each other to improve tracking of signal features over time. A typical overlap is 75%, i.e. a new frame starts every 1,024 samples. T may range, for example, between 0.1 and 2 seconds, thereby providing 1024 to 32768 samples for a 16 kHz sampling rate. The samples may also be called sound samples, which represent the sound signals received by the array of microphones during the time period T.

[0068] Each frame is transformed in 203 into the frequency domain by applying a Fourier transform or a variant of the Fourier transform such as the short-time Fourier transform (STFT), constant-Q transform (CQT), logarithmic Fourier transform (LFT), filter bank, etc. Several techniques such as windowing and zero-padding can also be applied to control the framing effect. By 203, M complex-valued vectors of length K are obtained. For example, if the array contains 7 microphones, 7 vectors are prepared, which are registered by the frame time index l. K is the number of frequency bins, which is determined by the frequency transform. For example, when using the regular STFT, K=T, which is the length of the buffer. The output of step 203 can also be called the frequency transformed signal.

[0069] The audio signals are clustered into different speakers at 204. The clusters can be called speaker-related clusters. Unlike prior art works that cluster speakers based on direction only, 204 deals with multiple speakers in a reverberation room, so that signals from different directions can be assigned to the same speaker by direct and indirect paths. The proposed solution suggests the use of a set of acoustic cues, such as pitch frequency and intensity, and its harmonic frequencies and intensity, in addition to a set of spatial cues, such as the direction (azimuth and elevation) and intensity of the signal at one of the microphones. The pitch and one or more of the spatial cues serve as state vectors for tracking algorithms such as Kalman filters and their variants, multiple hypothesis tracking (MHT) or particle filters, which are used to track this state vector and to assign the individual tracks to different speakers.

[0070] All these tracking algorithms use a model describing the dynamics of the state vector in time, so that when measurements of the state vector are missing or contaminated by noise, the tracking algorithms compensate for this use of a dynamic model and simultaneously update the model parameters. The output of this stage is a vector that assigns the individual frequency components at a given time l to the individual speakers. 204 is explained in more detail in Figure 3.

[0071] At 205, an RTF estimator is applied to the data in the frequency domain. This stage results in a set of RTFs, each of which is registered to its associated speaker. The registration process is performed using the clustered array from the clustered speakers 204. This set of RTFs can also be referred to as speaker-related relative transfer functions.

[0072] The MIMO beamformer 206 reduces the energy of noise and interference signals with respect to the energy of the wanted speech signal by spatial filtering. The output of step 206 can also be called a beamformed signal. The beamformed signal is then sent to an inverse frequency transform 207 to produce a continuous speech signal in the form of a stream of samples, which in turn is forwarded 208 to other elements such as a speech recognition system, a communication system and a recording device.

[0073] In a preferred embodiment of the present invention, keyword spotting 209 can be used to improve the performance of the clustering block 204. The frames from 202 are searched for predefined keywords (e.g., "Alexa today" or "Okay Google"). Once a keyword is spotted in the stream of frames, the acoustic cues of the speaker are extracted, such as pitch frequency and intensity, as well as its harmonic frequencies and intensities. Also, features of the path that individual frequency components took to reach the microphone array 201 are extracted. These features are used by the clustering speaker 204 as seeds for the desired cluster of speakers. Seeds are initial guesses for the initial parameters of the cluster, such as the cluster centroid, radius, and statistics for centroid-based clustering algorithms such as K-means, PSO, and 2KPM. Another example is the subspace basis for subspace-based clustering.

[0074] Figure 3 illustrates the loudspeaker clustering algorithm. It is assumed that each loudspeaker is endowed with a different set of acoustic cues, such as pitch frequency and intensity, as well as its harmonic frequencies and intensities, and thus the set of acoustic cues is a bijective indicator for the loudspeakers. Acoustic cue detection is known to be a challenging task, especially in noisy and reverberant multi-speaker environments. To address this challenge, spatial cues, for example in the form of signal strength, azimuth and elevation angles, are used. To overcome temporarily inactive loudspeakers and changes in acoustic cues, the acoustic cues are constantly tracked using filters such as spatial filters and Extended Kalman Filters (EKF), and the spatial cues are also used to segment the frequency components among different sources. To facilitate the clustering of frequency components into specific loudspeakers with specific pitches, the EKF and segmentation results are combined by cross-correlation.

[0075] At 31, as an example of a preferred embodiment, potential acoustic cues in the form of pitch frequency are detected. First, a time-frequency map is prepared using the frequency transform of the buffers from the individual microphones, calculated at 203. Then, the absolute values ​​of each of the M complex-valued vectors of length K are weighted-averaged using some weighting factors that can be determined to reduce artifacts in some microphones. This results in a single real vector of length K. In this vector, values ​​that are greater than a given threshold μ are extracted, while the remaining elements are discarded. The threshold μ is often three times the standard deviation of the noise and is adaptively chosen not to fall below a certain value that depends on the electrical parameters of the system, especially the number of effective bits of the sampled signal. Values ​​whose frequency indices are in the range [k_min, k_max] are defined as candidates for the pitch frequency. The variables k_min and k_max are typically 85Hz and 2550Hz, respectively, since a typical adult male has a fundamental frequency between 85Hz and 1800Hz, and a typical adult female has a fundamental frequency between 165Hz and 2550Hz. Then, each pitch candidate is verified by searching its higher harmonics. The presence of the second and third harmonics may be a prerequisite for a candidate pitch to be detected as a reasonable pitch with a confidence of R (e.g. R=10). If higher harmonics (e.g. 4th and 5th) are present, the confidence of the pitch increases, e.g. double per harmonic. An example can be found in FIG. 4. In a preferred embodiment of the invention, the pitch 32 of the desired speaker is provided by 210 using the keyword pronounced by the desired speaker. The provided pitch 32 is added to the list with the highest possible confidence, e.g. R=1000.

[0076] At 33, an Extended Kalman Filter (EKF) is applied to the pitch from 31. As noted by the Wikipedia entry for the Extended Kalman Filter (www.wikipedia.org / wiki / Extended_Kalman_filter), the Kalman filter has a state transition equation and an observation model. The state transition equation for the discrete computation is x k =f(x k-1 , u k )+w k (1)

[0077] Also, the observation model for discrete calculation is z k =h(x k )+v k (2) In the above formula, x k is a state vector containing parameters that (partially) describe the state of the system, and u k is a vector of external inputs that provide information about the state of the system, and w k and v k are the process and observation noises. The time updater of the extended Kalman filter can predict the next state using the prediction equation, and the detected pitch can be calculated using the following type of equation, y k =z k -h(x k|k+1 ) (3) You can update the variables by comparing the actual measurements with the predicted measurements using z k is the detected pitch, and y k is the error between the measured and predicted pitch.

[0078] 33, the individual trajectories are fitted to a model f(x k , u k ) The input of the model is the past state vector x k(either one state vector or more state vectors), and any external inputs u that affect the pitch dynamics, such as voice rate, voice intensity, and emotional voicing. k The elements of the state vector x can quantitatively describe the pitch. For example, the state vector of pitch can include, among other things, the pitch frequency, the magnitude of the first harmonic, and the frequencies and magnitudes of the higher harmonics. A vector function f(x k , u k ) can be used to predict the state vector x at some predetermined time k+1 prior to the current time. An exemplary implementation of the dynamic model in the EKF can include a time update equation (aka prediction equation) as described in the book "Lessons in Digital Estimation Theory" by Jerry M. Mendel, which is incorporated herein by reference.

[0079] For example, the three-term state vector

number

[0080] An example state vector model for pitch is

number

[0081] This describes a model that assumes constant pitch all the time. In a preferred embodiment of the present invention, voice rate, voice intensity and emotional voicing are continuously monitored using speech recognition algorithms known in the art, and an external input u is used to refine the time update stage of the EKF. kEmotional speech methods are known in the art. See, for example, "New Features for Emotional Speech Recognition" by Palo et. al.

[0082] Each track is endowed with a confidence field that is inversely proportional to the time the track evolves using only time updates. When the confidence of a track falls below some confidence threshold, e.g., ρ, which represents 10 seconds of undetected pitch, the track is defined as dead, meaning that the respective speaker is no longer active. On the other hand, the appearance of a new measurement (pitch detection) that cannot be assigned to any existing track will start a new track.

[0083] At 34, spatial cues are extracted from the M frequency transformed frames. As in 31, the L most recent vectors are saved for analysis using correlation in time. This results in a time-frequency-Cue (TFC) map for each of the M microphones, which is a 3-dimensional array of size LxKxP, where P=M-1. TFCs are illustrated in FIG. 5.

[0084] At 35, the spatial cues of individual frequency components in the TFC are segmented. The idea is that frequency components can come from different speakers along L frames, and this can be observed by comparing the spatial cues. However, at a single frame time l, it is assumed that the frequency components come from one speaker due to the W-DO assumption. The segmentation can be performed using any known method in the literature used for clustering, such as K nearest neighbors (KNN). The clustering is done by assigning an index to each cell in A that indicates the cluster to which the cell (k, l) belongs.

number

[0085] At 36, the frequency components of the signal are classified such that each frequency component is assigned to a particular pitch listed in the list of pitches tracked by the EKF and is active according to its reliability. This means that the kth line of the time-frequency map (see FIG. 4) is assigned to one of the pitches and a particular cluster index c o This is done by calculating the sample cross-correlation between all values ​​with (j, l). This is done for all cluster indices. The sample cross-correlation is

number

[0086] where A is the time-frequency map, k is the index of the line to which one of the pitches belongs, j is any other line of A, and L is the number of columns of A. After calculation of the sample cross-correlation between each pitch and each of the clusters in the other lines, the cluster c1 in the line j1 with the highest cross-correlation is sorted with the respective pitch, then the cluster c2 in the line j2 with the second highest cross-correlation is sorted with the respective pitch, and so on. This process is repeated until the sample cross-correlation is below some threshold k, which can be adaptively set, for example, as 0.5x (the average energy of the signal at a single frequency). 35 gives a set of groups of frequencies endowed with each pitch frequency.

[0087] FIG. 4 illustrates an example of pitch detection on a time-frequency map. 41 is the time axis, represented by parameter l, and 42 is the frequency axis, described by parameter k. Each column in this two-dimensional array is a real-valued vector of length K extracted in 31 after averaging the absolute values ​​of the M frequency-transformed buffers in time l. For correlation analysis in time, the L most recent vectors are stored in a two-dimensional array of size KxL. In 43, two pitches are represented by diagonals in different directions. Pitch k=2, which has its harmonics at k=4, 6, 8, has confidence R=20 since the fourth harmonic is present, and pitch k=3, which has its harmonics at k=6, 9, has confidence R=10. In 44, pitch k=3 is inactive and only k=2 is active. However, the confidence of pitch k=2 is reduced to R=10 since the fourth harmonic is not detected (below threshold μ). At 45 the k=3 pitch is again active and k=2 is inactive. At 46 a new pitch candidate k=4 appears but only its second harmonic has been detected, so it is not detected as a pitch. At 47 the k=3 pitch is inactive and no pitch is detected.

[0088] 5 illustrates a TFC-map, whose axes are frame index (time) 51, frequency components 52, and spatial cues 53, which may be complex-valued, representing for example the direction (azimuth and altitude) from which the individual frequency components arrive and the strength of the components. Once a frame of index l has been processed and transferred to the frequency domain, the frequency components

number

number

number

[0089] appendix The performance of audio enhancement modules is determined by their ability to filter out all interfering signals, leaving only the desired audio signal. Interfering signals can be, for example, other speakers, noise from air conditioning, music, motor noise (e.g. in a car or airplane), and large crowd noise, also known as "cocktail party noise". The performance of audio enhancement modules is usually judged by their ability to improve the Speech-to-Noise Ratio (SNR) or Speech-to-Interference Ratio (SIR), which reflects the ratio (often in dB scale) of the power of the desired audio signal to the total power of noise and other interfering signals, respectively.

[0090] If the acquisition module contains a single microphone, the method is called single-microphone speech enhancement and is often based on statistical features of the signal itself in the time-frequency domain, such as single-channel spectral subtraction, minimum variance distortionless response (MVDR) and spectral prediction with echo-cancelation. If several microphones are used, the acquisition module is often called microphone array and the method is called multi-microphone speech enhancement. Many of these methods exploit the difference between the signals captured simultaneously by the microphones. An established method is beamforming, which sums the signals from the microphones after multiplying the individual signals by weighting coefficients. The purpose of the weighting coefficients is to average out the interfering signals to condition the important signal.

[0091] Beamforming, in other words, is a method of creating a spatial filter that algorithmically increases the power of signals emanating from a given location in space (the desired signal from the desired speaker) and decreases the power of signals emanating from other locations in space (the interfering signals from other sources), thereby increasing the SIR at the beamformer output.

[0092] The delay-and-sum beamformer (DSB), which requires the use of DSB weighting coefficients, is composed of counter delays necessitated by the different paths that the desired signal travels from its source to each of the microphones in the array. DSB is limited to signals coming from each single direction, such as in an anechoic environment. Thus, in reverberant environments where signals from the same source travel along different paths to the microphones and arrive at the microphones from multiple directions, DSB performance is typically inadequate.

[0093] To mitigate the shortcomings of DSB in reverberant environments, beamformers can use more complex acoustic transfer functions (ATFs) that represent the direction (azimuth and altitude) from which individual frequency components arrive at a particular microphone from a given source. The single direction of arrival (DOA) assumed by DSB and other DOA-based methods often does not hold true in reverberant environments, where components of the same audio signal arrive from different directions. This is due to the different frequency responses of physical elements in the reverberant environment, such as walls, furniture, and people. The ATF in the frequency domain is a vector that assigns complex numbers to individual frequencies in the Nyquist bandwidth. The absolute value represents the gain of the path associated with this frequency, and the phase indicates the phase added to the frequency component along the path.

[0094] The prediction of the ATF between a given point in space and a given microphone can be performed by using a loudspeaker emitting a known signal, placed at the given point. By simultaneously acquiring the signal from the input of the loudspeaker and the output of the microphone, the ATF can be easily predicted. The loudspeaker can be placed at one or more positions where a human speaker will be present during operation of the system. This method produces a map of the ATF for each point in space, or more practically for each point on a grid. The ATF of points not included in the grid is approximated using interpolation. However, this method has major drawbacks. First, the system needs to be calibrated for each installation, which makes it impractical. Second, there are acoustic differences between human and electronic speakers, which make the measured ATF deviate from the real ATF. Third, there is the complexity of the huge number of measurements of the ATF, especially when the speaker orientation is also taken into account, and fourth, there are possible errors due to changes in the environment.

[0095] A more practical alternative to ATF is the relative transfer function (RTF) as an improvement over the shortcomings of ATF prediction methods in practical applications. RTF is the difference between the ATFs of a given source for two of the microphones in the array, and in the frequency domain takes the form of a ratio between the spectral representations of the two ATFs. Like ATF, RTF in the frequency domain assigns complex numbers to individual frequencies. The absolute value is the gain difference between the two microphones, which is often close to unity when the microphones are close to each other, and the phase reflects the source's angle of incidence under some conditions.

[0096] Transfer function based linear constrained minimum variance (TF-LCMV) beamformers can reduce noise while limiting speech distortion in multi-microphone applications by minimizing the output energy under the constraint that the speech component in the output signal is equal to the speech component in one of the microphone signals. d +N i Given sources, N i N disturbance sources and N polluted by stationary noise d Consider the problem of extracting m desired sound sources. Each of the contained signals propagates through an acoustic medium before being picked up by an arbitrary array with M microphones. The signals of the individual microphones are segmented into frames of length T, and an FFT is applied to each frame. In the frequency domain, the kth frequency component of the lth frame for the mth microphone and the nth source is

number

number

number

[0097] In the above formula

number

number

number

number

number

number

number

number

number

[0098] In the above formula

number

[0099] Given an array measurement z(l, k), N d The mixture of desired sources must be predicted. The extraction of the desired signal is performed using a beamformer.

number

number

number

number

number

[0100] Possible solutions to (9) are

number

[0101] Based on (7) and (8) and the constraint set, the component of the desired signal at the beamformer output is

number

[0102] From the l-th set of RTFs and for each frequency component k, the intensity a obtained from one of the microphones, defined as the reference microphone, for example p Using a phase difference based algorithm coupled with (l, k), the incidence angle is θ p (l, k), a set of at most M-1 sources with p=1, .., P≦M-1, and an elevation angle φ p (l, k) can be extracted.

number

[0103] TF-LCMV is an applicable method to extract M-1 sound sources impinging on an array of M sensors from different locations in a reverberant environment. However, a necessary condition for TF-LCMV to work is that the RTF matrix H(l,k), whose columns are the RTF vectors of all active sources in the environment, is known and available to TF-LCMV. To do so, it is necessary to associate individual frequency components with their source speakers.

[0104] Several methods can be used to assign sources to signals without the need for additional information. The main family of methods is called blind source separation (BSS), which recovers unknown signals or sources from their observed mixtures. A key weakness of BSS in the frequency domain is that at each frequency, the column vectors of the mixing matrix (predicted by BSS) are randomly permuted, and without knowledge of this random permutation, combining the results across frequencies becomes difficult as disclosed.

[0105] BSS can be aided by pitch information, however the gender of the speaker is required a-priori.

[0106] BSS can be used in the frequency domain while resolving ambiguities in the predicted mixing matrix using the maximum-magnitude method, which assigns a particular column of the mixing matrix to the source corresponding to the largest element in the vector. However, this method is highly dependent on the spectral distribution of the sources, since it is assumed that the strongest components at each frequency actually belong to the strongest source. However, this condition is rarely encountered, since different speakers will introduce power peaks at different frequencies. Alternatively, source activity detection, also known as voice activity detection (VAD), can be used so that ambiguities in the mixing matrix are resolved using information about the active sources at a particular time. A drawback of VAD is that it cannot robustly detect voice pauses, especially in a multi-speaker environment. Also, this method is effective when only one speaker at a time participates in the conversation, which is sensitive to motion, during this period, as it requires a relatively long training period.

[0107] The TF-LCMV beamformer can also be used as an extended version for bi-aural speech enhancement systems, in conjunction with a bi-aural cue generator. Acoustic cues are used to separate speech from noise components in the input signal. This technique is based on auditory scene analysis theory, which suggests the use of distinct cues to cluster signals from distinct speech sources in a "cocktail party" environment. 1 It is based on the idea that the proposed method ...

[0108] It should be noted that even if multiple loudspeakers are simultaneously active, the spectral contents of the loudspeakers do not overlap at most time-frequency points. This is called W-separation orthogonality, or W-DO for short. This can be justified by the sparseness of the audio signal in the time-frequency domain, according to which the probability of simultaneous activity of two loudspeakers at a particular time-frequency point is extremely low. In other words, in the case of multiple simultaneous loudspeakers, each time-frequency point is likely to correspond mostly to the spectral content of one of the loudspeakers.

[0109] The W-DO can be used to facilitate BSS by defining a certain class of signals that are to some extent W-DO. This is computationally economical as only the first order statistics required can be used. Furthermore, any number of signal sources can be demixed using only two microphones, provided that the sources are W-DO and do not occupy the same spatial location. However, this method assumes exactly the same underlying mixing matrix across all frequencies. This assumption is essential for using histograms of predicted mixing coefficients across different frequencies. However, this assumption does not hold true in reverberant environments, and often only holds true in anechoic environments. Extension of this method to the multipath case is limited to either negligible energy from the multipaths, or a sufficiently smooth convolutional mixing filter, thus smearing the histogram but still maintaining a single peak. This assumption also does not hold true in reverberant environments, where the differences between different paths are often too large to produce a smooth histogram.

[0110] The suggested solution has been shown to perform well in reverberant environments and does not need to rely on unnecessary assumptions and constraints: it can operate without a priori information, without an extensive training process, without constraining the prediction of attenuation and delay for a given source at each frequency to a single point in the attenuation-delay space, without constraining the prediction of attenuation-delay values ​​for a single source to produce a single cluster, and without limiting the number of mixed sounds to two.

[0111] Source separation for speech recognition engine A voice user interface (VUI) is an interface between a human speaker and a machine. A VUI receives speech signals using one or more microphones and converts them into a digital signature, often by transcribing the speech signal into text, which it uses to infer the speaker's intent. The machine can then respond to the speaker's intent based on the application for which it is designed.

[0112] A key component of a VUI is an automatic speech recognition engine (ASR) that converts digitized speech signals into text. The performance of the ASR, how accurately the text describes the acoustic speech signal, is highly dependent on the matching of the input signal to the requirements of the ASR. Other components of the VUI are therefore designed to enhance the captured speech signal before feeding it to the ASR. Such components may be noise suppression, echo cancellation, and source separation, to name a few.

[0113] One of the crucial components in sound enhancement is source separation (SS), which is intended to separate sound signals arriving from several sources. Given an array of two or more microphones, the signal acquired by each of the microphones is a mixture of all the sound signals in the environment plus noise and other disturbances such as music. The SS algorithm takes the mixed signals from all the microphones and breaks them down into their components. That is, the output of source separation is a set of signals, each representing a signal of a specific source, be it a sound signal from a specific speaker, music or even noise.

[0114] There is an increasing need to improve source separation.

[0115] FIG. 6 illustrates an example of a voice recognition chain for offline training. The chain often includes an array of microphones 511 providing a set of digitized acoustic signals. The number of digitized acoustic signals is equal to the number of microphones that make up the array 511. Each digitized acoustic signal includes a mixture of all acoustic sources in the vicinity of the microphone array 511, whether human speakers or synthetic speakers such as TV, music and noise. The digitized acoustic signals are passed to a pre-processing stage 512. The purpose of the pre-processing stage 512 is to improve the quality of the digitized acoustic signals by removing disturbances such as echo, reverberation and noise. The pre-processing stage 512 is usually implemented using a multi-channel algorithm employing statistical associations between the digitized acoustic signals. The output of the pre-processing stage 512 is a set of processed signals, usually having the same number of signals as the number of digitized acoustic signals at the input to this stage. This set of processed signals is sent to a source separation (SS) stage 513 whose purpose is to extract the acoustic signals from the individual sources in the vicinity of the microphone array. In other words, the SS stage 513 takes the set of signals where the individual signals are different mixtures of acoustic signals received from different sources and produces a set of signals where each signal mainly contains a single acoustic signal from a single specific source. Source separation of the audio signals can be performed using a geometric consideration of the source deployment, such as beam forming, or by considering the characteristics of the audio signals, such as independent component analysis. The number of signals to be separated is usually equal to the number of active sources in the vicinity of the microphone array 511, but less than the number of microphones. The set of separated signals is sent to a source selector 514. The purpose of the source selector is to select the relevant source of the audio whose audio signal is to be recognized. The source selector 514 can use a trigger word detector so that a source that utters a predefined trigger word is selected. Alternatively, the source selector 514 may consider the location of the source in the vicinity of the microphone array 511 , such as a predefined direction relative to the microphone array 511 .The source selector 514 can also use a predefined acoustic signature of the speech signal to select a source that matches this signature. The output of the source selector 514 is a single speech signal that is sent to the speech recognition engine 515. The speech recognition engine 515 converts the digitized speech signal into text. There are many methods for speech recognition known in the art, most of which are based on extracting features from the speech signal and comparing these features to a predefined vocabulary. The main output of the speech recognition engine 515 is a text string 516 that is associated with the input speech signal. A predefined text 518 is spoken into a microphone in offline training. An error 519 is calculated by comparing the output 516 of the ASR against this text. The comparison 517 can be performed using simple word counting or using more highly complex comparison methods that take into account the meaning of the words and appropriately weight the false positives of different words. The error 519 is then used by the SS 513 to modify a set of parameters to find the values ​​that minimize the error. This can be done by any supervised forecast or by optimization methods such as least squares, stochastic gradients, neural networks (NNs) and their variants.

[0116] FIG. 7 illustrates an example of a voice recognition chain in real-time training, i.e., training during regular operation of the system. While the VUI is running, the true text pronounced by the human speaker is unknown and the supervised error 519 is not available. An alternative is a confidence score 521 developed for real-time applications when there is no reference to the actual text spoken and when the application can benefit from knowing the confidence level of the ASR output. For example, if the confidence score is low, the system can proceed to an appropriate branch where a more controlled dialogue is conducted with the user. There are many methods to predict the confidence score, most of which target a high correlation with the error that can be calculated once the spoken text is known. In real-time training, the confidence score 521 is converted to the supervised error 519 by an error estimator 522. If the confidence score is highly correlated with the theoretical supervised error, the error estimator can be a simple axis transformation. The confidence score 521 ranges from 0 to 100 and the goal is to make the confidence score 521 higher, while the supervised error ranges from 0 to 100 and the goal is to make the supervised error smaller. A simple axis transformation of the form estimated_error=100-confidence_score can be used as the error estimator 522. The predicted error 519 can be used to train the parameters of the SS as in the offline training case.

[0117] FIG. 8 illustrates the training mechanism of a typical SS513. The source separator (SS) 513 receives a set of mixed signals from the pre-processing stage 512 and provides the separated signals to the source selector 514. Typically, source separation of acoustic signals, and especially of speech signals, is performed in the frequency domain. The mixed signals from the pre-processing stage 512 are first transformed 553 into the frequency domain. This is performed by dividing the mixed signals into segments of exactly the same length and having an overlap period between the resulting segments. For example, if the length of the segments is 1024 samples and the overlap period is determined as 25%, then each mixed signal is divided into segments of 1024 samples each. The current set of segments from different mixed signals is called a batch. Each batch of segments starts 768 samples after the previous batch. It is noted that the segments across the set of mixed signals are synchronized, i.e. the starting points of all segments belonging to the same batch are exactly the same. The length of the segments and the overlap period within a batch are obtained from the model parameters 552.

[0118] A demixing algorithm 554 separates the batches of segments arriving from the frequency transform 553. Like many other algorithms, the source separation (SS) algorithm includes a set of mathematical models accompanied by a set of model parameters 552. The mathematical models establish how the SS operates, such as how it handles physical phenomena, e.g., multipath. The set of model parameters 552 adjusts the operation of the SS to the specific characteristics of the source signals, the architecture of the automatic speech recognition engine (ASR) that receives these signals, the geometry of the environment, and even human speakers.

[0119] The demixed batch of segments is sent to an inverse frequency transform 555, where the batch is transformed back to the time domain. In the inverse frequency transform stage 555, the same set of model parameters 552 used in the frequency transform stage 553 is used. The output signal in the time domain from the resulting batch is reconstructed, for example using the overlapping periods. This is done, for example, using the overlap-add method, where after the inverse frequency transform, the resulting output signal is reconstructed by overlapping, possibly using a suitable weighting function ranging between 0 and 1 over the entire overlapping region, and adding the overlapped time intervals, thus saving total energy. In other words, the overlapped segments from the previous batch fade out, while the overlapped segments from the later batch fade in. The output of the inverse frequency transform block is sent to a source selector 514.

[0120] The model parameters 552 are a set of parameters used by the frequency transformation block 553, the demixing block 554, and the inverse frequency transformation block 555. The division of the mixed signal into segments of identical length performed by the frequency transformation block 553 is paced by a clock mechanism such as a real-time clock. At each pace, the frequency transformation block 553, the demixing block 554, and the inverse frequency transformation block 555 each extract parameters from the model parameters 552. These parameters are then substituted into the mathematical models implemented in the frequency transformation block 553, the demixing block 554, and the inverse frequency transformation block 555.

[0121] A collector 551 optimizes the set of model parameters 552 with the goal of reducing the error 519 from the error estimator. The collector 551 receives the error 519 and the current set of model parameters 552 and outputs a modified set of model parameters 552. The modification of the set of parameters can be performed a priori (offline) or while the VUI is running (real-time). In offline training, the error 519 used to modify the set of model parameters 552 is derived using a predefined text spoken to the microphone and comparing the output of the ASR against this text. In real-time training, the error 519 is derived from the confidence score of the ASR.

[0122] The error is then used to modify the set of parameters to find the values ​​that minimize the error, which can be done by any supervised prediction or optimization method, preferably derivative-free methods such as golden section search, lattice search and Nelder-Mead.

[0123] The Nelder-Mead method (also called the downhill simplex method, the amoeba method or the polytope method) is a widely applied numerical method used to find the minimum or maximum of an objective function in a multidimensional space. It is a direct search method (based on function comparison) and is often applied to nonlinear optimization problems where the derivatives are not known.

[0124] Nelder-Mead iteratively finds a local minimum of the error 519 as a function of several parameters. The method starts with a set of values ​​that define a simplex (a generalized triangle in N dimensions). It is assumed that the local minimum exists within the simplex. At each iteration, the error at the vertices of the simplex is calculated. The vertex with the maximum error is replaced by a new vertex, thus reducing the volume of the simplex. This is repeated until the simplex volume is smaller than a predefined volume and the optimum value is one of the vertices. This process is performed by a collector 551.

[0125] The golden section search finds a minimum of the error 519 by successively narrowing the range of values ​​within which the minimum is found to lie. The golden section search requires a strictly unimodal error as a function of the parameters. The range narrowing operation is performed by the corrector 551.

[0126] Golden section search is a technique for finding the extreme value (minimum or maximum) of a strictly unimodal function by successively narrowing the range of values ​​within which the extreme value is found to lie (www.wikipedia.org).

[0127] The grid search iterates through a set of values ​​associated with one or more of the parameters to be optimized. If multiple parameters are being optimized, each value in the set is a vector whose length is equal to the number of parameters. For each value, an error 519 is calculated and the value corresponding to the smallest error is selected. The iteration through the set of values ​​is performed by a collector 551.

[0128] Lattice Search - The traditional way to perform hyperparameter optimization is lattice search or parameter sweep, which is simply an exhaustive search through a manually defined subset of the hyperparameter space of a learning algorithm. Lattice search algorithms must be guided by some performance metric, typically determined by cross-validation on a training set or by evaluation on a held-out validation set. Because the parameter space of machine learning can include real-valued or unbounded value spaces for certain parameters, manually setting bounds and truncations may be necessary before applying a lattice search (www.wikipedia.org).

[0129] All optimization methods require continuous calculation of the error 519 using the same set of separated acoustic signals. This is a time consuming process and therefore may not be performed continuously, but only when the error 519 (which is calculated continuously) exceeds some predefined threshold, for example 10% error. When this occurs, two approaches can be taken.

[0130] One approach is to operate the optimization in parallel with the normal operation of the system using parallel threads or multiple cores. That is, there are one or more parallel tasks that blocks 513, 514, 515, 522 perform in parallel with the tasks of the normal operation of the system. In the parallel tasks, batches of mixed signals of 1-2 seconds in length are obtained from preprocessing 512 and are iteratively separated 513 and interpreted 514, 515 using different sets of model parameters 552. An error 519 is calculated for each such cycle. At each cycle, the set of model parameters is selected by a collector 551 according to an optimization method.

[0131] The second approach is to operate the optimization when no voice is present in the room. Periods where there is no human voice can be detected using a Voice Activity Detection (VAD) algorithm. These periods are used to optimize the model parameters 552 in the same way as in the first approach, saving the need for parallel threads or multiple cores.

[0132] An appropriate optimization method must be selected for each parameter in 552. Some of the methods apply to single parameters and some to groups of parameters. The following text suggests some parameters that affect the performance of speech recognition. Also, optimization methods based on the properties of the parameters are suggested.

[0133] Length of segment parameters The segment parameter length is related to the FFT / IFFT. Typically, ASR using isolated phoneme features requires short segments of around 20 ms, while ASR using the resulting set of phoneme features uses segments of around 100-200 ms. On the other hand, the length of the segment is influenced by scenarios such as the reverberation time of the room. The segment length should be around the reverberation time, which can be around 200-500 ms. There is no sweet point for the segment length, so this value has to be optimized for the scenario and ASR. Typical values ​​are 100-500 ms in terms of samples. For example, a sampling rate of 8 kHz implies a segment length of 800-4000 samples. This is a continuous parameter.

[0134] Optimization of this parameter can be performed using various optimization methods such as golden section search, or Nelder-Mead in conjunction with overlap period. When using golden section search, the inputs to the algorithm are the minimum and maximum possible lengths, e.g., 10 ms to 500 mSec, and an error function 519. The output is the length of the segment that minimizes the error function 519. When using Nelder-Mead with overlap period, the inputs are a set of three binaries of segment length and overlap period, e.g., (10 ms, 0%), (500 ms, 10%), and (500 ms, 80%), and the error function 519, and the output is the optimal length of the segment and the optimal overlap period.

[0135] Overlap Period The overlap period parameter is related to the FFT / IFFT. It is used to avoid missing phonemes due to segmentation, i.e. splitting phonemes between the resulting segments. Due to the length of the segments, the overlap period is determined by the features employed by the ASR. Typical range is 0-90% of the segment length. It is a continuous parameter.

[0136] Optimization of this parameter can be performed using various optimization methods such as golden section search, Nelder-mead with segment length, etc. When using golden section search, the inputs to the algorithm are the minimum and maximum possible overlap terms, e.g., 0% to 90%, and an error function 519. The output is the overlap term that minimizes the error function 519.

[0137] Window. Window parameters are associated with FFT / IFFT. Frequency transforms 553 often use windowing to mitigate the effects of segmentation. Some windows, such as Kaiser and Chebyshev, are parameterized. This means that the effect of the window can be controlled by changing the window parameters. The typical range depends on the type of window. This is a continuous parameter. Optimization of this parameter can be performed using various optimization methods, such as golden section search. When using golden section search, the input to the algorithm is the minimum and maximum of the window parameters, which depend on the window type, and the error function 519. For example, for the Kaiser window, the minimum and maximum are (0, 30). The output is the optimal window parameters.

[0138] Sampling Rate The sampling rate parameter is associated with the FFT / IFFT. The sampling rate is one of the critical parameters that affect the performance of speech recognition. There are ASRs that have demonstrated poor results for sampling rates below 16 kHz for example. Other ASRs can work well even at 4 kHz or 8 kHz. Typically, this parameter is optimized once the ASR is selected. Typical ranges are 4 kHz, 8 kHz, 16 kHz, 44.1 kHz, 48 kHz. This parameter is a discrete parameter. The optimization of this parameter can be performed using various optimization methods such as lattice search. The input to the algorithm is the sampling rate, for example (4, 8, 16, 44.1, 48) kHz, which is the value at which the lattice search is performed, and the error function 519. The output is the optimal sampling rate.

[0139] filtering The filtering parameters are related to the demixing. Some ASRs use features that represent limited frequencies. Thus, filtering the separated signals after source separation 513 can enhance the specific features used by the ASR, thereby improving its performance. Furthermore, filtering out spectral components not used by the ASR can improve the signal to noise ratio (SNR) of the separated signals, which in turn can improve the performance of the ASR. A typical range is 4-8 kHz. Optimization of this parameter can be performed using various optimization methods, such as golden section search. This parameter is continuous. When applying the golden section search, the inputs to the algorithm are the error function 519 and an initial guess of the division of the cutoff frequency, e.g. 1000 Hz and 0.5X sampling rate. The output is the optimal filtering parameters.

[0140] Weighting coefficients per microphone. The weighting coefficients per microphone are associated with demixing. Theoretically, the sensitivities of different microphones on a particular array should be similar up to 3 dB. In practice, however, the span of sensitivities of different microphones can be much wider. Furthermore, microphone sensitivities can vary over time due to dust and moisture. A typical range is 0-10 dB. This is a continuous parameter. Optimization of this parameter can be performed using various optimization methods such as Nelder-mead with or without weighting coefficients per microphone. When applying the Nelder-mead method, the input to the algorithm is the error function 519 and an initial guess of the vertices of the simplex. For example, the size of each n-term is the number of microphones - N: (1, 0, .., 0, 0), (0, 0, ..., 0, 1) and (1 / N, 1 / N, ..., 1 / N). The output is the optimal weights per microphone.

[0141] Number of microphones The number of microphones is related to the demixing. It affects on the one hand the number of sources that can be separated and on the other hand the complexity and numerical accuracy. Also, practical experiments show that too many microphones can lead to a small output SNR. A typical range is 4-8. It is a discrete parameter. The optimization of this parameter can be performed using various optimization methods such as lattice search or Nelder-mead with weighting coefficients per microphone. When applying a lattice search, the input to the algorithm is the error function 519 and the number of microphones for which the search is performed, for example 4, 5, 6, 7, 8 microphones. The output is the optimal number of microphones.

[0142] FIG. 9 illustrates the method 600.

[0143] The method 600 may begin at step 610 with receiving or calculating an error associated with a speech recognition process applied to a previous output of a source selection process.

[0144] Step 610 may be followed by step 620 of revising at least one parameter of the source separation process based on the error.

[0145] Step 620 may be followed by step 630 of receiving signals representative of audible signals emitted from a plurality of sources and detected by an array of microphones.

[0146] Step 630 may be followed by step 640 of performing a source separation process to separate the audio signals emitted from different sources of the plurality of sources to provide a source-separated signal, and to send the source-separated signal to a source selection process.

[0147] Step 640 may be followed by step 630 .

[0148] After one or more iterations of steps 630 and 640, step 610 (not shown) may be followed, and the output of step 640 may be provided to the source selection process and the ASR to provide the preceding output of the ASR.

[0149] Note that the initial iterations of steps 630 and 640 may be performed without receiving any errors.

[0150] Step 640 may include applying a frequency transform (such as, but not limited to, an FFT), demixing, and applying an inverse frequency transform (such as, but not limited to, an IFFT).

[0151] Step 620 may include at least one of the following steps. a. revising at least one parameter of a frequency transformation b. revising at least one parameter of the inverse frequency transform c. revising at least one parameter of the demixing d. revising the length of the segment of the signal representing the audio signal to which the frequency transformation is applied e. revising the overlap between successive segments of a signal representative of an audio signal, a frequency transformation being applied on a segment-by-segment basis; f. Step of revising the sampling rate of the frequency conversion g. revising the windowing parameters of the window applied by the frequency transform; h. revising the cutoff frequency of the filter applied during demixing i. revising the weights applied to individual microphones of the array of microphones during demixing j. revising the number of microphones in the microphone array k. determining a revised value of at least one parameter using a golden section search; l. Determining a revised value of at least one parameter using the Nedler Mead algorithm. m. determining a revised value of at least one parameter using a lattice search. n. determining a revised value of the at least one parameter based on a predefined mapping between the error and the at least one parameter; determining in real time a mapping between the error and at least one parameter;

[0152] Although the invention has been described herein with reference to specific examples of embodiments thereof, it will be apparent, however, that various modifications and changes can be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims.

[0153] Moreover, the terms "front," "rear," "top," "bottom," "upper," "lower," and the like in the description and claims, when used, are used for descriptive purposes and not necessarily to describe permanent relative positions, and it is understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein can operate in orientations other than those illustrated or otherwise described herein.

[0154] Any arrangement of components to achieve the same functionality is effectively "associated" such that the desired functionality is achieved. Thus, any two components herein that are combined to achieve a particular functionality can be viewed as being "associated" with one another such that the desired functionality is achieved, regardless of architecture or intermediate components. Likewise, any two components so associated can equally be viewed as being "operably connected" or "operably coupled" with one another to achieve the desired functionality.

[0155] Moreover, those skilled in the art will recognize that the boundaries between operations described above are merely illustrative. Multiple operations may be combined into a single operation, a single operation may be distributed among additional operations, and operations may be performed with at least partial time overlap. Furthermore, alternative embodiments may include multiple instances of a particular operation, and the order of operations may be altered in various other embodiments.

[0156] However, other modifications, variations, and alternatives are also possible, and the specification and drawings are accordingly to be regarded in an illustrative rather than a restrictive sense.

[0157] The phrase "may X" indicates that condition X may be met. It also suggests that condition X may not be met. Any reference to a system, e.g., as including a particular component, is intended to also encompass scenarios in which the system does not include that particular component. Any reference to a method, e.g., as including a particular step, is intended to also encompass scenarios in which the method does not include that particular component. As yet another example, any reference to a system configured to perform a particular operation is intended to also encompass scenarios in which the system is not configured to perform that particular operation.

[0158] The terms "including," "comprising," "having," "consisting," and "consisting essentially of" are used interchangeably. For example, a method may include at least the steps all included in the figures and / or specification, or may include only the steps included in the figures and / or specification. The same applies to a system.

[0159] The system may include an array of microphones, a memory device, and one or more hardware processors, such as a digital signal processor, an FPGA, an ASIC, a general purpose processor programmed to perform any of the methods mentioned above, etc. The system may not include an array of microphones, but may be fed from sound signals generated by an array of microphones.

[0160] It will be appreciated that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, dimensions of some of the elements may be exaggerated relative to other elements for clarity. Further, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or similar elements.

[0161] Although the invention has been described herein with reference to specific examples of embodiments thereof, it will be apparent, however, that various modifications and changes can be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims.

[0162] Moreover, the terms "front," "rear," "top," "bottom," "upper," "lower," and the like in the description and claims, when used, are used for descriptive purposes and not necessarily to describe permanent relative positions, and it is understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein can operate in orientations other than those illustrated or otherwise described herein.

[0163] Those skilled in the art will recognize that the boundaries between logic blocks are merely illustrative, and that alternative embodiments may combine logic blocks or circuit elements or enforce alternative decompositions of functionality among various logic blocks or circuit elements. It should therefore be understood that the architectures depicted herein are merely exemplary, and that in fact many other architectures may be implemented which achieve the same functionality.

[0164] Any arrangement of components to achieve the same functionality is effectively "associated" such that the desired functionality is achieved. Thus, any two components herein that are combined to achieve a particular functionality can be viewed as being "associated" with one another such that the desired functionality is achieved, regardless of architecture or intermediate components. Likewise, any two components so associated can equally be viewed as being "operably connected" or "operably coupled" with one another to achieve the desired functionality.

[0165] Moreover, those skilled in the art will recognize that the boundaries between operations described above are merely illustrative. Multiple operations may be combined into a single operation, a single operation may be distributed among additional operations, and operations may be performed with at least partial time overlap. Furthermore, alternative embodiments may include multiple instances of a particular operation, and the order of operations may be altered in various other embodiments.

[0166] Also, for example, in one embodiment, the illustrated examples may be implemented as circuitry located on a single integrated circuit or within the same device, or alternatively, the examples may be implemented as any number of separate integrated circuits or separate devices interconnected together in any suitable manner.

[0167] Also, for example, the illustrations, or portions thereof, may be implemented as soft or code representations of physical circuitry, or as logical representations that can be translated into physical circuitry, such as in any suitable type of hardware description language.

[0168] Furthermore, the present invention is not limited to physical devices or units realized in non-programmable hardware, but may equally be applied within programmable devices or units, such as mainframes, minicomputers, servers, workstations, personal computers, notepads, personal digital assistants, electronic games, automobiles and other embedded systems, cellular telephones and various other wireless devices, generally referred to in this application as "computer systems", which are capable of performing desired device functions by operating in accordance with appropriate program code.

[0169] However, other modifications, variations, and alternatives are also possible, and the specification and drawings are accordingly to be regarded in an illustrative rather than a restrictive sense.

[0170] In the claims, any reference signs placed between parentheses shall not be construed as limiting the scope of the claim. The word "comprising" does not exclude the presence of other elements or steps than those recited in the claim. Furthermore, the singular expressions of unspecified elements used in this specification are defined as one or more. Also, the use of introductory clauses such as "at least one" and "one or more" in the claims shall not be construed as implying that the introduction of another claim element by the singular expression of an unspecified element limits any particular claim containing such introduced claim element to an invention containing only one such element, even if the same claim contains the introductory clause "one or more" or "at least one" and the singular expression of an unspecified element. The same applies to the use of expressions referring to specific elements. Unless otherwise specifically stated, terms such as "first" and "second" are used arbitrarily to arbitrarily distinguish between the elements that such terms describe. Thus, these terms are not necessarily intended to imply any temporal or other ordering of such elements, and the mere fact that certain measures are recited in mutually different claims does not indicate that a combination of these measures cannot be used to advantage.

[0171] The invention may also be embodied in a computer program for running on a computer system, the computer program comprising at least code portions for performing the steps of a method according to the invention or for enabling the programmable device to perform functions of a device or system according to the invention when the computer program is run on a programmable device such as a computer system. The computer program may cause the storage system to allocate disk drives into disk drive groups.

[0172] A computer program is a listing of instructions for a particular application program and / or operating system, etc. A computer program may include, for example, one or more of a subroutine, function, procedure, method, implementation, executable application, applet, servlet, source code, object code, shared library / dynamic load library, and / or other sequence of instructions designed to be executed on a computer system.

[0173] A computer program may be stored internally on a non-transitory computer readable medium. All or a portion of a computer program may be provided on a computer readable medium that is permanently, removably, or remotely coupled to an information processing system. Computer readable media may include, for example, any number of, but not limited to, magnetic storage media, including disk and tape storage media; optical storage media, such as compact disk media (e.g., CD-ROM, CD-R, etc.) and digital video disk storage media; non-volatile memory storage media, including semiconductor-based storage devices such as FLASH memory, EEPROM, EPROM, ROM, etc.; ferromagnetic digital memory; MRAM; volatile storage media, including registers, buffers or caches, main memory, RAM, and the like. A computer process typically includes the running program or portions of a program, current program values ​​and state information, and resources used by the operating system to manage the execution of the process. An operating system (OS) is software that manages the sharing of the computer's resources and provides the programmer with the interface used to access these resources. An operating system processes system data and user input and responds by allocating and managing tasks and internal system resources as services to users and programs of the system. When a computer system executes a computer program, which may include, for example, at least one processing unit, associated memory, and a number of input / output (I / O) devices, the computer system processes information in accordance with the computer program and generates resulting output information via the I / O devices.

[0174] All systems related to this patent application include at least one hardware component.

[0175] While certain features of the invention have been illustrated and described herein, many modifications, substitutions, changes and equivalents will occur to those skilled in the art, and it is therefore to be understood that the appended claims are intended to cover all such modifications and changes as fall within the true spirit of the invention.

Claims

1. 1. A method for audio enhancement, comprising: receiving or generating sound samples representative of sound signals received by an array of microphones during a given period of time; frequency transforming the sound samples to provide frequency transformed samples; clustering the frequency transformed samples into speakers to provide speaker-related clusters, the clustering being based on (i) spatial cues associated with the received sound signal, and (ii) acoustic cues associated with the speakers; determining a relative transfer function for each individual one of the speakers to provide a speaker related relative transfer function; applying a multiple-input multiple-output (MIMO) beamforming operation to the speaker-associated relative transfer functions to provide beamformed signals; inverse frequency converting the beamformed signal to provide an audio signal; The method includes:

2. The method of claim 1 , further comprising generating the acoustic cue associated with the speaker.

3. generating said acoustic cues comprises searching for keywords in said sound samples; extracting the acoustic cues from the keywords; The method of claim 2 , comprising:

4. The method of claim 3 further comprising extracting spatial cues associated with the keywords.

5. The method of claim 4 , further comprising using the spatial queries associated with the keywords as clustering seeds.

6. The method of claim 2 , wherein the acoustic cues include a pitch frequency, a pitch intensity, one or more harmonics of the pitch frequency, and an intensity of the one or more harmonics of the pitch frequency.

7. The method of claim 5 , comprising associating a reliability attribute with each pitch and determining that a speaker associated with said pitch becomes silent when the reliability of said pitch falls below a predefined threshold.

8. 2. The method of claim 1, wherein the clustering step comprises the steps of: processing the frequency transformed samples to provide the acoustic cues and the spatial cues; keeping track of speaker states using the acoustic cues; segmenting spatial cues of individual frequency components of the frequency transformed signals into groups; and assigning acoustic cues associated with currently active speakers to each group of frequency transformed signals.

9. 9. The method of claim 8, wherein the assigning step comprises the step of calculating, for each group of frequency transformed signals, a cross-correlation between elements of iso-frequency lines of the time-frequency map that belong to other lines of the time-frequency map and have elements associated with the group of frequency transformed signals.

10. The method of claim 8 , wherein the tracking step includes applying an extended Kalman filter.

11. The method of claim 8 , wherein the tracking step comprises applying multiple hypothesis tracking.

12. The method of claim 8 , wherein the tracking step includes applying a particle filter.

13. The method of claim 8 , wherein the segmenting step comprises the step of assigning a single frequency component associated with a single time frame to a single speaker.

14. 9. The method of claim 8, further comprising the step of monitoring at least one of the following monitored acoustic features: speech rate, speech intensity, and emotional voicing.

15. The method of claim 14 comprising the step of feeding the at least one monitored acoustic feature to an extended Kalman filter.

16. 2. The method of claim 1, wherein the frequency transformed samples are arranged in a plurality of vectors, one vector for each microphone of the array of microphones, the method comprising: calculating an intermediate vector by weighted averaging the plurality of vectors; and searching for acoustic cue candidates by ignoring elements of the intermediate vector that have values ​​below a predefined threshold.

17. 17. The method of claim 16, comprising determining the predefined threshold to be three times the standard deviation of the noise.

18. 11. A non-transitory computer-readable medium storing instructions that, when executed by a computerized system, cause the computerized system to: receive or generate sound samples representing sound signals received by an array of microphones during a given period of time; frequency convert the sound samples to provide frequency converted samples; cluster the frequency converted samples into speakers to provide speaker-related clusters, the clustering being based on (i) spatial cues associated with the received sound signals and (ii) acoustic cues associated with the speakers; determine relative transfer functions for each individual one of the speakers to provide speaker-related relative transfer functions; apply a multiple-input multiple-output (MIMO) beamforming operation to the speaker-related relative transfer functions to provide a beamformed signal; and inverse frequency convert the beamformed signal to provide an audio signal.

19. 20. The non-transitory computer readable medium of claim 18, wherein the non-transitory computer readable medium stores instructions for generating the acoustic cue associated with the speaker.

20. 1. A computerized system comprising an array of microphones, a storage device, and a processor, the processor being configured to receive or generate sound samples representing sound signals received by the array of microphones during a given period of time, frequency converting the sound samples to provide frequency converted samples, clustering the frequency converted samples into speakers to provide speaker-related clusters, the clustering being based on (i) spatial cues associated with the received sound signals and (ii) acoustic cues associated with the speakers, determining relative transfer functions for each individual speaker of the speakers to provide speaker-related relative transfer functions, applying a multiple-input multiple-output (MIMO) beamforming operation to the speaker-related relative transfer functions to provide a beamformed signal, and inverse frequency converting the beamformed signal to provide an audio signal, the storage device being configured to store at least one of the sound samples and the audio signal.

21. A method for adjusting a source separation algorithm to a speech recognition algorithm, the method comprising the steps of receiving or calculating an error associated with a speech recognition process applied to a previous output of a source selection process, revising at least one parameter of the source separation process based on the error, receiving signals representative of audible signals originating from a plurality of sources and detected by an array of microphones, performing a source separation process to separate the audible signals originating from different ones of the plurality of sources to provide a source-separated signal, and transmitting the source-separated signal to the source selection process.

22. 22. The method of claim 21, wherein the step of performing the source separation process comprises the steps of applying a frequency transform, demixing, and applying an inverse frequency transform.

23. The method of claim 22 , wherein at least one parameter of the source separation process includes at least one parameter of the frequency transformation.

24. 24. The method of claim 23, wherein at least one parameter is the length of a segment of a signal representing an audio signal to which a frequency transformation is applied.

25. 24. The method of claim 23, wherein at least one parameter is an overlap between successive segments of a signal representing an audio signal, and the frequency transformation is applied on a segment-by-segment basis.

26. 24. The method of claim 23, wherein at least one parameter is a sampling rate of the frequency transformation.

27. The method of claim 23, wherein at least one parameter is a windowing parameter of a window applied by the frequency transform.

28. 23. The method of claim 22, wherein at least one parameter of the source separation process comprises at least one parameter of the demixing step.

29. 28. The method of claim 27, wherein at least one parameter is a cutoff frequency of a filter applied during the demixing step.

30. 28. The method of claim 27, wherein at least one parameter is a weight applied to each microphone of the array of microphones during the unmixing step.

31. 28. The method of claim 27, wherein at least one parameter is the number of microphones in the array of microphones.

32. 22. The method of claim 21, wherein the revising step comprises determining a revised value of the at least one parameter using a golden section search.

33. 22. The method of claim 21, wherein the revising step includes determining a revised value of the at least one parameter using a Nedler Mead algorithm.

34. 22. The method of claim 21, wherein the revising step comprises determining a revised value of at least one parameter using a lattice search.

35. 22. The method of claim 21, wherein the revising step comprises determining a revised value of at least one parameter based on a predefined mapping between the error and the at least one parameter.

36. 22. The method of claim 21, wherein the revising step comprises determining in real time a mapping between the error and the at least one parameter.

37. A non-transitory computer-readable medium storing instructions for separating sources by performing the steps of receiving or calculating an error associated with a speech recognition process applied to a previous output of a source selection process, revising at least one parameter of the source separation process based on the error, receiving signals representative of audible signals originating from a plurality of sources and detected by an array of microphones, performing a source separation process to separate the audible signals originating from different sources of the plurality of sources to provide a source-separated signal, and transmitting the source-separated signal to the source selection process.

Images