Speech Enhancement Using Clustering of Cues

Abstract

A method for speech enhancement, the method may include: receiving or generating sound samples, the sound samples representing sound signals received by a microphone array during a given time period; frequency transforming the sound samples to provide frequency transformed samples; clustering the frequency transformed samples according to speakers to provide speaker-related clusters, wherein the clustering is based on (i) spatial cues associated with the received sound signals and (ii) acoustic cues associated with the speakers; determining a relative transfer function for each of the speakers to provide a speaker-related relative transfer function; applying a multiple-input multiple-output (MIMO) beamforming operation on the speaker-related relative transfer functions to provide a beamformed signal; and inversely frequency transforming the beamformed signal to provide a speech signal.

Description

Background Art

[0001] The performance of the speech enhancement module depends on the ability to filter out all interfering signals and leave only the desired speech signal. Interfering signals may be, for example, other speakers, noise from air conditioners, music, motor noise (e.g., in cars or airplanes), and large crowd noises also known as "cocktail party noises." The performance of the speech enhancement module is usually measured by its ability to improve the speech-to-noise ratio (SNR) or the speech-to-interference ratio (SIR), which reflects the ratio of the power of the desired speech signal to the total power of the noise and the ratio of the power of the desired speech signal to the total power of other interfering signals (usually in dB).

[0002] There is an increasing need to perform speech enhancement in reverberant environments. Summary of the invention

[0003] A method for speech enhancement may be provided, the method may include: receiving or generating sound samples, the sound samples representing sound signals received by a microphone array during a given time period; frequency transforming the sound samples to provide frequency transformed samples; clustering the frequency transformed samples according to speakers to provide speaker-related clusters, wherein the clustering may be based on (i) spatial cues associated with the received sound signals and (ii) acoustic cues associated with the speakers; determining a relative transfer function for each of the speakers to provide a speaker-related relative transfer function; applying a multiple-input multiple-output (MIMO) beamforming operation on the speaker-related relative transfer functions to provide a beamformed signal; and inversely frequency transforming the beamformed signal to provide a speech signal.

[0004] The method may include generating acoustic cues associated with the speaker.

[0005] The generating of the acoustic clues may include: searching for keywords in the sound sample; and

[0006] The acoustic clues are extracted from the keywords.

[0007] The method may include extracting spatial cues related to the keyword.

[0008] The method may include using spatial cues associated with the keyword as clustering seeds.

[0009] The acoustic cues may include pitch frequency, pitch intensity, one or more harmonics of the pitch frequency, and intensity of the one or more harmonics of the pitch frequency.

[0010] The method may include associating a reliability attribute with each pitch and determining that a speaker associated with the pitch may be silent when the reliability of the pitch falls below a predefined threshold.

[0011] The clustering may include processing the frequency transformed samples to provide the acoustic cues and the spatial cues; using the acoustic cues to track the time-varying state of the speaker; segmenting the spatial cues for each frequency component of the frequency transformed signal into groups; and assigning acoustic cues associated with a currently active speaker to each group of frequency transformed signals.

[0012] The assigning may comprise calculating, for each set of frequency transformed signals, cross-correlations between elements of equal-frequency lines of the time-frequency diagram and elements belonging to other lines of the time-frequency diagram and which may be correlated with the set of frequency transformed signals.

[0013] The tracking may include applying an extended Kalman filter.

[0014] The tracking may include applying multi-hypothesis tracking.

[0015] The tracking may include applying a particle filter.

[0016] The segmentation may include assigning a single frequency component associated with a single time frame to a single speaker.

[0017] The method may include monitoring at least one of speech rate, speech intensity, and emotional utterance as an acoustic feature.

[0018] The method may include feeding the at least one monitored acoustic feature to an extended Kalman filter.

[0019] The frequency transformed samples may be arranged in a plurality of vectors, one vector for each microphone in the microphone array; wherein the method may include calculating an intermediate vector by weighted averaging the plurality of vectors; and searching for acoustic cue candidates by ignoring elements of the intermediate vector having values ​​that may be below a predefined threshold.

[0020] The method may comprise determining the predefined threshold as three times the standard deviation of the noise.

[0021] A non-transitory computer-readable medium may be provided that stores instructions that, once executed by a computerized system, cause the computerized system to: receive or generate sound samples that represent sound signals received by a microphone array during a given time period; frequency transform the sound samples to provide frequency transformed samples; cluster the frequency transformed samples according to speakers to provide speaker-related clusters, wherein the clustering may be based on (i) spatial cues associated with the received sound signals and (ii) acoustic cues associated with the speakers; determine a relative transfer function for each of the speakers to provide a speaker-related relative transfer function; apply a multiple-input multiple-output (MIMO) beamforming operation on the speaker-related relative transfer functions to provide a beamformed signal; and inversely frequency transform the beamformed signal to provide a speech signal.

[0022] The non-transitory computer-readable medium may store instructions for generating the acoustic cues associated with the speaker.

[0023] The generation of the acoustic cues may include searching for keywords in the sound sample; and

[0024] The acoustic clues are extracted from the keywords.

[0025] The generation of the acoustic cues may include searching for keywords in the sound sample; and

[0026] The acoustic clues are extracted from the keywords.

[0027] The non-transitory computer-readable medium may store instructions for extracting spatial cues related to the keyword.

[0028] The non-transitory computer-readable medium may store instructions for using the spatial cues associated with the keyword as clustering seeds.

[0029] The acoustic cues may include pitch frequency, pitch intensity, one or more harmonics of the pitch frequency, and intensity of the one or more harmonics of the pitch frequency.

[0030] The non-transitory computer-readable medium may store instructions for associating a reliability attribute with each pitch and determining that a speaker associated with the pitch may be silent when the reliability of the pitch falls below a predefined threshold.

[0031] The clustering may include processing the frequency transformed samples to provide the acoustic cues and the spatial cues; using the acoustic cues to track the time-varying state of the speaker; segmenting the spatial cues for each frequency component of the frequency transformed signals into groups; and assigning acoustic cues associated with a currently active speaker to each group of frequency transformed signals.

[0032] The assigning may comprise calculating, for each set of frequency transformed signals, a cross-correlation between elements of equal-frequency rows of the time-frequency diagram and elements belonging to other rows of the time-frequency diagram and which may be correlated with the set of frequency transformed signals.

[0033] The tracking may include applying an extended Kalman filter.

[0034] The tracking may include applying multi-hypothesis tracking.

[0035] The tracking may include applying a particle filter.

[0036] The segmentation may include assigning a single frequency component associated with a single time frame to a single speaker.

[0037] The non-transitory computer-readable medium may store instructions for monitoring at least one monitored acoustic feature of speech rate, speech intensity, and emotional expression.

[0038] The non-transitory computer readable medium may store instructions for feeding the at least one monitored acoustic feature to an extended Kalman filter.

[0039] The frequency transformed samples may be arranged in a plurality of vectors, one vector for each microphone in the microphone array; wherein the non-transitory computer-readable medium may store instructions for calculating an intermediate vector by weighted averaging the plurality of vectors and searching for acoustic cue candidates by ignoring elements of the intermediate vector having values ​​that may be below a predefined threshold.

[0040] The non-transitory computer-readable medium may store instructions for determining the predefined threshold as three times a standard deviation of noise.

[0041] A computerized system may be provided, which may include a microphone array, a memory unit, and a processor. The processor may be configured to receive or generate sound samples, the sound samples representing sound signals received by the microphone array during a given time period; frequency transform the sound samples to provide frequency transformed samples; cluster the frequency transformed samples according to speakers to provide speaker-related clusters, wherein the clustering may be based on (i) spatial cues associated with the received sound signals and (ii) acoustic cues associated with the speakers; determine a relative transfer function for each of the speakers to provide a speaker-related relative transfer function; apply a multiple-input multiple-output (MIMO) beamforming operation on the speaker-related relative transfer function to provide a beamformed signal; inverse frequency transform the beamformed signal to provide a speech signal; and wherein the memory unit may be configured to store at least one of the sound samples and the speech signal.

[0042] The computerized system may not include the microphone array, but may receive a signal from the microphone array representative of the sound signals received by the microphone array during a given time period.

[0043] The processor may be configured to generate the acoustic cues associated with the speaker.

[0044] The generation of the acoustic cues may include searching for keywords in the sound sample; and

[0045] The acoustic clues are extracted from the keywords.

[0046] The processor may be configured to extract spatial cues related to the keyword.

[0047] The processor may be configured to use the spatial cues associated with the keyword as clustering seeds.

[0048] The acoustic cues may include pitch frequency, pitch intensity, one or more harmonics of the pitch frequency, and intensity of the one or more harmonics of the pitch frequency.

[0049] The processor may be configured to associate a reliability attribute with each pitch and determine that a speaker associated with the pitch may be silent when the reliability of the pitch falls below a predefined threshold.

[0050] The processor may be configured to cluster by processing the frequency-transformed samples to provide the acoustic cues and the spatial cues; use the acoustic cues to track the speaker's state over time; segment the spatial cues for each frequency component in the frequency-transformed signal into groups; and assign acoustic cues associated with a currently active speaker to each group of frequency-transformed signals.

[0051] The processor may be configured to allocate by computing, for each set of frequency transformed signals, a cross-correlation between elements of equal-frequency rows of the time-frequency diagram and elements belonging to other rows of the time-frequency diagram and which may be correlated with the set of frequency transformed signals.

[0052] The processor may be configured to track by applying an extended Kalman filter.

[0053] The processor may be configured to track by applying multi-hypothesis tracking.

[0054] The processor may be configured to track by applying a particle filter.

[0055] The processor may be configured to segment by assigning a single frequency component associated with a single time frame to a single speaker.

[0056] The processor may be configured to monitor at least one of speech rate, speech intensity, and emotional expression as an acoustic feature.

[0057] The processor may be configured to feed the at least one monitored acoustic feature to an extended Kalman filter.

[0058] The frequency transformed samples may be arranged in a plurality of vectors, one vector for each microphone in the microphone array; wherein the processor may be configured to calculate an intermediate vector by weighted averaging the plurality of vectors; and to search for acoustic cue candidates by ignoring elements of the intermediate vector having values ​​that may be below a predefined threshold.

[0059] The processor may be configured to determine the predefined threshold as three times a standard deviation of the noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to understand the invention and to see how it can be put into practice, a preferred embodiment will now be described, by way of non-limiting example only, with reference to the accompanying drawings.

[0061] Figure 1 Multipath is illustrated;

[0062] Figure 2 An example of a method is illustrated;

[0063] Figure 3 Illustrated Figure 2 An example of the clustering step of the method;

[0064] Figure 4 An example of pitch detection on a time-frequency graph is illustrated;

[0065] Figure 5 An example of a time-frequency-Cue map is illustrated;

[0066] Figure 6 An example of a voice recognition chain in offline training is illustrated;

[0067] Figure 7 An example of a speech recognition chain in real-time training is illustrated;

[0068] Figure 8 An example of a training mechanism is illustrated; and

[0069] Fig. 9 An example of a method is illustrated. DETAILED DESCRIPTION

[0070] Any reference to a system shall apply mutatis mutandis to a method performed by the system and / or a non-transitory computer-readable medium storing instructions which, when executed by the system, will cause the system to perform the method.

[0071] Any reference to a method should apply mutatis mutandis to a system configured to perform the method and / or to a non-transitory computer-readable medium storing instructions which, when executed by the system, will cause the system to perform the method.

[0072] Any reference to a non-transitory computer-readable medium shall apply mutatis mutandis to methods performed by the system and / or systems configured to execute instructions stored in the non-transitory computer-readable medium.

[0073] The term "and / or" is additionally or alternatively.

[0074] The term "system" refers to a computerized system.

[0075] Speech enhancement methods focus on extracting speech signals from the desired source (speaker) when the signal is interfered by noise and other speakers. In a free-field environment, spatial filtering in the form of directional beamforming is effective. However, in a reverberant environment, the speech from each source smears in several directions and is not necessarily continuous, thereby reducing the advantages of ordinary beamformers. Using a transfer function (TF)-based beamformer to solve this problem, or using the relative transfer function (RTF) as the TF itself is a promising direction. However, in a multi-talker environment, when the speech signals are captured simultaneously, the ability to estimate the RTF for each speaker remains a challenge. A solution is provided that involves tracking acoustic cues and spatial cues to cluster simultaneous speakers, thereby facilitating the estimation of the speaker's RTF in a reverberant environment.

[0076] A speaker clustering algorithm is provided that assigns each frequency component to its original speaker, especially in a multi-talker reverberant environment. This provides the necessary condition for the RTF estimator to work properly in a multi-talker reverberant environment. The estimate of the RTF matrix is ​​then used to calculate the weight vector of a transfer function-based linear constrained minimum variance (TF-LCMV) beamformer (see equation (10) below), and thus satisfies the necessary condition for TF-LCMV to work. It is assumed that each human speaker is assigned a different pitch, so that the pitch is a bijective indicator for the speaker. It is known that multi-pitch detection is a challenging task, especially in a noisy, reverberant multi-talker environment. To address this challenge, the W-Disjoint Orthogonality (W-DO) assumption is adopted, and a set of spatial cues (such as signal strength, azimuth and elevation) are used as additional features. An extended Kalman filter (EKF) is used to track the acoustic clue - pitch value over time to overcome temporarily inactive speakers and pitch changes, and spatial clues are used to segment the final L frequency components and assign each frequency component to a different source. The results of the EKF and segmentation are combined by cross-correlation to facilitate clustering of frequency components according to specific speakers with specific pitches.

[0077] Figure 1Describes the paths that the frequency components of a speech signal follow from a human speaker 11 to a microphone array 12 in a reverberant environment. Walls 13 and other elements 14 in the environment reflect the impinging signal with attenuation and reflection angles that depend on the material and texture of the wall. Different frequency components of human speech may take different paths. They may be direct paths 15, or indirect paths 16, 17, which are located on the shortest path between the human speaker 11 and the microphone array 12. Note that a frequency component may propagate along one or more paths.

[0078] Figure 2 The algorithm is described. The signal is acquired by a microphone array 201, which contains M≥2 microphones, where M=7 microphones is an example. The microphones can be deployed in a series of constellations such as equidistantly on a straight line, on a circle or on a sphere, or even non-equidistantly to form an arbitrary shape. The signal from each microphone is sampled, digitized and stored in M ​​frames, each frame containing T consecutive samples 202. The size of the frame T can be selected to be large enough so that the short-time Fourier transform (STFT) is accurate, but short enough so that the signal is stationary along the equivalent duration. For a sampling rate of 16kHz, a typical value of T is 4096 samples, in other words, the frame is equivalent to 1 / 4 second. Typically, consecutive frames overlap each other to improve tracking after the characteristics of the signal change over time. The typical overlap is 75%, in other words, a new frame is started every 1024 samples. The range of T can be, for example, between 0.1 seconds and 2 seconds-thus sampling 1024-32768 for a sampling rate of 16kHz. The samples are also referred to as sound samples representing the sound signals received by the microphone array during the time period T.

[0079] Each frame is transformed to the frequency domain in 203 by applying a Fourier transform or a variant of the Fourier transform such as a short-time Fourier transform (STFT), a constant-Q transform (CQT), a log Fourier transform (LFT), a filter bank, etc. A variety of techniques, such as windowing and zero-padding, can be applied to control the framing effect. The result of 203 is M complex-valued vectors of length K. For example, if the array includes 7 microphones, 7 vectors are prepared, which are indexed by the frame time Register. K is the number of frequency bins and is determined by the frequency transform. For example, when using a normal STFT, K=T which is the length of the buffer. The output of step 203 may be referred to as a frequency transformed signal.

[0080] In 204, the speech signals are clustered according to different speakers. The clustering may be referred to as speaker-dependent clustering. Unlike prior art works that cluster speakers based only on direction, 204 handles multiple speakers in a reverberant room so that signals from different directions can be assigned to the same speaker due to direct and indirect paths. The proposed solution suggests using a set of acoustic cues (e.g., pitch frequency and intensity and its harmonic frequencies and intensities) over a set of spatial cues (e.g., direction (azimuth and elevation) and intensity of the signal in one of the microphones). One or more of the pitch and spatial cues are used as a state vector for a tracking algorithm (such as a Kalman filter and its variants, multi-hypothesis tracking (MHT) or a particle filter), which is used to track this state vector and assign each track to a different speaker.

[0081] All of these tracking algorithms use a model that describes the dynamics of the state vector in time, so that when measurements of the state vector are lost or corrupted by noise, the tracking algorithm uses the dynamic model to compensate for this and update the model parameters at the same time. The output of this stage is a vector that represents each frequency component at a given time Assigned to each speaker. 204 Figure 3 was further elaborated in.

[0082] The RTF estimator is applied to the data in the frequency domain in 205. The result of this stage is a collection of RTFs, each of which is registered to an associated speaker. The registration process is completed using the cluster array from clustering speakers 204. The collection of RTFs is also called speaker-dependent relative transfer function.

[0083] The MIMO beamformer 206 reduces the energy of noise and interference signals relative to the energy of the desired speech signal by spatial filtering. The output of step 206 can be referred to as a beamformed signal. The beamformed signal is then forwarded to an inverse frequency transform 207 to create a continuous speech signal in the form of a sample stream, which in turn is passed to other elements such as speech recognition, communication systems, and recording devices 208.

[0084] In a preferred embodiment of the present invention, keyword spotting 209 can be used to improve the performance of clustering block 204. Frames from 202 are searched for predefined keywords (e.g., "hello Alexa" or "ok Google"). Once the keyword is identified in the stream of frames, acoustic clues of the speaker are extracted, such as pitch frequency and intensity and its harmonic frequency and intensity. In addition, the characteristics of the path of each frequency component to the microphone array 201 are extracted. These features are used by speaker clustering 204 as seeds for clustering of the desired speaker. The seed is an initial guess about the initial parameters of the clustering. For example, the centroid, radius, and statistical values ​​of the clustering for a centroid-based clustering algorithm, such as K-means, PSO, and 2KPM. Another example is the base of a subspace for subspace-based clustering.

[0085] Figure 3 An algorithm for clustering speakers is described. It is assumed that each speaker is assigned a different set of acoustic cues, e.g., pitch frequency and intensity and its harmonic frequencies and intensities, such that the set of acoustic cues is a bijective indicator for the speaker. Acoustic cue detection is known to be a challenging task, especially in noisy, reverberant multi-speaker environments. To address this challenge, spatial cues in the form of, for example, signal strength, azimuth, and elevation are used. Acoustic cues are tracked over time using filters such as particle filters and extended Kalman filters (EKFs) to overcome temporarily inactive speakers and variations in acoustic cues, and spatial cues are used to segment frequency components between different sources. The results of the EKF and segmentation are combined by cross-correlation to facilitate clustering of frequency components according to specific speakers with specific pitches.

[0086] As an example of a preferred embodiment, in 31, potential acoustic clues in the form of pitch frequencies are detected. First, a time-frequency map is prepared using a frequency transform from a buffer of each microphone, which is calculated in 203. Next, the absolute value of each of the M K-length complex-valued vectors is weighted averaged with some weighting factors, which can be determined to reduce artifacts in some microphones. The result is a single K-length real vector. In this vector, values ​​above a given threshold μ are extracted, while the remaining elements are discarded. The threshold μ is typically adaptively selected to be three times the standard deviation of the noise, but not less than a constant value, which depends on the electrical parameters of the system and, in particular, on the number of effective bits of the sampled signal. Values ​​with frequency indices in the range of [k_min, k_max] are defined as candidates for pitch frequencies. The variables k_min and k_max are typically 85 Hz and 2550 Hz, respectively, because a typical adult male will have a fundamental frequency from 85 Hz to 1800 Hz, and a typical adult female has a fundamental frequency from 165 Hz to 2550 Hz. Each pitch candidate is then verified by searching for its higher harmonics. The presence of 2nd and 3rd harmonics may be a prerequisite for the candidate pitch to be detected as a legitimate pitch with reliability R (e.g., R=10). If higher harmonics (e.g., 4th and 5th) are present, the reliability of the pitch may be increased—e.g., the reliability may be doubled for each harmonic. Figure 4 An example can be found in . In a preferred embodiment of the invention, the desired speaker's pitch 32 is supplied via 210 using a key word uttered by the desired speaker. The supplied pitch 32 is added to the list with the greatest possible reliability (eg R=1000).

[0087] In 33, an extended Kalman filter (EKF) is applied to the pitch from 31. As pointed out in the Wikipedia entry on the extended Kalman filter (www.wikipedia.org / wiki / extended_Kalman_filter), the Kalman filter has a state transition equation and an observation model. For discrete calculations, the state transition equation is:

[0088] x k =f(x k-1 ,u k )+w k (1) And for discrete computation, the observation model is:

[0089] z k =h(x k )+vk (2)

[0090] where x k is the state vector, which contains the parameters that (partially) describe the state of the system, u k is a vector of external inputs that provide information about the state of the system, w k and v k is the process noise and the observation noise. The time updater of the extended Kalman filter can use the prediction equation to predict the next state, and the detected pitch can update the variable by comparing the actual measurement to the predicted measurement using an equation of the following type:

[0091] y k =z k -h(x k|k+1 ) (3)

[0092] where z k is the detected pitch, and y k is the error between the measured and predicted pitch.

[0093] In 33, each trajectory can start with a detected pitch, followed by the model f(x k ,u k ), which reflects the temporal behavior of pitch, which may become higher or lower due to emotion. The input to this model can be the past state vector x k (a state vector or multiple), and any external input u that affects the dynamics of the pitch k , the dynamics of the pitch such as speech speed, speech intensity and emotional expression. The elements of the state vector x can quantitatively describe the pitch. For example, the state vector of the pitch may include, among other things, the pitch frequency, the intensity of the first-order harmonic, and the frequency and intensity of higher-order harmonics. The vector function f(x k ,u k ) can be used to predict the state vector x at a given time k+1 before the current time. An exemplary implementation of the dynamic model in the EKF may include a time update equation (also called a prediction equation), as described in Jerry M. Mendel's book "Lessons in Digital Estimation Theory", which is incorporated herein by reference.

[0094] For example, consider the 3-tuple state vector:

[0095]

[0096] where f k is the frequency of the pitch (1st harmonic) at time k, a kis the intensity of the pitch (1st harmonic) at time k, and b k is the intensity of the second harmonic at time k.

[0097] An exemplary state vector model for pitch could be:

[0098]

[0099] It describes a model that assumes constant pitch. In a preferred embodiment of the invention, speech speed, speech intensity and emotional expression are continuously monitored using speech recognition algorithms known in the art, thereby providing external input u to improve the time update stage of the EKF k Emotional expression methods are known in the art. See, for example, Palo et al., “New Features for Emotional Speech Recognition”.

[0100] Each track is assigned a reliability field that is inversely proportional to the time over which the track evolves using only time updates. When the reliability of a track drops below a certain reliability threshold ρ, e.g. representing 10 seconds of undetected pitches, the track is defined as dead (no longer relevant), meaning that the corresponding speaker is not active. On the other hand, when a new measurement result (pitch detection) occurs that cannot be assigned to any existing track, a new track is started.

[0101] In 34, spatial cues are extracted from the M frequency transformed frames. As in 31, the most recent L vectors are saved for analysis using correlation in time. The result is a Time-Frequency-Cue (TFC) map, which is a 3-dimensional array of size LxKxP (where P=M-1) for each of the M microphones. Figure 5 TFC is described.

[0102] In 35, the spatial cues for each frequency component in the TFC are segmented. The idea is that along L frames, a frequency component may originate from different speakers, and this can be observed by comparing the spatial cues. However, due to the W-DO assumption, it is assumed that at a single frame time l, the frequency component originates from a single speaker. The segmentation can be performed using any known method for clustering in the literature, such as K-nearest neighbors (KNN). Clustering assigns an index to each cell in A. It indicates to which cluster the unit (k,l) belongs.

[0103] At 36, the frequency components of the signal are grouped so that each frequency component is assigned to a specific pitch in the list of pitches tracked by the EKF and is active for its reliability. This is done by calculating the time-frequency graph assigned to one of the pitches (see Figure 4 ) with all values ​​with a particular cluster index c0(j,l) in other rows of the time-frequency diagram. This is done for each cluster index. The sample cross-correlation is given by:

[0104]

[0105] where A is a time-frequency graph, k is the index of the row belonging to one of the pitches, j is any other row of A, and L is the number of columns of A. After computing the cross-correlation between each pitch and each of the clusters in the other rows, the cluster c1 in row j1 with the highest cross-correlation is grouped together with the corresponding pitch, and then the cluster c2 in row j2 with the second highest cross-correlation is grouped together with the corresponding pitch, and so on. This process is repeated until the sample cross-correlation drops below a certain threshold κ, which can be adaptively set to, say, 0.5x (the average energy of the signal at a single frequency). The result is a collection of frequency groups that are assigned corresponding pitch frequencies.

[0106] Figure 4 An example of pitch detection on a time-frequency graph is described. 41 is a time axis, which is composed of parameters denoted by , and 42 is the frequency axis, which is described by parameter k. Each column in this two-dimensional array is at time A real-valued vector of length K extracted in 31 after averaging the absolute values ​​of the M frequency-transformed buffers. For temporal correlation analysis, the L nearest vectors are stored in a two-dimensional array of size KxL. In 43, two pitches are represented by diagonals in different directions. Due to the presence of 4th harmonics, the pitch k=2 - whose harmonics are at k=4, 6, 8 - has a reliability R=20, and the pitch at k=3 - whose harmonics are at k=6, 9 - has a reliability R=10. In 44, the pitch at k=3 is inactive, and only k=2 is active. However, since the 4th harmonic is not detected (below the threshold μ), the reliability of the pitch at k=2 is reduced to R=10. In 45, the pitch at k=3 is active again, and k=2 is inactive. In 46, a new pitch candidate at k=4 appears, but only its 2nd harmonic is detected. Therefore, it is not detected as a pitch. In 47, the pitch of k=3 is inactive and no pitch is detected.

[0107] Figure 5 A TFC diagram is depicted whose axes are frame index (time) 51, frequency components 52 and spatial cues 53, which may be complex values ​​expressing, for example, the direction of arrival (azimuth and elevation) of each frequency component and the strength of the component. When the frame is processed and transferred to the frequency domain, for each frequency element A vector of M complex numbers is received. Up to M-1 spatial cues are extracted from each vector. In the example of the direction and strength of each frequency component, this may be done using any direction finding algorithm known in the art for array processing, such as MUSIC or ESPRIT. The result of this algorithm is a set of up to M-1 directions in three-dimensional space, each represented by two angles and an estimated strength of the arriving signal. The cues are arranged in the TFC diagram so that In by The unit indexed by k0, p0.

[0108] appendix

[0109] The performance of the speech enhancement module depends on the ability to filter out all interfering signals and leave only the desired speech signal. Interfering signals may be, for example, other speakers, noise from air conditioners, music, motor noise (e.g., in cars or airplanes), and large crowd noises also known as "cocktail party noises." The performance of the speech enhancement module is usually measured by its ability to improve the speech-to-noise ratio (SNR) or the speech-to-interference ratio (SIR), which reflects the ratio of the power of the desired speech signal to the total power of the noise and the ratio of the power of the desired speech signal to the total power of other interfering signals (usually in dB).

[0110] When the acquisition module contains a single microphone, the method is called single-microphone speech enhancement and is usually based on the statistical characteristics of the signal itself in the time-frequency domain, such as single channel spectral subtraction, spectral estimation using minimum variance distortion-free response (MVDR), and echo cancellation. When more than a single microphone is used, the acquisition module is usually called a microphone array, and the method is called multi-microphone speech enhancement. Many of these methods exploit the differences between the signals captured simultaneously by the microphones. One method that has been confirmed is beamforming, which calculates the total number of signals from the microphones after multiplying each signal by a weighting factor. The goal of the weighting factor is to average out interfering signals in order to adjust the signal of interest.

[0111] In other words, beamforming is a method of creating a spatial filter that algorithmically increases the power of signals transmitted from a given location in space (the desired signal from the desired speaker) and reduces the power of signals transmitted from other locations in space (interfering signals from other sources), thereby increasing the SIR at the output of the beamformer.

[0112] Delay-and-sum beamformer (DSB) involves using weighting factors for DSB that consist of counter delays implied by the different paths that a desired signal takes from its source to each microphone in the array. DSB is limited to signals that each come from a single direction, such as in a free-field environment. Therefore, in a reverberant environment, where signals from the same source propagate to the microphones along different paths and arrive at the microphones from multiple directions, DSB performance is typically inadequate.

[0113] To mitigate the disadvantages of DSB in reverberant environments, beamformers can use more complex acoustic transfer functions (ATFs), which represent the direction (azimuth and elevation) from which each frequency component arrives at a specific microphone from a given source. The single direction of arrival (DOA) assumed by DSB and other DOA-based methods is generally not applicable in reverberant environments, where components of the same speech signal arrive from different directions. This is because of the different frequency responses of physical elements in a reverberant environment, such as walls, furniture, and people. The ATF in the frequency domain is a vector that assigns complex numbers to each frequency in the Nyquist bandwidth. The absolute value represents the path gain associated with this frequency, and the phase indicates the phase added to the frequency component along that path.

[0114] Estimating the ATF between a given point in space and a given microphone can be accomplished by using a loudspeaker positioned at the given point and emitting a known signal. By simultaneously taking the signal from the speaker's input and the signal from the microphone's output, one can easily estimate the ATF. The loudspeaker can be located at one or more locations where a human speaker may reside during system operation. This method creates a graph of the ATF for each point in space or, more practically, for each point on a grid. The ATF for points not included in the grid is approximated using interpolation. Nevertheless - this method has serious disadvantages. First, the system needs to be calibrated for each installation, making this method impractical. Second, the acoustic differences between human speakers and electronic speakers, which deviate the measured ATF from the actual ATF. Third, the complexity of measuring a large number of ATFs, especially when the direction of the speaker is also considered, and fourth, possible errors due to environmental changes.

[0115] A more practical alternative to the ATF is the relative transfer function (RTF) as a remedy to the shortcomings of the ATF estimation method in practical applications. The RTF is the difference between the ATFs between a given source and two of the microphones in the array, and in the frequency domain, the difference takes the form of a ratio between the spectral representations of the two ATFs. Like the ATF, the RTF in the frequency domain assigns a complex number to each frequency. The absolute value is the gain difference between the two microphones, which is usually close to unity when the microphones are close to each other, and under some conditions, the phase reflects the angle of incidence of the source.

[0116] The transfer function-based linear constrained minimum variance (TF-LCMV) beamformer can reduce noise while limiting speech distortion, in multi-microphone applications, by minimizing the output energy under the constraint that the speech component in the output signal is equal to the speech component in one of the microphone signals. d +N i sources, considering the extraction of i N interference sources and stationary noise pollution d The problem is to find a desired speech source. Each of the signals involved propagates through an acoustic medium and is then picked up by an arbitrary array of M microphones. The signal from each microphone is split into frames of length T, and an FFT is applied to each frame. In the frequency domain, let us use and represents the mth microphone and the nth source respectively. The kth frequency component of the frame. Similarly, the ATF between the nth source and the mth microphone is And the noise at the mth microphone is The received signal in matrix form is given by:

[0117]

[0118] in is the sensor vector, is the source vector, is the ATF matrix such that and is an additive stationary noise that is uncorrelated with any source. Equivalently, (7) can be formulated using RTF. Without loss of generality, the RTF of the nth speech source can be defined as the ratio between the nth speech component at the mth microphone and its corresponding component at the first microphone, i.e. The signals in the RTF matrix can be used To formulate In vector notation:

[0119]

[0120] in is the source signal of the change.

[0121] Considering the array measurement results Need to estimate N d By using the beamformer Applied to microphone signal The desired signal can be extracted. Assuming M ≥ N, Can be selected to meet LCMV criteria:

[0122]

[0123] in yes The power spectral density (PSD) matrix of is the constraint vector.

[0124] A possible solution to (9) is:

[0125]

[0126] Based on (7) and (8) and the constraints set, the components of the desired signal at the beamformer output are given by Given, in other words, the output of the beamformer is a mixture of the components of the desired signal measured by the first (reference) microphone.

[0127] From the RTF sets and for each frequency component k, a set of up to M-1 sources—with incident angles p=1,..,P≤M-1 and elevation angle - an algorithm based on phase difference can be used, for example, together with the intensity obtained from the microphone defined as the reference microphone To extract these triplets Often referred to as spatial cues.

[0128] TF-LCMV is a suitable method for extracting M-1 speech sources impinging on an array consisting of M sensors from different locations in a reverberant environment. However, a necessary condition for TF-LCMV to work is that the RTF matrix whose columns are the RTF vectors of all active sources in the environment is known and available for TF-LCMV. This requires the association of each frequency component with its source speaker.

[0129] A variety of methods can be used to assign sources to signals without complementary information. The main family of methods is called blind source separation (BSS), which recovers the unknown signal or source from the observed mixture. The key weakness of BSS in the frequency domain is that at each frequency, the column vectors of the mixing matrix (estimated by BSS) are randomly arranged, and without knowledge of this random arrangement, it becomes difficult to combine the results across frequencies, as disclosed.

[0130] BSS can be assisted by pitch information. However, the gender of the speaker must be known a-priory.

[0131] BSS can be used in the frequency domain, while resolving ambiguities in the estimated mixing matrix using the maximum-magnitude method, which assigns a specific column of the mixing matrix to the source corresponding to the largest element in the vector. Nevertheless - this method depends heavily on the spectral distribution of the sources, since it is assumed that the strongest component at each frequency does belong to the strongest source. However, this condition is not often met, since different speakers may introduce intensity peaks at different frequencies. Alternatively, source activity detection, also known as voice activity detection (VAD), can be used, so that information about the sources that are active at a specific time is used to resolve ambiguities in the mixing matrix. The disadvantage of VAD is that it is not able to robustly detect voice pauses, especially in multi-speaker environments. In addition, this method is only effective when no more than a single speaker joins the conversation at a time, requires a relatively long training period, and is sensitive to motion during this period.

[0132] The TF-LCMV beamformer and its extensions can be used in binaural speech enhancement systems together with binaural cue generators. Acoustic cues are used to separate the speech component from the noise component in the input signal. This technique is based on the theory of auditory scene analysis. 1 , which suggests using unique perceptual cues to cluster signals from different speech sources in a "cocktail party" environment. Examples of primitive grouping cues that can be used for speech separation include common onset / offset across frequency bands, pitch (fundamental frequency), co-location in space, temporal and spectral modulation, pitch and energy continuity and smoothness. However, an implicit assumption of this approach is that all components of the desired speech signal have almost the same direction. In other words, almost free-field conditions, saving the impact of the head-shadow effect, which is suggested to be compensated by using head-related transfer functions. This is unlikely to happen in a reverberant environment.

[0133] It is important to note that even when multiple speakers are active simultaneously, the spectral contents of the speakers do not overlap at most time-frequency points. This is called W-separation orthogonality, or W-DO for short. This can be demonstrated by the sparsity of speech signals in the time-frequency domain. According to this sparsity, the probability of simultaneous activity of two speakers at a particular time-frequency point is very low. In other words, in the case of multiple simultaneous speakers, each time-frequency point most likely corresponds to the spectral content of one of the speakers.

[0134] W-DO can be used to facilitate BSS by defining a specific class of signals that are W-DO to some extent. This can be used only when first-order statistics are needed, which is computationally economical. In addition, if the source is W-DO and does not occupy the same spatial position, only two microphones can be used to demix any number of signal sources. However, this method assumes that the underlying mixing matrix across all frequencies is the same. This assumption is necessary for using a histogram of estimated mixing coefficients across different frequencies. However, this assumption is usually not established in a reverberant environment, but only established in a free field. The extension of this method to the multipath situation is limited to negligible energy from the multipath, or is limited to a sufficiently smooth convolution mixing filter so that the histogram is tailed, but a single peak is maintained. This assumption is also not established in a reverberant environment, in which the difference between different paths is usually too large to create a smooth histogram.

[0135] It has been found that the proposed solution performs in reverberant environments and does not have to rely on unnecessary assumptions and constraints. The solution works even without a priori information, even without a large training process, even without constraining the estimates of the attenuation and delay of a given source at each frequency to a single point in the attenuation-delay space, even without constraining the estimates of the attenuation-delay values ​​of a single source to create a single cluster, and even without limiting the number of mixed sounds to two.

[0136] Source separation to speech recognition engine

[0137] A voice user interface (VUI) is an interface between a human speaker and a machine. A VUI receives a voice signal using one or more microphones and converts the voice signal into a digital signature, typically by transcribing the voice signal into text, which is used to infer the speaker's intent. The machine can then respond to the speaker's intent based on the application the machine is designed for.

[0138] A key component of a VUI is an automatic speech recognition engine (ASR), which converts digitized speech signals into text. The performance of the ASR, in other words, how accurately the text describes the acoustic speech signal, depends largely on how well the input signal matches the requirements of the ASR. Therefore, other components of the VUI are designed to enhance the acquired speech signal before feeding it to the ASR. Such components may be noise suppression, echo cancellation, and source separation, to name a few.

[0139] One of the key components in speech enhancement is source separation (SS), which is intended to separate speech signals from several sources. Assuming an array of two or more microphones, the signal acquired by each of the microphones is a mixture of all speech signals in the environment plus other interferences (such as noise and music). The SS algorithm takes the mixed signal from all microphones and decomposes them into their components. In other words, the output of source separation is a set of signals, each representing a signal from a specific source, whether it is a speech signal from a specific speaker or music or even noise.

[0140] There is an increasing need to improve source separation.

[0141] Figure 6An example of a speech recognition chain in offline training is illustrated. The chain generally includes a microphone array 511, which provides a collection of digitized acoustic signals. The number of digitized acoustic signals is equal to the number of microphones that make up the array 511. Each digitized acoustic signal contains a mixture of all acoustic sources near the microphone array 511, whether it is a human speaker, a synthetic speaker such as TV, music and noise. The digitized acoustic signal is delivered to a pre-processing stage 512. The purpose of the pre-processing stage 512 is to improve the quality of the digitized acoustic signal by removing interference such as echo, reverberation and noise. The pre-processing stage 512 is usually performed using a multi-channel algorithm that uses statistical associations between digitized acoustic signals. The output of the pre-processing stage 512 is a collection of processed signals, which generally has the same number of signals as the number of digitized acoustic signals at the input to the stage. The collection of processed signals is forwarded to a source separation (SS) stage 513, which is intended to extract acoustic signals from each source near the microphone array. In other words, the SS stage 513 takes a collection of signals, each of which is a different mixture of acoustic signals received from different sources, and creates a collection of signals so that each signal contains mainly a single acoustic signal from a single specific source. Source separation of speech signals can be performed using geometric considerations of the deployment of sources such as beamforming or by considering the characteristics of speech signals such as independent component analysis. The number of separated signals is usually equal to the number of active sources near the microphone array 511, but less than the number of microphones. The collection of separated signals is forwarded to the source selector 514. The purpose of the source selector is to select the relevant speech source whose speech signal should be recognized. The source selector 514 can use a trigger word detector so that a source that emits a predefined trigger word sound is selected. Alternatively, the source selector 514 can consider the location of the source near the microphone array 511, such as a predefined direction relative to the microphone array 511. In addition, the source selector 514 can use a predefined acoustic signature of the speech signal to select a source that matches this signature. The output of the source selector 514 is a single speech signal, which is forwarded to a speech recognition engine 515. The speech recognition engine 515 converts the digitized speech signal into text. There are many methods known in the art for speech recognition, most of which are based on extracting features from the speech signal and comparing these features with predefined vocabulary. The main output of the speech recognition engine 515 is a text string 516 associated with the input speech signal. In offline training, a predefined text 518 is sounded to the microphone. The error 519 is calculated by comparing the output of the ASR 516 with this text. The comparison 517 can be performed using a simple word count or a more complex comparison method that takes into account the meaning of the word and appropriately weights the error detection of different words. The SS513 then uses the error 519 to modify the set of parameters to find the value that minimizes the error.This can be done by any supervised estimation or optimization method such as least squares, stochastic gradient, neural network (NN), and their variants.

[0142] Figure 7 An example of a speech recognition chain in real-time training, in other words, during normal operation of the system, is illustrated. During the operation of the VUI, the real text pronounced by the human speaker is unknown, and the supervised error 519 is unavailable. An alternative is a confidence score 521 developed for real-time applications, when there is no reference to the real spoken text, and the application can benefit from knowing the reliability level of the ASR output. For example, when the confidence score is low, the system may enter an appropriate branch in which a more direct conversation with the user is conducted. There are many methods for estimating confidence scores, most of which target a high correlation with the error that can be calculated when the spoken text is known. In real-time training, the confidence score 521 is converted into a supervised error 519 by an error estimator 522. When the confidence score is highly correlated with the theoretical supervised error, the error estimator 522 can be a simple axis conversion. When the confidence score 521 ranges from 0 to 100, the goal is to make it higher, the supervised error ranges from 0 to 100, and the goal is to make it lower. A simple axis transformation in the form of estimated error = 100 - confidence score is used as the error estimator 522. The estimated error 519 can be used to train the parameters of the SS, as well as offline training.

[0143] Figure 3 The training mechanism of a typical SS 513 is illustrated. The source separator (SS) 513 receives a set of mixed signals from the preprocessing stage 512 and feeds the separated signals to the source selector 514. In general, source separation of acoustic signals and in particular speech signals is performed in the frequency domain. The mixed signal from the preprocessing stage 512 is first transformed into the frequency domain 553. This is done by dividing the mixed signal into segments of the same length, with overlapping periods between the resulting segments. For example, when the length of the segment is determined to be 1024 samples and the overlapping period is 25%, then each of the mixed signals is divided into segments of 1024 samples each. A collection of concurrent segments from different mixed signals is called a batch. Each batch of segments starts 768 samples after the previous batch. Note that the segments across the set of mixed signals are synchronized, in other words, the starting points of all segments belonging to the same batch are the same. The length and overlapping period of the segments in a batch are obtained from the model parameters 552.

[0144] The unmixing algorithm 554 separates a batch of segments arriving from the frequency transform 553. Like many other algorithms, the source separation (SS) algorithm includes a set of mathematical models - accompanied by a set of model parameters 552. The mathematical model establishes the method of operation, such as the way SS copes with physical phenomena (e.g., multipath). The set of model parameters 552 adjusts the operation of SS to the specific characteristics of the source signals, to the architecture of the automatic speech recognition engine (ASR) that receives these signals, to the geometry of the environment, and even to the human speaker.

[0145] The unmixed batch of segments is forwarded to the inverse frequency transform 555, where it is transformed back to the time domain. The same set of model parameters 552 used in the frequency transform stage 553 is used in the inverse frequency transform stage 555. For example, the overlap period is used to reconstruct the output signal in the time domain from the resulting batch. This is done, for example, using an overlap-add method, in which after the inverse frequency transform, the resulting output signal is reconstructed by overlapping and adding the overlapping intervals, possibly using an appropriate weighting function ranging between 0 and 1 across the overlapping region so that the total energy is conserved. In other words, the overlapping segments from the previous batch fade out and the overlapping segments from the next batch fade in. The output of the inverse frequency transform block is forwarded to the source selector 514.

[0146] Model parameters 552 are a set of parameters used by frequency transform block 553, demixing block 554, and inverse frequency transform block 555. The mixed signal is divided into segments of equal length, which is done by frequency transform 553, and paced by a clocking mechanism such as a real-time clock. At each pace, each of frequency transform block 553, demixing block 554, and inverse frequency transform block 555 extracts parameters from model parameters 552. These parameters are then substituted into the mathematical models executed in frequency transform block 553, demixing block 554, and inverse frequency transform block 555.

[0147] Corrector 551 optimizes a set of model parameters 552 with the goal of reducing the error 519 from the error estimator. Corrector 551 receives the error 519 and the current set of model parameters 552, and outputs a set of corrected model parameters 552. Correction of the set of parameters can be done a priori (offline) or during VUI operation (real time). In offline training, the error 519 used to correct the set of model parameters 552 is extracted using a predefined text pronounced to the microphone and comparing the output of the ASR with this text. In real-time training, the error 519 is extracted from the confidence score of the ASR.

[0148] The error is then used to modify the set of parameters to find the value that minimizes the error. This can be done by any supervised estimation or optimization method, preferably a derivative-free method such as golden section search, grid search, and Nelder-Mead.

[0149] The Nelder-Mead method (also downhill simplex method, amoeba method or polytope method) is a commonly used numerical method for finding the minimum or maximum of an objective function in multidimensional space. It is a direct search method (based on function comparison) and is generally applicable to nonlinear optimization problems where the derivatives may not be known.

[0150] Nelder-Mead iteratively finds the local minimum of the error 519 as a function of several parameters. The method starts with a set of values ​​that determine a simplex (a generalized triangle in N dimensions). There is an assumption that a local minimum exists within the simplex. In each iteration, the errors at the vertices of the simplex are calculated. The vertex with the largest error is replaced with a new vertex so that the volume of the simplex is reduced. This is repeated until the simplex volume is less than a predefined volume and the optimal value is one of the vertices. The process is performed by a corrector 551.

[0151] The golden section search seeks the minimum of the error 519 by successively narrowing the range of values ​​within which the minimum is known to exist. The golden section search requires a strictly unimodal error as a function of the parameters. The narrowing operation is performed by the corrector 551.

[0152] The golden section search is a technique for finding the extrema (minimum or maximum) of a strictly unimodal function by successively narrowing the range of values ​​within which the extrema are known to exist. . (www.wikipedia.org).

[0153] The grid search iterates through a set of values ​​associated with one or more of the parameters to be optimized. When more than one parameter is optimized, each value in the set is a vector of length equal to the number of parameters. For each value, the error is calculated 519, and the value corresponding to the minimum error is selected. The iteration of the set of traversal values ​​is performed by a corrector 551.

[0154] Grid Search - The traditional way to perform hyperparameter optimization is a grid search or parameter sweep, which is simply an exhaustive search through a manually specified subset of a learning algorithm's hyperparameter space. The grid search algorithm must be guided by some performance metric, usually measured by cross-validation on a training set or evaluation on a hold-out validation set. Since the parameter space of a machine learning machine can include real-valued or unbounded value spaces for some parameters, it may be necessary to manually set bounds and discretization before applying the grid search. (www.wikipedia.org).

[0155] All optimization methods require continuous calculation of the error 519 with the same set of separated acoustic signals. This is a time consuming process and therefore may not be performed continuously, but only when the error 519 (which is continuously calculated) exceeds a certain predefined threshold (e.g., 10% error). When this occurs, two approaches can be taken.

[0156] One approach is to use parallel threads or multiple cores to run the optimization in parallel with the normal operation of the system. In other words, there are one or more parallel tasks that execute blocks 513, 514, 515, 522 in parallel with the tasks of the normal operation of the system. In the parallel tasks, a batch of mixed signals with a length of 1-2 seconds is obtained from the preprocessing 512, and it is repeatedly separated 513 and interpreted 514, 515 with different sets of model parameters 552. For each such cycle, the error 519 is calculated. In each cycle, the corrector 551 selects a set of model parameters according to the optimization method.

[0157] The second method is to run the optimization when there is no speech in the room. A voice activity detection (VAD) algorithm can be used to detect periods of missing human speech. These periods are used to optimize the model parameters 552 in the same manner as in the first method to save the need for parallel threads or multiple cores.

[0158] A suitable optimization method should be selected for each parameter in 552. Some methods are applicable to a single parameter, and some are applicable to a group of parameters. Several parameters that affect the performance of speech recognition are suggested below. In addition, optimization methods are suggested based on the characteristics of the parameters.

[0159] Segment length parameter.

[0160] The segment length parameter is related to FFT / IFFT. Typically, ASR using features of isolated phonemes requires short segments of about 20 milliseconds, while ASR using features of a series of resulting phonemes uses segments of about 100-200 milliseconds. On the other hand, the length of the segment is affected by the scene, such as the reverberation time of the room. The segment length should be approximately the reverberation time, which can be as much as 200-500 milliseconds. Since there is no sweet point for the length of the segment, this value should be optimized for the scene and ASR. In terms of samples, a typical value is 100-500 milliseconds. For example, a sampling rate of 8kHz means a segment length of 800-4000 samples. This is a continuous parameter.

[0161] Optimization of this parameter can be accomplished using various optimization methods, such as golden section search or Nelder-Mead together with overlap period. When using golden section search, the input to the algorithm is the minimum and maximum possible lengths, such as 10 milliseconds to 500 milliseconds, and error function 519. The output is the length of the segment that minimizes the error function 519. When using Nelder-Mead with overlap period, the input is a set of three pairs of overlap period and segment length, such as (10 milliseconds, 0%), (500 milliseconds, 10%) and (500 milliseconds, 80%) and error function 519, and the output is the optimal segment length and the optimal overlap period.

[0162] Overlapping time

[0163] The overlap period parameter is related to FFT / IFFT. The overlap period is used to avoid missing phonemes due to segmentation. In other words, phonemes are divided between the resulting segments. Due to the length of the segment, the overlap period depends on the features used by the ASR. Typical range - 0-90% of the length of the segment. This is a continuous parameter.

[0164] Optimization of this parameter can be done using various optimization methods, such as golden section search or Nelder-mead and segment length. When using golden section search, the input to the algorithm is the minimum and maximum possible overlap period, for example 0% to 90%, and the error function 519. The output is the overlap period that minimizes the error function 519.

[0165] Window. The window parameters are related to FFT / IFFT. Frequency transform 553 usually uses windowing to mitigate the effects of segmentation. Some windows such as Kaiser and Chebyshev are parameterized. This means that the effect of the window can be controlled by changing the parameters of the window. The typical range depends on the type of window. This is a continuous parameter. Various optimization methods (such as golden section search) can be used to optimize this parameter. When using the golden section search, the input to the algorithm is the minimum and maximum values ​​of the parameters of the window, which depends on the window type, and the error function 519. For example, for the Kaiser window, the minimum and maximum values ​​are (0,30). The output is the optimal window parameters.

[0166] Sampling rate

[0167] The sampling rate parameter is related to FFT / IFFT. Sampling rate is one of the key parameters that affect the performance of speech recognition. For example, there are ASRs that show poor results for sampling rates below 16kHz. Others can work well even at 4kHz or 8kHz. Usually, when ASR is selected, this parameter is optimized once. The typical range is 4, 8, 16, 44.1, 48kHz. This parameter is a discrete parameter. Various optimization methods (such as grid search) can be used to optimize this parameter. The input to the algorithm is the value for performing the grid search - for example, the sampling rate of (4, 8, 16, 44.1, 48)kHz, and the error function 519. The output is the optimal sampling rate.

[0168] Filtering

[0169] The filtering parameters are related to unmixing. Some ASRs use features that represent a limited frequency range. Therefore, filtering the separated signal after source separation 513 can emphasize the specific features used by the ASR, thereby improving its performance. More importantly, filtering out spectral components not used by the ASR can improve the signal-to-noise ratio (SNR) of the separated signal, which in turn can improve the performance of the ASR. The typical range is 4kHz-8kHz. Various optimization methods (such as golden section search) can be used to optimize this parameter. This parameter is continuous. When applying the golden section search, the input to the algorithm is an initial guess of the error function 519 and the portion of the cutoff frequency, such as 1000Hz and 0.5X sampling rate. The output is the optimal filtering parameters.

[0170] Weighting factors for each microphone. The weighting factors for each microphone are related to unmixing. In theory, the sensitivity of different microphones on a particular array should be similar, up to 3dB. However, in practice, the span of sensitivity of different microphones can be larger. In addition, the sensitivity of the microphones may change over time due to dust and humidity. The typical range is 0-10dB. This is a continuous parameter. Whether or not there is a weighting factor for each microphone, optimization of this parameter can be accomplished using various optimization methods (such as Nelder-mead). When the Nelder-mead method is applied, the input to the algorithm is the error function 519 and the initial guess of the simplex vertices. For example, the size of each n-tuple is the number of microphones - N: (1,0,...,0,0), (0,0,...,0,1) and (1 / N,1 / N,...,1 / N). The output is the optimal weighting for each microphone.

[0171] Number of microphones

[0172] The number of microphones is related to unmixing. The number of microphones affects the number of sources that can be separated on the one hand, and affects complexity and numerical precision on the other hand. Actual experiments also show that too many microphones may lead to a reduction in output SNR. The typical range is 4-8. This is a discrete parameter. Various optimization methods (such as grid search or Nelder-mead) can be used to optimize this parameter. When applying grid search, the input of the algorithm is the error function 519 and the number of microphones performing the search. For example, 4, 5, 6, 7, 8 microphones. The output is the optimal number of microphones.

[0173] Fig. 9 Method 600 is illustrated.

[0174] The method 600 may begin at step 610 of receiving or calculating an error associated with a speech recognition process applied to a previous output of a source selection process.

[0175] Step 610 may be followed by step 620 of modifying at least one parameter of the source separation process based on the error.

[0176] Step 620 may be followed by step 630 of receiving a signal representing audio signals originating from a plurality of sources and detected by the microphone array.

[0177] Step 630 may be followed by step 640 of performing a source separation process for separating audio signals originating from different sources of the plurality of sources to provide source separated signals and sending the source separated signals to the source selection process.

[0178] Step 640 may be followed by step 630 .

[0179] Each iteration or iterations of steps 630 and 640 may be followed by step 610 (not shown) - wherein the output of step 640 may be fed to the source selection process and ASR to provide the previous output of the ASR.

[0180] It should be noted that initial iterations of step 630 and step 640 may be performed without receiving an error.

[0181] Step 640 may include applying a frequency transform (such as but not limited to an FFT), demixing, and applying an inverse frequency transform (such as but not limited to an IFFT).

[0182] Step 620 may include at least one of the following:

[0183] a. Modify at least one parameter of the frequency conversion.

[0184] b. Modify at least one parameter of the inverse frequency conversion.

[0185] c. Modify at least one parameter of the unmixing.

[0186] d. Modifying the length of a segment of the signal representing the audio signal to which the frequency transform is applied.

[0187] e. Modifying the overlap between consecutive segments of a signal representing an audio signal, wherein a frequency transformation is applied on a segment-to-segment basis.

[0188] f. Modify the sampling rate of frequency conversion.

[0189] g. Modify the windowing parameters of the window applied by the frequency conversion.

[0190] h. Modify the cutoff frequency of the filter applied during demixing.

[0191] i. Modify the weighting applied to each microphone in the microphone array during demixing.

[0192] j. Modify the number of microphones in the microphone array.

[0193] k. Determine a modified value of at least one parameter using a golden section search.

[0194] 1. Determine a modified value of at least one parameter using the Nelder-Mead algorithm.

[0195] m. Determine a modified value of at least one parameter using a grid search.

[0196] n. Determining a modified value of one parameter of the at least one parameter based on a predefined mapping between the error and the at least one parameter.

[0197] o. Determine in real time a mapping between the error and at least one parameter.

[0198] In the foregoing specification, the invention has been described with reference to specific examples of embodiments of the invention. It will, however, be evident that various modifications and changes may be made therein without departing from the broader spirit and scope of the invention as set out in the appended claims.

[0199] Furthermore, the terms "front", "rear", "top", "bottom", "above", "below", etc., if any, in the specification and claims are used for descriptive purposes and are not necessarily used to describe permanent relative positions. It is understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein are, for example, capable of operation in other orientations than those illustrated or otherwise described herein.

[0200] Any arrangement of components that achieve the same function is effectively "associated" so that the desired function is achieved. Therefore, any two components that are combined to achieve a specific function in this article can be regarded as "associated" with each other so that the desired function is achieved, regardless of the architecture or intermediate components. Similarly, any two components so associated can also be regarded as "operably connected" or "operably coupled" to each other to achieve the desired function.

[0201] In addition, those skilled in the art will recognize that the boundaries between the operations described above are only illustrative. Multiple operations can be combined into a single operation, a single operation can be distributed in additional operations, and operations can be performed at least partially overlapping in time. In addition, alternative embodiments can include multiple instances of specific operations, and the order of operations can be changed in each other embodiment.

[0202] However, other modifications, changes and alternatives are possible. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive.

[0203] The phrase "may be X" indicates that condition X may be satisfied. This phrase also indicates that condition X may not be satisfied. For example - any reference to a system including a certain component should also cover scenarios where the system does not include the certain component. For example - any reference to a method including a certain step should also cover scenarios where the method does not include the certain component. For another example - any reference to a system configured to perform a certain operation should also cover scenarios where the system is not configured to perform the certain operation.

[0204] The terms "comprising", "including", "having", "consisting of", and "consisting essentially of" are used in an interchangeable manner. For example - any method may include at least the steps included in the drawings and / or description, or only the steps included in the drawings and / or description. The same applies to the system.

[0205] The system may include a microphone array, a memory unit and one or more hardware processors, such as a digital signal processor, FPGA, ASIC, general purpose processor, etc., programmed to perform any of the above mentioned methods. The system may not include a microphone array, but may be fed from a sound signal generated by a microphone array.

[0206] It should be understood that for simplicity and clarity of illustration, the elements shown in the drawings are not necessarily drawn to scale. For example, for clarity, the size of some elements may be exaggerated relative to other elements. In addition, reference numerals may be repeated among the drawings to indicate corresponding or similar elements where deemed appropriate.

[0207] In the foregoing specification, the invention has been described with reference to specific examples of embodiments of the invention. It will, however, be evident that various modifications and changes may be made therein without departing from the broader spirit and scope of the invention as set out in the appended claims.

[0208] Furthermore, the terms "front", "rear", "top", "bottom", "above", "below", etc., if any, in the specification and claims are used for descriptive purposes and are not necessarily used to describe permanent relative positions. It is understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein are, for example, capable of operation in other orientations than those illustrated or otherwise described herein.

[0209] Those skilled in the art will recognize that the boundaries between logic blocks are merely illustrative, and that alternative implementations may merge logic blocks or circuit elements or impose alternative decompositions of functionality on various logic blocks or circuit elements. It should therefore be understood that the architectures described herein are merely exemplary, and that in fact many other architectures that achieve the same functionality may be implemented.

[0210] Any arrangement of components that achieve the same function is effectively "associated" so that the desired function is achieved. Therefore, any two components that are combined to achieve a specific function in this article can be regarded as "associated" with each other so that the desired function is achieved, regardless of the architecture or intermediate components. Similarly, any two components so associated can also be regarded as "operably connected" or "operably coupled" to each other to achieve the desired function.

[0211] In addition, those skilled in the art will recognize that the boundaries between the operations described above are only illustrative. Multiple operations can be combined into a single operation, a single operation can be distributed in additional operations, and operations can be performed at least partially overlapping in time. In addition, alternative embodiments can include multiple instances of specific operations, and the order of operations can be changed in each other embodiment.

[0212] Also for example, in one embodiment, the illustrated examples may be implemented as circuit systems located on a single integrated circuit or within the same device. Alternatively, the examples may be implemented as any number of separate integrated circuits or separate devices interconnected with each other in a suitable manner.

[0213] Also for example, the examples or portions thereof may be implemented as physical circuitry or as soft or code representations of logical representations convertible into physical circuitry, such as in any suitable type of hardware description language.

[0214] Furthermore, the present invention is not limited to physical devices or units implemented in non-programmable hardware, but may be applied to programmable devices or units that perform desired device functions by operating according to appropriate program code, such as mainframes, minicomputers, servers, workstations, personal computers, notebook computers, personal digital assistants, electronic game consoles, automotive and other embedded systems, cellular telephones and various other wireless devices, generally denoted as "computer systems" in this application.

[0215] However, other modifications, changes and alternatives are possible. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive.

[0216] In the claims, any reference marks placed between brackets shall not be interpreted as limiting the claims. The word "comprising" does not exclude the presence of other elements or steps other than those listed in the claims. In addition, the terms "a" or "an" as used herein are defined as one or more than one. In addition, the use of introductory phrases such as "at least one" and "one or more" in the claims shall not be interpreted as implying that the introduction of another claim element by the indefinite article "a" or "an" will limit any particular claim containing such introduced claim elements to an invention containing only one such element, even if the same claim includes the introductory phrases "one or more" or "at least one" and indefinite articles such as "a" or "an". The same is true for the use of definite articles. Unless otherwise specified, terms such as "first" and "second" are used to arbitrarily distinguish the elements described by such terms. Therefore, these terms are not necessarily intended to indicate the time or other priority of such elements, and the fact that certain measures are recorded in mutually different claims does not indicate that the combination of these measures cannot be used advantageously.

[0217] The present invention may also be implemented in a computer program for running on a computer system, the computer program comprising at least code portions for executing the steps of the method according to the present invention or for enabling the programmable device to perform the functions of the device or system according to the present invention when running on a programmable device such as a computer system. The computer program may cause the storage system to assign the disk drive to the disk drive group.

[0218] A computer program is a list of instructions such as a specific application and / or operating system. A computer program may include, for example, one or more of the following: a subroutine, a function, a procedure, an object method, an object implementation, an executable application, an applet, a servlet, source code, object code, a shared library / dynamically loaded library, and / or other instruction sequences designed to be executed on a computer system.

[0219] Computer programs may be stored internally on non-transitory computer-readable media. All or some of the computer programs may be provided on a computer-readable medium that is permanently, removably or remotely coupled to an information processing system. Computer-readable media may include, for example but not limited to, any number of the following items: magnetic storage media, including disk and tape storage media; optical storage media, such as optical disk media (e.g., CD ROM, CD-R, etc.) and digital video disk storage media; non-volatile storage media, including semiconductor-based memory cells, such as flash memory, EEPROM, EPROM, ROM; ferromagnetic digital memory; MRAM; volatile storage media, including registers, buffers or caches, main memory, RAM, etc. A computer process typically includes executing (running) a program or a portion of a program, current program values ​​and state information, and an operating system to manage the resources used by the execution of the process. An operating system (OS) is software that manages the sharing of resources of a computer and provides an interface for programmers to access these resources. The operating system processes system data and user input, and responds by allocating and managing tasks and internal system resources as a service to users and programs of the system. A computer system may, for example, include at least one processing unit, associated memory, and a plurality of input / output (I / O) devices. When executing a computer program, the computer system processes information according to the computer program and generates resulting output information through I / O devices.

[0220] Any system mentioned in this patent application includes at least one hardware component.

[0221] Although certain features of the present invention have been illustrated and described herein, many modifications, substitutions, changes and equivalents will now occur to those skilled in the art. Therefore, it should be understood that the appended claims are intended to cover all such modifications and changes that fall within the true spirit of the invention.

Claims

1. A method for speech enhancement, the method comprising: frequency transforming the sound samples to provide frequency transformed samples, the sound samples representing the sound signals received by the microphone array during a given time period; clustering the frequency-transformed samples into a plurality of speaker-related clusters, the plurality of speaker-related clusters corresponding to a plurality of speakers, respectively, wherein the clustering is based on spatial cues associated with the sound signal and based on acoustic cues associated with the plurality of speakers, wherein a speaker-related cluster corresponding to a speaker of the plurality of speakers includes frequency-transformed samples, the frequency-transformed samples being associated with the speaker based on the spatial cues and the acoustic cues; determining a plurality of speaker-dependent relative transfer functions, the plurality of speaker-dependent relative transfer functions corresponding to the plurality of speakers, respectively, wherein determining the plurality of speaker-dependent relative transfer functions comprises determining a speaker-dependent relative transfer function corresponding to the speaker of the plurality of speakers based on the frequency-transformed samples in the speaker-dependent cluster corresponding to the speaker; applying a multiple-input multiple-output (MIMO) beamforming operation on the relative transfer functions associated with the plurality of speakers to provide a beamformed signal; as well as performing an inverse frequency transform on the beamformed signal to provide a speech signal, Wherein determining the speaker-dependent relative transfer function corresponding to the speaker comprises determining the speaker-dependent relative transfer function to represent a ratio in the frequency domain between two acoustic transfer functions of the speaker with respect to two corresponding microphones in the microphone array.

2. The method of claim 1, comprising generating the acoustic cues by: searching for a keyword in the sound sample; and The acoustic clues are extracted from the keywords. The method according to claim 2 , comprising extracting spatial clues related to the keyword. 4 . The method of claim 3 , comprising using spatial cues associated with the keyword as clustering seeds for clustering the frequency transformed samples into the plurality of speaker-related clusters.

5. The method of claim 1, wherein the acoustic cues include: A pitch frequency, a pitch intensity, one or more harmonics of the pitch frequency, and an intensity of the one or more harmonics of the pitch frequency. 6 . The method of claim 1 , comprising associating a reliability attribute with a pitch and determining that a speaker associated with the pitch is silent when the reliability of the pitch falls below a predefined threshold.

7. The method of claim 1 , wherein the clustering comprises processing the frequency-transformed samples to provide the acoustic cues and the spatial cues; using the acoustic cues to track the time-varying state of a speaker; segmenting the spatial cues of the frequency components of the frequency-transformed samples into groups; and assigning acoustic cues associated with an active speaker to a group of frequency-transformed samples.

8. The method according to claim 7, wherein the assigning comprises calculating, for the set of frequency transformed samples, cross-correlations between elements of equal-frequency rows of a time-frequency diagram and elements belonging to other rows of the time-frequency diagram and associated with the set of frequency transformed samples.

9. The method of claim 7, wherein the tracking comprises applying an extended Kalman filter.

10. The method of claim 7, wherein the tracking comprises applying multi-hypothesis tracking. The method of claim 7 , wherein the tracking comprises applying a particle filter.

12. The method of claim 7, wherein the segmenting comprises assigning a single frequency component associated with a single time frame to a single speaker.

13. The method of claim 7, comprising monitoring at least one of speech speed, speech intensity, and emotional expression as an acoustic feature.

14. The method of claim 13, comprising feeding the at least one monitored acoustic feature to an extended Kalman filter.

15. The method according to any one of claims 1-14, wherein clustering the frequency transformed samples into the plurality of speaker-dependent clusters comprises: processing the frequency transformed samples to detect the acoustic cues based on a time-frequency graph of the frequency transformed samples; processing the frequency transformed samples to extract the spatial cues in a time-frequency-cue map; as well as The frequency transformed samples are assigned to the plurality of speaker-related clusters based on the acoustic cues and the spatial cues in the time-frequency-cue map.

16. The method according to any one of claims 1 to 14, wherein clustering the frequency-transformed samples comprises: The acoustic cues are used to assign the sound signals received from both the direct path and the indirect path to the same speaker.

17. The method of any one of claims 1-14, comprising processing the frequency transformed samples arranged in a plurality of vectors corresponding to a respective plurality of microphones of the microphone array, wherein processing the frequency transformed samples comprises: Calculating an intermediate vector by weighted averaging the plurality of vectors; And searching for acoustic cue candidates by ignoring elements of the intermediate vector having values ​​below a predefined threshold. The method of claim 17 , wherein the predefined threshold is three times the standard deviation of the noise.

19. A non-transitory computer readable medium storing instructions which, once executed by a computerized system, cause the computerized system to perform the method according to any one of claims 1-18.

20. A computerized system comprising a microphone array, a memory unit and a processor, wherein the processor is configured to perform the method according to any one of claims 1-18.

Images