Method for source isolation, source isolation apparatus, and system

Abstract

To provide a method and a system for enhancing speech in echo environment.SOLUTION: A method includes: a step for receiving or generating a sound sample 202 indicating a sound signal which an array 201 of a microphone receives during a prescribed time period; a step 203 for frequency-converting a sound sample for providing a sample whose frequency is converted; a step 204 for clustering the sample whose frequency is converted to a speaker based on a spatial cue related to the received sound signal and an acoustic cue related to the speaker for providing a speaker-related cluster; a step 205 for determining a relative transfer function for each speaker for providing a speaker-related relative transfer function; a step 206 for applying multiple input multiple output MIMO beam formation operation to the speaker-related relative transfer function for providing a beam-formed signal; and a step 207 for inverse-frequency-converting the beam-formed signal for providing a voice signal.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] (Background technology) The performance of a speech enhancement module is determined by its ability to filter out all interfering signals, leaving only the desired speech signal. Interfering signals can include, for example, noise from other speakers, air conditioning, music, motor noise (e.g., inside a car or airplane), and crowd noise, also known as "cocktail party noise." The performance of a speech enhancement module is usually judged by its ability to improve the speech-to-noise ratio (SNR) or speech-to-interference ratio (SIR), which typically reflects the ratio (often on a dB scale) of the desired speech signal's power to the total power of the noise and other interfering signals, respectively. [Prior art documents] [Non-patent literature]

[0002] [Non-Patent Document 1] "Lessons in Digital Estimation Theory" by Jerry M. Mendel [Non-Patent Document 2] “New Features for Emotional Speech Recognition” by Palo et. al. [Overview of the project] [Problems that the invention aims to solve]

[0003] The need to enhance audio in reverberant environments is increasing. [Means for solving the problem]

[0004] A method for voice enhancement can be provided. The method includes receiving or generating sound samples representing sound signals received by an array of microphones during a given time period, frequency converting the sound samples to provide frequency-converted samples, clustering the frequency-converted samples to speakers to provide speaker-related clusters, where the clustering step can be based on (i) a spatial cue related to the received sound signal and (ii) an acoustic cue related to the speakers, determining a relative transfer function for each individual speaker of the speakers to provide speaker-related relative transfer functions, applying a multiple-input multiple-output (MIMO) beamforming operation to the speaker-related relative transfer functions to provide a beamformed signal, and inverse frequency converting the beamformed signal to provide a voice signal.

[0005] The method can include generating an acoustic cue related to the speakers.

[0006] The step of generating an acoustic cue can include searching for keywords in the sound samples and extracting an acoustic cue from the keywords.

[0007] The method can include extracting a spatial cue related to the keywords.

[0008] The method can include using the spatial cue related to the keywords as a clustering seed.

[0009] The acoustic cue can include a pitch frequency, a pitch intensity, one or more pitch frequency harmonics, and intensities of one or more pitch frequency harmonics.

[0010] The method can include steps of associating reliability attributes with individual pitches, and determining that a speaker associated with a pitch can become silent when the reliability of the pitch falls below a defined threshold.

[0011] The clustering step can include steps of processing frequency-converted samples to provide acoustic cues and spatial cues, constantly tracking the state of the speaker using the acoustic cues, segmenting spatial cues of individual frequency components of the frequency-converted signal into groups, and assigning acoustic cues related to speakers in the current active state to individual groups of the frequency-converted signal.

[0012] The assigning step can include steps of calculating the cross-correlation between elements of equal-frequency lines of a time-frequency map for each group of the frequency-converted signal, where the elements belong to other lines of the time-frequency map and can be associated with the group of the frequency-converted signal.

[0013] The tracking step can include steps of applying an extended Kalman filter.

[0014] The tracking step can include steps of applying multiple hypothesis tracking.

[0015] The tracking step can include steps of applying a particle filter.

[0016] The segmenting step can include steps of assigning a single frequency component related to a single time frame to a single speaker.

[0017] The method can include steps of monitoring at least one monitored acoustic feature selected from the group consisting of sound velocity, sound intensity, and emotional vocalization.

[0018] The method may include the step of supplying at least one acoustic feature to be monitored to an extended Kalman filter.

[0019] The frequency-converted samples can be arranged in multiple vectors, one vector for each microphone in an array of microphones, and the method may include the steps of calculating an intermediate vector by weight averaging the multiple vectors and searching for acoustic cue candidates by ignoring elements of the intermediate vector that may have values ​​below a predefined threshold.

[0020] The method may include a step of determining a predefined threshold such that it is three times the standard deviation of the noise.

[0021] A non-temporary computer-readable medium can be provided that stores instructions which, when executed by a computerized system, will receive or generate sound samples representing sound signals received by an array of microphones over a given time period, frequency-transform the sound samples to provide frequency-transformed samples, cluster the frequency-transformed samples to speakers to provide speaker-related clusters (this clustering may be based on (i) spatial cues associated with the received sound signals and (ii) acoustic cues associated with the speakers), determine the relative transfer function for each individual speaker of the speakers to provide speaker-related relative transfer functions, apply multiple-input multiple-output (MIMO) beamforming operations to the speaker-related relative transfer functions to provide beamformed signals, and inverse frequency-transform the beamformed signals to provide audio signals.

[0022] A non-temporary computer-readable medium can store instructions for generating acoustic cues associated with a speaker.

[0023] The generation of acoustic cues may include searching for keywords in sound samples and extracting acoustic cues from those keywords.

[0024] The generation of acoustic cues may include searching for keywords in sound samples and extracting acoustic cues from those keywords.

[0025] Non-temporary computer-readable media can store instructions for extracting spatial queues associated with keywords.

[0026] Non-temporary computer-readable media can store instructions for using spatial cures associated with keywords as clustering seeds.

[0027] An acoustic cue may include a pitch frequency, a pitch intensity, one or more pitch frequency harmonics, and the intensity of one or more pitch frequency harmonics.

[0028] A non-temporary computer-readable medium can associate reliability attributes with individual pitches, and a speaker that can be associated with a pitch can store instructions for determining whether it may become silent when the reliability of the pitch falls below a defined threshold.

[0029] Clustering may include processing frequency-transformed samples to provide acoustic and spatial cues, continuously tracking speaker status using acoustic cues, segmenting spatial cues of individual frequency components of the frequency-transformed signal into groups, and assigning acoustic cues to each group of frequency-transformed signals that are associated with the currently active speaker.

[0030] The assignment may include calculating the cross-correlation between elements of isofrequency lines in a time-frequency map that belong to other lines in the time-frequency map and can also be associated with the group of frequency-converted signals, for each group of frequency-converted signals.

[0031] Tracking can include applying an extended Kalman filter.

[0032] Tracking can include applying multiple hypothesis tracking.

[0033] Tracking can include applying particle filters.

[0034] Segmentation can include assigning a single frequency component associated with a single time frame to a single speaker.

[0035] A non-temporary computer-readable medium can store commands for monitoring at least one of the following acoustic features: speech speed, speech intensity, and emotional utterance.

[0036] A non-temporary computer-readable medium can store instructions for supplying at least one monitored acoustic feature to an extended Kalman filter.

[0037] The frequency-converted samples can be arranged in multiple vectors, one vector for each microphone in an array of microphones, and a non-temporary computer-readable medium can store instructions for searching for acoustic cue candidates by calculating an intermediate vector by weight-averaging the multiple vectors and by ignoring elements of the intermediate vector that may have values ​​below a predefined threshold.

[0038] Non-temporary computer-readable media can store instructions for determining a predefined threshold such that the noise standard deviation is three times the threshold.

[0039] A computerized system can be provided that includes an array of microphones, a storage device, and a processor. The processor can be configured to receive or generate sound samples representing sound signals received by the array of microphones during a given time period, to frequency-convert the sound samples to provide frequency-converted samples, to cluster the frequency-converted samples to speakers to provide speaker-associated clusters (this clustering may be based on (i) spatial cues associated with the received sound signals and (ii) acoustic cues associated with the speakers), to determine a relative transfer function for each individual speaker of the speakers to provide a speaker-associated relative transfer function, to apply a multiple-input multiple-output (MIMO) beamforming operation to the speaker-associated relative transfer function to provide a beamformed signal, and to inverse frequency-convert the beamformed signal to provide an audio signal, and the storage device can be configured to store at least one of the sound samples and the audio signal.

[0040] A computerized system cannot include an array of microphones, but it can receive signals from an array of microphones that represent the sound signals received by the array of microphones during a given time period.

[0041] The processor can be configured to generate acoustic cues related to the speaker.

[0042] The generation of acoustic cues may include searching for keywords in sound samples and extracting acoustic cues from those keywords.

[0043] The processor can be configured to extract spatial queues related to keywords.

[0044] The processor can be configured to use spatial cures associated with keywords as clustering seeds.

[0045] An acoustic cue may include a pitch frequency, a pitch intensity, one or more pitch frequency harmonics, and the intensity of one or more pitch frequency harmonics.

[0046] The processor can be configured to associate reliability attributes with individual pitches, and speakers that may be associated with a pitch can be configured to determine that they may become silent if the pitch reliability falls below a predefined threshold.

[0047] The processor can be configured to cluster frequency-transformed samples to provide acoustic and spatial cues, to continuously track speaker states using acoustic cues, to segment spatial cues of individual frequency components of the frequency-transformed signal into groups, and to assign acoustic cues to each group of frequency-transformed signals that are associated with the currently active speaker.

[0048] The processor can be configured to assign each group of frequency-converted signals by calculating the cross-correlation between elements of isofrequency lines in the time-frequency map, which have elements that belong to other lines in the time-frequency map and can also be associated with the group of frequency-converted signals.

[0049] The processor can be configured to track by applying an extended Kalman filter.

[0050] The processor can be configured to track by applying multiple hypothesis tracking.

[0051] The processor can be configured to track particles by applying a particle filter.

[0052] The processor can be configured to segment by assigning a single frequency component associated with a single time frame to a single speaker.

[0053] The processor can be configured to monitor at least one of the following acoustic features: speech velocity, speech intensity, and emotional utterance.

[0054] The processor can be configured to supply at least one monitored acoustic feature to the extended Kalman filter.

[0055] The frequency-converted samples can be arranged in multiple vectors, one vector for each microphone in an array of microphones, and the processor can be configured to search for acoustic cue candidates by calculating an intermediate vector by weight-averaging the multiple vectors and by ignoring elements of the intermediate vector that may have values ​​below a predefined threshold.

[0056] The processor can be configured to determine a predefined threshold that is three times the standard deviation of the noise.

[0057] To better understand the present invention and to explore ways in which it can be put into practice, preferred embodiments will be described below by reference to the accompanying drawings, in a manner that is merely non-limiting. [Brief explanation of the drawing]

[0058] [Figure 1] This diagram shows multiple passages. [Figure 2] This diagram shows an example of the method. [Figure 3] This figure shows an example of the clustering step in the method shown in Figure 2. [Figure 4] This figure shows an example of pitch detection on a time-frequency map. [Figure 5] This figure shows an example of a time-frequency-queue map. [Figure 6] This figure shows an example of a voice recognition chain in offline training. [Figure 7] This figure shows an example of a voice recognition chain in real-time training. [Figure 8]This figure shows an example of a training mechanism. [Figure 9] This diagram shows an example of the method. [Modes for carrying out the invention]

[0059] All references to the system shall apply, with any necessary modifications, to non-temporary computer-readable media that store the methods performed by the system and / or the instructions that, when executed by the system, cause the system to perform those methods.

[0060] All references to the methods shall apply to systems configured to perform the methods, with the necessary modifications, and to non-temporary computer-readable media that store instructions that, when executed by the systems, would cause those systems to perform the methods.

[0061] All references to non-temporary computer-readable media shall apply to methods executed by the system and / or by systems configured to execute instructions stored in non-temporary computer-readable media, with the necessary modifications.

[0062] The terms "and / or" are additional or alternative.

[0063] The term "system" refers to a computerized system.

[0064] Audio enhancement methods focus on extracting audio signals from a desired source (speaker) when the signal is interfered with by noise and other speakers. In anechoic environments, spatial filtering in the form of directional beamforming is effective. However, in reverberant environments, audio from individual sources is blurred in several directions and is not necessarily continuous, thus not taking advantage of the benefits of conventional beamformers. Addressing this problem using transfer function (TF) based beamformers, or using the relative transfer function (RTF) as the TF itself, is a promising approach. However, in multi-speaker environments, the ability to predict the RTF for each speaker remains a challenge when audio signals are captured simultaneously. Solutions are provided that include tracking acoustic and spatial cues to cluster simultaneous speakers, thereby facilitating the prediction of speaker RTF in reverberant environments.

[0065] In particular, in a multi-speaker reverberation environment, a speaker clustering algorithm is provided that assigns individual frequency components to their original speakers. This clustering algorithm provides the necessary conditions for an RTF estimator (RTF predictor) to operate properly in a multi-speaker reverberation environment. Next, using the prediction of the RTF matrix, the weight vector of a transfer function-based linear constrained minimum variance (TF-LCMV) beamformer is calculated (see equation (10) later), thus satisfying the conditions necessary for the TF-LCMV to operate. It is assumed that each individual human speaker is assigned a different pitch, and therefore pitch is a bijective indicator for the speaker. Multi-pitch detection is known to be a challenging task, especially in noisy, reverberant multi-speaker environments. To address this challenge, the W-Disjoint Orthogonality (W-DO) assumption is adopted, and a set of spatial cues, such as signal intensity, azimuth, and elevation, are used as additional features. To overcome temporarily inactive speakers and pitch changes, an extended Kalman filter (EKF) is used to constantly track acoustic cues—pitch values—and spatial cues are used to segment the last L frequency components, with each frequency component assigned to a different source. To facilitate the clustering of frequency components to specific speakers with specific pitches, the results of the EKF and segmentation are combined by cross-correlation.

[0066] Figure 1 illustrates the paths taken by the frequency components of an audio signal as they travel from the human speaker 11 to the microphone array 12 in an acoustic environment. The walls 13 and other elements in the environment 14 reflect incoming signals, and their attenuation and reflection angles are determined by the wall material and weave. Different frequency components of human speech will follow different paths. These paths may be direct paths 15 that lie on the shortest path between the human speaker 11 and the microphone array 12, or indirect paths 16, 17. Note that frequency components will travel along one or more paths.

[0067] Figure 2 illustrates the algorithm. The signal is acquired by a microphone array 201 containing M≧2 microphones, where M=7 microphones is one example. The microphones can be arranged in a series of equally spaced microphones, for example, on a straight line, circle, or sphere, or even non-uniformly spaced microphones forming any shape. The signals from each microphone are sampled, digitized, and stored in M ​​frames, each containing T consecutive samples 202. The size of the frame T can be chosen to be large enough so that the short-time Fourier transform (STFT) is accurate, but sufficiently short, and therefore the signal remains stationary along its equivalent time duration. A typical value for T is 4,096 samples for a sampling rate of 16 kHz, i.e., the frame is equivalent to 1 / 4 second. Consecutive frames are often superimposed on each other to improve the tracking of signal features over time. A typical superposition is 75%, meaning a new frame starts every 1,024 samples. T can be in the range of, for example, between 0.1 and 2 seconds, thereby providing 1,024 to 32,768 samples for a 16 kHz sampling rate. A sample can also be called an audio sample, representing the sound signal received by the array of microphones during a time period T.

[0068] Each frame is transformed in step 203 into the frequency domain by applying a Fourier transform, or a variation of the Fourier transform such as the Short-Time Fourier Transform (STFT), Constant-Q Transform (CQT), Logarithmic Fourier Transform (LFT), filter bank, etc. It is also possible to control the framing effect by applying several techniques such as windowing and zero-padding. Step 203 yields M complex-valued vectors of length K. For example, if the array contains 7 microphones, 7 vectors are prepared, which are registered by the frame time exponent l. K is the number of frequency bins and is determined by the frequency transform. For example, when using a normal STFT, K=T, which is the length of the buffer. The output of step 203 can also be called the frequency-transformed signal.

[0069] In 204, audio signals are clustered to different speakers. These clusters can be called speaker-associated clusters. Unlike conventional work that clusters speakers based solely on direction, 204 deals with multiple speakers in a reverberation chamber, and therefore signals from different directions can be assigned to the same speaker via direct and indirect paths. The proposed solution suggests the use of a set of acoustic cues, such as pitch frequency and intensity, as well as its harmonic frequencies and intensity, in addition to a set of spatial cues, such as the direction (azimuth and altitude) and intensity of the signal in one of the microphones. The pitch and one or more of the spatial cues act as state vectors for tracking algorithms such as Kalman filters and their variations, multiple hypothesis tracking (MHT), or particle filters, which are used to track this state vector and to assign individual tracks to different speakers.

[0070] All of these tracking algorithms use a model that describes the dynamics of the state vector in time, and therefore, if the measurements of the state vector are lost or contaminated by noise, the tracking algorithm compensates for this use of the dynamic model and, at the same time, updates the model parameters. The output of this stage is a vector that assigns the individual frequency components at a given time l to individual speakers. 204 is explained in more detail in Figure 3.

[0071] In step 205, an RTF estimator is applied to the data in the frequency domain. This stage yields a set of RTFs, each of which is registered with the associated speaker. The registration process is carried out using a clustered array from clustered speaker 204. This set of RTFs can also be called speaker-associated relative transfer functions.

[0072] The MIMO beamformer 206 reduces the energy of noise and interference signals by spatial filtering with respect to the energy of the required audio signal. The output of step 206 can also be called the beamformed signal. The beamformed signal is then sent to an inverse frequency converter 207 to produce a continuous audio signal in the form of a stream of samples, which is then transferred to other elements such as a speech recognition system, a communication system, and a recording device 208.

[0073] In a preferred embodiment of the present invention, keyword spotting 209 can be used to improve the performance of the clustering block 204. A predefined keyword (e.g., "Today is Alexa" or "OK ​​Google") is searched for within the frames from 202. Once the keyword is spotted within the stream of frames, the speaker's acoustic cues, such as pitch frequency and intensity, and their harmonic frequencies and intensity, are extracted. The characteristics of the path through which individual frequency components reached the microphone array 201 are also extracted. These characteristics are used by the clustering speaker 204 as a seed for a desired cluster of speakers. The seed is an initial guess regarding the cluster's initial parameters, such as the cluster's centroid, radius, and centroid-based clustering algorithm statistics such as K-mean, PSO, and 2KPM. Another example is the subspace basis for subspace-based clustering.

[0074] Figure 3 illustrates the speaker clustering algorithm. It is assumed that each speaker is assigned a different set of acoustic cues, such as pitch frequency and intensity, and its harmonic frequencies and intensity, and thus this set of acoustic cues is a bijective indicator for the speaker. Acoustic cue detection is known to be a challenging task, especially in noisy, reverberant, multi-speaker environments. To address this challenge, spatial cues, such as signal intensity, azimuth, and elevation, are used. To overcome temporarily inactive speakers and changes in acoustic cues, acoustic cues are constantly tracked using filters such as spatial filters and extended Kalman filters (EKF), and frequency components are segmented between different sources using spatial cues. To facilitate the clustering of frequency components to specific speakers with specific pitches, the EKF and segmentation results are combined by cross-correlation.

[0075] In step 31, potential acoustic cues in the form of pitch frequencies are detected, as in an example of a preferred embodiment. First, a time-frequency map is prepared using the frequency transformation of buffers from individual microphones, calculated in step 203. Next, the absolute values ​​of each of M complex-valued vectors of length K are weighted and averaged using some weighting coefficient which can be determined to minimize artifacts in some microphones. This yields a single real vector of length K, in which values ​​greater than a given threshold μ are extracted, while the remaining elements are discarded. The threshold μ is often three times the standard deviation of the noise and is adaptively chosen so as not to be less than a constant value determined by the electrical parameters of the system, in particular the number of effective bits of the sampled signal. Values ​​whose frequency exponents are within the range [k_minimum, k_maximum] are defined as candidates for pitch frequencies. The variables k_minimum and k_maximum are typically 85Hz and 2550Hz, respectively, since a typical adult male has a fundamental frequency between 85Hz and 1800Hz, and a typical adult female has a fundamental frequency between 165Hz and 2550Hz. Individual pitch candidates are then validated by exploring their higher harmonics. The presence of second and third harmonics may be a prerequisite for a candidate pitch that will be detected as a reasonable pitch with a reliability of R (e.g., R=10). If higher harmonics (e.g., fourth and fifth) are present, the reliability of the pitch increases, and can be doubled for each harmonic, for example. An example can be found in Figure 4. In a preferred embodiment of the present invention, the pitch 32 of a desired speaker is supplied by 210 using a keyword pronounced by the desired speaker. The supplied pitch 32 is added to a list having the highest possible reliability, e.g., R=1000.

[0076] At 33, an Extended Kalman Filter (EKF) is applied to the pitch from 31. As annotated by the Wikipedia entry for the Extended Kalman Filter (www.wikipedia.org / wiki / Extended_Kalman_filter), the Kalman filter has a state transition equation and an observation model. The state transition equation for discrete calculations is x k = f(x k-1 , u k ) + w k (1)

[0077] Also, the observation model for discrete calculations is z k = h(x k ) + v k (2) where x k is a state vector containing parameters that (partially) describe the state of the system, u k is a vector of external inputs that provide information about the state of the system, w k and v k are process and observation noise. The time update of the Extended Kalman Filter can predict the next state using the prediction equation, and the detected pitch can update the variables by comparing the actual measurement with the predicted measurement using an equation of the following type y k = z k - h(x k|k+1 ) (3) where z k is the detected pitch and y k is the error between the measurement and the predicted pitch.

[0078] At 33, individual trajectories can start from the detected pitch, followed by a model f(x k , u k ) that reflects the transient behavior of the pitch that may increase or decrease for emotional reasons. The input to the model is the past state vector x k(one or more state vectors), and any external inputs that affect pitch dynamics such as speech velocity, speech intensity and emotional utterance. k This may also be the case. The elements of the state vector x can quantitatively describe the pitch. For example, the pitch state vector can include, in particular, the pitch frequency, the intensity of the first harmonic, and the frequencies and intensity of higher harmonics. Vector function f(x k u k Using this, it is possible to predict the state vector x at some given time k+1 prior to the current time. An exemplary realization of the dynamic model in EKF may include the time update equation (aka prediction equation), as described in the book "Lessons in Digital Estimation Theory" by Jerry M. Mendel, which is incorporated herein by reference.

[0079] For example, a 3-term state vector

number

[0080] An exemplary state vector model for pitch is:

number

[0081] This describes a model that assumes a constant pitch at all times. In a preferred embodiment of the present invention, the velocity, intensity, and emotional utterance of speech are continuously monitored using speech recognition algorithms known in the art, and an external input u improves the time update stage of the EKF. kThis provides the following. Emotional speech methods are known in this field. See, for example, "New Features for Emotional Speech Recognition" by Palo et al.

[0082] Each track is assigned a reliability field that is inversely proportional to the time the track is deployed, using only time updates. When the reliability of a track falls below a certain reliability threshold, e.g., ρ representing 10 seconds of undetected pitch, the track is defined as dead, meaning that the respective speaker is not active. On the other hand, when a new measure (pitch detection) appears that cannot be assigned to any existing track, a new track is initiated.

[0083] In step 34, spatial cues are extracted from M frequency-transformed frames. As in step 31, the most recent L vectors are saved for analysis using correlation in time. This yields a time-frequency-cue (TFC) map for each of the M microphones, which is a three-dimensional array of size LxKxP (where P=M-1). The TFC is explained in Figure 5.

[0084] In step 35, the spatial cues of individual frequency components in TFC are segmented. The idea is that frequency components can originate from different speakers along L frames, which can be observed by comparing the spatial cues. However, for a single frame time l, it is assumed that the frequency components originate from one speaker due to the W-DO assumption. Segmentation can be performed using any known method in the literature used for clustering, such as K-nearest neighbors (KNN). Clustering is performed by assigning an index to each cell in A that indicates the cluster to which that cell (k, l) belongs.

number

[0085] In step 36, individual frequency components are assigned to specific pitches listed in the list of pitches tracked by the EKF, and the frequency components of the signal are classified such that each individual frequency component is active according to its reliability. This is done by assigning the k-th line of the time-frequency map (see Figure 4) to one of the pitches, and a specific cluster index c on the other lines in the time-frequency map. o This is done by calculating the sample cross-correlation between all values ​​having (j, l). This is done for all cluster indices. The sample cross-correlation is,

number

[0086] In the above equation, A is a time-frequency map, k is the index of the line to which one of the pitches belongs, j is any other line in A, and L is the number of columns in A. After calculating the sample cross-correlation between each individual pitch and each of the clusters on the other lines, cluster c1 on line j1 with the highest cross-correlation is classified together with its respective pitch, then cluster c2 on line j2 with the second highest cross-correlation is classified together with its respective pitch, and so on. This process is repeated until the sample cross-correlation falls below some threshold k which can be adaptively set, for example, 0.5x (average energy of the signal at a single frequency). 35 gives a set of groups of frequencies to which each pitch frequency is assigned.

[0087] Figure 4 illustrates an example of pitch detection on a time-frequency map. 41 is the time axis, represented by parameter l, and 42 is the frequency axis, represented by parameter k. Each column in this two-dimensional array is a real-valued vector of length K extracted in 31 after averaging the absolute values ​​of M frequency-transformed buffers over time l. For correlation analysis in time, L nearest vectors are stored in a two-dimensional array of size KxL. In 43, two pitches are represented by diagonals in different directions. Pitch k=2, with its harmonics at k=4, 6, and 8, has a reliability R=20 because a fourth harmonic is present, while pitch k=3, with its harmonics at k=6 and 9, has a reliability R=10. In 44, pitch k=3 is inactive, and only k=2 is active. However, the reliability of pitch k=2 drops to R=10 because the fourth harmonic is not detected (below the threshold μ). In 45, the k=3 pitch is active again, while the k=2 pitch is inactive. In 46, a new pitch candidate for k=4 appears, but only its second harmonic is detected. Therefore, this candidate is not detected as a pitch. In 47, the k=3 pitch is inactive, and no pitch is detected.

[0088] Figure 5 illustrates the TFC map, whose axes are the frame index (time) 51, frequency components 52, and spatial cues 53 which may be complex values ​​representing, for example, the direction (azimuth and altitude) to which each frequency component arrives and the intensity of the component. When a frame of index l is processed and moved to the frequency domain, the frequency elements

number

number

number

[0089] appendix The performance of an audio enhancement module is determined by its ability to filter out all interfering signals, leaving only the desired audio signal. Interfering signals can include, for example, noise from other speakers, air conditioning, music, motor noise (e.g., inside a car or airplane), and crowd noise, also known as "cocktail party noise." The performance of an audio enhancement module is usually judged by its ability to improve the speech-to-noise ratio (SNR) or speech-to-interference ratio (SIR), which typically reflects the ratio (often on a dB scale) of the power of the desired audio signal to the total power of the noise and other interfering signals, respectively.

[0090] When the acquisition module contains a single microphone, the method is called single-microphone speech enhancement and is often based on the statistical features of the signal itself in the time-frequency domain, such as single-channel spectral deduction, minimum variance distortionless response (MVDR), and spectral prediction using echo cancellation. When multiple microphones are used, the acquisition module is often called a microphone array, and the method is called multi-microphone speech enhancement. Many of these methods take advantage of the differences between signals captured simultaneously by the microphones. An established method is beamforming, which sums the signals from the microphones after multiplying each individual signal by a weighting coefficient. The purpose of the weighting coefficient is to average out interfering signals in order to condition the important signals.

[0091] Beamforming, in other words, is a method of creating a spatial filter that increases the power of a signal emitted from a given location in space (a desired signal from a desired speaker) and decreases the power of signals emitted from other locations in space (interfering signals from other sources), thereby increasing the SIR at the beamformer output.

[0092] A delay-and-sum beamformer (DSB), which requires the use of DSB weighting factors, consists of a counter-delay that is inevitably included by the different paths the desired signal takes to travel from its source to each of the microphones in the array. DSB is limited to signals coming from a single direction, such as in an anechoic environment. Therefore, in reverberant environments where signals from the same source travel to the microphones along different paths and reach the microphones from multiple directions, DSB performance is typically insufficient.

[0093] To mitigate the shortcomings of DSB in reverberant environments, beamformers can use a more complex acoustic transfer function (ATF) that represents the direction (azimuth and altitude) in which individual frequency components arrive from a given source to a specific microphone. The single direction of arrival (DOA) assumed by DSB and other DOA-based methods often does not hold true in reverberant environments where components of the same audio signal arrive from different directions. This is due to the different frequency responses of physical elements in the reverberant environment, such as walls, furniture, and people. The ATF in the frequency domain is a vector that assigns complex numbers to individual frequencies in the Nyquist bandwidth. The absolute value represents the gain of the path associated with this frequency, and the phase indicates the phase added to the frequency component along the path.

[0094] Predicting the ATF between a given point in space and a given microphone can be done by using a loudspeaker that emits a known signal and is placed at the given point. By simultaneously acquiring signals from the speaker input and the microphone output, the ATF can be easily predicted. The loudspeaker can be placed at one or more locations where a human speaker would be present during the system's operation. This method creates an ATF map for each point in space, or more practically, for each point on a grid. The ATF for points not included in the grid is approximated using interpolation. However, this method has significant drawbacks. Firstly, the system needs to be calibrated for each installation, which makes it impractical. Secondly, there is the acoustic difference between a human speaker and an electronic speaker, and this difference deviates the measured ATF from the actual ATF. Thirdly, there is the complexity of measuring a vast number of ATFs, especially when considering the direction of the speaker, and fourthly, there are possible errors due to changes in the environment.

[0095] A more practical alternative to ATF is the relative transfer function (RTF), which improves upon the shortcomings of ATF prediction methods in practical applications. The RTF is the difference between the ATFs of two given sources for two microphones in an array, and in the frequency domain, it takes the form of a ratio between the spectral representations of the two ATFs. Similar to ATF, the RTF in the frequency domain assigns a complex number to each frequency. Its absolute value is the gain difference between the two microphones, which is often close to one if the microphones are close together, and its phase reflects the incident angle of the source under certain conditions.

[0096] A linear-constrained minimum-variance (TF-LCMV) beamformer based on a transfer function can reduce noise while limiting speech distortion in multiplex microphone applications by minimizing output energy under the constraint that the speech component in the output signal is equal to the speech component in one of the microphone signals. N=N d +N i Given individual sources, N i N contaminated by individual interference sources and steady-state noise d Consider the problem of extracting n desirable sound sources. Each of the included signals propagates through an acoustic medium before being picked up by an arbitrary array of M microphones. The signal from each microphone is segmented into frames of length T, and an FFT is applied to each frame. In the frequency domain, each of the m-th microphone and the k-th frequency component of the l-th frame of the n-th source

number

number

number

[0097] In the above formula

number

number

number

number

number

number

number

number

number

[0098] In the above formula

number

[0099] Given array measures z(l, k), N d It is necessary to predict the mixing of individual desired sources. Extraction of the desired signal is done using a beamformer.

number

number

number

number

number

[0100] Possible solutions for (9) are

number

[0101] Based on (7) and (8) and the constraint set, the desired signal components in the beamformer output are:

number

[0102] From the l-th set of RTFs, and for each frequency component k, the intensity a obtained from, for example, one of the microphones defined as the reference microphone. p Using a phase difference-based algorithm in conjunction with (l, k), the angle of incidence is θ p A set of up to M-1 sources where (l, k), p=1, ..., P≦M-1, and elevation angle φ p (l, k) can be extracted. These three terms

number

[0103] TF-LCMV is an applicable method for extracting M-1 sound sources colliding from different locations in an echogenic environment into an array of M sensors. However, a prerequisite for TF-LCMV to operate is that the RTF matrix H(l, k), whose column is the RTF vector of all active sources in the environment, is known and available for use in TF-LCMV. To achieve this, it is necessary to associate each frequency component with its corresponding source speaker.

[0104] Several methods can be used to assign a source to a signal without requiring supplemental information. The main family of methods is called blind source separation (BSS), which recovers an unknown signal or source from its observed mixture. A key weakness of BSS in the frequency domain is that, at individual frequencies, the column vectors of the mixture matrix (predicted by BSS) are randomly rearranged, and without knowledge of this random rearrangement, it becomes difficult to combine and disclose the results across the entire frequency range.

[0105] Pitch information can be used to supplement BSS. However, the gender of the speaker must be a priori.

[0106] BSS can be used in the frequency domain while elucidating the ambiguity of the predicted mixing matrix using a maximum-magnitude method that assigns specific columns of the mixing matrix to sources corresponding to the largest elements in the vector. However, this method is highly dependent on the spectral distribution of the sources, as it is assumed that the strongest components at individual frequencies actually belong to the strongest sources. However, this condition is rarely encountered, as different speakers will introduce intensity peaks at different frequencies. Alternatively, source activity detection, also known as voice activity detection (VAD), can be used to elucidate the ambiguity in the mixing matrix using information about the source's activity state at a particular time. A drawback of VAD is that it cannot robustly detect voice pauses, particularly in multi-speaker environments. Also, this method is effective only when it requires a relatively long training period and when only one speaker at a time is participating in the conversation, which is sensitive to movement.

[0107] The TF-LCMV beamformer, in conjunction with a bi-auditory cue generator, can also be used as an extended version for bi-auditory speech enhancement systems. Acoustic cues are used to separate speech components from noise components in the input signal. This technique suggests the use of cues from entirely different perspectives to cluster signals from entirely different speech sources in a "cocktail party" environment, as suggested by auditory scene analysis theory. 1 This method is based on the following: Examples of primitive classification cues that can be used for speech separation include a common onset / offset across the entire frequency band, pitch (fundamental frequency), same location in space, transient spectral modulation, pitch and energy continuity, and smoothing. However, the assumption underlying this method is that all components of the desired speech signal have approximately the same direction; that is, a near-anechoic state that preserves the effect of the head shadow effect, which is suggested to be compensated for by using a head-related transfer function, which is unlikely to occur in a reverberant environment.

[0108] It should be noted that even when multiple speakers are active simultaneously, the spectral contents of the speakers do not superimpose at most time-frequency points. This is called W-separation orthogonality, or W-DO for short. This can be justified by the sparseness of the audio signal in the time-frequency domain. Due to this sparseness, the probability of simultaneous activity of two speakers at a particular time-frequency point is extremely low. In other words, in the case of multiple simultaneous speakers, each time-frequency point seems to correspond almost entirely to the spectral contents of one of the speakers.

[0109] Using W-DO, BSS can be facilitated by defining a specific class of signal that is W-DO to a certain extent. This is computationally economical as it only requires the use of the necessary first-order statistics. Furthermore, any number of signal sources can be demixed using only two microphones, provided that the sources are W-DO and do not occupy the same spatial location. However, this method assumes the exact same underlying mixing matrix across all frequencies. This assumption is essential for using histograms of predicted mixing coefficients across different frequencies. However, this assumption often does not hold true in reverberant environments, only in anechoic environments. An extension of this method to the case of multiple paths is limited to either negligible energy from the multiple paths or a sufficiently smooth convolutional mixing filter, thus the histogram becomes blurred, but still retains a single peak. This assumption also likewise does not hold true in reverberant environments, where the differences between different paths are often too large to produce a smooth histogram.

[0110] The proposed solution has been shown to perform well in reverberant environments and does not rely on unnecessary assumptions and constraints. This solution can operate without a priori information, without a large-scale training process, without constraining the prediction of the attenuation and delay of a given source at individual frequencies to a single point in the attenuation-delay space, without constraining the predicted attenuation-delay values ​​of a single source to the creation of a single cluster, and without limiting the number of mixed sounds to two.

[0111] Source separation for speech recognition engine A voice user interface (VUI) is an interface between a human speaker and a machine. A VUI uses one or more microphones to receive audio signals and often transcribes them into text, converting them into a digital signature, which is then used to infer the speaker's intent. The machine can then respond to the speaker's intent based on the application for which it is designed.

[0112] A key component of a VUI is the automatic speech recognition engine (ASR), which converts digitized speech signals into text. The performance of the ASR largely depends on how accurately the text describes the acoustic speech signal, and how well the input signal matches the ASR's requirements. Therefore, other components of the VUI are designed to enhance the acquired speech signal before feeding it to the ASR. Such components may include, to name a few, noise suppression, echo cancellation, and source isolation.

[0113] One of the most crucial components in voice enhancement is source separation (SS), which is intended to separate voice signals arriving from several sources. Assuming an array of two or more microphones, the signal acquired by each microphone is a mixture of all voice signals in the environment plus other interferences such as noise and music. The SS algorithm takes the mixed signal from all microphones and decomposes it into its components. That is, the output of source separation is a set of signals, each representing a signal from a specific source, whether it be a voice signal from a specific speaker, music, or even noise.

[0114] The need to improve source isolation is becoming increasingly important.

[0115] Figure 6 illustrates an example of a voice recognition chain in offline training. The chain often includes an array of microphones 511 that provides a set of digitized acoustic signals. The number of acoustic signals to be digitized is equal to the number of microphones that make up the array 511. Each digitized acoustic signal includes a mixture of all acoustic sources in the vicinity of the microphone array 511, whether human speakers or composite speakers such as TV, music, and noise. The digitized acoustic signals are passed to a pre-processing stage 512. The purpose of the pre-processing stage 512 is to improve the quality of the digitized acoustic signals by removing interferences such as echoes, reverberations, and noise. The pre-processing stage 512 is typically performed using a multi-channel algorithm that employs statistical associations between the digitized acoustic signals. The output of the pre-processing stage 512 is a set of processed signals, which typically have the same number of signals as the number of acoustic signals digitized at the input to this stage. This set of processed signals is sent to a source separation (SS) stage 513, which aims to extract acoustic signals from individual sources in the vicinity of the microphone array. In other words, the SS stage 513 takes a set of signals, each individual signal being a different mixture of acoustic signals received from different sources, and creates a set of signals such that each individual signal primarily contains a single acoustic signal from a single specific source. Source separation of an audio signal can be performed using geometric considerations of source deployment, such as beamforming, or by considering the characteristics of the audio signal, such as independent component analysis. The number of signals to be separated is usually equal to the number of active sources in the vicinity of the microphone array 511, but less than the number of microphones. The separated set of signals is sent to a source selector 514. The purpose of the source selector is to select the relevant source of the audio signal from which the audio signal should be recognized. The source selector 514 may use a trigger word detector to select a source that emits a predefined trigger word. Alternatively, the source selector 514 may also consider the location of sources near the microphone array 511, such as a predefined direction relative to the microphone array 511.The source selector 514 can also use a predefined acoustic signature of the audio signal to select a source that matches this signature. The output of the source selector 514 is a single audio signal sent to the speech recognition engine 515. The speech recognition engine 515 converts the digitized audio signal into text. Many methods for speech recognition exist that are known in the art, most of which are based on extracting features from the audio signal and comparing these features to a predefined vocabulary. The main output of the speech recognition engine 515 is a text string 516 associated with the input audio signal. The predefined text 518 is spoken into the microphone during offline training. The error 519 is calculated by comparing the output 516 of the ASR against this text. The comparison 517 can be performed using simple word counting, or using more complex comparison methods that take into account the meaning of words and appropriately weight the false positives of different words. The error 519 is then used by the SS 513 to modify a set of parameters to find the values ​​that minimize the error. This can be achieved by arbitrary monitored predictions, or by optimization methods such as least squares, stochastic gradients, neural networks (NN), and their variations.

[0116] Figure 7 illustrates an example of the voice recognition chain during real-time training, i.e., training while the system is in normal operation. While the VUI is running, the true text spoken by the human speaker is unknown, and the monitored error 519 is also unavailable. An alternative is the confidence score 521, developed for real-time applications when there is no reference to the spoken real text and when the application can benefit from knowing the confidence level of the ASR output. For example, a low confidence score allows the system to proceed to an appropriate branch where a more controlled dialogue takes place with the user. Many methods exist for predicting the confidence score, most of which aim for a high correlation with the error, which can be calculated once the spoken text is known. In real-time training, the confidence score 521 is converted to the monitored error 519 by an error estimator 522. If the confidence score is highly correlated with the theoretical monitored error, the error estimator may be a simple axis transformation. The confidence score 521 ranges from 0 to 100, and the objective is to increase the confidence score 521, while the monitored error ranges from 0 to 100, and the objective is to decrease the monitored error. A simple axis transformation of the form estimated_error=100-confidence_score can be used as the error estimator 522. The predicted error 519 can be used to train the SS parameters, as in the case of offline training.

[0117] Figure 8 illustrates a typical training mechanism of SS513. The source separator (SS) 513 receives a set of mixed signals from the pre-processing stage 512 and supplies the separated signals to the source selector 514. Typically, source separation of acoustic signals, and especially voice signals, is performed in the frequency domain. The mixed signals from the pre-processing stage 512 are first converted to the frequency domain 553. This is done by dividing the mixed signals into segments of exactly the same length and giving an overlap period between the resulting segments. For example, if the segment length is determined to be 1024 samples and the overlap period to be 25%, then each of the mixed signals is divided into segments of 1024 samples. A current set of segments from different mixed signals is called a batch. Each batch of segments starts with 768 samples after the preceding batch. Note that the segments across the entire set of mixed signals are synchronized, i.e., all segments belonging to the same batch have exactly the same starting point. The segment length and overlap period within a batch are obtained from model parameter 552.

[0118] The demixing algorithm 554 separates batches of segments that have arrived from the frequency conversion 553. Like many other algorithms, the source separation (SS) algorithm includes a set of mathematical models accompanied by a set of model parameters 552. The mathematical models establish how SS operates, such as how it handles physical phenomena, e.g., multiple paths. The set of model parameters 552 above coordinates the specific characteristics of the source signals, the architecture of the automatic speech recognition engine (ASR) that receives these signals, the geometry of the environment, and even the operation of SS on a human speaker.

[0119] The demixed batch of segments is sent to the inverse frequency transform 555, where the batch is transformed back into the time domain. The same set of model parameters 552 used in the frequency transform stage 553 are used in the inverse frequency transform stage 555. The time domain output signal from the resulting batch is reconstructed, for example, using superposition periods. This is done, for example, using a superposition-adding method, in which the resulting output signal is reconstructed by superimposing after the inverse frequency transform using an appropriate weighting function, which is probably in the range of 0 and 1 across the entire superposition domain, and by adding the superimposed time intervals, thus saving total energy. In other words, the superimposed segments from the previous batch fade out, while the superimposed segments from the later batch fade in. The output of the inverse frequency transform block is sent to the source selector 514.

[0120] Model parameters 552 are a set of parameters used by the frequency conversion block 553, the demixing block 554, and the inverse frequency conversion block 555. The division of the mixed signal into segments of exactly the same length, performed by the frequency conversion 553, is paced by a timekeeping mechanism such as a real-time clock. At each pace, each of the frequency conversion block 553, the demixing block 554, and the inverse frequency conversion block 555 extracts parameters from model parameters 552. These parameters are then substituted in mathematical models performed within the frequency conversion block 553, the demixing block 554, and the inverse frequency conversion block 555.

[0121] The collector 551 optimizes the above set of model parameters 552 with the aim of reducing the error 519 from the error estimator. The collector 551 receives the error 519 and the current set of model parameters 552 and outputs the modified set of model parameters 552. The modification of the above set of parameters can be performed a priori (offline) or while the VUI is running (real time). In offline training, the error 519 used to modify the above set of model parameters 552 is extracted by comparing the output of the ASR against a predefined text spoken to the microphone. In real-time training, the error 519 is extracted from the confidence score of the ASR.

[0122] Next, the set of parameters described above are modified using the error to find the values ​​that minimize the error. This can be done by any monitored prediction or optimization method, preferably a derivative-less method such as golden ratio search, lattice search, and Nelder-Mead.

[0123] The Nelder-Mead method (also known as the downhill simplex method, amoeba method, or polytope method) is a widely applied numerical method used to find the minimum or maximum of an objective function in multidimensional space. It is a direct search method (based on function comparison) and is often applied to nonlinear optimization problems where the derivative is unknown.

[0124] Nelder-Mead iteratively finds a local minimum of error 519 as a function of several parameters. The method begins with a set of values ​​that determine the simplex (a generalized triangle in N dimensions). It is assumed that the local minimum lies within the simplex. In each iteration, the error at a vertex of the simplex is calculated. The vertex with the largest error is replaced by a new vertex, thus reducing the volume of the simplex. This is iterated until the simplex volume is less than a predefined volume and the optimal value lies at one of the vertices. This process is carried out by collector 551.

[0125] The golden section search finds the minimum error 519 by continuously narrowing the range of values ​​within which the minimum exists. The golden section search requires a strictly unimodal error as a function of parameters. The range narrowing operation is performed by collector 551.

[0126] The golden ratio search is a technique for finding the extrema (minimum or maximum) of a unimodal function by continuously narrowing the range of values ​​within which an extremum is known to exist (www.wikipedia.org).

[0127] The grid search iterates through a set of values ​​associated with one or more of the parameters to be optimized. If multiple parameters are to be optimized, the individual values ​​in the set are vectors whose length is equal to the number of parameters. An error 519 is calculated for each value, and the value corresponding to the minimum error is selected. The iteration through the above set of values ​​is performed by the collector 551.

[0128] The traditional method for performing lattice search-hyperparameter optimization is lattice search, or parameter sweep, which is simply an exhaustive search through a manually defined subset of the hyperparameter space of a learning algorithm. Lattice search algorithms must be guided by some performance metric, typically determined by cross-validation against a training set or by evaluation against a held-out validation set. Since the parameter space of machine learning can include a real-valued space or an unbounded-valued space for specific parameters, manually defined boundaries and truncation may be necessary before applying lattice search (www.wikipedia.org).

[0129] All optimization methods require the continuous calculation of error 519 using the same set of separated acoustic signals. This is a time-consuming process and therefore cannot be performed continuously, but can only be performed if error 519 (which is calculated continuously) exceeds some predefined threshold, e.g., 10% error. When this occurs, two approaches can be taken.

[0130] One approach is to use parallel threads or multiple cores to perform the optimization in parallel with the normal operation of the system. That is, there are one or more parallel tasks that blocks 513, 514, 515, and 522 perform in parallel with the task of the normal operation of the system. In the parallel tasks, batches of mixed signals of length 1-2 seconds are obtained from preprocessing 512, iteratively separated 513, and interpreted using different sets of model parameters 552 514, 515. The error 519 is calculated for each such cycle. In each individual cycle, the collector 551 selects the above set of model parameters according to the optimization method.

[0131] The second method involves performing optimization when there is no sound in the room. Periods without human voice can be detected using a Voice Activity Detection (VAD) algorithm. Using these periods, the model parameters 552 are optimized in the same way as in the first method, saving the need for parallel threads or multiple cores.

[0132] An appropriate optimization method must be selected for each parameter in 552. Some methods apply to a single parameter, while others apply to a group of parameters. The following text suggests several parameters that affect speech recognition performance, and also suggests optimization methods based on the characteristics of these parameters.

[0133] Segment parameter length The segment parameter length is related to the FFT / IFFT. Typically, ASRs using separated phoneme features require short segments of around 20 milliseconds, while ASRs using a resulting sequence of phoneme features use segments of around 100-200 milliseconds. The segment length is also influenced by the scenario, such as room reverberation time. The segment length should be roughly equal to the reverberation time, which can be around 200-500 milliseconds. Since there is no sweet spot for segment length, this value must be optimized for the specific scenario and ASR. Typical values ​​are 100-500 milliseconds in terms of samples. For example, an 8kHz sampling rate implicitly means a segment length of 800-4000 samples. This is a continuous parameter.

[0134] The optimization of these parameters can be performed using various optimization methods, such as golden section search or Nelder-Mead in combination with the superposition period. When using golden section search, the input to the algorithm is the minimum and maximum possible lengths, e.g., 10 milliseconds to 500 msec, and the error function 519. The output is the segment length that minimizes the error function 519. When using Nelder-Mead with the superposition period, the input is the segment length and a set of three binary terms of the superposition period, e.g., (10 milliseconds, 0%), (500 milliseconds, 10%), and (500 milliseconds, 80%), and the error function 519, and the output is the optimal segment length and the optimal superposition period.

[0135] Overlapping period The superposition period parameter is associated with the FFT / IFFT. The superposition period is used to avoid phoneme loss due to segmentation; that is, phonemes are divided between the resulting segments. Due to the segment length, the superposition period is determined by the features adopted by the ASR. A typical range is 0-90% of the segment length. This is a continuous parameter.

[0136] The optimization of these parameters can be performed using various optimization methods, such as golden ratio search and Nelder-mead with segment length. When using golden ratio search, the inputs to the algorithm are the minimum and maximum possible superposition periods, e.g., 0% to 90%, and the error function 5¹⁹. The output is the superposition period that minimizes the error function 5¹⁹.

[0137] Window. Window parameters are associated with FFT / IFFT. Frequency transforms (553) often use windowing to mitigate the effects of segmentation. Several windows, such as Kaiser and Chebyshev, are parameterized. This means that the effect of the window can be controlled by changing the window parameters. A typical range is determined by the type of window. This is a continuous parameter. Optimization of this parameter can be performed using various optimization methods, such as golden ratio search. When using golden ratio search, the input to the algorithm is the minimum and maximum values ​​of the window parameters, determined by the window type, and the error function (519). For example, in the case of a Kaiser window, the minimum and maximum values ​​are (0, 30). The output is the optimal window parameter.

[0138] Sampling rate The sampling rate parameter is associated with the FFT / IFFT. The sampling rate is one of the critical parameters that affect the performance of speech recognition. For example, some ASRs have demonstrated poor results at sampling rates below 16kHz. Other ASRs can perform well even at 4kHz or 8kHz. Typically, this parameter is optimized when an ASR is selected. Typical ranges are 4kHz, 8kHz, 16kHz, 44.1kHz, and 48kHz. This parameter is discrete. Optimization of this parameter can be performed using various optimization methods, such as grid search. The input to the algorithm is the sampling rate (e.g., 4, 8, 16, 44.1, 48)kHz, which is the value at which the grid search is performed, and the error function 519. The output is the optimal sampling rate.

[0139] filtering The filtering parameter is related to demixing. Some ASRs use features that represent restricted frequencies. Therefore, filtering of the separated signal after source separation 513 can be coordinated with the specific features used by the ASR, thereby improving its performance. Furthermore, by filtering out spectral components not used by the ASR, the signal-to-noise ratio (SNR) of the separated signal can be improved, and consequently, the performance of the ASR can be improved. A typical range is 4-8 kHz. Optimization of this parameter can be carried out using various optimization methods, such as golden ratio search. This parameter is continuous. When applying golden ratio search, the input to the algorithm is the error function 519 and an initial estimate of the cutoff frequency division, e.g., 1000 Hz and 0.5X sampling rate. The output is the optimal filtering parameter.

[0140] Weighting coefficients for each microphone. The weighting coefficients for each microphone are associated with demixing. Theoretically, the sensitivities of different microphones on a given array should be similar up to a maximum of 3 dB. However, in practice, the sensitivities of different microphones can span a wider range. Furthermore, microphone sensitivity can change over time due to dust and moisture. A typical range is 0 to 10 dB. This is a continuous parameter. Optimization of this parameter can be performed using various optimization methods, such as Nelder-mead, which may or may not have weighting coefficients for each microphone. When applying the Nelder-mead method, the input to the algorithm is the error function 519 and the initial estimate of the simplex vertices. For example, the size of each n term is the number of microphones - N: (1, 0, ..., 0, 0), (0, 0, ..., 0, 1) and (1 / N, 1 / N, ..., 1 / N). The output is the optimal weight for each microphone.

[0141] Number of microphones The number of microphones is related to demixing. The number of microphones affects, on the one hand, the number of sources that can be separated, and on the other hand, the complexity and numerical precision. Furthermore, practical experiments have shown that too many microphones can result in a low output SNR. A typical range is 4 to 8. This is a discrete parameter. Optimization of this parameter can be performed using various optimization methods, such as grid search or Nelder-mead, which has weighting coefficients for each microphone. When applying grid search, the input to the algorithm is the error function 519 and the number of microphones to be searched, e.g., 4, 5, 6, 7, or 8 microphones. The output is the optimal number of microphones.

[0142] Figure 9 illustrates method 600.

[0143] Method 600 can begin with step 610, which involves receiving or calculating an error related to the speech recognition process applied to the preceding output of the source selection process.

[0144] Step 610 may be followed by step 620, which modifies at least one parameter of the source isolation process based on the error.

[0145] Step 620 may be followed by step 630, which receives a signal representing an audible signal transmitted from multiple sources and detected by an array of microphones.

[0146] Step 630 may be followed by step 640, which performs a source separation process to separate audible signals emitted from different sources of multiple sources in order to provide a source separation signal, and to transmit the source separation signal to the source selection process.

[0147] Step 640 can be followed by step 630.

[0148] Steps 630 and 640 can be followed by one or more iterations of step 610 (not shown) to supply the output of step 640 to the source selection process and the ASR in order to provide a preceding output of the ASR.

[0149] Note that the initial iterations of steps 630 and 640 can be performed without receiving any errors.

[0150] Step 640 may include steps of applying a frequency transform (but not limited to an FFT), demixing, and applying an inverse frequency transform (but not limited to an IFFT).

[0151] Step 620 may include at least one of the following steps: a. Step of revising at least one parameter of the frequency conversion. b. Step of revising at least one parameter of the inverse frequency transform. c. Step of revising at least one parameter of demixing. d. Step of revising the length of the signal segment representing the audible signal to which the frequency conversion is applied. e. A step of revising the superposition between consecutive segments of a signal representing an audible signal, wherein a frequency conversion is applied on a segment-by-segment basis. f. Steps to revise the sampling rate for frequency conversion. g. Step of revising the windowing parameters of the window applied by the frequency conversion. h. Step of revising the cutoff frequency of the filter applied during demixing. i. A step of revising the weights applied to individual microphones in the microphone array during demixing. j. Step to revise the number of microphones in the microphone array. k. A step to determine the revised value of at least one parameter using golden ratio search. l. A step of determining the revised value of at least one parameter using the Nedler-Mead algorithm. Step 1: Determine the revised value of at least one parameter using m. lattice search. n. A step of determining the revised value of at least one parameter based on a predefined mapping between the error and at least one parameter. o. A step to determine the mapping between the error and at least one parameter in real time.

[0152] In this specification, the present invention has been described with reference to specific examples of embodiments of the present invention. However, it will be apparent that various modifications and changes can be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims.

[0153] Furthermore, terms such as “front,” “rear,” “top,” “bottom,” “up,” “down,” etc., in the description and claims are used for descriptive purposes where applicable and not necessarily to describe permanent relative positions. It is understood that such terms are interchangeable under appropriate circumstances so that embodiments of the invention described herein may operate, for example, in the illustrated orientation or in orientations other than those described herein.

[0154] Any arrangement of components to achieve the same functionality is effectively "related" in such a way that the desired functionality is achieved. Therefore, any two components in this specification combined to achieve a particular functionality can be considered "related" to one another in such a way that the desired functionality is achieved independently of the architecture or intermediate components. Similarly, any two such related components can likewise be considered "operably connected" or "operably coupled" to one another in such a way that the desired functionality is achieved.

[0155] Furthermore, those skilled in the art will recognize that the boundaries between the operations described above are merely illustrative. Multiple operations can be combined into a single operation, a single operation can be distributed among additional operations, and operations can be performed with at least partial time overlap. Moreover, alternative embodiments may include multiple examples of a particular operation, and the order of operations can be modified in various other embodiments.

[0156] However, other forms of modification, variations, and substitutions are equally possible. Therefore, this specification and the drawings should be considered illustrative, not restrictive.

[0157] The phrase "X may be" indicates that condition X may be satisfied. It also suggests that condition X may not be satisfied. For example, every reference to a system containing a specific component includes a scenario in which the system does not contain that specific component. Similarly, every reference to a method containing a specific step includes a scenario in which the method does not contain that specific component. And in yet another example, every reference to a system configured to perform a specific operation includes a scenario in which the system is not configured to perform that specific operation.

[0158] The terms “include,” “equip,” “have,” “consist of,” and “essentially consist of” are used interchangeably. For example, all methods may include at least the steps included in the figures and / or herein, or may include only the steps included in the figures and / or herein. The same applies to systems.

[0159] The system may include an array of microphones, a memory device, and one or more hardware processors such as a digital signal processor, FPGA, ASIC, a general-purpose processor programmed to perform any of the methods mentioned above, etc. The system may not include an array of microphones, but may be supplied by sound signals generated by an array of microphones.

[0160] For the sake of simplicity and clarity, it should be noted that the elements shown in the diagrams are not necessarily drawn to scale. For example, the dimensions of some elements may be exaggerated relative to others for clarity. Furthermore, where deemed appropriate, reference numbers may be repeated between diagrams to indicate corresponding or similar elements.

[0161] In this specification, the present invention has been described with reference to specific examples of embodiments of the present invention. However, it will be apparent that various modifications and changes can be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims.

[0162] Furthermore, terms such as “front,” “rear,” “top,” “bottom,” “up,” “down,” etc., in the description and claims are used for descriptive purposes where applicable and not necessarily to describe permanent relative positions. It is understood that such terms are interchangeable under appropriate circumstances so that embodiments of the invention described herein may operate, for example, in the illustrated orientation or in orientations other than those described herein.

[0163] Those skilled in the art will recognize that the boundaries between logic blocks are merely illustrative, and that alternative embodiments may integrate logic blocks or circuit elements, or force alternative functional decompositions on various logic blocks or circuit elements. Therefore, it should be understood that the architectures described herein are merely illustrative, and that many other architectures can indeed achieve the same functionality.

[0164] Any arrangement of components to achieve the same functionality is effectively "related" in such a way that the desired functionality is achieved. Therefore, any two components in this specification combined to achieve a particular functionality can be considered "related" to one another in such a way that the desired functionality is achieved independently of the architecture or intermediate components. Similarly, any two such related components can likewise be considered "operably connected" or "operably coupled" to one another in such a way that the desired functionality is achieved.

[0165] Furthermore, those skilled in the art will recognize that the boundaries between the operations described above are merely illustrative. Multiple operations can be combined into a single operation, a single operation can be distributed among additional operations, and operations can be performed with at least partial time overlap. Moreover, alternative embodiments may include multiple examples of a particular operation, and the order of operations can be modified in various other embodiments.

[0166] Furthermore, in one embodiment, for example, the illustrated example can be implemented as a circuit mechanism located on a single integrated circuit or within the same device. Alternatively, the example can be implemented as any number of separate integrated circuits or separate devices interconnected in an appropriate manner.

[0167] Furthermore, for example, examples or parts thereof can be implemented as a software, i.e., code representation of a physical circuit mechanism, or as a logical representation that can be converted into a physical circuit mechanism, such as in any appropriate type of hardware description language.

[0168] Furthermore, the present invention is not limited to physical devices or units implemented in non-programmable hardware, but may also be applied to programmable devices or units, such as mainframes, minicomputers, servers, workstations, personal computers, notepads, personal digital assistants, electronic games, automobiles and other embedded systems, cell phones and various other wireless devices, which can perform desired device functions by being operated according to appropriate program code, and are generally referred to in this application as "computer systems."

[0169] However, other forms of modification, variations, and substitutions are equally possible. Therefore, this specification and the drawings should be considered illustrative, not restrictive.

[0170] In the claims, all reference numerals placed in parentheses should not be construed as limiting the claims. The word “equipped with” does not exclude the existence of other elements or steps other than those listed in the claims. Furthermore, the singular expressions of unspecified elements used herein are defined as one or plural. Also, the use of introductory clauses such as “at least one” and “one or more” in the claims should not be construed as implicitly meaning that the introduction of another claim element by a singular expression of an unspecified element limits any particular claim containing such introduced claim element to an invention containing only one such element, even if the same claim contains an introductory clause “one or more” or “at least one” and a singular expression of an unspecified element. The same applies to the use of expressions referring to specific elements. Unless otherwise specifically mentioned, terms such as “first” and “second” are used arbitrarily to distinguish between the elements they describe. Therefore, these terms are not necessarily intended to indicate any temporary or other prioritization of such elements, and the mere fact that certain means are described in different claims does not mean that combinations of these means cannot be used to one's advantage.

[0171] Furthermore, the present invention can also be implemented in a computer program for running on a computer system, which, when running on a programmable device such as a computer system, includes at least a code portion for performing steps of the method according to the present invention or enabling a programmable device to perform the functions of the device or system according to the present invention. The computer program can cause a storage system to allocate disk drives into disk drive groups.

[0172] A computer program is a list of instructions for a particular application program and / or operating system. A computer program may include, for example, one or more of the following: subroutines, functions, procedures, intended methods, intended embodiments, executable applications, applets, servlets, source code, intended code, shared libraries / dynamic load libraries, and / or other sequences of instructions designed to run on a computer system.

[0173] Computer programs can be stored internally on non-temporary computer-readable media. All or part of a computer program can be provided on computer-readable media permanently, removablely, or remotely linked to an information processing system. Computer-readable media can include, for example, any number of magnetic storage media, including disks and tapes; optical storage media, such as compact disk media (e.g., CD-ROMs, CD-Rs, etc.) and digital video disk media; non-volatile memory storage media, including semiconductor-based storage devices such as flash memory, EEPROMs, EPROMs, and ROMs; ferromagnetic digital memory; MRAM; and volatile storage media, including registers, buffers, or caches, main memory, RAM, etc. A computer process typically includes an executing program or part of a program, current program values ​​and state information, and resources used by the operating system to manage the execution of the process. An operating system (OS) is software that manages the sharing of computer resources and provides programmers with interfaces used to access these resources. An operating system processes system data and user input, and responds by allocating and managing tasks and internal system resources as services to the system's users and programs. When a computer system runs a computer program that includes, for example, at least one processing unit, associated memory, and many input / output (I / O) devices, the computer system processes information according to the computer program and generates the resulting output information via the I / O devices.

[0174] All systems related to this patent application include at least one hardware component.

[0175] While specific features of the present invention have been illustrated and described herein, many modifications, substitutions, alterations, and equivalents will come to mind for those skilled in the art. Therefore, it should be understood that the appended claims are intended to encompass all such modifications and alterations as within the true spirit of the invention.

Claims

1. A method for separating a source of light, performed by a source separation device, A step of determining a real-time error based on a real-time output related to a speech recognition process applied in real time to a preceding speech recognition input based on a preceding output of a source separation process, the real-time output including a confidence score representing the confidence level of the output text of the speech recognition process, A step of adjusting at least one parameter of the source isolation process in real time based on the real-time error, The steps include receiving a signal representing a real-time audio signal transmitted from multiple sources and detected in real time by multiple microphones, A method comprising the step of performing the source separation process based on the signals in order to provide a plurality of source separation signals corresponding to real-time audio signals transmitted from different sources of the plurality of sources.

2. The method according to claim 1, comprising the step of providing the plurality of source separation signals to a source selection process, wherein the real-time error is related to the speech recognition process applied to the preceding output of the source selection process.

3. The method according to claim 1, further comprising the step of updating at least one parameter of the source separation process in real time based on the real-time change of the real-time error.

4. The method according to claim 1, wherein the execution of the source separation process includes the steps of: applying a frequency transform to convert the signal into a plurality of segments in the frequency domain; separating the plurality of segments to provide a plurality of separated segments; and applying an inverse frequency transform to convert the plurality of separated segments into the time domain.

5. The method according to claim 4, wherein at least one parameter of the source separation process includes at least one parameter of the frequency conversion.

6. The method according to claim 5, wherein at least one parameter of the frequency conversion includes at least one of the following: the segment length of the frequency conversion, the segment overlap of the frequency conversion, the sampling rate of the frequency conversion, and the window parameter of the frequency conversion.

7. The method according to claim 4, wherein at least one parameter of the source separation process includes at least one parameter of the separation.

8. The method according to claim 7, wherein at least one parameter of the separation includes at least one of the following: the cutoff frequency of the filter, the weighting coefficient of the microphone, and the number of microphones in the plurality of microphones.

9. The method according to claim 1, wherein the step of adjusting at least one parameter of the source separation process includes the step of adjusting the at least one parameter using at least one of the golden ratio search, the Nedler Med algorithm, and the lattice search.

10. The method according to claim 1, wherein the step of adjusting the at least one parameter of the source isolation process includes the step of adjusting the at least one parameter based on a predetermined mapping between the real-time error and the at least one parameter.

11. The method according to claim 1, comprising converting the confidence score to the real-time error according to the axis transformation.

12. A source isolation device comprising processing means configured to perform the method described in any one of claims 1 to 11.

13. A non-temporary computer-readable medium for storing instructions that are executed by a computer system to perform the method described in any one of claims 1 to 11.

14. It is a speech recognition system, Multiple microphones, A speech recognition engine that performs the speech recognition process, A source isolation device configured to perform the method described in any one of claims 1 to 11, A system equipped with the following features.

Images