Distance-based single-channel purpose speech extraction method

Abstract

This paper aims to achieve single-channel target speech extraction (TSE) in a closed space using only distance information. [Solution] This is the first study to utilize distance cues alone for single-channel TSEs without using speaker physiological information. Inspired by recent single-channel distance-based separation and extraction methods, a novel model is introduced that efficiently fuses time-frequency (TF) bins with distance information for TSEs. Experimental results in both single-room and multi-room scenarios demonstrate the feasibility and effectiveness of this approach. This method can also be used to estimate the distance between different speakers in mixed speech.

Description

[Technical Field]

[0001] The present invention relates to a distance-based single-channel objective speech extraction method. [Background technology]

[0002] Target speech extraction (TSE) is a task that utilizes speaker-related cues to extract a target speech from a mixture of speeches consisting of multiple speakers. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] International Publication No. 2024 / 006514 [Patent Document 2] Japanese Patent Publication No. 2004-184856 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] This paper aims to achieve single-channel target speech extraction (TSE) in a closed space using only distance information. [Means for solving the problem]

[0005] This is the first study to utilize distance cues alone for single-channel TSEs without using speaker physiological information. Inspired by recent single-channel distance-based separation and extraction methods, it introduces a novel model that efficiently fuses time-frequency (TF) bins with distance information for TSEs. Experimental results in both single-room and multi-room scenarios demonstrate the feasibility and effectiveness of this approach. This method can also be used to estimate the distance between different speakers in mixed speech. [Brief explanation of the drawing]

[0006] [Figure 1]Query distance dq and speaker distance dk. [Figure 2] (a) Overall structure, (b) Structure of distance query blocks. [Figure 3] Inactive SDR of speech versus different query distances. The red dotted line represents the ground truth speaker distance, the blue dots represent the query distance, and a higher iSDR indicates the presence of a speaker. [Modes for carrying out the invention]

[0007] 1. Introduction Target speech extraction (TSE) is a task that utilizes speaker-related cues to extract a target speech from a speech mixture consisting of multiple speakers. Such speaker cues include registered speech, facial images, and lip movements [1], [2]. However, in real-world scenarios, ideal speaker-related information is not always readily available, and such sensitive data that can be used for biometrics can raise privacy concerns [3]. To date, several studies have reported successful results using spatial and distance cues for speech separation and TSE, which do not explicitly import speaker-related information, but instead utilize the transfer characteristics between the speaker and the microphone in a closed room. The main reason distance cues work is that the direct reverberation ratio (DRR) decreases as the distance from the target to the microphone increases in a closed space [4], [5]. The late reverberation component is less affected by distance, while the direct and early components are inversely proportional to distance.

[0008] The first study using distance cues for deep learning-based single-channel speech separation or extraction is distance-based speech separation (DSS)[5], which uses a basic recurrent neural network (RNN) model to separate single-channel mixed speech into two groups, a close group and a far group, according to a static distance threshold such as 1.5 m. This model implicitly learns the acoustic characteristics of mixed reverberant speech located at close and far positions, and separates them. However, due to the static threshold, users need to retrain the model for a new threshold.

[0009] To make such threshold-based methods more flexible, reference [6] proposes region-customizable speech extraction. It uses a dynamic distance threshold to directly control the boundary between the close group and the far group. The target distance threshold is converted into a learnable embedding, which is then fused with the multi-channel signal intermediate representation to obtain the target close-range sound group. However, when the expected sound source is located in a specific distance range, the model needs to perform inference multiple times for the target sound.

[0010] Following DSS, reference [7] also adopts the concept of static distance threshold, but implements the method in a monaural target speaker extraction (TSE) scheme where the model outputs only one target speech. Specifically, the target speaker embedding is learned by a speaker encoder of the network, and this encoder only outputs the embeddings of speakers within the distance threshold. This study only tests scenarios that include a single speaker within the threshold range, and this method, similar to reference [5], is also limited by static thresholds. Furthermore, during the training process, speaker information is required to classify the learned speaker embeddings, which reduces the emphasis on distance information. Distance is mainly used to train the model to generate desired nearby speaker embeddings, and the generation of target speech depends only on the obtained speaker embeddings instead of distance information. In summary, very few studies have considered distance cues [8], [9], and there is still much room for research in this field.

[0011] In this study, instead of using speaker embeddings to extract the target speech, we propose distance-based single-channel target speech extraction (TSE) that relies only on distance information as a cue for TSE without prior physiological information about the speaker. The proposed method leverages spatial reverberation cues inherent to enclosed rooms that vary between speakers at different distances from the microphone. This allows the use of raw distance information to extract speech at a specific distance. Furthermore, since the same distance can correspond to multiple room impulse responses (RIRs) from different directions, distance functions as an ambiguous cue. This ambiguity enables distance-based TSE to be implemented in a single-channel manner, which forms the basis of the proposed method. To the best of our knowledge, this study is the first to propose using only distance information for single-channel TSE in an enclosed space. For this task, we introduce a model that efficiently fuses distance information and time-frequency bins, and the results demonstrate the feasibility of our approach. The remainder of this document is organized as follows: Section 2 describes the proposed method. Section 3 presents the dataset and experimental results, and Section 4 concludes this document.

[0012] II. METHOD A. Problem Formulation Assume there are K speakers in an enclosed room. Let the anechoic speech, reverberant speech, and corresponding RIR of the k-th speaker be s k (t), x k (t), and h k (t), respectively. The reverberant speech x k (t) can be formulated as follows. [[Mathematical Expression]] where * denotes the convolution operator in the time domain. This is for a single-channel microphone in the room. The collected mixed signal can be expressed by the following formula. [[Mathematical Expression]]

[0013] Each target speech x k(t) is the impulse response h influenced by the physical characteristics of the room, as well as the positions of the microphone and the speaker k (t), and their positions are represented by their relative distance d k . Given a query distance d q , the TSE model outputs the target speech located in the range |d k -d q | ≤ r spk , where r spk represents the speaker distance range centered on the speaker distance, as shown in (a) of FIG. 1, and can be expressed as follows. [Mathematical formula]

[0014] Speakers often maintain similar distances, and when the query distance d q is within the range of one or more speakers, the model needs to output the sum of the speech signals, as shown in (b) of FIG. 1. When there is no speaker near the query distance, the model outputs zero, as shown in (c) of FIG. 1. [Mathematical formula]

[0015] B. Proposed Model The main task of target speaker extraction (TSE) is to identify the target speaker given a cue, so a general TSE framework always consists of a cue encoder and a speech extraction module. The cue encoder is designed to project the cue into a target speaker embedding, and the speech extraction module is mainly used to fuse the obtained embedding with the representation of the mixed speech [1]. This method follows this framework in the time-frequency (TF) domain. The overall configuration is shown in (a) of FIG. 2. First, by transforming y(t) into the TF domain via short-time Fourier transform (STFT), the input features of the model are the concatenated real and imaginary (RI) components [Mathematical formula] Next, it is processed by an encoder consisting of an initial 2D convolutional layer with a 3x3 kernel and a global layer normalization, followed by D-dimensional TF embedding.

number

[10] is obtained. Next, TF embedded H Y This feeds into the distance query (DQ) block and the time and spectral (TS) block stack. Finally, a decoder containing a 3x3 kernel and a 2D convolutional layer with a sigmoid function outputs the target speech mask. The DQ block contains a learnable distance embedding that fuses with the intermediate representation in both the time and spectral domains, while the TS block is designed to output the target speech. These two types of blocks share the same basic structure, which will be explained in this section.

[0016] 1) Distance Query Block: The main functions of the DQ module are target query distance and TF embedding H Y The objective is to receive the target speech and output an intermediate representation of it. The specific structure is shown in Figure 2(b) and includes two learnable distance embeddings, an intra-subband fusion module, and an intra-frame fusion module. The learnable distance embedding generator (DEG) includes three linear layers following a tanh function. This is the original query distance d q It takes as input and embeds distance

number

[10] -

[12] . For details, see the input TF embedding H Y First, the distance embedding D is replicated along the spectral dimension. s It is connected to H. s The data is then processed by layer normalization (LN) and BLSTM layers.

number

number

[0017] The in-frame fusion module uses distance embedding D f The coupling is aligned with the spectral dimension, and it shares the same structure as the subband fusion module, except that the BLSTM's scanning direction spans subbands within each frame.

[0018] 2) Time and spectral blocks: After fusing the target distance embedding and the speech representation, a TS block is used to improve the quality of the output speech. The TS block shares its basic structure with the DQ block, which does not have a fusing portion, and repeatedly uses a single RNN across all frames or subbands

[12] .

[0019] C. Loss function The model uses two types of loss functions and outputs both the target audio and a zero value, conditional on the target distance. Target audio x s , estimated voice x^ s Given the parameters, the signal-to-distortion ratio (SDR) is adopted as the loss function when the speaker is within the query distance.

number

[13] is not considered because we want the model to be able to handle inactive source conditions where the model can output zero. In this experiment, we found that SI-SDR usually causes the model to output a small scale under all conditions (presence of target speech and zero output), but it is important to construct the signal scale correctly in tasks where it is necessary to detect active or inactive sources

[14] . For the condition that there are no speakers near the query distance, the loss function adopted is inactive SDR (iSDR)

[14]

[15] .

number

[0020] III. Experiments A. Dataset generation In this study, the mixed speech input is a superposition of two reverberation speech signals, each convolved with a location-specific RIR, utilizing both simulated and realistic RIRs. For the simulated dataset, we use the randomized image method

[16] . For the realistic dataset, we adopt the BUT ReverbDB dataset

[17] . In total, we construct four datasets: three with simulated RIRs and one with a real RIR.

[0021] 1) D1 Fixed Room Size and Fixed Microphone Position: This simulated dataset includes a fixed room measuring 7 x 8 x 3 meters and a fixed microphone position measuring 3.5 x 4 x 1.1 meters. The reverberation time (RT60) is set to 0.2 seconds. The speaker's position is randomly initialized to be at least 0.5 meters from the wall and at a height of 1.2 to 2.0 meters. A total of 10,000 RIRs are generated. The dataset is split into training, validation, and test datasets in a ratio of 0.9:0.02:0.08.

[0022] 2) D2 Fixed room size and random microphone positions: This simulated dataset includes a fixed room measuring 7 x 8 x 3 meters. Each room randomly generates 100 microphone positions corresponding to 2,000 randomly generated speaker positions, resulting in 200,000 RIRs. The dataset is split into training, validation, and test datasets in a ratio of 0.9:0.01:0.09.

[0023] 3) D3 Random Room Size and Random Microphone Position: This simulated dataset has 50,000 rooms randomly initialized within a range of 4×5×2.5 meters to 8×10×3.0 meters. RT60 is randomly initialized from 0.2 seconds to 0.5 seconds. Each room contains one randomly placed microphone and 10 speakers, resulting in 489,813 RIRs, because smaller rooms cannot have samples from longer distances. The training, validation, and test datasets are split based on different rooms in a ratio of 0.9:0.02:0.08.

[0024] 4) D4 BUT Real RIR from ReverbDB dataset: We will adopt actual RIRs collected from three rooms in the dataset: Q301, L207, and L212. The distances range from 0.266 meters to 10.521 meters. A total of 624 RIRs will be used for fine-tuning, and 182 RIRs will be reserved for testing.

[0025] The speech was taken from the LibriLight

[18] dataset, with 128, 48, and 64 different speakers used for training, validation, and testing. Each utterance was convolved from a randomly selected speech and RIR, and the root mean square energy of the convolved signals was randomly adjusted from -25 to -20 dB.

[0026] B. Training Process and Configuration The model includes four DQ blocks and four TS blocks, with the dimensional size D set to 64 and the hidden state size H set to 64. The three linear layers in each distance embedding generator contain 32, 64, and 64 units, respectively. For efficient structure, the parameter quantity is 1.25M. STFT uses a frame length of 32ms with a 16ms frame shift. spk The gradient is set to 0.5 meters for D1 through D3 and 0.1 meters for D4. During the training phase, the Adam

[19] optimizer is used with gradient clipping set to a maximum norm of 5. The batch size is set to 14. The initial learning rate is 0.001, and if no lower loss is found for 14 consecutive epochs, the learning rate is reduced to 80%. Training is 500 epochs, with the target audio presence / absence ratio set to 0.9:0.1 for the first 250 epochs and 0.7:0.3 for the last 250 epochs. In scenario D4, due to insufficient data, the model is fine-tuned using the model pre-trained in D3.

[0027] C. Experimental Results 1) Results in different scenarios: The evaluation must consider both the presence and absence of the speaker at the query distance. For the presence condition, speech performance and scale accuracy are examined using SDR, SDR improvement (SDRi), and PESQ. For the absence condition, iSDR is used as the evaluation metric. The evaluation results for two mixed speakers are shown in Table I. The results are obtained from five repeated tests and are expressed as mean ± standard deviation.

[0028] In four test conditions, the proposed method achieved an SDR of 12.47 dB in a fixed room and microphone scenario. Performance degrades as the number of rooms and microphones increases. This method must handle a large number of combinations of room and microphone positions for a given query distance, so varying room sizes and random microphone placement pose significant challenges. Results with actual datasets are lower, mainly due to the multi-room conditions and limited data availability. A broader dataset is needed for further improvement.

[0029] [Table 1]

[0030] 2) Comparison results: The proposed method is compared with a basic LSTM model similar to DSS[5], which has 5.98M parameters and includes four Bi-LSTM layers and a final linear layer with a sigmoid function for mask prediction. DEG is preserved, and a mixed spectrogram of distance embeddings and TSE is concatenated. In addition, the number of DQ blocks and TS blocks is adjusted in the proposed method. The results in Table II demonstrate the performance of the proposed method.

[0031] [Table 2]

[0032] 3) Speaker Distance Estimation in Mixed Speech: One notable advantage of the proposed distance-based TSE model is its ability to output speech within a range centered on the query distance. This feature enables speech detection at various distances by iteratively processing all possible input distances. The sum of iSDRs at adjacent distances is used as the metric. For example, using the model trained in scenario D1, the query distance interval is set to 0.5m. The results are shown in Figure 3. The blue shaded area represents the sum of iSDRs within 1m, and the orange circle represents the detected peak value, which signifies the detected speaker distance. Table III shows the mean absolute error (MAE), quantifying the difference between the highest detected peak and the nearest actual speaker distance.

[0033] [Table 3]

[0034] IV. Conclusion We propose an innovative task: distance-based single-channel target speech extraction and the design of a parameter-efficient model to achieve it. The results demonstrate the feasibility and future potential of the task, including the ability to estimate speaker distance from mixed speech. Based on these findings, future research will focus on improving performance in real-world scenarios and multi-speaker situations.

[0035] References [1] K. Zmolikova, M. Delcroix, T. Ochiai, K. Kinoshita, J. C ernocky', and D. Yu, “Neural Target Speech Extraction: An overview,” IEEE Signal Processing Magazine, vol. 40, no. 3, pp. 8-29, May 2023. [2] Z. Mu and X. Yang, “Separate in the speech chain: Cross-modal conditional audio-visual target speech extraction,” in Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24, K. Larson, Ed. International Joint Conferences on Artificial Intelligence Organization, 8 2024, pp. 6415-6423, main Track. [3] P.-G. No’e, M. Mohammadamini, D. Matrouf, T. Parcollet, A. Nautsch, and J.-F. Bonastre, “Adversarial disentanglement of speaker representation for attribute-driven privacy preservation,” arXiv preprint arXiv:2012.04454, 2020. [4] S. S. Kushwaha, I. R. Roman, M. Fuentes, and J. P. Bello, “Sound Source Distance Estimation in Diverse and Dynamic Acoustic Conditions,” in 2023 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), Oct. 2023, pp. 1-5. [5] K. Patterson, K. Wilson, S. Wisdom, and J. R. Hershey, “Distance-Based Sound Separation,” in Interspeech 2022. ISCA, Sep. 2022, pp. 901-905. [6] R. Gu and Y. Luo, “Rezero: Region-customizable sound extraction,” IEEE / ACM Transactions on Audio, Speech, and Language Processing, 2024. [7] J. Lin, P. Wang, H. Dinkel, J. Chen, Z. Wu, Z. Yan, Y. Wang, J. Zhang, and Y. Wang, “Focus on the sound around you: Monaural target speaker extraction via distance and speaker information,” arXiv preprint arXiv:2306.16241, 2023. [8] H. Taherian, K. Tan, and D. Wang, “Multi-channel talker-independent speaker separation through location-based training,” IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 2791-2800, 2022. [9] D. Petermann and M. Kim, “Hyperbolic distance-based speech separation,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 1191-1195.

[10] Z.-Q. Wang, S. Cornell, S. Choi, Y. Lee, B.-Y. Kim, and S. Watanabe, “TF-GridNet: Integrating Full- and Sub-Band Modeling for Speech Separation,” IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 3221-3236, 2023.

[11] S. Wang, X. Kong, X. Peng, H. Movassagh, V. Prakash, and Y. Lu, “Dasformer: Deep Alternating Spectrogram Transformer For Multi / Single-Channel Speech Separation,” in ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Jun. 2023, pp. 1-5.

[12] Y. Luo, Z. Chen, and T. Yoshioka, “Dual-path rnn: efficient long sequence modeling for time-domain single-channel speech separation,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 46-50.

[13] J. L. Roux, S. Wisdom, H. Erdogan, and J. R. Hershey, “SDR - Halfbaked or Well Done?” in ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), May 2019, pp. 626-630.

[14] M. Delcroix, J. B. V’azquez, T. Ochiai, K. Kinoshita, Y. Ohishi, and S. Araki, “Soundbeam: Target sound extraction conditioned on soundclass labels and enrollment clues for increased performance and continuous learning,” IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 121-136, 2022.

[15] S. Wisdom, H. Erdogan, D. P. Ellis, R. Serizel, N. Turpault, E. Fonseca, J. Salamon, P. Seetharaman, and J. R. Hershey, “What’s all the fuss about free universal sound separation data?” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 186-190.

[16] R. Scheibler, E. Bezzam, and I. Dokmani’c, “Pyroomacoustics: A Python package for audio room simulations and array processing algorithms,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Apr. 2018, pp. 351-355.

[17] I. Szo¨ke, M. Ska’cel, L. Mos ner, J. Paliesek, and J. C ernocky’, “Building and evaluation of a real room impulse response dataset,” IEEE Journal of Selected Topics in Signal Processing, vol. 13, no. 4, pp. 863-876, Aug. 2019.

[18] J. Kahn, M. Riviere, W. Zheng, E. Kharitonov, Q. Xu, P.-E. Mazar’e, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen et al., “Libri-light: A benchmark for asr with limited or no supervision,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 7669-7673.

[19] D. P. Kingma, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.

Claims

[Claim 1] Distance-based single-channel purpose speech extraction method.

Images