End-to-end speech separation algorithm based on speech language model

Abstract

The invention discloses an end-to-end speech separation algorithm based on a speech language model, and the algorithm comprises the steps: discretizing a continuous audio into a 32-order discrete codebook sequence through a residual vector quantization coder-decoder, and introducing a transcription start symbol lt through an SOT strategy; sOSgt, SOSgt; a special separator is lt; sCgt; and a termination symbol lt; eOSgt, EOSgt; splicing a multi-person voice sequence; extracting audio depth features by using a pre-trained WavLM model, and guiding an autoregression decoder to output a separated zero-order codebook sequence in combination with a cross attention mechanism; predicting a high-order codebook sequence step by step through a non-autoregression model, configuring an independent embedding layer to fuse low-order information, and introducing a task embedding mechanism to optimize modeling; based on a special separator lt; sCgt; and slicing the multi-order discrete codebook sequence, and outputting an independent voice source through an Encodec decoder. According to the method, the intelligibility of voice separation and the audio restoration quality can be effectively improved, the decoding speed is high, the subjective hearing experiment result and the downstream task performance are excellent, the scene that the number of speakers is unknown is supported, and the industrialization application prospect is wide.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech signal processing, and particularly relates to an end-to-end speech separation algorithm based on a speech language model. BACKGROUND

[0002] Speech separation is a key task in speech processing, aiming to separate independent speech sources from mixed and overlapped signals, and its application fields cover automatic speech recognition, speaker recognition and hearing aids, etc., and it is an important basic support for realizing high-performance landing of the above downstream tasks.

[0003] In the development process of speech separation technology, discriminative methods are traditionally used to process speech separation problems. Such discriminative methods usually take scale-invariant signal distortion ratio as the training target, and optimize model parameters to reduce the difference between the separated signals and the real speech at the signal level. Although this method has certain advantages at the signal level, it can achieve preliminary separation of mixed signals, but it is often difficult to ensure speech intelligibility, and it is easy to cause distortion and artifacts in the separated speech, which further negatively affects the performance of downstream tasks such as automatic speech recognition and speaker recognition, and cannot meet the demand for speech quality in actual application.

[0004] In contrast, generative methods provide another technical solution for speech separation. Such methods model the underlying data distribution of speech and construct a separation model from the data generation perspective, which is theoretically expected to generate more coherent and clear separated signals and has potential in improving speech intelligibility. However, generative methods also have obvious limitations: on the one hand, since the separated signals are generated step by step through iterative sampling, the decoding speed is slow, which makes it difficult to adapt to real-time speech separation scenarios; on the other hand, such methods are prone to "hallucination", that is, generating speech content that does not exist in the real mixed speech. Such false information will interfere with the judgment of downstream tasks, further limiting their range of practical application.

[0005] Meanwhile, the latest progress of large language models brings new breakthroughs to the performance improvement of speech processing tasks. With powerful processing capabilities, large language models achieve deep integration of acoustic information and language information in speech by using techniques such as quantization or discretization of speech tokens, providing a new direction for solving technical problems in the field of speech processing. Specifically, in the text-to-speech task, methods such as VALL-E and Seed-TTS use neural encoder-decoder-based and discrete token-based language models to process, improving the naturalness and fidelity of speech synthesis; in the automatic speech recognition task, methods such as Seed-ASR and FireRedASR integrate large language model technology to achieve top recognition performance; in the speech enhancement task, the SELM method uses discrete tokens extracted from the WavLM model to model through a speech language model, effectively improving the speech enhancement effect. These technological advances show that introducing speech language model-related technologies into speech separation tasks has the potential to significantly improve speech intelligibility, which in turn is expected to enhance the separation performance of downstream tasks and provide the possibility of breaking through the limitations of existing speech separation methods.

[0006] Therefore, the skilled person in the art is committed to developing an end-to-end speech separation algorithm based on a speech language model. SUMMARY

[0007] In view of the above-mentioned defects of the prior art, the present application at least solves the following technical problems: 1. The speech intelligibility of traditional discriminative speech separation models is poor, which seriously affects the performance of downstream tasks; 2. Continuous audio signals cannot be directly modeled by speech language models, and efficient discretization schemes are needed; 3. The self-recurrent decoder is slow in inference, and the single-order codebook restoration capability is limited, making it difficult to balance separation efficiency and audio restoration quality; 4. It is difficult to deeply integrate with downstream speech processing tasks, cannot output multiple results end-to-end, and cannot solve the separation problem in the scenario where the number of speakers is unknown.

[0008] The present application discloses an end-to-end speech separation algorithm based on a speech language model, which comprises the following steps: S1: Discretize the continuous audio signal into a multi-order discrete codebook sequence using a residual vector quantization encoder-decoder; S2: Extract the deep features of the continuous audio signal through a pre-trained speech composite feature encoder, guide the self-recurrent decoder with a cross-attention mechanism, and output the zero-order codebook sequence after separation according to the deep features and historical token sequence; S3: Use a non-self-recurrent model to predict high-order codebook sequences step by step, and fuse the embedding information of all low-order codebook sequences through an independent embedding layer to obtain a total embedding vector; S4: By special separator <sc>The multi-order discrete codebook sequence is sliced to extract a single-person speech sequence, and the single-person speech sequence is input into an Encodec decoder for decoding to obtain an independent speech source after separation; Further, the residual vector quantization encoder-decoder is an Encodec encoder-decoder, the Encodec encoder clusters audio features step by step through a residual connection to generate the multi-order discrete codebook sequence, and the Encodec decoder restores the multi-order discrete codebook sequence to a continuous audio signal; Further, the multi-order discrete codebook sequence obtained through discretization in step S1 is spliced into a multi-person speech sequence by using an SOT strategy, the multi-order discrete codebook sequence is 32 orders, and a transcription start symbol is introduced in the splicing process <sos>, the special delimiter <sc>and a terminal symbol <eos>, concatenates in the order of the first-in first-out of the speech, the transcription start symbol <sos>, special separator <sc>and a terminal symbol <eos>Having the same meaning in each codebook, the single-order codebook sequence format of the multi-person speech sequence satisfies:

[0009]

[0010] in, Indicates the first A sequence of sequential codebooks. Indicates the first In the nth sequential codebook sequence, the nth The speaker's first 1 tag and =1024, Indicates the first The sequence length corresponding to each speaker Indicates the number of codebook entries; Furthermore, the pre-trained speech composite feature encoder is a WavLM model based on the Transformer architecture; Furthermore, the process of extracting the depth features in step S2 includes: fusing the features of each hidden layer of the WavLM model through a linear layer, and performing layer normalization on the fused features to obtain the depth features. ; Furthermore, the autoregressive decoder is built based on the Whisper decoder architecture and trained from scratch. The process of outputting the zero-order codebook sequence in step S2 satisfies:

[0011] in, Indicates the first The probability distribution of each word element Represents a historical zero-order lexical sequence. Represents a vocabulary and satisfies ; Furthermore, the non-autoregressive model adopts the same architecture as the autoregressive decoder, and the one-way mask is removed; Furthermore, the non-autoregressive model is configured with eight independent word embedding layers that share the same positional embeddings; Furthermore, the non-autoregressive model introduces a task embedding mechanism and combines it with position embedding to construct a total embedding vector. The task embedding mechanism is used to enable the model to identify the current codebook prediction task type; the process of obtaining the total embedding vector in step S3 satisfies:

[0012] in, Indicates the first Word embeddings of a codebook sequence, denote a position embedding, denote a task embedding of a th task; Further, the process of step S3 of predicting the high-order codebook sequence stage by stage further comprises: inputting the total embedding vector into a Transformer layer for deep modeling, and then projecting to a word embedding space of a th codebook through a Softmax function to obtain a probability distribution of all word units in the th codebook sequence, satisfying:

[0013]

[0014] wherein, denote an output feature of the Transformer layer, denote a word unit probability distribution of a th codebook sequence, denote a projection matrix of a th codebook.

[0015] Compared with the traditional discriminative model, the performance index of the downstream task and the subjective listening experiment result of the human are greatly ahead; With the help of the residual vector quantization encoder-decoder, it is possible to use the speech language model for speech separation, and the decoder has low information loss of the audio, and the information required by the downstream task is retained to the maximum extent; Through the design of combining the autoregressive model with the non-autoregressive model, the decoding speed is greatly accelerated compared with the pure autoregressive scheme, and the introduction of the high-order codebook greatly improves the audio restoration quality; At the same time, the algorithm has natural compatibility because of using the speech language model architecture, can be combined with automatic speech recognition and other downstream tasks for joint training, can output the transcribed text and single clear voice in parallel end to end; and because the speaker conversion is directly represented by a special symbol, it can directly process the situation where the number of speakers is unknown, and the performance is also better than other discriminative models in the subjective listening experiment of the human. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 It is a step flowchart of the end-to-end speech separation algorithm based on the speech language model of the application; Figure 2 It is an algorithm component architecture and data flow direction schematic diagram of the end-to-end speech separation algorithm based on the speech language model of the application. DETAILED DESCRIPTION

[0017] The technical content of the present application will be more clearly understood and facilitated to be understood by referring to the following description of preferred embodiments of the present application with reference to the accompanying drawings. The present application can be embodied in many different forms and the scope of the present application is not limited to the embodiments set forth herein.

[0018] In the drawings, components of the same structure are denoted by the same reference numerals, and components similar in structure or function are denoted by similar reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present application is not limited to the size and thickness of each component. In order to make the drawing clearer, the thickness of the components is appropriately exaggerated in some places in the drawing.

[0019] The present application discloses an end-to-end speech separation algorithm based on a speech language model, as shown in Figure 1 The algorithm comprises the following steps: S1: Discretize the continuous audio signal into a multi-order discrete codebook sequence using a residual vector quantization encoder-decoder; S2: Extract the deep features of the continuous audio signal through a pre-trained speech composite feature encoder, combine a cross-attention mechanism to guide a self-recursive decoder, and output a zero-order codebook sequence after separation according to the deep features and historical word sequence; S3: Use a non-self-recursive model to predict high-order codebook sequences step by step, and fuse the embedding information of all low-order codebook sequences through an independent embedding layer to obtain a total embedding vector; S4: Through a special separator <sc>The multi-order discrete codebook sequence is sliced to extract single speaker sequences, which are then input into an Encodec decoder for decoding to obtain separated independent speech sources.

[0020] The complete processing flow of the above-mentioned end-to-end speech separation algorithm is shown in Figure 2 The flow clearly presents the whole-link logic of multi-speaker audio from encoding, codebook sequence processing to the final output of single independent speech.

[0021] Single speaker speech is first encoded into multi-codebook sequences by Encodec, and then spliced into multi-speaker sequences using the SOT (Sequence of Talkers, SOT) strategy. The zero-order codebook is modeled using an Auto-Encoder Decoder (AED) framework, in which the decoder performs autoregressive inference through cross-attention mechanisms across multi-speaker features. Then, a non-autoregressive (NAR) model with the same architecture is used to predict high-order codebooks: an independent word embedding layer is configured for each low-order input, and they are combined to generate the final embedding vector. Finally, the codebook sequence is segmented by a special symbol, and the separated single speaker signal is output by the Encodec decoder.

[0022] This embodiment uses the Encodec algorithm to map continuous audio into a compact discrete representation. The encoding process generates a 32-order codebook with a size of = 1024. Within the encoder-decoder framework, this embodiment extracts and encodes single speaker segments from multi-speaker segments provided by the dataset. Then, the SOT strategy is used to splice the multi-codebook sequence: a transcription start symbol <sos>, concatenates the sequences in the order of the first-in first-out of the outgoing calls, special separator <sc>marker speaker switch, end symbol <eos>The end of the label, the specific format as follows:

[0023]

[0024] Wherein, represents the first order codebook sequence, represents the first order codebook sequence in the first speaker, the first label, represents the first speaker corresponding to the sequence length, represents the number of codebooks. It should be noted that although all codebooks have the same vocabulary and identification space, the same value may represent completely different features in different codebooks, while special symbols have the same meaning in each codebook.

[0025] After obtaining the complete m order SOT sequence through the SLM-SS method, the special separator <sc>The multi-order discrete codebook sequences are sliced to extract single speaker speech sequences. These sequences are then input into the Encodec decoder to recover clean single speaker speech data. Since speaker transition is directly represented by special symbols, it is possible to directly handle scenarios where the number of speakers is unknown.

[0026] For the encoder, the pre-trained WavLM model, a powerful feature encoder based on Transformer, is selected, and fine-tuning is performed on the basis of the WavLM pre-training weights. For the decoder, the model is constructed in accordance with the design of the Whisper decoder, and is trained from scratch. The vocabulary V contains the codebook size of the encoder model and three special symbols for the SOT task:

[0027] When obtaining the zero-order codebook sequence, the encoder first encodes the multi-speaker audio into deep features. To integrate the capabilities of the hidden layers in WavLM, a linear layer is designed to fuse the features of all layers. After layer normalization, the deep features are input into the decoder. Then, the autoregressive decoder predicts the th word based on the encoded audio features , the cross-attention mechanism, and the historical word sequence :

[0028] where represents the probability distribution of the th word, represents the historical zero-order word sequence.

[0029] The same architecture as the AED model is adopted in this embodiment, but the one-way mask is removed to implement the NAR model. To realize the cooperative training of multi-order codebooks, eight independent word embedding layers that share the same position embedding are designed. In addition, the task embedding mechanism is introduced to enable the model to identify the current codebook prediction task type.

[0030] When predicting the th-order codebook sequence, information from all lower-order codebook sequences must be considered. This requires embedding and summing all lower-order sequences, combining the position embedding and task embedding to obtain the total embedding vector , which is specifically as follows:

[0031] where represents the word embedding of the th codebook sequence, denotes a position embedding, denotes a task embedding for the th task. Subsequently, is input to a series of Transformer layers for deep modeling. Finally, the vector is projected to the th vocabulary of the codebook, thereby generating a probability distribution over all words in the th codebook sequence:

[0032]

[0033] where, denotes the output feature of the Transformer layer, denotes the word token probability distribution of the th codebook sequence, denotes the projection matrix of the th codebook.

[0034] The experimental results of the embodiment are shown in Table 1: Table 1: Speech separation performance comparison experimental data table

[0035] Even if the complete 32-order codebook of Encodec is recovered, the generated audio still has obvious information loss, and this loss is more significant when only the first 8 orders are used. This irreversible distortion introduced by feature discretization leads to a slight degradation in objective indicators compared to the original audio, but it is still not easy to detect in subjective listening experiments, which further confirms the hypothesis of the present application. Similar phenomena are observed in the TER index: the audio reconstructed based on the 32-order codebook still has label errors after re-encoding, confirming the persistence of distortion. When only the predicted 8-order codebook is decoded for re-encoding, SLM-SS shows more serious mismatch phenomena; but since it uses the framework centered on Encodec, its degradation is still lower than that of Sepformer and BSRNN.

[0036] In contrast, the distortion introduced by SLM-SS in speaker identification and speech consistency is relatively mild. The algorithm shows strong reconstruction capability, with a word error rate (WER) close to the true value. In contrast, BSRNN and Sepformer have a larger performance gap in WER, which can be attributed to the mismatch between the pre-trained ASR model and the separation output, which is usually imperceptible to human listeners. LPS and SBS models show similar trends.

[0037] Overall, both SLM-SS and baseline model have mismatch and distortion, leading to inconsistent results of different evaluation indicators. To more accurately evaluate its real impact, more attention should be paid to subjective listening experiments. In this regard, the score of the algorithm is always higher than BSRNN and Sepformer, while the Encodec reconstructed audio is still comparable to the original recording. These findings confirm that, from a perceptual point of view, the present invention can achieve better speech and feature reconstruction, where minor and imperceptible distortions can be safely ignored.

[0038] The preferred embodiments of the present application have been described in detail above. It should be understood that modifications and variations can be made by those of ordinary skill in the art without departing from the spirit and scope of the application. Therefore, it is intended that the application be given a broad interpretation in accordance with the principles thereof as defined by the following claims.< / sc> < / eos> < / sc> < / sos> < / sc> < / eos> < / sc> < / sos> < / eos> < / sc> < / sos> < / sc>

Claims

1. An end-to-end speech separation algorithm based on a phonetic language model, characterized in that, The method comprises the following steps: S1: discretize continuous audio signals into multi-order discrete codebook sequences using a residual vector quantization coding decoder; S2: extract deep features of the continuous audio signals through a pre-trained speech composite feature encoder, combine a cross-attention mechanism to guide a self-recurrent decoder, and output separated zero-order codebook sequences according to the deep features and historical token sequences; S3: predict high-order codebook sequences step by step using a non-self-recurrent model, and fuse embedding information of all low-order codebook sequences through an independent embedding layer to obtain a total embedding vector; S4: By special separator <sc>slice the multi-order discrete codebook sequences, extract single-person speech sequences, and then input the single-person speech sequences into an Encodec decoder for decoding to obtain separated independent speech sources.< / sc> 2. The end-to-end speech separation algorithm based on a phonetic language model of claim 1, wherein, The residual vector quantization coding decoder is an Encodec encoder-decoder, the Encodec encoder clusters audio features step by step through a residual connection to generate the multi-order discrete codebook sequences, and the Encodec decoder restores the multi-order discrete codebook sequences into continuous audio signals.

3. The end-to-end speech separation algorithm based on a phonetic language model of claim 1, wherein, The multi-order discrete codebook sequence obtained by discretization in step S1 is spliced into a multi-person speech sequence using an SOT strategy, the multi-order discrete codebook sequence is 32 orders, and a transcription start symbol is introduced in the splicing process <sos>, the special delimiter <sc>and a terminal symbol <eos>, concatenates in the order of the first-in first-out of the speech, the transcription start symbol <sos>, special separator <sc>and a terminal symbol <eos>In each codebook, the single-order codebook sequence format of the multi-person speech sequence has the same meaning and satisfies:< / eos> < / sc> < / sos> < / eos> < / sc> < / sos> wherein, denotes the th sequential codebook sequence, denotes the th sequential codebook sequence, th speaker in the th sequential codebook sequence and = 1024, denotes the th speaker's corresponding sequence length, denotes the number of codebooks.

4. The end-to-end speech separation algorithm based on a speech language model as described in claim 1, characterized in that, The pre-trained speech composite feature encoder is a WavLM model based on a Transformer architecture.

5. The end-to-end speech separation algorithm based on a phonetic language model as defined in claim 4, characterized in that, The process of extracting the deep feature in step S2 includes: fusing features of hidden layers of the WavLM model through a linear layer, performing layer normalization processing on the fused features, and obtaining the deep feature .

6. The end-to-end speech separation algorithm based on a phonetic language model of claim 1, wherein, The self-recurrent decoder is constructed based on a Whisper decoder architecture and is trained from scratch, and the process of outputting the zero-order codebook sequence in step S2 satisfies: wherein, denotes the probability distribution of the th word piece, denotes the historical zeroth order word piece sequence, denotes the vocabulary and satisfies .

7. The end-to-end speech separation algorithm based on a phonetic language model of claim 1, wherein, The non-self-recurrent model adopts the same architecture as the self-recurrent decoder, and removes the one-way mask.

8. The end-to-end speech separation algorithm based on a phonetic language model of claim 1, wherein, The non-self-recurrent model is configured with eight independent word embedding layers that share the same position embedding.

9. The end-to-end speech separation algorithm based on a phonetic language model of claim 1, wherein, The non-self-recurrent model introduces a task embedding mechanism to identify the current codebook prediction task type, and combines position embedding to jointly construct a total embedding vector, and the process of obtaining the total embedding vector in step S3 satisfies: wherein, denotes a word embedding of the th codebook sequence, denotes a position embedding, denotes a task embedding of the th task.

10. The end-to-end speech separation algorithm based on a phonetic language model of claim 9, wherein, The process of step S3 of sequentially predicting the high-order codebook sequence further includes: projecting the total embedding vector into a word embedding space of the high-order codebook through an input Transformer layer for deep modeling and a Softmax function, to obtain a probability distribution of all word units in the high-order codebook sequence, satisfying: ​​ wherein, denotes the output feature of a Transformer layer, denotes the word probability distribution of the order codebook sequence, denotes the projection matrix of the order codebook.

Images