Voice separation method and device, equipment and storage medium
By combining semantic pseudo-label training of speech recognition models and large language models, speech separation technology improves the accuracy and robustness of speech separation in complex acoustic environments, solving the problem of poor separation performance caused by relying solely on acoustic features in existing technologies.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-26
- Publication Date
- 2026-04-03
AI Technical Summary
Existing speech separation technologies suffer from decreased accuracy in complex acoustic environments, primarily due to their reliance on acoustic features, which leads to poor separation results.
By combining speech recognition models and large language models, and fusing semantic features of speech recognition text with acoustic features of mixed speech through a cross-attention mechanism, the speech separation model is trained using semantic pseudo-labels, thereby enhancing the semantic feature capture capability of the speech separation model and reducing the reliance on manual annotation.
It improves the accuracy and robustness of multi-speaker mixed speech separation, ensuring the quality of the separated speech data, and performs exceptionally well, especially in complex acoustic environments.
Smart Images
Figure CN121789710A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of speech processing technology, and in particular to a speech separation method, apparatus, device, and storage medium. Background Technology
[0002] With the rapid development of voice interaction technology, the demand for speech separation in multi-speaker mixed speech is increasing, such as speaker speech separation in conference recordings, speech extraction in public environments, and multi-user voice interaction on smart devices. The goal of speech separation technology is to accurately extract the independent speech data of each speaker from mixed speech, providing high-quality input for subsequent tasks such as speech recognition, speech storage, and speech analysis.
[0003] However, in current speech separation, speech separation usually relies solely on the acoustic features of the speech signal for separation, which leads to a decrease in the accuracy of the separated speech in complex acoustic environments. Summary of the Invention
[0004] This invention provides a speech separation method, apparatus, device, and storage medium to address the problem that in the prior art, speech separation typically relies solely on the acoustic features of the speech signal, leading to a decrease in the accuracy of the separated speech in complex acoustic environments.
[0005] This invention provides a speech separation method, the method comprising: Obtain the mixed speech to be separated; The mixed speech is input into a speech recognition model to obtain a first speech recognition text output by the speech recognition model; the first speech recognition text includes the speech text of each speaker identified in the mixed speech. The first speech recognition text and the mixed speech are input into the speech separation model to obtain the speech data of each speaker output by the speech separation model. The speech recognition model is obtained by training a first initial model on a first mixed speech sample labeled with speech recognition text tags. The speech separation model is obtained by training a second initial model on a second mixed speech sample labeled with speech data tags and its corresponding semantic pseudo-tags. The semantic pseudo-tags are obtained by processing the second mixed speech sample based on the speech recognition model and the large language model.
[0006] According to the speech separation method provided in the embodiments of the present invention, the step of inputting the first speech recognition text and the mixed speech into a speech separation model to obtain the speech data of each speaker output by the speech separation model includes: The first speech recognition text and the mixed speech are input into the speech separation model. Based on the encoder in the speech separation model, the first semantic features corresponding to the speech text of each speaker in the first speech recognition text and the acoustic features of the mixed speech are extracted respectively. For each speaker, the first semantic feature and the acoustic feature are interacted through a cross-attention mechanism to obtain semantic guidance fusion features; The semantic-guided fusion features of each speaker are input into the separator in the speech separation model to obtain the target acoustic features of each speaker. The target acoustic features of each speaker are input into the decoder in the speech separation model to obtain the speech data of each speaker output by the speech separation model.
[0007] According to the speech separation method provided in the embodiments of the present invention, the step of interacting the first semantic feature and the acoustic feature through a cross-attention mechanism to obtain semantically guided fusion features includes: The first semantic feature is used as the query vector, and the acoustic feature is used as the key vector and value vector; Attention weights are calculated based on the similarity between the query vector and the key vector, and semantically guided fusion features are obtained based on the attention weights and the value vector.
[0008] According to the speech separation method provided in the embodiments of the present invention, the speech recognition model is trained in the following manner: Obtain first speech recognition training data; the first speech recognition training data includes multiple first mixed speech samples labeled with speech recognition text tags, and the number of speakers associated with the multiple first mixed speech samples is different; The first mixed speech sample is input into the first initial model to obtain the speech recognition text prediction result output by the first initial model; Based on the speech recognition text prediction results and the speech recognition text labels, determine the first loss of the first initial model; Based on the first loss, update the model parameters of the first initial model until the training termination condition is met, and obtain the trained speech recognition model.
[0009] According to the speech separation method provided in this embodiment of the invention, the speech separation model is trained in the following manner: Acquire second speech recognition training data; the second speech recognition training data includes multiple second mixed speech samples labeled with speech data tags; The second mixed speech sample is sequentially input into the speech recognition model and the large language model to obtain the semantic pseudo-labels corresponding to the second mixed speech sample; The second mixed speech sample and the semantic pseudo-label are input into the second initial model to obtain the speech data prediction results of each speaker output by the second initial model; Based on the speech data prediction results, the speech data labels, and the semantic pseudo-labels, the joint loss of the second initial model is determined using the speech recognition model and the large language model. Based on the joint loss, the model parameters of the second initial model are updated until the training termination condition is met, resulting in the trained speech separation model.
[0010] According to the speech separation method provided in the embodiments of the present invention, the step of sequentially inputting the second mixed speech sample into the speech recognition model and the large language model to obtain the semantic pseudo-labels corresponding to the second mixed speech sample includes: The second mixed speech sample is input into the speech recognition model to obtain the second speech recognition text output by the speech recognition model; The second speech recognition text is input into the large language model to obtain the second semantic features of each speaker in the second speech recognition text output by the large language model; The second semantic features of each speaker in the second speech recognition text are used as the semantic pseudo-labels corresponding to the second mixed speech samples.
[0011] According to the speech separation method provided in the embodiments of the present invention, the step of determining the joint loss of the second initial model using the speech recognition model and the large language model based on the speech data prediction result, the speech data label, and the semantic pseudo-label includes: The speech data prediction results are sequentially input into the speech recognition model and the large language model to obtain the semantic prediction results corresponding to the second mixed speech sample; Based on the speech data prediction results and the speech data labels, a second loss of the second initial model is determined; and based on the semantic prediction results and the semantic pseudo-labels, a third loss of the second initial model is determined. The joint loss of the second initial model is determined based on the second loss and the third loss.
[0012] The present invention also provides a speech separation device, the device comprising: The acquisition module is used to acquire the mixed speech to be separated; The speech recognition module is used to input the mixed speech into the speech recognition model to obtain the first speech recognition text output by the speech recognition model; the first speech recognition text includes the speech text of each speaker identified in the mixed speech; The speech separation module is used to input the first speech recognition text and the mixed speech into the speech separation model to obtain the speech data of each speaker output by the speech separation model. The speech recognition model is obtained by training a first initial model on a first mixed speech sample labeled with speech recognition text tags. The speech separation model is obtained by training a second initial model on a second mixed speech sample labeled with speech data tags and its corresponding semantic pseudo-tags. The semantic pseudo-tags are obtained by processing the second mixed speech sample based on the speech recognition model and the large language model.
[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the above-described speech separation methods.
[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech separation method as described above.
[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the speech separation methods described above.
[0016] The speech separation method, apparatus, device, and storage medium provided by this invention input the first speech recognition text output by the speech recognition model and the mixed speech into the speech separation model. During the separation process, the semantic features of the mixed speech are introduced as constraints and guidance, overcoming the limitations of relying solely on acoustic features for separation. Simultaneously, the speech separation model is trained using semantic pseudo-labels generated by a large language model. This not only enhances the speech separation model's ability to capture the semantic features of the mixed speech but also overcomes the reliance on manual annotation during model training, ensuring consistency between the training and inference processes. Therefore, by fusing the acoustic and semantic features of the mixed speech, this invention enables the model to utilize the textual semantic content of the first speech recognition text output by the speech recognition model to assist in distinguishing the speech data of different speakers when facing complex acoustic environments. This improves the accuracy and robustness of multi-speaker mixed speech separation and ensures the quality of the separated speech data. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0018] Figure 1 This is one of the flowcharts of the speech separation method provided by the present invention.
[0019] Figure 2 This is the second flowchart of the speech separation method provided by the present invention.
[0020] Figure 3 This is the third flowchart of the speech separation method provided by the present invention.
[0021] Figure 4 This is the fourth flowchart of the speech separation method provided by the present invention.
[0022] Figure 5 This is a schematic diagram of the speech separation device provided by the present invention.
[0023] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0025] This invention provides a speech separation method applicable to various scenarios requiring the separation of different speakers' voices in mixed speech, such as meeting recording transcription, public speech interaction, and multi-speaker transcription. This method can be executed by an electronic device, such as a personal computer, server, mobile terminal (e.g., smartphone, tablet), smart home device, or dedicated speech processing device. The entity executing this method can be a locally deployed application or part of a cloud computing service; there are no limitations on this.
[0026] Figure 1 This is one of the flowcharts illustrating the speech separation method provided in this embodiment of the invention, such as... Figure 1 As shown, the method includes the following steps 110, 120 and 130.
[0027] Step 110: Obtain the mixed speech to be separated.
[0028] Here, mixed speech refers to a speech signal that contains the voices of two or more speakers. In addition to the speech signal containing the voices of the speakers, mixed speech may also contain background noise, echoes and other interfering speech signals.
[0029] In this embodiment, the mixed speech to be separated can be a speech signal collected by a microphone array on a terminal device such as a mobile phone, smart speaker, or conference recorder, or a speech signal from a pre-recorded audio file read from a storage device, or a speech signal received from other devices via a network device. There are no restrictions on which one is included.
[0030] It should be noted that the number of speakers included in the mixed speech in this embodiment is not limited, and it can be a mixed speech with two, three, or other numbers of speakers.
[0031] Step 120: Input the mixed speech into the speech recognition model to obtain the first speech recognition text output by the speech recognition model; the first speech recognition text includes the speech text of each speaker identified in the mixed speech.
[0032] In this embodiment, the Automatic Speech Recognition (ASR) model is trained on a first initial model based on a first mixed speech sample labeled with speech recognition text tags. It is used to process mixed speech containing an unknown number of speakers and, while outputting text, distinguish the content spoken by each speaker.
[0033] Specifically, a large number of first mixed speech samples labeled with speech recognition text are pre-constructed. The number of speakers in each first mixed speech sample is not fixed; it can be two, three, or any other number of speakers, ensuring that the trained speech recognition model can recognize mixed speech with any number of speakers. The speech recognition text labels are the manually annotated real text corresponding to the first mixed speech samples. This text includes not only the transcribed text content but also the speaker roles associated with each piece of text. Through supervised learning, such as using a connection-time classification loss function or an attention-based loss function, the parameters of the initial model are continuously optimized, making its output increasingly closer to the real speech recognition text labels. In this way, a speech recognition model capable of handling mixed speech from any number of speakers and performing content recognition and speaker differentiation can be trained.
[0034] Here, the first initial model is a neural network structure with sequence modeling capabilities that can map speech acoustic features into text sequences, such as a hybrid sequence model based on convolutional neural networks and recurrent neural networks, or a Transformer model, etc. The specific model can be flexibly selected according to the needs.
[0035] In this embodiment, the mixed speech to be separated is input into the trained speech recognition model. After internal acoustic feature extraction and other processing steps, the speech recognition model outputs the processed first speech recognition text.
[0036] Here, the first speech recognition text is structured text data, including not only the transcribed speech text of all speech signals, but also the speaker attribution information for each speech text. For example, for a mixed speech containing a dialogue between speaker A and speaker B, the content of the first speech recognition text is as follows: Speaker 1: Hello, how's the weather today? Speaker 2: It's fine, the weather is sunny and the wind is gentle.
[0037] The speech recognition model in this embodiment can not only recognize the speech content, but also the number of speakers in the mixed speech and the speech content of each speaker. The number of speakers is dynamically determined based on the mixed speech and does not need to be preset.
[0038] Step 130: Input the first speech recognition text and the mixed speech into the speech separation model to obtain the speech data of each speaker output by the speech separation model.
[0039] It should be understood that the speech separation model receives two parts of input: the original mixed speech and the first speech recognition text containing speaker attribution information. Through its internal network structure, the speech separation model uses the semantic features of the first speech recognition text as guidance to locate and extract the speech data corresponding to each speaker from the acoustic features of the mixed speech.
[0040] In practical applications, the speech data output by the speech separation model can be the separated audio waveform data, or the separated time spectrum or other acoustic feature representation speech data. The specific settings can be flexibly configured according to the requirements. The separated speech data can be used for subsequent storage, playback, or other speech analysis tasks.
[0041] In this embodiment, the speech separation model is obtained by training a second initial model based on a second mixed speech sample labeled with speech data tags and its corresponding semantic pseudo-tags. Here, the second mixed speech sample may be the same as or different from the first mixed speech sample. The speech data tags are the manually annotated clean speech signals of each speaker corresponding to the second mixed speech sample.
[0042] Here, the second initial model can be a convolutional model based on the time domain or frequency domain, such as a temporal convolutional network (TCN), a frequency domain convolutional model, a sequence modeling model based on a recurrent neural network, or a Transformer model based on a self-attention mechanism, etc., without any restrictions.
[0043] It should be noted that the semantic pseudo-labels in this embodiment are not pre-annotated manually, but are obtained during the model training phase by processing the second mixed speech sample based on the speech recognition model and the large language model. Specifically, during the training phase of the speech separation model, the pre-trained speech recognition model is used to recognize the second mixed speech sample. The recognition result of the second mixed speech sample is input into the large language model with semantic understanding capabilities, and the deep semantic features extracted by the large language model are the semantic pseudo-labels.
[0044] In this embodiment, when training the speech separation model, semantic pseudo-labels generated by the speech recognition model and the large language model are combined. This introduces high-level semantic supervision signals into the speech separation model during the training phase, constructs joint constraints of acoustic features and semantic features, and uses non-manually labeled semantic pseudo-labels during training. Therefore, even without manual labeling during the inference phase, the speech separation model can maintain consistency between training and inference based on the text recognized by the speech recognition model, thereby achieving more accurate separation results in complex multi-person aliasing scenarios.
[0045] The speech separation method provided in this invention inputs the first speech recognition text output by the speech recognition model and the mixed speech into the speech separation model. During the separation process, the semantic features of the mixed speech are introduced as constraints and guidance, overcoming the limitations of relying solely on acoustic features for separation. Simultaneously, semantic pseudo-labels generated by a large language model are used to train the speech separation model, which not only enhances the model's ability to capture the semantic features of the mixed speech but also overcomes the dependence on manual annotation during model training, ensuring consistency between the training and inference processes. Therefore, by fusing the acoustic and semantic features of the mixed speech, this invention enables the model to utilize the semantic content of the first speech recognition text output by the speech recognition model to help distinguish the speech data of different speakers when facing complex acoustic environments. This improves the accuracy and robustness of multi-speaker mixed speech separation and ensures the quality of the separated speech data.
[0046] In some embodiments, reference Figure 2 , Figure 2 This is a second schematic flowchart of the speech separation method provided in this embodiment of the invention, as shown below. Figure 2As shown, the method includes the following steps 210, 220, 230 and 240.
[0047] Step 210: Input the first speech recognition text and the mixed speech into the speech separation model, and extract the first semantic features corresponding to the speech text of each speaker in the first speech recognition text and the acoustic features of the mixed speech based on the encoder in the speech separation model.
[0048] In this embodiment, the speech separation model employs a network architecture consisting of an encoder, a separator, and a decoder. The encoder transforms the raw input data into processable high-dimensional features. In one example, the encoder may employ a dual-branch structure, including a text encoder and an acoustic encoder. The text encoder typically consists of word embedding layers and positional encoding layers, used to extract semantic features from the text data. The acoustic encoder may include convolutional layers, used to extract acoustic features from the speech data.
[0049] For each speaker's speech text in the first speech recognition text, it is first segmented into words. Then, the segmented words are mapped to the corresponding initial feature vectors through the word embedding layer in the text encoder. After that, the initial feature vector corresponding to each word is input to the position encoding layer of the text encoder. The position encoding layer generates a position vector based on the position of each word in the speech text and then superimposes and fuses it with the initial feature vector to obtain the first semantic feature.
[0050] For mixed speech, a preprocessing step is first performed, which cuts the mixed speech into fixed-length speech frames and converts them into frequency domain feature maps such as Mel spectrum, completing the time domain to frequency domain conversion. Then, the frequency domain feature map is input into the convolutional layer in the acoustic encoder. The convolution kernel in the convolutional layer performs sliding convolution operation on the frequency domain feature map, extracting features layer by layer. Finally, after multi-layer convolution processing, the acoustic features of the mixed speech are output.
[0051] It should be understood that the encoder in this embodiment may also adopt other network structures, and there are no restrictions on this.
[0052] Step 220: For the first semantic feature corresponding to each speaker, the first semantic feature and the acoustic feature are interacted through a cross-attention mechanism to obtain semantic guidance fusion features.
[0053] In this embodiment, semantic-guided fusion features refer to features that match the speaker's first semantic features, selected from acoustic features through a cross-attention mechanism, with the speaker's first semantic features as the core guide. The parts of these features that are related to the speaker's text content are amplified, while the parts that are not related to the speaker's text content are suppressed.
[0054] Specifically, after extracting the first semantic and acoustic features of each speaker, for each speaker's first semantic feature, a cross-attention mechanism is used to interact with the speaker's first semantic feature and acoustic features. For example, cross-attention calculation is performed using speaker A's first semantic feature and acoustic features to obtain a semantic guidance fusion feature specifically for speaker A. In this semantic guidance fusion feature, the parts related to speaker A's text content are amplified, while the parts unrelated to speaker A's text content are suppressed. Similarly, the same operation is performed on speaker B to obtain its corresponding semantic guidance fusion feature.
[0055] Step 230: Input the semantic guidance fusion features of each speaker into the separator in the speech separation model to obtain the target acoustic features of each speaker.
[0056] In this embodiment, the separator is used to separate the acoustic features of the target speaker from the acoustic features of interference. It should be understood that the semantic guidance fusion feature is a high-dimensional vector that combines the semantic features and acoustic features of the target speaker. It contains the speech information of the target speaker and may also be mixed with other interference speech information. Therefore, in this embodiment, the semantic guidance fusion feature is processed by the separator to capture the long-term dependence of the speech signal and remove residual background noise or interference speaker signals.
[0057] In one example, the separator can be composed of multiple complex neural networks, such as temporal convolutional networks, long short-term memory networks, or gated recurrent units, which perform deep processing on semantically guided fusion features to obtain the target acoustic features of each speaker. These target acoustic features contain only the pure acoustic information of the corresponding speaker.
[0058] For example, the separator adopts a multi-module cascaded temporal convolutional network architecture. After obtaining the semantic guidance fusion features of each speaker, the semantic guidance fusion features are input into the separator. The semantic guidance fusion features are subjected to multi-layer dilated convolution operations through multiple cascaded temporal convolutional networks in the separator. Each temporal convolutional network expands the convolution layer by layer with the network depth by setting an increasing dilation rate, thereby accurately capturing the full-scale temporal dependencies in the features from short-term syllables to long-term sentences. At the same time, while strengthening the acoustic feature response corresponding to the target semantics, the interference speech feature components remaining in the fusion features are suppressed, and the final output is the target acoustic feature containing only the acoustic attributes of a single speaker.
[0059] Step 240: Input the target acoustic features of each speaker into the decoder in the speech separation model to obtain the speech data of each speaker output by the speech separation model.
[0060] After obtaining the target acoustic features of each speaker, the target acoustic features of each speaker are input into the decoder in the speech separation model. The decoder performs the opposite operation to the encoder, such as inverse Fourier transform or transposed convolution, to restore the features to the time domain waveform, and finally obtain the speech data of each speaker.
[0061] The speech separation method provided in this embodiment of the invention combines the semantic features of the first speech recognition text with the acoustic features of the mixed speech, achieves semantically guided feature fusion through a cross-attention mechanism, and is then processed by a separator and a decoder to accurately separate the independent speech data of each speaker in the mixed speech, thereby improving the accuracy of speech separation.
[0062] In some embodiments, the step of interacting the first semantic feature and the acoustic feature through a cross-attention mechanism to obtain semantically guided fusion features includes: The first semantic feature is used as the query vector, and the acoustic feature is used as the key vector and value vector; Attention weights are calculated based on the similarity between the query vector and the key vector, and semantically guided fusion features are obtained based on the attention weights and the value vector.
[0063] It should be understood that the query vector represents the vector of the object to be retrieved, the key vector represents the vector of the candidate objects, and the value vector represents the vector of the actual valid information corresponding to the key vector.
[0064] In this embodiment, the first semantic feature corresponding to each speaker is set as the query vector. The query vector carries the semantic attributes corresponding to the text content of the target speaker and serves as the retrieval standard for subsequent acoustic feature filtering. The acoustic features of the mixed speech are used as key vectors and value vectors. The key vectors are used to calculate the similarity with the query vectors, while the value vectors are the acoustic feature ontology that will be weighted in the final filtering. It should be understood here that since the speech signal includes multiple consecutive signal frames, the key vectors and value vectors corresponding to the acoustic features of the mixed speech in this embodiment are both a set of sequence data, that is, the acoustic features corresponding to each signal frame are respectively used as a key vector and a value vector.
[0065] In this embodiment, the feature processing process of speaker A is used as an example for explanation. The feature processing processes of other speakers are the same as those of speaker A, and will not be described in detail here.
[0066] Specifically, the similarity between speaker A's query vector (i.e., semantic features) and speaker A's key vector (i.e., mixed acoustic features) is first calculated. For example, the semantic features and mixed acoustic features are multiplied by a dot product to obtain a similarity value sequence. The length of the similarity value sequence is the same as the number of frames in the mixed speech signal. If the sound features of a certain signal frame are similar to the semantic features of a certain speaker, for example, if the sound signal corresponding to the 5th signal frame includes the pronunciation of the word "weather" in speaker A's text content, then the similarity value between the 5th signal frame and speaker A's semantic features will be relatively large.
[0067] After calculating the similarity between the query vector and the key vector of speaker A, all similarity values of speaker A are normalized to obtain a set of attention weights with a sum of 1. The higher the attention weight, the more similar the corresponding key vector and query vector are.
[0068] Finally, the acoustic features corresponding to speaker A's value vector are multiplied by their corresponding attention weights, and then all weighted vectors are summed to obtain speaker A's semantic guidance fusion features. Therefore, the parts of speaker A's semantic guidance fusion features that are related to speaker A's text content are amplified, while the parts that are not related to speaker A's text content are suppressed.
[0069] The speech separation method provided in this embodiment of the invention uses the speaker's semantic features as query vectors and acoustic features as key and value vectors. It uses an attention mechanism to calculate the matching weights of semantic and acoustic features and weighted filters the acoustic features to finally generate accurate semantic guidance fusion features.
[0070] In some embodiments, reference Figure 3 , Figure 3 This is the third flowchart illustrating the speech separation method provided in this embodiment of the invention, as shown below. Figure 3 As shown, the method includes the following steps 310, 320, 330 and 340.
[0071] Step 310: Obtain first speech recognition training data; the first speech recognition training data includes multiple first mixed speech samples labeled with speech recognition text tags, and the number of speakers associated with the multiple first mixed speech samples is different.
[0072] It should be noted that, in order to train a speech recognition model capable of handling any number of speakers, this embodiment constructs a diverse training dataset. This dataset contains a large number of first mixed speech samples, and the number of speakers associated with each first mixed speech sample varies. For example, the dataset contains first mixed speech samples where two people speak simultaneously, as well as first mixed speech samples where three or more people speak alternately or overlappingly.
[0073] For each first mixed speech sample, an accurate speech recognition text label is required. This label includes not only all spoken content but also speaker identifiers, such as speaker A, speaker B, and speaker C. The speech recognition text label allows the model to learn to distinguish the speech content of different speakers.
[0074] In practical applications, the first mixed speech sample can be synthesized using real-recorded multi-person dialogue data, or it can be synthesized by manually mixing speech from a single-person clean speech dataset in different combinations. There are no restrictions on this.
[0075] Step 320: Input the first mixed speech sample into the first initial model to obtain the speech recognition text prediction result output by the first initial model.
[0076] In this embodiment, the first initial model can be a deep neural network capable of processing sequential data, such as an end-to-end initial speech recognition model (Automatic Speech Recognition, ASR) based on Transformer. The first initial model obtains the speech recognition text prediction result by performing forward propagation prediction processing on the input first mixed speech sample. The speech recognition text prediction result includes the speech text prediction result corresponding to each speaker predicted by the model.
[0077] Step 330: Determine the first loss of the first initial model based on the speech recognition text prediction result and the speech recognition text label.
[0078] Here, the first loss can be either cross-entropy loss or CTC (Connectionist Temporal Classification Loss), depending on the model structure of the initial model. For example, if the initial model is based on CTC, then the CTC loss function is used to calculate the first loss between the speech recognition text prediction result and the speech recognition text label. If the initial model is based on an attention mechanism, then the cross-entropy loss function is used to calculate the first loss between the speech recognition text prediction result and the speech recognition text label.
[0079] In this embodiment, the method of calculating loss is consistent with the method of calculating loss in the prior art, and will not be described in detail here.
[0080] Step 340: Update the model parameters of the first initial model according to the first loss until the training termination condition is met, and obtain the trained speech recognition model.
[0081] After obtaining the first loss, gradient descent methods, such as Adam and SGD optimizers, are used to calculate the gradient of the loss with respect to each parameter of the initial model, and the parameters are updated in the opposite direction of the gradient until the training termination condition is met.
[0082] Specifically, the training termination condition can be that the training reaches the preset maximum number of epochs, or the model's performance on the validation set no longer improves, or the training loss converges to below a certain threshold. When the training termination condition is met, the training process stops, and a satisfactory speech recognition model is obtained.
[0083] The speech separation method provided in this embodiment of the invention trains a first initial model using training data containing different numbers of speakers, so that the final speech recognition model can handle mixed speech from any number of speakers during the inference stage.
[0084] In some embodiments, reference Figure 4 , Figure 4 This is the fourth flowchart of the speech separation method provided in this embodiment of the invention, as shown below. Figure 4 As shown, the method includes the following steps 410, 420, 430, 440 and 450.
[0085] Step 410: Obtain second speech recognition training data; the second speech recognition training data includes multiple second mixed speech samples labeled with speech data tags.
[0086] It should be noted that, in order to train a speech separation model capable of separating mixed speech, this embodiment constructs a diverse training dataset. This dataset contains a large number of second mixed speech samples, where the number of speakers associated with each second mixed speech sample can be the same or different.
[0087] For each second mixed speech sample, an accurate speech data label is required. This label refers to the original clean speech signal of each speaker corresponding to the mixed speech sample. For example, a label for clean speech A labeled by speaker A and a label for clean speech B labeled by speaker B.
[0088] In practical applications, the second mixed speech sample can be synthesized using real-recorded multi-person dialogue data, or it can be synthesized by manually mixing the speech from a single-person clean speech dataset in different orders. There are no restrictions on this.
[0089] Step 420: Input the second mixed speech sample into the speech recognition model and the large language model in sequence to obtain the semantic pseudo-labels corresponding to the second mixed speech sample.
[0090] Here, semantic pseudo-labels refer to labels that are not pre-annotated manually, but rather obtained by processing the second mixed speech sample through speech recognition models and large language models.
[0091] In this embodiment, the second mixed speech sample is input into a pre-trained speech recognition model to obtain the recognized text with speaker attribution. Then, this text, or the text of each speaker within it, is input into a pre-trained large language model, such as GPT or BERT. The large language model encodes the text and extracts its deep semantic representation, which is used as a semantic pseudo-label.
[0092] Step 430: Input the second mixed speech sample and the semantic pseudo-label into the second initial model to obtain the speech data prediction results of each speaker output by the second initial model.
[0093] After obtaining the semantic pseudo-labels, the second mixed speech sample and the semantic pseudo-labels generated in the previous step are input into the second initial model. The second initial model extracts the time-frequency features or time-domain features of the mixed speech in the second mixed speech sample, and then, combined with the guidance of the semantic pseudo-labels, distinguishes the speech feature boundaries of different speakers, and finally outputs the speech data prediction results for each speaker.
[0094] Step 440: Based on the speech data prediction results, the speech data labels, and the semantic pseudo-labels, determine the joint loss of the second initial model using the speech recognition model and the large language model.
[0095] In this embodiment, the joint loss refers to the total loss resulting from the fusion of acoustic feature loss and semantic feature loss. Specifically, the speech data prediction result of the second mixed speech sample is compared with the speech data label of the second mixed speech sample to calculate the corresponding acoustic feature loss; the speech data prediction result of the second mixed speech sample is converted into text, and then semantic features are extracted by a large language model. Subsequently, this semantic feature is compared with the semantic pseudo-label of the second mixed speech sample to calculate the semantic feature loss. Finally, the acoustic feature loss and semantic feature loss are weighted and fused to obtain the joint loss.
[0096] In some embodiments, determining the joint loss of the second initial model based on the speech data prediction result, the speech data label, and the semantic pseudo-label, using the speech recognition model and the large language model, includes: The speech data prediction results are sequentially input into the speech recognition model and the large language model to obtain the semantic prediction results corresponding to the second mixed speech sample; Based on the speech data prediction results and the speech data labels, a second loss of the second initial model is determined; and based on the semantic prediction results and the semantic pseudo-labels, a third loss of the second initial model is determined. The joint loss of the second initial model is determined based on the second loss and the third loss.
[0097] In this embodiment, after the speech data prediction results are input into the speech recognition model, the speech recognition model will convert these predicted speech waveforms into corresponding speech text; then the speech text is input into the large language model, which performs semantic parsing and feature extraction on the speech text, and finally outputs a feature vector that can represent the core semantics of the predicted speech, that is, the semantic prediction result.
[0098] After obtaining the semantic prediction result, the predicted speech data is compared with the actual speech data labels to calculate the loss, resulting in the corresponding second loss. Here, the signal-to-noise ratio (SNR) can be used as the loss function, and its goal is to maximize the SNR between the separated speech and the clean speech. The higher the SNR value, the better the acoustic separation effect, and the smaller the corresponding loss value.
[0099] In addition to calculating the second loss, this embodiment also calculates the loss between the semantic prediction result and the corresponding semantic pseudo-label to obtain the corresponding third loss. Here, the third loss can be cross-entropy loss, cosine similarity loss, etc., and its goal is to make the directions of the two vectors as consistent as possible. In this embodiment, the loss calculation method is consistent with the loss calculation method in the prior art, and will not be described in detail here.
[0100] After obtaining the second and third losses, a weighted sum is performed on the second and third losses to obtain the final joint loss. In practical applications, different weight coefficients can be assigned to the two losses before weighted summation according to the actual separation requirements.
[0101] Step 450: Update the model parameters of the second initial model according to the joint loss until the training termination condition is met, and obtain the trained speech separation model.
[0102] After obtaining the joint loss, gradient descent methods, such as Adam and SGD optimizers, are used to calculate the gradient of the loss with respect to each parameter of the second initial model, and the parameters are updated in the opposite direction of the gradient until the training termination condition is met.
[0103] Specifically, the training termination condition can be that the training reaches the preset maximum number of epochs, or the model's performance on the validation set no longer improves, or the training loss converges to below a certain threshold. When the training termination condition is met, the training process stops, and a speech separation model that meets the requirements is obtained.
[0104] The speech separation method provided in this invention improves the separation accuracy of the trained speech separation model through joint training based on acoustics and semantics.
[0105] In some embodiments, the step of sequentially inputting the second mixed speech sample into the speech recognition model and the large language model to obtain the semantic pseudo-label corresponding to the second mixed speech sample includes: The second mixed speech sample is input into the speech recognition model to obtain the second speech recognition text output by the speech recognition model; The second speech recognition text is input into the large language model to obtain the second semantic features of each speaker in the second speech recognition text output by the large language model; The second semantic feature of each speaker in the second speech recognition text is used as the semantic pseudo-label corresponding to the second mixed speech sample.
[0106] In this embodiment, the second mixed speech sample is input into a pre-trained speech recognition model. The model will extract acoustic features, perform temporal modeling and character decoding on the second mixed speech sample based on the correspondence between the speech acoustic features and text characters it has learned. Finally, it outputs the text content corresponding to each speaker in the second mixed speech sample, which is the second speech recognition text.
[0107] Next, after inputting the second speech recognition text corresponding to each speaker into the large language model, the large language model will first perform semantic parsing on the second speech recognition text and extract feature vectors that can represent its core semantics, that is, the second semantic features.
[0108] After the above processing, the second semantic features of each speaker in the second speech recognition text are used as the semantic pseudo-labels corresponding to the second mixed speech samples, so as to provide semantic supervision signals for the training of the subsequent speech separation model.
[0109] The speech separation method provided in this invention converts mixed speech into speech text using a speech recognition model, then uses a large language model to split and extract the semantic features of each speaker, and finally generates semantic pseudo-labels for each speaker, providing accurate semantic supervision for subsequent speech separation models.
[0110] Based on any of the above embodiments, the present invention also provides a speech separation device, with reference to... Figure 5 The device includes: an acquisition module 510, a speech recognition module 520, and a speech separation module 530.
[0111] The acquisition module 510 is used to acquire the mixed speech to be separated; The speech recognition module 520 is used to input the mixed speech into the speech recognition model to obtain the first speech recognition text output by the speech recognition model; the first speech recognition text includes the speech text of each speaker identified in the mixed speech; The speech separation module 530 is used to input the first speech recognition text and the mixed speech into the speech separation model to obtain the speech data of each speaker output by the speech separation model; The speech recognition model is obtained by training a first initial model on a first mixed speech sample labeled with speech recognition text tags. The speech separation model is obtained by training a second initial model on a second mixed speech sample labeled with speech data tags and its corresponding semantic pseudo-tags. The semantic pseudo-tags are obtained by processing the second mixed speech sample based on the speech recognition model and the large language model.
[0112] The speech separation device provided in this invention inputs the first speech recognition text output by the speech recognition model and the mixed speech into the speech separation model. During the separation process, the semantic features of the mixed speech are introduced as constraints and guidance, overcoming the limitations of relying solely on acoustic features for separation. Simultaneously, the speech separation model is trained using semantic pseudo-labels generated by a large language model. This not only strengthens the speech separation model's ability to capture the semantic features of the mixed speech but also overcomes the reliance on manual annotation during model training, ensuring consistency between the training and inference processes. Therefore, by fusing the acoustic and semantic features of the mixed speech, this invention enables the model to utilize the textual semantic content of the first speech recognition text output by the speech recognition model to assist in distinguishing the speech data of different speakers when facing complex acoustic environments. This improves the accuracy and robustness of multi-speaker mixed speech separation and ensures the quality of the separated speech data.
[0113] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a voice separation method, the method including: Obtain the mixed speech to be separated; The mixed speech is input into a speech recognition model to obtain a first speech recognition text output by the speech recognition model; the first speech recognition text includes the speech text of each speaker identified in the mixed speech. The first speech recognition text and the mixed speech are input into the speech separation model to obtain the speech data of each speaker output by the speech separation model. The speech recognition model is obtained by training a first initial model on a first mixed speech sample labeled with speech recognition text tags. The speech separation model is obtained by training a second initial model on a second mixed speech sample labeled with speech data tags and its corresponding semantic pseudo-tags. The semantic pseudo-tags are obtained by processing the second mixed speech sample based on the speech recognition model and the large language model.
[0114] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0115] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is capable of performing the speech separation methods provided by the above methods, the methods comprising: Obtain the mixed speech to be separated; The mixed speech is input into a speech recognition model to obtain a first speech recognition text output by the speech recognition model; the first speech recognition text includes the speech text of each speaker identified in the mixed speech. The first speech recognition text and the mixed speech are input into the speech separation model to obtain the speech data of each speaker output by the speech separation model. The speech recognition model is obtained by training a first initial model on a first mixed speech sample labeled with speech recognition text tags. The speech separation model is obtained by training a second initial model on a second mixed speech sample labeled with speech data tags and its corresponding semantic pseudo-tags. The semantic pseudo-tags are obtained by processing the second mixed speech sample based on the speech recognition model and the large language model.
[0116] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the speech separation methods provided by the methods described above, the methods comprising: Obtain the mixed speech to be separated; The mixed speech is input into a speech recognition model to obtain a first speech recognition text output by the speech recognition model; the first speech recognition text includes the speech text of each speaker identified in the mixed speech. The first speech recognition text and the mixed speech are input into the speech separation model to obtain the speech data of each speaker output by the speech separation model. The speech recognition model is obtained by training a first initial model on a first mixed speech sample labeled with speech recognition text tags. The speech separation model is obtained by training a second initial model on a second mixed speech sample labeled with speech data tags and its corresponding semantic pseudo-tags. The semantic pseudo-tags are obtained by processing the second mixed speech sample based on the speech recognition model and the large language model.
[0117] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0118] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A speech separation method, characterized in that, The method includes: Obtain the mixed speech to be separated; The mixed speech is input into a speech recognition model to obtain a first speech recognition text output by the speech recognition model; the first speech recognition text includes the speech text of each speaker identified in the mixed speech. The first speech recognition text and the mixed speech are input into the speech separation model to obtain the speech data of each speaker output by the speech separation model. The speech recognition model is obtained by training a first initial model on a first mixed speech sample labeled with speech recognition text tags. The speech separation model is obtained by training a second initial model on a second mixed speech sample labeled with speech data tags and its corresponding semantic pseudo-tags. The semantic pseudo-tags are obtained by processing the second mixed speech sample based on the speech recognition model and the large language model.
2. The speech separation method according to claim 1, characterized in that, The step of inputting the first speech recognition text and the mixed speech into the speech separation model to obtain the speech data of each speaker output by the speech separation model includes: The first speech recognition text and the mixed speech are input into the speech separation model. Based on the encoder in the speech separation model, the first semantic features corresponding to the speech text of each speaker in the first speech recognition text and the acoustic features of the mixed speech are extracted respectively. For each speaker, the first semantic feature and the acoustic feature are interacted through a cross-attention mechanism to obtain semantic guidance fusion features; The semantic-guided fusion features of each speaker are input into the separator in the speech separation model to obtain the target acoustic features of each speaker. The target acoustic features of each speaker are input into the decoder in the speech separation model to obtain the speech data of each speaker output by the speech separation model.
3. The speech separation method according to claim 2, characterized in that, The step of interacting the first semantic feature and the acoustic feature through a cross-attention mechanism to obtain semantically guided fusion features includes: The first semantic feature is used as the query vector, and the acoustic feature is used as the key vector and value vector; Attention weights are calculated based on the similarity between the query vector and the key vector, and semantically guided fusion features are obtained based on the attention weights and the value vector.
4. The speech separation method according to claim 1, characterized in that, The speech recognition model was trained in the following way: Obtain first speech recognition training data; the first speech recognition training data includes multiple first mixed speech samples labeled with speech recognition text tags, and the number of speakers associated with the multiple first mixed speech samples is different; The first mixed speech sample is input into the first initial model to obtain the speech recognition text prediction result output by the first initial model; Based on the speech recognition text prediction results and the speech recognition text labels, determine the first loss of the first initial model; Based on the first loss, update the model parameters of the first initial model until the training termination condition is met, and obtain the trained speech recognition model.
5. The speech separation method according to claim 1, characterized in that, The speech separation model was trained in the following way: Acquire second speech recognition training data; the second speech recognition training data includes multiple second mixed speech samples labeled with speech data tags; The second mixed speech sample is sequentially input into the speech recognition model and the large language model to obtain the semantic pseudo-labels corresponding to the second mixed speech sample; The second mixed speech sample and the semantic pseudo-label are input into the second initial model to obtain the speech data prediction results of each speaker output by the second initial model; Based on the speech data prediction results, the speech data labels, and the semantic pseudo-labels, the joint loss of the second initial model is determined using the speech recognition model and the large language model. Based on the joint loss, the model parameters of the second initial model are updated until the training termination condition is met, resulting in the trained speech separation model.
6. The speech separation method according to claim 5, characterized in that, The step of sequentially inputting the second mixed speech sample into the speech recognition model and the large language model to obtain the semantic pseudo-labels corresponding to the second mixed speech sample includes: The second mixed speech sample is input into the speech recognition model to obtain the second speech recognition text output by the speech recognition model; The second speech recognition text is input into the large language model to obtain the second semantic features of each speaker in the second speech recognition text output by the large language model; The second semantic feature of each speaker in the second speech recognition text is used as the semantic pseudo-label corresponding to the second mixed speech sample.
7. The speech separation method according to claim 5, characterized in that, The step of determining the joint loss of the second initial model based on the speech data prediction result, the speech data label, and the semantic pseudo-label, using the speech recognition model and the large language model, includes: The speech data prediction results are sequentially input into the speech recognition model and the large language model to obtain the semantic prediction results corresponding to the second mixed speech sample; Based on the speech data prediction results and the speech data labels, a second loss of the second initial model is determined; and based on the semantic prediction results and the semantic pseudo-labels, a third loss of the second initial model is determined. The joint loss of the second initial model is determined based on the second loss and the third loss.
8. A speech separation device, characterized in that, include: The acquisition module is used to acquire the mixed speech to be separated; The speech recognition module is used to input the mixed speech into the speech recognition model to obtain the first speech recognition text output by the speech recognition model; the first speech recognition text includes the speech text of each speaker identified in the mixed speech; The speech separation module is used to input the first speech recognition text and the mixed speech into the speech separation model to obtain the speech data of each speaker output by the speech separation model. The speech recognition model is obtained by training a first initial model on a first mixed speech sample labeled with speech recognition text tags. The speech separation model is obtained by training a second initial model on a second mixed speech sample labeled with speech data tags and its corresponding semantic pseudo-tags. The semantic pseudo-tags are obtained by processing the second mixed speech sample based on the speech recognition model and the large language model.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the speech separation method as described in any one of claims 1 to 7.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the speech separation method as described in any one of claims 1 to 7.