A speech signal extraction method, device, equipment and computer readable medium

Abstract

The present disclosure provides a speech signal extraction method, device and equipment and computer readable medium, belonging to the technical field of signal processing, which comprises: inputting a mixed speech signal into a pre-trained encoder to obtain a latent space representation of the mixed speech signal, the mixed speech signal comprising speech signals of at least two speakers; obtaining an auxiliary feature representation of a target speaker; inputting the latent space representation of the mixed speech signal and the auxiliary feature representation of the target speaker into a pre-trained target speaker extraction model to obtain a mask of the target speaker; obtaining a latent feature representation of the target speaker according to the mask of the target speaker and the latent space representation; and decoding the latent space representation of the target speaker through a decoder to obtain the speech signal of the target speaker, thereby achieving real-time extraction, enhancing the generalization ability of the speech signal extraction of the target speaker, and improving the accuracy of the extraction.

Description

Technical Field

[0001] This disclosure relates to the field of signal processing technology, and in particular to a method, apparatus, device and computer-readable medium for extracting speech signals. Background Technology

[0002] Currently, the target speaker extraction technology framework in academia and industry generally consists of an extraction network composed of an encoder, extractor, and decoder, and an auxiliary network that specifically provides target speaker features. The encoder's role is to transform the time-domain signal of the input mixed speech into another latent space, forming a feature vector representation of the mixed speech in this latent space. The extractor's role is to extract the target speech in the latent space. The decoder's role is to inversely transform the extracted target speaker feature vector representation back into a time-domain signal. The auxiliary network's role is to provide the extraction network with information related to the target speaker, that is, to tell the extraction network which person's voice should be extracted. This is the existing target speaker extraction technology framework.

[0003] The limitations of existing target speaker extraction technology frameworks are as follows: the extraction network used for target speaker extraction is limited to the output of the target speaker's target speech, and does not separate the speech signals of each speaker in the mixed speech. Moreover, the performance of the extraction network depends entirely on the quality of the target speaker's voiceprint features transmitted by the auxiliary network. The closer the voiceprint features of the target speaker provided by the auxiliary network are to the features of that speaker in the mixed speech, the better the extraction effect. At the same time, the models of existing speaker extraction technologies are relatively large and cannot be used for real-time extraction. In practical engineering applications, it is often difficult to meet the requirements. If the quality of the features output by the auxiliary network is poor, it will directly affect the performance of target speaker extraction. Therefore, the versatility and robustness are not strong. Summary of the Invention

[0004] In view of this, the present disclosure aims to provide a speech signal extraction method, apparatus, device and computer-readable medium. The present disclosure no longer uses the above-mentioned target speaker extraction technology framework, but trains the speaker separation model and encoder separately to obtain the target speaker extraction model. By training a perfect target speaker extraction model, it is possible to extract in real time, enhance the generalization ability of speech signal extraction of target speakers, and improve the extraction accuracy.

[0005] A first aspect of this disclosure provides a method for extracting speech signals, the method comprising:

[0006] The mixed speech signal is input into a pre-trained encoder to obtain the latent spatial representation of the mixed speech signal, wherein the mixed speech signal includes the speech signals of at least two speakers, including the target speaker;

[0007] Obtain the auxiliary feature representation of the target speaker;

[0008] The latent spatial representation of the mixed speech signal and the auxiliary feature representation of the target speaker are input into the pre-trained target speaker extraction model to obtain the masking of the target speaker;

[0009] Based on the masking of the target speaker and the latent spatial representation of the mixed speech signal, the latent feature representation of the target speaker is obtained;

[0010] The latent spatial representation of the target speaker is decoded by a decoder to obtain the speech signal of the target speaker.

[0011] In one implementation, obtaining the auxiliary feature representation of the target speaker includes:

[0012] Acquire the original speech signal of the target speaker;

[0013] The original speech signal of the target speaker is input into the pre-trained encoder to obtain the auxiliary feature representation of the target speaker.

[0014] In one implementation, before inputting the latent spatial representation of the mixed speech signal and the auxiliary feature representation of the target speaker into the pre-trained target speaker extraction model, the method further includes:

[0015] The latent spatial representation of the mixed speech signal is concatenated with the auxiliary feature representation of the target speaker to obtain the concatenated feature representation.

[0016] The input dimension of the concatenated feature representation is changed by using a linear layer;

[0017] The latent spatial representation of the mixed speech signal and the auxiliary feature representation of the target speaker are input into a pre-trained target speaker extraction model, including:

[0018] The concatenated feature representation after the input dimension is changed is input into the pre-trained target speaker extraction model.

[0019] In one implementation, obtaining the latent feature representation of the target speaker based on the masking of the target speaker and the latent spatial representation of the mixed speech signal includes:

[0020] The masking of the target speaker is multiplied by the latent spatial representation of the mixed speech signal, wherein the result of the multiplication is the latent feature representation of the target speaker.

[0021] In one implementation, the trained target speaker extraction model includes a trained encoder and a trained speaker separation model, which are trained through the following steps:

[0022] Standard feature representations of each of the speech signals in the first test mixed speech signal are obtained in advance, wherein the first test mixed speech signal includes speech signals of multiple speakers;

[0023] The first test mixed speech signal is input into the encoder to be trained to obtain the latent spatial representation of the first test mixed speech signal.

[0024] The latent spatial representation of the first test mixed speech signal is input into the speaker separation model to be trained to obtain the masking of each speaker among multiple speakers;

[0025] Based on the masking of each speaker among the plurality of speakers and the latent spatial representation of the first test mixed speech signal, the latent feature representation of each speaker among the plurality of speakers is obtained;

[0026] Based on the latent feature representation and corresponding standard feature representation of each of the multiple speakers, the parameters of the speaker separation model to be trained and the parameters of the encoder to be trained are updated to obtain the trained speaker separation model and the trained encoder.

[0027] In one implementation, updating the parameters of the speaker separation model to be trained and the parameters of the encoder to be trained based on the latent feature representation and corresponding standard feature representation of each of the plurality of speakers to obtain the trained speaker separation model and the trained encoder includes:

[0028] Based on the standard feature representation and the corresponding latent feature representation of each of the plurality of speakers, calculate the loss value of each of the plurality of speakers, wherein the loss value is the absolute value of the difference between the standard feature representation and the latent feature representation;

[0029] The parameters of the speaker separation model and the encoder to be trained are updated using the loss values ​​of the multiple speakers until the loss value of each of the multiple speakers is less than a preset threshold. Then the training ends, and the trained speaker separation model and the trained encoder are obtained.

[0030] In one implementation, after the speaker separation model and encoder have been trained, the method further includes:

[0031] A second test mixed speech signal is pre-acquired, and a standard feature representation of the speech signal of the target speaker is obtained, wherein the second test mixed speech signal includes the speech signals of multiple speakers, and the speech signals of the multiple speakers include the speech signal of the target speaker;

[0032] The second test mixed speech signal is input into the pre-trained encoder to obtain the latent spatial representation of the second test mixed speech signal;

[0033] The latent spatial representation of the second test mixed speech signal and the standard feature representation of the target speaker are input into the trained speaker separation model to obtain the masking of the target speaker;

[0034] Based on the masking of the target speaker and the latent spatial representation of the second test mixed speech signal, the latent feature representation of the target speaker is obtained;

[0035] Based on the latent feature representation of the target speaker and the corresponding standard feature representation, the parameters of the speaker separation model to be trained and the parameters of the encoder to be trained are updated a second time to obtain the trained target speaker extraction model.

[0036] In one implementation, updating the parameters of the speaker separation model to be trained and the parameters of the encoder to be trained based on the latent feature representation and the corresponding standard feature representation of the target speaker to obtain the trained target speaker extraction model includes:

[0037] Based on the standard feature representation and latent feature representation of the target speaker, the loss value of the target speaker is calculated, wherein the loss value is the absolute value of the difference between the standard feature representation and latent feature representation of the target speaker;

[0038] The parameters of the speaker separation model to be trained and the parameters of the encoder to be trained are updated twice using the loss value of the target speaker until the loss value of the target speaker is less than a preset threshold. Then the training ends and the trained target speaker extraction model is obtained.

[0039] A second aspect of this disclosure provides a speech extraction device, the device comprising:

[0040] The first input module is used to input the mixed speech signal into a pre-trained encoder to obtain the latent spatial representation of the mixed speech signal, wherein the mixed speech signal includes the speech signals of at least two speakers, and the target speaker is among the at least two speakers;

[0041] The first acquisition module is used to acquire auxiliary feature representations of the target speaker;

[0042] The second input module is used to input the latent spatial representation of the mixed speech signal and the auxiliary feature representation of the target speaker into the pre-trained target speaker extraction model to obtain the masking of the target speaker;

[0043] The first calculation module is used to obtain the latent feature representation of the target speaker based on the masking of the target speaker and the latent spatial representation of the mixed speech signal;

[0044] The first decoding module is used to decode the latent spatial representation of the target speaker through a decoder to obtain the speech signal of the target speaker.

[0045] A third aspect of this disclosure provides an apparatus including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method as described in the first aspect.

[0046] A fourth aspect of this disclosure provides a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in the first aspect.

[0047] The beneficial effects of this disclosure are:

[0048] This disclosure provides a speech signal extraction method, apparatus, device, and computer-readable medium. The method involves inputting a mixed speech signal into a pre-trained encoder to obtain a latent spatial representation of the mixed speech signal. The mixed speech signal includes speech signals from at least two speakers, one of whom is a target speaker. The method then acquires auxiliary feature representations of the target speaker. The latent spatial representation of the mixed speech signal and the auxiliary feature representation of the target speaker are input into a pre-trained target speaker extraction model to obtain a mask for the target speaker. Based on the mask and the latent spatial representation of the mixed speech signal, a latent feature representation of the target speaker is obtained. Finally, a decoder decodes the latent spatial representation of the target speaker to obtain the speech signal of the target speaker. This achieves real-time extraction, enhances the generalization ability of extracting the speech signal of the target speaker, and improves the accuracy of extraction. Attached Figure Description

[0049] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure. In the drawings:

[0050] Figure 1This is a flowchart of a speech signal extraction method provided in an embodiment of the present disclosure;

[0051] Figure 2 This is a flowchart of a method for training a speaker separation model and encoder provided in an embodiment of this disclosure;

[0052] Figure 3 This is a schematic diagram of the speaker separation model and encoder provided in one embodiment of the present disclosure;

[0053] Figure 4 This is a flowchart of a training target speaker extraction model method provided in an embodiment of this disclosure;

[0054] Figure 5 This is a schematic diagram of the framework of a target speaker extraction model provided in an embodiment of this disclosure;

[0055] Figure 6 This is a schematic diagram of the framework of a speech signal extraction device provided in one embodiment of the present disclosure. Detailed Implementation

[0056] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0057] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0058] Figure 1 This is a flowchart of a speech signal extraction method provided in one embodiment of the present disclosure, as follows: Figure 1 As shown in this embodiment, a speech signal extraction method is provided, the method comprising:

[0059] In step S101, the mixed speech signal is input into a pre-trained encoder to obtain the latent spatial representation of the mixed speech signal. The mixed speech signal includes the speech signals of at least two speakers, including the target speaker among the at least two speakers.

[0060] In step S102, the auxiliary feature representation of the target speaker is obtained;

[0061] In step S103, the latent spatial representation of the mixed speech signal and the auxiliary feature representation of the target speaker are input into the pre-trained target speaker extraction model to obtain the masking of the target speaker;

[0062] In step S104, the latent feature representation of the target speaker is obtained based on the masking of the target speaker and the latent spatial representation of the mixed speech signal;

[0063] In step S105, the latent spatial representation of the target speaker is decoded by a decoder to obtain the speech signal of the target speaker.

[0064] First, in step S101, the mixed speech signal is input into a pre-trained encoder to obtain the latent space representation of the mixed speech signal. The mixed speech signal includes speech signals from at least two speakers, one of whom is the target speaker. We need to extract the target speaker's speech signal from the mixed speech signal. Here, the speech signal refers to the time-domain signal. The mixed speech signal is formed by combining the speech signals of at least two speakers. We need the target speaker's speech signal to be mixed into the mixed speech signal. Therefore, we can input the mixed speech signal into the pre-trained encoder to obtain the latent space representation of the mixed speech signal. The encoder consists of multiple networks, such as a certain number of convolutional neural networks. The encoder transforms the time-domain signal form of the mixed speech signal into a latent space representation with high separability, better distinguishing different speakers. This latent space representation is expressed as a vector in this latent space.

[0065] Further, in step S102, the auxiliary feature representation of the target speaker is obtained. We also need to obtain the auxiliary feature representation of the target speaker. The auxiliary feature representation can serve as auxiliary information in the process of extracting the target speaker's speech signal from the mixed speech signal. This auxiliary information can be understood as an identifier that corresponds to the characteristics of the target speaker, so that we can identify the target speaker's speech signal that we need to extract from the mixed speech signal based on this auxiliary feature representation.

[0066] Further, in step S103, the latent spatial representation of the mixed speech signal and the auxiliary feature representation of the target speaker are input into the pre-trained target speaker extraction model to obtain the masking of the target speaker. By simultaneously inputting the latent spatial representation of the mixed speech signal to be processed and the auxiliary feature representation of the target speaker into the pre-trained target speaker extraction model, the masking of the target speaker is obtained. The masking includes various independent features of the speech signal, such as pitch features, voiceprint features, timbre features, and frequency features. Therefore, the masking of the target speaker consists of various independent features that match the target speaker, and these independent features differ from the independent features of the other speakers' speech signals in the mixed speech signal.

[0067] In other embodiments, the original speech signal of the target speaker can also be directly input into the target speaker extraction model.

[0068] Further, in step S104, the latent feature representation of the target speaker is obtained based on the masking of the target speaker and the latent spatial representation of the mixed speech signal. The latent feature representation of the target speaker is obtained by calculating the masking of the target speaker and the latent spatial representation of the mixed speech signal.

[0069] Finally, in step S105, the latent spatial representation of the target speaker is decoded by the decoder to obtain the speech signal of the target speaker. The obtained latent spatial representation of the target speaker is input into the decoder, and the latent spatial representation is decoded by the decoder to obtain the final speech signal of the target speaker, thus completing the entire process of extracting the speech signal of the target speaker from a mixed speech signal containing the speech signals of multiple speakers.

[0070] Optionally, in step S102 above, obtaining the auxiliary feature representation of the target speaker includes:

[0071] Acquire the original speech signal of the target speaker;

[0072] The original speech signal of the target speaker is input into the pre-trained encoder to obtain the auxiliary feature representation of the target speaker.

[0073] First, obtain the original voice signal of the target speaker. We can obtain the original voice signal of the target speaker in advance. This original voice signal can be obtained by pre-recording the voice of the target speaker. That is, we can pre-record the voices of everyone and obtain the original voice signals of everyone. From this, we can select the original voice signal of the target speaker that we need.

[0074] Then, the original speech signal of the target speaker is input into the pre-trained encoder to obtain the auxiliary feature representation of the target speaker. We can input the original speech signal of the target speaker into the trained encoder to obtain the auxiliary feature representation of the target speaker. This auxiliary feature representation is a better feature representation for identifying the target speaker and can serve as auxiliary information in the process of extracting the target speaker's speech signal from a mixed speech signal containing the speech signals of multiple speakers. The auxiliary feature representation of the target speaker is not a feature representation of the target speaker's speech signal in the mixed speech signal, but rather an auxiliary speech signal obtained in advance from the original speech signal; that is, the original speech signal of the target speaker is not the same as the speech signal of the target speaker in the mixed speech signal.

[0075] Optionally, before step S103, the method further includes:

[0076] The latent spatial representation of the mixed speech signal is concatenated with the auxiliary feature representation of the target speaker to obtain the concatenated feature representation.

[0077] The input dimension of the concatenated feature representation is changed by using a linear layer;

[0078] The latent spatial representation of the mixed speech signal and the auxiliary feature representation of the target speaker are input into a pre-trained target speaker extraction model, including:

[0079] The concatenated feature representation after the input dimension is changed is input into the pre-trained target speaker extraction model.

[0080] First, the latent spatial representation of the mixed speech signal is concatenated with the auxiliary feature representation of the target speaker to obtain a concatenated feature representation. Before inputting the latent spatial representation of the mixed speech signal and the auxiliary feature representation of the target speaker into the pre-trained target speaker extraction model, it is necessary to concatenate them to obtain the concatenated feature representation.

[0081] Furthermore, the input dimension of the concatenated feature representation is changed through a linear layer. Because the addition of auxiliary feature information of the target speaker alters the model dimension of the pre-trained target speaker extraction model, the input dimension of the concatenated feature representation needs to be changed through a linear layer before being input into the target speaker extraction model.

[0082] Finally, the latent spatial representation of the mixed speech signal and the auxiliary feature representation of the target speaker are input into the pre-trained target speaker extraction model. By changing the input dimension of the concatenated feature representation in the above steps, a concatenated feature representation with modified input dimensions is obtained. The latent spatial representation of the mixed speech signal and the auxiliary feature representation of the target speaker are then input into the pre-trained target speaker extraction model to ensure that the pre-trained target speaker extraction model can operate normally.

[0083] Specifically, the concatenated feature representation with altered input dimensions is input into the pre-trained target speaker extraction model. It is understood that the concatenated feature representation with altered input dimensions needs to be input into the pre-trained target speaker extraction model to ensure its smooth operation.

[0084] Optionally, step S104 includes:

[0085] The masking of the target speaker is multiplied by the latent spatial representation of the mixed speech signal, wherein the result of the multiplication is the latent feature representation of the target speaker.

[0086] Since both the masking of the target speaker and the latent spatial representation of the mixed speech signal are represented in vector form, it is necessary to perform a dot product between the masking of the target speaker and the latent spatial representation of the mixed speech signal. That is, multiply each element in the vector correspondingly, and then accumulate the multiplications at each position to obtain the final result. In the embodiment, the result of the dot product is the latent feature representation of the target speaker.

[0087] The technical solution disclosed herein includes two independent pre-trained models: a speaker separation model and a target speaker extraction model. Both of the above pre-trained models are trained using a Mask method.

[0088] Figure 2 This is a flowchart of a method for training a speaker separation model and encoder according to an embodiment of this disclosure, as follows: Figure 2 As shown, this embodiment provides a method for training a speaker separation model and an encoder, the method comprising:

[0089] In step S201, the standard feature representation of each of the speech signals in the first test mixed speech signal is obtained in advance, wherein the first test mixed speech signal includes speech signals of multiple speakers;

[0090] In step S202, the first test mixed speech signal is input into the encoder to be trained to obtain the latent spatial representation of the first test mixed speech signal.

[0091] In step S203, the latent spatial representation of the first test mixed speech signal is input into the speaker separation model to be trained to obtain the masking of each speaker among multiple speakers;

[0092] In step S204, the latent feature representation of each speaker among the plurality of speakers is obtained based on the masking of each speaker among the plurality of speakers and the latent spatial representation of the first test mixed speech signal;

[0093] In step S205, the parameters of the speaker separation model to be trained and the parameters of the encoder to be trained are updated according to the latent feature representation and the corresponding standard feature representation of each of the plurality of speakers, so as to obtain the trained speaker separation model and the trained encoder.

[0094] First, in step S201, the standard feature representation of each speech signal in the first test mixed speech signal is pre-obtained, wherein the first test mixed speech signal includes speech signals from multiple speakers. When pre-training the speaker separation model, a first test mixed speech signal needs to be pre-configured. This first test mixed speech signal contains a sufficiently large number of speech signals from speakers, and the standard feature representation of each speaker's speech signal is known. Specifically, the standard feature representation of each speaker's speech signal is the feature representation of each speaker's speech signal in the first test mixed speech signal.

[0095] The speaker separation model uses concatenation to link multiple RNNs into a stack, and then combines the stacks into a complete network. When combining RNNs into stacks and stacks into a network, the number of RNNs and the number of stacks can be adjusted according to performance and algorithm requirements.

[0096] Further, in step S202, the first test mixed speech signal is input into the encoder to be trained to obtain the latent spatial representation of the first test mixed speech signal. The first test mixed speech signal pre-configured in step S201 is input into the encoder to be trained, and the latent spatial representation of the first test mixed speech signal is obtained through the output of the encoder to be trained.

[0097] Further, in step S203, the latent spatial representation of the first test mixed speech signal is input into the speaker separation model to be trained to obtain the mask for each speaker among multiple speakers. The latent spatial representation of the first test mixed speech signal obtained in step S202 is input into the speaker separation model to be trained, and then the mask for each speaker among multiple speakers in the first test mixed speech signal is obtained through the speaker separation model to be trained. The mask includes independent features of the speech signals of each speaker among multiple speakers in the first test mixed speech signal, and distinguishes the features of each speaker's speech signal. The features include, for example, pitch features, voiceprint features, timbre features, frequency features, etc. Compared with traditional separators, the speaker separation model in this disclosure can include more speaker speech signals and has full generalization capability. It is no longer limited to the speech signals of speakers in a closed set. Therefore, it can also be used in projects related to speaker speech signal suppression, but it can also be used for noise reduction.

[0098] Further, in step S204, the latent feature representation of each speaker is obtained based on the masking of each speaker among the plurality of speakers and the latent spatial representation of the first test mixed speech signal. This is achieved by performing a dot product between the masking of each speaker in the first test mixed speech signal obtained in step S203 and the latent spatial representation of the first test mixed speech signal obtained in step S202. Since both the masking of each speaker and the latent spatial representation of the first test mixed speech signal are represented in vector form, it is necessary to perform a dot product between them. This involves multiplying each element of the vector and then summing the products at each position to obtain the final result. The result of this dot product is the latent feature representation corresponding to each speaker in the first test mixed speech signal.

[0099] Finally, in step S205, the parameters of the speaker separation model to be trained and the parameters of the encoder to be trained are updated according to the latent feature representation and corresponding standard feature representation of each of the plurality of speakers, to obtain the trained speaker separation model and the trained encoder. Since the latent feature representation corresponding to each speaker in the first test mixed speech signal obtained by the speaker separation model to be trained is the output result of the training process before the speaker separation model is trained, and is only an accurate latent feature representation after the separation model is trained, it is necessary to compare the latent feature representation corresponding to each speaker in the first test mixed speech signal obtained by the speaker separation model to be trained with the standard feature representation of each speech signal in the first test mixed speech signal obtained in advance in step S201. According to the comparison result, the parameters of the speaker separation model to be trained and the parameters of the encoder to be trained are updated, so as to obtain the trained speaker separation model and the trained encoder.

[0100] The above step S205 includes:

[0101] Based on the standard feature representation and the corresponding latent feature representation of each of the plurality of speakers, calculate the loss value of each of the plurality of speakers, wherein the loss value is the absolute value of the difference between the standard feature representation and the latent feature representation;

[0102] The parameters of the speaker separation model and the encoder to be trained are updated using the loss values ​​of the multiple speakers until the loss value of each of the multiple speakers is less than a preset threshold. Then the training ends, and the trained speaker separation model and the trained encoder are obtained.

[0103] First, since the current pre-trained model is still a type of prediction model, it needs to be inspired by the pre-training of speech models in natural speech processing. Therefore, in this disclosure, L1 loss function is used as the training objective between the model's predicted output latent feature representation and the input standard feature representation. Then, L1 loss function is fed back into the network to correct the parameters of the speaker separation model to be trained and the parameters of the encoder to be trained.

[0104] The loss value is obtained by calculating the absolute value of the difference between each speaker in the first test mixed speech signal using a loss function. Before calculating the loss value, a preset threshold needs to be set for the loss value, and the loss value of each speaker in the first test mixed speech signal is compared with the preset threshold. Only when the loss value is less than the preset threshold is it considered that the latent feature representation of the speaker's speech signal obtained by the encoder to be trained and the speaker separation model to be trained is accurate.

[0105] Therefore, when the loss value of each speaker in the first test mixed speech signal is less than the preset threshold, it indicates that the speaker separation model and the encoder have been trained.

[0106] In practical applications, Figure 3 This is a schematic diagram of the speaker separation model and encoder provided in an embodiment of the present disclosure. Inspired by pre-training methods in natural speech processing, the speaker separation model uses an unsupervised training method, trains the separation model using an autoregressive model, and uses L1 as the loss function.

[0107] like Figure 3 As shown, the left side represents the main network of the encoder, which consists of multiple networks. In the embodiments of this disclosure, the encoder can be constructed using a certain number of convolutional neural networks, or other networks can be replaced as needed or based on performance. The right side represents the speaker separation model, which consists of multiple stacked blocks, each composed of multiple recurrent neural networks (RNNs) connected in series.

[0108] First, a mixed speech signal consisting of the time-domain speech signals of as many speakers as possible is input into the encoder on the left side. The encoder maps the mixed speech signal into a latent space, and the latent space representation of the mixed speech signal output by the encoder is obtained.

[0109] Among them, the mixed speech signal is C represents the number of speakers, S i (t) represents the speech signal of the i-th speaker;

[0110] The latent space is represented as Y = {y1, ... y2} c ,...y C Let Y represent the latent spatial representation of C speakers, y c This represents the latent spatial representation of the c-th speaker.

[0111] Next, the encoder splits the output latent spatial representation into two signals. One signal is transmitted to a speaker separation model composed of multiple recurrent neural networks (RNNs). The output of the speaker separation model is then fed into an activation function to obtain the mask w for each speaker in the mixed speech signal. i Furthermore, the masking of the mixed speech signal and the latent spatial representation of the mixed speech signal output from the other signal of the encoder are combined using the dot product formula d. i =Y⊙w i The latent feature representation d of each speaker in the mixed speech signal is obtained. i ;

[0112] Furthermore, d is represented by latent features. i The standard feature representation d0 is obtained through the loss function formula L=|d i -d0| Calculate the loss value L for each speaker, update the parameters of the encoder and the speaker separation model based on the obtained loss value L, and indicate that the encoder and speaker separation model have been trained when the loss value of all speakers is less than the preset threshold.

[0113] Figure 4 This is a flowchart of a training target speaker extraction model method provided in an embodiment of this disclosure, as follows: Figure 4 As shown in the figure, this embodiment provides a method for training a target speaker extraction model, the method comprising:

[0114] In step S401, a second test mixed speech signal is acquired in advance, and a standard feature representation of the speech signal of the target speaker is acquired, wherein the second test mixed speech signal includes the speech signals of multiple speakers, and the speech signals of the multiple speakers include the speech signal of the target speaker;

[0115] In step S402, the second test mixed speech signal is input into the pre-trained encoder to obtain the latent spatial representation of the second test mixed speech signal.

[0116] In step S403, the latent spatial representation of the second test mixed speech signal and the standard feature representation of the target speaker are input into the trained speaker separation model to obtain the masking of the target speaker;

[0117] In step S404, the latent feature representation of the target speaker is obtained based on the masking of the target speaker and the latent spatial representation of the second test mixed speech signal;

[0118] In step S405, based on the latent feature representation of the target speaker and the corresponding standard feature representation, the parameters of the speaker separation model to be trained and the parameters of the encoder to be trained are updated a second time to obtain the trained target speaker extraction model.

[0119] First, in step S401, a second test mixed speech signal is pre-acquired, and the standard feature representation of the target speaker's speech signal is obtained. The second test mixed speech signal includes speech signals from multiple speakers, including the target speaker's speech signal. During pre-training of the target speaker extraction model, a second test mixed speech signal needs to be pre-configured. This second test mixed speech signal differs from the first test mixed speech signal in that it contains a sufficiently large number of speaker signals, including the target speaker, and the target speaker's standard feature representation is known.

[0120] Further, in step S402, the second test mixed speech signal is input into the pre-trained encoder to obtain the latent spatial representation of the second test mixed speech signal. The second test mixed speech signal obtained in step S401 is input into the encoder trained in the above embodiments of this disclosure, and the latent spatial representation of the speech signals of each speaker in the second test mixed speech signal is obtained through the trained encoder.

[0121] Further, in step S403, the latent spatial representation of the second test mixed speech signal and the standard feature representation of the target speaker are input into the trained speaker separation model to obtain the masking of the target speaker. The latent spatial representations of the speech signals of each speaker in the second test mixed speech signal obtained in step S402 and the standard feature representation of the target speaker obtained in step S401 are input into the trained speaker separation model in the above embodiments of this disclosure to obtain the masking of the target speaker.

[0122] In other embodiments, before inputting the latent spatial representations of the speech signals of each speaker in the second test mixed speech signal obtained in step S402 and the standard feature representation of the target speaker obtained in step S401 into the speaker separation model trained in the above embodiments of this disclosure, it is necessary to concatenate the latent spatial representation of the second test mixed speech signal with the standard feature representation of the target speaker to obtain a concatenated feature representation. Then, the concatenated feature representation of the latent spatial representation of the second test mixed speech signal and the standard feature representation of the target speaker is modified by a linear layer. The concatenated feature representation with the changed input dimension is then input into the trained speaker separation model so that the pre-trained speaker separation model can run smoothly.

[0123] Further, in step S404, the latent feature representation of the target speaker is obtained based on the masking of the target speaker and the latent spatial representation of the second test mixed speech signal. Since both the masking of the target speaker and the latent spatial representation of the second test mixed speech signal are represented in vector form, it is necessary to perform a dot product between the masking of the target speaker and the latent spatial representation of the second test mixed speech signal. That is, multiply each element in the vector correspondingly, and then sum the multiplications at each position to obtain the final result. In this embodiment, the result of the dot product is the latent feature representation of the target speaker.

[0124] Finally, in step S405, based on the latent feature representation of the target speaker and the corresponding standard feature representation, the parameters of the speaker separation model to be trained and the parameters of the encoder to be trained are updated a second time to obtain the trained target speaker extraction model. The obtained latent feature representation of the target speaker is compared with the standard feature representation of the target speaker, and the parameters of the speaker separation model to be trained and the encoder to be trained are updated a second time based on the comparison results. The speaker separation model to be trained and the encoder to be trained are the trained speaker separation model and the trained encoder in the above embodiments of this disclosure. Here, a second update is needed based on the comparison results to ensure the training effect of the model.

[0125] Step S405 further includes:

[0126] Based on the standard feature representation and latent feature representation of the target speaker, the loss value of the target speaker is calculated, wherein the loss value is the absolute value of the difference between the standard feature representation and latent feature representation of the target speaker;

[0127] The parameters of the speaker separation model to be trained and the parameters of the encoder to be trained are updated twice using the loss value of the target speaker until the loss value of the target speaker is less than a preset threshold. Then the training ends and the trained target speaker extraction model is obtained.

[0128] Before updating the parameters of the speaker separation model and the encoder to be trained, it is necessary to calculate the loss value of the target speaker. The calculation method of the loss value has been explained above and will not be repeated here. The loss value of the target speaker is used to update the parameters of the speaker separation model and the encoder to be trained. The loss value of the target speaker is calculated repeatedly until the loss value of the target speaker is less than the preset threshold. This indicates that the target speaker extraction model has been trained. At this time, the training ends and the trained target speaker extraction model is obtained.

[0129] In practical applications, Figure 5 This is a schematic diagram of the framework of a target speaker extraction model provided in an embodiment of the present disclosure. Inspired by pre-training methods in natural speech processing, the target speaker extraction model uses an unsupervised training method, employs an autoregressive model to train the target speaker extraction model, and uses L1 as the loss function.

[0130] like Figure 5 As shown, the left side represents the main network of the encoder, which consists of multiple networks. In the embodiments of this disclosure, the encoder can be constructed using a certain number of convolutional neural networks, or other networks can be replaced as needed or based on performance. The right side represents the speaker separation model, which consists of multiple stacked blocks, each composed of multiple recurrent neural networks (RNNs) connected in series. The LNs are linear layers used to change the input dimension of the feature representation.

[0131] First, a mixed speech signal consisting of the time-domain speech signals of as many speakers as possible is input into the encoder on the left side. The encoder maps the mixed speech signal into a latent space, and the latent space representation of the mixed speech signal output by the encoder is obtained.

[0132] Next, the encoder splits the output latent space representation into two signals. One signal is transmitted to a speaker separation model composed of multiple recurrent neural networks (RNNs), where it is concatenated with the standard feature representation of the target speaker to obtain a concatenated feature representation. This concatenated feature representation is then input into a linear layer (LN) to change the input dimension of the concatenated feature representation. Finally, the speaker separation model trained in the above embodiment is output to the activation function to obtain the mask w of the target speaker. j And to cover up the target speaker's identity. jThe latent spatial representation of the mixed speech signal with the other signal output of the encoder is obtained through the dot product formula d. j =Y⊙w j The latent feature representation d of the target speaker in the mixed speech signal is obtained. j ;

[0133] Furthermore, by representing the latent features of the target speaker d j The standard feature representation d0 of the target speaker is compared with the loss function formula L=|d j -d0|Calculate the loss value L of the target speaker. Based on the obtained loss value L, update the parameters of the encoder and the parameters of the speaker separation model. When the loss value of the target speaker is less than the preset threshold, it means that the target speaker extraction model has been trained.

[0134] Based on the same inventive concept, another embodiment of this disclosure provides a schematic diagram of the framework of a speech signal extraction device, as shown below. Figure 6 As shown in the figure, this embodiment provides a speech signal extraction device, the device comprising:

[0135] The first input module 11 is used to input the mixed speech signal into a pre-trained encoder to obtain the latent spatial representation of the mixed speech signal, wherein the mixed speech signal includes the speech signals of at least two speakers, and the target speaker is among the at least two speakers;

[0136] The first acquisition module 12 is used to acquire the auxiliary feature representation of the target speaker;

[0137] The second input module 13 is used to input the latent spatial representation of the mixed speech signal and the auxiliary feature representation of the target speaker into the pre-trained target speaker extraction model to obtain the masking of the target speaker;

[0138] The first calculation module 14 is used to obtain the latent feature representation of the target speaker based on the masking of the target speaker and the latent spatial representation of the mixed speech signal;

[0139] The first decoding module 15 is used to decode the latent spatial representation of the target speaker through a decoder to obtain the speech signal of the target speaker.

[0140] Optionally, the first acquisition module 11 further includes:

[0141] The first acquisition unit is used to acquire the original speech signal of the target speaker;

[0142] The first operation unit is used to input the original speech signal of the target speaker into the pre-trained encoder to obtain the auxiliary feature representation of the target speaker.

[0143] Optionally, the device further includes:

[0144] The first splicing unit is used to splice the latent spatial representation of the mixed speech signal and the auxiliary feature representation of the target speaker before inputting the latent spatial representation of the mixed speech signal and the auxiliary feature representation of the target speaker into the pre-trained target speaker extraction model, so as to obtain the spliced ​​feature representation.

[0145] The first modification unit is used to modify the input dimension of the concatenated feature representation through a linear layer;

[0146] The second operation unit is used to input the latent spatial representation of the mixed speech signal and the auxiliary feature representation of the target speaker into the pre-trained target speaker extraction model.

[0147] Optionally, the second operation unit includes:

[0148] The second operation subunit is used to input the spliced ​​feature representation after the input dimension has been changed into the pre-trained target speaker extraction model.

[0149] Optionally, the first computing module 14 further includes:

[0150] The first computing unit is configured to perform a dot product between the masking of the target speaker and the latent spatial representation of the mixed speech signal, wherein the result of the dot product is the latent feature representation of the target speaker.

[0151] Optionally, the device further includes:

[0152] The second acquisition unit is used to acquire in advance the standard feature representation of each of the speech signals in the first test mixed speech signal, wherein the first test mixed speech signal includes speech signals of multiple speakers;

[0153] The third operation unit is used to input the first test mixed speech signal into the encoder to be trained to obtain the latent spatial representation of the first test mixed speech signal.

[0154] The fourth operation unit is used to input the latent spatial representation of the first test mixed speech signal into the speaker separation model to be trained, so as to obtain the mask of each speaker among the multiple speakers;

[0155] The fifth operation unit is used to obtain the latent feature representation of each of the plurality of speakers based on the masking of each speaker among the plurality of speakers and the latent spatial representation of the first test mixed speech signal;

[0156] The first update unit is used to update the parameters of the speaker separation model to be trained and the parameters of the encoder to be trained based on the latent feature representation and the corresponding standard feature representation of each of the plurality of speakers, so as to obtain the trained speaker separation model and the trained encoder.

[0157] Optionally, the first update unit further includes:

[0158] The second calculation unit is used to calculate the loss value of each speaker among the plurality of speakers based on the standard feature representation and the corresponding latent feature representation of each speaker among the plurality of speakers, wherein the loss value is the absolute value of the difference between the standard feature representation and the latent feature representation;

[0159] The first update subunit is used to update the parameters of the speaker separation model to be trained and the parameters of the encoder to be trained using the loss values ​​of the multiple speakers, until the loss value of each of the multiple speakers is less than a preset threshold, then the training ends and the trained speaker separation model and the trained encoder are obtained.

[0160] Optionally, the device further includes:

[0161] The third acquisition unit is used to acquire the second test mixed speech signal in advance and acquire the standard feature representation of the speech signal of the target speaker, wherein the second test mixed speech signal includes the speech signals of multiple speakers, and the speech signals of the multiple speakers include the speech signal of the target speaker;

[0162] The sixth operation unit is used to input the second test mixed speech signal into the pre-trained encoder to obtain the latent spatial representation of the second test mixed speech signal;

[0163] The seventh operation unit is used to input the latent spatial representation of the second test mixed speech signal and the standard feature representation of the target speaker into the trained speaker separation model to obtain the masking of the target speaker;

[0164] The eighth operation unit is used to obtain the latent feature representation of the target speaker based on the masking of the target speaker and the latent spatial representation of the second test mixed speech signal;

[0165] The second update unit is used to update the parameters of the speaker separation model to be trained and the parameters of the encoder to be trained a second time based on the latent feature representation of the target speaker and the corresponding standard feature representation, so as to obtain the trained target speaker extraction model.

[0166] Optionally, the second update unit further includes:

[0167] The third calculation unit is used to calculate the loss value of the target speaker based on the standard feature representation and the latent feature representation of the target speaker, wherein the loss value is the absolute value of the difference between the standard feature representation and the latent feature representation of the target speaker;

[0168] The second update subunit is used to update the parameters of the speaker separation model to be trained and the parameters of the encoder to be trained using the loss value of the target speaker, until the loss value of the target speaker is less than a preset threshold, then the training ends and the trained target speaker extraction model is obtained.

[0169] Based on the same inventive concept, another embodiment of this disclosure also provides an apparatus including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement any of the methods described above.

[0170] Based on the same inventive concept, another embodiment of this disclosure also provides a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements any of the methods described above.

[0171] As the apparatus is basically similar to the method embodiment, it is described in a relatively simple way. For relevant details, please refer to the description of the method embodiment.

[0172] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0173] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0174] This disclosure describes embodiments of methods, terminal devices (systems), and computer program products according to embodiments of this disclosure with reference to flowchart illustrations and / or block diagrams. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0175] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0176] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0177] While preferred embodiments of the present disclosure have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the present disclosure.

[0178] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0179] The above provides a detailed description of a speech signal extraction method, apparatus, device, and computer-readable medium provided by this disclosure. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this disclosure. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims

1. A method for extracting speech signals, characterized in that, The method includes: The mixed speech signal is input into a pre-trained encoder to obtain the latent spatial representation of the mixed speech signal, wherein the mixed speech signal includes the speech signals of at least two speakers, including the target speaker; Obtain the auxiliary feature representation of the target speaker; The latent spatial representation of the mixed speech signal and the auxiliary feature representation of the target speaker are input into the pre-trained target speaker extraction model to obtain the masking of the target speaker; Based on the masking of the target speaker and the latent spatial representation of the mixed speech signal, the latent feature representation of the target speaker is obtained; The latent spatial representation of the target speaker is decoded by a decoder to obtain the speech signal of the target speaker.

2. The extraction method according to claim 1, characterized in that, The acquisition of the auxiliary feature representation of the target speaker includes: Acquire the original speech signal of the target speaker; The original speech signal of the target speaker is input into the pre-trained encoder to obtain the auxiliary feature representation of the target speaker.

3. The extraction method according to claim 1, characterized in that, Before inputting the latent spatial representation of the mixed speech signal and the auxiliary feature representation of the target speaker into the pre-trained target speaker extraction model, the method further includes: The latent spatial representation of the mixed speech signal is concatenated with the auxiliary feature representation of the target speaker to obtain the concatenated feature representation. The input dimension of the concatenated feature representation is changed by using a linear layer; The latent spatial representation of the mixed speech signal and the auxiliary feature representation of the target speaker are input into a pre-trained target speaker extraction model, including: The concatenated feature representation after the input dimension is changed is input into the pre-trained target speaker extraction model.

4. The extraction method according to claim 1, characterized in that, The step of obtaining the latent feature representation of the target speaker based on the masking of the target speaker and the latent spatial representation of the mixed speech signal includes: The masking of the target speaker is multiplied by the latent spatial representation of the mixed speech signal, wherein the result of the multiplication is the latent feature representation of the target speaker.

5. The extraction method according to claim 1, characterized in that, The trained target speaker extraction model includes a trained encoder and a trained speaker separation model, which are trained through the following steps: Standard feature representations of each of the speech signals in the first test mixed speech signal are obtained in advance, wherein the first test mixed speech signal includes speech signals of multiple speakers; The first test mixed speech signal is input into the encoder to be trained to obtain the latent spatial representation of the first test mixed speech signal. The latent spatial representation of the first test mixed speech signal is input into the speaker separation model to be trained to obtain the masking of each speaker among multiple speakers; Based on the masking of each speaker among the plurality of speakers and the latent spatial representation of the first test mixed speech signal, the latent feature representation of each speaker among the plurality of speakers is obtained; Based on the latent feature representation and corresponding standard feature representation of each of the multiple speakers, the parameters of the speaker separation model to be trained and the parameters of the encoder to be trained are updated to obtain the trained speaker separation model and the trained encoder.

6. The extraction method according to claim 5, characterized in that, The step of updating the parameters of the speaker separation model to be trained and the parameters of the encoder to be trained based on the latent feature representation and corresponding standard feature representation of each of the plurality of speakers, to obtain the trained speaker separation model and the trained encoder, includes: Based on the standard feature representation and the corresponding latent feature representation of each of the plurality of speakers, calculate the loss value of each of the plurality of speakers, wherein the loss value is the absolute value of the difference between the standard feature representation and the latent feature representation; The parameters of the speaker separation model and the encoder to be trained are updated using the loss values ​​of the multiple speakers until the loss value of each of the multiple speakers is less than a preset threshold. Then the training ends, and the trained speaker separation model and the trained encoder are obtained.

7. The extraction method according to claim 6, characterized in that, After the speaker separation model and encoder have been trained, the method further includes: A second test mixed speech signal is pre-acquired, and a standard feature representation of the speech signal of the target speaker is obtained, wherein the second test mixed speech signal includes the speech signals of multiple speakers, and the speech signals of the multiple speakers include the speech signal of the target speaker; The second test mixed speech signal is input into the pre-trained encoder to obtain the latent spatial representation of the second test mixed speech signal; The latent spatial representation of the second test mixed speech signal and the standard feature representation of the target speaker are input into the trained speaker separation model to obtain the masking of the target speaker; Based on the masking of the target speaker and the latent spatial representation of the second test mixed speech signal, the latent feature representation of the target speaker is obtained; Based on the latent feature representation of the target speaker and the corresponding standard feature representation, the parameters of the speaker separation model to be trained and the parameters of the encoder to be trained are updated a second time to obtain the trained target speaker extraction model.

8. The extraction method according to claim 7, characterized in that, The step of updating the parameters of the speaker separation model to be trained and the parameters of the encoder to be trained based on the latent feature representation and corresponding standard feature representation of the target speaker to obtain the trained target speaker extraction model includes: Based on the standard feature representation and latent feature representation of the target speaker, the loss value of the target speaker is calculated, wherein the loss value is the absolute value of the difference between the standard feature representation and latent feature representation of the target speaker; The parameters of the speaker separation model to be trained and the parameters of the encoder to be trained are updated twice using the loss value of the target speaker until the loss value of the target speaker is less than a preset threshold. Then the training ends and the trained target speaker extraction model is obtained.

9. A speech signal extraction device, characterized in that, The device includes: The first input module is used to input the mixed speech signal into a pre-trained encoder to obtain the latent spatial representation of the mixed speech signal, wherein the mixed speech signal includes the speech signals of at least two speakers, and the target speaker is among the at least two speakers; The first acquisition module is used to acquire auxiliary feature representations of the target speaker; The second input module is used to input the latent spatial representation of the mixed speech signal and the auxiliary feature representation of the target speaker into the pre-trained target speaker extraction model to obtain the masking of the target speaker; The first calculation module is used to obtain the latent feature representation of the target speaker based on the masking of the target speaker and the latent spatial representation of the mixed speech signal; The first decoding module is used to decode the latent spatial representation of the target speaker through a decoder to obtain the speech signal of the target speaker.

10. A device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method as claimed in any one of claims 1-8.

11. A computer-readable medium, characterized in that, It stores computer programs, among which, When the program is executed by the processor, it implements the method as described in any one of claims 1-8.

Images