Target speaker extraction method and device based on historical dialogue text, medium and product

Abstract

The invention provides a target speaker extraction method and device based on a historical dialogue text, a medium and a product, and the method comprises the steps: processing a mixed voice signal through a voice encoder, and obtaining frequency spectrum feature embedding; wherein the mixed voice signal comprises a voice signal of a target speaker, a voice signal of an interference speaker and a noise signal; processing the text prompt information through a text encoder to obtain text feature embedding; the text prompt information is associated with text content corresponding to the voice signal of the target speaker; fusing spectrum feature embedding and text feature embedding through a fusion layer to obtain fusion features; processing the fusion feature through a mask estimator to obtain a target mask corresponding to the fusion feature; and extracting a voice signal of the target speaker according to the target mask and the mixed voice signal. The recognition effect of the target speaker can be improved, voice does not need to be registered in advance, and implementation is convenient.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to a method, device, medium and product for extracting a target speaker based on historical dialogue text. Background Art

[0002] In complex multi-speaker scenarios, extracting the target speaker’s speech remains a huge challenge due to heavy speaker overlap and significant background noise. This is particularly important in dialogue-based applications, such as AI voice assistants, where accurate understanding of human intent is crucial.

[0003] However, existing deep learning-based speech enhancement (SE) and speech separation (SS) methods perform poorly when dealing with complex acoustic scenes, especially in distinguishing similar sound sources, maintaining speech integrity, and solving the identity permutation problem.

[0004] Traditional speech enhancement methods such as spectral subtraction and Wiener filtering have difficulty dealing with rapidly changing noise, resulting in speech distortion. Recent deep learning methods such as MP-SENet and FRCRN perform well in denoising, but do not handle interfering speakers. To cope with multi-speaker scenarios, speech separation methods include TF-GridNet, Conv-TasNet, DPRNN, DPTNet, SPGM, Sep-Former, and MossFormer2. However, these methods require the number of speakers to be known in advance, and it is difficult to accurately associate the separated signals with the speaker identities, i.e., the ID permutation problem.

[0005] Although Permutation Invariant Training (PIT) was originally proposed to solve the identity permutation problem in speech separation tasks, it still cannot accurately identify which separated speech corresponds to the target speaker. As an alternative, Target Speaker Extraction (TSE) uses auxiliary information to extract the target speech from mixed and noisy speech, thereby overcoming the limitations of speech enhancement and speech separation methods.

[0006] Target speaker extraction methods aim to extract the target speaker's speech without prior knowledge. However, since pre-registered speech is not available, this method cannot be well applied in conversation-based AI voice assistants. Summary of the invention

[0007] The first object of the present invention is to provide a target speaker extraction method based on historical conversation text with better performance and more convenient implementation.

[0008] The second object of the present invention is to provide a computer device for implementing the above-mentioned target speaker extraction method based on historical conversation text.

[0009] The third object of the present invention is to provide a computer-readable storage medium for implementing the above-mentioned target speaker extraction method based on historical conversation text.

[0010] A fourth object of the present invention is to provide a computer program product for implementing the above-mentioned method for extracting a target speaker based on historical conversation text.

[0011] In order to achieve the above-mentioned first purpose, the present invention provides a method for extracting a target speaker based on historical conversation text, which includes the following steps: processing a mixed speech signal through a speech encoder to obtain a spectral feature embedding; wherein the mixed speech signal includes a speech signal of a target speaker, a speech signal of an interfering speaker, and a noise signal; processing text prompt information through a text encoder to obtain a text feature embedding; matching the text content corresponding to the speech signal of the target speaker with the text prompt information; fusing the spectral feature embedding and the text feature embedding through a fusion layer to obtain a fused feature; processing the fused feature through a mask estimator to obtain a target mask corresponding to the fused feature; extracting the speech signal of the target speaker according to the target mask and the mixed speech signal.

[0012] As can be seen from the above scheme, the present invention processes speech dialogue scenarios with overlapping speech and strong background noise, uses historical dialogue text as clues, explores the semantic association between questions and human answers, and uses prior knowledge of contextual prompts as clues to effectively extract the speech of the target speaker. Since no speech or pre-registration is required, it is more convenient to implement and has a wider range of application scenarios than existing solutions.

[0013] A further solution is that the mask estimator includes multiple DPRNN modules. When processing the fused features through the mask estimator, it includes: dividing the fused features into segments, connecting all the segments to form blocks, the blocks are processed in turn by multiple DPRNN modules and multiple echo prompt blocks of the echo prompt module, and finally reconstructing the blocks through the aggregation layer, and generating the target mask through the activation layer.

[0014] It can be seen that by designing the echo prompt module, low signal-to-noise ratio noise can be effectively removed at multiple granularity levels to improve the recognition effect.

[0015] A further solution is that the number of echo prompt blocks is 3.

[0016] It can be seen that when the number of echo prompt blocks is 3, a better recognition effect can be obtained.

[0017] A further solution is that when the mixed speech signal is processed by a speech encoder to obtain the spectral feature embedding, the method includes: processing the mixed speech signal by a one-dimensional convolution layer, and then applying a ReLU layer to obtain the spectral feature embedding.

[0018] A further solution is that when processing the text prompt information through a text encoder to obtain text feature embedding, the method includes: generating text feature embedding corresponding to the text prompt information by comparing a language audio pre-training model.

[0019] It can be seen that the text feature embedding corresponding to the text prompt information can be obtained more accurately.

[0020] A further solution is that the fusion layer, the mask estimator and the echo prompt module are obtained through a deep learning training method, including: setting a data set, the data set includes multiple voice segments and multiple noise segments, each of the voice segments is provided with a corresponding unique matching text generation problem, and during training, two of the voice segments and one of the noise segments are randomly selected and mixed using a random SNR mixing method.

[0021] A further solution is that the speech segment and most noise segments are sampled to 8 kHz.

[0022] This shows that it can be more suitable for the use scenarios of artificial intelligence voice assistants.

[0023] In order to achieve the above-mentioned second purpose, the present invention provides a computer device, including a processor and a memory, wherein: a computer program is stored in the memory, and when the computer program is executed by the processor, the above-mentioned target speaker extraction method based on historical conversation text is implemented.

[0024] In order to achieve the third objective mentioned above, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, the target speaker extraction method based on historical conversation text is implemented.

[0025] In order to achieve the fourth objective described above, a computer program product is provided, comprising computer instructions, wherein: when the computer instructions are executed by a processor, the target speaker extraction method based on historical conversation text is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the application of the present invention in an artificial intelligence voice assistant scenario.

[0027] Figure 2It is a flowchart of the first embodiment of the target speaker extraction method based on historical conversation text of the present invention.

[0028] Figure 3 It is a structural diagram of the TPEech model in the first embodiment of the target speaker extraction method based on historical conversation text of the present invention.

[0029] Figure 4 This is a comparison chart of scores of different indicators between the TPEech model and other models in the first embodiment of the target speaker extraction method based on historical conversation text of the present invention.

[0030] Figure 5 It is a schematic diagram of the results of the ablation experiment on the TPEech model in the first embodiment of the target speaker extraction method based on historical conversation text of the present invention.

[0031] Figure 6 It is a schematic diagram of the normalized metric difference of the TPEech model under different signal-to-noise ratios in the first embodiment of the target speaker extraction method based on historical conversation text of the present invention.

[0032] The present invention is further described below in conjunction with the accompanying drawings and embodiments. DETAILED DESCRIPTION

[0033] The target speaker extraction method based on historical conversation text of the present invention uses historical conversation text as a clue to extract the speech signal of the target speaker from the mixed speech signal in a scene with multiple speakers and obvious background noise. The present invention also provides a computer device, a computer-readable storage medium and a computer program product for implementing the target speaker extraction method based on historical conversation text.

[0034] See also Figure 1 , Figure 1The application of the present invention in the application scenario of artificial intelligence voice assistant is shown. In this application scenario, the user interacts with the AI ​​voice assistant. At this time, the AI ​​voice assistant asks the question "What actions did he take after these differences occurred?" The user answers according to the question "He happened to leave his name..." The AI ​​voice assistant needs to receive the content of the user's answer for further feedback. However, since the user is in a noisy environment, the AI ​​voice assistant will receive a mixed voice signal including a noise signal and a voice signal of an interfering speaker, and cannot extract the user's answer from the received answer, that is, cannot extract the voice signal of the target speaker. The present invention uses the question "What actions did he take after these differences occurred?" raised by the AI ​​voice assistant as text prompt information to extract the target speaker. Since the text content corresponding to the question raised by the AI ​​voice assistant and the voice signal of the target speaker are related, while the noise and the voice signal of the interfering speaker are not related to the question raised by the AI ​​voice assistant, the text prompt information can be used as a clue to extract the voice signal of the target speaker from the mixed voice signal, so that the AI ​​voice assistant can answer based on the voice signal of the target speaker extracted by the present invention, so that the AI ​​voice assistant can still be used reliably in a noisy environment.

[0035] First embodiment of the target speaker extraction method based on historical conversation text: Let s(τ) denote the speech signal of the target speaker, v i (τ) represents the interfering speech signal of the i-th interfering speaker, n(τ) is the noise signal, i∈{1,2,...,I}, I is the total number of interfering speakers, τ ∈ {1, 2, ..., T}, τ represents the time index, then the mixed speech signal x(τ) can be expressed as: , where α and β are used to adjust the interfering speech signal v i A scalar variable that represents the signal-to-noise ratio (SNR) of the noise signal n(τ).

[0036] This embodiment extracts the target speaker's speech signal s(τ) from the mixed speech signal x(τ) by executing a computer program. Figure 2 , specifically including the following steps: S1: Obtain mixed voice signal and text prompt information.

[0037] Among them, the text prompt information is the question asked by the currently interacting AI voice assistant, such as Figure 1 In the question “what actions did he take after these disagreements?”, the mixed speech signal x(τ) is similar to Figure 1 The waveform after the noise signal, the interfering speaker's speech signal, and the target speaker's speech signal are superimposed.

[0038] S2: Process the mixed speech signal and text prompt information to obtain the speech signal of the target speaker.

[0039] The text content corresponding to the target speaker's voice signal in the mixed voice signal is associated with the text prompt information. "Association" means that the text content corresponding to the target speaker's voice signal is directly given for the text prompt information. For example, Figure 1 The text content corresponding to the speech signal of the target speaker in “He happened to leave his name…” is given in response to the text prompt information “What actions did he take after these disagreements occurred”.

[0040] S3: Output the speech signal of the target speaker.

[0041] The above steps S1 to S3 are implemented based on a parameter trainable model, which is hereinafter referred to as the TPEech model.

[0042] See also Figure 3 The TPEech model includes a speech encoder 11, a text encoder 12, a fusion layer 13, a mask estimator 14, an echo prompt module 15 and a speech decoder 16.

[0043] The speech encoder 11 is used to convert the mixed speech signal x(τ) into a spectral feature embedding X(t). The speech encoder 11 processes the mixed speech signal x(τ) through a one-dimensional convolution layer Conv1D, and then applies the activation function of the ReLU layer to obtain the spectral feature embedding X(t). The mixed speech signal x(τ) is converted into a spectral feature embedding X(t) in the latent space: ,in, Speech represents a sequence of operations applied to the input. A 1D convolution is configured with input and output dimensions of 1 and D respectively. It uses a kernel of size L to capture local patterns in the data, and a stride of L / 2 allows for overlap in receptive fields.

[0044] The text encoder 12 is used to convert the text prompt information pt into a text feature embedding P text In this embodiment, the text encoder 12 generates a text feature embedding P corresponding to the speech by using a comparative language audio pre-training (CLAP) model. text : , in, text (·) represents the text encoder, P textis a D-dimensional text embedding. Among them, the contrastive language audio pre-training (CLAP) model does not participate in the training of the TPEech model, that is, all parameters of the CLAP model are fixed. Figure 3 The snowflake symbol indicates that all parameters of the model are frozen.

[0045] In some embodiments, the text prompt information pt is not presented in the form of voice, but directly in the form of text, so there is no need to convert the text prompt information pt into text feature embedding P through the CLAP model. text .

[0046] The fusion layer 13 is used to embed the spectral features into X(t) and the text features into P text The fused features F(t) are generated by merging. Layer normalization (LN) is applied to the spectral feature embedding X(t) to stabilize and normalize the speech embedding. In particular, the linear modulation (FiLM) layer modulation combines text and speech features, where the text feature embedding P text The normalized spectral feature embedding X(t) is modulated by scaling and shifting. Since the text feature embedding P text The dimension of the spectral feature embedding X(t) is 512, the dimension of the spectral feature embedding X(t) is 256, and the text feature embedding P is resampled through the re-sample layer. text After resampling, the final fusion feature F(t) is expressed as: ,γ(P text ) and β(P text ) is a learning parameter, which is embedded in P according to the text features text The spectral feature embedding X(t) is scaled and offset, where ⊙ represents element-wise multiplication. This process can be repeated multiple times to more deeply integrate text and speech, so that the TPEech model can better incorporate text information into speech. In this embodiment, Figure 3 The “×2” at the fusion layer 13 means it is repeated twice.

[0047] The mask estimator 14 is used to obtain the target mask M (t) corresponding to the fused feature F (t). The target mask M (t) is a mask used to filter out irrelevant noise and interference in the spectral feature embedding X (t). First, the fused feature F (t) passes through a one-dimensional convolution layer (Conv1D), and then is divided into segments with a window size of K and a jump size of K / 2 by the segmentation layer (Segmention). In order to minimize the total input length, K≈2T is set. seq, and then all segments are connected to form a block W. The DPRNN (Dual-Path Recurrent Neural Network) module is repeated R times, processing the block W through bidirectional and cross-sequence, and introducing a residual connection mechanism, namely the echo prompt module15. The output of each DPRNN module is called the dual-path feature, denoted by E dp Finally, the aggregation layer reconstructs the block W back to the shape of F (t), and the subsequent activation layer (PReLU) generates the target mask M (t).

[0048] The echo prompt module 15 includes a plurality of echo prompt blocks ECB, which are used to connect the mask estimator 14 to enhance the performance of target speaker extraction. By setting the echo prompt module 15, the text feature embedding P text The hierarchical integration of the spectral feature embedding X(t) enhances the output of each DPRNN block. Given that the outputs of different DPRNN blocks represent features of different granularities, this design improves the noise suppression capability. text And the spectral feature embedding X(t) is output to each echo prompt block ECB respectively, that is, Figure 3 The blue dashed line in the middle indicates that each echo prompt block ECB receives the spectral feature embedding X(t), and the red dashed line indicates that each echo prompt block ECB receives the text feature embedding P text , each echo prompt ECB receives the text feature embedding P text The sum spectral feature embedding X(t) is independent of other echo cue blocks ECB.

[0049] The echo prompt block ECB contains a cross attention layer. In this layer, the query (Q ech ), key (K ech ) and value (V ech ) matrix is ​​embedded by combining text features text and spectral features are embedded into X(t), and are represented by their respective weight matrices W s and W t This cross-attention mechanism focuses on preserving the semantic content of the spectral feature embedding X(t), making it very suitable for filtering out semantically irrelevant noise. Dual-path feature E in mask estimator 14 dp It is enhanced by residual connections, named H, where the attention weights are calculated by applying a softmax function to the scaled dot product of the query matrix and the key matrix. The overall design can be expressed as: , Among them, W s Represents text feature embedding P text The weight matrix, Wt represents the weight matrix of the spectral feature embedding X(t), and d is in the dimensions of the query, key, and value matrices.

[0050] The speech decoder 16 is used to reconstruct the speech signal s(τ) of the target speaker according to the target mask M(t). The speech decoder 16 outputs the masked speech embedding by performing element-by-element multiplication between the target mask M(t) and the spectral feature embedding X(t) output. : , Then, the linear layer maps the masked features from high dimensions back to 1 dimension, which facilitates the subsequent OnA layer operation. By performing an overlap-add operation (OnA), the masked speech can be embedded Convert the target speaker’s speech signal s(τ).

[0051] In different embodiments, the above-mentioned TPEech model can be integrated into the artificial intelligence voice intelligent assistant as a part of the artificial intelligence voice intelligent assistant, or it can be a part independent of the artificial intelligence voice intelligent assistant and perform data transmission with the artificial intelligence voice intelligent assistant.

[0052] The training process of the TPEech model: In terms of data set setting, we use the self-designed MNSpeech data set, which includes 272.28 hours of audio data from 417 speakers. The audio data is composed of the LibriHeavy-small data set, in which each speech segment is padded or cropped to 8 seconds in length.

[0053] Automatic speech recognition (ASR) is performed through the Whisper-large-v3 model to obtain the corresponding text for each speech segment. Then, the Mistral-7BInstruct-v0.2 model is used to generate questions based on the text generated by ASR. The answers to the questions generated by the Mistral-7B-Instructions-v0.2 model correspond to the text obtained from the ASR results. In order to better adapt to the real scene, all speech segments are sampled to 8kHz.

[0054] See also Figure 1 , the question is presented in the form of a text prompt "What action did he take after these disagreements?", which is based on the ASR result of the target speech "He happened to leave his name..." Therefore, each speech segment corresponds to a unique matching text.

[0055] The noise components in the dataset come from six datasets: MS-SNSD dataset, MUSAN dataset, Nonspeech dataset, QUT-NOISE dataset, UrbanSound 8k dataset, and WHAM! dataset. All noise segments are resampled to a sampling rate of 8 kHz to match the speech segments. In order to generate overlapping speech, two samples are randomly selected from the MNSpeech dataset and mixed using a random SNR mixing method with an SNR range of -5 to 10 dB. Then, a noise sample is added with an SNR value also selected from the same range. This mixing process is generated dynamically during the training process.

[0056] The dataset is divided into training set, validation set and test set, containing 200,000, 5,000 and 1,000 samples respectively. There is no overlap of speaker IDs between the sets, that is, the training set, validation set and test set do not contain the voices of repeated individuals.

[0057] For the implementation, the original DPRNN module design was used, with the input channels set to 256, the output channels set to 64, and the hidden channels set to 128, containing a total of six modules. Instead of the original core size of 2, a core size of 40 was used. Based on the discussion in the ablation study section, we empirically selected 3 echo cue modules.

[0058] For evaluation and training indicators, signal-to-noise ratio (SDR), perceptual evaluation of speech quality (PESQ), and short-term target intelligibility (STOI) are used as indicators. For each indicator, a higher value indicates better performance (↑). The Adam optimizer is used, and the initial learning rate is 5×10 −4 .

[0059] The negative inverse scaled signal-distortion ratio improvement (SI-SDRi) is used as the loss function to evaluate the signal quality of the extracted target speaker's speech signal. SI-SDRi is an improved version of the inverse scaled signal-distortion ratio (SI-SDR): , , For brevity, (τ) is omitted in the loss function.

[0060] See also Figure 4 , Figure 4The figure shows how the TPEech model compares to various SE models in terms of scores on different metrics. SE models such as the MP-SENet model and the FRCRN model demonstrate the ability to reduce background noise and partially suppress non-speaker voices, achieving scores of 2.076 dB and 2.135 dB for the SI-SDRi metric, respectively. However, their performance drops significantly when the target speaker is not the dominant speaker, especially in low SNR mixed conditions. On the other hand, the SGMSE model performs poorly when dealing with multi-speaker scenarios, achieving only a score of 1.721 dB for the SI-SDRi metric, which is slightly worse than the above models.

[0061] Figure 4 Multiple speech separation methods were also evaluated. In this context, PIT association involves manually selecting the optimal speech output based on the SI-SDR metric calculation to align with the target speaker's speech. Models like the DPTNet model and the SPGM model performed poorly in effectively separating speech in noisy environments, achieving scores of 3.554 dB and 4.323 dB for the SI-SDRi metric, respectively. On the other hand, the SepFormer model and the MossFormer2 model performed well in handling noisy speech. However, during the speech separation process, while the first output can effectively remove noise, the second output often retains residual noise. When using PIT association, the target speech may correspond to the second output, resulting in a decrease in the overall average performance. This problem limits their effectiveness, as can be seen from their relatively low SI-SDRi metric scores of 6.114 dB and 6.292 dB. Figure 4 The TPEech model of this embodiment is also compared with another TSE model AudioSep-CLAP, which is fine-tuned using the MNSpeech dataset of this embodiment. The AudioSep-CLAP model is also an audio extraction model that uses text as clues. However, it focuses more on audio rather than speech, which makes it less effective in accurately removing background noise and extracting the speech of the target speaker. Therefore, the AudioSep-CLAP model is inferior to the TPEech model of this embodiment in performance, scoring 7.969 dB lower on the SI-SDRi indicator.

[0062] See also Figure 5 , Figure 5The ablation experiment is shown. The ablation experiment reveals the necessity of the number of echo cue blocks and text cue embedding, and the results include: (1) When using 6 echo cue blocks, the performance of the TPEech model is slightly reduced compared to the full model, with a decrease of 0.272 dB in the SI-SDRi index score. (2) When using 2 echo cue blocks, the performance decreases more significantly, with a decrease of 1.09 dB in the SI-SDRi score. (3) When the echo cue block is not used, the performance decreases significantly, with a decrease of 2.961 dB in the SI-SDRi score. When using DPRNN (PIT association), TPEech outperforms DPRNN itself when using 3 echo cue blocks (i.e., the full model), with an improvement of 0.733 dB in the SI-SDRi index. (4) When the text cue information is removed, the TPEech model is completely unable to extract the target speech, resulting in an SI-SDRi index score of -6.227 dB.

[0063] See also Figure 6 , Figure 6 The effects of SNR mixing and SNR noise on TSE performance were revealed. The experiments were conducted under two settings using 500 fixed speakers, interference sources, and noise combinations. The results showed that when the SNR was negative, the effect of noise on all indicators was greater than the effect of interfering speech at the same SNR level. On the contrary, when the SNR was positive, the effect of interfering speech exceeded that of noise. This shows that in extreme cases, the effect of noise is greater.

[0064] Second embodiment of the target speaker extraction method based on historical conversation text: The difference between this embodiment and the first embodiment is that this embodiment is applied in a voice call scenario. In the voice call scenario, there is a first interlocutor and a second interlocutor who need to talk to each other, and the first interlocutor and the second interlocutor talk through an existing communication terminal. The communication terminal can be a smart phone, a tablet computer or a smart watch. The TPEech model of the first embodiment runs in the communication terminal of the first interlocutor and / or the second interlocutor.

[0065] At this time, the text prompt information is the current speech content of one interlocutor collected, and the mixed voice signal is the reply of the communication terminal of the other interlocutor in response to the current speech content. Therefore, one interlocutor can extract the voice signal of the target speaker based on the reply of the other interlocutor through the TPEech model, thereby improving the quality of voice call.

[0066] For example, suppose that the first interlocutor and the second interlocutor have a voice call in a strong noise environment. During the process, the first interlocutor says: "Where are you going this afternoon?" At this time, "Where are you going this afternoon?" is the extracted text prompt information, and the second interlocutor says: "I am going to see a performance at location A in the afternoon." "I am going to see a performance at location A in the afternoon" is the text content corresponding to the target speaker's voice signal. It can be seen that the text content is semantically associated with the text prompt information. The TPEech model processing running on the communication terminal of the first interlocutor combines the extracted text prompt information (corresponding to "Where are you going this afternoon?") and the mixed voice signal obtained from the second interlocutor to output the second interlocutor's voice signal. At this time, the mixed voice signal includes the second interlocutor's voice signal (corresponding to the text content "I am going to see a performance at location A in the afternoon"), the surrounding environmental noise of the second interlocutor, and the voice signal of the interfering speaker around the second interlocutor.

[0067] Then, the first interlocutor says: "What a coincidence! I am also going to see a performance at location A this afternoon." Similarly, at this time, "I am also going to see a performance at location A this afternoon" is the extracted text prompt information, and "What a coincidence! I am also going to see a performance at location A this afternoon" is the text content corresponding to the target speaker's voice signal. The TPEech model running on the communication terminal of the second interlocutor combines the extracted text prompt information (corresponding to "I am also going to see a performance at location A this afternoon") and the mixed voice signal obtained from the call terminal of the first interlocutor to output the voice signal of the first interlocutor. At this time, the mixed voice signal includes the voice signal of the first interlocutor (the corresponding text content is "What a coincidence! I am also going to see a performance at location A this afternoon"), the surrounding environmental noise of the first interlocutor, and the voice signals of the interfering speakers around the first interlocutor.

[0068] In summary, the present invention processes speech dialogue scenarios with overlapping speech and strong background noise, uses historical dialogue text as clues, explores the semantic association between questions and human answers, and uses prior knowledge of contextual prompts as clues to effectively extract the speech of the target speaker. In addition, by designing an echo prompt module, low signal-to-noise ratio noise is effectively removed at multiple granularity levels. The present invention improves the effect of extracting the speech signal of the target speaker, is easy to implement, and has a wide range of applications.

[0069] Computer device embodiment: The computer device of this embodiment includes a processor and a memory. The memory stores a computer program. When the processor executes the computer program, the above-mentioned target speaker extraction method embodiment based on historical conversation text is implemented.

[0070] The computer device may include but is not limited to a processor and a memory. Those skilled in the art will appreciate that the computer device may include more or fewer components, or a combination of certain components, or different components, for example, the computer device may also include input and output devices, network access devices, buses, etc.

[0071] For example, the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microcontroller or any conventional processor, etc. The processor is the control center of a computer device, and uses various interfaces and lines to connect various parts of the entire computer device.

[0072] The memory can be used to store computer programs and / or modules. The controller realizes various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. For example, the memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound receiving function, a sound conversion to text function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, text data, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0073] Computer readable storage medium embodiment: If the module integrated in the computer device of the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the implementation of all or part of the process of the target speaker extraction method embodiment based on the historical dialogue text can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the controller, the steps of the target speaker extraction method embodiment based on the historical dialogue text can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form. The storage medium can include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electrical carrier signals and telecommunications signals.

[0074] Computer program product embodiment: The computer program product of this embodiment includes computer instructions, which are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes each step of the above-mentioned target speaker extraction method embodiment based on historical conversation text.

[0075] Finally, it should be emphasized that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various changes and modifications. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A target speaker extraction method based on historical dialogue text, characterized in that: The following steps are involved: Processing a mixed speech signal through a speech encoder to obtain spectral feature embedding; wherein the mixed speech signal includes a speech signal of a target speaker, a speech signal of an interfering speaker, and a noise signal; Processing the text prompt information through a text encoder to obtain text feature embedding; associating the text content corresponding to the speech signal of the target speaker with the text prompt information; fusing the spectral feature embedding and the text feature embedding through a fusion layer to obtain a fusion feature; Processing the fused features through a mask estimator to obtain a target mask corresponding to the fused features; A speech signal of the target speaker is extracted according to the target mask and the mixed speech signal.

2. The target speaker extraction method based on historical dialogue text according to claim 1, characterized in that: The mask estimator includes a plurality of DPRNN modules. When the fusion feature is processed by the mask estimator, the mask estimator includes: The fused features are divided into segments, all the segments are connected to form blocks, the blocks are processed in turn by multiple echo prompt blocks of the DPRNN modules and the echo prompt module, and finally the blocks are reconstructed by the aggregation layer, and the target mask is generated by the activation layer.

3. The target speaker extraction method based on historical dialogue text according to claim 2, characterized in that: The number of the echo prompt blocks is 3.

4. The target speaker extraction method based on historical dialogue text according to claim 1, characterized in that: When the mixed speech signal is processed by the speech encoder to obtain the spectral feature embedding, the method includes: The mixed speech signal is processed through a one-dimensional convolutional layer and then a ReLU layer is applied to obtain the spectral feature embedding.

5. The target speaker extraction method based on historical dialogue text according to claim 1, characterized in that: When the text prompt information is processed by the text encoder to obtain the text feature embedding, the method includes: The text feature embedding corresponding to the text prompt information is generated by comparing the language audio pre-training model.

6. The target speaker extraction method based on historical dialogue text according to claim 1, characterized in that: The fusion layer, the mask estimator and the echo prompt module are obtained through a deep learning training method, including: setting a data set, the data set includes multiple voice segments and multiple noise segments, each of the voice segments is provided with a corresponding unique matching text generation problem, and during training, two of the voice segments and one of the noise segments are randomly selected and mixed using a random SNR mixing method.

7. The method for extracting a target speaker based on historical conversation text according to claim 6, characterized in that: The speech segments and most noise segments are sampled to 8 kHz.

8. A computer device comprising a processor and a memory, characterized in that: The memory stores a computer program, and when the computer program is executed by the processor, the method for extracting a target speaker based on historical conversation text according to any one of claims 1 to 7 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for extracting a target speaker based on historical conversation text as described in any one of claims 1 to 7 is implemented.

10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by the processor, the method for extracting a target speaker based on historical conversation text as described in any one of claims 1 to 7 is implemented.

Images