Speech extraction methods, devices, equipment and media

Abstract

This invention relates to the field of artificial intelligence technology and discloses a speech extraction method, apparatus, device, and medium. The method includes: first, acquiring reference speech of the target speaker and mixed speech of all speakers; preprocessing and encoding the reference speech and mixed speech to generate two discrete token sequences; fusing the two discrete token sequences to form a fused discrete token sequence; using a language model to predict the fused discrete token sequence to generate candidate discrete token sequences for the target speaker; calculating the probability distribution of the candidate token sequences using a linear classifier and selecting the sequence with the highest probability as the target discrete token sequence; and then reconstructing the target discrete token sequence into a speech waveform to obtain the speech of the target speaker. This invention transforms the complex audio generation problem into a classification problem, simplifying model training; and utilizes the sequence modeling capability of a language model to capture long-term dependencies between speech tokens, achieving high-quality speech reconstruction.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to speech extraction methods, apparatus, devices, and media. Background Technology

[0002] Target speaker extraction technology aims to accurately extract the voice of a specific speaker from mixed speech containing multiple speakers. This technology is particularly important in speech recognition in financial and medical scenarios. For example, in telephone customer service systems in the financial industry, this technology helps the system accurately identify and extract customer voice, ensuring the clarity and accuracy of the call content even in noisy environments, thereby improving customer experience and service quality. Similarly, in remote doctor-patient communication scenarios in the medical field, this technology can effectively separate doctors' instructions or patients' symptom descriptions, assisting in medical diagnosis and ensuring accurate information transmission.

[0003] Currently, target speaker extraction methods are mainly divided into two categories: discriminative models and generative models. Discriminative models typically employ masking strategies to directly minimize the difference between the estimated speech and the clean speech. However, these methods struggle to generalize to unseen data and may introduce unnecessary distortion. In contrast, generative models aim to learn the latent distribution of the target speaker's speech and use this knowledge to generate the target speaker's speech from the mixed speech. However, current target speaker extraction processes are complex and fail to effectively capture long-term dependencies between speech tokens, resulting in low-quality obtained speech. Summary of the Invention

[0004] This invention provides an artificial intelligence-based speech extraction method, apparatus, computer equipment, and medium to solve the technical problems of complex and low-quality speech extraction techniques in existing target speaker extraction technologies.

[0005] Firstly, a speech extraction method is provided, including:

[0006] Acquire the reference speech of the target speaker and the mixed speech to be extracted, in which the target speaker is involved;

[0007] The reference speech and the mixed speech are preprocessed and encoded respectively to obtain the corresponding first discrete token sequence and second discrete token sequence;

[0008] The first discrete token sequence and the second discrete token sequence are fused using a cross-attention mechanism to obtain a fused discrete token sequence.

[0009] A language model is used to predict the target speaker's speech from the fused discrete token sequence, resulting in a candidate discrete token sequence for the target speaker.

[0010] A linear classifier is used to predict the probability distribution of each token sequence in the candidate discrete token sequence, and the token sequence with the highest probability distribution is output as the target discrete token sequence;

[0011] The target discrete token sequence is reconstructed into a speech waveform to obtain the speech of the target speaker.

[0012] Secondly, a speech extraction device is provided, comprising:

[0013] The acquisition module is used to acquire the reference speech of the target speaker and the mixed speech to be extracted in which the target speaker participates;

[0014] The encoding module is used to preprocess and encode the reference speech and the mixed speech respectively to obtain the corresponding first discrete token sequence and second discrete token sequence;

[0015] The sequence fusion module is used to fuse the first discrete token sequence and the second discrete token sequence using a cross-attention mechanism to obtain a fused discrete token sequence.

[0016] The candidate sequence prediction module is used to predict the target speaker's speech by using a language model on the fused discrete token sequence, so as to obtain the candidate discrete token sequence of the target speaker.

[0017] The target sequence output module is used to predict the probability distribution of each token sequence in the candidate discrete token sequence using a linear classifier, and output the token sequence with the high probability distribution as the target discrete token sequence.

[0018] The speech reconstruction module is used to reconstruct the target discrete token sequence into a speech waveform to obtain the speech of the target speaker.

[0019] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described speech extraction method.

[0020] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described speech extraction method.

[0021] The aforementioned speech extraction method, apparatus, device, and medium, and the scheme implemented by the speech extraction method, involve: firstly, acquiring the reference speech of the target speaker and the mixed speech of all speakers; preprocessing and encoding the reference speech and the mixed speech to generate two discrete token sequences; fusing the two discrete token sequences using a cross-attention mechanism to form a fused discrete token sequence; using a language model to predict the fused discrete token sequence to generate candidate discrete token sequences for the target speaker; calculating the probability distribution of the candidate token sequences using a linear classifier, and selecting the sequence with the highest probability as the target discrete token sequence; and then reconstructing the target discrete token sequence into a speech waveform to obtain the speech of the target speaker. This invention transforms the complex audio generation problem into a classification problem, simplifying model training; it effectively captures the long-term dependencies between speech tokens by utilizing the sequence modeling capabilities of a language model, achieving speech reconstruction and effectively solving the problem of target speech prediction in multi-speaker speech mixing situations. This improves the accuracy of speech recognition and prediction, not only accurately capturing the speech features of a specific speaker in complex multi-person dialogue environments but also generating high-quality prediction results with high speech quality and intelligibility. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of an application environment for the speech extraction method in one embodiment of the present invention;

[0024] Figure 2 This is a schematic flowchart of a speech extraction method according to an embodiment of the present invention;

[0025] Figure 3 yes Figure 2 A flowchart illustrating a specific implementation of step S202;

[0026] Figure 4 yes Figure 2 A schematic diagram of a specific implementation method for step S203;

[0027] Figure 5 yes Figure 2 A flowchart illustrating a specific implementation of step S204;

[0028] Figure 6 yes Figure 2A schematic diagram of a specific implementation method for step S205;

[0029] Figure 7 yes Figure 2 A schematic diagram of a specific implementation method for step S206;

[0030] Figure 8 This is a schematic diagram of the structure of a speech extraction device in one embodiment of the present invention;

[0031] Figure 9 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0032] Figure 10 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] The speech extraction method provided in this embodiment of the invention can be applied to, for example, Figure 1In this application environment, the client communicates with the server via a network. The server can receive the reference speech of the target speaker and the mixed speech of all speakers from the client, generate a target speaker speech extraction task, and send it to the server. The server preprocesses and encodes the reference speech and mixed speech respectively to obtain corresponding first discrete token sequences and second discrete token sequences. A cross-attention mechanism is used to fuse the first and second discrete token sequences to obtain a fused discrete token sequence. A language model is used to predict the target speaker's speech from the fused discrete token sequence to obtain a candidate discrete token sequence for the target speaker. A linear classifier is used to predict the probability distribution of each token sequence in the candidate discrete token sequence, and the token sequence with the high probability distribution is output as the target discrete token sequence. The target discrete token sequence is reconstructed into a speech waveform to obtain the target speaker's speech. This invention transforms the complex audio generation problem into a classification problem, simplifying model training; it effectively captures the long-term dependencies between speech tokens by utilizing the sequence modeling capabilities of the language model, achieving high-quality speech reconstruction. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0035] Please see Figure 2 As shown, Figure 2 A flowchart illustrating the speech extraction method provided in this embodiment of the invention includes the following steps:

[0036] S201. Obtain the reference speech of the target speaker and the mixed speech to be extracted in which the target speaker participated;

[0037] In this step, a reference voice of the target speaker (e.g., registered voice) is obtained in advance; the reference voice and the mixed voice can be spliced ​​together to update the mixed voice, and the updated mixed voice can enhance the attention of subsequent modeling to the target speaker.

[0038] S202. Preprocess and encode the reference speech and the mixed speech respectively to obtain the corresponding first discrete token sequence and second discrete token sequence;

[0039] S203. The first discrete token sequence and the second discrete token sequence are fused using a cross-attention mechanism to obtain a fused discrete token sequence.

[0040] S204. Use a language model to predict the target speaker's speech from the fused discrete token sequence to obtain the candidate discrete token sequence of the target speaker.

[0041] S205. Use a linear classifier to predict the probability distribution of each token sequence in the candidate discrete token sequence, and output the token sequence with the high probability distribution as the target discrete token sequence.

[0042] S206. Reconstruct the target discrete token sequence into a speech waveform to obtain the speech of the target speaker.

[0043] In this embodiment, the target speaker extraction method mainly includes three stages: encoding, modeling, and decoding. Encoding stage: The reference speech and mixed speech are preprocessed and encoded separately to obtain corresponding discrete token sequences. Modeling stage: The reference speech information is integrated into the mixed speech token sequence using a cross-attention mechanism, and a language model is used to model the dependencies between token sequences to predict the target speaker's token sequence. Decoding stage: A HiFi-GAN vocoder can be used to reconstruct the target speaker's token sequence into a speech waveform.

[0044] For example, in a specific scenario within the financial sector, a financial institution needs to perform speech recognition on customer service calls to extract key information. Due to the large number of customer service personnel and the complex background of the calls, traditional speech recognition methods struggle to accurately extract the target speaker's voice. In this case, the target speaker extraction method described in this embodiment can be used. First, the reference voice of the customer service personnel and the mixed voice to be extracted are obtained. Then, the reference voice and the mixed voice are preprocessed and encoded to obtain corresponding first and second discrete token sequences. Next, a cross-attention mechanism is used to integrate the reference voice information into the mixed voice token sequence, and a language model is used to model the dependencies between the token sequences to predict the token sequence of the customer service personnel's voice. Finally, a HiFi-GAN vocoder is used to reconstruct the predicted token sequence into a speech waveform, obtaining the customer service personnel's voice. In this way, financial institutions can accurately extract the customer service personnel's voice, providing strong support for subsequent information processing.

[0045] This invention transforms the complex audio generation problem into a classification problem, simplifying model training. By utilizing the sequence modeling capabilities of language models, it effectively captures long-term dependencies between speech tokens, achieving speech reconstruction. It effectively solves the challenge of target speech prediction in multi-speaker speech mixture scenarios, improving the accuracy of speech recognition and prediction. It can not only accurately capture the speech features of specific speakers in complex multi-person dialogue environments, but also generate high-quality prediction results with high speech quality and intelligibility.

[0046] In one embodiment, such as Figure 3 As shown, step S202 includes:

[0047] S301. The reference speech is input into the WavLM model and features are extracted from multiple hidden layers. The outputs of multiple hidden layers are used as the first speech features. The first speech features are then quantized using the K-means clustering algorithm to obtain the first discrete token sequence.

[0048] S302. The mixed speech is input into the WavLM model and features are extracted from multiple hidden layers. The outputs of the multiple hidden layers are used as the second speech features. The K-means clustering algorithm is used to quantize the second speech features to obtain the second discrete token sequence.

[0049] In this embodiment, the WavLM model is used to extract features from the input reference speech and the mixed speech, respectively. The WavLM model has powerful sequence modeling capabilities, capturing long-term dependencies between speech tokens, which is crucial for subsequent speech reconstruction and target speech prediction. Through multiple hidden layers of the WavLM model, we can obtain feature representations containing rich speech information. These feature representations form the basis for subsequent feature quantization and discrete token sequence generation. In the device, the WavLM model works collaboratively with other components to achieve efficient extraction of the target speaker's speech.

[0050] For example, in a specific medical scenario, it's necessary to extract the speech of a doctor communicating with a patient in a noisy environment. The WavLM model can be used for accurate feature extraction of the doctor's voice. First, the doctor's speech is input into the WavLM model as a reference, and the model performs deep analysis to extract key feature information. Then, the mixed speech of the doctor and patient is also input into the WavLM model, which performs feature extraction again. In both steps, the WavLM model utilizes its powerful sequence modeling capabilities to capture the long-term dependencies between the doctor's speech tokens, thus ensuring that the extracted features accurately reflect the characteristics of the doctor's speech.

[0051] In one embodiment, such as Figure 4 As shown, step S203 includes:

[0052] S401. Extract features from the first discrete token sequence and the second discrete token sequence respectively to obtain the corresponding first sequence features and second sequence features;

[0053] S402. The first sequence feature and the second sequence feature are interacted through the cross-computation module to generate an interaction feature matrix;

[0054] S403. Use an attention mechanism to assign weights to the interaction feature matrix and determine the association weights of the first sequence features and the second sequence features.

[0055] S404. Align the first sequence features and the second sequence features according to the association weights to generate an aligned feature vector;

[0056] S405. Map the aligned feature vectors to a unified feature space to obtain the mapped feature vectors;

[0057] S406. Reconstruct the mapped feature vector to generate a fused discrete token sequence.

[0058] In this embodiment, steps S401-S406 extract features from the first discrete token sequence and the second discrete token sequence, respectively, to obtain corresponding first sequence features and second sequence features. The cross-computation module is responsible for exchanging information between these features to generate an interaction feature matrix. Next, the attention mechanism module assigns weights to the interaction feature matrix to determine the association weights between the first sequence features and the second sequence features. The sequence alignment module aligns the features according to these association weights to generate aligned feature vectors. The feature space mapping module maps the aligned feature vectors to a unified feature space to obtain mapped feature vectors. Finally, the sequence reconstruction module reconstructs the mapped feature vectors to generate a fused discrete token sequence, thereby efficiently extracting the speech information of the target speaker.

[0059] In one embodiment, such as Figure 5 As shown, step S204 includes:

[0060] S501. Use a pre-trained language model to extract features from the fused discrete token sequence and generate a total vector representation containing the speech features of each speaker.

[0061] S502. Use a pre-trained language model to extract features from the first discrete token sequence and generate a reference vector representation of the target speaker's speech features.

[0062] S503. The total vector representation is matched with the reference vector representation using a matching algorithm, the matching degree score is calculated, and the feature sequences with high matching degree with the reference vector are selected from the total vector representation and used as candidate discrete token sequences for the target speaker.

[0063] In this embodiment, steps S501-S503 extract features from the fused discrete token sequence and the first discrete token sequence using a pre-trained language model, generating a total vector representation and a reference vector representation, respectively. These two vector representations contain the speech features of each speaker and the speech features of the target speaker, respectively. Subsequently, a matching algorithm is used to match the total vector representation with the reference vector representation, and feature sequences with high matching degrees are selected by calculating the matching score. These feature sequences are used as candidate discrete token sequences for the target speaker, providing crucial information for subsequent processing. Through this design, the speech extraction process can efficiently and accurately extract the speech information of the target speaker, providing strong support for applications in speech recognition, speech analysis, and other fields.

[0064] In one embodiment, such as Figure 6 As shown, step S205 includes:

[0065] S601. Extract features from each token sequence in the candidate discrete token sequence to form a set of feature vectors;

[0066] S602. Construct a linear classifier model and set model parameters including weight vectors and bias terms;

[0067] S603. Input the feature vector set into a linear classifier, and obtain the probability distribution of each token sequence through weight calculation and bias term adjustment;

[0068] S604. Select the token sequence whose probability distribution is higher than the preset classification threshold as the target discrete token sequence.

[0069] In this embodiment, steps S601-S604 further analyze and process the candidate discrete token sequences through a specific processing module. First, feature extraction is performed on each token sequence. This step delves into the key information within each token sequence and transforms it into a set of feature vectors, providing data support for subsequent classification processing. Next, a linear classifier model is constructed, and model parameters, including weight vectors and bias terms, are set. The settings of these parameters have a crucial impact on the classifier's performance.

[0070] After the model is built, the feature vector set is input into a linear classifier. Through weight calculation and bias adjustment, the classifier outputs a probability distribution for each token sequence. This probability distribution reflects the likelihood that each token sequence belongs to the target speaker's speech. Finally, based on a preset classification threshold, token sequences with probability distributions higher than the threshold are selected as target discrete token sequences. These target discrete token sequences accurately reflect the speech features of the target speaker, providing a reliable foundation for subsequent processing and applications.

[0071] Through steps S601-S604, this speech extraction process can efficiently filter out the speech information of the target speaker, providing strong support for applications in speech recognition, speech analysis, and other fields. At the same time, the device also has high accuracy and robustness, and can adapt to the speech extraction needs of different scenarios.

[0072] In one embodiment, such as Figure 7 As shown, step S206 includes:

[0073] S701. Extract the acoustic features of the target discrete token sequence;

[0074] S702. Input the acoustic features into the HiFi-GAN vocoder to reconstruct the sequence and generate the initial speech waveform;

[0075] S703. The initial speech waveform is denoised, and the speech waveform of the target speaker is output.

[0076] In this embodiment, steps S701-S703 together constitute a complete speech reconstruction and optimization process. S701 extracts acoustic features from the target discrete token sequence, providing crucial input information for subsequent steps. These acoustic features accurately reflect the speech characteristics of the target speaker and are the foundation of speech reconstruction. S702 uses a HiFi-GAN vocoder for sequence reconstruction, generating an initial speech waveform. The HiFi-GAN vocoder, with its powerful generation capabilities and high-quality output, has wide applications in speech synthesis. Through this vocoder, we can convert acoustic features into realistic speech waveforms, providing high-quality input for subsequent processing. S703 then performs noise reduction processing on the initial speech waveform to eliminate potential background noise and interference. This step is crucial for improving the clarity and purity of the output speech. After noise reduction, we finally obtain the speech waveform of the target speaker, which accurately reflects the speech characteristics of the target speaker, providing a reliable foundation for subsequent applications and processing.

[0077] This embodiment, combining steps S701-S703 described above, achieves efficient and accurate speech extraction and reconstruction. This device not only possesses high accuracy and robustness but also adapts to speech extraction needs in different scenarios, providing strong support for applications in speech recognition, speech analysis, and other fields.

[0078] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0079] In some other embodiments, the present invention can be effectively applied to the dual tasks of speech synthesis and identity verification.

[0080] For example, in a financial telephone customer service scenario, when a customer initiates a business inquiry through an IVR system, the Mel spectrum extraction module first converts the customer service representative's standard response text (first speech feature) and the customer's real-time speech (second speech feature) into a discrete token sequence. The cross-computation module then establishes a dynamic correlation between the prosodic features of the customer service representative's response (average fundamental frequency 230Hz, speech rate 4.2 words / second) and the personalized features of the customer's speech (specific voiceprint fingerprint) through a multi-head cross-modal attention mechanism.

[0081] In the identity verification stage, this invention uses an attention weight allocation module to compare the interaction feature matrix frame by frame. When the formant features of the customer's voice (e.g., F1 = 500Hz, F2 = 1500Hz) match the reserved voiceprint template to a preset threshold, the sequence alignment module generates a time-synchronized fusion feature vector. After being reconstructed by the WaveGlow vocoder, this vector retains the standard business script of the customer service (e.g., "Your account balance is...") while embedding the customer's unique vocal characteristics, ensuring that the synthesized speech complies with financial compliance requirements.

[0082] In one embodiment, a speech extraction device is provided, which corresponds one-to-one with the speech extraction methods described in the above embodiments. For example... Figure 8 As shown, the speech extraction device includes an acquisition module 801, an encoding module 802, a sequence fusion module 803, a candidate sequence prediction module 804, a target sequence output module 805, and a speech reconstruction module 806. Detailed descriptions of each functional module are as follows:

[0083] The acquisition module 801 is used to acquire the reference speech of the target speaker and the mixed speech to be extracted in which the target speaker participates;

[0084] Encoding module 802 is used to preprocess and encode the reference speech and mixed speech respectively to obtain the corresponding first discrete token sequence and second discrete token sequence;

[0085] The sequence fusion module 803 is used to fuse the first discrete token sequence and the second discrete token sequence using a cross-attention mechanism to obtain a fused discrete token sequence.

[0086] The candidate sequence prediction module 804 is used to predict the target speaker's speech by using a language model on the fused discrete token sequence, so as to obtain the candidate discrete token sequence of the target speaker.

[0087] The target sequence output module 805 is used to predict the probability distribution of each token sequence in the candidate discrete token sequence using a linear classifier, and output the token sequence with the high probability distribution as the target discrete token sequence.

[0088] The speech reconstruction module 806 is used to reconstruct the target discrete token sequence into a speech waveform to obtain the speech of the target speaker.

[0089] In one embodiment, the encoding module 802 is specifically used for:

[0090] The reference speech is input into the WavLM model and features are extracted from multiple hidden layers. The outputs of the multiple hidden layers are used as the first speech features. The first speech features are then quantized using the K-means clustering algorithm to obtain the first discrete token sequence.

[0091] The mixed speech is input into the WavLM model and features are extracted from multiple hidden layers. The outputs of the multiple hidden layers are used as the second speech features. The K-means clustering algorithm is then used to quantize the second speech features to obtain the second discrete token sequence.

[0092] In one embodiment, the sequence fusion module 803 is specifically used for:

[0093] Feature extraction is performed on the first discrete token sequence and the second discrete token sequence respectively to obtain the corresponding first sequence features and second sequence features;

[0094] The cross-computation module interacts the first sequence features and the second sequence features to generate an interaction feature matrix.

[0095] An attention mechanism is used to assign weights to the interaction feature matrix to determine the association weights between the first sequence features and the second sequence features.

[0096] The first sequence features and the second sequence features are aligned according to the association weights to generate an aligned feature vector.

[0097] The aligned feature vectors are mapped to a unified feature space to obtain the mapped feature vectors;

[0098] The mapped feature vectors are reconstructed to generate a fused discrete token sequence.

[0099] In one embodiment, the candidate sequence prediction module 804 is specifically used for:

[0100] A pre-trained language model is used to extract features from the fused discrete token sequence to generate a total vector representation containing the speech features of each speaker.

[0101] A pre-trained language model is used to extract features from the first discrete token sequence to generate a reference vector representation of the target speaker's speech features;

[0102] The total vector representation is matched with the reference vector representation using a matching algorithm. The matching score is calculated, and feature sequences with high matching scores with the reference vector are selected from the total vector representation and used as candidate discrete token sequences for the target speaker.

[0103] In one embodiment, the target sequence output module 805 is specifically used for:

[0104] Extract features from each token sequence in the candidate discrete token sequence to form a set of feature vectors;

[0105] Construct a linear classifier model and set model parameters including weight vectors and bias terms;

[0106] The feature vector set is input into a linear classifier, and the probability distribution of each token sequence is obtained through weight calculation and bias term adjustment;

[0107] Select the token sequence whose probability distribution is higher than the preset classification threshold as the target discrete token sequence.

[0108] In one embodiment, the speech reconstruction module 806 is specifically used for:

[0109] Extract acoustic features from the target discrete token sequence;

[0110] The acoustic features are input into the HiFi-GAN vocoder for sequence reconstruction to generate the initial speech waveform.

[0111] The initial speech waveform is denoised to output the speech waveform of the target speaker.

[0112] This invention provides a speech extraction device. The target speaker extraction method in this device mainly includes three stages: encoding, modeling, and decoding. The encoding stage involves preprocessing and encoding the reference speech and the mixed speech separately to obtain corresponding discrete token sequences. The modeling stage utilizes a cross-attention mechanism to integrate the reference speech information into the mixed speech token sequence and uses a language model to model the dependencies between token sequences, predicting the token sequence of the target speaker's speech. The decoding stage uses a HiFi-GAN vocoder to reconstruct the speech waveform from the token sequence of the target speaker's speech.

[0113] This invention transforms the complex audio generation problem into a classification problem, simplifying model training. By utilizing the sequence modeling capabilities of language models, it effectively captures long-term dependencies between speech tokens, achieving speech reconstruction. It effectively solves the challenge of target speech prediction in multi-speaker speech mixture scenarios, improving the accuracy of speech recognition and prediction. It can not only accurately capture the speech features of specific speakers in complex multi-person dialogue environments, but also generate high-quality prediction results with high speech quality and intelligibility.

[0114] Specific limitations regarding the speech extraction device can be found in the limitations of the speech extraction method described above, and will not be repeated here. Each module in the aforementioned speech extraction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0115] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a speech extraction method on the server side.

[0116] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 10As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the functions or steps of a speech extraction method on the client side.

[0117] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0118] Acquire the reference speech of the target speaker and the mixed speech to be extracted, in which the target speaker is involved;

[0119] The reference speech and the mixed speech are preprocessed and encoded respectively to obtain the corresponding first discrete token sequence and second discrete token sequence;

[0120] A cross-attention mechanism is used to fuse the first discrete token sequence and the second discrete token sequence to obtain a fused discrete token sequence.

[0121] A language model is used to predict the speech of the target speaker from the fused discrete token sequence, resulting in a candidate discrete token sequence of the target speaker.

[0122] A linear classifier is used to predict the probability distribution of each token sequence in the candidate discrete token sequence, and the token sequence with the highest probability distribution is output as the target discrete token sequence.

[0123] The target discrete token sequence is reconstructed into a speech waveform to obtain the speech of the target speaker.

[0124] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0125] Acquire the reference speech of the target speaker and the mixed speech to be extracted, in which the target speaker is involved;

[0126] The reference speech and the mixed speech are preprocessed and encoded respectively to obtain the corresponding first discrete token sequence and second discrete token sequence;

[0127] A cross-attention mechanism is used to fuse the first discrete token sequence and the second discrete token sequence to obtain a fused discrete token sequence.

[0128] A language model is used to predict the speech of the target speaker from the fused discrete token sequence, resulting in a candidate discrete token sequence of the target speaker.

[0129] A linear classifier is used to predict the probability distribution of each token sequence in the candidate discrete token sequence, and the token sequence with the highest probability distribution is output as the target discrete token sequence.

[0130] The target discrete token sequence is reconstructed into a speech waveform to obtain the speech of the target speaker.

[0131] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0132] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0133] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0134] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A voice extraction method characterized by, The method comprises the following steps: obtaining reference speech of a target speaker and mixed speech in which the target speaker participates and from which speech is to be extracted; preprocessing and encoding the reference speech and the mixed speech respectively to obtain corresponding first discrete token sequences and second discrete token sequences; fusing the first discrete token sequences and the second discrete token sequences by using a cross-attention mechanism to obtain fused discrete token sequences; predicting target speaker speech from the fused discrete token sequences by using a language model to obtain candidate discrete token sequences of the target speaker; predicting probability distribution of each token sequence in the candidate discrete token sequences by using a linear classifier, and outputting a token sequence with high probability distribution as a target discrete token sequence; specifically comprising: extracting features of each token sequence in the candidate discrete token sequences to form a feature vector set; constructing a linear classifier model and setting model parameters containing a weight vector and a bias term; inputting the feature vector set into the linear classifier to obtain the probability distribution of each token sequence through weight calculation and bias adjustment; selecting a token sequence with a probability distribution higher than a preset classification threshold as the target discrete token sequence; reconstructing the target discrete token sequence into a speech waveform to obtain speech of the target speaker.

2. The voice extraction method of claim 1, wherein, The preprocessing and encoding of the reference speech and the mixed speech respectively to obtain corresponding first discrete token sequences and second discrete token sequences comprises: inputting the reference speech into a WavLM model to extract features of multiple hidden layers to obtain outputs of the multiple hidden layers as first speech features, and quantizing the first speech features by using a K-means clustering algorithm to obtain the first discrete token sequences; inputting the mixed speech into the WavLM model to extract features of multiple hidden layers to obtain outputs of the multiple hidden layers as second speech features, and quantizing the second speech features by using a K-means clustering algorithm to obtain the second discrete token sequences.

3. The voice extraction method of claim 1, wherein, The fusing of the first discrete token sequences and the second discrete token sequences by using a cross-attention mechanism to obtain fused discrete token sequences comprises: extracting features of the first discrete token sequences and the second discrete token sequences respectively to obtain corresponding first sequence features and second sequence features; interacting information of the first sequence features and the second sequence features by using a cross calculation module to generate an interaction feature matrix; determining correlation weights of the first sequence features and the second sequence features by using an attention mechanism to distribute weights of the interaction feature matrix; aligning the first sequence features and the second sequence features according to the correlation weights to generate aligned feature vectors; mapping the aligned feature vectors to a unified feature space to obtain mapped feature vectors; reconstructing the mapped feature vectors to generate the fused discrete token sequences.

4. The voice extraction method of claim 2, wherein, The target speaker's candidate discrete token sequence is obtained by predicting target person speech of the fused discrete token sequence using a language model. The pre-trained language model is used to extract features of the fused discrete token sequence to generate a total vector representation containing speech features of each speaker; The pre-trained language model is used to extract features of the first discrete token sequence to generate a reference vector representation of the target speaker's speech features; The total vector representation and the reference vector representation are matched by a matching algorithm to calculate a matching score, and a feature sequence with high matching degree with the reference vector is selected from the total vector representation as the candidate discrete token sequence of the target speaker.

5. The voice extraction method of claim 1, wherein, The target discrete token sequence is reconstructed into a speech waveform to obtain the speech of the target speaker, including: Extracting the acoustic features of the target discrete token sequence; The initial speech waveform is generated by inputting the acoustic features into the HiFi-GAN vocoder for sequence reconstruction; The initial speech waveform is denoised to output the speech waveform of the target speaker.

6. The voice extraction method of claim 1, wherein, It also includes: The reference speech and the mixed speech are spliced to update the mixed speech.

7. A voice extraction apparatus characterized by comprising: It includes: An acquisition module is configured to acquire reference speech of a target speaker and mixed speech to be extracted in which the target speaker participates; An encoding module is configured to preprocess and encode the reference speech and the mixed speech respectively to obtain corresponding first and second discrete token sequences; A sequence fusion module is configured to fuse the first and second discrete token sequences by using a cross-attention mechanism to obtain a fused discrete token sequence; A candidate sequence prediction module is configured to predict target person speech of the fused discrete token sequence using a language model to obtain a candidate discrete token sequence of the target speaker; A target sequence output module is configured to predict a probability distribution of each token sequence in the candidate discrete token sequence using a linear classifier, and output a token sequence with high probability distribution as a target discrete token sequence; A speech reconstruction module is configured to reconstruct the target discrete token sequence into a speech waveform to obtain the speech of the target speaker; The target sequence output module is specifically configured to: extract features of each token sequence in the candidate discrete token sequence to form a feature vector set; construct a linear classifier model and set model parameters including a weight vector and a bias term; input the feature vector set into the linear classifier to obtain the probability distribution of each token sequence through weight calculation and bias adjustment; Select a token sequence with a probability distribution higher than a preset classification threshold as the target discrete token sequence.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the speech extraction method of any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the steps of the speech extraction method of any one of claims 1-6.

Images