Voice generation method based on two-channel semantic token and block condition flow matching

Abstract

The invention relates to the technical field of speech synthesis, and particularly discloses a speech generation method based on two-channel semantic token and block condition flow matching. According to the method, the first semantic token sequence and the second semantic token sequence are generated through double-channel semantic parallel processing, the two channels can mutually perceive interaction dynamic information, interaction information between speakers can be accurately obtained, the problem that interaction dynamic information cannot be extracted through a single channel is solved, and naturalness and interactivity of synthesized voice are improved; besides, the long token sequence is divided into token blocks with fixed lengths, and each token block is processed by adopting a conditional stream matching model, so that high-quality synthesis is realized, and the naturalness of the synthesized speech is further improved. When the method is applied to business scenes of medical knowledge popularization, financial product interpretation and the like, natural and smooth dialogue audio with high interactivity can be generated, and the popularization, introduction, interpretation and promotion video voice quality is improved.

Description

Technical Field

[0001] This application relates to the field of speech synthesis technology, and in particular to a speech generation method based on dual-channel semantic tokens and block conditional stream matching. Background Technology

[0002] With the deepening of digital transformation, digital technology is increasingly being applied to business scenarios in the financial and medical fields. For example, in medical knowledge dissemination, videos are generated to popularize medical information, and videos are also used for introducing, interpreting, and promoting financial products. The characters in the videos can be generated using virtual technology, and the dialogue can be synthesized using text-to-speech technology. However, existing speech synthesis technologies cannot accurately capture the voice interaction relationships of the speakers in multi-turn dialogue scenarios, resulting in unnatural transitions, interruptions, and overlaps in the generated dialogue. Therefore, improving the naturalness and interactivity of synthesized speech has become an urgent problem to be solved. Summary of the Invention

[0003] This application provides a speech generation method based on dual-channel semantic tokens and block conditional stream matching to improve the naturalness and interactivity of synthesized speech.

[0004] In a first aspect, this application provides a speech generation method based on dual-channel semantic tokens and block conditional stream matching, the method comprising: The dialogue text and the speaker's speech feature vector are obtained, and the dialogue text and the speaker's speech feature vector are processed based on a preset dual-channel large language model to generate a first semantic token sequence corresponding to the first speaker and a second semantic token sequence corresponding to the second speaker in parallel. According to the first preset length, the first semantic token sequence and the second semantic token sequence are divided into a first token block set and a second token block set respectively. Based on a preset conditional flow matching model, the first speech feature vector and the first token block set are reconstructed to obtain the first Mel spectrum, and the second speech feature vector and the second token block set are reconstructed in parallel to obtain the second Mel spectrum. The first Mel spectrum and the second Mel spectrum are converted to obtain a first waveform and a second waveform, respectively. The first waveform and the second waveform are then aligned and merged to obtain the target dialogue audio.

[0005] Secondly, this application also provides a speech generation apparatus based on dual-channel semantic tokens and block conditional stream matching, the apparatus comprising: The text and speech feature acquisition module is used to acquire dialogue text and speaker speech feature vectors, wherein the speaker speech feature vectors include a first speech feature vector corresponding to a first speaker and a second speech feature vector corresponding to a second speaker. The semantic token sequence generation module is used to process the dialogue text and the speaker's speech feature vector based on a preset dual-channel large language model, and generate in parallel a first semantic token sequence corresponding to the first speaker and a second semantic token sequence corresponding to the second speaker. The semantic token sequence segmentation module is used to segment the first semantic token sequence and the second semantic token sequence according to a first preset length, respectively, to obtain a first token block set and a second token block set; The Mel spectrum generation module is used to perform spectrum reconstruction on the first speech feature vector and the first token block set based on a preset conditional flow matching model to obtain the first Mel spectrum, and to perform spectrum reconstruction on the second speech feature vector and the second token block set in parallel to obtain the second Mel spectrum. The target dialogue audio acquisition module is used to convert the first Mel spectrum and the second Mel spectrum respectively to obtain a first waveform and a second waveform, and to align and merge the first waveform and the second waveform to obtain the target dialogue audio.

[0006] Thirdly, this application also provides a computer device, the computer device including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the speech generation method based on dual-channel semantic tokens and block conditional stream matching as described above.

[0007] Fourthly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the speech generation method based on dual-channel semantic tokens and block conditional stream matching as described above.

[0008] This application discloses a speech generation method based on dual-channel semantic tokens and block-based conditional flow matching. The method obtains dialogue text and speaker speech feature vectors, wherein the speaker speech feature vectors include a first speech feature vector corresponding to a first speaker and a second speech feature vector corresponding to a second speaker. Based on a preset dual-channel large language model, the method processes the dialogue text and the speaker speech feature vectors, generating in parallel a first semantic token sequence corresponding to the first speaker and a second semantic token sequence corresponding to the second speaker. The first and second semantic token sequences are segmented according to a first preset length to obtain a first token block set and a second token block set. Based on a preset conditional flow matching model, the method performs spectral reconstruction on the first speech feature vectors and the first token block set to obtain a first Mel spectrum, and simultaneously performs spectral reconstruction on the second speech feature vectors and the second token block set to obtain a second Mel spectrum. The first and second Mel spectra are then converted to obtain a first waveform and a second waveform, respectively. Finally, the first and second waveforms are aligned and merged to obtain the target dialogue audio. This application generates a first semantic token sequence and a second semantic token sequence through dual-channel semantic parallel processing. The two channels can perceive each other's interactive dynamic information and accurately obtain the interactive information between speakers, solving the problem that single-channel cannot extract interactive dynamic information and improving the naturalness and interactivity of synthesized speech. In addition, by dividing the long token sequence into fixed-length token blocks and processing each token block with a conditional flow matching model, high-quality synthesis is achieved, further improving the naturalness of synthesized speech. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a first schematic flowchart of a speech generation method based on dual-channel semantic tokens and block conditional stream matching provided by an embodiment of this application; Figure 2 This is a second schematic flowchart of a speech generation method based on dual-channel semantic tokens and block conditional stream matching provided by an embodiment of this application; Figure 3 This is a third schematic flowchart of a speech generation method based on dual-channel semantic tokens and block conditional stream matching provided by an embodiment of this application; Figure 4A schematic block diagram of a speech generation device based on dual-channel semantic tokens and block conditional stream matching provided for embodiments of this application; Figure 5 A schematic block diagram of the structure of a computer device provided for an embodiment of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the described order. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0013] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0014] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0015] This application provides a speech generation method based on dual-channel semantic tokens and block-based conditional flow matching. This method can be applied to a server. Through parallel processing of dual-channel semantics, a first semantic token sequence and a second semantic token sequence are generated. The two channels can mutually perceive interactive dynamic information, accurately obtaining the interaction information between speakers. This solves the problem that a single channel cannot extract interactive dynamic information, improving the naturalness and interactivity of the synthesized speech. Furthermore, by dividing the long token sequence into fixed-length token blocks and processing each token block using a conditional flow matching model, high-quality synthesis is achieved, further improving the naturalness of the synthesized speech. The server can be a standalone server or a server cluster.

[0016] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0017] Please see Figure 1 , Figure 1 This is a schematic flowchart illustrating a speech generation method based on dual-channel semantic tokens and block-based conditional flow matching, provided in an embodiment of this application. This speech generation method, based on dual-channel semantic tokens and block-based conditional flow matching, can be applied in a server. It generates a first semantic token sequence and a second semantic token sequence through parallel processing of dual-channel semantics. The two channels can mutually perceive interactive dynamic information, accurately obtaining the interactive information between speakers. This solves the problem that a single channel cannot extract interactive dynamic information, improving the naturalness and interactivity of the synthesized speech. Furthermore, by dividing the long token sequence into fixed-length token blocks and processing each token block using a conditional flow matching model, high-quality synthesis is achieved, further improving the naturalness of the synthesized speech.

[0018] like Figure 1 As shown, the speech generation method based on dual-channel semantic tokens and block conditional stream matching specifically includes steps S101 to S105.

[0019] S101. Obtain the dialogue text and the speaker's speech feature vector, wherein the speaker's speech feature vector includes a first speech feature vector corresponding to a first speaker and a second speech feature vector corresponding to a second speaker. In one embodiment, the dialogue text is a structured dialogue script input by the user, containing speaker identifiers and switching control tags (such as [spkchange]) to indicate speaker switching. For example, in a medical knowledge dissemination video, from the perspective of communication between a doctor and a patient, disseminating knowledge about a certain disease, the dialogue script would include the dialogue between the doctor (first speaker) and the patient (second speaker), with the control tag [spkchange] indicating the switching of speakers between the doctor and the patient.

[0020] Speaker voice feature vectors are voiceprint features extracted from speaker reference audio using a voiceprint feature extraction model (such as the ECAPA-TDNN (Enhanced Channel Attention Propagation and Convergence Delay Neural Network) model). These include a first voice feature vector corresponding to the first speaker and a second voice feature vector corresponding to the second speaker. For example, to humanize the video audio while preserving the voices of the doctor and patient, audio recordings of the doctor and patient's speech are used as reference audio to extract voiceprint features.

[0021] S102. Based on a preset dual-channel large language model, the dialogue text and the speaker's speech feature vector are processed to generate, in parallel, a first semantic token sequence corresponding to the first speaker and a second semantic token sequence corresponding to the second speaker. In one embodiment, the dual-channel large language model adopts a Transformer architecture similar to LLaMA (Large Language Model MetaAI), and is improved to a dual-channel encoder layer and decoder layer to achieve parallel output of dual channels.

[0022] Specifically, when the dual-channel large language model receives the dialogue text and the speaker's speech feature vector, it first encodes the dialogue text into word embeddings and maps them into text embedding vectors of the same dimension as the speaker's speech feature vectors, and then fuses them to obtain the conditional vectors for each channel.

[0023] Then, the dual-channel encoder and decoder layers in the dual-channel large language model implement parallel processing of the two channels to generate a first semantic token stream and a second semantic token stream. Specifically, semantic tokens are generated for active speakers (i.e., those currently speaking), and silence tokens are generated for inactive speakers (i.e., those not speaking). <sil>Explicitly representing temporal dynamics, such as S1: [token, token, <sil> ]，S2: [ <sil> , <sil>, token).

[0024] In another embodiment, the encoder layer of the dual-channel large language model also introduces a causal cross-attention mechanism, enabling the two channels to perceive each other, solving the interactive dynamics problem that traditional single-stream cannot express, and improving the interactivity of dialogue speech.

[0025] S103. Divide the first semantic token sequence and the second semantic token sequence according to the first preset length to obtain the first token block set and the second token block set; In one embodiment, a block-based mechanism is employed to handle long sequences and reduce memory introduction in subsequent processing. Specifically, the first semantic token sequence and the second semantic token sequence are divided according to a first preset length, and an index is assigned to each token block to obtain the first token block set corresponding to the first semantic token sequence and the second token block set corresponding to the second semantic token sequence. The first preset length *b* is the length of each token block, which can be set by the user according to actual needs. For example, the first semantic token sequence S1 is divided into n token blocks: A1, A2, ..., An, and the first token block set is {A1, A2, ..., An}. The second semantic token sequence S1 is divided into n token blocks: B1, B2, ..., Bn, and the second token block set is {B1, B2, ..., Bn}, with each token block containing *b* tokens.

[0026] S104. Based on a preset conditional flow matching model, the first speech feature vector and the first token block set are reconstructed to obtain a first Mel spectrum. In parallel, the second speech feature vector and the second token block set are reconstructed to obtain a second Mel spectrum. In one embodiment, token blocks from the first and second token block sets are transmitted sequentially to the conditional flow matching model. Upon receiving a token block, the conditional flow matching model constructs a conditional vector based on the current token block, the speaker's speech feature vector corresponding to the current token block, and the preceding and following token blocks corresponding to the current token block.

[0027] For the current token block, the conditional flow matching model uses the conditional vector as a guide to predict the vector field, performs vector field estimation, and obtains the Mel spectrum block corresponding to the current token block by stepwise integration from the noise along the vector field through the ODE (Ordinary Differential Equation) solver.

[0028] After obtaining the Mel spectrum blocks of all token blocks in the first token block set and all token blocks in the second token block set, inter-block fusion is performed on the Mel spectrum blocks corresponding to the first token block set and the second token block set to eliminate the acoustic discontinuity of the block boundary, and the first Mel spectrum and the second Mel spectrum are obtained respectively.

[0029] S105. The first Mel spectrum and the second Mel spectrum are converted to obtain the first waveform and the second waveform respectively, and the first waveform and the second waveform are aligned and merged to obtain the target dialogue audio.

[0030] In one embodiment, a pre-trained neural vocoder (such as the BigVGAN (Large-scale Vocoder Generative Adversarial Network) model) is used to convert the first Mel spectrum and the second Mel spectrum into a first waveform and a second waveform, respectively.

[0031] Based on the timestamp information of the first and second semantic token sequences, the first and second waveforms are aligned. Specifically, waveform segments that do not overlap in time are directly spliced ​​according to their timestamp information. For waveform segments that overlap in time, short-time windowing or acoustic energy weighted mixing is performed.

[0032] In another embodiment, the audio obtained after waveform alignment and merging is post-processed, including loudness normalization, residual noise suppression, etc., to obtain the target dialogue audio.

[0033] Further, the step of aligning and merging the first waveform and the second waveform to obtain the target dialogue audio includes: aligning the first waveform and the second waveform in time based on the timestamp information of the first semantic token sequence and the second semantic token sequence, and determining the waveform segments with overlapping times, wherein the time overlap includes partial overlap and complete overlap; performing weighted mixing processing on the partially overlapping waveform segments and windowing superposition processing on the completely overlapping waveform segments to obtain the target dialogue audio.

[0034] In one embodiment, token timestamps are mapped to waveform sampling points, and a global timestamp lookup table is constructed between the first semantic token sequence and the second semantic token sequence to ensure that the waveform segments corresponding to each token are precisely aligned on the time axis. Dynamic time warping is then performed to correct minor time offsets caused by vocoder synthesis delays or token prediction errors, and the time axis is further fine-tuned by detecting the energy peak positions of speech segments (such as word stress and sentence-ending pauses).

[0035] In one embodiment, based on the aligned timeline, the interaction state (non-overlapping, partially overlapping, completely overlapping) of the first and second waveforms is identified, and a differentiated mixing strategy is adopted to preserve the interaction intent (such as interruption, response, background noise) and avoid signal distortion. Partial overlap refers to partial temporal overlap of speech segments, while complete overlap refers to complete temporal overlap of the speech segments of two speakers or an overlap ratio greater than a preset threshold, such as when the first and second speakers are arguing and speaking simultaneously in a claims communication meeting.

[0036] For the non-overlapping waveform segments of the first and second waveforms, they are directly spliced ​​and filled with silence.

[0037] For partially overlapping waveform segments, an energy-weighted dynamic mixing strategy is adopted. Specifically, the short-time energy envelopes of the first and second waveforms within the overlapping segment are calculated, and their weights are also calculated. ,in, and Let be the short-time energy of the first and second waveforms at time t. To prevent division by zero, parameters are used to avoid division by zero. Weighted mixing is performed based on the calculated weights to obtain the target waveform. ,in, This is the first waveform. This is the second waveform.

[0038] For completely overlapping waveform segments, window functions (such as Hamming window and Hanning window) are used to divide the overlapping waveform segments into short-time windows. The signals in each window are weighted and superimposed to generate the final target waveform.

[0039] In one embodiment, the obtained target waveform is processed to obtain target dialogue audio.

[0040] The above embodiments provide a speech generation method based on dual-channel semantic tokens and block-based conditional flow matching. By performing parallel processing of semantics in both channels, a first semantic token sequence and a second semantic token sequence are generated. The two channels can perceive each other's interactive dynamic information and accurately obtain the interactive information between speakers, solving the problem that a single channel cannot extract interactive dynamic information and improving the naturalness and interactivity of the synthesized speech. In addition, by dividing the long token sequence into fixed-length token blocks and processing each token block using a conditional flow matching model, high-quality synthesis is achieved, further improving the naturalness of the synthesized speech.

[0041] Please see Figure 2 , Figure 2 This is a schematic flowchart illustrating a speech generation method based on dual-channel semantic tokens and block conditional flow matching, provided in an embodiment of this application. This speech generation method based on dual-channel semantic tokens and block conditional flow matching can be applied in a server to generate more natural speaker switching, interruption, and overlap behaviors through parallel dual-channel semantic modeling and explicit representation of silence / overlap markers at the semantic layer. This provides a token sequence containing interactive features for subsequent spectrogram reconstruction and audio synthesis, thereby improving the naturalness and interactivity of the synthesized speech.

[0042] like Figure 2 As shown, step S102 of the speech generation method based on dual-channel semantic tokens and block conditional stream matching specifically includes steps S201 to S203.

[0043] S201. Based on the input layer of the dual-channel large language model, the dialogue text and the speaker's speech feature vector are encoded and fused to generate the first condition vector of the first channel and the second condition vector of the second channel in the dual-channel encoder layer. In one embodiment, by fusing the dialogue text with the speaker's speech feature vector, an input condition vector for a dual-channel encoder is generated, thereby achieving a unified representation of semantic and identity features.

[0044] Specifically, the dialogue text contains <spkchange>The markers explicitly indicate the speaker switching sequence, for example: "First speaker: Hello!" <spkchange>Second speaker: Hello! How can I help you? The dialogue text is split into sub-word sequences to obtain the word sequences corresponding to the first speaker and the second speaker. The word sequences corresponding to the first speaker are mapped to the first text embedding vector aligned with the first speech feature vector, and the word sequences corresponding to the second speaker are mapped to the second text embedding vector aligned with the second speech feature vector, while preserving the contextual semantic structure (such as the difference in tone between questions and exclamations).

[0045] The mapped text embedding vector and the speaker's speech feature vector are fused by position-by-position addition to generate the input condition vector of the dual-channel encoder. Specifically, the first text embedding vector and the first speech feature vector are fused by position-by-position addition to generate the first condition vector of the first channel, and the second text embedding vector and the second speech feature vector are fused by position-by-position addition to generate the second condition vector of the second channel.

[0046] S202. Construct a causal cross-attention mask based on the dual-channel encoder layer, and extract features from the first conditional vector and the second conditional vector based on the causal cross-attention mask to obtain a first hidden state sequence and a second hidden state sequence. In one embodiment, the encoder layer extracts features based on conditional vectors and constructs the interaction dependency between the two channels through a causal cross-attention mechanism to model the speaker's interaction dynamics and generate a hidden state sequence containing interaction information.

[0047] In a specific embodiment, the dual-channel encoder layer first applies a self-attention mechanism to the first condition vector and the second condition vector, respectively. Based on self-attention, the encoder introduces causal cross-attention, enabling the two channels to perceive each other. The causal cross-attention mask matrix is ​​defined as follows: (The lower triangular mask ensures causality) That is, the query at time step i can only focus on the key and value of the other party's channel at time steps j ≤ i, thus avoiding future information leakage.

[0048] For example, the hidden state of the first channel is used as the query (Q1), and the hidden state of the second channel is used as the key (K2) and value (V2). The causal cross-attention weights are calculated based on the causal mask matrix. Constraints ensure that each location can only access current and historical information (to prevent future leaks). Symmetrically, the second channel also uses the first channel as a reference to achieve bidirectional interaction. For example, when the first speaker generates "I think...", the second channel can perceive this semantics through cross-attention and then generate response tokens such as "Yes, I agree..." to achieve a natural interjection response.

[0049] The dual-channel encoder outputs the first hidden state sequence and the second hidden state sequence, which contain text semantics, speaker identity and interaction dynamic features.

[0050] S203. The first hidden state sequence and the second hidden state sequence are decoded based on the dual-channel decoder layer to obtain the first semantic token sequence and the second semantic token sequence.

[0051] In one embodiment, based on the first hidden state sequence and the second hidden state sequence, a sequence containing hidden states is generated through parallel decoding. <sil>A two-channel semantic token stream that explicitly expresses the "say / stop / overlap" state.

[0052] The dual-channel decoder (corresponding one-to-one with the encoder channels) independently processes the first and second hidden state sequences, respectively. Each layer includes self-attention and causal cross-attention. It generates content tokens (such as semantic tokens corresponding to words) for active speakers and generates content tokens (such as semantic tokens corresponding to words) for inactive speakers. <sil>Silence marker. Specifically, starting from the initial time step (t=1), the current token is predicted based on the received hidden state sequence and historical tokens (such as the output of step t-1). The prediction result of each step is used as the input for the next time step, and the process is repeated until the end of the sequence.

[0053] In the above embodiments, by using dual-channel semantic parallel modeling and explicitly representing silence / overlap markers at the semantic layer, more natural speaker switching, interruption, and overlap behaviors are generated, providing token sequences containing interactive features for subsequent spectrogram reconstruction and audio synthesis, thereby improving the naturalness and interactivity of synthesized speech.

[0054] Further, before step S101, the method includes: acquiring training audio and segmenting the training audio into segments according to a second preset length to obtain at least one audio segment; performing quality screening on each audio segment to obtain at least one target audio segment, and preprocessing each target audio segment to obtain the dialogue text and speaker annotations corresponding to each target audio segment; performing speaker separation on each target audio segment based on the dialogue text and speaker annotations to obtain the audio corresponding to each speaker; encoding and extracting voiceprint features from the audio corresponding to each speaker to obtain the target semantic token sequence and voiceprint embedding vector corresponding to each speaker; and training the pre-trained model based on the dialogue text corresponding to the target audio segment, the target semantic token sequence corresponding to each speaker, and the voiceprint embedding vector to obtain the dual-channel large language model.

[0055] In one embodiment, training audio can be extracted from public dialogue datasets, podcast / interview long audios, and film / television dialogues. The extracted audio is then subjected to quality screening and preprocessing to filter out low-quality audio, single-speaker monologues, and unnatural dialogues (such as script readings), while retaining samples containing natural interactions (such as overlaps and interjections).

[0056] In one embodiment, FFmpeg (an audio and video processing toolkit) is used to cut long audio into at least one audio segment according to a second preset length (configurable to a fixed length of 10-30 minutes, which can be set by the user according to actual needs) to balance contextual integrity and processing efficiency. A 5-second overlap is reserved between segments to avoid semantic breaks in the dialogue (such as truncating sentences across segments).

[0057] In one embodiment, high-quality target audio segments are selected and retained through quality detection. Quality detection includes SNR (signal-to-noise ratio) detection, cluster consistency detection, embedding similarity detection, and DNSMOS scoring. High-quality target audio segments are selected based on threshold values ​​for each quality metric.

[0058] In one embodiment, VAD (Voice Activity Detection), ASR (Automatic Speech Recognition), Speaker Diarization, Punctuation Restoration, and OSD (Overlapping Speech Detection) are performed in parallel for each target audio segment to obtain speech segments, non-speech segments, dialogue text, speaker annotations, and overlapping speech regions.

[0059] Among them, VAD (Voice Activity Detection) is used to distinguish between speech and non-speech segments and mark silent intervals. <sil>ASR (Automatic Speech Recognition) is used to convert speech into text and output word-by-word timestamps; Speaker Diarization is used to identify the number of speakers and speaking intervals, and generate speaker labels (such as Speaker A / B); Punctuation Restoration is used to add punctuation (commas, periods, question marks) to the dialogue text obtained by speech recognition, and restore natural language pauses, which can be achieved using a BERT-based punctuation prediction model; OSD (Overlapping Speech Detection) is used to detect overlapping intervals when multiple speakers speak at the same time (such as interruptions).

[0060] In one embodiment, a speech separation tool (such as SpatialNet) is used to perform speaker separation on the target audio segment using spatial audio information or spectral masks, decomposing the mixed audio into independent streams to obtain the clean audio of each speaker. The separated single-speaker audio is then subjected to secondary VAD filtering to remove residual silence segments and retain valid speech.

[0061] In one embodiment, a pre-trained S3Tokenizer model (an improved discretization model based on HuBERT) is used to convert the audio waveform of each speaker into a Mel spectrum and extract hidden features. K-means clustering is then performed on the hidden features to discretize the features into semantic tokens, thereby obtaining the semantic token sequence corresponding to each speaker's audio.

[0062] In one embodiment, a voiceprint feature extraction model is used to extract the speaker's voiceprint embedding vector from the audio corresponding to each speaker through multi-layer convolution and attention mechanisms.

[0063] In one embodiment, the pre-trained model is based on an LLaMA-like Transformer architecture, extended to a dual-channel architecture, and incorporates causal cross-attention to enable mutual perception between the two channels at the encoder layer, modeling interaction dependencies. The pre-trained model is trained based on the dialogue text corresponding to each target audio segment, the target semantic token sequence corresponding to each speaker, and the speakerprint embedding vector. The training objective is to minimize the dual-channel cross-entropy. ,in, The training data consists of a sequence of real semantic tokens, and T is the input dialogue text. For the currently predicted token sequence, These are the model parameters.

[0064] Further, the step of training the pre-trained model based on the dialogue text corresponding to the target audio segment, the target semantic token sequence corresponding to each speaker, and the voiceprint embedding vector to obtain the dual-channel large language model includes: processing the dialogue text corresponding to the target audio segment and the voiceprint embedding vector based on the pre-trained model, and predicting the third semantic token sequence and the fourth semantic token sequence of the third speaker and the fourth speaker in parallel; determining the dual-channel cross loss value based on the third semantic token sequence, the fourth semantic token sequence, and the target semantic token sequence corresponding to each speaker; and obtaining the dual-channel large language model when the dual-channel cross loss value is minimized.

[0065] In one embodiment, the pre-trained model architecture is based on an LLaMA-like Transformer architecture, extended to a dual-channel approach at the encoder and decoder layers to simultaneously predict token sequences from two speakers. Causal cross-attention is introduced at the encoder layer, enabling the two channels to be mutually aware. The first channel focuses on historical information (already generated sequences) from the second channel, and vice versa, to model speaker interactions (such as interruptions and overlaps). The attention mask is set to a lower triangular matrix to ensure that only historical information is utilized, avoiding future token leakage.

[0066] At each time step, the pre-trained model generates content tokens (such as semantic tokens corresponding to words) for active speakers and silence tokens for inactive speakers. <sil>Explicitly representing the "speak / stop / overlap" state, we obtain the third semantic token sequence of the third speaker and the fourth semantic token sequence of the fourth speaker.

[0067] Based on the predicted third and fourth semantic token sequences and the corresponding real semantic token sequences of the third and fourth speakers, the two-channel cross-entropy loss is calculated. The two-channel cross-loss function is: .

[0068] In one embodiment, a gradient descent algorithm (such as AdamW) is used to iteratively update the parameters. During each forward propagation, the model predicts the probability distributions of the third and fourth semantic token sequences in parallel, calculates the two-channel cross-entropy loss, and then backpropagates the gradient. Training stops when the two-channel cross-entropy loss reaches its minimum, resulting in a two-channel large language model.

[0069] In another embodiment, a conditional flow matching architecture is used for token-spectrum reconstruction. Based on the semantic token sequence corresponding to the target audio segment and the target Mel spectrum, the model is trained to predict continuous noise samples. To the target Mel spectrum block Mapping vector field Obtain the prediction vector field The optimization objective is to minimize , For the flow matching path, t∈[0,1] is the time parameter. is the parameter of the conditional flow matching model, and c is the fusion conditional vector of the context embedding matrix corresponding to the token block and the speaker's speech feature vector.

[0070] In the above embodiments, high-quality, aligned dual-channel training data (including real overlap and interaction patterns) is extracted from publicly available long audio files to alleviate the problem of scarce high-quality dual-channel labeled data. Zero-sample speaker cloning is achieved by combining speaker speech feature vectors. Causal cross-attention is introduced to enable the two channels to perceive and model each other at the encoder level, enhancing the learning of interjections and collaborative speech behaviors and solving the interactive dynamics problem that traditional single-stream methods cannot express.

[0071] Please see Figure 3 , Figure 3 This is a schematic flowchart illustrating a speech generation method based on dual-channel semantic tokens and block-based conditional flow matching, provided in an embodiment of this application. This speech generation method based on dual-channel semantic tokens and block-based conditional flow matching can be applied to servers to achieve high-quality, low-latency Mel spectrum generation in long dialogue scenarios through block-based conditional flow matching and block-level attention masks, reducing memory and latency without sacrificing acoustic quality.

[0072] like Figure 3 As shown, step S103 of the speech generation method based on dual-channel semantic tokens and block conditional stream matching specifically includes steps S301 to S305.

[0073] S301. Transmit each first token block in the first token block set to the conditional flow matching model in sequence, and transmit each second token block in the second token block set to the conditional flow matching model in parallel in sequence; In one embodiment, the token blocks from the first and second token block sets obtained after block division are input into the conditional flow matching model block by block in chronological order to ensure the temporal consistency of block processing.

[0074] In one embodiment, the conditional flow matching model supports parallel processing, which can perform dual-channel parallel processing on token blocks of the first token block set and the second token block set to improve processing efficiency, but the token blocks within the set still need to be processed sequentially.

[0075] S302. Based on the conditional flow matching model, construct a block attention mask according to a preset inter-block offset threshold; In one embodiment, to reduce long sequence memory overhead while preserving cross-block dependencies, a mask is used to restrict the model to focus only on local contexts.

[0076] Specifically, a unique index is assigned to each token block, and an inter-block offset threshold τ (e.g., τ=2) is set to construct a block attention mask. This indicates that the current token block i only establishes attention connections with its τ adjacent blocks j before and after it. The mask mode can be Causal mode (only allowing access to historical adjacent blocks, supporting streaming inference) or History / Future mode (allowing access to the τ adjacent blocks before and after it, balancing context integrity). S303. Based on the block attention mask and the index of each first token block, obtain the first adjacent block corresponding to each first token block, and based on the block attention mask and the index of each second token block, obtain the second adjacent block corresponding to each second token block. In one embodiment, based on the block attention mask and the index of the first token block currently input to the conditional flow matching model from the first token block set, a token block within the context window of the first current token block is loaded as the adjacent block corresponding to the first current token block, i.e., the first adjacent block. Based on the block attention mask and the index of the second token block currently input to the conditional flow matching model from the second token block set, a token block within the context window of the second current token block is loaded as the adjacent block corresponding to the second current token block, i.e., the second adjacent block.

[0077] S304. Process each first token block, each first adjacent block and the first speech feature vector to obtain a first Mel spectrum block corresponding to each first token block, and process each second token block, each second adjacent block and the second speech feature vector to obtain a second Mel spectrum block corresponding to each second token block. Further, step S304 includes: based on the conditional flow matching model, performing context feature encoding on each first token block and the first adjacent block to obtain a first context embedding matrix, and performing context feature encoding on each second token block and the second adjacent block to obtain a second context embedding matrix; performing vector field estimation based on each first context embedding matrix and the first speech feature vector to obtain a first Mel spectrum block corresponding to each first token block, and performing vector field estimation based on each second context embedding matrix and the second speech feature vector to obtain a second Mel spectrum block corresponding to each second token block.

[0078] In one embodiment, the token blocks in the first and second token block sets currently received by the conditional flow matching model are used as the first and second current token blocks, respectively. A block-based attention layer is used to perform context encoding on the tokens of the first current token block and its neighboring blocks to obtain a first context embedding matrix. Similarly, a block-based attention layer is used to perform context encoding on the tokens of the second current token block and its neighboring blocks to obtain a second context embedding matrix. The first context embedding matrix and the first speech feature vector are fused using adaptive instance normalization to generate a conditional vector c. The second context embedding matrix and the second speech feature vector are also fused using adaptive instance normalization to generate another conditional vector c.

[0079] In one embodiment, the prediction vector field is obtained based on each condition vector c. Discretize the time interval [0,1] into K=64 steps, and solve for the ODE using the Euler method: Integrating from t=0 to 1 yields the first Mel spectrum block and the second Mel spectrum block corresponding to each first token block and each second token block.

[0080] In one embodiment, stream matching possesses the mathematical properties of convergence stability and efficient sampling. Conditional stream matching, by using semantics and speaker information as conditions, reduces the distribution mapping problem to a conditional vector field estimation problem, enabling high-quality synthesis while preserving fine-grained acoustic details.

[0081] S305. Perform continuous integration on each of the first Mel spectrum blocks to obtain the first Mel spectrum, and perform continuous integration on each of the second Mel spectrum blocks to obtain the second Mel spectrum.

[0082] In one embodiment, each first Mel spectrum block corresponding to the first semantic token sequence and each second Mel spectrum block corresponding to the second semantic token sequence are concatenated in chronological order. The discrete Mel spectrum blocks are concatenated into a complete and continuous Mel spectrum, eliminating acoustic abrupt changes at block boundaries. The first Mel spectrum is formed by integrating all Mel spectrum blocks of the first semantic token sequence, and the second Mel spectrum is formed by integrating all Mel spectrum blocks of the second semantic token sequence; these two spectra correspond to the acoustic feature trajectories of the first and second speakers, respectively.

[0083] In a specific embodiment, during the generation of Mel spectrum blocks, each block contains a small amount of overlapping regions, such as a 10% overlap between the end of one block and the beginning of the next. During stitching, the overlapping regions are windowed and cross-faded (e.g., using a Hanning window) to achieve a smooth transition. The windowing fusion formula is as follows: ,in, For continuous Mel spectrum, For the pre-Mel spectrum block The weight of the overlapping region, For post-Mel spectrum block The weight of the overlapping region.

[0084] In one embodiment, energy envelope alignment is performed on each complete Mel spectrum after splicing. Specifically, the average energy of the overlapping region of adjacent blocks is calculated, and energy jumps between blocks are corrected by linear interpolation (such as smoothly transitioning the starting energy of the later block to the ending energy of the previous block) to avoid "sudden volume changes" in the listening experience.

[0085] In the above embodiments, block-based conditional flow matching and block-level attention masks ensure that minute-level dialogues can be processed with fixed memory overhead. This achieves high-quality, low-latency Mel spectrum generation in long dialogue scenarios, reducing memory and latency without sacrificing acoustic quality. It is suitable for real-time or near-real-time dialogue synthesis scenarios (such as voice assistants and broadcast generation), and supports streaming inference, significantly expanding the system's engineering applicability and deployment flexibility.

[0086] Please see Figure 4 , Figure 4 This application provides a schematic block diagram of a speech generation apparatus based on dual-channel semantic tokens and block conditional stream matching, which is used to execute the aforementioned speech generation method based on dual-channel semantic tokens and block conditional stream matching. The speech generation apparatus based on dual-channel semantic tokens and block conditional stream matching can be configured on a server.

[0087] like Figure 4 As shown, the speech generation device 400 based on dual-channel semantic tokens and block conditional stream matching includes: The text and speech feature acquisition module 401 is used to acquire dialogue text and speaker speech feature vectors, wherein the speaker speech feature vectors include a first speech feature vector corresponding to a first speaker and a second speech feature vector corresponding to a second speaker. The semantic token sequence generation module 402 is used to process the dialogue text and the speaker's speech feature vector based on a preset dual-channel large language model, and generate in parallel a first semantic token sequence corresponding to the first speaker and a second semantic token sequence corresponding to the second speaker. The semantic token sequence segmentation module 403 is used to segment the first semantic token sequence and the second semantic token sequence according to a first preset length to obtain a first token block set and a second token block set. The Mel spectrum generation module 404 is used to perform spectrum reconstruction on the first speech feature vector and the first token block set based on a preset conditional flow matching model to obtain a first Mel spectrum, and to perform spectrum reconstruction on the second speech feature vector and the second token block set in parallel to obtain a second Mel spectrum. The target dialogue audio acquisition module 405 is used to convert the first Mel spectrum and the second Mel spectrum respectively to obtain a first waveform and a second waveform, and to align and merge the first waveform and the second waveform to obtain the target dialogue audio.

[0088] Further, the semantic token sequence generation module 402 includes: The condition vector generation unit is used to encode and fuse the dialogue text and the speaker's speech feature vector based on the input layer of the dual-channel large language model, and generate the first condition vector of the first channel and the second condition vector of the second channel in the dual-channel encoder layer. The state sequence generation unit is used to construct a causal cross-attention mask based on the dual-channel encoder layer, and to extract features from the first condition vector and the second condition vector based on the causal cross-attention mask to obtain a first hidden state sequence and a second hidden state sequence. The token sequence generation unit is used to decode the first hidden state sequence and the second hidden state sequence based on the dual-channel decoder layer to obtain the first semantic token sequence and the second semantic token sequence.

[0089] Furthermore, the Mel spectrum generation module 403 includes: The token block transmission unit is used to transmit each first token block in the first token block set to the conditional flow matching model in sequence, and to transmit each second token block in the second token block set to the conditional flow matching model in parallel. The attention mask construction unit is used to construct a block attention mask based on the conditional flow matching model and according to a preset inter-block offset threshold. The adjacent block acquisition unit is used to acquire the first adjacent block corresponding to each first token block according to the block attention mask and the index of each first token block, and to acquire the second adjacent block corresponding to each second token block according to the block attention mask and the index of each second token block. The Mel spectrum block acquisition unit is used to process each first token block, each first adjacent block and the first speech feature vector to obtain a first Mel spectrum block corresponding to each first token block, and to process each second token block, each second adjacent block and the second speech feature vector to obtain a second Mel spectrum block corresponding to each second token block. The Mel spectrum acquisition unit is used to continuously integrate each of the first Mel spectrum blocks to obtain the first Mel spectrum, and to continuously integrate each of the second Mel spectrum blocks to obtain the second Mel spectrum.

[0090] Furthermore, the Mel spectrum block obtaining unit includes: The context embedding matrix obtaining unit is used to perform context feature encoding on each first token block and the first adjacent block based on the conditional flow matching model to obtain a first context embedding matrix, and to perform context feature encoding on each second token block and the second adjacent block to obtain a second context embedding matrix. The Mel spectrum block acquisition unit is used to perform vector field estimation based on each of the first context embedding matrices and the first speech feature vectors to obtain the first Mel spectrum block corresponding to each of the first token blocks, and to perform vector field estimation based on each of the second context embedding matrices and the second speech feature vectors to obtain the second Mel spectrum block corresponding to each of the second token blocks.

[0091] Furthermore, the speech generation device 400 based on dual-channel semantic tokens and block conditional stream matching also includes a model acquisition module, which includes: An audio segment acquisition unit is used to acquire training audio and segment the training audio into segments according to a second preset length to obtain at least one audio segment. The target audio segment acquisition unit is used to perform quality screening on each of the audio segments to obtain at least one target audio segment, and to preprocess each of the target audio segments to obtain the dialogue text and speaker annotations corresponding to each of the target audio segments. The speaker audio acquisition unit is used to perform speaker separation on each of the target audio segments based on the dialogue text of the target audio segments and the speaker annotations, and obtain the audio corresponding to each speaker; The speaker audio processing unit is used to encode and extract voiceprint features from the audio of each speaker to obtain the target semantic token sequence and voiceprint embedding vector for each speaker. The model training unit is used to train the pre-trained model based on the dialogue text corresponding to the target audio segment, the target semantic token sequence corresponding to each speaker, and the voiceprint embedding vector to obtain the dual-channel large language model.

[0092] Furthermore, the model training unit includes: The token sequence prediction subunit is used to process the dialogue text corresponding to the target audio segment and the voiceprint embedding vector based on the pre-trained model, and to predict the third semantic token sequence and the fourth semantic token sequence of the third speaker and the fourth speaker in parallel. The loss value determination subunit is used to determine the dual-channel cross loss value based on the third semantic token sequence, the fourth semantic token sequence, and the target semantic token sequence corresponding to each speaker; The target model determination sub-unit is used to obtain the dual-channel large language model when the dual-channel cross loss value is minimized.

[0093] Furthermore, the target dialogue audio acquisition module 405 includes: A time alignment unit is used to perform time alignment on the first waveform and the second waveform based on the timestamp information of the first semantic token sequence and the second semantic token sequence, and to determine the waveform segments with time overlap, wherein the time overlap includes partial overlap and complete overlap; A waveform segment processing unit is used to perform weighted mixing processing on the partially overlapping waveform segments and windowing superposition processing on the completely overlapping waveform segments to obtain the target dialogue audio.

[0094] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus and modules can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0095] The aforementioned device can be implemented as a computer program, which can be used in, for example... Figure 5 It runs on the computer device shown.

[0096] Please see Figure 5 , Figure 5 This is a schematic block diagram illustrating the structure of a computer device according to an embodiment of this application. The computer device may be a server.

[0097] See Figure 5 The computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0098] The non-volatile storage medium can store an operating system and a computer program. This computer program includes program instructions that, when executed, cause the processor to perform any speech generation method based on dual-channel semantic tokens and block conditional stream matching.

[0099] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0100] Internal memory provides an environment for the execution of computer programs in non-volatile storage media. When executed by a processor, the computer program enables the processor to perform any speech generation method based on dual-channel semantic tokens and block conditional stream matching.

[0101] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0102] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0103] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps: Obtain the dialogue text and the speaker's speech feature vector, wherein the speaker's speech feature vector includes a first speech feature vector corresponding to a first speaker and a second speech feature vector corresponding to a second speaker; The dialogue text and the speaker's speech feature vector are processed based on a preset dual-channel large language model to generate a first semantic token sequence corresponding to the first speaker and a second semantic token sequence corresponding to the second speaker in parallel. According to the first preset length, the first semantic token sequence and the second semantic token sequence are divided into a first token block set and a second token block set respectively. Based on a preset conditional flow matching model, the first speech feature vector and the first token block set are reconstructed to obtain the first Mel spectrum, and the second speech feature vector and the second token block set are reconstructed in parallel to obtain the second Mel spectrum. The first Mel spectrum and the second Mel spectrum are converted to obtain a first waveform and a second waveform, respectively. The first waveform and the second waveform are then aligned and merged to obtain the target dialogue audio.

[0104] In one embodiment, when the processor processes the dialogue text and the speaker's speech feature vector based on a preset dual-channel large language model, and generates in parallel a first semantic token sequence corresponding to the first speaker and a second semantic token sequence corresponding to the second speaker, it is used to: The input layer of the dual-channel large language model encodes and fuses the dialogue text and the speaker's speech feature vector to generate the first condition vector of the first channel and the second condition vector of the second channel in the dual-channel encoder layer. A causal cross-attention mask is constructed based on the dual-channel encoder layer, and features are extracted from the first conditional vector and the second conditional vector based on the causal cross-attention mask to obtain the first hidden state sequence and the second hidden state sequence. The first hidden state sequence and the second hidden state sequence are decoded by a dual-channel decoder layer to obtain the first semantic token sequence and the second semantic token sequence.

[0105] In one embodiment, when the processor performs spectrum reconstruction on the first speech feature vector and the first token block set based on a preset conditional flow matching model to obtain a first Mel spectrum, and performs spectrum reconstruction on the second speech feature vector and the second token block set in parallel to obtain a second Mel spectrum, it is configured to: Each first token block in the first token block set is transmitted sequentially to the conditional flow matching model, and each second token block in the second token block set is transmitted sequentially to the conditional flow matching model in parallel. Based on the conditional flow matching model, a block attention mask is constructed according to a preset inter-block offset threshold; Based on the block attention mask and the index of each first token block, obtain the first adjacent block corresponding to each first token block, and based on the block attention mask and the index of each second token block, obtain the second adjacent block corresponding to each second token block. Each first token block, each first adjacent block, and the first speech feature vector are processed to obtain a first Mel spectrum block corresponding to each first token block, and each second token block, each second adjacent block, and the second speech feature vector are processed to obtain a second Mel spectrum block corresponding to each second token block. The first Mel spectrum blocks are continuously integrated to obtain the first Mel spectrum, and the second Mel spectrum blocks are continuously integrated to obtain the second Mel spectrum.

[0106] In one embodiment, when the processor processes each first token block, each first adjacent block, and the first speech feature vector to obtain a first Mel spectrum block corresponding to each first token block, and processes each second token block, each second adjacent block, and the second speech feature vector to obtain a second Mel spectrum block corresponding to each second token block, the processor is configured to: Based on the conditional flow matching model, context feature encoding is performed on each first token block and the first adjacent block to obtain a first context embedding matrix, and context feature encoding is performed on each second token block and the second adjacent block to obtain a second context embedding matrix. Vector field estimation is performed based on each of the first context embedding matrices and the first speech feature vectors to obtain the first Mel spectrum block corresponding to each of the first token blocks. Vector field estimation is performed based on each of the second context embedding matrices and the second speech feature vectors to obtain the second Mel spectrum block corresponding to each of the second token blocks.

[0107] In one embodiment, before acquiring the dialogue text and speaker speech feature vectors, processing the dialogue text and speaker speech feature vectors based on a preset dual-channel large language model, and generating in parallel a first semantic token sequence corresponding to the first speaker and a second semantic token sequence corresponding to the second speaker, the processor is further configured to: Acquire training audio and segment the training audio into segments according to a second preset length to obtain at least one audio segment; Each audio segment is subjected to quality screening to obtain at least one target audio segment, and each target audio segment is preprocessed to obtain the dialogue text and speaker annotations corresponding to each target audio segment. Based on the dialogue text of the target audio segment and the speaker annotation, speaker separation is performed on each target audio segment to obtain the audio corresponding to each speaker; The audio of each speaker is encoded and voiceprint features are extracted to obtain the target semantic token sequence and voiceprint embedding vector for each speaker. Based on the dialogue text corresponding to the target audio segment, the target semantic token sequence corresponding to each speaker, and the voiceprint embedding vector, the pre-trained model is trained to obtain the dual-channel large language model.

[0108] In one embodiment, when the processor trains the pre-trained model based on the dialogue text corresponding to the target audio segment, the target semantic token sequence corresponding to each speaker, and the voiceprint embedding vector to obtain the dual-channel large language model, it is used to implement: Based on the pre-trained model, the dialogue text corresponding to the target audio segment and the voiceprint embedding vector are processed, and the third semantic token sequence and the fourth semantic token sequence of the third speaker and the fourth speaker are predicted in parallel. Based on the third semantic token sequence, the fourth semantic token sequence, and the target semantic token sequence corresponding to each speaker, the dual-channel cross loss value is determined. The dual-channel large language model is obtained when the dual-channel cross-loss value is minimized.

[0109] In one embodiment, when the processor aligns and merges the first waveform and the second waveform to obtain the target dialogue audio, it is configured to: Based on the timestamp information of the first semantic token sequence and the second semantic token sequence, the first waveform and the second waveform are time-aligned, and waveform segments with time overlap are determined, wherein the time overlap includes partial overlap and complete overlap; The target dialogue audio is obtained by performing weighted mixing processing on the partially overlapping waveform segments and windowing superposition processing on the completely overlapping waveform segments.

[0110] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the speech generation methods based on dual-channel semantic tokens and block conditional stream matching provided in the embodiments of this application.

[0111] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.

[0112] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.< / sil> < / sil> < / sil> < / sil> < / spkchange> < / spkchange> < / sil> < / sil> < / sil> < / sil>

Claims

1. A speech generation method based on dual-channel semantic tokens and block conditional stream matching, characterized in that, include: Obtain the dialogue text and the speaker's speech feature vector, wherein the speaker's speech feature vector includes a first speech feature vector corresponding to a first speaker and a second speech feature vector corresponding to a second speaker; The dialogue text and the speaker's speech feature vector are processed based on a preset dual-channel large language model to generate a first semantic token sequence corresponding to the first speaker and a second semantic token sequence corresponding to the second speaker in parallel. According to the first preset length, the first semantic token sequence and the second semantic token sequence are divided into a first token block set and a second token block set respectively. Based on a preset conditional flow matching model, the first speech feature vector and the first token block set are reconstructed to obtain the first Mel spectrum, and the second speech feature vector and the second token block set are reconstructed in parallel to obtain the second Mel spectrum. The first Mel spectrum and the second Mel spectrum are converted to obtain a first waveform and a second waveform, respectively. The first waveform and the second waveform are then aligned and merged to obtain the target dialogue audio.

2. The speech generation method based on dual-channel semantic tokens and block conditional stream matching according to claim 1, characterized in that, The process, based on a preset dual-channel large language model, processes the dialogue text and the speaker's speech feature vector, and generates in parallel a first semantic token sequence corresponding to the first speaker and a second semantic token sequence corresponding to the second speaker, including: The input layer of the dual-channel large language model encodes and fuses the dialogue text and the speaker's speech feature vector to generate the first condition vector of the first channel and the second condition vector of the second channel in the dual-channel encoder layer. A causal cross-attention mask is constructed based on the dual-channel encoder layer, and features are extracted from the first conditional vector and the second conditional vector based on the causal cross-attention mask to obtain the first hidden state sequence and the second hidden state sequence. The first hidden state sequence and the second hidden state sequence are decoded by a dual-channel decoder layer to obtain the first semantic token sequence and the second semantic token sequence.

3. The speech generation method based on dual-channel semantic tokens and block conditional stream matching according to claim 1, characterized in that, The first speech feature vector and the first token block set are reconstructed based on a preset conditional flow matching model to obtain a first Mel spectrum, and the second speech feature vector and the second token block set are reconstructed in parallel to obtain a second Mel spectrum, including: Each first token block in the first token block set is transmitted sequentially to the conditional flow matching model, and each second token block in the second token block set is transmitted sequentially to the conditional flow matching model in parallel. Based on the conditional flow matching model, a block attention mask is constructed according to a preset inter-block offset threshold; Based on the block attention mask and the index of each first token block, obtain the first adjacent block corresponding to each first token block, and based on the block attention mask and the index of each second token block, obtain the second adjacent block corresponding to each second token block. Each first token block, each first adjacent block, and the first speech feature vector are processed to obtain a first Mel spectrum block corresponding to each first token block, and each second token block, each second adjacent block, and the second speech feature vector are processed to obtain a second Mel spectrum block corresponding to each second token block. The first Mel spectrum blocks are continuously integrated to obtain the first Mel spectrum, and the second Mel spectrum blocks are continuously integrated to obtain the second Mel spectrum.

4. The speech generation method based on dual-channel semantic tokens and block conditional stream matching according to claim 3, characterized in that, The step of processing each first token block, each first adjacent block, and the first speech feature vector to obtain a first Mel spectrum block corresponding to each first token block, and processing each second token block, each second adjacent block, and the second speech feature vector to obtain a second Mel spectrum block corresponding to each second token block, includes: Based on the conditional flow matching model, context feature encoding is performed on each first token block and the first adjacent block to obtain a first context embedding matrix, and context feature encoding is performed on each second token block and the second adjacent block to obtain a second context embedding matrix. Vector field estimation is performed based on each of the first context embedding matrices and the first speech feature vectors to obtain the first Mel spectrum block corresponding to each of the first token blocks. Vector field estimation is performed based on each of the second context embedding matrices and the second speech feature vectors to obtain the second Mel spectrum block corresponding to each of the second token blocks.

5. The speech generation method based on dual-channel semantic tokens and block conditional stream matching according to claim 1, characterized in that, Before processing the dialogue text and the speaker's speech feature vector based on a preset dual-channel large language model, and generating in parallel the first semantic token sequence corresponding to the first speaker and the second semantic token sequence corresponding to the second speaker, the method further includes: Acquire training audio and segment the training audio into segments according to a second preset length to obtain at least one audio segment; Each audio segment is subjected to quality screening to obtain at least one target audio segment, and each target audio segment is preprocessed to obtain the dialogue text and speaker annotations corresponding to each target audio segment. Based on the dialogue text of the target audio segment and the speaker annotation, speaker separation is performed on each target audio segment to obtain the audio corresponding to each speaker; The audio of each speaker is encoded and voiceprint features are extracted to obtain the target semantic token sequence and voiceprint embedding vector for each speaker. Based on the dialogue text corresponding to the target audio segment, the target semantic token sequence corresponding to each speaker, and the voiceprint embedding vector, the pre-trained model is trained to obtain the dual-channel large language model.

6. The speech generation method based on dual-channel semantic tokens and block conditional stream matching according to claim 5, characterized in that, The process of training a pre-trained model based on the dialogue text corresponding to the target audio segment, the target semantic token sequence corresponding to each speaker, and the voiceprint embedding vector to obtain the dual-channel large language model includes: Based on the pre-trained model, the dialogue text corresponding to the target audio segment and the voiceprint embedding vector are processed, and the third semantic token sequence and the fourth semantic token sequence of the third speaker and the fourth speaker are predicted in parallel. Based on the third semantic token sequence, the fourth semantic token sequence, and the target semantic token sequence corresponding to each speaker, the dual-channel cross loss value is determined. The dual-channel large language model is obtained when the dual-channel cross-loss value is minimized.

7. The speech generation method based on dual-channel semantic tokens and block conditional stream matching according to any one of claims 1 to 6, characterized in that, The step of aligning and merging the first waveform and the second waveform to obtain the target dialogue audio includes: Based on the timestamp information of the first semantic token sequence and the second semantic token sequence, the first waveform and the second waveform are time-aligned, and waveform segments with time overlap are determined, wherein the time overlap includes partial overlap and complete overlap; The target dialogue audio is obtained by performing weighted mixing processing on the partially overlapping waveform segments and windowing superposition processing on the completely overlapping waveform segments.

8. A speech generation device based on dual-channel semantic tokens and block conditional stream matching, characterized in that, include: The text and speech feature acquisition module is used to acquire dialogue text and speaker speech feature vectors, wherein the speaker speech feature vectors include a first speech feature vector corresponding to a first speaker and a second speech feature vector corresponding to a second speaker. The semantic token sequence generation module is used to process the dialogue text and the speaker's speech feature vector based on a preset dual-channel large language model, and generate in parallel a first semantic token sequence corresponding to the first speaker and a second semantic token sequence corresponding to the second speaker. The semantic token sequence segmentation module is used to segment the first semantic token sequence and the second semantic token sequence according to a first preset length, respectively, to obtain a first token block set and a second token block set; The Mel spectrum generation module is used to perform spectrum reconstruction on the first speech feature vector and the first token block set based on a preset conditional flow matching model to obtain the first Mel spectrum, and to perform spectrum reconstruction on the second speech feature vector and the second token block set in parallel to obtain the second Mel spectrum. The target dialogue audio acquisition module is used to convert the first Mel spectrum and the second Mel spectrum respectively to obtain a first waveform and a second waveform, and to align and merge the first waveform and the second waveform to obtain the target dialogue audio.

9. A computer device, characterized in that, The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and, in executing the computer program, implement the speech generation method based on dual-channel semantic tokens and block conditional stream matching as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the speech generation method based on dual-channel semantic tokens and block conditional stream matching as described in any one of claims 1 to 7.

Images